C: How to mount a regular filesystem created with mkfs using mount(2)?

The problem is that the mount library call does not set up a loop device. In order to mount your container, you would have needed to pass -o loop to the mount command, which effectively includes what losetup does (i.e. makes /dev/loopN point to your file and mounts that). When using the library call, you need to simulate that, i.e. you need to set up a loop device first.

To determine exactly what you want need to do, try:

strace mount -t ext4 -o loop container /mnt

and have a look at the system calls. You can see that what's happening is that first it makes a loop device (e.g. /dev/loop0 using the system call equivalent of losetup), then it mounts that loop device.

Here are the key lines from the strace output:

First allocate the loop device:

stat("/dev/loop", 0x7fffe90b00b0)       = -1 ENOENT (No such file or directory)open("/dev/loop0", O_RDONLY)            = 3fstat(3, {st_mode=S_IFBLK|0660, st_rdev=makedev(7, 0), ...}) = 0ioctl(3, LOOP_GET_STATUS, {number=39736224, offset=0x6f726763, encrypt_type=0x732f7075 /* LO_CRYPT_??? */, encrypt_key_size=1702130553, flags=LO_FLAGS_READ_ONLY|LO_FLAGS_AUTOCLEAR|LO_FLAGS_PARTSCAN|0x63206460, name="group rw,noexec,nosuid,nodev,none,name=systemd 0 0", encrypt_key="s 0 0\0\0e-agent.devices 0 0\0\0\320\177\0\0", ...}) = -1 ENXIO (No such device or address)close(3)                                = 0open("/home/amb/so/container", O_RDWR)  = 3open("/dev/loop0", O_RDWR)              = 4readlink("/home", 0x7fffe90adfa0, 4096) = -1 EINVAL (Invalid argument)readlink("/home/amb", 0x7fffe90adfa0, 4096) = -1 EINVAL (Invalid argument)readlink("/home/amb/so", 0x7fffe90adfa0, 4096) = -1 EINVAL (Invalid argument)readlink("/home/amb/so/container", 0x7fffe90adfa0, 4096) = -1 EINVAL (Invalid argument)ioctl(4, LOOP_SET_FD, 0x3)              = 0close(3)                                = 0ioctl(4, LOOP_SET_STATUS64, {offset=0, number=0, flags=LO_FLAGS_AUTOCLEAR, file_name="/home/amb/so/container", ...}) = 0ioctl(4, LOOP_GET_STATUS64, {offset=0, number=0, flags=LO_FLAGS_AUTOCLEAR, file_name="/home/amb/so/container", ...}) = 0

and only then do a mount:

mount("/dev/loop0", "/mnt", "ext4", MS_MGC_VAL, NULL) = 0

I would however advise you to shell out to mount unless you have a good reason not to. This ensures /etc/mtab etc. is maintained.

As a followup to abligh's answer, the code for setting up a loop device is

int device = open(device_path, O_RDWR);/* omitting error checking on device for clarity */char buf[20];int success = 0;for (i = 0; i < 999999999) {    sprintf(buf, "/dev/loop%d", i);    int h = open(buf, O_RDWR);    if (h > -1) {        int r= ioctl(h, LOOP_SET_FD, device);        close(h);        if (!r) {           success = 1;           break;        }    } else if (errno == ENOENT)        break;}if (!success) {     /* no more free loop devices or you're not root */}