'unshare' does not work as expected in C api

Despite your comment that

"sometimes" umount returns 0 "sometimes" -1, but in the end it does not unmount /proc at all

, in 10000 trials of the code from your pastebin, umount() always failed for me, returning -1 and not unmounting /proc. I am disinclined to believe that umount() ever returns 0 despite having failed to perform the requested unmounting, but if ever it does then that would constitute a bug in umount(). If you can in fact substantiate such a bug then the community-minded response would be to file a bug report against glibc.

The question then becomes why and how your bash script behaves differently. In fact, however, it does not seem to do.

In the first place, you have the wrong expectation of the unshare(1) command. Unlike the unshare(2) function, the unshare command does not affect the shell in which it is executed. Instead, it launches a separate process that has its own private copies of the specified namespaces. Normally you would specify the command to launch that process on the unshare command line, and in fact the program's manual page indicates that doing so is mandatory.

Empirically, I find that if I fail to specify such a command -- as you do -- then unshare launches a new shell as the target process. In particular, when I run your script (with sufficient privilege to use unshare), I immediately get a new prompt, but it is the prompt of the new shell, running in the foreground. This is immediately evident to me because the prompt is different (your prompt might not be any different under those circumstances, however). There is no error message, etc. from umount at that point because it has not yet run. If I manually attempt to umount proc in the (unshared) subshell, it fails with "device is busy" -- this is the analog of what your C program tries to do.

When I exit the subshell, the rest of the script runs, with both umounts and both mounts failing. That is to be expected, because the main script shares its mount namespace.

It is entirely plausible that /proc really is busy and therefore cannot be unmounted, even for a process that has a private copy of the mount namespace. It is likely that such a process is itself using its private copy of the mount of /proc. In contrast, I find that I can successfully unmount /dev/pts in a process with an unshared mount namespace, but not in a process that shares the system's copy of that namespace.

I found the problem checking the source code of unshare command. The /proc must be unmounted with MS_PRIVATE | MS_REC and mounted without them, this is essentially to ensure that the the mount has effect only in the current (the new) namespace. The second problem is it is not possible to umount /dev/pts without producing an effect on global namespace (this is caused by the internal routine of devpts driver). To have a private /dev/pts the only way is to mount it with the dedicated -o newinstance option. Finally /dev/ptmx should also be bind-remounted.

Therefore, this is the C working code as expected:

unshare(CLONE_NEWPID | CLONE_NEWNS );int pid = fork();if (pid != 0) {    int status;    waitpid(-1, &status, 0);    return status;}printf("New PID after unshare is %i", getpid());if (mount("none", "/proc", NULL, MS_PRIVATE|MS_REC, NULL)) {    printf("Cannot umount proc! errno=%i", errno);    exit(1);}if (mount("proc", "/proc", "proc", MS_NOSUID|MS_NOEXEC|MS_NODEV, NULL)) {    printf("Cannot mount proc! errno=%i", errno);    exit(1);}if (mount("devpts", "/dev/pts", "devpts", MS_MGC_VAL | MS_NOSUID | MS_NOEXEC, "newinstance") ) {    printf("Cannot mount pts! errno=%i", errno);    exit(1);}if (mount("/dev/pts/ptmx", "/dev/ptmx", NULL, MS_MGC_VAL | MS_NOSUID | MS_NOEXEC | MS_BIND, NULL) ) {    printf("Cannot mount ptmx! errno=%i", errno);    exit(1);}

I think the issue is with the system("mount") which is spawns a shell and doesn't carry over the umount. Try opening a file in /proc/ after umount and see it works as expected.

See this -

unshare(CLONE_NEWPID | CLONE_NEWNS );int rc = 0;int pid = fork();if (pid != 0) {        int status;        waitpid(-1, &status, 0);        return status;}printf(">>> My pid: %d\n", getpid()); // It prints 1 as expectedrc = umount2("/proc", MNT_FORCE); // Returns 0printf(">>> umount returned %d. errno = %d, desc = (%s)\n", rc, errno, strerror(errno));rc = open("/proc/cpuinfo", O_RDONLY);printf(">>> open returned %d. errno = %d, desc = (%s)\n", rc, errno, strerror(errno));