Creating a UNIX shell

system(3) does indeed use (possibly directly or indirectly via exec) a shell to do its work. exec(3) and friends, however, do not use a shell, but rather execute the specified program image directly. You can see this simply by reading their respective man pages.

One difference is that with system(), you will see sugar like wildcards being expanded, whereas if you pass * as an argument to your program using exec(), your program will see the literal asterisk (and probably not know what to do).

A shell can be implemented using exec() among other things. It gets its stdin and stdout from something called the TTY (teletype, or old-school terminal) or PTY (pseudo-terminal, as in modern systems). See posix_openpt(2).

There are several functions in the exec family: execve(2), execl(3), execle(3), execlp(3), execv(3), execvp(3). The first one, execve(2) is provided by the operating system kernel as a system call. (Well, okay, the function that programs call is provided by the system C library, but it is a simple wrapper around the system call.) The other functions provide slightly different semantics and are implemented in terms of the execve(2) function.

The shells could use execvp(3) or execlp(3) to provide the PATH search for executables, but at least bash(1) hashes the full pathname of executables to provide a performance benefit. (See bash(1) built-in hash for details.)

system(3) is implemented via /bin/sh -c, as you've surmised.

The standard input and output is set up by whichever program spawned the shell. If a user logs in on the console directly, it'll be handled by agetty(8) or mgetty(8) or whichever getty-alike program handles direct logins. If a user logs in via sshd(8), then sshd(8) is in charge of creating the pty and delegating the terminal slave to the shell. If a user creates their shells via xterm(1) or other terminal emulators, then those processes will be responsible for hooking up the standard input, output, and error for the shell.