How does the libc function isatty() work?

The general strategy for implementing isatty is to attempt a tty-specific ioctl operation on the file descriptor, and check for ENOTTY error result. Traditionally, TCGETS, which is the backend for the tcgetattr function, is used, but this is slightly dangerous since the ioctl number for it on Linux clashes with legacy OSS sound devices, and if the file descriptor actually refers to a certain type of MIDI device, it will make changes to the device. In musl libc, we use TIOCGWINSZ, the "get window size" operation, whose number was not inadvertently reused for any other types of devices and which reliably fails with ENOTTY for non-tty devices.

In theory, you could do it with fstat and check the st_rdev field for the device major number, but this would require a hard-coded list of all device majors which are ttys, and would have broken when new types were added (e.g. USB-serial/ACM devices, uartlite devices, etc.).

isatty(3) is a library function (you won't find anything about in the linux kernel), and is usually implemented by calling tcgetattr(3) and checking its return value.

For example, in the GNU C library (glibc):

/* Return 1 if FD is a terminal, 0 if not.  */int__isatty (int fd){  struct termios term;  return __tcgetattr (fd, &term) == 0;}

tcgetattr(3) itself will resolve to some ioctl like TCGETA or TCGETS.

Notice that isatty(3) will also return true for a master side of a pseudo-tty, which isn't really a tty -- most tty related ops performed on it will actually apply to its slave side.

On linux, isatty(3) will also return true for /dev/console, which again, isn't a real tty (it cannot be made the controlling tty of a process).

On linux, you can obtain a list of all the tty drivers on your system with their major and minor numbers via cat /proc/tty/drivers. Of course, that only reflects the modules which have been loaded.