Are libc and malloc part of the operating system?

This question is best asked with another question: what is an Operating System? Or if you prefer: where do you print the line between OS and standard libraries?

Technically, malloc is part of the standard C library. And since the Linux is mainly written in C, and that the same library also includes many system calls, not in the C language, it is reasonable to think that this library is part of the OS.

But, on the other hand, there are several implementations of the C library, and also, the GNU C library is available for others operating systems, such as Windows. And I'm sure that there are other languages out there that call the OS without using the standard C library. So, from that POV, it is not part of the OS.

But then, Linux is the kernel, the OS should be named GNU/Linux (citation needed). But again, there are Linux systems without GNU, such as Android...

The conclusion is: the term "Operating System" is not a technical one. If you want to be precise, use kernel or standard C library, etc.

Yes... and no. C malloc() is usually a sub-allocator to memory areas provided by OS calls. The OS manages all virtual memory - that is part of it's job.