How can I build a small operating system on an old desktop computer? [closed]
First things first. Read, read, read, read, read. You need to have a firm understanding of how the OS works before you can hope to implement your own.
Grab one of Andrew Tanenbaum's books on operating systems. This is the one we used in my OS class in college:
Modern Operating Systems on Amazon
Despite the ridiculous cover, it's a fantastic read, especially for a textbook. Tanenbaum is really an expert in this area and his explanations of how the OS works underneath the hood are clear and easy to understand. This book is mostly theory, but I believe he also has a book that discusses more of the implementation. I've never read it, though, so I can't comment on it.
That should help you bone up on process management, memory management, filesystems, and everything else your OS kernel needs to do to get it up to a bootable state. From that point on it's basically a matter of writing device drivers for the hardware you need to support, and offering implementations of the C library functions to make kernel calls for things like opening files and devices, reading and writing, passing messages between processes, etc.
Read up on x86 assembly (assuming you are designing this for an x86 machine). That should answer a lot of your questions with regards to moving between processor operating modes.
If you've got any electronics knowledge, it may be easier to start with writing an operating system for an embedded device that has ample documentation, because it will generally be simpler than an x86 PC. I've always wanted to write my own OS as well, and I'm starting with writing a microkernel embedded OS for This Development Board from Digilent. It can run the soft-core MicroBlaze processor from Xilinx, which has very thorough documentation. It's also got some RAM, flash data storage, LEDs, switches, buttons, VGA output, etc. Plenty of stuff to play around with writing simple drivers for.
One of the benefits of an embedded device is also that you may be able to avoid writing a VGA driver for a long time. In my case, the Digilent development board has an onboard UART, so I can effectively use the serial output as my console to get the whole thing up and booting to a command line with minimal fuss.
Just make sure that whatever you choose to target has a readily available and well-tested compiler for it. You do not want to be writing an OS and a compiler at the same time.
http://www.osdev.org/andhttp://www.osdever.net/
welcome to the OS Development world.
See also other x86 documentation links in SO's x86 tag wiki: Intel and AMD manuals, compiler / assembler docs, and various guides.
It also suggests using BOCHS or other virtual environment for debugging, since you can single-step your bootloader and examine registers.
I would suggest working, at least at first, on Bochs or some other virtual machine the reason being that you can take it with you wherever you want, it's easier to debug (you can see the exact state of the hardware), and if you need outside help debugging they can use the exact same 'hardware' as you.
The most useful advice I have is to get yourself into a state where you can be running C code as quickly as possible -- i.e. boot up, setup your descriptor tables, and get yourself to a point where it's safe to run compiled C. Most if not all the kernel should be in C if you want to stay sane and keep working on it. Assembly, while required in some places, is tedious and tends to be hard to debug.