How did Chrome create a different GUI for the browser?
I would advise against taking this approach. While Chrome does have an attractive GUI, breaking the design guidelines for your target platform is rarely (but sometimes) the path to creating an attractive, usable, intuitive user interface.
If you're looking to understand the process for developing a UI, then I would recommend that you explore the design guidelines for your target platform (Windows, it would appear, in your case).
Likely purely Windows API.
There is are functions in Win32 to override almost any aspect of Window behavior / and or look. For example, you can intercept the WM_NC_PAINT message to paint the "non-client area" (title bar and max/min buttons). You can do this on any app, in fact.
Chrome is open source. They have also published pretty comprehensive design documentation as well.
You should be able to have a look at exactly what they do!