Fastest method for screen capturing on Linux

This is possible using VirtualGL in a server with hardware acceleration. Basically just configure the server appropiately, then either on the same machine or on a machine in the same network, run

export DISPLAY=<your xvfb display>vglrun <your_app>

This will have the following advantages:

1) your app will render using virtualGL, which will use the hardware

2) VirtualGL will display your 3D context inside the Xfvb display, which will only render the 2D widgets in CPU

3) configure Xvfb to render to a framebuffer

4) profit!

This is not possible with pure X11.

You can get what you want with compositing, but only on servers which support it (most modern ones do). This actually has nothing to do with window managers. A WM is just another client, albeit with some special abilities, but those are unrelated to compositing. You can use the same compositing API.

man xcomposite should get you started.

You can use xwd(1) to dump a X11 window (and xwud(1) to display it). ImageMagick and the Netpbm tools know how to deal with the XWD format. But these are not especially fast. And they will not reveal hidden portions of a window.