Android Graphics Internals
There are two core pieces to Android graphics: SurfaceFlinger and Skia. SurfaceFlinger is Android's compositor, used by the window manager to create and display windows (actually called surfaces.) SurfaceFlinger is implemented on top of OpenGL ES 1.x currently and can also use other hardware acceleration techniques when available (MDP, a 2D blitter on the T-Mobile G1, or hardware overlays on the Xoom.)
Each application renders into its windows (or surfaces) using primarily Skia. Skia is Android's 2D graphics library. You can also use OpenGL ES 1.x and 2.0 to render into a surface.
Android doesn't use DirectFB or X11 or any other existing Linux solution.
There is a brief introduction presentation about Android Graphics: http://www.slideshare.net/jserv/design-and-concepts-of-android-graphics
Romain Guy was definitely right. But thing changed since Android 3.0. Skia is not that important anymore , most 2D drawing are accelerated using openGL, a.k.a HWUI component.