Android view animation and hardware layers
The delay happens when there's no layer in the cache, you should not see this delay for subsequent calls to setLayerType(NONE)/setLayerType(HARDWARE). You could call buildLayer() from onSizeChanged() to force a layer to be built and then put in the cache (call setLayerType(NONE) to move the layer to the cache.)
Note that the delay you are seeing depends greatly on the device you are running on.
The reason why you shouldn't keep layers enabled is that it doubles the amount of drawing work every time the view update. For instance, if you move a ListView into a layer and then scroll the list, each frame update during the scroll animation will cause: (a) the list to repaint into the layer (b) the layer to be drawn on screen. It's extremely wasteful and may cause performance issues depending on the complexity of your UI.