That's a normal behavior, if I've got your situation right. Since each column family represents a separate Store on RegionServer, accessing multiple stores takes more time.
You can limit your scan to specific column families, useaddFamily on your scan object.