Advanced queries in HBase
The query as described is better suited to a relational database. You can answer the query quickly, however, by precomputing the result. For example, you might have a table where the key is the number of classes in common, and the cells are individual students that have key-many classes in common.
You could use a variant on this to answer questions like "which students are in class X and class Y": use the classes as pieces of the key (in alphabetical ordering, or something at least consistent), and again, each column is a student.
This type of querying is not available through the 0.20.0 API. I'm not sure if there are any plans for it (I doubt it would appear anytime soon). You'll find some roadmap details on the HBase website that might answer that question.
You'll need to compute the answer in your own application (although I'd love to be proved wrong).