Sorting a list of RGB triplets into a spectrum

You're trying to convert a three-dimensional space into a one-dimensional space. There's no guarantee that you can make a pleasing rainbow out of it, as Oli says.

What you can do is "bucket" the colors into a few different categories based on saturation and value/lightness, and then sort within the categories, to get several independent gradients. For example, high-saturation colors first for the classic rainbow, then mid-saturation high-value colors (pastels), then low-saturation (grays).

Alternately, if all you care about is the rainbow, convert to hsl, then slam saturation to 1.0 and value to 0.5, convert back to rgb and render that instead of the original color.

Presumably you are sorting by hue (i.e. H)? That will give a nice result if S and L (or V) are constant, but if they are varying independently, then you will get a bit of a mess!

An interesting method for reducing dimensionality of color spaces uses the space-filling Hilbert curve. Two relevant articles are:

• Color Space Dimension Reduction - overview of several methods for reducing dimensionality of color data
• Portrait of the Hilbert Curve - detailed article about Hilbert curves and application to color-space dimensionality reduction

They both consider 3d -> 2d reduction, but the intermediate step of mapping to the 1d curve could be a solution to your problem.