Image upload storage strategies

I've answered a similar question before but I can't find it, maybe the OP deleted his question...

Anyway, Adams solution seems to be the best so far, yet it isn't bulletproof since images/c/cf/ (or any other dir/subdir pair) could still contain up to 16^30 unique hashes and at least 3 times more files if we count image extensions, a lot more than any regular file system can handle.

AFAIK, SourceForge.net also uses this system for project repositories, for instance the "fatfree" project would be placed at projects/f/fa/fatfree/, however I believe they limit project names to 8 chars.

I would store the image hash in the database along with a DATE / DATETIME / TIMESTAMP field indicating when the image was uploaded / processed and then place the image in a structure like this:

images/  2010/                                      - Year    04/                                      - Month      19/                                    - Day        231c2ee287d639adda1cdb44c189ae93.png - Image Hash

Or:

images/  2010/                                    - Year    0419/                                  - Month & Day (12 * 31 = 372)      231c2ee287d639adda1cdb44c189ae93.png - Image Hash

Besides being more descriptive, this structure is enough to host hundreds of thousands (depending on your file system limits) of images per day for several thousand years, this is the way Wordpress and others do it, and I think they got it right on this one.

Duplicated images could be easily queried on the database and you'd just have to create symlinks.

Of course, if this is not enough for you, you can always add more subdirs (hours, minutes, ...).

Personally I wouldn't use user IDs unless you don't have that info available in your database, because:

1. Disclosure of usernames in the URL
2. Usernames are volatile (you may be able to rename folders, but still...)
3. A user can hypothetically upload a large number of images
4. Serves no purpose (?)

Regarding the CDN I don't see any reason this scheme (or any other) wouldn't work...

MediaWiki generates the MD5 sum of the name of the uploaded file, and uses the first two letters of the MD5 (say, "c" and "f" of the sum "cf1e66b77918167a6b6b972c12b1c00d") to create this directory structure:

images/c/cf/Whatever_filename.png

You could also use the image ID for a predictable upper limit on the number of files per directory. Maybe take floor(image unique ID / 1000) to determine the parent directory, for 1000 images per directory.

Yes, yes I know this is an ancient topic. But the problem to store large amount of images and how the underlying folder structure should be organized. So I present my way to handle it in the hope this might help some people.

The idea using md5 hash is the best way to handle massive image storage. Keeping in mind that different values might have the same hash I strongly suggest to add also the user id or nicname to the path to make it unique. Yep that's all what's needed. If someone has different users with the same database id - well, there is something wrong ;) So root_path/md5_hash/user_id is everything you need to do it properly.

Using DATE / DATETIME / TIMESTAMP is not the optimal solution by the way IMO. You end up with big clusters of image folders on a buisy day and nearly empty ones on less frequented ones. Not sure this leads to performance problems but there is something like data aesthetics and a consistent data distribution is always superior.

So I clearly go for the hash solution.

I wrote the following function to make it easy to generate such hash based storage paths. Feel free to use it if you like it.

/*** Generates directory path using $user_id md5 hash for massive image storing * @author Hexodus * @param string $user_id numeric user id* @param string $user_root_raw root directory string* @return null|string*/function getUserImagePath($user_id = null, $user_root_raw = "images/users", $padding_length = 16,                             $split_length = 3, $hash_length = 12, $hide_leftover = true){    // our db user_id should be nummeric    if (!is_numeric($user_id))        return null;    // clean trailing slashes      $user_root_rtrim = rtrim( $user_root_raw, '/\\' );    $user_root_ltrim = ltrim( $user_root_rtrim, '/\\' );    $user_root = $user_root_ltrim;    $user_id_padded = str_pad($user_id, $padding_length, "0", STR_PAD_LEFT); //pad it with zeros      $user_hash = md5($user_id); // build md5 hash    $user_hash_partial = $hash_length >=1 && $hash_length < 32                         ? substr($user_hash, 0, $hash_length) : $user_hash;    $user_hash_leftover = $user_hash_partial <= 32 ? substr($user_hash, $hash_length, 32) : null;    $user_hash_splitted = str_split($user_hash_partial, $split_length); //split in chunks    $user_hash_imploded = implode($user_hash_splitted,"/"); //glue aray chunks with slashes    if ($hide_leftover || !$user_hash_leftover)        $user_image_path = "{$user_root}/{$user_hash_imploded}/{$user_id_padded}"; //build final path    else        $user_image_path = "{$user_root}/{$user_hash_imploded}/{$user_hash_leftover}/{$user_id_padded}"; //build final path plus leftover    return $user_image_path;}

Function test calls:

$user_id = "1394";$user_root = "images/users"; $user_hash = md5($user_id);$path_sample_basic = getUserImagePath($user_id);$path_sample_advanced = getUserImagePath($user_id, "images/users", 8, 4, 12, false);echo "<pre>hash: {$user_hash}</pre>";echo "<pre>basic:<br>{$path_sample_basic}</pre>";echo "<pre>customized:<br>{$path_sample_advanced}</pre>";echo "<br><br>";

The resulting output - colorized for your convenience ;):