Redis: the best way to get all hash values

How about an unpside down change. Transpose the way you store the data.

Instead of having 50k hashes each with 5 values. Have 5 hashes each with 50k values.

For example your hash depends on eventid and you store new_code, old_code and other stuffs inside that hash

Now, for new_code have a hash map which will contain eventid as a member and it's value as value. So new_code alone is a hash map containing 50k member value pair.

So looping through 5 instead of 50k will be relatively quicker.

I have done a little experiment and following are the numbers

50k hashes * 5 elements Memory : ~12.5 MBTime to complete loop through of elements : ~1.8 seconds5 hashes * 50k elementsMemory : ~35 MBTime to complete loop through of elements : ~0.3 seconds.

I have tested with simple strings like KEY_i and VALUE_i (where i is the incrementer) so memory may increase in your case. And also I have just walked through the data, I haven't done any manipulations so time also will vary in your case.

As you can see this change can give you 5x performance boost up, and 2 times more memory.

Redis does compression for hashes within a range (512 - default). Since we are storing more than that range (50k) we have this spike in memory.

Basically it's a trade off and it's upto you to choose the best one that would suit for your application.

For your 1st question:

1. you are getting values of new_code in each hashes, now you haveeverything in a single hash -> just a single call.
2. Then you are updating old_code and new_code one by one. Now you can do them using hmset using a single call.

Hope this helps.

For your first problem, using a Lua script will definitely improve performance. This is untested, but something like:

update_hash = r.register_script("""    local key = KEYS[1]    local new_code = ARGS[1]    local old_code = redis.call("HGET", key, "new_code")    if old_code then        redis.call("HMSET", key, "old_code", old_code, "new_code", new_code)    else        redis.call("HSET", key, "new_code", new_code)    end""")# You can use transaction=False here if you don't need all the # hashes to be updated together as one atomic unit.pipe = r.pipeline()for availability in availabilities:    keys = [availability["EventId"]]    args = [availability["MsgCode"]]    update_hash(keys=keys, args=args, client=pipe)pipe.execute()

For your second problem you could again make it faster by writing a short Lua script. Instead of getting all the keys and returning them to the client, your script would get the keys and the data associated with them and return it in one call.

(Note, though, that calling keys() is inherently slow wherever you do it. And note that in either approach you're essentially pulling your entire Redis dataset into local memory, which might or might not become a problem.)

There is no command like that, redis hashes work within the hash, so HMGET work inside one hash and give all the fields in that hash. There is no way to access all the fields in multiple hashes at ones.

There are 2 options

1. Using pipeline
2. Using LUA

However both of this are workarounds, not a solution to your problem. To know how to do this check My Answer in this Question: Is there a command in Redis for HASH data structure similar to MGET?