MongoDB and Redis as cache layer architecture

It really depends on your needs, but here's a fairly common one:

on_get_request  if data_in_redis    serve_data_from _redis  else    get_data_from_mongo    set_data_in_redis    set_expire_in_redis    serve_data_from_memory

The data will be a bit stale at times, but that's ok for most use cases. It works well in combination with some cache invalidation when important data is written:

on_important_data  delete_invalid_redis_keys

But that all assumes low write, high read, and a stable set of queries.

What does your high load use case look like?

This is already implemented in reference architecture for MongoDB open source project called "Socialite" though it's in Java and not node.js so my answers are based on my experience stress and load-testing that code.

As you can see from its implementation of status feed, the feed has option fanoutOnWrite cache which will create a cache (limited size document) for active users, capping the number of most recent entries in the cache document (that number is configurable).

The key principles from that implementation is that content requirements are in fact different from timeline cache requirements, and the write to content database is first as that is the system of record for all content, then you update the cache (if it exists). This part can be done asynchronously, if desired. The update utilizes "capped arrays" aka update $slice functionality to atomically push a new value/content to the array and chop the oldest one off at the same time.

Don't create cache for a user if it doesn't already exist (if they never log in then you're wasting the effort). Optionally you can expire caches based on some TTL parameter.

When you go to read a cache for a user when they log in and it's not there, then fall back on "fanoutOnRead" (which is querying all content of users they follow) and then build their cache out of that result.

The Socialite project used MongoDB for all back end, but when benchmarking it we found that the timeline cache did not need to be replicated or persisted so its MongoDB servers were configured to be "in memory" only (no journal, no replication, no disk flushing) which is analogous to your Redis use. If you lose the cache, it will just get rebuilt from permanent content DB "on demand".

Idel approach is write back cache way.You can write mongodb first and then write to redis. It is most common way.

Another option is,You can write redis first and send async message by using redis (like Q) Some thread can consume the message and read it, write it to mongoDB.

First option is more easy to implement.Second option can support huge # of write transaction.As i know, mongodb lock problem is not solved yet (it has been fixed from global lock to db level lock)Second option can be considerable to reduce such a lock contention.