Save Followers count in a field or query each time if needed [closed] Save Followers count in a field or query each time if needed [closed] mongoose mongoose

Save Followers count in a field or query each time if needed [closed]


Well, if you want to know what is right, there is only one answer.

  1. Each of the separate fields in the user record contains derived data (data that can be easily derived via a query). Therefore it constitutes a duplication. Therefore it fails Normalisation.

  2. The consequence of failed Normalisation is, you have an Update Anomaly. You no longer have One Fact in One Place, you have one fact in two places. And you have to update them every time one fact changes, every time the Followers/Followed per User changes. Within a Transaction.

  3. That isn't a "trade-off" against performance concerns, that is a crime. When the fact in two places gets "out of synch", your crimes will be exposed. You will have to re-visit the app and database and perform some hard labour to make amends. And you may have to do that several times. Until you remove the causative problem.

Performance

As for the load on the database, if your application is serious, and you expect to be in business next year, get a real SQL platform.

Population or load for this requirement is simply not an issue on a commercial platform. You always get what you pay for, so pay something of value, and get something of value.

Note that if you have millions of Users, that does not mean you have millions of Followers per User. Note that your files will be indexed, so you will not chase down 16 million Users to count 25 Followers, your index will allow you to identify 25 Followers in a maximum of 25 index rows, in very few pages. This kind of concern simply does not exist on a commercial platform, it is the concern of people with no platform.


Well, it depends who is it for?

If it's for your users - they can see how many followers they are having. I would do this Twitter API call only when user logs in to your service.

If for some reason it must be done for all users. I think best way would be to do this followers-count-api-call for example once in an hour, every second hour or just daily. This could be achieved by a script that runs in cron.

Do you really need followers or just followers count? Or both?

If both, you can request Twitter user's followers and limit it to 100 (if your cron runs every minute to every fifteen minutes). Then loop those follower ids against your database and keep inserting them, until there is match. Twitter returns all the newest follower id:s by default. So this is possible at this moment.

Just remember you can make only 15 request per user tokens agains Twitter API when requesting Followers. This limit could vary between different endpoints.

Good to mention that I assumed that you are getting only follower ids. Those you can get 5000 at a time. If you want to request follower objects, there the limit is only 200 per request.

Hope this helps :D