Business data querying/reporting in service oriented architecture Business data querying/reporting in service oriented architecture database database

Business data querying/reporting in service oriented architecture


The solution is to aggregate data from different services into a central reporting database - this is feasible if the data collected is versioned by time - i.e you can go to the reporting data and get a point-in-time data which is correct (for that time)

Getting that data to the service can be via events published by the various service or periodic imports, "log" aggregation or combinations of them.

I call this pattern aggregated reporting

Note that in addition to that you still to get data from individual services for things that needs to be up-to-date as an aggregation solution has inherent delay (reduced freshness)

Edit:Considering the edits you've made and the comments you've made (ad-hoc queries) I'd say you need to treat this as a journey , that is, you want to get to option 4 so start by pulling data from sources you have to answer you current ad-hoc needs, convert to messages as you move forward with development and add more sources.

Also you may want to think about the difference between services (that don't share internal data structures between them and have strict boundaries) and aspects (semi-independent parts of service that can use the same source)

PS I also wrote that InfoQ piece on BI & SOA Tom mentioned in the comments - which essentially talks about this idea - this article is from 2007, i.e. I've successfully applied it this for more than a decade now (different technologies , moving from schema on write to schema on read etc. but same principles)


So, I'm not sure this would answer your needs - but i'll describe our overall approach to BI:

  1. Everything in our system generates an event: actions in the backend, actions in the mobile apps - everything we want to track produces event with the relevant data (ids, time, name etc).
  2. All the events are sent to some common funnel for collection - its a backend app that takes events - makes sure they're valid - and stores them.
  3. You can store the events in some no-sql storage (like Elasticsearch) or on a cloud (like google's BigQuery).
  4. Once they're in, its just a matter of querying and cross-referencing to get the overall picture you want. That's what our BI people do: they generate a picture from the heaps of events we collect.