Strategies for dealing with concurrency issues caused by stale Domain Objects (Grails/GORM/Hibernate)

Here is my own attempt to answer this question.

(ADDED 9/24/2014 There is simply no good solution to this problem. Sadly, HHH-9367 JIRA ticket has been rejected by Hibernate as 'not a bug'. The only solution suggested in that ticket was to use refresh (I assume that would require changing all queries to something that looks like:

BankAccount.findAllBy...(...).each{ it.refresh() }

Personally, I do not agree that this is a meaningful solution.)

As I have explained above, if Hibernate/GORM query returns a set of DomainObjects and some of theseobjects are already in hibernate session (populated by previous queries) the query will return these old objects, and these objects will not be automatically refreshed. This can cause some hard to spot concurrency issues. I call it Repeatable Finder problem.

This has nothing to do with 2nd level cache. This problem is caused by how hibernate works even without the 2nd level cache configured. (EDITED 9/24/2014: And, this is not any ORM, any DB application issue, the issue is specific to use of Hibernate).

Implications to your application:

(I can only explain the impacts I know of, I am not claiming that these are the only impacts).

Domain objects typically have a set of associated constraints/logical rules that need to hold across typically several records and are enforced by either the application or the database itself. I will borrow a term from FP and testing and will call these 'properties'.

Example properties:In the above BankAccount example, name uniqueness (enforced by DB) is a property (for example, you may use it in defining equals() method), if money is transferred between accounts, the total amount of money in these accounts needs to be a constant - this is a property.
If I modify my BankAccount class and add 'branch' association to it:

BankBranch branch

Then this is a property as well:

assert BankAccount.findAllByBranch(b).every{it.branch == b}.

(EDITED, This property should technically be enforced by DB and implementation of the finder method and developer may assume it is 'safe' and not breakable. In fact most 'where' criteria and 'joins' used by your app somewhere underneath hibernate define properties of similar nature.).

Repeatable finder problem can cause most of the properties to break under concurrent use (scary stuff!). For example I reiterate here a piece of code I wrote inthe relevant JIRA ticket linked in the question:

... a1 has branch b1BankAccount.findByName('a1')... concurrently a1 is moved to branch b2//fails because stale a1.branch == b1assert BankAccount.findAllByBranch(b2).every{it.branch == b2} 

Your application probably uses explicit and implicit properties and may have logic to enforce them.For example, application may rely on names being unique and will exception or return wrong results if they are not unique (maybe name on its own is used to define equals()). This is explicit usage.Application may provide list views and it will be very embarrassing if list shows property violated (list for accounts under branch b2 shows some accounts with branch b1 - this is implicit usage). Any of such cases will be affected by "repeatable finder'.

If Grails code (not DB constraint) is used to enforce a property, then in addition to 'repeatable finder' more obvious concurrency concerns need to be addressed. (I am not discussing these here.)

Finding problems:

(This pertains to broken properties only. I do not know if repeatable finder causes other issues.)

So, I think the first step is to identify all properties in the app (EDITED: there will be many of them, potentially too many to examine - so, focusing on domain objects that are likely to change concurrently maybe the key here.), the second step is to identify where and how the application (implicitly or explicitly) uses these properties and how are they enforced. Code for each of these needs to be examined to verify that repeatable finder is not the issue.

It maybe a good idea, to simply enable SQL tracing (as well as tracing of where each HTTP request starts and ends) and examine log traces from identified areas of concern for any table name in the 'from' part of SQL. If such table shows up more than onceper request this may be a good indication of a problem. Good Functional Test coverage could help in generating such log files.

This is obviously a not trivial process and there are no bullet proof solutions here.

Fixing problems:

Using discard() on objects from previous queries or running the query which relies on certain application property/properties in a separate hibernate session should solve the problem. Using new session approach should be more bullet proof. I do not recommend using refresh() here.(Note, hibernate provides no public API to query for objects attached to its session.)
Using new session will expose the application to some new problems like LazyInitalizationException or DupicateKeyException. These are trivial by comparison.

SIDE NOTE: I personally consider framework design decision which causes code to break when additional query is added: a terrible design flaw.

It is interesting to compare Hibernate against Active Record (which I know much less about). Hibernate took the ORM purist approach of trying to make RDBMS into OO, Active Record took the 'share nothing' approach of staying closer to DB and having DB deal with more complex concurrency issues.
Sure, in Active Record node.children.first().parent != parent but is that such a bad thing?
I admit to not understanding the reasons behind hibernate decision to not refresh objects in its cache when the new query is executed.Have they been concerned about side effects? Can Hibernate and Grails be lobbied to change that? Because that seems to be the BEST long term solution. (Edited 9/24/2014: my efforts to have Hibernate resolve the issue have failed.)

ADDED (2014/08/12):It could also helpful to rethink design of your Grails app and use GORM/Hibernate only as a very thin persistence layer. Designing such layer with a tight control over what queries are issued during each request should minimize this problem. This is obviously not what Grails framework advocates, (EDITED 9/24/2014 and it will only reduce not eliminate the problem.)

After lot of thinking about it, I seems to me that this maybe a major logical hole in Grails/Hibernate technology stack. There is really no good solution if you care about concurrency, you should be concerned.

Repeatable reads are a way of preventing lost updates in a database transaction. Most applications employ a read-modify-write data access pattern breaking database transaction boundaries and pushing transactions to the application-layer.

Hibernate employs a transactional write-behind policy, so entity state transitions are delayed as much as possible, to reduce database locking associated to DML statements.

In an application-level transaction, the first level cache acts as a application-level repeatable read mechanism. But while database locking ensures repeatable read consistency when using physical transactions, for application-level transactions you need an application-level locking mechanism. That's why you should always use optimistic locking in the first place.

Optimistic locking allows others to modify your previously loaded data while preventing you from updating stale data.

It's not the equals that's broken. The database constraints should always enforce unique business keys anyway.

For operations concerning account updates you should either use a single database transaction that ensures safety through lock acquisitions (SELECT FOR UPDATE) or use optimistic locking, so when others update your data you will get a stale entity exception.

I could replicate your use case. The entity is reused from the 1st-level cache. For SQL queries you have the freedom of loading concurrent changes. As long as you load entities for updating them later you should be fine, because the optimistic locking mechanism will prevent you from saving stale data.

If you use HQL/JPQL just for viewing then you might want to use projections instead.

A good article from Marc Palmer about these problems. I found it Very interesting. At the end of the article he gives somes "solutions" that could fit the needs of some of you.

The false optimism of GORM and Hibernate