Does it make sense to use BOTH mongodb and mysql in the same rails application?

That's what we do with CouchDB and PostgreSQL.

All our users and groups kinds of things are in the postgresql database.
Anything else (in our case, some records with statistical datas) are in the couchdb database.

In our case, it allows us to have one couchdb database per client (the app connects itself to one or an other depending of the user's host).
And only one postgresql database having all the users in it.

So yes I think it's a good idea to have a SQL and a NOSQL database in the same application.

There's no reason you can't keep your users table in MongoDB. Whether you decide to keep a transactions table in MongoDB is another question. If you need multi-object transactions, then you have to use a relational database that supports transactions. If you don't need this style of transaction, then using MongoDB for this can still be a good idea.

Keep in mind that if you use MongoDB, we recommend running with replication for failover.

The overall thing to keep in mind is that if you're using a relational database because of the consistency guarantees, you still need to make sure that your hardware is configured to take advantage of those features. See the cautionary note here:

http://dev.mysql.com/doc/refman/4.1/en/innodb-configuration.html

If you're not taking these precautions, then there's not a huge difference between MongoDB's durability and that of a relational database.

Think also about durability: http://blog.mongodb.org/post/381927266/what-about-durability, for example when you lose power (electricity).