how can I connect to a remote mongo server from Mac OS terminal
You are probably connecting fine but don't have sufficient privileges to run show dbs
.
You don't need to run the db.auth if you pass the auth in the command line:
mongo somewhere.mongolayer.com:10011/my_database -u username -p password
Once you connect are you able to see collections?
> show collections
If so all is well and you just don't have admin privileges to the database and can't run the show dbs
With Mongo 3.2 and higher just use your connection string as is:
mongo mongodb://username:password@somewhere.mongolayer.com:10011/my_database