Using Google Calendar API returns 401 (unauthorized)
Here's the answer after quite a bit of research, I just found and used this guide for using google calendar api http://googleappsdeveloper.blogspot.com/2011/12/using-new-js-library-to-unlock-power-of.html
Then I still was confused since the api key wasn't working every I tried regenerating it a couple of times. Then I had to use the browser key without any referrers just as mentioned here: Getting Error 403: Access Not Configured. Please use Google Developers Console to activate the API for your project
and finally I got everything working.
Here's also a main article for understanding the OAuth2https://developers.google.com/accounts/docs/OAuth2
In order to use Google Calendar API, you not only need the client ID and client secret but in most cases (other than accessing public calendars) you also need to have your user authenticated. One way to achieve that is using Oauth2.
I recommend using the client library that you can download here: https://developers.google.com/google-apps/calendar/downloads
Also there is an example on how to use the library here: https://developers.google.com/api-client-library/javascript/samples/samples