Bad JWT signature trying to use the Google Drive API
I was having the same issue for a while until I came across using the OWIN libraries to do it.
Here's the link to my Get-GSToken function that's part of my PSGSuite module. You can tear this apart if you'd like to do what you need (the purpose of that function is to grab a token), but you'll need the nuget folder as well in order for it to work (OWIN libraries are contained in there):
https://github.com/nferrell/PSGSuite/blob/master/Public/Get-GSToken.ps1
Feel free to grab the entire module for samples! I've wrapped almost all of the Google API calls into Powershell functions in there, all leveraging Get-GSToken to first grab a token at the scope the function needs.
Something to note: this leverages a service account and a P12 key file, not the clientsecrets.json file key type. I'm not sure how to build a working JWT with pure Powershell using the JSON file, but the P12 + OWIN libraries work amazingly well for it.