Java Keystore PrivateKeyEntry vs trustedCertEntry
I eventually figured out that I have to supply the private key as well (As Roshith mentioned in the link he supplied).
So I started with first creating a pfx file:
openssl pkcs12 -export -out newcertbundle.pfx -inkey myprivate.key -in newcertfile.crt
And then converted it to jks:
keytool -importkeystore -srckeystore newcertbundle.pfx -srcstoretype PKCS12 -srcstorepass mypass -deststorepass mypass -destkeypass mypass -destkeystore newkeystore.jks
The only thing I couldn't figure out (but wasn't too important to me) was how to use an alias, so I went with a default one (when I tried specifying one I got: Alias does not exist. This is discussed here).