SSL handshake alert: unrecognized_name error since upgrade to Java 1.7.0

Java 7 introduced SNI support which is enabled by default. I have found out that certain misconfigured servers send an "Unrecognized Name" warning in the SSL handshake which is ignored by most clients... except for Java. As @Bob Kerns mentioned, the Oracle engineers refuse to "fix" this bug/feature.

As workaround, they suggest to set the jsse.enableSNIExtension property. To allow your programs to work without re-compiling, run your app as:

java -Djsse.enableSNIExtension=false yourClass

The property can also be set in the Java code, but it must be set before any SSL actions. Once the SSL library has loaded, you can change the property, but it won't have any effect on the SNI status. To disable SNI on runtime (with the aforementioned limitations), use:

System.setProperty("jsse.enableSNIExtension", "false");

The disadvantage of setting this flag is that SNI is disabled everywhere in the application. In order to make use of SNI and still support misconfigured servers:

1. Create a SSLSocket with the host name you want to connect to. Let's name this sslsock.
2. Try to run sslsock.startHandshake(). This will block until it is done or throw an exception on error. Whenever an error occurred in startHandshake(), get the exception message. If it equals to handshake alert: unrecognized_name, then you have found a misconfigured server.
3. When you have received the unrecognized_name warning (fatal in Java), retry opening a SSLSocket, but this time without a host name. This effectively disables SNI (after all, the SNI extension is about adding a host name to the ClientHello message).

For the Webscarab SSL proxy, this commit implements the fall-back setup.

I had what I believe the same issue is.I found that I needed to adjust the Apache configuration to include a ServerName or ServerAlias for the host.

This code failed:

public class a {   public static void main(String [] a) throws Exception {      java.net.URLConnection c = new java.net.URL("https://mydomain.com/").openConnection();      c.setDoOutput(true);      c.getOutputStream();   }}

And this code worked:

public class a {   public static void main(String [] a) throws Exception {      java.net.URLConnection c = new java.net.URL("https://google.com/").openConnection();      c.setDoOutput(true);      c.getOutputStream();   }}

Wireshark revealed that during the TSL/SSL Hello the warningAlert (Level: Warning, Description: Unrecognized Name), Server HelloWas being sent from the server to the client.It was only a warning, however, Java 7.1 then responded immediately back with a "Fatal, Description: Unexpected Message", which I assume means the Java SSL libraries don't like to see the warning of unrecognized name.

From the Wiki on Transport Layer Security (TLS):

112 Unrecognized name warning TLS only; client's Server Name Indicator specified a hostname not supported by the server

This led me to look at my Apache config files and I found that if I added a ServerName or ServerAlias for the name sent from the client/java side, it worked correctly without any errors.

<VirtualHost mydomain.com:443>  ServerName mydomain.com  ServerAlias www.mydomain.com

You can disable sending SNI records with the System property jsse.enableSNIExtension=false.

If you can change the code it helps to use SSLCocketFactory#createSocket() (with no host parameter or with a connected socket). In this case it will not send a server_name indication.