AJAX calls to untrusted (self-signed) HTTPS fail silently
XMLHttpRequests (AJAX requests) are only permitted on same-origin servers. That means the scheme://host:port part of the target URL has to match that of the current document. According to the spec, you shouldn't even be allowed to make a request on the SSL URL from the non-SSL one.
The less hackish solution that I see is that you just force-redirect all users to the SSL site. That way they will be forced to see the certificate warning before any AJAX request can be made.
Note: The spec also says that in case of TLS handshake failure (which I assume this case falls under, in a way) it should throw a NETWORK_ERR (code 19) exception. You could try to catch the exception when initiating the AJAX request. Refer to the spec on error handling for more details.