The problem
Microsoft software (biztalk, wfetch, IE,…) all have a problem when performing a HTTPS TLSv1 handshake to Webmethods 8.0.1 running on a JVM 1.6 server. The problem manifests itself as an error on the client side (biztalk):“The underlying connection was closed: An unexpected error occurred on a send.”
Note: IE will actually catch this error and retry using SSLv3 so to the end-user it seems that IE works but it actually exhibits the same behavior.
Server specifications:
- Webmethods: windows 2003, JSE 1.664 bit, v8.0.1
- Biztalk: windows 2008
Recognizing the error
The first step is recognizing the error on our system so we can identify it when trying to reproduce it.Webmethods
First step is setting all the logging to “TRACE” for thewebmethodsserver logger. The server log/error log does not show any indication that the failed connection took place.JVM
The JVM offers a few system attributes to enable security debugging:- java.security.debug
- javax.net.debug
Setting “javax.net.debug=all”(or ssl)in the server.bat actually results in no additional output whatsoever.This may point to a custom webmethods implementation for HTTPS.
Wireshark
We installed wireshark on the server to capture the actual traffic going back and forth.We needed some additional configuration to actually make heads and tails from the encrypted traffic (source:http://support.citrix.com/article/CTX116557)- Go to edit > preferences > protocols > SSL
- Add <ip-of-webmethods-server>,<https-port-of-webmethods-server>,TCP,c:/path/to/private-key.pem
- E.g. 10.0.0.1,443,TCP,c:\my-private-key.pem
- Use “;” to separate multiple entires and make sure there are no spaces after each comma
- Or in newer versions of wireshark:
- Fill in an SSL debug file (e.g.c:\ssl-debug.log)
- Restart wireshark and the debug file you configured should show something like:
- ssl_init found host entry<configured-entry>
- ssl_init addr‘<ip>' port '<port>' filename '<name>' password(only for p12 file) '(null)'
- Private key imported: KeyID<id>..
- ssl_init private key file<path>successfully loaded
- If the key could not be imported (wrong format or the likes) you will see an error in the log file
$ openssl rsa -in private-key.der -out private-key.pem -outform PEM–inperform DER
Now that we can capture the HTTPS traffic,we add a filter to only see the traffic from the client:
ip.src == <client-ip> || ip.dst == <client-ip>
You can also add a port filter if necessary:
tcp.port == <local-port>
You can see the followingtrafficin wiresharkwhen the client attempts to connect:
Reproducing the error
Now that we can recognize the error on our system, we need to reproduce the error so we can narrow down the problem.Firefox
Firefox 9.0.1 will perform the entire conversation in SSLv3 which presents no problem to the server:
Internet Explorer
Internet explorer would seem to work from the client perspective as the HTTPS connection issuccessful;however wireshark tells us a slightly different story:
As you can see, it actually tries a TLSv1 connection which fails in the same way as the BizTalk one did, but then simply retries using SSLv3 which is successful.
WFetch
Microsoft provides a tool to test HTTPS connections: WFetch. If we tell it to connect to the server using TLSv1 (labeled TLS 3.1 because SSL 3.1 == TLS 1.0)we get an error as you can see:
Wireshark shows exactly the same pattern as the BizTalk failed connection.
If you toggle the connect option to “https” it will actually work on windows XP because the initial handshake uses SSLv2:
If you toggle it to “https” on windows 2008(and presumably vista/7)it will try the initial handshake with TLSv1 and fail(note that SSLv2 is disabledin windows 2008 due to security concerns).
OpenSSL
With OpenSSL you can also emulate a client (using the s_client option) to test a server:$openssl s_client -verify 6 -state -msg -showcerts -connect 10.32.54.5:4483 > ssl-output.txt
The output in “ssl-output.txt” shows us that this command uses SSL 2.0 for the initial handshakeand thenproceeds in TLSv1:
However you can force openssl to use tls1:
$openssl s_client -verify 6 -state -msg -showcerts-tls1-connect 10.32.54.5:4483 > ssl-output.txt
Surprisingly, the TLSv1 handshake works for OpenSSL:
The server shows us this:
Java Client
The following code will allow you tomake a rather low level SSL socket connection in java where you can choose the supported cypher suites and protocols:publicstaticvoidmain(String...args)throwsUnknownHostException, IOException {
SSLSocketFactory factory = (SSLSocketFactory) SSLSocketFactory.getDefault();
SSLSocket socket = (SSLSocket) factory.createSocket("10.32.54.5", 4483);
socket.setEnabledProtocols(newString [] {"TLSv1"});
socket.setEnabledCipherSuites(newString [] {"TLS_RSA_WITH_AES_128_CBC_SHA"});
socket.startHandshake();
Writer writer =newOutputStreamWriter(socket.getOutputStream());
PrintWriter printer =newPrintWriter(writer);
printer.println("GET /invoke/wm.server:ping HTTP/1.1");
printer.println();
printer.flush();
System.out.println(StreamUtils.toString(socket.getInputStream()));
writer.close();
}
There does not seem to be a way to enable or set specific TLS extensionsfor the requestthough.
Analysis of the error
We surmise from the above that:- It is not just a bug in TLSv1 handshakes as OpenSSL works
- All Microsoft based software seems to be impacted suggesting they use a central configuration which leads to this error
TheRFC (http://www.ietf.org/rfc/rfc2246.txt)does not really shed any more light on the “unexpected message” error.
Comparing the request to the successful OpenSSL TLSv1 handshake, we notice that OpenSSL sends along 27 suggested cipher suites (a selection of them in the screenshot):
As opposed to theMicrosoftclientswhichsend a much shorter list:
None of these suits ismentioned in the JSE6 documentationhttp://docs.oracle.com/javase/6/docs/technotes/guides/security/SunProviders.html#SupportedCipherSuitesbut it may be that there are functional aliases that should work.
Fixing the error
Enable Renegotiation
More about the renegotiation bug can be found in the sources at the end of this document, but hereis a very short summary:- A renegotiation security error was discovered recently (http://www.oracle.com/technetwork/java/javase/documentation/tlsreadme2-176330.html) which triggered an update to both the JVM and to Webmethods to disable renegotiation by default, you can reenable them using the following system properties:
- JVM:sun.security.ssl.allowUnsafeRenegotiation=true
- Webmethods:
- watt.ssl.iaik.clientAllowUnboundRenegotiate=true
- watt.ssl.iaik.serverAllowUnboundRenegotiate=true
- Renegotiation has to be enabled on both client and server to work
However, even if enabling renegotiation was the fix, itdoes not negate the inherent security risk in doing so in a production environment.
Upgrade the JVM
Java 7 was released a little while ago and it contains a number of improvements for TLS communication:http://docs.oracle.com/javase/7/docs/technotes/guides/security/enhancements7.htmlHowever Webmethods does not support it so we can’t run it on a production system.Additionally it seems that Webmethods bypasses the default implementation anyway so it may not impact the error.
Add support for the windows ciphers to the existing JVM
Just in case the lack of a common cipher is indeed the problem, we tried adding another security provider: bouncy castle. We addeditto the security file (jvm\w64_160\jre\lib\security\java.security):#
# List of providers and their preference orders (see above):
#
security.provider.1=sun.security.provider.Sun
security.provider.2=sun.security.rsa.SunRsaSign
security.provider.3=com.sun.net.ssl.internal.ssl.Provider
security.provider.4=com.sun.crypto.provider.SunJCE
security.provider.5=sun.security.jgss.SunProvider
security.provider.6=com.sun.security.sasl.Provider
security.provider.7=org.jcp.xml.dsig.internal.dom.XMLDSigRI
security.provider.8=sun.security.smartcardio.SunPCSC
security.provider.9=org.bouncycastle.jce.provider.BouncyCastleProvider
And added it to the classpath in the server.bat filein the hopes that it would contain a working cipher. No luck there.
Add support for the JVM ciphers to windows
In the same vain, we looked into adding ciphers to the windows machine (as a client-side fix).There seems to be no easy way to add a cipher to windows though the process in C is explained here:http://msdn.microsoft.com/en-us/library/windows/desktop/bb870930%28v=vs.85%29.aspxInstall latest core patch in Webmethods
Although it would seem like the most logical step, installing a core patch for webmethods is also the most invasive solution which is why it is not at the top of our list.Empower contains (at least) one reference to a handshake failure:https://empower.softwareag.com/sl24sec/SecuredServices/KCFullTextASP/viewing/view.asp?KEY=081375-10412598&DSN=PIVOTAL&DST=TCD&HL=1&QUERY=tls|1.0|handshake&SessionID=608289137
It actually points to a problemfor integration server 7.1.3 instead of 8.0.1, but it advises the installation of the latest core fix to solve the problem.
Installation of core fix 20 fixes the problem though we do still do not know exactly which part of the client handshake triggered the error.
Sources
An incomplete list of sources:- TLSRFC:http://www.ietf.org/rfc/rfc2246.txt
- TLS Extensions:
- Microsoft cryptography:
- Cryptography next generation(windows 7+):http://technet.microsoft.com/en-us/library/cc730763%28WS.10%29.aspx
- http://technet.microsoft.com/en-us/library/cc771361%28WS.10%29.aspx
- SChannel:http://msdn.microsoft.com/en-us/library/aa374757%28VS.85%29.aspx
- Windows 2003:http://technet.microsoft.com/en-us/library/cc785811%28WS.10%29.aspx
- http://technet.microsoft.com/en-us/library/cc766285%28WS.10%29.aspx
- http://msdn.microsoft.com/en-us/library/windows/desktop/bb870930%28v=vs.85%29.aspx
- http://support.microsoft.com/kb/245030
- Wireshark:
- Biztalk:
- General
- http://www.ipa.go.jp/security/rfc/RFC2246-AEEN.html
- http://java.sun.com/javase/javaseforbusiness/docs/TLSReadme.html
- http://en.wikipedia.org/wiki/Transport_Layer_Security#TLS_handshake_in_detail
- http://www.g-sec.lu/practicaltls.pdf
- http://docs.oracle.com/javase/1.5.0/docs/guide/security/jsse/ReadDebug.html
Author: Alexander Verbruggen
