Troubleshooting a HTTPS TLSv1 handshake between Microsoft software and Webmethods 8

The problem

Microsoft software (biztalk, wfetch, IE,…) all have a problem when performing a HTTPS TLSv1 handshake to Webmethods 8.0.1 running on a JVM 1.6 server. The problem manifests itself as an error on the client side (biztalk):
“The underlying connection was closed: An unexpected error occurred on a send.”
Note: IE will actually catch this error and retry using SSLv3 so to the end-user it seems that IE works but it actually exhibits the same behavior.
Server specifications:

• Webmethods: windows 2003, JSE 1.664 bit, v8.0.1
• Biztalk: windows 2008

Recognizing the error

The first step is recognizing the error on our system so we can identify it when trying to reproduce it.

Webmethods

First step is setting all the logging to “TRACE” for thewebmethodsserver logger. The server log/error log does not show any indication that the failed connection took place.

JVM

The JVM offers a few system attributes to enable security debugging:

• java.security.debug
• javax.net.debug

Setting “java.security.debug=all” in the server.bat (and running from CLI) produces a stupendous amount of logging at startup which made it ratherunusable (after 10minutes it wasn’t even at the first startup log you expect in server.log), especially because I couldn’t redirect the output of the CLI to a file (for unknown reasons)
Setting “javax.net.debug=all”(or ssl)in the server.bat actually results in no additional output whatsoever.This may point to a custom webmethods implementation for HTTPS.

Wireshark

We installed wireshark on the server to capture the actual traffic going back and forth.We needed some additional configuration to actually make heads and tails from the encrypted traffic (source:http://support.citrix.com/article/CTX116557)

• Go to edit > preferences > protocols > SSL
• Add <ip-of-webmethods-server>,<https-port-of-webmethods-server>,TCP,c:/path/to/private-key.pem
  
  • E.g. 10.0.0.1,443,TCP,c:\my-private-key.pem
  • Use “;” to separate multiple entires and make sure there are no spaces after each comma
  • Or in newer versions of wireshark:

• Fill in an SSL debug file (e.g.c:\ssl-debug.log)
• Restart wireshark and the debug file you configured should show something like:
  
  • ssl_init found host entry<configured-entry>
  • ssl_init addr‘<ip>' port '<port>' filename '<name>' password(only for p12 file) '(null)'
  • Private key imported: KeyID<id>..
  • ssl_init private key file<path>successfully loaded
• If the key could not be imported (wrong format or the likes) you will see an error in the log file

In our case the private key was in DER format so we converted it to PEM using openssl:
$ openssl rsa -in private-key.der -out private-key.pem -outform PEM–inperform DER
Now that we can capture the HTTPS traffic,we add a filter to only see the traffic from the client:
ip.src == <client-ip> || ip.dst == <client-ip>
You can also add a port filter if necessary:
tcp.port == <local-port>
You can see the followingtrafficin wiresharkwhen the client attempts to connect:

Reproducing the error

Now that we can recognize the error on our system, we need to reproduce the error so we can narrow down the problem.

Firefox

Firefox 9.0.1 will perform the entire conversation in SSLv3 which presents no problem to the server:

Internet Explorer

Internet explorer would seem to work from the client perspective as the HTTPS connection issuccessful;however wireshark tells us a slightly different story:

As you can see, it actually tries a TLSv1 connection which fails in the same way as the BizTalk one did, but then simply retries using SSLv3 which is successful.

WFetch

Microsoft provides a tool to test HTTPS connections: WFetch. If we tell it to connect to the server using TLSv1 (labeled TLS 3.1 because SSL 3.1 == TLS 1.0)we get an error as you can see:

Wireshark shows exactly the same pattern as the BizTalk failed connection.
If you toggle the connect option to “https” it will actually work on windows XP because the initial handshake uses SSLv2:

If you toggle it to “https” on windows 2008(and presumably vista/7)it will try the initial handshake with TLSv1 and fail(note that SSLv2 is disabledin windows 2008 due to security concerns).

OpenSSL

With OpenSSL you can also emulate a client (using the s_client option) to test a server:
$openssl s_client -verify 6 -state -msg -showcerts -connect 10.32.54.5:4483 > ssl-output.txt
The output in “ssl-output.txt” shows us that this command uses SSL 2.0 for the initial handshakeand thenproceeds in TLSv1:

However you can force openssl to use tls1:
$openssl s_client -verify 6 -state -msg -showcerts-tls1-connect 10.32.54.5:4483 > ssl-output.txt
Surprisingly, the TLSv1 handshake works for OpenSSL:

The server shows us this:

Java Client

The following code will allow you tomake a rather low level SSL socket connection in java where you can choose the supported cypher suites and protocols:
publicstaticvoidmain(String...args)throwsUnknownHostException, IOException {
SSLSocketFactory factory = (SSLSocketFactory) SSLSocketFactory.getDefault();
SSLSocket socket = (SSLSocket) factory.createSocket("10.32.54.5", 4483);
socket.setEnabledProtocols(newString [] {"TLSv1"});
socket.setEnabledCipherSuites(newString [] {"TLS_RSA_WITH_AES_128_CBC_SHA"});
socket.startHandshake();

Writer writer =newOutputStreamWriter(socket.getOutputStream());
PrintWriter printer =newPrintWriter(writer);
printer.println("GET /invoke/wm.server:ping HTTP/1.1");
printer.println();
printer.flush();

System.out.println(StreamUtils.toString(socket.getInputStream()));

writer.close();
}
There does not seem to be a way to enable or set specific TLS extensionsfor the requestthough.

Analysis of the error

We surmise from the above that:

• It is not just a bug in TLSv1 handshakes as OpenSSL works
• All Microsoft based software seems to be impacted suggesting they use a central configuration which leads to this error

Analysing the “unexpected error” in wireshark shows us this:

TheRFC (http://www.ietf.org/rfc/rfc2246.txt)does not really shed any more light on the “unexpected message” error.

Comparing the request to the successful OpenSSL TLSv1 handshake, we notice that OpenSSL sends along 27 suggested cipher suites (a selection of them in the screenshot):

As opposed to theMicrosoftclientswhichsend a much shorter list:

None of these suits ismentioned in the JSE6 documentationhttp://docs.oracle.com/javase/6/docs/technotes/guides/security/SunProviders.html#SupportedCipherSuitesbut it may be that there are functional aliases that should work.

Fixing the error

Enable Renegotiation

More about the renegotiation bug can be found in the sources at the end of this document, but hereis a very short summary:

• A renegotiation security error was discovered recently (http://www.oracle.com/technetwork/java/javase/documentation/tlsreadme2-176330.html) which triggered an update to both the JVM and to Webmethods to disable renegotiation by default, you can reenable them using the following system properties:
  
  • JVM:sun.security.ssl.allowUnsafeRenegotiation=true
  • Webmethods:
    
    • watt.ssl.iaik.clientAllowUnboundRenegotiate=true
    • watt.ssl.iaik.serverAllowUnboundRenegotiate=true
• Renegotiation has to be enabled on both client and server to work

We enabled all renegotiation parameters on the server to no avail, the BizTalk client still failed.While we weren’t able to check if the client had renegotiation enabled, their TLS extensions would seem to indicate that they do:

However, even if enabling renegotiation was the fix, itdoes not negate the inherent security risk in doing so in a production environment.

Upgrade the JVM

Java 7 was released a little while ago and it contains a number of improvements for TLS communication:http://docs.oracle.com/javase/7/docs/technotes/guides/security/enhancements7.html
However Webmethods does not support it so we can’t run it on a production system.Additionally it seems that Webmethods bypasses the default implementation anyway so it may not impact the error.

Add support for the windows ciphers to the existing JVM

Just in case the lack of a common cipher is indeed the problem, we tried adding another security provider: bouncy castle. We addeditto the security file (jvm\w64_160\jre\lib\security\java.security):
#
# List of providers and their preference orders (see above):
#
security.provider.1=sun.security.provider.Sun
security.provider.2=sun.security.rsa.SunRsaSign
security.provider.3=com.sun.net.ssl.internal.ssl.Provider
security.provider.4=com.sun.crypto.provider.SunJCE
security.provider.5=sun.security.jgss.SunProvider
security.provider.6=com.sun.security.sasl.Provider
security.provider.7=org.jcp.xml.dsig.internal.dom.XMLDSigRI
security.provider.8=sun.security.smartcardio.SunPCSC
security.provider.9=org.bouncycastle.jce.provider.BouncyCastleProvider
And added it to the classpath in the server.bat filein the hopes that it would contain a working cipher. No luck there.

Add support for the JVM ciphers to windows

In the same vain, we looked into adding ciphers to the windows machine (as a client-side fix).There seems to be no easy way to add a cipher to windows though the process in C is explained here:http://msdn.microsoft.com/en-us/library/windows/desktop/bb870930%28v=vs.85%29.aspx

Install latest core patch in Webmethods

Although it would seem like the most logical step, installing a core patch for webmethods is also the most invasive solution which is why it is not at the top of our list.
Empower contains (at least) one reference to a handshake failure:https://empower.softwareag.com/sl24sec/SecuredServices/KCFullTextASP/viewing/view.asp?KEY=081375-10412598&DSN=PIVOTAL&DST=TCD&HL=1&QUERY=tls|1.0|handshake&SessionID=608289137
It actually points to a problemfor integration server 7.1.3 instead of 8.0.1, but it advises the installation of the latest core fix to solve the problem.
Installation of core fix 20 fixes the problem though we do still do not know exactly which part of the client handshake triggered the error.

Sources

An incomplete list of sources:

• TLSRFC:http://www.ietf.org/rfc/rfc2246.txt
• TLS Extensions:
  
  • http://www.networksorcery.com/enp/protocol/tls.htm
  • http://www.ietf.org/rfc/rfc3546.txt
  • http://tools.ietf.org/html/rfc4366
  • http://tools.ietf.org/html/rfc4492
• Microsoft cryptography:
  
  • Cryptography next generation(windows 7+):http://technet.microsoft.com/en-us/library/cc730763%28WS.10%29.aspx
  • http://technet.microsoft.com/en-us/library/cc771361%28WS.10%29.aspx
  • SChannel:http://msdn.microsoft.com/en-us/library/aa374757%28VS.85%29.aspx
  • Windows 2003:http://technet.microsoft.com/en-us/library/cc785811%28WS.10%29.aspx
  • http://technet.microsoft.com/en-us/library/cc766285%28WS.10%29.aspx
  • http://msdn.microsoft.com/en-us/library/windows/desktop/bb870930%28v=vs.85%29.aspx
  • http://support.microsoft.com/kb/245030
• Wireshark:
  
  • http://support.citrix.com/article/CTX116557
  • http://wiki.wireshark.org/SSL
• Biztalk:
  
  • http://www.codeproject.com/Articles/88638/HTTPs-using-BizTalk-2009-HTTP-adapter-on-Windows-2
  • http://jason.agostoni.net/2010/10/07/biztalk-2009-https-connection-closed-issue/
• General
  
• http://www.ipa.go.jp/security/rfc/RFC2246-AEEN.html
• http://java.sun.com/javase/javaseforbusiness/docs/TLSReadme.html
• http://en.wikipedia.org/wiki/Transport_Layer_Security#TLS_handshake_in_detail
• http://www.g-sec.lu/practicaltls.pdf
• http://docs.oracle.com/javase/1.5.0/docs/guide/security/jsse/ReadDebug.html

Author: Alexander Verbruggen