Monday, February 27, 2012

Troubleshooting a HTTPS TLSv1 handshake between Microsoft software and Webmethods 8

The problem

Microsoft software (biztalk, wfetch, IE,…) all have a problem when performing a HTTPS TLSv1 handshake to Webmethods 8.0.1 running on a JVM 1.6 server. The problem manifests itself as an error on the client side (biztalk):
“The underlying connection was closed: An unexpected error occurred on a send.”
Note: IE will actually catch this error and retry using SSLv3 so to the end-user it seems that IE works but it actually exhibits the same behavior.
Server specifications:
  • Webmethods: windows 2003, JSE 1.664 bit, v8.0.1
  • Biztalk: windows 2008

Recognizing the error

The first step is recognizing the error on our system so we can identify it when trying to reproduce it.


First step is setting all the logging to “TRACE” for thewebmethodsserver logger. The server log/error log does not show any indication that the failed connection took place.


The JVM offers a few system attributes to enable security debugging:
Setting “” in the server.bat (and running from CLI) produces a stupendous amount of logging at startup which made it ratherunusable (after 10minutes it wasn’t even at the first startup log you expect in server.log), especially because I couldn’t redirect the output of the CLI to a file (for unknown reasons)
Setting “”(or ssl)in the server.bat actually results in no additional output whatsoever.This may point to a custom webmethods implementation for HTTPS.


We installed wireshark on the server to capture the actual traffic going back and forth.We needed some additional configuration to actually make heads and tails from the encrypted traffic (source:
  • Go to edit > preferences > protocols > SSL
  • Add <ip-of-webmethods-server>,<https-port-of-webmethods-server>,TCP,c:/path/to/private-key.pem
    • E.g.,443,TCP,c:\my-private-key.pem
    • Use “;” to separate multiple entires and make sure there are no spaces after each comma
    • Or in newer versions of wireshark:

  • Fill in an SSL debug file (e.g.c:\ssl-debug.log)
  • Restart wireshark and the debug file you configured should show something like:
    • ssl_init found host entry<configured-entry>
    • ssl_init addr‘<ip>' port '<port>' filename '<name>' password(only for p12 file) '(null)'
    • Private key imported: KeyID<id>..
    • ssl_init private key file<path>successfully loaded
  • If the key could not be imported (wrong format or the likes) you will see an error in the log file
In our case the private key was in DER format so we converted it to PEM using openssl:
$ openssl rsa -in private-key.der -out private-key.pem -outform PEM–inperform DER
Now that we can capture the HTTPS traffic,we add a filter to only see the traffic from the client:
ip.src == <client-ip> || ip.dst == <client-ip>
You can also add a port filter if necessary:
tcp.port == <local-port>
You can see the followingtrafficin wiresharkwhen the client attempts to connect:

Reproducing the error

Now that we can recognize the error on our system, we need to reproduce the error so we can narrow down the problem.


Firefox 9.0.1 will perform the entire conversation in SSLv3 which presents no problem to the server:

Internet Explorer

Internet explorer would seem to work from the client perspective as the HTTPS connection issuccessful;however wireshark tells us a slightly different story:

As you can see, it actually tries a TLSv1 connection which fails in the same way as the BizTalk one did, but then simply retries using SSLv3 which is successful.


Microsoft provides a tool to test HTTPS connections: WFetch. If we tell it to connect to the server using TLSv1 (labeled TLS 3.1 because SSL 3.1 == TLS 1.0)we get an error as you can see:

Wireshark shows exactly the same pattern as the BizTalk failed connection.
If you toggle the connect option to “https” it will actually work on windows XP because the initial handshake uses SSLv2:

If you toggle it to “https” on windows 2008(and presumably vista/7)it will try the initial handshake with TLSv1 and fail(note that SSLv2 is disabledin windows 2008 due to security concerns).


With OpenSSL you can also emulate a client (using the s_client option) to test a server:
$openssl s_client -verify 6 -state -msg -showcerts -connect > ssl-output.txt
The output in “ssl-output.txt” shows us that this command uses SSL 2.0 for the initial handshakeand thenproceeds in TLSv1:

However you can force openssl to use tls1:
$openssl s_client -verify 6 -state -msg -showcerts-tls1-connect > ssl-output.txt
Surprisingly, the TLSv1 handshake works for OpenSSL:

The server shows us this:

Java Client

The following code will allow you tomake a rather low level SSL socket connection in java where you can choose the supported cypher suites and protocols:
publicstaticvoidmain(String...args)throwsUnknownHostException, IOException {
SSLSocketFactory factory = (SSLSocketFactory) SSLSocketFactory.getDefault();
SSLSocket socket = (SSLSocket) factory.createSocket("", 4483);
socket.setEnabledProtocols(newString [] {"TLSv1"});
socket.setEnabledCipherSuites(newString [] {"TLS_RSA_WITH_AES_128_CBC_SHA"});

Writer writer =newOutputStreamWriter(socket.getOutputStream());
PrintWriter printer =newPrintWriter(writer);
printer.println("GET /invoke/wm.server:ping HTTP/1.1");


There does not seem to be a way to enable or set specific TLS extensionsfor the requestthough.

Analysis of the error

We surmise from the above that:
  • It is not just a bug in TLSv1 handshakes as OpenSSL works
  • All Microsoft based software seems to be impacted suggesting they use a central configuration which leads to this error
Analysing the “unexpected error” in wireshark shows us this:

TheRFC ( not really shed any more light on the “unexpected message” error.

Comparing the request to the successful OpenSSL TLSv1 handshake, we notice that OpenSSL sends along 27 suggested cipher suites (a selection of them in the screenshot):

As opposed to theMicrosoftclientswhichsend a much shorter list:

None of these suits ismentioned in the JSE6 documentation it may be that there are functional aliases that should work.

Fixing the error

Enable Renegotiation

More about the renegotiation bug can be found in the sources at the end of this document, but hereis a very short summary:
  • A renegotiation security error was discovered recently ( which triggered an update to both the JVM and to Webmethods to disable renegotiation by default, you can reenable them using the following system properties:
    • Webmethods:
      • watt.ssl.iaik.clientAllowUnboundRenegotiate=true
      • watt.ssl.iaik.serverAllowUnboundRenegotiate=true
  • Renegotiation has to be enabled on both client and server to work
We enabled all renegotiation parameters on the server to no avail, the BizTalk client still failed.While we weren’t able to check if the client had renegotiation enabled, their TLS extensions would seem to indicate that they do:

However, even if enabling renegotiation was the fix, itdoes not negate the inherent security risk in doing so in a production environment.

Upgrade the JVM

Java 7 was released a little while ago and it contains a number of improvements for TLS communication:
However Webmethods does not support it so we can’t run it on a production system.Additionally it seems that Webmethods bypasses the default implementation anyway so it may not impact the error.

Add support for the windows ciphers to the existing JVM

Just in case the lack of a common cipher is indeed the problem, we tried adding another security provider: bouncy castle. We addeditto the security file (jvm\w64_160\jre\lib\security\
# List of providers and their preference orders (see above):
And added it to the classpath in the server.bat filein the hopes that it would contain a working cipher. No luck there.

Add support for the JVM ciphers to windows

In the same vain, we looked into adding ciphers to the windows machine (as a client-side fix).There seems to be no easy way to add a cipher to windows though the process in C is explained here:

Install latest core patch in Webmethods

Although it would seem like the most logical step, installing a core patch for webmethods is also the most invasive solution which is why it is not at the top of our list.
Empower contains (at least) one reference to a handshake failure:|1.0|handshake&SessionID=608289137
It actually points to a problemfor integration server 7.1.3 instead of 8.0.1, but it advises the installation of the latest core fix to solve the problem.
Installation of core fix 20 fixes the problem though we do still do not know exactly which part of the client handshake triggered the error.


An incomplete list of sources:
Author: Alexander Verbruggen

Wednesday, February 15, 2012

Invoking stored procedure from DataPower ESB

On a recent project, a few stored procedures had to be exposed as web services.  Actually, the stored procedures already took an XML message as input and returned an XML message as output.

Initial plan was to write some .Net components to expose the stored procedures in the SQL Server 2008 database as web services.  The web services themselves would then be exposed via a DataPower ESB.

But with the DataPower ESB already in place and being capable of directly invoking the stored procedure (see also), the .Net components and underlying Windows servers could be avoided.

A WSDL was created based on the XML input and output messages.  That WSDL was the starting point to create a Web Service Proxy (WSP) on DataPower.  Within a rule of the WSP policy, a Transform step (XSLT) is executed that will invoke the stored procedure using a DataPower specific extension function.

A standard Web Service Proxy will invoke a back-end web service.  The rule of the Web Service Proxy immediately returns - wihout invoking any back-end web service - by setting the variable service/mpgw/skip-backside to 1.

Below the XSLT used to invoke the stored procedure using the dp:sql-execute function.  We obtain the XML part from the soap:Body and pass that on to the stored procedure.  The respone XML is wrapped in a SOAP Envelope.
<?xml version="1.0" encoding="UTF-8"?>
<xsl:stylesheet version="1.0"
  extension-element-prefixes="dp" >
  <xsl:template match="/">
    <xsl:variable name="statement">exec stored_proc ?</xsl:variable>
    <xsl:variable name="result">
      <dp:sql-execute source="'DS_DB'" statement="$statement" >
          <argument name="request_id" type="SQL_XML" mode="INPUT">

              select="./soap:Envelope/soap:Body/xxx:Operation" />
        <xsl:copy-of select="dp:parse($result)" />

Note on the last few lines: the XML response from the stored procedure was contained in a CDATA segment.  With the utility function dp:parse() the XML content within the CDATA could be obtained.

Conclusion: one of the great features of Integration tools (ESB, EAI) is their built-in database connectivity. The database adapter of an ESB allows querying, updating and invoking of stored procedures. Database adapters typically also support polling (staging) tables for new or modified records.

Author: Guy