SAP PI to Oracle with batch insert - Improved performance

We were implementing a bulk-data transfer from SAP into an Oracle database. Easily upwards of 15 000 materials were being uploaded to the database via a stored procedure (company policy). Normally we like this approach because it decouples PI from the database’s underlying table structure, but we were getting terrible performance.

In testing, the entire workflow took almost 2 hours. Whilst this in itself wasn’t an issue (the process runs in the middle of the night), it was unnecessary load on both systems, and the extended duration put the process at increased risk of failure (e.g. due to network issues).

Keen to improve this, we looked at PI’s “batch insert” capabilities. In order to maintain the decoupling, and in order to protect the destination tables, we created an interface table to temporarily contain the material data, and a procedure that safely updated the destination table.

Testing showed a 30-to-60-fold performance improvement during the PI-DB exchange, and the entire process ended up taking just 10 minutes.

Author: Edwin

ESB = Erroneous Spaghetti Box?

While re-reading the Microservices article by Martin Fowler, I was triggered by the following footnote #7: We can't resist mentioning Jim Webber's statement that ESB stands for "Egregious Spaghetti Box".  I viewed the presentation - from 2008 - in which Jim Webber and Martin Fowler bash the Enterprise Service Bus and translate the acronym ESB into Erroneous Spaghetti Box.

I do agree that often, the integration platform simply contains a spaghetti of point-2-point integrations. But that's good! Way better than all that integration logic dispersed over many systems. With a wide variety of integration techniques, protocols and message formats. And spaghetti in a box is exactly what I tell when explaining what an integration platform is. Only by taking the next step of careful service and message design, one can arrive at a true Service Oriented Architecture.

Let's sum up the main advantages of an integration platform:

• A standardized way to have applications talk to one another
• No coding in a 3GL such as Java or C# but configuration in an application specifically built for the task of integrating systems
• Support for applications of different kinds and ages, including packaged applications
• Strongly reduced diversity in the tools and techniques used to integrate applications
• Support for reliable, asynchronous communication using queuing and message persistence (which Fowler doesn't seem to like either)
• Trivial connectivity through adapters
• Central place to monitor and manage the communication between systems, in particular the reliable message exchange
• Help turn packaged or older applications into services if desired (not everything is developed in-house)

With the disadvantages:

• That it is a central, separate platform,
• Requiring some specific skills (XML)
• The cost of the integration development and support becoming truly visible.

Where Webber and Fowler do have a point, is that middleware vendors come with a whole slew of products. Obviously one should only pick the parts that are useful. And the ESB will definitely not create the Service Oriented Architecture for you.

Author: Guy

Micro Services - Conway Law and Application Integration teams

"Micro Services" is a new buzzword in world of IT architects. As it talks about application components communicating over a network and contains "services", it probably has something to do with SOA and integration. So I had to look into it.


Let's bulletize the description from the article by Martin Fowler and James Lewis:

• The Microservices architectural style is an approach to
• developing a single application < application architecture
• as a suite of small services, < componentization, no libraries
• each (service) running in its own process
• and communicating with lightweight mechanisms, < over the network
• often an HTTP resource API. < REST, message bus
• These services are built around business capabilities < Domain Driven Design
• and independently deployable by fully automated deployment machinery.
• There is a bare minimum of centralized management of these services, 
• which may be written in different programming languages and
• use different data storage technologies < eventual consistency

Microservices are an architecture used by very large, modern IT systems such as LinkedIn, Netflix, Amazon and eBay. There's all sorts of interesting aspects to Micro Services, e.g. the GUI part, security, transactional integrity, versioning etc.

Conway law - Integration Competence Center
But there was one aspect that triggered me in particular when learning about Microservices: Conway Law: "any organization that designs a system (defined broadly) will produce a design whose structure is a copy of the organization's communication structure".

So this law states that an application architecture will reflect the way an IT department is organized. Microservices advocates refer a lot to it.

Service boundaries reinforced by team boundaries (picture from article by Martin Fowler)

For Microservices to focus and align with business functionality, the teams developing (and maintaining) the Microservices should therefore be cross-functional, including the full range of skills required for the development: user-experience, database, and project management.

Orthogonal to the view of the Microservices architects, that Conway Law confirms my personal view and opinion that any IT organization that wishes to leverage a central integration platform to a great extent, requires a separate team developing on and managing that integration platform.


How did I learn about MicroServices?

• The Microservices article by Martin Fowler and James Lewis
• Rough cut version of the O'Reilly book "Building Microservices" by Sam Newman
• Software Engineering Radio podcast episode with Stefan Tilkov
• Software Engineering Radio podcast with James Lewis
• SAI evening session I attended with great presentation by Stijn van den Enden and colleagues

PS: when searching for the term "micro service", I found the term also in the book "Java Web Services Architecture" back from 2003!

Author: Guy

Message modeling and XSD generation

As an integration consultant I work almost daily with XML messages. In my opinion in order to work efficiently with XML you need to have XML schemas. XML schemas makes it possible to validate your messages (including those hard to find typo’s in mappings), they can be used to generate documentation, they define your service contracts and can be used to generate a skeleton of your code. if and when validation should be enabled is a different discussion. Perhaps in the future I will write another article about it.

In order to benefit from XML schemas they need to be clear, precise, flexible and interoperable with the different technologies you are going to use on your project. Amongst us colleagues we regularly have lively discussions on how to achieve this. We all have the same ideas on the general guidelines but sometime we disagree about some details. Mostly it boils down to the choice of technology we are used of working with. But I am relatively sure I can work with the schemas created by my peers.

One major downside of XML schemas is that it is very technical and functional analyst don’t always understand it very well and why should they? They want to model the messages in their favorite modelling tool. In the perfect world you can generate the XSD’s from the model. This way you can enforce the policy you have defined to which the XSD’s should conform.

So what is wrong with this? Nothing! I even encourage you to do it. If it is done correctly and you keep in mind that in the end a developer has to import the XSD’s in his tool and work with them.

On a recent project I had to import an XSD from a third party in order to interact with them. In their documentation they were very proud of their UML model and how clever they were modelling there data. With the generated XSD they were less concerned. From what the XSD should be: simple, flexible, easy to understand … nothing was achieved. I spend 2 days trying to import them in my tool (IBM Integration Toolkit). In the end I gave up as I could no longer justify the time spent to my client.  I wrote my own (very simple) XSD’s that conform to the messages we need to send and receive and used those within our tools.

For those thinking: then don’t use IBM Integration toolkit. I have quite some experience with IBM tooling and in my career I never before had so much problems importing XSD’s. I find the XML support of IBM tools excellent. We tried to import the XSD’s in different tools and they all failed.

So to conclude I want to give you some advice:

• Pay attention to your XML Schemas
• Define guidelines/rules to which you XML schemas should adhere within your organization
• For a public interface make sure the XML schema does not use too advanced schema features (UN/CEFACT Naming and Design Rules may help you there)
• Model your data and generate the XML Schemas from the (UML) model but let your developers validate the generated XSD’s

XML Schemas should be an aid not a burden ! Keep it Simple !

Author: Jef Jansen