Reading compressed files with Oracle Service Bus

Originally Posted by James Goddard

 reasonably common B2B interface convention is to compress the contents of exchanged files, to reduce the impact on network traffic and archiving requirements (see Figure below).  Although Oracle Middleware does not support such interfaces “out of the box,” it is reasonably straight-forward to piece together a simple adapter using existing tools.
01_B2B_Overview
This recipe will guide you through a sample implementation of a proxy service which polls a directory for XML files compressed using the gzip utility.
Gzip is a data compression software application, most commonly encountered as the version implemented by the GNU Project. Gzip only natively supports the compression of one file at a time (although of course that file may itself be the combination of several smaller files), and has a simple format consisting of compressed binary content between a standard header and footer.
J2SE includes a standard library of compression/decompression utilities in the java.util.zip package, which, among other things, includes functionality for working with the gzip data compression algorithm.

Getting ready...

Oracle Enterprise Pack for Eclipse

Although it is possible to develop for the Oracle Service Bus using only the Web Console interface, for a fully featured IDE (including support for version control, source code editing plug-ins, lower latency and independence of development environments) it is generally preferable to use the OSB Workshop perspective of the OEPE.
This recipe assumes the use of an existing OSB Configuration project within the OSB Workshop for development so ensure that you have installed and familiarised yourself with it prior to beginning.

Test Data

Prior to beginning this recipe, you will want to prepare some test data, consisting of XML files compressed using gzip.

How to do it...

Writing the Java method

Before jumping into creating anything for OSB, you’ll first need a java library which handles the extraction.
  • Start up Eclipse and switch to the Java perspective.
02_Java_Perspective
  • Right-click in the Project Explorer area and select New > Java Project.
03_new_project
  • Name the project GzExtractor and then select Next.
  • Select the Libraries tab and click Add External JARs....
04_add_library
  • Select com.bea.core.xml.xmlbeans_2.2.0.0.jar from the modules subdirectory of your Oracle middleware installation and click Open.
05_xmlbeans_selection
  • Click Finish in the Add External JARs dialog, and then Finish again in the New Java Project dialog box.
  • Right click on the new project and select New->Class.
06_new_class
  • Set the package to com.rubiconred and the class name to GzExtractor.  Leave everything else as default and then click Finish.
07_class_name
  • Replace the contents of the new file GzExtractor.java with the code below.
[code] package com.rubiconred; import org.apache.xmlbeans.*; import java.io.*; import java.util.zip.GZIPInputStream; public class GzExtractor { public static XmlObject processFile( Object param ) throws IOException, XmlException { byte[] bytes = (byte[])param; if (bytes != null) { InputStream input = new ByteArrayInputStream(bytes); InputStream gzipInput = new GZIPInputStream(input); Writer writer = new StringWriter(); char[] buffer = new char[1024]; try { BufferedReader reader = new BufferedReader( new InputStreamReader(gzipInput, "UTF-8") ); int n; while ((n = reader.read(buffer)) != -1) writer.write(buffer, 0, n); } finally { gzipInput.close(); } // return the contents of the file return XmlObject.Factory.parse( writer.toString() ); } else { // input parameter is null, return null return null; } } } [/code]

Incorporating the Jar File

The following steps explain how the java method created above can be incorporated into an OSB project.
  • Switch to the Oracle Service Bus perspective.
  • If you don’t have one already, create a new OSB Configuration Project and a new OSB project within it.  Name the project GzFileAdapter.
  • Right click on the GzExtractor Java project and select Export->Export....
08_export_java
  • Select Java->JAR file and then click Next.
09_export_to_jar
  • Provide the export destination as GzFileAdapter/GzExtractor.jar and then click Finish.
10_jar_destination
  • Right click on the GzFileAdapter OSB project and select Import->Import....
11_import_import
  • Select File System in the General category and then click Next.
12_import_file_system
  • Select the com.bea.core.xml.xmlbeans_2.2.0.0.jar XMLBeans library from the modules subdirectory of your Oracle middleware installation, as used earlier in the Java project, and then click Finish.
13_xmlbeans_selection
  • Right click on GzExtractor.jar in the Project Explorer and select Oracle Service Bus -> Modify Jar Dependencies from the context menu.
14_osb_modify_jar_dependencies
  • Select the XMLBeans library, then click Add, then OK.
15_jar_dependencies_selection

Creating the Proxy Service

The next step is to create the file adapter, as a binary file-based message proxy service, from the Oracle Service Bus IDE.
  • Right-click on the GzFileAdapter OSB project in the Project Explorer and select New -> Proxy Service.
16_new_proxy_service
  • Name it GzFileAdapter and click Finish.
  • In the General tab of the Proxy Service editor, select Message Service as the Service Type.
17_general_configuration
  • In the Messaging tab, select Binary as the Request Message Type.  Leave the Response Message Type as None.
18_message_type_configuration
  • In the Transport tab, select file as the Protocol and provide a directory name where you would like to poll for files from as the Endpoint URI.
19_transport_configuration
  • In the File Transport tab, customise the adapter to suit.  In this example, we’ll select a file mask of *.gz as well as configuring an archive Post Read Action and specifying all the necessary Directory file destinations.
20_file_transport_configuration

Defining the message flow

The next step is to incorporate a call to the GzExtractor java library into the Proxy Service Message Flow.
  • Start by dragging a new Pipeline Pair from the Design Palette into the Message Flow editor.
21_pipeline_pair
  • Add a new Stage to the Request Pipeline and name it Extract GZIP.
22_extract_gzip_stage
  • Drag an Assign Action into the new Stage.
23_assign
  • In the Properties tab for the Assign action, provide the Expression as $body/ctx:binary-content and theVariable as gzBinaryContent.
24_assign_properties
  • Drag a new Java Callout Action from the Design Palette to just under the Assign action.
25_java_callout
  • In the Properties tab for the new Java Callout, select Browse next to the Method field.
26_java_callout_properties
  • Select GzExtractor/com.rubiconred.GzExtractor.processFile.
  • Select the Expression for the only parameter and specify $gzBinaryContent.
27_java_callout_properties_part_2
  • Specify the Result Value variable as payload.

Testing the implementation

To test the new file adapter, you can place a GZIP’d file in the input directory and confirm the contents are read successfully following these steps:
  • Create a new Reporting Stage below the GZIP Extract Stage and drag a new Log Action in.
28_log_report
  • Set the Log Expression as $payload and the Annotation as “GzFileAdapter message payload”.
29_log_properties
  • Change the Severity to Error so that it is guaranteed to appear on your weblogic’s console output.
  • Deploy the service to your server.
  • Copy a GZIP’d text file to the input directory you specified earlier.
  • Observe that the contents are written to the console.

Email Notification with SOA Suite 11g

Oracle SOA Suite 11g supports a number of different notification channels through the User Messaging Service (UMS). This blog provides a step-by-step guide for configuring and testing email notification. Once email notification has been configured, emails can be sent from SOA Suite components such as BPEL processes and Human Tasks. For this example, Gmail will be used as the mail server.

Configuring Email Notification

To support email notification from SOA Suite 11g the UMS email driver and workflow notification properties to be configured with the mail server details. This can be done through the Enterprise Manager Fusion Middleware Control console (e.g. http://localhost:7001/em) as follows:
  1. Navigate to the Email Driver Properties page available under User Messaging Service >usermessagingdriver-email UMS-NavigationToEmailDriverProperties-2
  2. Update the following properties according to your mail server configuration:
    OutgoingMailServer - The SMTP hostname
    OutgoingMailServerPort - The SMTP port
    OutgoingMailServerSecurity - The type of security (e.g. SSL)
    OutgoingUsername - The mail user account
    OutgoingPassword - The mail user password
    UMS-EmailDriverProperties
  3. Click Apply to save the changes. UMS-EmailDriverApply
  4. Right-click on soa-infra under SOA and navigate to the Workflow Notification Properties page available under SOA Administration. UMS-NavigateToWorkflowNotificationProperties
  5. Set the Notification Mode to 'Email' and enter the default values which you wish to use for from, actionable and reply to email addresses. UMS-WorkflowNotificationProperties
  6. Click Apply to save the changes to the workflow notification properties.
  7. Restart the SOA server for the changes to take effect. If you are using Gmail or any other mail server that requires SSL you will need to perform the steps in the next section. In this case, you may wish to hold off on restarting the server before these steps are performed. This will save you the trouble of having to restart twice.

Import the SSL Certificates to the Java Keystore

Before SOA Suite can send emails with Gmail there are some SSL certificates that will need to be imported. This step can be ignored if sending from an alternative mail server that does not require SSL.
  1. Download the Gmail IMAP and SMTP certificates. This can be done with OpenSSL (Unix / Windows) by running the following from command prompt:
    • openssl s_client -connect host:port
      • where Gmail SMTP host:port is smtp.gmail.com:465
      • where Gmail IMAP host:port is imap.gmail.com:993
  2. Create a new file for the SMTP and IMAP certificates using the content between BEGIN CERTIFICATE & END CERTIFICATE Gmail-SSL-Certificate
  3. Use the Java Keytool to import the two certificates into a new or existing keystore. The keytool executable is available under the JAVA_HOME/bin directory. The following command can be used to import a certificate:
    • keytool -import -alias AliasName -keystore KeystoreLocation -file CertificateLocation
      • For example:
        • keytool -import -alias gmail-imap -keystore gmail-keystore.jks -file imap.txt
        • keytool -import -alias gmail-smtp -keystore gmail-keystore.jks -file smtp.txt
    • When prompted, enter a password for the keystore.
  4. Add the following system properties to the WebLogic server startup arguments:
    -Djavax.net.ssl.trustStore=CertificateLocation-Djavax.net.ssl.trustStorePassword=CertificatePassword
    This can be done in Arguments field available under the Server Start tab for the SOA Server (e.g. soa_server1) in the WebLogic Administration Console (e.g. http://localhost:7001/console) or by adding the properties to the DOMAIN_HOME/bin/setDomainEnv.cmd (or setDomainEnv.sh). For example:
    set EXTRA_JAVA_PROPERTIES=%EXTRA_JAVA_PROPERTIES% -Djavax.net.ssl.trustStore=%WL_HOME%\server\lib\gmail-keystore.jks -Djavax.net.ssl.trustStorePassword=changeit
  5. Start the SOA server (or restart it if it is already running).

Testing Email Notification

Now that we have configured outbound email notification, we want to test our configuration easily without having to create a BPEL process or human task to do this. Fortuntately, we can do this from Enterprise Manager Fusion Middleware Control.
  1. Right-click on soa-infra under SOA and navigate to the Human Workflow page available under Service Engines. Test-NavigateToHumanWorkflow
  2. Select the Notification Management tab.
  3. Click Send Test Notification Test-SendTestNotification
  4. The Send Test Notification dialog box should now appear. Enter the email subject, content, to address and be sure to select 'Email' for the channel. Click Send. Once the email has been sent, you should receive a value of 'SENT' for Response. You can now close the dialog box. Test-Notification
  5. Ensure that the email has been received.

Troubleshooting

If you encounter an issue with the outgoing email notification, it is a good idea to check the SOA server log for error messages. There are log files available at:
  • DOMAIN_HOME/servers/<soa server name>/logs/<soa server name>.log
  • DOMAIN_HOME/servers/<soa server name>/logs/<soa server name>.out
  • DOMAIN_HOME/servers/<soa server name>/logs/<soa server name>-diagnostic.log
Furthermore, it is a good idea to tick the UMS email driver debug flag.
Listed below are some common errors with their solution:
  • Error Message: javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target Diagnosis: There is an issue with the SSL certificate. Remedy: Ensure all of the steps have been followed as described in "Import the SSL Certificates to the Java Keystore". Also make sure that the mail server certificates in the Java Keystore have not expired. You can ensure a fresh copy of the certificates are downloaded by executing the openssl s_client -connect command followed by the host and port details for the mail server.
  • Error Message: ORABPEL-31023 - Cannot send email notification to <email address> address. The address is being marked as invalid and no further notification would be sent to this address. Diagnosis: Due to a number of previous failed send attempts to an email address, SOA Suite has added the address to the Bad Address List thus preventing subsequent sends to this address from failing. The previous failed attempts may been caused by a configuration issue. Remedy: Remove the email address from the Bad Address List. This can be done by selecting View Bad Addresses from the Notification Management page in Enterprise Manager. When the Bad Address List dialog box is displayed, remove the email address from this list.
  • Error Message: No matching drivers found for sender address Diagnosis: The UMS Driver for the appropriate channel is configured with a specific list of SenderAddresses, and the message sent by the application has a Sender Address that does not match. Remedy: Make the Sender Address blank. This will ensure that all outbound messages will be sent regardless of the sender address.

Caching Service Responses with OSB

Originally posted by James Goddard

Over the course of a complex service orchestration, it may be useful or necessary to query the same set of configuration data for an entity or process multiple times during and/or between executions. Where retrieving this configuration data involves interfacing with an external system or database, there can be reasonably significant I/O and network latency overheads associated which create bottlenecks in a service’s execution.
A popular technique for eliminating such bottlenecks is the concept of caching results, keeping a copy in memory paired with the key of the original request in case the same request is made again. The assumption of course is that configuration data changes infrequently enough to make these “old” results still valid for use for some time after the first query.
Oracle offers a distributed in-memory cache in a product called Coherence. Coherence features fast and reliable performance, redundancy across clustered servers, an intuitive API and many options for customising how it uses and retains information. Coherence is also packaged as a component of the Oracle Weblogic server and its functionality is incorporated into Oracle Service Bus.
This recipe will walk you through caching the responses from a business service using OSB.

Getting Ready...

Oracle Service Bus 11g

This recipe assumes you have already configured an OSB environment on which to deploy the service.

Oracle Enterprise Pack for Eclipse

Although it is possible to develop for the Oracle Service Bus using only the Web Console interface, for a fully featured IDE (including support for version control, source code editing plug-ins, lower latency and independence of development environments) it is generally preferable to use the OSB Workshop perspective of the OEPE.
This recipe assumes the use of an existing OSB Configuration project within the OSB Workshop for development so ensure that you have installed and familiarised yourself with it prior to beginning.

External Service

Prior to beginning this recipe, it is assumed that you already have access to the service you wish to cache.
If you wish to follow along exactly with the “Fruit info” example in these instructions you will require an Oracle database and a copy of the JCA database adapter used in the example.  You can obtain a copy of this, along with the database schema creation scripts from the following URL:
You will need to execute the database schema creation script, and then create a corresponding connection factory on your weblogic administration server  identified by “jndi.cf.cookbook”.

How to do it...

Check the server configuration

In order to allow caching, the OSB weblogic server must be appropriately configured.  Log into the OSB Administration Console and confirm the setting.
  • Direct a web browser to http://localhost:7001/sbconsole/
  • Log in using a system administrator account (by default, weblogic).
  • Select Global Settings from the Operations menu.
  • Confirm that Enable Result Caching is check (by default, it should be).
result_caching_enabled
If it is not enabled, complete the following steps to enable it.
  • Click the Create button in the top left.  Check the Enable Result Caching option and then click Update.enable_step_1
  • Click the Activate button, type a Description of the change and then click Submit.enable_step_2

Create the Business Service

In order to cache results using OSB, the first thing you need to do is wrap the service you wish to cache using an OSB Business Service.  If you’ve done this already, skip ahead to the next section "Configure the Business Service".
  • If you don't have one already, create a new OSB Configuration Project.
  • Create a new OSB Project.
  • Add a subfolder for the business service and create a new OSB Business Service, configured to wrap the service you wish to cache the responses from.  For example, for the provided "FruitInfo" example:
    • Import all the files from the “Database Adapter” folder provided into a subfolder called fruitinfo_db.import_importimport_filesimport_files_choice
    • Right-click on the fruitinfo_db.jca adapter definition and select Generate Service.generate_service
    • Accept the defaults and click OK.

Configure the Business Service

This is the key configuration step.
enable_caching
  1. Open your Business Service file for editing (the .biz file).
  2. Select the Message Handling tab.
  3. Expand the Advanced Settings.
  4. Check the Supported box next to Result Caching.
  5. Select the Duration radio button to indicate that cached results should expire after a set duration.
  6. Key in the amount of time you’d like for the Duration.  (In this example we’ve chosen 5 minutes to make it easy to test).
  7. Click on the <Expression> link next to Cache Token Expression.
  8. From the variable structures on the right, locate the appropriate request operation and drag the key field from your request over into the text field on the left.cache_expression
  9. Manually add /text() to the end of the expression to ensure that the contents of the node are used as the cache key.
  10. Select OK and then save all changes.

Expose using a Proxy Service

If you haven’t already done so, you will need to expose your Business Service via an OSB Proxy Service.
  1. Right click on the OSB Project and select New-> Proxy Service giving it an appropriate name. This will be the external facing service which wraps your cached service. Open the service for editing.  For the simplest implementation you can pass the payload straight through by re-using the WSDL from the business service.select_wsdl
  2. Select WSDL Web Service as the Service Type.
  3. Browse
to select an existing WSDL.
  1. In the resulting dialog browse to the WSDL of the business service and select its port.
  2. Click OK.
  3. When prompted to change the transport configuration, select No to retain the HTTP transport.
  4. Select the Message Flow tab.
  5. Drag a Route Node across into the flowroute_node.
  6. Drag a Routing Action into the Route Node. routing
  7. Select the Routing Action to bring up the Properties tab. route_to_biz
  8. Click Browse next to Service.
  9. In the resulting dialog, navigate to the business service which encapsulates the caching mechanism.
  10. Click OK.
  11. Ensure that the correct operation is invoked in the Properties tab. invoking
  12. Save all changes.
You’re done!  You can now use the Proxy Service to access your cached business service operation.

Testing and confirming the Cache

Deploy and test your service.  Then complete the following steps to confirm the caching behaviour.
  1. Navigate to the OSB Console at http://localhost:7001/sbconsole and login. osb_console
  2. Select Project Explorer from the main menu on the left.
  3. Navigate to the OSB Project you just created.
  4. Click on the Debug symbol to test the Proxy Service.
  5. In the resulting window, select the cached operation.  Note that the other operations will not correctly route. execute_test
  6. Modify the payload to specify the key for your request.  In this example we chose “Apple”.
  7. Call the service by clicking Execute.
  8. Confirm the response.
response
  • Now modify the state of your reference data. E.g. in our example we will modify the database entry for “Apple” to have color “Green”. change_color
  • Re-run the same Proxy Service test and note the response.
response
The color in the database is “Green”, yet the service still returns “Red”.  This is because the result was successfully cached.
  • Wait for the cached result to expire and then call the proxy service again with the same test to confirm that it will eventually refresh the result.
new_result.

How it works...

As you have seen, caching using Coherence in OSB is very simple to activate and use.  The following figure illustrates what is going on behind the scenes.
coherence_simplified
Because for most cases Coherence will be able to retrieve the result from the in-memory grid on the same application server, there will be no latency introduced by network or database I/O.  This should greatly reduce the response time of your service, assuming frequent requests for the same data are made.

There's more...

More complex cache tokens

Your desired cache token key may not be as straight forward as selecting the contents of a single, predictably placed node. It is worth noting that the Cache Token may be specified as any expression you like, using the XQuery functional programming language. XQuery has a rich syntax for constructing queries, including the use of many system libraries with common functions such as numeric operations and string manipulation.

Clear the cache explicitly

It might be that you can’t risk the cache becoming stale under certain circumstances.  If that is the case, rather than using a time parameter to manage the lifetime of the cached responses, mark critical requests with an additional flag and select XQuery Expression in the expiration time options to test for the existence of this flag.
expiration_xquery