Skip to main content

Processing large payloads with the esb script mediator iteratively


Overview

WSO2 ESB uses Rhino engine to execute JavaScripts. Rhino engine converts the script to a method inside a Java class. Therefore, when processing large JSON data volumes, the code length must be less than 65536 characters, since the Script mediator converts the payload into a Java object. However, you can use the following alternative options to process large JSON data volumes.

The script mediator which is used in ESB is powered by the Rhino engine. Therefore, when processing large JSON data volumes, the code length must be less than 65536 characters which is a limitation in the script mediator being used in the esb versions less than 5.0.0. In ESB 5.0.0 there is a higher capability to process larger payloads using script mediator. In order to process such large payloads we can follow the below two approaches.

1. Replace the javascript tranformation logic using java code by writing a custom mediator. [1]
2. Break down the large payload and execute them as sections using the script mediator.

This blog post will be demonstrating how to carry out the use case #2 using wso2 esb.

Let's get started

The approach we would be using is to iterate over the large response and pass each section through the script mediator. After which we will be aggregating the tranformed sections to display the modified and transformed response.

1) Let's create a mock service which returns the response mentioned in this text file. I have used the online tool http://www.mocky.io/ to generate this response. The invocation URL for my mock response is http://www.mocky.io/v2/57a881a3110000b6171d4546

2) Then we need to create the In sequence. In the ESB in the Service bus section , select the Sequences option and add a new sequence. This is the source view of the in sequence we will be using which simply has a endpoint invocation. This is the in sequence configuration and you can also download it from HERE

    
<sequence name="sendSequence" xmlns="http://ws.apache.org/ns/synapse">
    <log level="custom">
        <property name="test" value="Execution Started"/>
    </log>
    <send>
        <endpoint>
            <http method="GET" uri-template="http://www.mocky.io/v2/57a881a3110000b6171d4546"/>
        </endpoint>
    </send>
</sequence>


3) Next we need to create the out sequence which does the needed transformation. I have listed what happens within the sequence to get a better understanding.

  • I have used the response generated from the above invocation here which has an array of json objects. Since the objects inside the array are repititive we can split it using the iterate mediator. (In this sequence the splitting is done by the ARR_ITEM expression)
  • After splitting it using the iterate mediator we can manipulate each JSON object by passing it through the script mediator.
  • At the end we are placing an aggregate mediator which will aggregate all the modified JSON elements which went through the iterate mediator and display the response.
  • We will be storing the aggregated response inside the rootElement property as json objects. Follow is the sequence which executes this logic. This is the outsequence of our API.
Simillar to above you need to add another sequence which we would be using as the out sequence. This is the sequence source view corresponding to the out sequence described above. You can also download the sequence HERE

       
<sequence xmlns="http://ws.apache.org/ns/synapse" name="iterOutTest">
   <property name="rootElement" scope="default">
      <jsonObject xmlns=""/>
   </property>
   <iterate xmlns:ns="http://org.apache.synapse/xsd"
            continueParent="true"
            expression="//ARR_ITEM">
      <target>
         <sequence>
            <log level="custom">
               <property name="iterator" value="in the iterate"/>
            </log>
            <payloadFactory media-type="json">
               <format>
                $1
            </format>
               <args>
                  <arg evaluator="json" expression="$."/>
               </args>
            </payloadFactory>
            <script language="js">
               mc.setProperty('CONTENT_TYPE', 'application/json');
               var payLoad = mc.getPayloadJSON();
               var catalogItem = payLoad.ARR_ITEM;
               catalogItem.CT.CT_ITEM.ED = "456"; 
               catalogItem.DISCRIPTIONS = "This is the modified description";                
               mc.setPayloadJSON(payLoad);</script>
            <log level="full"/>
            <aggregate>
      <completeCondition>
         <messageCount min="-1" max="-1"/>
      </completeCondition>
      <onComplete xmlns:ns="http://org.apache.synapse/xsd"
                  expression="//jsonObject/ARR_ITEM"
                  enclosingElementProperty="rootElement">
         <log level="full">
            <property name="MSG" value="Aggregated Msg"/>
         </log>
         <respond/>
      </onComplete>
   </aggregate>
         </sequence>
      </target>
   </iterate>
</sequence>

4) After we have created the sequences we need to then add it to the API. Refer the below screenshot. We need to add the above created in and out sequences to the API respectively


Here you can find the xml configuration corresponding to the above API. Also you can download the API from HERE.


<api xmlns="http://ws.apache.org/ns/synapse" name="IterateAggregateAPI" context="/iterate">
   <resource methods="POST" uri-template="/*" inSequence="sendSequence" outSequence="iterateOUT"/>
</api>

5) Using the soap UI or any other rest client you can invoke the API and observe how the response payload is iterated and processed to display the aggregated and tranformed response.


I hope that following this tutorial you were able to understand how large payloads could be split using common elements to process using the script mediator.

References:

[1] https://docs.wso2.com/display/ESB490/Creating+Custom+Mediators
[2] https://docs.wso2.com/display/ESB490/Iterate+Mediator
[3] https://docs.wso2.com/display/ESB490/Aggregate+Mediator
[4] https://docs.wso2.com/display/ESB490/Creating+APIs
[5] https://docs.wso2.com/display/ESB490/Mediation+Sequences


Comments

Popular posts from this blog

Configure WSO2 Identity Server for single sign on with Moodle - Part I

Introduction Moodle is a learning platform designed to provide educators, administrators and learners with a single robust, secure and integrated system to create personalized learning environments.[1]. This is commonly used in the education sector and there might be occurrences where it would require to use SAML to login to Moodle instead of the default basic authentication. For this we would need to have an Identity Provider capable of issuing such SAML tokens and Moodle configured to accept the tokens in order to do the authentication I will be discussing this in two parts. Part I - How to configure moodle for SSO with WSO2 Identity Server Part II - How to carry our user provisioning and attribute profile mapping with Moodle and WSO2 Identity Server Let's see how this can be achieved using a SAML authentication plugin provided by one login. Prerequisite  You need to have a working version of Moodle installed. If you want to try this tutorial from scratch you can

Exposing a SOAP service as a REST API

In this post i will be explaining how we can transform a SOAP based backend to receive requests in a restful manner through the WSO2 API Cloud. Steps. First log into the WSO2 Cloud and navigate to the API Cloud. In the API cloud select the option to add a new API. We will be creating an API to demonstrate an invocation to the backend soap service ws.cdyne.com/phoneverify/phoneverify.asmx?wsdl Give a name, context and version to the API and add a resource name with a GET Method. The resource name can be anything which you like since we will invoke the actual service usiing a custom sequence. Mention the URI template as indicated in the below screenshot. Next go to the implement tab. And select the endpoint type as HTTP/SOAP Endpoint and specify the endpoint as http://ws.cdyne.com/phoneverify/phoneverify.asmx. There is an important step we need to do here. We need to set the SOAP version for this request. In order to do that we need to select the advanced option for the e