Camel development series 9

Welcome again to another post regarding development with the Apache Camel framework. In this series I will cover a couple of different things that I have encountered or developed which may be of benefit to you.

Camel demo project on Github

I have updated my github page at https://github.com/SoucianceEqdamRashti/Integration and removed some old projects and added two new ones.

CamelDemo

The CamelDemo project contains a larger maven project based on camel-blueprint-archetype version 2.17.0. It consists of a series of routes that aim to show in a very simple manner the basics of some of the most common Camel functionalities. This includes features such as:

  1. Content-based routing
  2. Splitting csv, xml and json data
  3. Rest-dsl
  4. File handling
  5. Timer based routes
  6. Camel unit tests

More features will be added in the near future. But I think it is a pretty good place to start if you are a complete newbie to Camel to just get a feel for the framework and what you can do with very little code. It is indeed a very powerful framework.

CustomKaraf

This is also a maven project based on Karaf 4.0.2. This is a really cool and powerful feature that I discovered a month or so ago and is extremely powerful when developing micro-services in isolation.

In short what this does is to allow you to build and distribute a Karaf distribution customized to your needs and include all the bundles  you desire on boot level. In practical terms, this means that you tell the maven karaf plugin which bundles/features you are interested in and add them to boot level.

Then you run maven:clean install and the plugin creates a zip and a tar.gz file in the target directory. Choose whichever format you desire. Then all you do is move that distribution to your server or docker image and unzip and run /bin/karaf. Then when Karaf boots up the bundles, including Camel and your Camel routes will already be started.

Take a look at the project for an overview. You should be able to build it and have Camel installed at boot level.

Running Camel for OSGI in Intellij

Normally when I develop a project with Camel I always know I will run it in Karaf. Therefore I start with template based on camel-blueprint-archetype. Before version 2.17.0 I didn’t have to change anything with logging. Just let maven do its magic and add the components I want and develop.

However since version 2.17.0 it seems new features to the archetype have been added. Now when you create a new project based on that archetype and then run the generated project you get this error:


[INFO] --- camel-maven-plugin:2.17.0:run (default-cli) @ email-bibsent ---
[INFO] camel-blueprint detected on classpath
[INFO] OSGi Blueprint XML files detected in directory C:\Work\repo\LSP\integration-platform\integrations\PollEmailFromBibsent\src\main\resources\OSGI-INF\blueprint
[INFO] Using org.apache.camel.test.blueprint.Main to initiate a CamelContext
[INFO] Starting Camel ...
SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
SLF4J: Defaulting to no-operation (NOP) logger implementation
SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.

Essentially Camel won’t start since the sl4j logger is not found. I then found that you can solve this by adding the following dependencies:


    <dependency>
      <groupId>org.slf4j</groupId>
      <artifactId>slf4j-api</artifactId>
      <version>${slf4j-version}</version>
    </dependency>
    <dependency>
      <groupId>org.slf4j</groupId>
      <artifactId>slf4j-simple</artifactId>
      <version>${slf4j-version}</version>
    </dependency> 

But then I thought this can’t be quit right so I posted this question on the Camel nabble forum. The answer I got from Claus Ibsen was that the logger dependency from Camel is not present in that archetype because Karaf or any other OSGI container has its own logger.

What this means is, add those dependencies when you want to run the route in Intellij for testing, but then change the scope to

<test>

when you want to deploy them to Karaf or to your maven repository.

Camel development series part 8

In this series I will simply touch upon a few issues I encountered during coding that you might find useful. For reference this is on Camel version 2.17.0.

PollEnrich will not move file after completion

So I had a route where in the middle of the route I had to enrich it with a file, and after consuming the file, and the route finished it should move it to the a specific folder with a timestamp. The route looked like this:

 

from(....)
  ...
  ...
 .pollEnrich().simple("file:" + location + "?fileName=${header.File}&charset=iso-8859-1&" +
            "move=../archive/${file:name.noext}-${date:now:yyyyMMddHHmm}.${file:ext}")

We are using simple because we want pollEnrich to evaulate dynamic endpoints from headers. This can be found in the documentation. However, what actually happened was that the pollEnrich created the archive folder and the completely ignore the file expression language and just created a folder with the timestamp. No matter how hard I tried to change it still produced an output similar to this. My intention was to create a folder called archive and in there save the files with a timestamp attached to their name.

The problem was that the ”?move=” part doesn’t evaluate to anything meaningful at the point it’s evaluated. The ”move=” expressions need to be evaluated after the file has been picked up, but they are being evaluated before the file is read.

So my workaround was to replace the pollEnrich with the following:

 ConsumerTemplate consumer = exchange.getContext().createConsumerTemplate();
    String fileName = exchange.getIn().getHeader("File", String.class);
    String fileUri = "file://" + location + "?fileName=" + fileName + "&charset=iso-8859-1&move=archive/${file:name.noext}-${date:now:yyyyMMddHHmmssSSS}.${file:ext}&moveFailed=failed/${file:name.noext}-${date:now:yyyyMMddHHmmssSSS}.${file:ext}";
byte[] fileBody = consumer.receiveBody(fileUri,5000, byte[].class);

The above code resides in a processor. Basically I had to revert to a ConsumerTemplate instead. This works fine, it was just confusing that the pollEnrich didn’t work with dynamic properties.

Building predicates from exchange properties

Exchange properties are very useful when you want to pass properties from one exchange to another. Predicates are very powerful to build rich expressions in order to make to complex statements. These two can then be combined into something as simple as:

    Predicate success = exchangeProperty("Status").isEqualTo("true");
Predicate failure = exchangeProperty("Status").isEqualTo("false");
.......
.........
from(..)
..
..
.choice()
.when(success)
.otherwise()
.when(failure)
..

The code statement is easier to read and you have replaced a complex statement with a simpler one that can be used in the dsl. This is something really cool about Camel.

Splitting large files

Suppose you have a big file that you want to read. Maybe you have millions of rows of CSV data and you want to split them by a certain number or simply one by one.Since each row should be handled the same, you also want things to be done in parallel.

The most performance optimal way can be expressed like this:

from("file:...")
.split().tokenize(System.lineSeparator(), 1000).streaming()
//use threading to handle the different split chunks in parallel
.threads(20, 50)
.to("someProcessor");

So, what we have done is to split based on a token. In this case it is by the operating system new line token e.g. in windows and in linux world. We also group them such that each split shall contain 1000 rows. Then we do streaming so that lazy parsing is done to save memory and finally we add threads so things can be done in parallel.

Specify exchange pattern and delay

The last two simple tips are for when you want to specify the exchange pattern directly as an endpoint and for introducing a simple delay in ms.

//specify the exhange pattern
.inOnly("seda:outputQueue")
//introduce a delay of 100 ms
 .delay(100)

As you can see we can write .inOnly or .inOut and then provide the endpoint. This can be good if you want to do some processing but don’t want to wait for a response, instead want to send an immediate reply to caller.

The delay is simple, simply add the value and you have your delay!

Camel development series part 7

In this post I will go through how I found a usefully way of handling asynchronous error handling when working with Apache Camel.

The requirements are as follows.

  1. We have a bunch of routes chained together who exist in different routeBuilder classes.
  2. The client does not require a response back so there is no need for an synchronous connection.
  3. We want a generic error handling strategy as we don’t want to replicate the error handling code in each routeBuilder class.

Essentially we want a global exception handler applied to all routeBuilder classes in a uniform way. Some months ago I faced the above requirements and googled a bit and found this post on stackoverflow:

http://stackoverflow.com/questions/9283861/camel-exception-handling-doesnt-work-if-exception-clause-is-defined-in-a-separat

This helped a lot accomplishing what I wanted. So let’s break it down:

Standard routeBuilder class

 

public class MyRouteBuilder extends RouteBuilder {
  
  @Override
  public void configure() throws Exception {
	ExceptionBuilder.setup(this);
	from()...to();
	}
}

Notice, that we have added the line

 ExceptionBuilder.setup(this);

which sends this instance of the RouteBuilder class to our ExceptionBuilder.

Now off course we need to create our ExceptionBuilder class. This is how I have done:

public class ExceptionBuilder {

    public static void setup(RouteBuilder routeBuilder) {
    routeBuilder.onException(Exception.class).useOriginalMessage().handled(true).to("direct://ErrorHandler");
  }  
}

I have created a standard java class which has a setup method accepting a RouteBuilder object as a parameter.

It then configures the .onException handling and directs the exchange to the ErrorHandler route. The configuration I have used is based on my needs, you can configure it your way. I want to listen for all possible exceptions, and use the original message and handle the exception which is why it looks like that. Now all we have left is to create our ErrorHandler route.

public class ErrorHandler extends RouteBuilder {

  @Override
  public void configure() throws Exception {
    String hashSeparator = "##################################################################################################################";
    String lineBreak = System.lineSeparator();
    from("direct://ErrorHandler").startupOrder(1).routeId("ErrorHandler")
      .log(LoggingLevel.ERROR, "ErrorHandler", hashSeparator)
      .log(LoggingLevel.ERROR, "ErrorHandler", "An error occured in the integration integration" + lineBreak)
      .setHeader("CamelMessageId", simple("${id}"))
      .setHeader("CamelContextId", simple("${camelId}"))
      .setHeader("CamelCreatedTimestamp", simple("${property.CamelCreatedTimestamp}"))
      .setHeader("CamelExceptionCaught", simple("Exception Object: ${property.CamelExceptionCaught}"))
      .setHeader("CamelStacktrace", simple("${exception.stacktrace}"))
      .setHeader("ExceptionMessage", simple("${exception.message}"))
      .setHeader("CamelFailureEndpoint", simple("${property.CamelFailureEndpoint}"))
      .setHeader("CamelFailureRouteId", simple("${property.CamelFailureRouteId}"))
      .log(LoggingLevel.ERROR, "ErrorHandler", " Headers: ${headers}" + lineBreak + lineBreak)
      .log(LoggingLevel.ERROR, "ErrorHandler", " Body: ${body}" + lineBreak)
      .log(LoggingLevel.ERROR, "ErrorHandler", hashSeparator);
  }
}

The ErrorHandler route is a normal route but with an explicit startup order and then I log a bunch of specific headers and set other headers. You can add and remove stuff as you please and as you want your error log output to look.

Combining the three steps we have seen gives us a powerful way to do error handling in a generic manner applied to all our RouteBuilder classes. This is especially useful for integrations where the client doesn’t need a response and the error handling is the same for all routes.
 

Camel development series part 6

It has been months since I wrote a blog on Camel. This has been mainly due to a lot of activity in my personal and professional life but I thought would get back on track with the blog.

In today post we will look at some basic beginner ”best practice”. Note, if someone reading disagree then please do comment as it is always interesting to receive constructive feedback.

Always name your CamelContext.

  • It does not matter if you are writing in pure java or a mix of blueprint and java. Always name your CamelContext.
  • It makes writing and reading logs easier as there is now a user friendly name to look out for.
  • It gives your reader an understanding of what this Context is supposed to do.
  • It helps to get into the spirit of naming your Camel objects such as routes.
  • Code example:

    CamelContext context = new DefaultCamelContext();
    context.setName("MyCamelContext");
    

    #Blueprint version

    <blueprint xmlns="http://www.osgi.org/xmlns/blueprint/v1.0.0">
     
        <camelContext xmlns="http://camel.apache.org/schema/blueprint" name="MyCamelContext">
            <route>
                <from uri="timer:test" />
                <to uri="log:test" />
            </route>
        </camelContext>
     
    </blueprint>
    &nbsp;
    

    Always give a routeId to your routes.

  • Give a meaningful id to your route. Amongst others, it helps to show which routes have started when the camel context starts. Otherwise you will simply see route1, route2, route3 without know which of these match your routes.
  • It makes it much easier to write log statements if you have a routeId. Each log statement that belongs to a route can use the routeId to put the statements in their category. This makes it much easier to know which route publish which log statement.
  • It gives you an enormous benefit during testing because you can intercept, replace and do other things based on your routeId. I do a lot of replacement during tests for example I replace my eventbus route with a hardcoded response. You need a routeId for this.
  • Always give an id to statements that are configurable.

  • For example, I may send my exchange to a bean method or to a processor or to another route via direct. Give meaningful id names to each .to() statement. It makes testing so much easier as you can intercept and replace .to() endpoints and that is a big bonus during test.
  • Code example:

    from("inputUri).routeId("MyRouteId").to("outputUri).id("MyId");
    

    Keep inline processors short

  • In camel you can create inline processors to write normal java code and manipulate whatever is on the exchange or the headers. This is good for quick and dirty operations but if you are doing complex business logic or operations that should be configurable or hidden then create a Processor class instead and invoke that.
  • Keeping shorter inline processor code makes reading the camel dsl code easier as well.
  • By putting complex resuable code in a Processor class you ensure that other routes can call it as well. This means you write code once and call it from other parts of the CamelContext. You don’t then need to write same inline code everywhere.
  • Code example:

    from("activemq:myQueue").process(new Processor() {
        public void process(Exchange exchange) throws Exception {
            String payload = exchange.getIn().getBody(String.class);
            // do operations on the payload that should only happen at this //point in the route and only in this route.
    // Add simple logic for e.g. header or body manipulation.
    //Complex logic goes in separate Processor class.
           exchange.getIn().setBody("Changed body");
       }
    }).to("activemq:myOtherQueue");
    
    #Processor class
    public class MyProcessor implements Processor {
      public void process(Exchange exchange) throws Exception {
        String body = exchange.getIn().getBody().toString();
        body="changed";
    exchange.getIn().setBody(body);
    //Create a class when you need to class the Processor class from several parts of your route or routes or has configurable parts or contains complex
    business logic.
    
      }
    }
    

    Generate your Camel endpoint uris

  • Your endpoint uri whether in a from() or to() should be generated or injected rather than hardcoded.
  • Avoid from(”file://test/?fileName=test.txt”). Instead do from(fileUri) where fileUri is created in a utility class, some bean method or injected via a property. The same goes for .to() endpoints
  • Generating uri makes testing much easier because you can generate other test endpoints and simply inject those instead and never touch your main uris. If you have hardcoded uri the same is possible but cumbersome.
  • In particular if you have different uri for different environment then you cannot hardcode them otherwise you are creating different code for different environments and that is a bad practise. Here you defintely need to inject them via property file determined by some environment variable that tells you which environment the context is running in.
  • Code example:

    @PropertyInject("{{fileUri}}")
    private String fileUri;
    @PropertyInject("{{toUri}}")
    private String toUri;
    ...
    ...
    from(fileUri).to(toUri);
    

    Don’t complicate your logs

  • Your log files should contain at least two levels. One at debug level intended for technical minded people who wanted in depth knowledge of what is going on. Then you have logs at info level which should only give a brief description of what has happened. The main steps in a route should be at info level. The details of the exchange should be at debug level.
  • Do not put exchange headers, body or properties at info level at least it is of buisiness importance. Info level logs should be of the form:
    Order 123 generated. Customer request received. Message published. No technical details present.
  • At debug level you should log all aspects of the exchange to provide maximum details for debugging and troubleshooting. That means log ${headers}, ${body} and ${properties} where applicable. Because debug levels are not on by default the log files will not get massive straight away.
  • Link the log statements with the routeId so you know which route generated that statement. This makes it easier to backtrack from the logs to your code.
  • Code example:

    #Don't do this. It contains unnecessary amount of information.
    from(fileUri).routeId("MyRouteId").log(LoggingLevel.INFO,"MyRouteId", "Order ${body} received").to(toUri);
    #Log like this.
    from(fileUri).routeId("MyRouteId").log(LoggingLevel.INFO,"MyRouteId", "Order from customer received").log(LoggingLevel.DEBUG, "MyRouteId", "Order from customer with body: ${body} and headers ${headers}").to(toUri);
    

    Split your logic into several routes

  • Yes, you can create a giantic 1000 line log Camel route just as you can with normal java class but it is not considered good practice.
  • Decide how your problem can be proken into individual parts, set each parts action to be done by a specific route and chain the routes together.
  • Chaining routes means you can replace routes during tests or due to requirements changes which is harder to do when all the code is in one giant route
  • Chaining routes makes it easier to understand the different components and you can programmatically or via external commands stop individual routes for troubleshooting. You cannot do this if all the code exist in one giant route.
  • Use common sense off course. You can always start with a big route in order to check your code and then split the code in smaller routes. It is more important to get the coding working first and then get it more coherent then start with the optimization.
  • Code example:

    #Don't do this. It is a massive route.
    from(fileUri).routeId("MyRouteId").log(LoggingLevel.INFO,"MyRouteId", "Order ${body} received").to(eventBusUri).convertBodyTo(String.class).removeHeaders("eventbusHeaders").to()....;
    #Do more like this as each step is broken down and the implementation is written in individual routes
    from(fileUri).routeId("MyRouteId").to(eventBusRoute).to(processOrderRoute).to(generateReceiptRoute).to(eventbusRoute).to(saveReceiptRoute);
    

    Camel development series part 5

    Hello, and welcome back to the Camel development series and to part 5. Its been a couple of weeks since I wrote and this has been mainly due to lack of time but hopefully I can get back to track.

    In this series we are going to look at a classic case of content based routing. This means, based on some criteria the input data should be transformed/routed in one way and on another criteria transformed/routed in another way.

    We will build up on our previous work but this will be in a new project. You can find all the source code here https://github.com/SoucianceEqdamRashti/Integration/tree/master/CamelContentbasedRouter .

    Problem

    Let’s say we have a file placed in some folder. Based on the file name we want to either map it to XML or to JSON. The input format is basic CSV.

    1. If the file name contains the string ”xml” then the data should be mapped to XML.
    2. If the file name contains the string ”json” then the data should be mapped to JSON.

    Our end result should produce an XML or a JSON based on the file name.

    Solution

    We will introduce a few new concepts there. One of the most powerful ideas in the Camel world is the use of Predicates. You can read more about it here http://camel.apache.org/predicate.html . Predicates allow you to define complicated conditions upon which to based your logic and choices. We will use them here to make choices. We will also look at how to send and use bean transformation to work together with JAXB. Finally we will also look at how to read the file name and how to actually map to XML from CSV.

    Main RouteBuilder class

    Let us first look at the main RouteBuilder class to get an idea how the main flow looks like.

    package org.souciance.integration.contentbasedrouter;
    
    import org.apache.camel.LoggingLevel;
    import org.apache.camel.Predicate;
    import org.apache.camel.PropertyInject;
    import org.apache.camel.builder.RouteBuilder;
    import org.apache.camel.converter.jaxb.JaxbDataFormat;
    import org.apache.camel.dataformat.bindy.csv.BindyCsvDataFormat;
    import java.lang.String;
    
    public class MainRouteBuilder extends RouteBuilder {
    @PropertyInject("{{input.folder}}")
    private String inputFolder;
    @PropertyInject("{{output.folder}}")
    private String outputFolder;
    @PropertyInject("{{input.filename}}")
    private String inputFileName;
    @PropertyInject("{{output.filename}}")
    private String outputFileName;
    
    @Override
    public void configure() throws Exception {
    
    Predicate toXML = header("CamelFileName").contains(String.valueOf("xml"));
    Predicate toJSON = header("CamelFileName").contains(String.valueOf("json"));
    
    BindyCsvDataFormat bindy = new BindyCsvDataFormat(Person.class);
    JaxbDataFormat jaxbDataFormat = new JaxbDataFormat(Persons.class.getPackage().getName());
    jaxbDataFormat.setPrettyPrint(true);
    from("file://" + inputFolder+"?include=.*.csv").routeId("ContentbasedRouter")
    .log(LoggingLevel.INFO, "body","body: ${body}")
    .unmarshal(bindy)
    .choice()
    .when(toXML)
    .log(LoggingLevel.INFO, "Data will be mapped to xml structure")
    .bean(Transform.class)
    .marshal(jaxbDataFormat)
    .log(LoggingLevel.INFO, "Data will be mapped to xml structure: ${body}")
    .to("file://"+outputFolder+"/?fileName=output.xml")
    .when(toJSON)
    .log(LoggingLevel.INFO, "Data will be mapped to json structure.")
    .unmarshal(bindy)
    .to("PersonMapperToJson")
    .log(LoggingLevel.INFO, "PersonMapperToJson", "Mapped Pojo to json: ${body}")
    .to("file://"+outputFolder+"/?fileName="+"output.json")
    .endChoice()
    .end();
    
    }
    }
    
    

    The main idea should be readable even if you don’t know all the details. We are polling a directory. If a file with extension .csv is placed in the directory we look at the file name. Here is where predicates come in handy. Let’s look at it in more detail:

    Predicate toXML = header("CamelFileName").contains(String.valueOf("xml"));
    Predicate toJSON = header("CamelFileName").contains(String.valueOf("json"));
    

    We have defined two predicates, and each one of them looks at the header named CamelFileName. This is a standard Camel header where the name of the file is placed. In the rule we check if the value xml or json is found. These are examples of simple predicates, you can build much more complicated rules and also combine them with property files to make dynamic rules.

    Next let’s look at the new idea of DataFormat and especially the jaxb data format.

    JaxbDataFormat jaxbDataFormat = new JaxbDataFormat(Persons.class.getPackage().getName());
    jaxbDataFormat.setPrettyPrint(true);
    

    DataFormats are what allows Camel to recognize formats and transform or otherwise known as marshal/unmarshal data. We need two types of data formats in our case. One is the one we have introduced earlier which is the Bindy format to parse CSV into POJO. The second seen above is the JAXB dataformat to transform the list of POJOs to XML. For more on data formats read here https://camel.apache.org/data-format.html .

    The second new part and where all the interesting logic happens is here:

    .choice()
    .when(toXML)
    .log(LoggingLevel.INFO, "Data will be mapped to xml structure")
    .bean(Transform.class)
    .marshal(jaxbDataFormat)
    .log(LoggingLevel.INFO, "Data will be mapped to xml structure: ${body}")
    .to("file://"+outputFolder+"/?fileName=output.xml")
    

    You can see that we use the .choice() tag to start our content based routing. We use the .when(toXML) to use the predicate defined earlier. Essentially this means in plain English, when the predicate toXML holds true enter this block. After the log statement you notice the .bean() tag and this is an interesting tag. It allows us to send the exchange to the bean and do whatever we want with it and then continue. In our cause we are going to do some JAXB logic in the transform bean.
    Then finally we have the marshal statement which refers to the jaxb data format. The last two are simple log and writing to file statements.

    Let’s look at our Transform class

    Transform Bean

    Our transform bean looks as follows:

    package org.souciance.integration.contentbasedrouter;
    
    import java.util.List;
    import org.apache.camel.Exchange;
    import org.apache.camel.Message;
    
    public class Transform {
    
    public void process(Exchange exchange) throws Exception {
    Message msg = exchange.getIn();
    
    @SuppressWarnings("unchecked")
    List personList = (List) msg.getBody();
    
    ObjectFactory objectFactory = new ObjectFactory();
    
    Persons persons = objectFactory.createPersons();
    
    for (Person aPerson : personList) {
    persons.setPerson(aPerson);
    
    exchange.getIn().setBody(persons);
    }
    
    }
    
    }
    

    The code is essentially receiving a list of Person POJOs which were created when marshalling to the bindy format. Then an instance of ObjectFactory is created and we are adding each POJO from the list to the ArrayList in Persons class. Finally we set the exchange to contain the Persons list.

    The remaining parts are the annotated classes and the ObjectFactory class. So let’s look at them.

    ObjectFactory

    The object factory class and the annotated classes were inspired by the post found here http://www.developpez.net/forums/d1216465/java/serveurs-conteneurs-java-ee/autres/camel-convertir-csv-xml-bindy-jaxb/ . Although it is in French, the gist of the idea should be comprehensible.

    Here is our class:

    //
    // This file was generated by the JavaTM Architecture for XML Binding(JAXB) Reference Implementation, v2.2.8-b130911.1802
    // See <a href="http://java.sun.com/xml/jaxb">http://java.sun.com/xml/jaxb</a>
    // Any modifications to this file will be lost upon recompilation of the source schema.
    // Generated on: 2016.03.05 at 06:27:39 PM CET
    //
    
    package org.souciance.integration.contentbasedrouter;
    
    import javax.xml.bind.annotation.XmlRegistry;
    
    /**
    * This object contains factory methods for each
    * Java content interface and Java element interface
    * generated in the org.souciance.integration.contentbasedrouter.jaxb package.
    *
    
    An ObjectFactory allows you to programatically
    * construct new instances of the Java representation
    * for XML content. The Java representation of XML
    * content can consist of schema derived interfaces
    * and classes representing the binding of schema
    * type definitions, element declarations and model
    * groups. Factory methods for each of these are
    * provided in this class.
    *
    */
    @XmlRegistry
    public class ObjectFactory {
    
    /**
    * Create a new ObjectFactory that can be used to create new instances of schema derived classes for package: org.souciance.integration.contentbasedrouter.jaxb
    *
    */
    public ObjectFactory() {
    }
    
    /**
    * Create an instance of {@link Persons }
    *
    */
    public Persons createPersons() {
    return new Persons();
    }
    
    /**
    * Create an instance of {@link Persons.Person }
    *
    */
    public Person createPerson() {
    return new Person();
    }
    
    }
    

    Annotated classes

    We need annotated classes for bindy and jaxb processing. For completeness sake here they are.

    Persons class

    //
    // This file was generated by the JavaTM Architecture for XML Binding(JAXB) Reference Implementation, v2.2.8-b130911.1802
    // See <a href="http://java.sun.com/xml/jaxb">http://java.sun.com/xml/jaxb</a>
    // Any modifications to this file will be lost upon recompilation of the source schema.
    // Generated on: 2016.03.05 at 06:27:39 PM CET
    //
    
    package org.souciance.integration.contentbasedrouter;
    
    import java.util.ArrayList;
    import java.util.List;
    import javax.xml.bind.annotation.XmlAccessType;
    import javax.xml.bind.annotation.XmlAccessorType;
    import javax.xml.bind.annotation.XmlElement;
    import javax.xml.bind.annotation.XmlRootElement;
    import javax.xml.bind.annotation.XmlType;
    
    /**
    *
    
    Java class for anonymous complex type.
    *
    *
    
    The following schema fragment specifies the expected content contained within this class.
    *
    *
    <pre> * &lt;complexType&gt;
     *   &lt;complexContent&gt;
     *     &lt;restriction base="{http://www.w3.org/2001/XMLSchema}anyType"&gt;
     *       &lt;sequence&gt;
     *         &lt;element name="person" maxOccurs="unbounded" minOccurs="0"&gt;
     *           &lt;complexType&gt;
     *             &lt;complexContent&gt;
     *               &lt;restriction base="{http://www.w3.org/2001/XMLSchema}anyType"&gt;
     *                 &lt;sequence&gt;
     *                   &lt;element name="firstname" type="{http://www.w3.org/2001/XMLSchema}string"/&gt;
     *                   &lt;element name="lastname" type="{http://www.w3.org/2001/XMLSchema}string"/&gt;
     *                   &lt;element name="identity" type="{http://www.w3.org/2001/XMLSchema}string"/&gt;
     *                   &lt;element name="country" type="{http://www.w3.org/2001/XMLSchema}string"/&gt;
     *                   &lt;element name="phone" type="{http://www.w3.org/2001/XMLSchema}string"/&gt;
     *                 &lt;/sequence&gt;
     *               &lt;/restriction&gt;
     *             &lt;/complexContent&gt;
     *           &lt;/complexType&gt;
     *         &lt;/element&gt;
     *       &lt;/sequence&gt;
     *     &lt;/restriction&gt;
     *   &lt;/complexContent&gt;
     * &lt;/complexType&gt;
     *</pre>
    *
    *
    */
    @XmlAccessorType(XmlAccessType.FIELD)
    @XmlType(name = "", propOrder = {
    "person"
    })
    @XmlRootElement(name = "persons")
    public class Persons {
    @XmlElement(required = false, name = "person")
    protected List person;
    
    public List getPerson() {
    if (person == null) {
    person = new ArrayList();
    }
    return this.person;
    }
    public void setPerson(Person person){
    this.getPerson().add(person);
    }
    
    }
    

    Person class

    package org.souciance.integration.contentbasedrouter;
    
    import javax.xml.bind.annotation.XmlAccessorType;
    import javax.xml.bind.annotation.XmlElement;
    import javax.xml.bind.annotation.XmlType;
    import javax.xml.bind.annotation.XmlAccessType;
    import org.apache.camel.dataformat.bindy.annotation.CsvRecord;
    import org.apache.camel.dataformat.bindy.annotation.DataField;
    
    @CsvRecord(separator = ",")
    
    @XmlAccessorType(XmlAccessType.FIELD)
    @XmlType(name="person")
    public class Person {
    
    @XmlElement(name = "identity")
    @DataField(pos=1)
    private int identity;
    
    @XmlElement(name = "firstname")
    @DataField(pos=2)
    private String firstname;
    
    @XmlElement(name = "lastname")
    @DataField(pos=3)
    private String lastname;
    
    @XmlElement(name = "phonenumber")
    @DataField(pos=4)
    private int phone;
    
    @XmlElement(name = "country")
    @DataField(pos=5)
    private String country;
    
    public int getIdentity() {
    return identity;
    }
    public void setIdentity(int identity) {
    this.identity = identity;
    }
    public String getFirstname() {
    return firstname;
    }
    public void setFirstname(String firstname) {
    this.firstname = firstname;
    }
    public String getLastname() {
    return lastname;
    }
    public void setLastname(String lastname) {
    this.lastname = lastname;
    }
    public int getPhone() {
    return phone;
    }
    public void setPhone(int phone) {
    this.phone = phone;
    }
    public String getCountry() {
    return country;
    }
    public void setCountry(String country) {
    this.country = country;
    }
    
    }
    

    Summary

    In this project we have looked at a common problem, how to transform and route data in different ways based on some condition. We defined predicates to define our condition rules, we used the choice and when tags to check our predicates, we use bean transformation to connect our jaxb processing and then we marshalled  the list of objects to XML using jaxb marshalling.

    The input data looked like this:
    12345,souciance,rashti,012458478,Sweden
    12345,souciance,rashti,012458478,Sweden

    The filename was csvtoxml.csv

    The final output is:

    <?xml version="1.0" encoding="UTF-8" standalone="yes"?>
    <persons>
        <person>
            <identity>12345</identity>
            <firstname>souciance</firstname>
            <lastname>rashti</lastname>
            <phonenumber>12458478</phonenumber>
            <country>Sweden</country>
        </person>
        <person>
            <identity>12345</identity>
            <firstname>souciance</firstname>
            <lastname>rashti</lastname>
            <phonenumber>12458478</phonenumber>
            <country>Sweden</country>
        </person>
    </persons>
    

    It works similarly for the second when block for the JSON which is exactly as the previous series.

    Camel development series – part 4

    Welcome again to this fourth installed of the Camel development series. It is almost 1am but am determined to write one post a week as to not fall behind my original goal. This post will not cover a lot of functionality but will touch upon two parts which I think are essentially to developing production ready code. These are:

    1. ensuring single copy of deployment. What I mean here is that we don’t want to develop one copy for our test environment, one copy for our QA and one for production environment. We want a single copy which can be deployed to all of our environments, thereby ensuring consistency and knowing that what is in production is the same as what is in test in QA. It has therefore been tested and verified. To achieve this, we need a way to hide away endpoint parameters and hence deal with property files.
    2. producing log files. To track our integrations we need to know what they have done and off course if something has gone wrong, the simplest way is to produce log files and write logs during important steps. We will cover this step. In a later post I will go through MDC logging when you deploy to a Karaf environment so that you can produce log files based on a route or context id which gives you better granularity.

    Ok let’s start with the actually code and return to our standard project. You will find the source here https://github.com/SoucianceEqdamRashti/Integration/tree/master/MapCsvsToJson

    Blueprint code

     

    Here is our blueprint.xml code:

    <?xml version="1.0" encoding="UTF-8"?>
    <blueprint xmlns="http://www.osgi.org/xmlns/blueprint/v1.0.0"        xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"        xsi:schemaLocation="        http://www.osgi.org/xmlns/blueprint/v1.0.0 http://www.osgi.org/xmlns/blueprint/v1.0.0/blueprint.xsd        http://camel.apache.org/schema/blueprint http://camel.apache.org/schema/blueprint/camel-blueprint.xsd">
    
      <bean id="CsvToJsonRouteBuilder" class="org.souciance.integration.csvstojson.CsvsToJsonRouteBuilder">
      </bean>
      <bean id="IdentityToJson" class="org.souciance.integration.csvstojson.IdentityToJson">
      </bean>
      <bean id="properties"	class="org.apache.camel.component.properties.PropertiesComponent">
    		<property name="location" value="classpath:MapCsvsToJson.properties" />
    </bean>
    
      <camelContext xmlns="http://camel.apache.org/schema/blueprint" useMDCLogging="true" id="CsvToJson">
        <routeBuilder ref="CsvToJsonRouteBuilder" />    
    
      </camelContext>
    
    </blueprint>
    

    There are two additions here. One is that we have added useMDCLogging=”true” attribute on the camelContext tag. We will use this later in another post. But more importantly we have added the bean for reading properties. You can see that there is bean declared with id=”properties” and the name of the properties field is MapCsvsToJson.properties and it should be present somewhere on the classpath. Off course in real life you follow some naming convention but this is just tho show how it works. The beauty of this approach is that it enables us to inject any kind of property values onto java class member fields. This is in essence how we enable our goal of single copy.

    MapCsvsToJson.properties

    We have also created a properties file called MapCsvsToJson.properties and it is in our src/resources folder. The properties defined are:
    input.folder =C:/test
    output.folder =C:/test
    input.filename =input.csv
    output.filename =output.json

    As you can see we have written down filepath and filenames for our input and output. You can easily see that this can be extended to any endpoint be it HTTP, FTP, WMQ, Rabbit, TCP etc. This means that the actual integration code remains the same, you just change the endpoint when you move your package from test environment to qa and add the qa endpoints and same when you go to production. You can add other system parameters here which may need to change from one environment to another or to make your code more flexible.

    MapCsvsToJsonRouteBuilder.java

    In our route builder class we have made some additions. Here is how it looks now:

    package org.souciance.integration.csvstojson;
    
    import org.apache.camel.LoggingLevel;
    import org.apache.camel.PropertyInject;
    import org.apache.camel.builder.RouteBuilder;
    import org.apache.camel.dataformat.bindy.csv.BindyCsvDataFormat;
    import org.slf4j.Logger;
    import org.slf4j.LoggerFactory;
    
    /**
     * A bean which we use in the route
     */
    public class CsvsToJsonRouteBuilder extends RouteBuilder {
    	@PropertyInject("{{input.folder}}")
    	private String inputFolder;
    	@PropertyInject("{{output.folder}}")
    	private String outputFolder;
    	@PropertyInject("{{input.filename}}")
    	private String inputFileName;
    	@PropertyInject("{{output.filename}}")
    	private String outputFileName;
    
    	@Override
    	public void configure() throws Exception {
    		// TODO Auto-generated method stub
    		BindyCsvDataFormat bindy = new BindyCsvDataFormat(org.souciance.integration.csvstojson.Identity.class);		
    
    		from("file:" + inputFolder+"/?fileName="+inputFileName).routeId("CsvsToJson")
    		.log(LoggingLevel.INFO, "CsvToJson", "Processing csv message ${id} with body ${body}")
    		.unmarshal(bindy)
    		.log(LoggingLevel.INFO, "CsvToJson", "Mapped CSV to Pojo ${body}")
    		.to("IdentityToJson")
    		.log(LoggingLevel.INFO, "CsvToJson", "Mapped Pojo to json: ${body}")
    		.to("file:"+outputFolder+"/?fileName="+outputFileName)
    		.log(LoggingLevel.INFO, "CsvToJson", "Json file written to output folder found in headers ${in.headers}")
    		.end();
    
    	}
    }
    

    As you can see we have now written the following when declaring our class member fields:

    @PropertyInject("{{input.folder}}")
    	private String inputFolder;
    

    What this does is to inject the value of property input.folder which is C:/test to the field inputFolder. The same goes for the other variables.

    Further down you can see this:

    from("file:" + inputFolder+"/?fileName="+inputFileName).routeId("CsvsToJson")
    

    This is now different from our previous code. We have hidden the input folder and the input file name. In some sense we are creating a dynamic endpoint because the actual endpoint URI are retrieved from our property file. So again, imagine having a properties file for test, qa and prod and simply deploying a single copy with the right property file rather than multiple copies of your integration for each environment. The same applies to the ”to” endpoint at the bottom. This demonstrates the main functionality of property injection, you can off course inject to normal POJO classes or even to input fields of a method. There is a rich complexity here not found in the commercial tools.

    Ok let us also mention the code we added for the logging. You have noticed several statements of the form:

    .log(LoggingLevel.INFO, "CsvToJson", "Processing csv message ${id} with body ${body}")
    

    Here we are using the Log dsl to do a few things. First, we set the logging level to INFO as we are just logging steps. We are giving the logger a name corresponding to the appender name in the log4j properties file. Finally we have some log text. Here we are logging the camel id of the message and the body. The language is called simple language and is a kind of camel query language where you can access the body, headers and camel properties. To do that you write ${variable}. As you see we are logging when we receive the file, after we mapped to a POJO, after we mapped to Json and after we have written the output file.

    The log4j properties looks like this:

    log4j.rootLogger=INFO, out, CsvsToJson
    #log4j.logger.org.apache.camel=DEBUG, file

    # CONSOLE appender
    log4j.appender.out=org.apache.log4j.ConsoleAppender
    log4j.appender.out.layout=org.apache.log4j.PatternLayout
    log4j.appender.out.layout.ConversionPattern=[%30.30t] %-30.30c{1} %-5p %m%n
    log4j.appender.out.layout.ConversionPattern=%d [%-15.15t] %-5p %-30.30c{1} – %m%n

    log4j.appender.CsvsToJson=org.apache.log4j.RollingFileAppender
    log4j.appender.CsvsToJson.layout=org.apache.log4j.PatternLayout
    log4j.appender.CsvsToJson.layout.ConversionPattern=%d{ISO8601} | %-5.5p | %-16.16t | %-32.32c{1} | %X{camel.routeId} | %m%n
    log4j.appender.CsvsToJson.file=C:/test/logs/camel.log
    log4j.appender.CsvsToJson.MaxFileSize=50MB
    log4j.appender.CsvsToJson.MaxBackupIndex=10
    log4j.appender.CsvsToJson.append=true

    You can see the log name CsvsToJson matches the name in teh rootLogger and the actual appender. There are several options and ways to write. This is just one example.

    Running our integration..

    So when you run this, you will see in your console the following log statements to begin with.

    [INFO] Using org.apache.camel.test.blueprint.Main to initiate a CamelContext
    [INFO] Starting Camel ...
    2016-02-14 20:35:39,223 [int.Main.main()] INFO  Activator                      - Camel activator starting
    2016-02-14 20:35:39,236 [int.Main.main()] INFO  Activator                      - Camel activator started
    2016-02-14 20:35:39,367 [int Extender: 1] INFO  BlueprintContainerImpl         - Bundle csvstojson/0.0.1.SNAPSHOT is waiting for namespace handlers [http://camel.apache.org/schema/blueprint]
    2016-02-14 20:35:40,374 [int Extender: 1] INFO  BlueprintCamelContext          - Apache Camel 2.16.1 (CamelContext: CsvToJson) is starting
    2016-02-14 20:35:40,374 [int Extender: 1] INFO  BlueprintCamelContext          - MDC logging is enabled on CamelContext: CsvToJson
    2016-02-14 20:35:40,375 [int Extender: 1] INFO  ManagedManagementStrategy      - JMX is enabled
    2016-02-14 20:35:40,623 [int Extender: 1] INFO  DefaultRuntimeEndpointRegistry - Runtime endpoint registry is in extended mode gathering usage statistics of all incoming and outgoing endpoints (cache limit: 1000)
    2016-02-14 20:35:40,623 [int Extender: 1] INFO  BlueprintCamelContext          - AllowUseOriginalMessage is enabled. If access to the original message is not needed, then its recommended to turn this option off as it may improve performance.
    2016-02-14 20:35:40,623 [int Extender: 1] INFO  BlueprintCamelContext          - StreamCaching is not in use. If using streams then its recommended to enable stream caching. See more details at http://camel.apache.org/stream-caching.html
    2016-02-14 20:35:40,623 [int Extender: 1] INFO  BlueprintCamelContext          - Total 0 routes, of which 0 is started.
    2016-02-14 20:35:40,624 [int Extender: 1] INFO  BlueprintCamelContext          - Apache Camel 2.16.1 (CamelContext: CsvToJson) started in 0.250 seconds
    2016-02-14 20:35:40,798 [int Extender: 1] INFO  BlueprintCamelContext          - Route: CsvsToJson started and consuming from: Endpoint[file://C:/test/?fileName=input.csv]
    2016-02-14 20:35:55,426 [file://C:/test/] INFO  CsvToJson                      - Processing csv message ID-moeed-Dator-63752-1455478540192-0-1 with body 12345,souciance,rashti,012458478,Sweden
    12346,souciance,eqdam rashti,012458478,Sweden
    12347,souciance,eqdam,012458478,Sweden
    12348,Moeed,eqdam rashti,012458478,Sweden
    2016-02-14 20:35:55,441 [file://C:/test/] INFO  CsvToJson                      - Mapped CSV to Pojo [org.souciance.integration.csvstojson.Identity@14718c6, org.souciance.integration.csvstojson.Identity@1b2d615, org.souciance.integration.csvstojson.Identity@1857aaa, org.souciance.integration.csvstojson.Identity@1dfdcc0]
    2016-02-14 20:35:55,615 [file://C:/test/] INFO  CsvToJson                      - Mapped Pojo to json: {"ListOfRows":[{"firstname":"souciance","lastname":"rashti","phone":12458478,"country":"Sweden"},{"firstname":"souciance","lastname":"eqdam rashti","phone":12458478,"country":"Sweden"},{"firstname":"souciance","lastname":"eqdam","phone":12458478,"country":"Sweden"},{"firstname":"Moeed","lastname":"eqdam rashti","phone":12458478,"country":"Sweden"}]}
    2016-02-14 20:35:55,674 [file://C:/test/] INFO  CsvToJson                      - Json file written to output folder found in headers {breadcrumbId=ID-moeed-Dator-63752-1455478540192-0-1, CamelFileAbsolute=true, CamelFileAbsolutePath=C:\test\input.csv, CamelFileContentType=application/vnd.ms-excel, CamelFileLastModified=1454862757199, CamelFileLength=169, CamelFileName=input.csv, CamelFileNameConsumed=input.csv, CamelFileNameOnly=input.csv, CamelFileNameProduced=C:\test\output.json, CamelFileParent=C:\test, CamelFilePath=C:\test\input.csv, CamelFileRelativePath=input.csv}
    

    As you can see the logs highlight the different steps but near the end the different log statements that we are are also visible. The actual log file produced contains the same log.

    [INFO] Using org.apache.camel.test.blueprint.Main to initiate a CamelContext
    [INFO] Starting Camel ...
    2016-02-14 20:35:39,223 [int.Main.main()] INFO Activator - Camel activator starting
    2016-02-14 20:35:39,236 [int.Main.main()] INFO Activator - Camel activator started
    2016-02-14 20:35:39,367 [int Extender: 1] INFO BlueprintContainerImpl - Bundle csvstojson/0.0.1.SNAPSHOT is waiting for namespace handlers [http://camel.apache.org/schema/blueprint]
    2016-02-14 20:35:40,374 [int Extender: 1] INFO BlueprintCamelContext - Apache Camel 2.16.1 (CamelContext: CsvToJson) is starting
    2016-02-14 20:35:40,374 [int Extender: 1] INFO BlueprintCamelContext - MDC logging is enabled on CamelContext: CsvToJson
    2016-02-14 20:35:40,375 [int Extender: 1] INFO ManagedManagementStrategy - JMX is enabled
    2016-02-14 20:35:40,623 [int Extender: 1] INFO DefaultRuntimeEndpointRegistry - Runtime endpoint registry is in extended mode gathering usage statistics of all incoming and outgoing endpoints (cache limit: 1000)
    2016-02-14 20:35:40,623 [int Extender: 1] INFO BlueprintCamelContext - AllowUseOriginalMessage is enabled. If access to the original message is not needed, then its recommended to turn this option off as it may improve performance.
    2016-02-14 20:35:40,623 [int Extender: 1] INFO BlueprintCamelContext - StreamCaching is not in use. If using streams then its recommended to enable stream caching. See more details at http://camel.apache.org/stream-caching.html
    2016-02-14 20:35:40,623 [int Extender: 1] INFO BlueprintCamelContext - Total 0 routes, of which 0 is started.
    2016-02-14 20:35:40,624 [int Extender: 1] INFO BlueprintCamelContext - Apache Camel 2.16.1 (CamelContext: CsvToJson) started in 0.250 seconds
    2016-02-14 20:35:40,798 [int Extender: 1] INFO BlueprintCamelContext - Route: CsvsToJson started and consuming from: Endpoint[file://C:/test/?fileName=input.csv]
    2016-02-14 20:35:55,426 [file://C:/test/] INFO CsvToJson - Processing csv message ID-moeed-Dator-63752-1455478540192-0-1 with body 12345,souciance,rashti,012458478,Sweden
    12346,souciance,eqdam rashti,012458478,Sweden
    12347,souciance,eqdam,012458478,Sweden
    12348,Moeed,eqdam rashti,012458478,Sweden
    2016-02-14 20:35:55,441 [file://C:/test/] INFO CsvToJson - Mapped CSV to Pojo [org.souciance.integration.csvstojson.Identity@14718c6, org.souciance.integration.csvstojson.Identity@1b2d615, org.souciance.integration.csvstojson.Identity@1857aaa, org.souciance.integration.csvstojson.Identity@1dfdcc0]
    2016-02-14 20:35:55,615 [file://C:/test/] INFO CsvToJson - Mapped Pojo to json: {"ListOfRows":[{"firstname":"souciance","lastname":"rashti","phone":12458478,"country":"Sweden"},{"firstname":"souciance","lastname":"eqdam rashti","phone":12458478,"country":"Sweden"},{"firstname":"souciance","lastname":"eqdam","phone":12458478,"country":"Sweden"},{"firstname":"Moeed","lastname":"eqdam rashti","phone":12458478,"country":"Sweden"}]}
    2016-02-14 20:35:55,674 [file://C:/test/] INFO CsvToJson - Json file written to output folder found in headers {breadcrumbId=ID-moeed-Dator-63752-1455478540192-0-1, CamelFileAbsolute=true, CamelFileAbsolutePath=C:\test\input.csv, CamelFileContentType=application/vnd.ms-excel, CamelFileLastModified=1454862757199, CamelFileLength=169, CamelFileName=input.csv, CamelFileNameConsumed=input.csv, CamelFileNameOnly=input.csv, CamelFileNameProduced=C:\test\output.json, CamelFileParent=C:\test, CamelFilePath=C:\test\input.csv, CamelFileRelativePath=input.csv}
    

    If you use Eclipse and you just want test things out you can right click on your project –>run as–> run configurations –>jre and add

    -Dlog4j.configuration=file:<pathto yourlogfile> and then run with camel:run as your maven goal. This should create the log file for you.

    Summary

    So in this post we have covered how to inject properties in your java class fields which can have great value when you want to hide your endpoints and make them more dynamic to make your integrations more consistent and easier to maintain.  We have also see how log files can be created and how you can write custom log statements to log headers and the message paylaod. Thanks for tuning in.

    Camel development series – Part 3

    Hello again and welcome to the third part of the Camel development series.

    In part 2 we touched upon a very basic mapping scenario, namely mapping a single row of CSV data to a simple json structure. It touched upon some basic Camel ideas such as combing blueprint and the java dsl, injecting beans, using processors and the jackson json API.

    In this part we will create a new project and handle multiple rows of CSV data and map it to a json array structure. You can find all the source code and test data from here https://github.com/SoucianceEqdamRashti/Integration

    Input and Output

    12345,souciance,rashti,012458478,Sweden
    12346,souciance,eqdam rashti,012458478,Sweden
    12347,souciance,eqdam,012458478,Sweden
    12348,Moeed,eqdam rashti,012458478,Sweden

    This above is our input. As you can see these are simple fields and each row is delimited by a control line feed and each field is delimited by a comma. Our desired output is:

    {
    	"ListOfRows": [{
    		"firstname": "souciance",
    		"lastname": "rashti",
    		"phone": 12458478,
    		"country": "Sweden"
    	},
    	{
    		"firstname": "souciance",
    		"lastname": "eqdam rashti",
    		"phone": 12458478,
    		"country": "Sweden"
    	},
    	{
    		"firstname": "souciance",
    		"lastname": "eqdam",
    		"phone": 12458478,
    		"country": "Sweden"
    	},
    	{
    		"firstname": "Moeed",
    		"lastname": "eqdam rashti",
    		"phone": 12458478,
    		"country": "Sweden"
    	}]
    }
    

    Our output contains a JSON array called ListOfRows which contains the data from every row in the CSV file. Let’s go through the code to see how we can acheive this.

    Blueprint code

    The blueprint.xml file is a copy of the one found in part 2. It looks like this:

    <?xml version="1.0" encoding="UTF-8"?>
    <blueprint xmlns="http://www.osgi.org/xmlns/blueprint/v1.0.0"        xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"        xsi:schemaLocation="        http://www.osgi.org/xmlns/blueprint/v1.0.0 http://www.osgi.org/xmlns/blueprint/v1.0.0/blueprint.xsd        http://camel.apache.org/schema/blueprint http://camel.apache.org/schema/blueprint/camel-blueprint.xsd">
    
      <bean id="CsvToJsonRouteBuilder" class="org.souciance.integration.csvstojson.CsvToJsonRouteBuilder">
      </bean>
      <bean id="IdentityToJson" class="org.souciance.integration.csvstojson.IdentityToJson">
      </bean>
    
      <camelContext xmlns="http://camel.apache.org/schema/blueprint">
        <routeBuilder ref="CsvToJsonRouteBuilder" />    
    
      </camelContext>
    
    </blueprint>
    

    The only thing that has changed is the package names. I should point out one thing. I am probably the worst person in creating good names for classes, variables and packages and have to constantly refactor. So, don’t pay too much attention to the naming😉

    RouteBuilder

    Again the RouteBuilder class is pretty much a copy of the one in part 2. It looks like this:

    package org.souciance.integration.csvstojson;
    
    import org.apache.camel.builder.RouteBuilder;
    import org.apache.camel.dataformat.bindy.csv.BindyCsvDataFormat;
    
    /**
     * A bean which we use in the route
     */
    public class CsvToJsonRouteBuilder extends RouteBuilder {
    
    	@Override
    	public void configure() throws Exception {
    		// TODO Auto-generated method stub
    		BindyCsvDataFormat bindy = new BindyCsvDataFormat(org.souciance.integration.csvstojson.Identity.class);		
    
    		from("file:C:/test/?fileName=input.csv")
    		.unmarshal(bindy)
    		.log("${body}")
    		.to("IdentityToJson")
    		.to("file:C:/test/?fileName=output.json")
    		.log("done!")
    		.end();
    
    	}
    }
    

    What we are doing is to define our bindy dataformat as CSV. We let it know to use the Identity POJO as our model class to parse the CSV rows. You can see that we are doing the unmarshal directly after the ”from” statement. Since this file contains multiple rows the important part is to understand that the exchange now contains a list. This means that we are dealing with a list of objects rather than the objects themselves. Here we can approach it in different ways.

    1. We can use the Splitter EIP to split the file and produce a file for each row. But this is not what we want.
    2. We can split and aggregate at the end – possible but a bit too complicated for our scenario.
    3. We send the entire list to our mapping class and deal with the list there. This is our approach.

    Identity class – our model object

    Just for reference our model class for the bindy dataformat looks like this:

    package org.souciance.integration.csvstojson;
    
    import org.apache.camel.dataformat.bindy.annotation.CsvRecord;
    import org.apache.camel.dataformat.bindy.annotation.DataField;
    
    @CsvRecord(separator = ",")
    public class Identity {
    
    	@DataField(pos=1)
    	private int identity;
    	@DataField(pos=2)
    	private String firstname;
    	@DataField(pos=3)
    	private String lastname;
    	@DataField(pos=4)
    	private int phone;
    	@DataField(pos=5)
    	private String country;
    	public int getIdentity() {
    		return identity;
    	}
    	public void setIdentity(int identity) {
    		this.identity = identity;
    	}
    	public String getFirstname() {
    		return firstname;
    	}
    	public void setFirstname(String firstname) {
    		this.firstname = firstname;
    	}
    	public String getLastname() {
    		return lastname;
    	}
    	public void setLastname(String lastname) {
    		this.lastname = lastname;
    	}
    	public int getPhone() {
    		return phone;
    	}
    	public void setPhone(int phone) {
    		this.phone = phone;
    	}
    	public String getCountry() {
    		return country;
    	}
    	public void setCountry(String country) {
    		this.country = country;
    	}
    
    }
    

    As you can see we have not changed anything there.

    IdentityToJson class – Where we do all the json magic

    In this class we do all the json conversion. It looks as follows:

    package org.souciance.integration.csvstojson;
    
    import java.io.ByteArrayOutputStream;
    import java.util.List;
    
    import org.apache.camel.Exchange;
    import org.apache.camel.Processor;
    
    import com.fasterxml.jackson.core.JsonFactory;
    import com.fasterxml.jackson.core.JsonGenerator;
    import com.fasterxml.jackson.databind.ObjectMapper;
    import com.fasterxml.jackson.databind.node.ArrayNode;
    import com.fasterxml.jackson.databind.node.JsonNodeFactory;
    import com.fasterxml.jackson.databind.node.ObjectNode;
    
    public class IdentityToJson implements Processor {
    
    	@Override
    	public void process(Exchange exchange) throws Exception {
    		// TODO Auto-generated method stub
    
    		/*initialize Jackson
    		 *we need to create an outer object, an inner array and node objects for individual array elements
    		 */
    		JsonNodeFactory factory = new JsonNodeFactory(false);
    		JsonFactory jsonFactory = new JsonFactory();
    		ByteArrayOutputStream outputStream = new ByteArrayOutputStream();
    		JsonGenerator generator = jsonFactory.createGenerator(outputStream);
    		ObjectMapper mapper = new ObjectMapper();
    		ObjectNode listOfPersons = mapper.createObjectNode();
    		ArrayNode persons = factory.arrayNode();
    
    		/*get the current row and add it to the json array item by
    		*traversing the incoming List of Identity objects
    		*person is the json object containing each row
    		*persons is the array that contains multiple person items
    		*/
    		if (exchange.getIn().getBody() instanceof java.util.List) {
    			@SuppressWarnings("unchecked")
    			List<Identity> listOfIdentities = ((List<Identity>)exchange.getIn().getBody());
    			for (Identity identity : listOfIdentities ) {
    				ObjectNode person = factory.objectNode();
    				person.put("firstname", identity.getFirstname());
    				person.put("lastname", identity.getLastname());
    				person.put("phone", identity.getPhone());
    				person.put("country", identity.getCountry());
    				persons.add(person);
    			}
    		}
    		//finally we create the entire json structure by adding the array to the root object ListOfRows
    		listOfPersons.putArray("ListOfRows").addAll(persons);
    
    		//write the json string to the exchange
    		mapper.writeTree(generator,  listOfPersons);
    		String json = new String(outputStream.toString());
    		exchange.getIn().setBody(json);
    
    	}
    
    }
    

    There are several changes made here.

    1. In the json initialization block you can see that we have added not only normal json objects but also array nodes. Essentially listOfPersons contains Persons which contains mulitple persons.
    2. The crucial part is the second part where we check if the object in the exchange is an instaneof List – that is if it contains a list. If it does, we iterate and extract the values and add it the person node. At the end of the iteration the person node is added to persons array.
    3. Once the iteration is done, we add the array to the listOfPersons object using the putArray and addAll jackson methods.

    Summary

    In essence this is one approach to map Csv to JSON format and customising your json structure. There are I am sure better ways or other approaches. If you know of others please do post them😉 Stay tuned for the next series.