Camel development series part 5

Hello, and welcome back to the Camel development series and to part 5. Its been a couple of weeks since I wrote and this has been mainly due to lack of time but hopefully I can get back to track.

In this series we are going to look at a classic case of content based routing. This means, based on some criteria the input data should be transformed/routed in one way and on another criteria transformed/routed in another way.

We will build up on our previous work but this will be in a new project. You can find all the source code here https://github.com/SoucianceEqdamRashti/Integration/tree/master/CamelContentbasedRouter .

Problem

Let’s say we have a file placed in some folder. Based on the file name we want to either map it to XML or to JSON. The input format is basic CSV.

  1. If the file name contains the string ”xml” then the data should be mapped to XML.
  2. If the file name contains the string ”json” then the data should be mapped to JSON.

Our end result should produce an XML or a JSON based on the file name.

Solution

We will introduce a few new concepts there. One of the most powerful ideas in the Camel world is the use of Predicates. You can read more about it here http://camel.apache.org/predicate.html . Predicates allow you to define complicated conditions upon which to based your logic and choices. We will use them here to make choices. We will also look at how to send and use bean transformation to work together with JAXB. Finally we will also look at how to read the file name and how to actually map to XML from CSV.

Main RouteBuilder class

Let us first look at the main RouteBuilder class to get an idea how the main flow looks like.

package org.souciance.integration.contentbasedrouter;

import org.apache.camel.LoggingLevel;
import org.apache.camel.Predicate;
import org.apache.camel.PropertyInject;
import org.apache.camel.builder.RouteBuilder;
import org.apache.camel.converter.jaxb.JaxbDataFormat;
import org.apache.camel.dataformat.bindy.csv.BindyCsvDataFormat;
import java.lang.String;

public class MainRouteBuilder extends RouteBuilder {
@PropertyInject("{{input.folder}}")
private String inputFolder;
@PropertyInject("{{output.folder}}")
private String outputFolder;
@PropertyInject("{{input.filename}}")
private String inputFileName;
@PropertyInject("{{output.filename}}")
private String outputFileName;

@Override
public void configure() throws Exception {

Predicate toXML = header("CamelFileName").contains(String.valueOf("xml"));
Predicate toJSON = header("CamelFileName").contains(String.valueOf("json"));

BindyCsvDataFormat bindy = new BindyCsvDataFormat(Person.class);
JaxbDataFormat jaxbDataFormat = new JaxbDataFormat(Persons.class.getPackage().getName());
jaxbDataFormat.setPrettyPrint(true);
from("file://" + inputFolder+"?include=.*.csv").routeId("ContentbasedRouter")
.log(LoggingLevel.INFO, "body","body: ${body}")
.unmarshal(bindy)
.choice()
.when(toXML)
.log(LoggingLevel.INFO, "Data will be mapped to xml structure")
.bean(Transform.class)
.marshal(jaxbDataFormat)
.log(LoggingLevel.INFO, "Data will be mapped to xml structure: ${body}")
.to("file://"+outputFolder+"/?fileName=output.xml")
.when(toJSON)
.log(LoggingLevel.INFO, "Data will be mapped to json structure.")
.unmarshal(bindy)
.to("PersonMapperToJson")
.log(LoggingLevel.INFO, "PersonMapperToJson", "Mapped Pojo to json: ${body}")
.to("file://"+outputFolder+"/?fileName="+"output.json")
.endChoice()
.end();

}
}

The main idea should be readable even if you don’t know all the details. We are polling a directory. If a file with extension .csv is placed in the directory we look at the file name. Here is where predicates come in handy. Let’s look at it in more detail:

Predicate toXML = header("CamelFileName").contains(String.valueOf("xml"));
Predicate toJSON = header("CamelFileName").contains(String.valueOf("json"));

We have defined two predicates, and each one of them looks at the header named CamelFileName. This is a standard Camel header where the name of the file is placed. In the rule we check if the value xml or json is found. These are examples of simple predicates, you can build much more complicated rules and also combine them with property files to make dynamic rules.

Next let’s look at the new idea of DataFormat and especially the jaxb data format.

JaxbDataFormat jaxbDataFormat = new JaxbDataFormat(Persons.class.getPackage().getName());
jaxbDataFormat.setPrettyPrint(true);

DataFormats are what allows Camel to recognize formats and transform or otherwise known as marshal/unmarshal data. We need two types of data formats in our case. One is the one we have introduced earlier which is the Bindy format to parse CSV into POJO. The second seen above is the JAXB dataformat to transform the list of POJOs to XML. For more on data formats read here https://camel.apache.org/data-format.html .

The second new part and where all the interesting logic happens is here:

.choice()
.when(toXML)
.log(LoggingLevel.INFO, "Data will be mapped to xml structure")
.bean(Transform.class)
.marshal(jaxbDataFormat)
.log(LoggingLevel.INFO, "Data will be mapped to xml structure: ${body}")
.to("file://"+outputFolder+"/?fileName=output.xml")

You can see that we use the .choice() tag to start our content based routing. We use the .when(toXML) to use the predicate defined earlier. Essentially this means in plain English, when the predicate toXML holds true enter this block. After the log statement you notice the .bean() tag and this is an interesting tag. It allows us to send the exchange to the bean and do whatever we want with it and then continue. In our cause we are going to do some JAXB logic in the transform bean.
Then finally we have the marshal statement which refers to the jaxb data format. The last two are simple log and writing to file statements.

Let’s look at our Transform class

Transform Bean

Our transform bean looks as follows:

package org.souciance.integration.contentbasedrouter;

import java.util.List;
import org.apache.camel.Exchange;
import org.apache.camel.Message;

public class Transform {

public void process(Exchange exchange) throws Exception {
Message msg = exchange.getIn();

@SuppressWarnings("unchecked")
List personList = (List) msg.getBody();

ObjectFactory objectFactory = new ObjectFactory();

Persons persons = objectFactory.createPersons();

for (Person aPerson : personList) {
persons.setPerson(aPerson);

exchange.getIn().setBody(persons);
}

}

}

The code is essentially receiving a list of Person POJOs which were created when marshalling to the bindy format. Then an instance of ObjectFactory is created and we are adding each POJO from the list to the ArrayList in Persons class. Finally we set the exchange to contain the Persons list.

The remaining parts are the annotated classes and the ObjectFactory class. So let’s look at them.

ObjectFactory

The object factory class and the annotated classes were inspired by the post found here http://www.developpez.net/forums/d1216465/java/serveurs-conteneurs-java-ee/autres/camel-convertir-csv-xml-bindy-jaxb/ . Although it is in French, the gist of the idea should be comprehensible.

Here is our class:

//
// This file was generated by the JavaTM Architecture for XML Binding(JAXB) Reference Implementation, v2.2.8-b130911.1802
// See <a href="http://java.sun.com/xml/jaxb">http://java.sun.com/xml/jaxb</a>
// Any modifications to this file will be lost upon recompilation of the source schema.
// Generated on: 2016.03.05 at 06:27:39 PM CET
//

package org.souciance.integration.contentbasedrouter;

import javax.xml.bind.annotation.XmlRegistry;

/**
* This object contains factory methods for each
* Java content interface and Java element interface
* generated in the org.souciance.integration.contentbasedrouter.jaxb package.
*

An ObjectFactory allows you to programatically
* construct new instances of the Java representation
* for XML content. The Java representation of XML
* content can consist of schema derived interfaces
* and classes representing the binding of schema
* type definitions, element declarations and model
* groups. Factory methods for each of these are
* provided in this class.
*
*/
@XmlRegistry
public class ObjectFactory {

/**
* Create a new ObjectFactory that can be used to create new instances of schema derived classes for package: org.souciance.integration.contentbasedrouter.jaxb
*
*/
public ObjectFactory() {
}

/**
* Create an instance of {@link Persons }
*
*/
public Persons createPersons() {
return new Persons();
}

/**
* Create an instance of {@link Persons.Person }
*
*/
public Person createPerson() {
return new Person();
}

}

Annotated classes

We need annotated classes for bindy and jaxb processing. For completeness sake here they are.

Persons class

//
// This file was generated by the JavaTM Architecture for XML Binding(JAXB) Reference Implementation, v2.2.8-b130911.1802
// See <a href="http://java.sun.com/xml/jaxb">http://java.sun.com/xml/jaxb</a>
// Any modifications to this file will be lost upon recompilation of the source schema.
// Generated on: 2016.03.05 at 06:27:39 PM CET
//

package org.souciance.integration.contentbasedrouter;

import java.util.ArrayList;
import java.util.List;
import javax.xml.bind.annotation.XmlAccessType;
import javax.xml.bind.annotation.XmlAccessorType;
import javax.xml.bind.annotation.XmlElement;
import javax.xml.bind.annotation.XmlRootElement;
import javax.xml.bind.annotation.XmlType;

/**
*

Java class for anonymous complex type.
*
*

The following schema fragment specifies the expected content contained within this class.
*
*
<pre> * &lt;complexType&gt;
 *   &lt;complexContent&gt;
 *     &lt;restriction base="{http://www.w3.org/2001/XMLSchema}anyType"&gt;
 *       &lt;sequence&gt;
 *         &lt;element name="person" maxOccurs="unbounded" minOccurs="0"&gt;
 *           &lt;complexType&gt;
 *             &lt;complexContent&gt;
 *               &lt;restriction base="{http://www.w3.org/2001/XMLSchema}anyType"&gt;
 *                 &lt;sequence&gt;
 *                   &lt;element name="firstname" type="{http://www.w3.org/2001/XMLSchema}string"/&gt;
 *                   &lt;element name="lastname" type="{http://www.w3.org/2001/XMLSchema}string"/&gt;
 *                   &lt;element name="identity" type="{http://www.w3.org/2001/XMLSchema}string"/&gt;
 *                   &lt;element name="country" type="{http://www.w3.org/2001/XMLSchema}string"/&gt;
 *                   &lt;element name="phone" type="{http://www.w3.org/2001/XMLSchema}string"/&gt;
 *                 &lt;/sequence&gt;
 *               &lt;/restriction&gt;
 *             &lt;/complexContent&gt;
 *           &lt;/complexType&gt;
 *         &lt;/element&gt;
 *       &lt;/sequence&gt;
 *     &lt;/restriction&gt;
 *   &lt;/complexContent&gt;
 * &lt;/complexType&gt;
 *</pre>
*
*
*/
@XmlAccessorType(XmlAccessType.FIELD)
@XmlType(name = "", propOrder = {
"person"
})
@XmlRootElement(name = "persons")
public class Persons {
@XmlElement(required = false, name = "person")
protected List person;

public List getPerson() {
if (person == null) {
person = new ArrayList();
}
return this.person;
}
public void setPerson(Person person){
this.getPerson().add(person);
}

}

Person class

package org.souciance.integration.contentbasedrouter;

import javax.xml.bind.annotation.XmlAccessorType;
import javax.xml.bind.annotation.XmlElement;
import javax.xml.bind.annotation.XmlType;
import javax.xml.bind.annotation.XmlAccessType;
import org.apache.camel.dataformat.bindy.annotation.CsvRecord;
import org.apache.camel.dataformat.bindy.annotation.DataField;

@CsvRecord(separator = ",")

@XmlAccessorType(XmlAccessType.FIELD)
@XmlType(name="person")
public class Person {

@XmlElement(name = "identity")
@DataField(pos=1)
private int identity;

@XmlElement(name = "firstname")
@DataField(pos=2)
private String firstname;

@XmlElement(name = "lastname")
@DataField(pos=3)
private String lastname;

@XmlElement(name = "phonenumber")
@DataField(pos=4)
private int phone;

@XmlElement(name = "country")
@DataField(pos=5)
private String country;

public int getIdentity() {
return identity;
}
public void setIdentity(int identity) {
this.identity = identity;
}
public String getFirstname() {
return firstname;
}
public void setFirstname(String firstname) {
this.firstname = firstname;
}
public String getLastname() {
return lastname;
}
public void setLastname(String lastname) {
this.lastname = lastname;
}
public int getPhone() {
return phone;
}
public void setPhone(int phone) {
this.phone = phone;
}
public String getCountry() {
return country;
}
public void setCountry(String country) {
this.country = country;
}

}

Summary

In this project we have looked at a common problem, how to transform and route data in different ways based on some condition. We defined predicates to define our condition rules, we used the choice and when tags to check our predicates, we use bean transformation to connect our jaxb processing and then we marshalled  the list of objects to XML using jaxb marshalling.

The input data looked like this:
12345,souciance,rashti,012458478,Sweden
12345,souciance,rashti,012458478,Sweden

The filename was csvtoxml.csv

The final output is:

<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<persons>
    <person>
        <identity>12345</identity>
        <firstname>souciance</firstname>
        <lastname>rashti</lastname>
        <phonenumber>12458478</phonenumber>
        <country>Sweden</country>
    </person>
    <person>
        <identity>12345</identity>
        <firstname>souciance</firstname>
        <lastname>rashti</lastname>
        <phonenumber>12458478</phonenumber>
        <country>Sweden</country>
    </person>
</persons>

It works similarly for the second when block for the JSON which is exactly as the previous series.

Annonser

Camel development series – part 4

Welcome again to this fourth installed of the Camel development series. It is almost 1am but am determined to write one post a week as to not fall behind my original goal. This post will not cover a lot of functionality but will touch upon two parts which I think are essentially to developing production ready code. These are:

  1. ensuring single copy of deployment. What I mean here is that we don’t want to develop one copy for our test environment, one copy for our QA and one for production environment. We want a single copy which can be deployed to all of our environments, thereby ensuring consistency and knowing that what is in production is the same as what is in test in QA. It has therefore been tested and verified. To achieve this, we need a way to hide away endpoint parameters and hence deal with property files.
  2. producing log files. To track our integrations we need to know what they have done and off course if something has gone wrong, the simplest way is to produce log files and write logs during important steps. We will cover this step. In a later post I will go through MDC logging when you deploy to a Karaf environment so that you can produce log files based on a route or context id which gives you better granularity.

Ok let’s start with the actually code and return to our standard project. You will find the source here https://github.com/SoucianceEqdamRashti/Integration/tree/master/MapCsvsToJson

Blueprint code

 

Here is our blueprint.xml code:

<?xml version="1.0" encoding="UTF-8"?>
<blueprint xmlns="http://www.osgi.org/xmlns/blueprint/v1.0.0"        xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"        xsi:schemaLocation="        http://www.osgi.org/xmlns/blueprint/v1.0.0 http://www.osgi.org/xmlns/blueprint/v1.0.0/blueprint.xsd        http://camel.apache.org/schema/blueprint http://camel.apache.org/schema/blueprint/camel-blueprint.xsd">

  <bean id="CsvToJsonRouteBuilder" class="org.souciance.integration.csvstojson.CsvsToJsonRouteBuilder">
  </bean>
  <bean id="IdentityToJson" class="org.souciance.integration.csvstojson.IdentityToJson">
  </bean>
  <bean id="properties"	class="org.apache.camel.component.properties.PropertiesComponent">
		<property name="location" value="classpath:MapCsvsToJson.properties" />
</bean>

  <camelContext xmlns="http://camel.apache.org/schema/blueprint" useMDCLogging="true" id="CsvToJson">
    <routeBuilder ref="CsvToJsonRouteBuilder" />    

  </camelContext>

</blueprint>

There are two additions here. One is that we have added useMDCLogging=”true” attribute on the camelContext tag. We will use this later in another post. But more importantly we have added the bean for reading properties. You can see that there is bean declared with id=”properties” and the name of the properties field is MapCsvsToJson.properties and it should be present somewhere on the classpath. Off course in real life you follow some naming convention but this is just tho show how it works. The beauty of this approach is that it enables us to inject any kind of property values onto java class member fields. This is in essence how we enable our goal of single copy.

MapCsvsToJson.properties

We have also created a properties file called MapCsvsToJson.properties and it is in our src/resources folder. The properties defined are:
input.folder =C:/test
output.folder =C:/test
input.filename =input.csv
output.filename =output.json

As you can see we have written down filepath and filenames for our input and output. You can easily see that this can be extended to any endpoint be it HTTP, FTP, WMQ, Rabbit, TCP etc. This means that the actual integration code remains the same, you just change the endpoint when you move your package from test environment to qa and add the qa endpoints and same when you go to production. You can add other system parameters here which may need to change from one environment to another or to make your code more flexible.

MapCsvsToJsonRouteBuilder.java

In our route builder class we have made some additions. Here is how it looks now:

package org.souciance.integration.csvstojson;

import org.apache.camel.LoggingLevel;
import org.apache.camel.PropertyInject;
import org.apache.camel.builder.RouteBuilder;
import org.apache.camel.dataformat.bindy.csv.BindyCsvDataFormat;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;

/**
 * A bean which we use in the route
 */
public class CsvsToJsonRouteBuilder extends RouteBuilder {
	@PropertyInject("{{input.folder}}")
	private String inputFolder;
	@PropertyInject("{{output.folder}}")
	private String outputFolder;
	@PropertyInject("{{input.filename}}")
	private String inputFileName;
	@PropertyInject("{{output.filename}}")
	private String outputFileName;

	@Override
	public void configure() throws Exception {
		// TODO Auto-generated method stub
		BindyCsvDataFormat bindy = new BindyCsvDataFormat(org.souciance.integration.csvstojson.Identity.class);		

		from("file:" + inputFolder+"/?fileName="+inputFileName).routeId("CsvsToJson")
		.log(LoggingLevel.INFO, "CsvToJson", "Processing csv message ${id} with body ${body}")
		.unmarshal(bindy)
		.log(LoggingLevel.INFO, "CsvToJson", "Mapped CSV to Pojo ${body}")
		.to("IdentityToJson")
		.log(LoggingLevel.INFO, "CsvToJson", "Mapped Pojo to json: ${body}")
		.to("file:"+outputFolder+"/?fileName="+outputFileName)
		.log(LoggingLevel.INFO, "CsvToJson", "Json file written to output folder found in headers ${in.headers}")
		.end();

	}
}

As you can see we have now written the following when declaring our class member fields:

@PropertyInject("{{input.folder}}")
	private String inputFolder;

What this does is to inject the value of property input.folder which is C:/test to the field inputFolder. The same goes for the other variables.

Further down you can see this:

from("file:" + inputFolder+"/?fileName="+inputFileName).routeId("CsvsToJson")

This is now different from our previous code. We have hidden the input folder and the input file name. In some sense we are creating a dynamic endpoint because the actual endpoint URI are retrieved from our property file. So again, imagine having a properties file for test, qa and prod and simply deploying a single copy with the right property file rather than multiple copies of your integration for each environment. The same applies to the ”to” endpoint at the bottom. This demonstrates the main functionality of property injection, you can off course inject to normal POJO classes or even to input fields of a method. There is a rich complexity here not found in the commercial tools.

Ok let us also mention the code we added for the logging. You have noticed several statements of the form:

.log(LoggingLevel.INFO, "CsvToJson", "Processing csv message ${id} with body ${body}")

Here we are using the Log dsl to do a few things. First, we set the logging level to INFO as we are just logging steps. We are giving the logger a name corresponding to the appender name in the log4j properties file. Finally we have some log text. Here we are logging the camel id of the message and the body. The language is called simple language and is a kind of camel query language where you can access the body, headers and camel properties. To do that you write ${variable}. As you see we are logging when we receive the file, after we mapped to a POJO, after we mapped to Json and after we have written the output file.

The log4j properties looks like this:

log4j.rootLogger=INFO, out, CsvsToJson
#log4j.logger.org.apache.camel=DEBUG, file

# CONSOLE appender
log4j.appender.out=org.apache.log4j.ConsoleAppender
log4j.appender.out.layout=org.apache.log4j.PatternLayout
log4j.appender.out.layout.ConversionPattern=[%30.30t] %-30.30c{1} %-5p %m%n
log4j.appender.out.layout.ConversionPattern=%d [%-15.15t] %-5p %-30.30c{1} – %m%n

log4j.appender.CsvsToJson=org.apache.log4j.RollingFileAppender
log4j.appender.CsvsToJson.layout=org.apache.log4j.PatternLayout
log4j.appender.CsvsToJson.layout.ConversionPattern=%d{ISO8601} | %-5.5p | %-16.16t | %-32.32c{1} | %X{camel.routeId} | %m%n
log4j.appender.CsvsToJson.file=C:/test/logs/camel.log
log4j.appender.CsvsToJson.MaxFileSize=50MB
log4j.appender.CsvsToJson.MaxBackupIndex=10
log4j.appender.CsvsToJson.append=true

You can see the log name CsvsToJson matches the name in teh rootLogger and the actual appender. There are several options and ways to write. This is just one example.

Running our integration..

So when you run this, you will see in your console the following log statements to begin with.

[INFO] Using org.apache.camel.test.blueprint.Main to initiate a CamelContext
[INFO] Starting Camel ...
2016-02-14 20:35:39,223 [int.Main.main()] INFO  Activator                      - Camel activator starting
2016-02-14 20:35:39,236 [int.Main.main()] INFO  Activator                      - Camel activator started
2016-02-14 20:35:39,367 [int Extender: 1] INFO  BlueprintContainerImpl         - Bundle csvstojson/0.0.1.SNAPSHOT is waiting for namespace handlers [http://camel.apache.org/schema/blueprint]
2016-02-14 20:35:40,374 [int Extender: 1] INFO  BlueprintCamelContext          - Apache Camel 2.16.1 (CamelContext: CsvToJson) is starting
2016-02-14 20:35:40,374 [int Extender: 1] INFO  BlueprintCamelContext          - MDC logging is enabled on CamelContext: CsvToJson
2016-02-14 20:35:40,375 [int Extender: 1] INFO  ManagedManagementStrategy      - JMX is enabled
2016-02-14 20:35:40,623 [int Extender: 1] INFO  DefaultRuntimeEndpointRegistry - Runtime endpoint registry is in extended mode gathering usage statistics of all incoming and outgoing endpoints (cache limit: 1000)
2016-02-14 20:35:40,623 [int Extender: 1] INFO  BlueprintCamelContext          - AllowUseOriginalMessage is enabled. If access to the original message is not needed, then its recommended to turn this option off as it may improve performance.
2016-02-14 20:35:40,623 [int Extender: 1] INFO  BlueprintCamelContext          - StreamCaching is not in use. If using streams then its recommended to enable stream caching. See more details at http://camel.apache.org/stream-caching.html
2016-02-14 20:35:40,623 [int Extender: 1] INFO  BlueprintCamelContext          - Total 0 routes, of which 0 is started.
2016-02-14 20:35:40,624 [int Extender: 1] INFO  BlueprintCamelContext          - Apache Camel 2.16.1 (CamelContext: CsvToJson) started in 0.250 seconds
2016-02-14 20:35:40,798 [int Extender: 1] INFO  BlueprintCamelContext          - Route: CsvsToJson started and consuming from: Endpoint[file://C:/test/?fileName=input.csv]
2016-02-14 20:35:55,426 [file://C:/test/] INFO  CsvToJson                      - Processing csv message ID-moeed-Dator-63752-1455478540192-0-1 with body 12345,souciance,rashti,012458478,Sweden
12346,souciance,eqdam rashti,012458478,Sweden
12347,souciance,eqdam,012458478,Sweden
12348,Moeed,eqdam rashti,012458478,Sweden
2016-02-14 20:35:55,441 [file://C:/test/] INFO  CsvToJson                      - Mapped CSV to Pojo [org.souciance.integration.csvstojson.Identity@14718c6, org.souciance.integration.csvstojson.Identity@1b2d615, org.souciance.integration.csvstojson.Identity@1857aaa, org.souciance.integration.csvstojson.Identity@1dfdcc0]
2016-02-14 20:35:55,615 [file://C:/test/] INFO  CsvToJson                      - Mapped Pojo to json: {"ListOfRows":[{"firstname":"souciance","lastname":"rashti","phone":12458478,"country":"Sweden"},{"firstname":"souciance","lastname":"eqdam rashti","phone":12458478,"country":"Sweden"},{"firstname":"souciance","lastname":"eqdam","phone":12458478,"country":"Sweden"},{"firstname":"Moeed","lastname":"eqdam rashti","phone":12458478,"country":"Sweden"}]}
2016-02-14 20:35:55,674 [file://C:/test/] INFO  CsvToJson                      - Json file written to output folder found in headers {breadcrumbId=ID-moeed-Dator-63752-1455478540192-0-1, CamelFileAbsolute=true, CamelFileAbsolutePath=C:\test\input.csv, CamelFileContentType=application/vnd.ms-excel, CamelFileLastModified=1454862757199, CamelFileLength=169, CamelFileName=input.csv, CamelFileNameConsumed=input.csv, CamelFileNameOnly=input.csv, CamelFileNameProduced=C:\test\output.json, CamelFileParent=C:\test, CamelFilePath=C:\test\input.csv, CamelFileRelativePath=input.csv}

As you can see the logs highlight the different steps but near the end the different log statements that we are are also visible. The actual log file produced contains the same log.

[INFO] Using org.apache.camel.test.blueprint.Main to initiate a CamelContext
[INFO] Starting Camel ...
2016-02-14 20:35:39,223 [int.Main.main()] INFO Activator - Camel activator starting
2016-02-14 20:35:39,236 [int.Main.main()] INFO Activator - Camel activator started
2016-02-14 20:35:39,367 [int Extender: 1] INFO BlueprintContainerImpl - Bundle csvstojson/0.0.1.SNAPSHOT is waiting for namespace handlers [http://camel.apache.org/schema/blueprint]
2016-02-14 20:35:40,374 [int Extender: 1] INFO BlueprintCamelContext - Apache Camel 2.16.1 (CamelContext: CsvToJson) is starting
2016-02-14 20:35:40,374 [int Extender: 1] INFO BlueprintCamelContext - MDC logging is enabled on CamelContext: CsvToJson
2016-02-14 20:35:40,375 [int Extender: 1] INFO ManagedManagementStrategy - JMX is enabled
2016-02-14 20:35:40,623 [int Extender: 1] INFO DefaultRuntimeEndpointRegistry - Runtime endpoint registry is in extended mode gathering usage statistics of all incoming and outgoing endpoints (cache limit: 1000)
2016-02-14 20:35:40,623 [int Extender: 1] INFO BlueprintCamelContext - AllowUseOriginalMessage is enabled. If access to the original message is not needed, then its recommended to turn this option off as it may improve performance.
2016-02-14 20:35:40,623 [int Extender: 1] INFO BlueprintCamelContext - StreamCaching is not in use. If using streams then its recommended to enable stream caching. See more details at http://camel.apache.org/stream-caching.html
2016-02-14 20:35:40,623 [int Extender: 1] INFO BlueprintCamelContext - Total 0 routes, of which 0 is started.
2016-02-14 20:35:40,624 [int Extender: 1] INFO BlueprintCamelContext - Apache Camel 2.16.1 (CamelContext: CsvToJson) started in 0.250 seconds
2016-02-14 20:35:40,798 [int Extender: 1] INFO BlueprintCamelContext - Route: CsvsToJson started and consuming from: Endpoint[file://C:/test/?fileName=input.csv]
2016-02-14 20:35:55,426 [file://C:/test/] INFO CsvToJson - Processing csv message ID-moeed-Dator-63752-1455478540192-0-1 with body 12345,souciance,rashti,012458478,Sweden
12346,souciance,eqdam rashti,012458478,Sweden
12347,souciance,eqdam,012458478,Sweden
12348,Moeed,eqdam rashti,012458478,Sweden
2016-02-14 20:35:55,441 [file://C:/test/] INFO CsvToJson - Mapped CSV to Pojo [org.souciance.integration.csvstojson.Identity@14718c6, org.souciance.integration.csvstojson.Identity@1b2d615, org.souciance.integration.csvstojson.Identity@1857aaa, org.souciance.integration.csvstojson.Identity@1dfdcc0]
2016-02-14 20:35:55,615 [file://C:/test/] INFO CsvToJson - Mapped Pojo to json: {"ListOfRows":[{"firstname":"souciance","lastname":"rashti","phone":12458478,"country":"Sweden"},{"firstname":"souciance","lastname":"eqdam rashti","phone":12458478,"country":"Sweden"},{"firstname":"souciance","lastname":"eqdam","phone":12458478,"country":"Sweden"},{"firstname":"Moeed","lastname":"eqdam rashti","phone":12458478,"country":"Sweden"}]}
2016-02-14 20:35:55,674 [file://C:/test/] INFO CsvToJson - Json file written to output folder found in headers {breadcrumbId=ID-moeed-Dator-63752-1455478540192-0-1, CamelFileAbsolute=true, CamelFileAbsolutePath=C:\test\input.csv, CamelFileContentType=application/vnd.ms-excel, CamelFileLastModified=1454862757199, CamelFileLength=169, CamelFileName=input.csv, CamelFileNameConsumed=input.csv, CamelFileNameOnly=input.csv, CamelFileNameProduced=C:\test\output.json, CamelFileParent=C:\test, CamelFilePath=C:\test\input.csv, CamelFileRelativePath=input.csv}

If you use Eclipse and you just want test things out you can right click on your project –>run as–> run configurations –>jre and add

-Dlog4j.configuration=file:<pathto yourlogfile> and then run with camel:run as your maven goal. This should create the log file for you.

Summary

So in this post we have covered how to inject properties in your java class fields which can have great value when you want to hide your endpoints and make them more dynamic to make your integrations more consistent and easier to maintain.  We have also see how log files can be created and how you can write custom log statements to log headers and the message paylaod. Thanks for tuning in.

Camel development series – Part 3

Hello again and welcome to the third part of the Camel development series.

In part 2 we touched upon a very basic mapping scenario, namely mapping a single row of CSV data to a simple json structure. It touched upon some basic Camel ideas such as combing blueprint and the java dsl, injecting beans, using processors and the jackson json API.

In this part we will create a new project and handle multiple rows of CSV data and map it to a json array structure. You can find all the source code and test data from here https://github.com/SoucianceEqdamRashti/Integration

Input and Output

12345,souciance,rashti,012458478,Sweden
12346,souciance,eqdam rashti,012458478,Sweden
12347,souciance,eqdam,012458478,Sweden
12348,Moeed,eqdam rashti,012458478,Sweden

This above is our input. As you can see these are simple fields and each row is delimited by a control line feed and each field is delimited by a comma. Our desired output is:

{
	"ListOfRows": [{
		"firstname": "souciance",
		"lastname": "rashti",
		"phone": 12458478,
		"country": "Sweden"
	},
	{
		"firstname": "souciance",
		"lastname": "eqdam rashti",
		"phone": 12458478,
		"country": "Sweden"
	},
	{
		"firstname": "souciance",
		"lastname": "eqdam",
		"phone": 12458478,
		"country": "Sweden"
	},
	{
		"firstname": "Moeed",
		"lastname": "eqdam rashti",
		"phone": 12458478,
		"country": "Sweden"
	}]
}

Our output contains a JSON array called ListOfRows which contains the data from every row in the CSV file. Let’s go through the code to see how we can acheive this.

Blueprint code

The blueprint.xml file is a copy of the one found in part 2. It looks like this:

<?xml version="1.0" encoding="UTF-8"?>
<blueprint xmlns="http://www.osgi.org/xmlns/blueprint/v1.0.0"        xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"        xsi:schemaLocation="        http://www.osgi.org/xmlns/blueprint/v1.0.0 http://www.osgi.org/xmlns/blueprint/v1.0.0/blueprint.xsd        http://camel.apache.org/schema/blueprint http://camel.apache.org/schema/blueprint/camel-blueprint.xsd">

  <bean id="CsvToJsonRouteBuilder" class="org.souciance.integration.csvstojson.CsvToJsonRouteBuilder">
  </bean>
  <bean id="IdentityToJson" class="org.souciance.integration.csvstojson.IdentityToJson">
  </bean>

  <camelContext xmlns="http://camel.apache.org/schema/blueprint">
    <routeBuilder ref="CsvToJsonRouteBuilder" />    

  </camelContext>

</blueprint>

The only thing that has changed is the package names. I should point out one thing. I am probably the worst person in creating good names for classes, variables and packages and have to constantly refactor. So, don’t pay too much attention to the naming 😉

RouteBuilder

Again the RouteBuilder class is pretty much a copy of the one in part 2. It looks like this:

package org.souciance.integration.csvstojson;

import org.apache.camel.builder.RouteBuilder;
import org.apache.camel.dataformat.bindy.csv.BindyCsvDataFormat;

/**
 * A bean which we use in the route
 */
public class CsvToJsonRouteBuilder extends RouteBuilder {

	@Override
	public void configure() throws Exception {
		// TODO Auto-generated method stub
		BindyCsvDataFormat bindy = new BindyCsvDataFormat(org.souciance.integration.csvstojson.Identity.class);		

		from("file:C:/test/?fileName=input.csv")
		.unmarshal(bindy)
		.log("${body}")
		.to("IdentityToJson")
		.to("file:C:/test/?fileName=output.json")
		.log("done!")
		.end();

	}
}

What we are doing is to define our bindy dataformat as CSV. We let it know to use the Identity POJO as our model class to parse the CSV rows. You can see that we are doing the unmarshal directly after the ”from” statement. Since this file contains multiple rows the important part is to understand that the exchange now contains a list. This means that we are dealing with a list of objects rather than the objects themselves. Here we can approach it in different ways.

  1. We can use the Splitter EIP to split the file and produce a file for each row. But this is not what we want.
  2. We can split and aggregate at the end – possible but a bit too complicated for our scenario.
  3. We send the entire list to our mapping class and deal with the list there. This is our approach.

Identity class – our model object

Just for reference our model class for the bindy dataformat looks like this:

package org.souciance.integration.csvstojson;

import org.apache.camel.dataformat.bindy.annotation.CsvRecord;
import org.apache.camel.dataformat.bindy.annotation.DataField;

@CsvRecord(separator = ",")
public class Identity {

	@DataField(pos=1)
	private int identity;
	@DataField(pos=2)
	private String firstname;
	@DataField(pos=3)
	private String lastname;
	@DataField(pos=4)
	private int phone;
	@DataField(pos=5)
	private String country;
	public int getIdentity() {
		return identity;
	}
	public void setIdentity(int identity) {
		this.identity = identity;
	}
	public String getFirstname() {
		return firstname;
	}
	public void setFirstname(String firstname) {
		this.firstname = firstname;
	}
	public String getLastname() {
		return lastname;
	}
	public void setLastname(String lastname) {
		this.lastname = lastname;
	}
	public int getPhone() {
		return phone;
	}
	public void setPhone(int phone) {
		this.phone = phone;
	}
	public String getCountry() {
		return country;
	}
	public void setCountry(String country) {
		this.country = country;
	}

}

As you can see we have not changed anything there.

IdentityToJson class – Where we do all the json magic

In this class we do all the json conversion. It looks as follows:

package org.souciance.integration.csvstojson;

import java.io.ByteArrayOutputStream;
import java.util.List;

import org.apache.camel.Exchange;
import org.apache.camel.Processor;

import com.fasterxml.jackson.core.JsonFactory;
import com.fasterxml.jackson.core.JsonGenerator;
import com.fasterxml.jackson.databind.ObjectMapper;
import com.fasterxml.jackson.databind.node.ArrayNode;
import com.fasterxml.jackson.databind.node.JsonNodeFactory;
import com.fasterxml.jackson.databind.node.ObjectNode;

public class IdentityToJson implements Processor {

	@Override
	public void process(Exchange exchange) throws Exception {
		// TODO Auto-generated method stub

		/*initialize Jackson
		 *we need to create an outer object, an inner array and node objects for individual array elements
		 */
		JsonNodeFactory factory = new JsonNodeFactory(false);
		JsonFactory jsonFactory = new JsonFactory();
		ByteArrayOutputStream outputStream = new ByteArrayOutputStream();
		JsonGenerator generator = jsonFactory.createGenerator(outputStream);
		ObjectMapper mapper = new ObjectMapper();
		ObjectNode listOfPersons = mapper.createObjectNode();
		ArrayNode persons = factory.arrayNode();

		/*get the current row and add it to the json array item by
		*traversing the incoming List of Identity objects
		*person is the json object containing each row
		*persons is the array that contains multiple person items
		*/
		if (exchange.getIn().getBody() instanceof java.util.List) {
			@SuppressWarnings("unchecked")
			List<Identity> listOfIdentities = ((List<Identity>)exchange.getIn().getBody());
			for (Identity identity : listOfIdentities ) {
				ObjectNode person = factory.objectNode();
				person.put("firstname", identity.getFirstname());
				person.put("lastname", identity.getLastname());
				person.put("phone", identity.getPhone());
				person.put("country", identity.getCountry());
				persons.add(person);
			}
		}
		//finally we create the entire json structure by adding the array to the root object ListOfRows
		listOfPersons.putArray("ListOfRows").addAll(persons);

		//write the json string to the exchange
		mapper.writeTree(generator,  listOfPersons);
		String json = new String(outputStream.toString());
		exchange.getIn().setBody(json);

	}

}

There are several changes made here.

  1. In the json initialization block you can see that we have added not only normal json objects but also array nodes. Essentially listOfPersons contains Persons which contains mulitple persons.
  2. The crucial part is the second part where we check if the object in the exchange is an instaneof List – that is if it contains a list. If it does, we iterate and extract the values and add it the person node. At the end of the iteration the person node is added to persons array.
  3. Once the iteration is done, we add the array to the listOfPersons object using the putArray and addAll jackson methods.

Summary

In essence this is one approach to map Csv to JSON format and customising your json structure. There are I am sure better ways or other approaches. If you know of others please do post them 😉 Stay tuned for the next series.

 

Camel development series – Part 2

CSV file map to Json file

The is the second part of the series. If you are a complete beginner go to part 1 . This series will touch upon mapping, specifically how to map from a very simple CSV (comma separated file) to a Json structure.

I will keep everything simple and add more complexity in subsequent series. So some prerequisites:

  1. The CSV file will only contain a single row. Handling mulitple rows will be in part 3 of the series.
  2. The Json structure will only contain field names and values.
  3. No error handling is done.

So, what do we want to do? We want to pick up a file from a specific folder, map the single row to an object in order to extract certain values and then map it to the output json format. Finally we save the json string to a file.

Camel vs IBM Integration on mapping

There is something to be said for those like me coming from another background in the integration world, specifically having use IBM WMB/Integration Bus. I think learning Camel and how routes and routebuilder works is somewhat ok once you get used to it. The error handling and logging is ok too, but the big jump from a conceptual point of view is how you do mapping.

Message Parsers

In the IBM Integration Bus you use parsers. Parsers are the key to mapping. You have the XMLNSC parser for parsing XML data, JSON parser for parsing JSON data and DFDL parser for parsing anything else. These are extremely powerful parsers and specifically the DFDL parser is incredible because you are able to model any kind of data as long as it can be described in some way. You can in practise model EDIFACT, MARC, or other complicated formats. Yes, it takes time but it can be done and then you use the parser to match the incoming data with the model you created. The DFDL parser can be debugged and tested as well which makes it very versatile.  What is the big drawback? Price. It is great for large corporations who don’t mind paying large license fees and have a heavy IBM presence. Off course ultimately it means you are locked down to IBM as well since it is proprietary tool. These are factors to consider when you choose your integration tool.

Everything is an object

In the Camel world, since it is java based, there is no concept of parsers or data modelling. This was a big conceptual change for me. Instead it is going back to basics and understanding that everything is an object. This includes a single row in a CSV file. That row is an object is as well. Multiple rows are simple list of objects. With objects we off course mean java objects defined in a class. Understanding this helps to understand the mapping. The big advantages are that for relatively simple formats there is built in support in Camel such as bindy or beanio so you can create your object model and do your mapping. However these components lack to the powerful modelling that exist in the DFDL parser which can model any type of data format. So for very complicated data formats you may need to go back to standard java code implemented in a Processor. There is off course an advantage to this as well. It is easy to read, it is not tied to a tool and it is cheaper. Again, this is something to consider when selecting your integration tool.

Project POM file

So let’s start with the actual code. As in series 1, I am basing this on structure of letting blueprint start my routebuilder and inject beans whilst I write the main code in java.

My POM file dependencies look as follows:

<dependency>
      <groupId>org.apache.camel</groupId>
      <artifactId>camel-core</artifactId>
      <version>2.16.1</version>
    </dependency>
    <dependency>
      <groupId>org.apache.camel</groupId>
      <artifactId>camel-blueprint</artifactId>
      <version>2.16.1</version>
    </dependency>
        <dependency>
      <groupId>org.apache.camel</groupId>
      <artifactId>camel-bindy</artifactId>
      <version>2.16.1</version>
    </dependency>
        <dependency>
      <groupId>org.apache.camel</groupId>
      <artifactId>camel-jackson</artifactId>
      <version>2.16.1</version>
    </dependency>

The two new dependencies are camel-bindy for modeling the CSV data and camel-jackson for creating the Json structure.

Blueprint file

My blueprint.xml is as follows:

<?xml version="1.0" encoding="UTF-8"?>
<blueprint xmlns="http://www.osgi.org/xmlns/blueprint/v1.0.0"        xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"        xsi:schemaLocation="        http://www.osgi.org/xmlns/blueprint/v1.0.0 http://www.osgi.org/xmlns/blueprint/v1.0.0/blueprint.xsd        http://camel.apache.org/schema/blueprint http://camel.apache.org/schema/blueprint/camel-blueprint.xsd">

  <bean id="CsvToJsonRouteBuilder" class="org.souciance.integration.CsvToJson.CSVToJsonRouteBuilder">
  </bean>
  <bean id="IdentityToJson" class="org.souciance.integration.CsvToJson.IdentityToJson">
  </bean>

  <camelContext xmlns="http://camel.apache.org/schema/blueprint">
    <routeBuilder ref="CsvToJsonRouteBuilder" />
  </camelContext>
</blueprint>

The main thing to notice is that we are using the routeBuilder tag to reference our CsvToJsonRouteBuilder class and injecting the bean IdentityToJson which is referring to the class IdentityToJson. This class is where we map to the Json structure.

RouteBuilder class

The CsvToJsonRouteBuilder class looks like this:

package org.souciance.integration.CsvToJson;

import org.apache.camel.builder.RouteBuilder;
import org.apache.camel.dataformat.bindy.csv.BindyCsvDataFormat;

/**
 * A bean which we use in the route
 */
public class CsvToJsonRouteBuilder extends RouteBuilder {

	@Override
	public void configure() throws Exception {
		// TODO Auto-generated method stub
		BindyCsvDataFormat bindy = new BindyCsvDataFormat(org.souciance.integration.CsvToJson.Identity.class);
		from("file:C:/test/?fileName=input.csv")
		.unmarshal(bindy)
		.to("IdentityToJson")
		.to("file:C:/test/?fileName=output.json")
		.log("done!")
		.end();

	}
}

A couple of things to notice here.

Firstly, we are creating a new bindy data format which is CSV based and we are referring to our Identity class. Then we pick up the file using the file component and the file is called input.csv.

Secondly, in the dsl we do an unmarshal(bindy) which means, we want to map the incoming data to the structure defined in the Identity class.

Thirdly, we sent the Identity object to the bean ”IdentityToJson” as defined in the blueprint.xml to map to the Json structure.

Finally we save it as as file called output.json and log ”done!”.

That is all that is needed.

Data model class : Identity

Let us look at the Identity model class where bindy is used.

package org.souciance.integration.CsvToJson;

import org.apache.camel.dataformat.bindy.annotation.CsvRecord;
import org.apache.camel.dataformat.bindy.annotation.DataField;

@CsvRecord(separator = ",")
public class Identity {

	@DataField(pos=1)
	private int identity;
	@DataField(pos=2)
	private String firstname;
	@DataField(pos=3)
	private String lastname;
	@DataField(pos=4)
	private int phone;
	@DataField(pos=5)
	private String country;
	public int getIdentity() {
		return identity;
	}
	public void setIdentity(int identity) {
		this.identity = identity;
	}
	public String getFirstname() {
		return firstname;
	}
	public void setFirstname(String firstname) {
		this.firstname = firstname;
	}
	public String getLastname() {
		return lastname;
	}
	public void setLastname(String lastname) {
		this.lastname = lastname;
	}
	public int getPhone() {
		return phone;
	}
	public void setPhone(int phone) {
		this.phone = phone;
	}
	public String getCountry() {
		return country;
	}
	public void setCountry(String country) {
		this.country = country;
	}

}

This is a very simple way of modelling data in Camel using the bindy component.

Step 1), you use the annotation @CsvRecord(separator = ”,”) to say that this is a CSV model and the separator to delimit each field is a comma. This needs to be done above the class declaration.

Step 2) you write down the fields in the CSV row. Above each field you annotate it with @DataField(pos=X) where pos is the order in which the field appears in the actual row.

Step 3) You add the getter and setters for each field.

That is all that is required for bindy for our simple row of data. For more on bindy and all the annotations you can use see here.

Mapping to Json

Finally, we have the IdentityToJson class where we map to Identity object to json format. Here is how that class looks like:

package org.souciance.integration.CsvToJson;

import java.io.ByteArrayOutputStream;

import org.apache.camel.Exchange;
import org.apache.camel.Processor;

import com.fasterxml.jackson.core.JsonFactory;
import com.fasterxml.jackson.core.JsonGenerator;
import com.fasterxml.jackson.databind.ObjectMapper;
import com.fasterxml.jackson.databind.node.JsonNodeFactory;
import com.fasterxml.jackson.databind.node.ObjectNode;

public class IdentityToJson implements Processor {

@Override
public void process(Exchange exchange) throws Exception {
// TODO Auto-generated method stub

//initialize Jackson
JsonNodeFactory factory = new JsonNodeFactory(false);
JsonFactory jsonFactory = new JsonFactory();
ByteArrayOutputStream outputStream = new ByteArrayOutputStream();
JsonGenerator generator = jsonFactory.createGenerator(outputStream);
ObjectMapper mapper = new ObjectMapper();

//get the Pojo with the CSV data
Identity identity = (Identity)exchange.getIn().getBody();

//map to Json
ObjectNode id = factory.objectNode();
id.put("identity", identity.getIdentity());
id.put("firstname", identity.getFirstname());
id.put("lastname", identity.getLastname());
id.put("phone", identity.getPhone());
id.put("country", identity.getCountry());

//write the json string to the exchange
mapper.writeTree(generator, id);
String json = new String(outputStream.toString());
exchange.getIn().setBody(json);
}
}

The main aspects are related to Jackson code. The first part is Jackson initialization where we create a JsonNodeFactory, JsonFactory and ByteOutputStream because we want to write the data to the stream which will be transformed to a string at the end.

In the second part we convert the body in the exchange to an instance of Identity. This is the crucial part because now we have access to the row data and can do our mapping.

The next part creates a node and starts putting in values in the json structure.

Finally we write the json string to the exchange.

Input and Output

When you run this with input.csv containing the row:
12345,souciance,eqdam rashti,012458478,Sweden

your output.json should be
{
”identity”:12345,
”firstname”:”souciance”,
”lastname”:”eqdam rashti”,
”phone”:12458478,
”country”:”Sweden”
}

Feel free to leave comments. In part 3 we will go through mapping multiple rows of data. Stay tuned!

Camel development series – Part 1

Hello again!

It has been a while since I wrote so I thought I’d start this year with some basic Camel development series. The main reason is that I feel one aspect lacking in the documentation is a good step by step introduction for some basic mapping. How do you go from Csv (comma separated format) to Json format? After all mapping is a big part of integration. They can be very simple from field to field mapping to extremely complicated business logic. Although personally adding any business logic to your integration code is very risky since now you have another place to keep track of your business domain.

Anyway, I am not sure how many parts this will be but we will start with basics and then add additional mapping complexity as well as error handling, and logging until we get a somewhat reasonable and stable integration. For this I am  hoping to add to the series once a week. Feel free to add suggestions or thoughts in the comments section!

Hello world

So let’s get started with the basics. All the code for this example can be found here https://github.com/SoucianceEqdamRashti/Integration/tree/master/HelloWorld

We want to build an extremely simple HTTP listener on a specific URL and port and returns a simple string when you call it. Nothing more complicated than that.

For my development environment I am using Eclipse Mars and importing the camel archetypes from Maven. I am also using Camel version 2.16.1. You can off course use another IDE if you prefer. That should not matter.

The first thing is my pom file and specifically my dependencies. Here is what I have imported:

  <dependencies>
    <dependency>
      <groupId>org.apache.camel</groupId>
      <artifactId>camel-core</artifactId>
      <version>2.16.1</version>
    </dependency>
    <dependency>
      <groupId>org.apache.camel</groupId>
      <artifactId>camel-blueprint</artifactId>
      <version>2.16.1</version>
    </dependency>
    <dependency>
      <groupId>org.apache.camel</groupId>
      <artifactId>camel-restlet</artifactId>
      <version>2.16.1</version>
      <type>bundle</type>
    </dependency>

As you can see most of it is standard camel core. The extra dependency is camel-restlet and we will use that for the HTTP communication.

So how do we start our camelcontext and create a route? Well there is the Java DSL and the blueprint dsl. One thing to bear in mind is where you want to deploy this. That determines a lot how you choose your dsl. Since I mainly work with Karaf I will select blueprint to create my beans and start the routebuilder but I will use the java dsl for writing the routes and all other code. This way, I get the best from both worlds.

This means when you start from scratch you should base your maven project on the archetype camel-blueprint.

So here is how my blueprint.xml looks like:

<?xml version="1.0" encoding="UTF-8"?>
<blueprint xmlns="http://www.osgi.org/xmlns/blueprint/v1.0.0"
       xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
       xsi:schemaLocation="
       http://www.osgi.org/xmlns/blueprint/v1.0.0 http://www.osgi.org/xmlns/blueprint/v1.0.0/blueprint.xsd
       http://camel.apache.org/schema/blueprint http://camel.apache.org/schema/blueprint/camel-blueprint.xsd">

  <bean id="HelloWorld" class="org.souciance.integration.helloworld.HelloWorld">      
 </bean>
  <camelContext xmlns="http://camel.apache.org/schema/blueprint">
    <routeBuilder ref="HelloWorld" />    
    </camelContext>
</blueprint>

The main part is the bean injection where we are injecting the class HelloWorld as bean into the Camel registry. The actual class name is HelloWorld and the prefix is the package name.

Since we want to write the main code in the java dsl we need to refer to it.
Hence we use the routeBuilder tag and refer to the bean ”HelloWorld” declared earlier.

That is all the blueprint we need.

In my HelloWorld class I have written the following code:

package org.souciance.integration.helloworld;

import org.apache.camel.builder.RouteBuilder;

/**
 * A bean which we use in the route
 */
public class HelloWorld extends RouteBuilder {

	@Override
	public void configure() throws Exception {
		// TODO Auto-generated method stub
		
		from("restlet:http://localhost:5000/hello")
		.log("request received")
		.setBody().simple("Hello to you too")
		.end();
		
	}
}

The main thing to pick up is that we are extending RouteBuilder and overriding the method configure() where all the action takes place. Inside the method we write our dsl code.

The actual dsl code is very simple. We listen on the url http://localhost:5000/hello, then we write a simple log text, and finally we set the response body to ”Hello to you too”. As you can see we are building an expression using the simple language to putting data in the exchange body. The simple expression language is very powerful and we will come back to it in more detail in other parts of the series.

That is all the code necessary.

How do you run it?

Well before deploying it you want to test that it works. The way I do in eclipse is the following:
1. Right click on the project.
2. Select Run As –> Maven Build
3. In the new build configuration as my goal I write camel:run.
4. If you no tests you can select skip tests.

Then simply run it with that build configuration.

If you run it and do a http get on that url you should see this in the console:

INFO: Starting the internal [HTTP/1.1] server on port 5000
[         Blueprint Extender: 1] BlueprintCamelContext          INFO  Route: route1 started and consuming from: Endpoint[http://localhost:5000/hello?restletMethods=GET]
[         Blueprint Extender: 1] BlueprintCamelContext          INFO  Total 1 routes, of which 1 is started.
[         Blueprint Extender: 1] BlueprintCamelContext          INFO  Apache Camel 2.16.1 (CamelContext: camel-1) started in 0.413 seconds
[              Restlet-26556173] route1                         INFO  request received
jan 31, 2016 1:32:04 EM org.restlet.engine.log.LogFilter afterHandle
INFO: 2016-01-31	13:32:04	0:0:0:0:0:0:0:1	-	-	5000	GET	/hello	-	200	16	0	16	http://localhost:5000	Mozilla/5.0 (Windows NT 10.0; WOW64; rv:43.0) Gecko/20100101 Firefox/43.0	-

The log shows that we have started the camelcontext, listening on that url, then we receive a HTTP request, we write the log text ”request received” and return the response.

This is a simple example how to get started with Camel.

In the next series we will look more into mapping. Feel free to comment if you have any thoughts or questions.

Camel basic examples

Hope you guys have had a good holiday and good start to the new year. I have finally managed to get my github account organized and promised myself to put more code there to have publicly available and to access it whenever needed. This off course means that you guys can get hold of the code too 😉 So first of the link to the main account is:

https://github.com/SoucianceEqdamRashti/Integration

You will find two projects there. Let’s go through them one by one.

Heartbeat

This is a basic hello world style integration. It consists of two parts.

  1. blueprint.xml which initiates the actual integration and references the java class Heartbeat.java in bean style.
  2. Heartbeat.java which extends the RouteBuilder class and simply exposes a REST url which the client performs a GET operation and gets a simple string as a response.
  3. Import the project in your IDE and run it.
  4. Write http://0.0.0.0/heartbeat in your browser and you should get the string ”working” as response.

Blueprint code

The blueprint code can be found under OSGI/blueprint.xml.

The core idea is the following code:

<?xml version="1.0" encoding="UTF-8"?>
<blueprint xmlns="http://www.osgi.org/xmlns/blueprint/v1.0.0"
       xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
       xsi:schemaLocation="
       http://www.osgi.org/xmlns/blueprint/v1.0.0 http://www.osgi.org/xmlns/blueprint/v1.0.0/blueprint.xsd
       http://camel.apache.org/schema/blueprint http://camel.apache.org/schema/blueprint/camel-blueprint.xsd">

	<bean id="heartbeat" class="com.souciance.integration.camel.Heartbeat">
	</bean>


	<camelContext xmlns="http://camel.apache.org/schema/blueprint">
		<routeBuilder ref="heartbeat"/>
	</camelContext>

</blueprint>

This is the approach I recommend as a best practise if you are developing integrations with Camel and Blueprint. Basically let blueprint initialize and then write all your main code using the java dsl inside the RouteBuilder class.

Java code

The java code is very simple and looks as follows:

public class Heartbeat extends RouteBuilder {

    @Override
    public void configure() throws Exception {
    	from("restlet:http://0.0.0.0/heartbeat?restletMethod=GET")
    	.setBody().constant("Working")
    	.log("response sent");
    	
    }
}

We are essentially defining our ”from” to be a REST URL and the response is a constant string defined as ”Working”. Finally we write a simple log output. To access this write http://localhost/heartbeat in the browser.

This concludes the basic heartbeat integration. You could use this in more advanced setting to deploy this to your Karaf environment or elsewhere and simply let your monitoring software call this integration to act as a heartbeat monitor.

Synchronous ErrorHandler

This is a more advanced example. Here we define several concepts that are core to Camel and essential to any production ready integration. You will find the code here:
https://github.com/SoucianceEqdamRashti/Integration/tree/master/synchronousErrorHandler

Again it consists of two parts, one blueprint and one java code.

Blueprint code

The blueprint code can be found under OSGI/blueprint.xml.

The core idea is the following code:

<?xml version="1.0" encoding="UTF-8"?>
<blueprint xmlns="http://www.osgi.org/xmlns/blueprint/v1.0.0"
       xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
       xsi:schemaLocation="
       http://www.osgi.org/xmlns/blueprint/v1.0.0 http://www.osgi.org/xmlns/blueprint/v1.0.0/blueprint.xsd
       http://camel.apache.org/schema/blueprint http://camel.apache.org/schema/blueprint/camel-blueprint.xsd">

    <bean id="synchronousErrorHandler" class="com.souciance.integration.camel.synchronousErrorHandler.ErrorHandler">
    </bean>
   

  <camelContext xmlns="http://camel.apache.org/schema/blueprint">
    <routeBuilder ref="synchronousErrorHandler"/>
  </camelContext>

</blueprint>

This is identical to the previous example. The only difference is the name of the java class we are referring to.

Let’s move on the java code.

Java code

The entire java code can be seen here:

package com.souciance.integration.camel.synchronousErrorHandler;

import org.apache.camel.Exchange;
import org.apache.camel.Processor;
import org.apache.camel.builder.RouteBuilder;
import org.apache.camel.model.dataformat.JsonLibrary;
import org.apache.camel.model.rest.RestBindingMode;

import com.fasterxml.jackson.annotation.JsonAutoDetect.Visibility;
import com.fasterxml.jackson.annotation.PropertyAccessor;
import com.fasterxml.jackson.core.JsonProcessingException;
import com.fasterxml.jackson.databind.ObjectMapper;


/**
 * @author Souciance Eqdam Rashti
 * Showing ways of handling synchronous processing
 *
 */
public class ErrorHandler extends RouteBuilder {

    @Override
    public void configure() throws Exception {
    	
    	onException(com.fasterxml.jackson.core.JsonParseException.class).handled(true)
    	.setHeader("CamelHttpResponseCode",simple("500"))
    	.setHeader("Content-Type" ,simple("text/plain"))
    	.setBody().constant("Internal Server Error");
    	
    	onException(InvalidMediaTypeException.class).handled(true)
    	.setHeader("CamelHttpResponseCode", simple("415"))
    	.setHeader("Content-Type", simple("text/plain"))
    	.setBody().constant("Invalid Media Type");
    	    	
    	restConfiguration().component("restlet").host("localhost").port(7070).bindingMode(RestBindingMode.auto).componentProperty("chunked", "true");
    	
    	rest().path("integration").get("/object/1234").produces("application/json").bindingMode(RestBindingMode.json).to("direct:getObject");
    	
    	rest().path("integration").get("/object/12345").produces("application/json").bindingMode(RestBindingMode.auto).to("direct:getObjectError");
    	
    	rest().path("integration").post("/object").bindingMode(RestBindingMode.off).to("direct:postObject");
    	
    	from("direct:getObject")
    	.setBody().constant(createJsonString()).unmarshal().json(JsonLibrary.Jackson)    	
    	.log("get request received");
    	
    	from("direct:getObjectError")
    	.setBody().constant("sdfsdf:sdf").unmarshal().json(JsonLibrary.Jackson)
    	.log("get request error received");
    	
    	from("direct:postObject")
    	.process(new Processor() {

			@Override
			public void process(Exchange exchange) throws Exception {
				// TODO Auto-generated method stub
				String contentType=(String) exchange.getIn().getHeader("Content-Type");
				if (contentType.equals("application/json")) {
					exchange.getIn().removeHeaders("*");			
					exchange.getOut().setBody("OK");
					exchange.getOut().setHeader("Content-Type", "text/plain");
					exchange.getOut().setHeader("CamelHttpResponseCode", "201");
				}
				else {
					throw new InvalidMediaTypeException("Invalid media type received!");
				}	
			}						
    		
    	})
    	.log("Message received: ${body}");

    
    }
    
    private String createJsonString() throws JsonProcessingException {
    	ObjectMapper mapper = new ObjectMapper();
    	mapper.setVisibility(PropertyAccessor.FIELD, Visibility.ANY);
    	dummyClassForJson jsonObj = new dummyClassForJson();
    	
    	String json = mapper.writeValueAsString(jsonObj);
    	return json;
    }
}

final class InvalidMediaTypeException extends Exception {


	public InvalidMediaTypeException() {
		
	}
	
	public InvalidMediaTypeException(String message) {
		super(message);
	}
	
}

class dummyClassForJson {
	  private String firstname="souciance";
	  private String lastname="eqdam rashti";
	  private String country="sweden";
	  private int age=34;
	  dummyClassForJson() {
	    // no-args constructor
	  }
	}

This is significantly more code than previously. Let’s go through some of the key parts.

onException(com.fasterxml.jackson.core.JsonParseException.class).handled(true)
    	.setHeader("CamelHttpResponseCode",simple("500"))
    	.setHeader("Content-Type" ,simple("text/plain"))
    	.setBody().constant("Internal Server Error");
    	
    	onException(InvalidMediaTypeException.class).handled(true)
    	.setHeader("CamelHttpResponseCode", simple("415"))
    	.setHeader("Content-Type", simple("text/plain"))
    	.setBody().constant("Invalid Media Type");
    	

The above defines the key concept of exception handling. I am defining an onException handler based on certain exceptions, set the handler to be true and then set specific headers and response body based on the error I am catching. The example also shows how to set header values, how to customize the return header and value of REST calls. The first onException returns a HTTP 500 because the Json string returned in the internal processing is not correctly formatted. However we don’t want the caller to receive the stack trace and internal error trace. That is why we set handled=true and set the HTTP response headers and body.

The second main part is the REST dsl which is new to Camel where using a dsl style code you can write REST components.

restConfiguration().component("restlet").host("localhost").port(7070).bindingMode(RestBindingMode.auto).componentProperty("chunked", "true");
    	
    	rest().path("integration").get("/object/1234").produces("application/json").bindingMode(RestBindingMode.json).to("direct:getObject");
    	
 

You see that we define a restConfiguration and define RESTLET as our main HTTP component, the host, port, and the binding mode to be auto. You can select JETTY or other instead of RESTLET and there are many other options to add to the restConfiguration. Finally using rest() you define the path, operation and the producer and consumer options and where the request should get directed to. In the example above the request goes to ”direct:getObject”.

One important part to note. RestBindingMode.json essentially means that Camel expects the request and response to be JSON. It means that if you happen to write plain text as error string Camel will still format it as Json. For example if you return OK the caller will actually see ”OK” if you set RestBindingMode.json. If you know you will return plain text set RestBindingMode.off. Then the response will be OK as well.

The final part we will go through is the POST operation.

from("direct:postObject")
    	.process(new Processor() {

			@Override
			public void process(Exchange exchange) throws Exception {
				// TODO Auto-generated method stub
				String contentType=(String) exchange.getIn().getHeader("Content-Type");
				if (contentType.equals("application/json")) {
					exchange.getIn().removeHeaders("*");			
					exchange.getOut().setBody("OK");
					exchange.getOut().setHeader("Content-Type", "text/plain");
					exchange.getOut().setHeader("CamelHttpResponseCode", "201");
				}
				else {
					throw new InvalidMediaTypeException("Invalid media type received!");
				}	
			}						
    		
    	})
    	.log("Message received: ${body}");    	

Here you see an example where we create a custom processor to investigate the incoming header on the exchange and evaluate the content-type. If the content-type is not application/json we through a custom exception. This exception is caught by the onException handler and returns a standard response back to the caller. Finally you see how we can log the body by using the ${body} syntax.

If you have any comments or questions please let me know!

Stay tuned for more examples.

Java dsl vs Blueprint

Recently I was thinking whether it would be good to stick to either the purse Java dsl or the blueprint one, especially when deploying to the Karaf environment.

One of the good things about the blueprint xml is that it makes the OSGI stuff much easier to handle since Karaf can work well with blueprint so there is no need to OSGI:fy your code.

One the other hand, the java dsl is simply easier to work with and most of the documentation is for the java dsl so you will find more help. Also, who knows, in 5 years maybe blueprint will be replaced so the java dsl feels more safe.

So I was curious to know how other Camel users felt hence I put this question the Camel Nabble forum here:

http://camel.465427.n5.nabble.com/java-dsl-vs-blueprint-xml-td5775085.html

The consensus seemed to be that one could use Blueprint as a start-up mechanism to load the CamelContext and then build the routes, onexception and all other code in a RouteBuilder class which can be referenced via a bean.

I think this makes best use of both worlds. You use only blueprint for loading the routes and use java to build the actual routes and all the extra layers for logging and error handling. This way, if blueprint is replaced it only affects your loading and not the main routing part.

See example here:

https://github.com/cschneider/Karaf-Tutorial/blob/master/camel/jms2rest/src/main/java/net/lr/tutorial/karaf/camel/jms2rest/Jms2RestRoute.java

https://github.com/cschneider/Karaf-Tutorial/blob/master/camel/jms2rest/src/main/resources/OSGI-INF/blueprint/blueprint.xml

There was a mention of the groovy dsl and I know there is a scala one as well. I haven’t used it and there is very little documentation on both so it is hard to learn quickly by examples but if anyone has used it feel free to comment on there ease of use.