Where to run Apache Camel?

So, you have no decided you want to use Apache Camel for your integration needs. That’s great, welcome to the party! However, once you start reading the API, the documentation and examples, one question that will soon come to your mind is, ”So where do I run my integration in?”.

In commercial platforms such as IBM Integration Bus you have an Integration Server with nodes or execution groups where your integrations get deployed and you manage them. Same goes for other commercial vendors, that is, you have a development part and a runtime part. So where is Camel’s runtime?

Actually Camel does not come with a runtime. There are probably many reasons for that, the main one as far as I can tell is that the project wants to be runtime agnostic. That is, it should support running on several different types of runtime environment rather than tied to a single one as you get with a commercial platform.

So, what options are there and what are some of their advantages and disadvantages?

Running as a standalone jar

Well you can always run as a standalone jar file, and add a installer and then run it as windows service so it is always running in the background. I did that once for a client project. It works great if what you are after is a single application to do one specific thing and you have no administrative needs or anything moving towards an integration platform. Suppose you just want to transfer a file from folder A to folder B and it should do this whenever a file ends up on folder A. Great, build your camel route and run it as a standalone jar. But this does approach does not work when you want to build an integration platform. You need much more functionality in terms of logging, error handling, retry, administrating your integrations, flexible start/stop, deployment, removing and much more.

Running inside ServiceMix

ServiceMix is an Apache project that can be found here. It was probably the first open source runtime environment for Camel and was used a lot by other companies such as Fuse Source to provide a runtime environment. I believe the project is still active and new functionality is being added.

However what sort of put me off going the ServiceMix direction is this blog by Claus Ibsen . Straight away the question is asked whether you should use ServiceMix or not and Claus’s answer is an affirmative no. The reasons are several but seem mostly to be that ServiceMix had connections to JBI and JBI was essentially dismissed along the way. Then the kernel of ServiceMix was extracted and put into a seperate project and became Apache Karaf. The hope was then that ServiceMix would improve on providing ESB-style functionality but this hasn’t really happened and the project hasn’t caught up. For this reason I decided not to test ServiceMix out however it could be that the project does make a major comeback.

Running inside Apache Karaf

This is the venture I went instead. Why? Well for several reasons, firstly because I started early on to look at JBoss Fuse which is built upon Apache Karaf. However since Fuse was using an older Karaf version I decided to look upon Karaf itself to see whether I could strip away some layers and get the lightweight runtime instead and simply add the functionality when needed.

Karaf can be found here . Although the website is quit sparse as is most Apache projects the most relevant things are there. I will go through Karaf in more detail and some important commands in another posting. But why did I decide go with it?

  1. It supported some very easy deployment mechanism that not even commercial vendors offer. I can simply take my OSGI bundle and put it straight in a folder and bam, the integration is deployed. I don’t need to run any commands or do anything. Just drag and drop. This is the simplest form of deployment. Off course I can deploy from commands as well but the fact that in a development environment it works so easily is great for a developer. Then, as you move to production you can automate the deployment. This feature is awesome!
  2. I have to say the development team and in particular Jean-Baptiste Onofré , have been really awesome and helpful. I got really good support when I wanted to install Apache Decanter and got really good help on this. This is off course very cool and shows how dedicated they are to their project.
  3. Karaf is also awesome because it makes certain administrative functionality very easy. I can easily configure MDC logging for camel so that my log files are based on my bundles. I can easily via camel commands inspect my routes. There is much more. You can even create Kar archive which is a complete package for your integration so you don’t need a maven repo. This is good for production environment in case you don’t want to access a maven repo for deployment. There is even a provisioning feature to start a Karaf instance with certain features so you can just quickly start Karaf with Camel pre-installed.
  4. Another great feature is Apache Decanter . It gives you so much functionality that you wonder how come it is free. You essentially get a customised Kibana/Elasticsearch dashboard that can monitor Karaf, ActiveMQ and Camel. Ontop that it has SLA alerts support so if something goes wrong e-mail someone to take some action. Usually these type of BAM like functionality costs lots of money and often you have to go to third-party applications. One thing, use version 1.0.1 when installing it. There was a memory bug that was fixed and it works really smoothly now. Like I said you get A LOT for free.

Well there is way more to say but I will go into Karaf in more detail in another posting. It is an OSGI container so you need to either OSGI:fy your java app or use blueprint or a mixture of the two. I personally use blueprint mixed with java to get the best from both worlds.

Running inside springboot

You can also run Camel inside springboot. See here and here. This is great for those with Spring background and know how to use it. Unfortunately I don’t have Spring competence and didn’t really want to learn another framework together with Camel. So that is why I cannot say much from experience on this approach. If you are big on Spring then defintely take a look!

Running with Fabric8

Fabric8 is the new integration platform developed by James Strachan and supported by Red Hat. It is essentially an answer to development and deployment the cloud, essentially with things like docker, kubernetes and other container like technology. I have not yet had time to play with fabric8 but have heard good things about it. It provides the administrative functionality you want from a platform but has lots of plugins to add extra components and is geared to support docker and kubernetes style deployment and scalability. If you are heading towards this approach then defintely look into Fabric8!

So as you can tell there are several approaches to running Camel. You need to clarify your requirements, your development environment, your production environment and what needs you foresee after 1-2 years. My recommendation is to start small. Perhaps pick up Karaf and learn how to development integrations and deploy and build up a best practise, then move into docker and build a docker container for your karaf installation and then finally look into Fabric8 for scalability.

 

Starting point

Hello,

Right so here comes some good links to bookmark and read upon for all things related to open source..

Apache Camel

Apache Karaf

HawtIO

Claus Ibsen’s blog

Camel Nabble forum

Apache Camel inside Apache Karaf

Jean-Baptiste’s blog on Camel and Karaf related material

Introduction to Apache Karaf

Blog explaining blueprint together with Apache Karaf

Performance tuning Apache Camel

Apache Conference presentation on camel performance optimization

I will update with more links continuously. Thanks.

Apache Camel vs Proprietary competitors

In this inaugural post I am going to describe a bit about what made me choose to work with Camel and Karaf and what you gain and lose compared to its commercial competitors such as IBM Integration Bus. In essence, what things should you look out for when deciding on your integration platform?

Having worked with the IBM WebSphere Message Broker for more than 8 years you get used to its philosophy, concepts and tooling. Its when you step outside that you realize other tools  work differently.

Most commercial tools follow a somewhat similar conceptual model but how they implement it can very significantly.  However if you are about to start choosing an integration platform, here are some things you should look out for:

  1. If you are not familiar with the vendor landscape for integration, start with reading Gartner and Forrest reports to just get a feel. Don’t base your decision on them, rather use them to get a feel for who are the players and then use the list for an evaluation.
  2. Forget about fancy vendor demos. They are done in an idealized situation with an extremely simple scenario. No real life integration is like that. Demos are good to peak but not to base your decision on. Ideally, download their tooling and see how long or hard it is to create your own demo. Get a feel for the tooling yourself.
  3. Clearly understand your integration requirements. Talk to system owners and people who need integration to understand why and what role it will play.
  4. Understand your message formats and protocols, will it be used to connect to legacy systems or use modern REST and JSON?
  5. Cost is off course important. Try to do a total cost of ownership analysis to see how much your tooling, license, OS etc will cost in the next year, three years, and 5 years to see where it aligns with your budget.
  6. Is the vendor innovating its integration platform and attracting new customers?
  7. How does the licensing and support model of the vendor work?

The above are just a handful of questions you need to address before making a decision.

Now, now what about Camel?

Let us get one thing straight first and that is cost. Apache Camel is free. It does not cost you one dollar to use. IBM software usually costs a lot, probably more than 80 000 USD and the pricing model can be very complicated. Already you have one big difference.

However, the comparison is not fair even if there is a huge price difference. IBM’s Integration Bus is complicated piece of software that is the complete packaging, from routing, transformation to tooling and run-time. It contains all aspects of development and run-time administration. Apache Camel is first and foremost a framework and its strength lies in its simplicity and extensive routing capability. However due to its lightweight nature you are also responsible for the development tooling, finding a suitable run-time and handling monitoring and administration. Here the differences depends very highly on your needs. Camel places a far greater responsibility on the developer. It see the developer as the main person working with integrations.

I would say from personal experience that the biggest difference between the two is in the transformation part. I have never seen a better transformation tool than the Integration Bus message parsers. The concept of parsers and the ability to use them and refer to fields and messages in real time is highly helpful when you are developing mapping of data. In contrast Camel, being purely a programming language framework cannot use the concept of parsers and hence for transformation you need to either map to a java object (pojo) first or use external tools such as smooks or similar. For me, this is the single biggest difference and the one part of tooling you miss when you leave the IBM world.

I finally decide on the following components for the platform that I am working with:

  1. Apache Camel for routing and development. Why?
    1. It is free.
    2. Avoiding vendor locking.
    3. Simple to use.
    4. Based on java so easy to get started.
    5. The team innovates and fixes the framework and it is standard for integration in the open source world.
    6. Uses patterns based on Enterprise Integration Patterns book.
    7. Not tied to a specific run-time.
    8. There are many more reasons but one final one is that it is very lightweight but at the same time contains exception handling, error handling, testing, mocking and many more advanced features.
  2. Apache Karaf for OSGI run-time environment.
    1. Karaf is very light weight, just a zip file and you are good to go.
    2. It contains many advanced features missing in commercial tools.
    3. Deployment is extremely easy.
    4. It works with Camel in a relatively straightforward manner.
    5. The logging framework is great.
    6. Administrating bundles is also very easy. There are even specific camel commands so that makes life really nice when deploying camel routes.
    7. It supports installing Hawtio and Apache ActiveMQ inside its run-time.
  3. Hawtio for webconsole.
    1. Hawtio is great to debug, profile and view your camel routes. You can also view the Karaf logs, manage your camel routes and much more.

Finally, after having used Camel now for about one month, I have to see I am pleasantly surprised because it has a lot of functionality for being open source and even functionality not found in commercial software. One huge benefit I think working with Camel and its ecosystem is that you gain a more in-depth understand of software, development and get to hone your skills more. You are not just an expert at a specific vendor tool, instead you can I definitely need to read the documentation more and understand some best practice but one step at a time 😉 Thanks for tuning in.