Monday, May 5, 2014

Continuous Integration with JBoss Fuse, Jenkins and Nexus

Recently I was putting together a quickstart Maven project to show a possible approach to the organization of a JBoss Fuse project.

The project is available on Github here: https://github.com/paoloantinori/fuse_ci

And it’s an slight evolution of what I have learnt working with my friend James Rawlings

The project proposes a way to organize your codebase in a Maven Multimodule project.
The project is in continuous evolution, thanks to feedback and suggestions I receive; but it’s key point is to show a way to organize all the artifacts, scripts and configuration that compose your project.

In the ci folder you will find subfolders like features or karaf_scripts with files you probably end up creating in every project and with inline comments to help you with tweaking and customization according to your specific needs.
The project makes also use of Fabric8 to handle the creation of a managed set of OSGi containers and to benefit of all its features to organize workflows, configuration and versioning of your deployments.
In this blogpost I will show you how to deploy that sample project in a very typical development setup that includes JBoss Fuse, Maven, Git, Nexus and Jenkins.
The reason why I decided to cover this topic is because I find that many times I meet good developers that tell me that even if they are aware of the added value of a continuous integration infrastructure, have no time to dedicate to the activity. With no extra time they focus only to development.

I don’t want you to evangelize around this topic or try to tell you what they should do. I like to trust them and believe they know their project priorities and that they accepted the trade-off among available time, backlog and the added overall benefits of each activity. Likewise I like to believe that we all agree that for large and long projects, CI best practices are definitely a must-do and that no one has to argue about their value.

With all this in mind, I want to show a possible setup and workflow, to show how quickly it is to invest one hour of your time for benefits that are going to last longer.

I will not cover step by step instructions. But to prove you that all this is working I have created a bash script, that uses Docker, and that will demonstrate how things can be easy enough to get scripted and, more important, that they really work!

If you want to jump straight to the end, the script is available here:
https://github.com/paoloantinori/fuse_ci/blob/master/ci/deploy_scripts/remote_nexus.sh

It uses some Docker images I have created and published as trusted builds on Docker Index:
https://index.docker.io/u/pantinor/fuse/
https://index.docker.io/u/pantinor/centos-jenkins/
https://index.docker.io/u/pantinor/centos-nexus/

They are a convenient and reusable way to ship executables and since they show the steps performed; they may also be seen as a way to document the installation and configuration procedure.
As mentioned above, you don’t necessarily need them. You can manually install and configure the services yourself. They are just an verified and open way to save you some time or to show you the way I did it.


Let’s start describing the component of our sample Continuous Integration setup:

1) JBoss Fuse 6.1
It’s the runtime we are going to deploy onto. It lives in a dedicated box. It interacts with Nexus as the source of the artifacts we produce and publish.

2) Nexus
It’s the software we use to store the binaries we produce from our code base. It is accessed by JBoss Fuse, that downloads artifacts from it but it is also accessed from Jenkins, that publishes binaries on it, as the last step of a successful build job.

3) Jenkins
It’s our build jobs invoker. It publishes its outputs to Nexus and it builds its output if the code it checked out with Git builds successfully.

4) Git Server
It’s the remote code repository holder. It’s accessed by Jenkins to download the most recent version of the code we want to build and it’s populated by all the developers when they share their code and when they want to build on the Continous Integration server. In our case, git server is just a filesystem accessed via ssh.
Interaction Diagram
http://yuml.me/edit/7e75fab5

git

 

First thing to do is to setup git to act as our source code management (SCM).
As you may guess we might have used every other similar software to do the job, from SVN to Mercurial, but I prefer git since it’s one of the most popular choices and also because it’s an officially supported tool to interact directly with Fabric8 configuration
We don’t have great requirements for git. We just need a filesystem to store our shared code and a transport service that allows to access that code.
To keep things simple I have decided to use SSH as the transport protocol.
This means that on the box that is going to store the code we need just sshd daemon started, some valid user, and a folder they can access.
Something like:

yum install -y sshd git
service sshd start
adduser fuse
mkdir -p /home/fuse/fuse_scripts.git
chmod a+rwx /home/fuse/fuse_scripts.git # or a better stratey based on guid
 
While the only git specific step is to initialize the git repository with

git init --bare /home/fuse/fuse_scripts.git

Nexus

 

Nexus OSS is a repository manager that can be used to store Maven artifacts.
It’s implemented as a java web application. For this reason installing Nexus is particularly simple.
Thanks to the embedded instance of Jetty that empowers it, it’s just a matter of extracting the distribution archive and starting a binary:

wget http://www.sonatype.org/downloads/nexus-latest-bundle.tar.gz /tmp/nexus-latest-bundle.tar.gz
tar -xzvf /tmp/nexus-latest-bundle.tar.gz -C /opt/nexus
/opt/nexus/nexus-*/bin/nexus 
 
Once started Nexus will be available by default at this endpoint:
http://your_ip/8081/nexus
with admin as user and admin123 as password.


Jenkins

 

Jenkins is the job scheduler we are going to use to build our project. We want to configure Jenkins in such a way that it will be able to connect directly to our git repo to download the project source.
To do this we need an additional plugin, Git Plugin.
We obviously also need java and maven installed on the box.
Being Jenkins configuration composed of various steps involving the interaction with multiple administrative pages, I will only give some hints on the important steps you are required to perform. For this reason I strongly suggest you to check my fully automated script that does everything in total automation.
Just like Nexus, Jenkins is implemented as a java web application.
Since I like to use RHEL compatible distribution like Centos or Fedora, I install Jenkins in a simplified way. Instead of manually extracting the archive like we did for Nexus, I just define the a new yum repo, and let yum handle the installation and configuration as a service for me:

wget http://pkg.jenkins-ci.org/redhat/jenkins.repo -O /etc/yum.repos.d/jenkins.repo
rpm --import http://pkg.jenkins-ci.org/redhat/jenkins-ci.org.key
yum install jenkins
service jenkins start
 
Once Jenkins is started you will find it’s web interface available here:
http://your_ip:8080/

By default it’s configured in single user mode, and that’s enough for our demo.
You may want to verify the http://your_ip:8080/configure to check if values for JDK, Maven and git look good. They are usually automatically picked up if you have those software already installed before Jenkins.

Then you are required to install Git Plugin:
http://your_ip:8080/pluginManager

Once you have everything configured, and after a restart of Jenkins instance, we will be able to see a new option in the form that allows us to create a Maven build job.
Under the section: Source Code Management there is now the option git. It’s just a matter of providing the coordinates of your SSH server, for example:

ssh://fuse@172.17.0.5/home/fuse/fuse_scripts.git



And in the section Build , under Goals and options, we need to explicitly tell Maven we want to invoke the deploy phase, providing the ip address of the Nexus insance:

clean deploy -DskipTests -Dip.nexus=172.17.0.3

The last configuration step, is to specify a different maven settings file, in the advanced maven properties , that is stored together with the source code:
https://github.com/paoloantinori/fuse_ci/blob/master/my_settings.xml

And that contains user and password to present to Nexus, when pushing artifacts there.

The configuration is done but we need an additional step to have Jenkins working with Git.
Since we are using SSH as our transport protocol, we are going to be asked, when connecting to the SSH server for the first time, to confirm that the server we are connecting to is safe and that its fingerprint is the one the we were expecting. This challenge operation will block the build job, since a batch job and there will not be anyone confirming SSH credentials.
To avoid all this, a trick is to connect to the Jenkins box via SSH, become the user that is used to run Jenkins process, jenkins in my case, and from there, manually connect to the ssh git server, to perform the identification operation interactively, so that it will no longer required in future:

ssh fuse@IP_GIT_SERVER
The authenticity of host '[172.17.0.2]:22 ([172.17.0.2]:22)' can't be established.
DSA key fingerprint is db:43:17:6b:11:be:0d:12:76:96:5c:8f:52:f9:8b:96.
Are you sure you want to continue connecting (yes/no)? 
 
The alternate approach I use my Jenkins docker image is to totally disable SSH fingerprint identification, an approach that maybe too insecure for you:

mkdir -p /var/lib/jenkins/.ssh ;  
printf "Host * \nUserKnownHostsFile /dev/null \nStrictHostKeyChecking no" >> /var/lib/jenkins/.ssh/config ; 
chown -R jenkins:jenkins /var/lib/jenkins/.ssh
 
If everything has been configured correctly, Jenkins will be able to automatically download our project, build it and publish it to Nexus.

But…

Before doing that we need a developer to push our code to git, otherwise there will not be any source file to build yet!
To to that, you just need to clone my repo, configure an additional remote repo (our private git server) and push:

git clone git@github.com:paoloantinori/fuse_ci.git
git remote add upstream ssh://fuse@$IP_GIT/home/fuse/fuse_scripts.git
git push upstream master
 
At this point you can trigger the build job on Jenkins. If it’s the first time you run it Maven will download all the dependencies, so it may take a while.
if everything is successful you will receive the confirmation that your artifacts have been published to Nexus.

JBoss Fuse

 

Now that our Nexus server is populated with the maven artifacts built from our code base, we just need to tell our Fuse instance to use Nexus as a Maven remote repository.
Teaches us how to do it:
In a karaf shell we need to change the value of a property,

fabric:profile-edit  --pid io.fabric8.agent/org.ops4j.pax.url.mvn.repositories=\"http://172.17.0.3:8081/nexus/content/repositories/snapshots/@snapshots@id=sample-snapshots\" default

And we can now verify that the integration is completed with this command:

cat  mvn:sample/karaf_scripts/1.0.0-SNAPSHOT/karaf/create_containers
 
If everything is fine, you are going to see an output similar to this:

# create broker profile
fabric:mq-create --profile $BROKER_PROFILE_NAME $BROKER_PROFILE_NAME
# create applicative profiles
fabric:profile-create --parents feature-camel MyProfile

# create broker
fabric:container-create-child --jvm-opts "$BROKER_01_JVM" --resolver localip --profile $BROKER_PROFILE_NAME root broker

# create worker
fabric:container-create-child --jvm-opts "$CONTAINER_01_JVM" --resolver localip root worker1
# assign profiles
fabric:container-add-profile worker1 MyProfile
 
Meaning that addressing a karaf script providing Maven coordinates worked well, and that now you can use shell:source, osgi:install or any other command you want that requires artifacts published on Nexus.

Conclusion

 

As mentioned multiple times, this is just a possible workflow and example of interaction between those platforms.
Your team may follow different procedures or using different instruments.
Maybe you are already implementing more advanced flows based on the new Fabric8 Maven Plugin.
In any case I invite everyone interested in the topic to post a comment or some link to different approach and help everyone sharing our experience.

52 comments:

  1. This is excellent Paolo. Looking forward to more :)

    ReplyDelete
  2. Thanks for the excellent post. I've learnt lots of things looking through your scripts. I especially like using shell:source with mvn co-ords, and I also didn't realise there were services we can wait on to ensure Fabric is ready to take commands. Good work!

    ReplyDelete
  3. Excellent Work @Paolo Antinori.
    Thanks! for sharing such a useful piece of information.
    Keep up the Good Work :))

    Integration Testing

    ReplyDelete
  4. Thanks Paolo,

    This is a great post, which provides a clear picture of different components being used and where they fit accordingly. Flow Diagram above makes it pretty easy to understand. Thanks again, keep sharing.

    ReplyDelete
  5. I simply couldn’t depart your site before suggesting that I really enjoyed the usual information an individual supply in your visitors? Is going to be again steadily to check out new posts.
    Office Interior Designers in Coimbatore
    Office Interior Designers in Bangalore
    Office Interior Designers in Hyderabad

    ReplyDelete
  6. Hello,
    The Article on Continuous Integration with JBoss Fuse, Jenkins and Nexus is informative. It gives detailed information about it.Thanks for Sharing the information on Software integration .For More information check the detail on Integration testing check, Software Testing Company

    ReplyDelete
  7. Hi,

    Well explained. Thanks for such a great post Paolo. For more information on testing service please go through the website. I found it intresting. Automation testing services

    ReplyDelete
  8. Hello,

    Indeed a great post. Thanks for sharing valuable information

    For more information on testing and test automation framework you can check it.

    ReplyDelete
  9. Good article, Jenkins is the modern DevOps tool for continuous integration solutions than with Nexus and JBoss Fuse. However Very neatly explained, thanks for this great article.

    Best Regards,
    DevOps Training in Hyderabad
    DevOps Online Training in Hyderabad
    DevOps Online Training
    DevOps Institutes in Hyderabad
    Learn DevOps Online

    ReplyDelete
  10. This comment has been removed by the author.

    ReplyDelete
  11. Paglaum ikaw adunay mas maayo ug mas maayo nga mga artikulo. Gusto ka nga usa ka nindot nga bag-ong adlaw

    bồn mát xa chân

    chậu ngâm chân giá rẻ

    máy ngâm chân giải độc

    bồn massage ngâm chân

    ReplyDelete
  12. Có lẽ cần phải trải qua tuổi thanh xuân( Phương pháp học toán tư duy ) mới có thể hiểu được tuổi xuân là khoảng thời gian ta( dạy trẻ học toán tư duy ) sống ích kỷ biết chừng nào. Có lúc nghĩ, sở dĩ tình yêu cần phải đi một vòng tròn lớn như vậy, phải trả một cái giá quá đắt như thế,( Học toán tư duy có tốt không ) là bởi vì nó đến không đúng thời điểm. Khi có được( Toán mầm non ) tình yêu, chúng ta thiếu đi trí tuệ. Đợi đến khi( Bé học đếm số ) có đủ trí tuệ, chúng ta đã không còn sức lực để yêu một tình yêu thuần khiết nữa.

    ReplyDelete
  13. A logo free of style patterns is a logo that won't need changing at regular intervals. logo design service

    ReplyDelete
  14. Si el agua cae al lago, desaparecerá( phụ kiện tủ bếp ). Pero si cae a la hoja de( phụ kiện tủ áo ) loto, brillará como una joya. Caer igual pero( thùng gạo thông minh ) estar con alguien es importante.

    ReplyDelete
  15. Vanskeligheter( van bi ) vil passere. På samme måte som( van điện từ ) regnet utenfor( van giảm áp ) vinduet, hvor nostalgisk( van xả khí ) er det som til slutt( van cửa ) vil fjerne( van công nghiệp ) himmelen.

    ReplyDelete
  16. Дээд чанар бол зүгээр л( đá ruby thiên nhiên ) санаатай биш юм. Энэ нь өндөр( đá ruby nam phi ) түвшний төвлөрөл, тусгай хүчин( Đá Sapphire ) чармайлт, ухаалаг ( đá sapphire hợp mệnh gì )чиг баримжаа, чадварлаг туршлага, ( đá ruby đỏ )саад тотгорыг даван туулах( lịch sử hình thành của đá emerald ) боломжийг хардаг.

    ReplyDelete
  17. This comment has been removed by the author.

    ReplyDelete
  18. Thank you for your attention to detail and great writing style. Your professionalism shows in your article. I like your interesting views and appreciate your unique ideas. This is quality.
    SAP training in Kolkata
    SAP training Kolkata
    Best SAP training in Kolkata
    SAP course in Kolkata

    ReplyDelete
  19. Nice Blog!! I like it and thanks for sharing that wonderful information.Thankyou.
    DevOps Training
    DevOps Online Training

    ReplyDelete
  20. Nice Blog !
    One such issue is QuickBooks Error 2107. Due to this error, you'll not be able to work on your software. Thus, to fix these issues, call us at 1-855-977-7463 and get the best ways to troubleshoot QuickBooks queries.

    ReplyDelete
  21. Nice & Informative Blog !
    In case you are searching for the best technical services for QuickBooks, call us at QuickBooks Error 15240 1-855-977-7463 and get impeccable technical services for QuickBooks. We make use of the best knowledge for solving your QuickBooks issues.

    ReplyDelete
  22. It is amazing and wonderful to visit your site. Thanks for sharing information; this is useful to us....
    graphic design training in Delhi

    FOR MORE INFO:

    ReplyDelete
  23. With today's modern society, the demanding needs of people are increasing. Not only beauty, eating and playing, but choosing a child's bedroom also requires a lot of factors. Because the bedroom is a place to rest, relax, study and sometimes also play a place for your baby. More: Phòng ngủ trẻ em, Giường tầng bé traiNội thất trẻ em

    ReplyDelete
  24. Hung was formerly an official distributor of industrial lubricants of Shell in the North. Currently, in addition to oil trading, we also trade in transportation and equipment trading. After nearly 12 years of establishment and development, Yen Hung is now a prestigious partner of nearly 10,000 large and small domestic and international factories. Main products:
    giá dầu truyền nhiệt
    dầu bánh răng
    dầu tuần hoàn
    dầu dẫn nhiệt
    dầu thủy lực shell
    mỡ bò bôi trơn chịu nhiệt

    ReplyDelete
  25. I am really thankful to the blog owner for helping us by giving valuable inputs.Good Post! Thank you so much for sharing this pretty post, it was so good to read and useful to improve my knowledge as updated one, keep blogging…Visit here for Software testing services and penetration testing services

    ReplyDelete
  26. RR Technosoft is the best Devops Training in Hyderabad and it provides Class room & Online Training by real time faculty with course material and Lab Facility.Devops training in hyderabad

    ReplyDelete
  27. this blog is very useful and relevan with article i've read, for more detail you can visit https://fh.unair.ac.id/kerja-sama-dengan-tiga-bumn-fh-unair-siap-bangun-gedung-baru-pringgodigdo/

    ReplyDelete