IT Press Review – July 2015

open-container-project-big

Dear followers,
here is the 6th press review of Jamkey, which is mainly press review around Continuous Integration, development and DevOps tooling.

As you know, Continuous Integration is not only a way to build automatically, but also a path to development industrialisation.
That’s why you will find here news on tools (like JHipster), architecture (API design) but also methods and processes (like DevOps).

This press review focus on development tools, and cloud with the Spring Boot 1.3 release and powerful integration with devtools and Docker and CoreOS unite to start the Open Container Project and standardize runtime, image format.

Geneva SonarQube Conference will take place the 23 & 24 of Spetember.

Facebook Infer bring open source static analysis to mobile Objective-C and JS developments.

Interesting article on the « Release Manager » role, which is often the invisible hero of development projects.

Google is setting up a GitHub rival : Google Cloud Source Repositories
Still lot of news on Jenkins, Sonar, Nexus. And also articles around Micro-Services and Cloud.

 

I hope you will find here some interesting information on your current investigations. Most of them are in English, but some are in French.

Don’t hesitate to comment these informations if you think they could be useful for our current challenges.

Development tools

GitLab

Web

DevOps

Mobile

Agile

Sonar and Static Analysis

Jenkins

Nexus

Java

Cloud

Misc

IT Press Review – June 2015

WebComponentsPolymerDear followers,

This press review focus on Web development with the release of Google Polymer the WebComponent framework of Google.

Swift (Apple programming language) version 2 is released …. on Open Source !
Angular 1.4 is out, with lots of new features!
Still lot of news on Jenkins, Sonar, Nexus.
And also articles around Micro-Services and Continuous Delivery.

I hope you will find here some interesting information on your current investigations. Most of them are in English, but some are in French.
Don’t hesitate to comment these informations if you think they could be useful for our current challenges.

Web

DevOps

Mobile

Agile

Sonar

Jenkins

Nexus

Testing

Architecture and languages

IDE and Dev Tools

Cloud

IT Press Review – may 2015

Dear followers,
This press review focus on Devoxx France 2015 feedbacks. Devoxx France is a set of conferences for developers which took place in Paris during 3 days in April 2015.
You will find also a new Cloud part which focus on new Cloud tooling and companies strategies.
A new part also on Architecture and Design which provide articles on API, micro-services, Java containers, Open Sources…
There are a lot of new articles on Build tools, mainly on Gradle.
The Java section mention also the end of Java 7 support by Oracle and the announcement on Java 9.

I hope you will find here some interesting information on your current investigations. Most of them are in English, but some are in French.
Don’t hesitate to comment these informations if you think they could be useful for our current challenges.

Feedback from Devoxx France 2015

Architecture and Design

Web development

Build tools

Java

Cloud

DevOps

Sonar and continuous inspection

Build and Continuous Integration

Nexus

Test

Mobile development

Agile

SCM

Devoxx France 2015 return of experience

Devoxx-logo

Devoxx France is a conference for developers which took place in Paris during 3 days in April 2015.

As explained by the Devoxx team, the goal of Devoxx is :

To provide a technological conference of quality,
where the enthusiastics developers can come to learn,
work in network, to hack code, to make monitoring,
and to come to be inspired during 3 days.

Summary quoted from Oracle website:

In this 20th year of Java, the theme of Devoxx France 2015 was about innovations in technology over the next 20 years.
In his keynote, Java architect Brian Goetz described a development approach that keeps up with the technical needs of developers.
Java 9 will deliver a modular platform tuned to hardware requirements.
CEO Quentin Adam of Clever Cloud considers that engineering roles are entering an age of specialization.
Dan Allen, an open source advocate, author, and world renowned speaker, discussed the responsibility of developers to preserve digital information for future generations.
CCO Stephan Tual discussed Ehereum, an open source and decentralized platform allowing the creation of P2P businesses with new infrastructure, communication and privacy.

 

Inside Devoxx France 2015

What is Devoxx ?

Devoxx-logo-panel

Devoxx is one of the most important conference in IT development.
The Devoxx France is:

  • Created by members of Paris JUG
  • Comes from Belgium (Stephan Janssen)
  • has more than 2500 visitors
  • speaks about Java, Agility, Web, Cloud Computing, Mobile…



What for ?

Devoxx-pic1

IT front line conference:

  • See avant-garde conference (Web components…)
  • To know Java roadmaps (Java EE 8, Java 9…)
  • See Development accelerator tools in action (Java Hipster, JBossForge…)
  • Get industrialisation return of experience (Groovy release…)
  • Lot of information : help vision on IT plan
  • Opportunity to share point of views with IT actors in a simple way
  • To be part of IT community



Devoxx « Word Of Mouth »

Here are 8 words most used in Devoxx conferences (based on track titles):
devoxx-words

As we can see, « Java » is obviously in good position.
But Web and Mobile represent an important part of the conference.

Devoxx tracks

Devoxx is composed of 8 differents tracks of topics :
devoxx

Tracks summaries

Java, infinite possibilities

icon_javase

  • 42 conferences
  • Java still very active despite the explosion of Web Front technologies

Topics of the conferences:

  • Java today and the next 20 years
  • Java Stream (data management)
  • JBossForge (Java EE dev accelerator)
  • RxJava (asynchronous and event-based programs)
  • CDI (Context & Dependency Injection)
  • Hystrix (Netflix – Latency and Fault Tolerance)
  • Micro Services (API and architecture)
  • Spring and Spring Boot
  • Hibernate

Architecture, Performance and Security

icon_architecture

  • 28 conferences
  • API in the middle of questions
  • Oriented to Microservices and Cloud

Topics of the conferences:

  • API
  • RxJava (asynchronous Java)
  • Akka persistence (toolkit for concurrent, distributed)
  • Circuit Breaker Pattern
  • AsciiDoc
  • Performance, Benchmarks and Optimisation
  • Lock free applications
  • WebCrypto API

Cloud & DevOps

icon_cloud

  • 30 conferences
  • Cloud and Docker in the middle

Topics of the conferences:

  • Big Data and Data Science
  • Release process (Groovy experience)
  • Cloud deployment tooling: Docker, Mesos, Marathon
  • Cloud application monitoring and development
  • Linux packaging for Docker/LXC orientation
  • Scaling with Kubernetes
  • Lot of new Cloud Solutions: Google Cloud, Amazon AWS Elastic Beanstalk, IBM Bluemix, Microsoft Azure, Cloud Foundry (OSS), RedHat OpenShift, Heroku, Clever Cloud…

Alternate Languages

icon_alternative

  • 16 conferences
  • Javascript full stack

Topics of the conferences:

  • JavaScript
    • Spread more and more
    • Full stack capacity
  • TypeScript : JavaScript of AngularJS v2
  • Ceylon : runs Java and JavaScript virtual machine
  • Scala : functional programming
  • Scripting combination : JavaScript, CoffeeScript, ClojureScript, GopherJs
  • Go : Google system language

Agility, Methodology & Tests

icon_startup

  • 33 conferences
  • New management
  • Build automation level up

Topics of the conferences:

  • BDD worst practices
  • Code review strategy
  • Geek management
  • Technical debt
  • Testing:
    • Unit testing concurrent code
    • WireMock
    • Build automation : Gradle
  • Ultimate Bug Fighting
  • Software craftsmanship

Big Data & Analytics

icon_mobile

  • 27 conferences
  • Data Science is one the big IT subject

Topics of the conferences:

  • Spark is replacing Hadoop/MapReduce
    • Machine Learning
    • SparkSQL
  • Streaming : Kafka
  • New databases: Cassandra, Datomic, InfluxDB
  • Data mapping : Elastic search
  • Log management : LogStash

Web, Mobile & UX

icon_web

  • 33 conferences
  • Lot of development tools
  • AngularJS in the middle of JS

Topics of the conferences:

  • JavaScript :
    • AngularJS: debugging, performance
    • Browserify: modularity
  • Vert.x : polyglot event-driven application framework
  • Jhipster : Spring Boot + AngularJS generator
  • Ionic Framework : powerful hybrid mobile
  • Web components : the future of Web
    • Polymer
    • Material Design

Future <Devoxx>

icon_future

  • 17 conferences
  • Connected objects : new fields of development

Topics of the conferences:

  • Cognitive application (IBM Watson)
  • Paris JUG and Duchess talks
  • The future of reading and writing
    • Robotics
    • Connected objects
    • Home automation
    • Aldebaran robots
    • Autonomous Drones
    • Robots could learn
  • IOT : Internet of things

What is coming next ?

DevOps in every conferences

Devoxx-pic2

  • DevOps very discussed in every conferences
  • Use case : Groovy release mechanism (use Bintray to deliver)
  • DevOps oriented to Continuous Delivery (Jenkins Delivery Pipeline, Tests Arquillian, Logstash)
  • Ops behaviour metrics : Logstash, Statsd and Graphite
  • Distributed systems tooling:
    • Docker
    • Apache Mesos
    • Marathon
  • Even for development environment : Docker Compose
  • Use native package to be ready to containers


Drove by the Cloud

Devoxx-pic3

  • In most of the presentations, Application Servers were replaced by Cloud solutions and deployment managed by Docker
  • Lot of new Cloud Solutions: Google Cloud, Amazon AWS Elastic Beanstalk, IBM Bluemix, Microsoft Azure, Cloud Foundry (OSS), RedHat OpenShift, Heroku, Clever Cloud…
  • Major Cloud companies push their tools to Open Source (NetFlix, Google)
  • IT architecture orientation : end of monolithic architecture, Microservices, asynchrone
  • Technologies: Docker, Mesos, RxJava, Spring Boot/Cloud, WildFly Swarm…
  • Manage cluster of Linux containers : Kubernetes
  • Adapted to multi technologies fronts : Mobile, Web…
  • Even Cloud IDE: Codenvy Eclipse Che



Development accelerators

Devoxx-pic4

  • Powerful stacks with simpler developments: Spring Boot, Fluent HTTP, Java EE CDI
  • Generator tools to shape and accelerate development: Jhipster, JBossForge
  • Spring Boot shape your developments
    • Simplification of Spring configuration
    • Possibility to create your own starters
  • Documentation industrialisation: AsciiDoctor
  • Environment setup : Docker (and Docker Compose), Mesos, Marathon

Web technologies spreading

  • Javascript and Web technologies spreading more and more
  • Google Stack prominent
    • Angular JS (framework)
    • Material Design (visual rules)
    • Polymer (web components)
    • Cloud endpoints (backend)
  • Ionic Framework for Mobile Hybrid development
  • Web components define future of web developments
  • Node.js : base platform for a lot of tools (Ionic, Jhipster…)
  • JS is still very dynamic new ecosystem : React.js (Facebook UI)
  • New norms : HTTP2 (performance enhancement)

Devoxx-pic5

Train of thoughts

  • Agility but with taking a step back
    • BDD difficulties due to lack of impact mapping
    • Cross design workshop (share the design of the application)
  • Responsibility and free-will of the developer
    • Private life
    • Security, back-door, technical fail
  • New way to IT management
    • Quash received wisdom (step back on general ideas)
    • Manage geeks by continuous improvement sharing
  • Internet of things
    • Connected objects are projects of today
    • More and more IT projects on Robotic

Sum up and conclusion

And Continuous Integration in all this ?

Devoxx-pic6

  • Inspiration to define better development use cases
  • Get fresh on best practices on tests, build, code analysis
  • Confirm current choices according to the development community
  • Understand the trends which lead to technology shifting
  • See other release management use case :
  • Groovy Release management
  • Tools examples:
    • Jenkins : Mesos plugin, Docker plugin
    • Environment setup: Docker Compose
    • Continuous Delivery : Jenkins delivery pipeline



Conclusion

Devoxx-pic7

Devoxx is a motivating experience, specially when you are asking yourself question on the IT vision to adopt.

  • New orientations are spreading:
    • Cloud oriented development is clearly a global shared vision
    • Microservices architecture will be associated to Cloud orientation
    • Fast development (Spring Boot, Jhipster, Javascript full stack) is mandatory to time to market innovations
  • New IT point of view
    • DevOps idea is spread on all IT subjects
  • New inspiration
    • Philosophy : as the quote said « with great power comes great responsabilities »
    • Robotic : is becoming a real field for IT projects
    • Learning : how to learn and how « to learn to learn » is becoming a big topic
    • Responsibility and security :
    • New way to manage projects : adopt management transparency, step back on received wisdom, agility and impact mapping is not a good choice = it is becoming mandatory



Links

External links:

 

 

IT Press Review – april 2015

Dear followers,

This press review focus on Development Accelerators (like JHipster or Ionic Framework) and AngularJS 2 (and the big choice of Google to use Microsoft TypeScript!).

You will find also the 2015 survey of StackOverflow on developer profiles and technologies.
Some contents refer to Devoxx France 2015 which took place last week (8-9-10 april) ! A special page will be published on that very soon!

I hope you will find here some interesting information on your current investigations. Most of them are in English, but some are in French.

Development accelerators

Build tools

Java

DevOps

Sonar and continuous inspection

Jenkins and Continuous Integration

Nexus

Test

Web development

Mobile development

Agile

IT Press Review – march 2015

Hi,

you will find here my press review around Continuous Integration and IT development solutions.
As you know, Continuous Integration is not only a way to build automatically, but also a path to development industrialisation.
That’s why you will find here news on tools (like Gradle or Maven), architecture (like Spring) but also methods and processes (like DevOps).

I hope you will find here some interesting information on your current investigations. Most of them are in English, but some are in French.

 

Java development tools

DevOps

Sonar and continuous inspection

Jenkins

Nexus

Gradle

Test

Web development

Mobile

Agile

Security

Jenkins memory scaling

Jenkins is a Java web application which needs to be tuned to avoid Out Of Memory Errors when dealing with large volume of jobs.
The tuning of Jenkins is something sensitive. But at the same time the core engine do simple things : Jenkins executes jobs which execute external processes and JVM (Maven, Gradle…).
Thus the role of Jenkins is based on the configuration of the jobs, their schedule and the management of the logs.

There are lot of examples of build environments with hundreds of slave of the hundreds of transform-scale-iconjobs without particular problems of memory, storage.
Initially it would be interesting to monitor Jenkins and its JVM to determine the reason of the potential problems.
The problem could be a insufficient memory configuration of the master Jenkins. Or some plugins memory consuming or memory leak.

 

Recommendations

In term of recommendations for a large build system :

  • To check the plugins usage in Jenkins (are they all used? Are they parameterized in adequate manner ?)
  • To have simple jobs (complexity remains in the build manager (Maven or other) not the continuous integration): main configs are SCM / 1 step of Build / some post-actions
  • To limit the history of the builds in the job (« discard old build »)
  • To use Jenkins UI plugins in order to filter the view of the jobs to limit their access
  • To introduce a master-slave architecture (what you did) (use of labels to distribute build in a transparent way)
  • To limit the number of executors by slave
  • To have potentially several masters

Be careful for the master-slave mode to take a more recent version of Jenkins which allows to optimize the memory consumption.

Before changing memory configuration, first monitor it

Resources consumption investigation is necessary in order to investigate and tune the JVM of Jenkins.

Monitoring by plugins

The Jenkins Monitoring plugin is interesting for that. It will give you Charts of memory, cpu, system load average, http response times…

Get live metrics

In the same way VisualVM connected on the JVM of Jenkins (see example http://java.dzone.com/articles/visualvm-monitoring-remote-jvm and http://docs.oracle.com/javase/1.5.0/docs/guide/management/agent.html).

Jenkins or plugins bugs

The problem could be a insufficient memory configuration of the master Jenkins. Or some plugins memory consuming or memory leak.
Check the updates of the plugins for that.
It can also be a Jenkins version which did not optimize enough the memory usage with a large number of jobs.

Performance tips

Build history

Keep only a subset of build records around

You can set Jenkins up to clean house by checking Discard Old Builds in the Configure menu and filling in one or both of
this option’s two text boxes. You can choose to throw the old records away if they are more than 30 days old, or you can
choose to keep only the last X records and discard everything else.

The whole point of controlling your build records is to avoid unbounded consumption. Sometimes you want to keep some
records for a while so people can look at problems when there are failures… so don’t make the number too low. You really
just need to have a fixed cap; 50 or 100 is a good number. Note that you can configure build record housekeeping on a
per-project basis.

Another way to keep specific records is to use the fingerprinting feature, which allows you to create an association be-tween jobs. As you enable fingerprinting, you can enable Jenkins to keep a build log of dependencies – this captures all
the upstream build log dependencies.

Number of slaves

Each slave maintains a connection to a master, and to service these connections a master launchs several threads, and each thread is costly (2MB or so memory just for a call stack.)

Exactly how many threads each slave incurs on a master depends on the mode of connection. Most connection mechanisms require two threads per slave (SSH slaves and Java Web Start slaves), but if a JWS slave falls back to the communication over HTTP, it’ll incur three. See « Obtaining a thread dump » for how to count the number of threads in your environment.

See Set up Master-Slave for more information.

Degree of concurrent HTTP accesses to the master

If you expect a lot of users accessing Jenkins user interfaces, this adds additional CPU overheads to a master. The memory impact on additional concurrent users should be negligible.

When you expect a large number of concurrent users, watch out for the upper bound of the maximum number of HTTP request handling threads. Contrary to the intuition, you do not want to give too big a value, which tends to make contended locks even more contended. You want to keep this value relatively small, and let incoming requests wait in the queue as opposed to try to serve them all in parallel.

Number of jobs

Impact of a larger number of jobs can be seen in several places. One is the start-up time of Jenkins, especially noticable if you have $JENKINS_HOME in a disk that has a large seek time (spinning HDDs, as opposed to SSDs.) Another is the UI organization. You need to utilize views, folders, and other means to organize jobs, and avoid using the « All » view that’d render everything.

Number of builds

Starting around 1.450 to 1.500 Jenkins started lazy-loading builds and discard them from memory if unneeded, but various plugins are still being updated to take advantages of this. Therefore, you’ll want to cap the number of builds each job occupies, by using the « discard old build » feature in Jenkins.

To prevent Jenkins from discarding build records only to load them back from the disk again, please give ample heap to Jenkins JVM.

Set up Master-Slave

Jenkins masters expectations:

  • Keep webui responsive
  • Running/Managing jobs/slaves take resources
  • Throw a beefy VM at it (IO less of an issue)
  • Don’t run jobs on it
  • Manage/Backup configuration programmaticaly (puppet)

Jenkins slaves expectations:

  • Should be quick to setup and throw away
  • Plan for high IO usage! (not single hypervisor)
  • Monitor Cpu and Disk
  • Segregate heavy jobs

Set up distributed Builds

You will eventually outgrow the ability to run builds on just one machine; as well, single systems do not take advantage
of the full power of Jenkins.

Dealing with the amount of compression load is not the only reason to use distributed builds. You also need isolation between builds. For example, when your tests depend on local resources like a local database or particular TCP/IP port, you can’t use the same machine to run tests
that access those resources. And while you can always work around these problems by tweaking your build script and tests
and such, it’s much easier if you have boxes that provide natural isolation.

Another driving factor for distributed builds is that if you’re testing against multiple platforms, you often want to have
more diversity in the environment. This situation inherently calls for multiple systems.

Set up scalable slaves with labels

Managing a diverse set of platforms and machines easily, and making machines interchangeable.

A very useful but under-used Jenkins feature is labels. Labels are simply tags you can assign to nodes to describe their capacities. Some typical useful labels include:

  • Operating system
  • 32 vs. 64-bit
  • Additional infrastructure that exists only on certain machines (for example, WebSphere)
  • Machine’s geographical location

Assign labels on the build machines themselves; then on the job side, specify that the job needs to run in a certain place based on label criteria. Instead of tying jobs to individual build machines, labels give Jenkins flexibility to choose where to run the builds, which results in better resource utilization and promoting manageability.

Using labels is so easy that even the Marketing team figured it out:

  1. Select a Slave machine and choose Configure
  2. Specify a label in the Labels field.
  3. Create a new job, name it and fill out any other necessary parameters, and click OK.
  4. Click the checkbox, Restrict where this project can be run (for example amd64, linux and Montreuil).
  5. Fill in the Label Expression that matches the label on your Slave machine (or any label).
  6. Click Save and then run the build. It will only run on machines whose labels match the job confi guration.

One final reason to use labels: if a machine goes down, Jenkins has the flexibility to shift the load to another machine with
a compatible label, which gives you time to diagnose and fi x the problem. This way you can increase the level of service
of your cluster to your users without service disruption.

Be careful of slave thread consumption

A connected JNLP slave used to occupy one thread on the master, but now it occupies none.
Combined with the earlier change that eliminated threads from idle executors, now you can connect thousands of slaves.
Be careful to use the correct Jenkins version.

See http://jenkins-ci.org/content/more-scalable-slaves

External links

See also

About

Jamkey est la société de Sébastien Morenosebastien-moreno

https://about.me/sebastien.moreno

https://www.linkedin.com/in/sebastienmoreno

http://www.viadeo.com/fr/profile/moreno.sebastien

https://fr.pinterest.com/sbastienmoreno/

 

Passionné par les nouvelles technologies, mon expérience est composée de nombreuses missions de conseil en tant que Développeur Java, Architecte d’Intégration et Consultant DevOps au sein de grands comptes comme Thales, BNP-Paribas, IBM, Alcatel, Orange, Gefco, Les Echos, la Caisse des Dépôts.

J’ai accompagné durant ces dernières années de nombreux clients pour les aider à définir et implémenter leur stratégie de développement, d’intégration et de livraison.

Grâce à ma formation en informatique, à mon expérience des nouvelles technologies et à mon implication dans le mouvement DevOps et les méthodes Agiles, je tire une excellente compréhension des enjeux que les entreprises rencontrent et je sais recommander les solutions qui permettent à mes clients d’atteindre leurs objectifs.

Mes responsabilités couvrent : l’expertise en intégration continue et DevOps (build, tests, packaging, déploiement, continuous delivery…), l’architecture technique, le développement, l’audit technique, le support, la mise en place et l’animation de formations, d’ateliers et de conférences.

Techniquement, ces années d’expérience m’ont fourni une très bonne connaissance du développement Java, des technologies Mobile (iOS et Android) et Front (JavaScript), des outils de build (Maven, Gradle…), et de l’implémentation des méthodes Agiles (Jira – Kanban).

J’ai eu la chance de travailler sur de nombreuses phases : avant-vente Analyse Développement Intégration Tests Livraison Mise en production exploitation… me permettant de construire une solide expertise DevOps.

J’ai pu développer une grande autonomie grâce à mon expérience dans le conseil mais aussi en devenant freelance depuis 2006. Ma volonté d’entreprendre m’a conduit à créer Jamkey mon entreprise de conseil.

 

La société JAMKEY a pour but de proposer des prestations de conseil en informatique et en système d’information.

Les services proposés par la société JAMKEY se situent sur 3 axes :

–             L’expertise et le conseil en architecture et en intégration logicielle

–             La conception et le développement logiciel

–             La formation en nouvelle technologie