Today’s article is serious.

I feel the need to say this because, as you know, I don’t always write about serious stuff. But this article is serious because I’m going to summarize how we continuously improve the Amazing Nuxeo Content Services Platform [1].

And this entails explaining our Quality Assurance (QA) and Continuous Integration (CI) processes. Here at Nuxeo, this is serious business.

To help reinforce that this is serious business, Nuxeo made sure I had as little involvement as possible. The engineering team is in charge of this effort: Developers (back-end as front-end developers), QA team, DevOps, SysAdmin…

Basically everybody at Nuxeo except for me… Hence, this is why this process is super solid.

At Nuxeo, our open-kitchen philosophy extends well beyond our source code - we open source everything, including our internal processes.

On a daily basis, we provide full transparency about how we’re building and improving Nuxeo, this incredibly powerful and amazing product you’re already using, or will be using soon [2]. This QA/CI documentation explains all of it, and if you read it, you’ll notice that everything in there is state of the art. Using (in order of appearance) GitHub, Jenkins, Maven, multiple testing tools (unit tests, functional tests, performance tests) as listed and described here.

The Big Picture

These processes are summarized in this chart [3]:

Nuxeo QA CI

This schema illustrates my main point, the core principles behind the continuous build of the Nuxeo platform. There are certainly many more details and sub-processes involved depending on which aspect of our platform we’re working on.

For example, modifying an optional add-on (Nuxeo Vision, Digital Signature, etc.) doesn’t have the same impact as modifying one of the core plugins (such as the ones handling the backend database storage). Also, the process isn’t always as sequential as displayed in the schema. Code review, brainstorming, and a “re-adjustment” can happen during any stage in the process, for example.

The Explanation of the Big Picture

So, here are the primary QA and CI processes here at Nuxeo:

  1. A Nuxeo developer completes his or her work on a plugin (say Plugin A), adding features or fixing bugs in a dedicated branch (nobody is supposed to directly work in the master branch). The platform is built on a Component Model architecture, so everything is a plugin and there is no need to work on the whole source code: Platform developers work on one of the plugins at a time (Nuxeo WebUI, Nuxeo S3 Storage, Nuxeo LiveConnect, Bulk Document Importer, etc.).

  2. The developer tests the changes locally, and if they’re happy with them, they’ll push the change to GitHub. If they’re not happy with them, then it’s back to the drawing board…

  3. Once on GitHub, a hook allows our Jenkins server to identify that Plugin A has been modified. Jenkins then pulls the changes and…

  4. … Asks Apache Maven to build Plugin A in the developer’s branch. This means: build and test. And this is where there can be other details and sub-processes involved than what I outlined in the summary diagram above: Depending on the plugin, a trigger to build of all dependent modules may also be initiated, as well as functional or performance tests on a full assembly of the platform and the plugin.

  5. Let’s say the plugin was successfully built and tested on our Jenkins servers. “On our servers” is very (really, very very) useful to start avoiding the “But, it works on my computer” syndrome… The developer can now trigger a pull request for the changes to be merged into the master branch. At least two fellow developers will then review the code and validate the changes. Which means that conceivably steps 1-4 will be repeated until everyone agrees on the change.

  6. This is when the developer’s branch is merged into the master branch, and the second level of serious stuff starts.

  7. Maven compiles the plugin into the master, and tests it.

  8. If everything is ok, the plugin is then packaged…

  9. …and the whole set of validation procedures can then begin: Functional testing, performance testing, matrix testing… everything.

  10. If “everything” is ok, the plugin is then sent to the Nuxeo Marketplace, which means it’s available for our customers. Our customers can use their own Continuous Integration/Deployment tools to get the plugin automatically. This is very easy to automate (basically, just ./nuxeoctl mp-install the-plugin-artifact-id and voilà.)

Also, after step #9, our different test-by-humans servers are updated in a continuous deployment process. Using Ansible scripts, containers, and all of the latest and cool stuff (well, cool for those of us who are a bit geeky).

Also, you probably noticed Sonar mentioned in the schema. We do involve Sonar to help us check the quality and the coverage of the code. So, once the master branch is built, Sonar will receive and reconcile the changes on GitHub and also tell us more about it (about the coverage and the possible issues that the developers and their reviewers may have missed).

More on this Process

You know what? All this is automated!

Well. Of course, it’s automated… Who wants to hire people to push a “Download from GitHub” button, a “Compile” button, a “Tell Sonar” button or a “Run test” button?

In all fairness, not every step is 100% automated. For example, the very first step requires the brain of our brilliant developers to conceive the best code possible, and then write it.

Also, if something goes wrong, the appropriate people are notified. The developer is alerted (of course), but other developers and managers are as well (because even though we’re a super-cool company, we still have bosses, even in engineering), so the plugin can be fixed ASAP.

In order to execute these processes, we basically launch (automatically as well as on demand) Jenkins in Docker containers within our cloud infrastructure on AWS.

You can follow all of this on qa.nuxeo.com, our publicly available QA site (did I say we are super open?). As we continuously build the product, feel free to connect with us at any time: Monthly, weekly, daily, hourly… Whenever you want. At any time, you can see the What/Why/Who as it relates to our QA and CI processes. What has been built, why it was built (JIRA ticket), what was modified in the code, and who modified the code. And if the tests failed, you can blame the developer if you wish (we recommend not doing so unless you also send some coupons for free beer in a geeky bar.)

The Future

At the beginning of this post, I said more details and sub-processes are involved depending on the plugin that has been built. One of these processes is called Test & Push. Basically, it’s when a developer changes something that may impact the entire platform.

Say, for example, a developer is working on changing the version of the JSON parser library Nuxeo uses. This could have a significant impact everywhere in the platform, and thus, just testing one plugin is not enough. The developer must run a Test & Push test, which requires building the entire platform - and testing the entire platform. Even if it can be automatically triggered, we like to keep most of it manual, on demand (as of today) because building the whole platform and testing it takes time and requires a lot of computing resources.

Our team is always thinking about how we can adapt and improve our processes. The idea here is to be able to dispatch the tasks. Testing the entire Nuxeo Platform takes time because, by essence, the tests are run in a linear way (one after the other). But, the vast majority of the tests aren’t contingent on each other. For example, there is no need for nuxeo-platform-renditions to wait for nuxeo-platform-tag to complete its tests. So we want to leverage the Jenkins pipeline and Docker to split the monolithic Test & Push into many smaller tests and checks that run in parallel. This allows for more optimal utilization of our CI resources, while also providing us with faster test results.

Conclusion

Nuxeo takes its QA and CI processes very seriously.
(A short conclusion, isn’t it?)


[1] This ANCSP acronym is not validated by the Marketing team, I just made it up for this article.

[2] For example, maybe you’re gathering information about Nuxeo and found this article. I wrote “you will use Nuxeo” because, as the best Content Services Platform in the universe, why wouldn’t you use it?

[3] My first idea was this kind of zip-the-summary.

The original diagram was:

Nuxeo QA CI Joke

But when I presented this to Thierry Delprat (Nuxeo CTO), I could sense that he wanted to throw his chair at me, so I included additional details. CTOs…