Our development team is hard at work on nuxeo.io. The content management system architecture is getting clearer as we try many different technologies revolving around Docker. One of the Linux distributions that stands out to host our containers is CoreOS. As we’re serious about this, we started working on CoreOS monitoring.
CoreOS is Linux for Massive Server Deployments.
This is the Linux distribution we use to host our docker containers for Nuxeo.io. The goal of CoreOS is to offer a minimal, HA Linux distribution with Docker. It’s also set up for clustering by default. That’s a perfect fit when building a PaaS. And something mandatory when you’re building a PAAS the right way is monitoring.
CoreOS Monitoring in Theory
We use different things to take care of our CoreOS monitoring. If you want a deeper overview of our monitoring stack you should read the post Mathieu wrote about it. It’s based on Diamond and Graphite for metrics collection, and it’s based on LogStash, ElasticSearch and Kibana for logs management.
I’ve been looking for various ways of doing CoreOS monitoring as it does not come with any collector as far as I can see. You have to know that CoreOS has no package manager whatsoever. If you want to install new things on CoreOS you have to rebuild them using their SDK. So it’s not easy to install Python based Diamond. But, it’s made to run Docker containers, so the approach I used was to run Diamond in a container on CoreOS. For this to work I needed to make sure Diamond had access to the /proc partition as this is where it collects most of its metrics.
Accessing the host filesystem from a container is easy thanks to the volume option. To access /proc from my container, I can run it like this:
sudo docker run -t -i -v /proc:/var/host_proc:ro ubuntu bash
Here _-v /proc:/var/host_proc:ro_ corresponds to [host_filesystem]:[container_filesystem]:read-only. This is the Volume option that gives me access to my host’s /proc.
The next step is to tell the different Diamond collectors that they should look for metrics in _/host_proc_ instead of /proc. Unfortunately, most of these collectors have the path hard-coded, so for the moment I forked it and hard-coded _/host_proc_ (yes I know, but I was eager to test it). Now that I know it works I will try to parameterize this and send a pull request.
In the meantime the source code is on GitHub. You’ll find different docker images for Nuxeo, Diamond and Graphite. If you want to test it, I suggest you do it with CoreOS. You can have it running in no time thanks to Vagrant. Just checkout https://github.com/coreos/coreos-vagrant and follow the Readme instructions.
Once you have a CoreOS session open, you can check out the monitoring images and start building them:
cd nuxeobase docker build -t nuxeo/nuxeobase . cd ../nuxeo docker build -t nuxeo/nuxeo . cd ../graphite docker build -t nuxeo/graphite . cd ../diamondBuild docker build -t nuxeo/diamond .
Then you can start your containers.
Start the graphite container:
docker run -h="graphiteServer" -p 8080:8080 -p 2030:2030 -p 2040:2040 -P -d nuxeo/graphite
Start the nuxeo container:
docker run -h="nuxeoServer" -p 80:80 -d -name nuxeoServer nuxeo/nuxeo
Start the diamond container:
docker run -h="diamondCollector" -d -v /proc:/host_proc:ro -link nuxeoServer:nuxeo -name collector nuxeo/diamond
Now with this particular setup you should have your graphite instance available on port 8080 of your host and Nuxeo available on port 80. If you take a look at your graphite instance, you should see Metrics getting stored in Graphite. That’s exactly what I was looking for.
Next step for me will be to use logStash to forward logs to ElasticSearch and browse them through Kibana.