We have a test to check our code behaviour if a file we need to backup is readable or not. It was roughly like the following:

And weirdly enough, this test was failing. By that I mean, there was no failure for the code above…​ ^[1]

We were running those tests on a shiny new infrastructure, and using wrong Docker images using root user as the default. For instance, if you use a base image like openjdk or basically any base image, you will hit this issue.

The thing is, when you are root, a bunch of things is not true anymore…​ For instance, permissions…​

If you don’t read Java, here’s a shell port of the Java code above:

But then replace the cat above by sudo cat:

I for one was slightly surprised root does not honor permissions at all. Had I been given a quiz about this, I would probably have thought that being root would still prevent you from reading it (but being root would allow you to call again chmod at will to set what you need), but that’s how it is.

I think we can say it is a very common and accepted practice that running a server using the root user is a bad idea. It is the same thing in Docker, for many reasons, and hopefully the examples given above will confirm it. At least it was a lesson for me, and I’ll be very cautious about it from now on.

So, if you care about your tests, and their ability to monitor and reveal issues and regressions, do NOT run your CI with the root user.

It is out of scope here as it has nothing specific to our current use case. Just create an instance. I am using the Microsoft Windows Server 2016 Base - ami-45e3ec52 for that article.

Your instance is now running.

This should quite slowly finally show the following screen:

Hope this helps!

If you run a production server, and you don’t have GC logs enabled, DO IT NOW. Really, it could save your ass.

It is unnoticeable in terms of performance impact, and is invaluable if you need to diagnose some performance issue. It was a bit of a mess with HotSpot many years ago, since log rotation was not available. But now it’s been added for quite a long time, there is no excuse anymore.

Then great, read below, this is the point of this article.

The Garbage Collector is that thing that is going to go after you, and try to find unreferenced objects to clean up your work.

That cleanup can be done using a lot of different strategies. It’s even pluggable in the JVM. So many are in use in the field. They are called Serial, Parallel, Concurrent-Mark-Sweep aka CMS, Garbage First aka G1, and so on.

Explaining each one of those strategies is way out the scope of that article. There is already a bunch of articles about them out there, and each one can easily require a whole article.

Still, knowing what the GC has been doing on a particular setup is essential for issue analysis and decision making.

The GC logs are non-standard outputs, depending on each strategy implementation, telling for example what was the previous available memory, what was freed, how much, and so on.

Now that you know you must always enable GC logging, the obvious way to do it is to use the options to output those logs into files. The historical way to do that was simply to use -Xloggc:gc.log. Fortunately, for a few years now, since:

There is now a standard way to do it with HotSpot ^[1].

To sum up: don’t do it. Unless you perfectly know what you do.

I have been very lucky to follow the famous "Java Performance Tuning" training with Kirk some years ago. He explained that nowadays the JVM has really become an impressive piece of engineering. Especially, it now contains many auto-adaptive algorithms. Fixing values like a pool size, the initial memory ^[3] and so on is likely to do more harm than good.

Unless it comes from actual testing, that is. Like GC logs analysis.

As always, you have to know what you’re doing and not blindly apply recipes, surprising isn’t it? :-)

I have also always liked to be part of opensource communities. Sharing knowledge and helping others, learning a lot in the process. Participating in things like Hibernate documentation translation, Hibernate forum, Apache Maven mailing lists, MOJO @ CodeHaus (now MojoHaus) Continuum, Archiva, M2E, Sonar[Qube], Hudson, and…​ Jenkins.

After switching from Apache Continuum, I started using Jenkins around 2009 when it was named a bit differently. My first message on the users mailing list seems to date back to early 2010.

My first code contribution to Jenkins seems to date back to 2011, where I had to git bisect a weird issue we were probably only very few to have, as we were using IBM AIX at that time.

In 2012, I published my first public plugin with Michaël.

Later, I took over the maintenance of some others, like the Radiator View Plugin, Parameterized Scheduler Plugin and most importantly the Chuck Norris Plugin.

About 2 years ago, I started participating in the Jenkins Governance Meetings on IRC.

All those years, I have been spending a fair amount of time on that project close to my heart. So, having the opportunity to work full-time in "a professional opensource company" on all-things-Jenkins is obviously something strongly motivating to me.

I am thrilled to say I will be joining CloudBees next August.

I am very proud to soon start working with so many great individuals, bringing my small contribution to smoothifying the software delivery process in our industry.

Follow me on @bmathus to get more updates if you are interested :-).

Management is too often badly considered. In my opinion, it’s because we actually rarely encounter good managers. You know, useful, inclusive, psychologist, able to catalyze work, and never get in the way, you have known plenty of them right?

Being a good manager is hard.

Like being a good developer.

It takes passion, time, experience, reading, and so many other things. And by trying to chase too many rabbits, you’re taking the risk to never even catch one.

I’ve been willing to write that down for a while. It started as a guts feeling, then as time went by I think I found more arguments to articulate my thoughts. It’s still ongoing, but I felt it is now somewhat clear enough to propose it here and possibly get some feedback.

I am talking here about human managers of many teams. I mean the type of position where your role is (should be?) to provide your team members with context so that they know where the company is going (i.e. alignment!).

Even if that may apply to other areas, what I have in mind is IT. The domain I work in and hence know the best. I also have in mind a modern/Agile organization where the goal is more to make the company succeed than to respect any form of historical hierarchical establishment.

Roughly, in my mind, that kind of manager can exist for at least 20 people. Below that number is heading towards micro-management.

Being and staying a good one is going to take time.

That time being a finite resource, you cannot be good and up-to-date in all areas.

I wanted to use Overlay storage driver for our Docker hosts (see why devicemapper should be avoided in my opinion. ).

The issue is: Overlayfs support has been merged in the Linux Kernel in 3.18 and Centos 7 is currently using a 3.10 one .^[1].

Hence, since this is not for a customer-facing production machine, I decided to upgrade the kernel.

The simplest way to do it is to use ELRepo and install the package called 'kernel-ml' (like in kernel mainline).

TL;DR:

If you use a remote VM like me, you may not have access to the grub UI when the machine reboots. And the thing is: Centos will by default use the previous kernel.

So, by default again, if you reboot without modifying it, you will stay on the same old kernel.

If you want your machine to use the newly installed one: execute the following command (it will select the first available kernel, which is the newly installed on centos by default)

To show the list of available kernels, you can verify that the new one is indeed the first:

Hope this helps!

As much as I love Docker, some may find a pity that the default behaviour is NOT to be used in production (on a non-ubuntu host)!

Indeed, by default, here’s what Docker will choose as a storage driver:

BUT, the thing is: though AUFS is apparently great for Docker (was used by DotCloud for their PaaS, before it became public), it’s not in the standard kernel. And is unlikely to be in the future.

For this reason, distributions like RedHat (which is upstream-first) chose to support devicemapper instead, in the so-called thin provisioning mode, aka thinp.

But by default, if AUFS is not found in the current kernel, Docker will fallback to the ubiquitous devicemapper driver. And for the default again, it will create loopback files. This is great for newcomers bootstrapping, but horrible from a least surprise principle perspective: since this mode MUST NOT be used in production.

Short answer: no.

Longer one: many Docker knowledgeable people have publicly stated that you should prefer other ones. Many have even recommended that you default to using Overlay ^[1]:

Even Daniel J Walsh aka rhdan, working for RedHat has stated ^[2]:

Or:

I can’t help thinking that this sentence may tell us more about the subject than his author was thinking. Complexity often being the reason software can not survive years? Might be.

I’ve started converting the Docker hosts I work with from devicemapper to overlay. My first impressions are good and everything seems to be working as expected ^[3].

From all I’ve read, my current wager is that overlay will soon become the default driver. It has the pros of devmapper (no need to create a FS for one) apparently without much of its cons.

Only some specific use cases will still make people choose other drivers like btrfs or zfs. But as these require to create and size real FS to be used, they are unlikely to be used as widely.

Though I retrospectively think I should maybe have pushed sometimes some more evolved/involved solutions, I’m not actually talking about a technical failure.

No, I’m talking about human/social one.

To simplify a bit, the management decided to reorganize the development with dev teams on one side, and a separate architecture team.

Because I had earned technical respect from (at least some of) my coworkers, it went not so badly initially. Some teams were asking for reviews, or even for solutions for issues/requirements they had.

But for some people, developers and maybe even more managers, we were intruders. Not welcome.

Mainly, I think I stayed in my office way too much, and took that position for granted. Kind of the Ivory Tower issue (well, without the tone I hope. I’ve tried hard to not be condescending especially because of how much I despised self-said Architects who didn’t code).

I thought the requests were going to flow naturally. How wrong, and dumb, I was.

Don’t get me wrong. I was not hiding and playing video-games on my computer :-). I was actually working for some teams. But even those teams eventually didn’t even ask us for help, and worked us around.

I should have hanged out more with the teams (which is somehow ironic I didn’t when you know me). Go see them, ask them if they were needing help. Get involved with them. Simply be more empathetic. Let them know what we did, why, for whom, constantly. Make that publicly available.

I should also have refused to work on some subjects supposed to be useful in 1 or 2 years, without any actual need. How many hours I lost on useless PoCs, studies, that will never get used.

That made me realize something. Something that may be obvious to more experienced people: the fact that the current management structure, the current organization will NOT stay as-is forever. And that you should always strive to break barriers, reach out the people who do the actual work and help them, work with them.

This way, people will know you’re basically useful wherever you are, and whatever position you hold. And that might also transitively prove your team is useful.

If you don’t, then you’re dead. At the next shakeup, you’ll be wiped out. And you will have deserved it.