Configure Hudson CI Server to automatically disable itself when main SVN repositories become unreachable

Note: I wrote this post some months ago, and just made it public since the problem making it impossible to use was fixed some weeks ago. In the meantime, you should also be aware that Hudson has recently been renamed to Jenkins, and its new house is now http://jenkins-ci.org/

Sometimes, we encounter erratic issues accessing our subversion repositories. Even apart from the server upgrade information that just dont reach the interested people, but only managers who didn’t forward (since there’re obviously not the ones that use the dev server…), we also have random problems like everyone.

The problem

When SVN becomes unreachable, every one starts receiving mails about it from Hudson… For example, last week-end I received 6000+ emails about that. So, I wrote this small script to update all our jobs to not run during both the night and the week-end.

But sure, this won’t solve everything. For example, if the server goes down during a working-day, and you’re not in front of your computer for some reason. When coming back to your box, you might discover the big amount of mails from Hudson, or even from the devs if you’re in charge of operating the CI server.

So I’ve been looking for a way to just automatically disable Hudson builds when a problem is detected.

The solution

For some days now, I’ve been playing with the Hudson script console since I discovered how greatly powerful it can be.

My starting point was the hudson command used to prepare a shutdown. How to do it through the groovy console? I gave it here in one tweet: hudson.model.Hudson.instance.doQuietDown(). Once I found this, I just had to find a way to interact with the SVN inside the groovy/hudson console system and build around it a small groovy script.

After some struggle about how to programmatically use SVNKit (Subversion pure Java API), and then how to use an anonymous class with Groovy, I was done.

Here’s the resulting script:

import hudson.model.*
import org.tmatesoft.svn.core.*
import org.tmatesoft.svn.core.wc.*

String[] repoToCheck = ['svn://svn/scle', 'svn://svn:3691/pgih']

class MyHandler implements ISVNDirEntryHandler
{
  def void handleDirEntry(SVNDirEntry dirEntry)
  {
    // nothing
  } 
}

org.tmatesoft.svn.core.internal.io.svn.SVNRepositoryFactoryImpl.setup();
Map<String, Throwable> problematicRepos = new LinkedHashMap<String, Throwable>();
for(String repo:repoToCheck)
{
  SVNURL url = SVNURL.parseURIDecoded(repo);

  SVNClientManager clientManager = SVNClientManager.newInstance();

  SVNLogClient c = clientManager.getLogClient();
  try
  {
    // Special groovy anonymous class construct
    def handler = new MyHandler()
    c.doList(url, SVNRevision.UNDEFINED, SVNRevision.HEAD, false, false, handler);
  }
  catch (Exception e)
  {
    problematicRepos.put(repo, e);
  }
}

if(!problematicRepos.isEmpty())
{
  for(Map.Entry<String, Throwable> entry:problematicRepos.entrySet())
  {
    println("Problem accessing \""+entry.getKey()+"\"");
    String s = entry.getValue();
    println(s)
  }
  println("Disabling hudson build")
  hudson.model.Hudson.instance.doQuietDown()
  return 1
}
else
{
  println("No problems with repos");
}

How to install and configure it

Install the Groovy Plugin for Hudson. This way, you’ll be able to add job directly written in Groovy. Then create a job that will run every minute! (”* * * * *”) and put the script above inside an “Execute system Groovy script”.

Then, configure the notification you like. It’s probably a good idea to target admin email when this jobs fails. That’s what I did.

Important note: there used to be a difference of behaviour with classloading between “groovy script console” and “groovy system script” in a job. This made the script above unable to work. The good news if that it was fixed with Hudson 1.352 and HUDSON-6068. So the bad news is that you can’t use this technique if you’re using an older version (time to upgrade? ;-)).

Possible improvements

Sure the script isn’t perfect, here’s a few thought of what’s currently missing:

Hope this helps!

comments powered by Disqus