First, obtain Buildbot on both master and slave.

master$ easy_install buildbot
slave$ easy_install buildbot-slave

Now, create the master.

master$ buildbot create-master master_dir

This will make a directory called master_dir and fill it with all sorts of goodies. In order to be operational, the master_dir requires a master.cfg file to be present within it. Luckily, Buildbot supplies a master.cfg.sample which can easily be tailored to fit our needs.

master$ cp master.cfg.sample master.cfg
master$ $EDITOR master.cfg

Now let's do the aforementioned tailoring. I'll step through configuration piece-by-piece.

c = BuildmasterConfig = {}

All that is required of each master.cfg is that it builds a dictionary called BuildmasterConfig (which we'll call c for brevity), which contains various configuration information for Buildbot.

branches = ['trunk', 
            'branches/bad_branch',
           ]

Because of Buildbot's architecture, we must specify (in some way) the branches that Buildbot should concern itself with; I define factories later on which generate the necessary Buildbot machinery for each branch specified here.

Instead of explicitly specifying the branches we want to monitor (which is usually all of them), it may be possible to poll SVN periodically and reconfigure Buildbot when a new branch is detected, but I haven't looked into that yet.

Specifying BuildSlaves

Now, we specify Buildslaves for c.

from buildbot.buildslave import BuildSlave
c['slaves'] = [BuildSlave("danke", "bitte"),
               BuildSlave("slug", "gross")]

This addition to c implies that we will have two BuildSlaves communicating with the master: one called danke (with a slave-password of bitte), and one called slug (with a slave-password of gross). We will configure the slaves (with their passwords) after we have configured the master.

It is important that these slave passwords remain relatively private, or else there are some security vulnerabilities that entail possible execution of arbitrary code on the master's host (alluded to ​here).

Then, pick a port over which the master and slave will communicate. This port must be publicly accessible on their respective networks. I chose 9989.

c['slavePortnum'] = 9989

Remember to make this port publicly accessible. This may necessitate forwarding ports.

Detecting source changes

Now we configure how the Buildmaster finds out about changes in the repository. There are a variety of ways the master can be configured to keep itself informed of changes, including polling the repository periodically, but I chose to manually inform the buildmaster of changes via an addition to SVN's post-commit hook which calls a buildbot/contrib script available ​here.

On the master.cfg side, we add the following:

from buildbot.changes.pb import PBChangeSource
c['change_source'] = PBChangeSource()

From here, I assume that the SVN repository is hosted on the same machine as the master. Digressing a moment from master.cfg, I navigate to the SVN repository's directory.

master$ cd /path/to/svn-repo
master$ $EDITOR hooks/post-commit

Once the editor is open, add the following to notify the buildbot master of changes upon commit.

# set up PYTHONPATH to contain Twisted/buildbot perhaps, if not already
# installed site-wide

/path/to/svn_buildbot.py --repository "$REPOS" --revision "$REV" \
--bbserver localhost --bbport 9989

If the master is hosted on a different machine than the repository, the bbserver flag above can be modified accordingly.

One caveat here is that we must modify the contrib/svn_buildbot.py script to propagate branch information on to Buildbot. Open svn_buildbot.py

master$ $EDITOR /path/to/svn_buildbot.py

and enact the following change.

# this should be around line 143
# split_file = split_file_dummy

split_file = split_file_branches

Now the SVN repo is set up to play nicely with the Buildbot installation and (hopefully) the master will be informed of each subsequent commit.

Schedulers

Back to master.cfg.

Next, we want to configure Schedulers. Whenever the master is informed of a change (via whatever object is held in c['change_source']) all Schedulers attached to the list held by c['schedulers'] will be informed of the change and, depending on the particular Scheduler in question, may or may not react by triggering a build or multiple builds.

Here, I'll configure a Scheduler for each branch to react to any change in that branch by running two builds.

from buildbot.scheduler import Scheduler
from buildbot.schedulers.filter import ChangeFilter

def buildSchedulerForBranch(branch):
    cf = ChangeFilter(branch=branch)
    return Scheduler(name="%s-scheduler" % branch,
                     change_filter=cf,
                     treeStableTimer=30,
                     builderNames=["full-build-%s" % branch,
                                   "smaller-build-%s" % branch])

c['schedulers'] = []

for branch in branches:
    c['schedulers'].append(buildSchedulerForBranch(branch))

The ChangeFilter is used to discriminate branches; note that branch isn't the only criterion a ChangeFilter can consider. The parameter treeStableTimer is the length of time that Buildbot waits before starting the build process.

Note that this Scheduler references "full-build-[branch]" and "smaller-build-[branch]", which are Builders we'll define right now.

Builders

Here, we specify the build procedures. First, we will define a number of BuildFactorys, which detail generic steps that a given Build will take, and then we create the Builds themselves and attach them to to the BuildmasterConfig dictionary. There is some machinery defined up front for the sake of generating separate builds for each branch with as little duplication as possible, so bear with me.

def addCheckoutStep(factory, defaultBranch='trunk'):
    """Ensure that `baseURL` has a forward slash at the end."""
    baseURL = 'svn://slug-jamesob.no-ip.org/home/job/tmp/fake_fipy_repo/' 
    factory.addStep(SVN(mode='update',
                        baseURL=baseURL))
 
def testAllForSolver(buildFact, solverType, parallel=False):
    """
    Add doctest steps to a build factory for a given solver type.
    """

    descStr = "for %s" % solverType
    solverArg = "--%s" % solverType
    
    buildFact.addStep(Doctest(description="testing modules " + descStr,
                              extraArgs=[solverArg, "--modules"]))

    buildFact.addStep(Doctest(description="testing examples " + descStr,
                              extraArgs=[solverArg, "--examples"]))

Here, addCheckoutStep takes a BuildFactory and adds a step which checks out whichever branch has been modified, as specified by the Scheduler reporting the changes. It does so by appending the name of the branch, as reported by the provoking Scheduler, to the baseURL that we pass in.

The function testAllForSolver takes a BuildFactory and adds steps which test both modules and examples for a given solver. I should mention at this point that here I use Doctest, a class I had to add to Buildbot.

Getting doctests to work

Buildbot integrates nicely with Trial, the unit-test framework that comes bundled with Twisted. Trial, unfortunately, doesn't seem to pick up on doctests, though there are scattered allusions speaking to the contrary (​here and ​here).

I tried a few approaches to getting Trial to process doctests (adding a __doctest__ module-level attribute, for one) to no avail. Even if there is some way to piggyback doctests onto Trial, it may involve a large modification of the FiPy source, which, if purely for the sake of making Buildbot happy, I don't think is worth it.

In light of this, I added an object that extends the class that handles Trial integration to handle doctests instead.

Within master.cfg, add

from buildbot.steps.python_twisted import Trial, TrialTestCaseCounter
from buildbot.steps.shell import ShellCommand
from twisted.python import log

class Doctest(Trial):
    """
    Add support for Python's doctests.
    """

    def __init__(self, description=None,
                       extraArgs=None,
                       **kwargs):

        ShellCommand.__init__(self, **kwargs)

        self.addFactoryArguments(description=description,
                                 extraArgs=extraArgs)

        self.testpath = "."
        self.command = ["python", "setup.py", "test"]
        self.extraArgs = extraArgs

        self.logfiles = {}

        if description is not None:
            self.description = [description]
            self.descriptionDone = [description + " done"]
        else:
            self.description = ["testing"]
            self.descriptionDone = ["tests"]
             
        # this counter will feed Progress along the 'test cases' metric
        self.addLogObserver('stdio', TrialTestCaseCounter())
             
    def start(self):
        if self.extraArgs is not None:
            if type(self.extraArgs) is list:
                self.command += self.extraArgs
            elif type(self.extraArgs) is str:
                self.command.append(self.extraArgs)
                
        self._needToPullTestDotLog = False
        log.msg("Doctest.start: command is '%s' with description '%s'." \
                    % (self.command, self.description))

        ShellCommand.start(self)
                 
    def _gotTestDotLog(self, cmd):
        Trial._gotTestDotLog(self, cmd)

        # strip out "tests" from the beginning 
        self.text[0] = self.description[0] + " " \
                        + " ".join(self.text[0].split(" ")[1:])

This allows us to run FiPy's test suite completely unmodified and have Buildbot interpret the results just as it would for Twisted's Trial.

Back to Builders

We never completed setting the Builders up, so let's do that now.

Let's first define a few Factory objects.

from buildbot.process import factory
from buildbot.steps.source import SVN

"""
Run a mostly-complete test-suite.
"""
fullFactory = factory.BuildFactory()

addCheckoutStep(fullFactory)
testAllForSolver(fullFactory, "pysparse")
testAllForSolver(fullFactory, "trilinos")

"""
Run a less-intensive test-suite.
"""
smallerFactory = factory.BuildFactory()

addCheckoutStep(smallerFactory)
smallerFactory.addStep(Doctest(extraArgs="--modules", 
                               description="testing modules"))

Now we will define and attach the Builders themselves.

c['builders'] = []

def makeBuildersForBranch(branch):
    builders = []

    builders.append({"name": "full-build-%s" % branch,
                     "slavename": "slug",
                     "builddir": "%s-slug" % branch,
                     "factory": fullFactory})

    builders.append({"name": "smaller-build-%s" % branch,
                     "slavename": "danke",
                     "builddir": "%s-danke" % branch,
                     "factory": smallerFactory})

    return builders

for branch in branches:
    for builder in makeBuildersForBranch(branch):
        c['builders'].append(builder)

The rest

Everything left in master.cfg concerns itself with how build information is reported back to us. Though there are very many possibilities open to us (e-mail reporting to blamed developers upon broken builds, IRC bots, Skynet, etc.), I haven't explored any of them other than the default web interface that Buildbot provides. With that in mind, here's the rest of master.cfg, basically unadulterated from the vanilla sample config.

####### STATUS TARGETS

# 'status' is a list of Status Targets. The results of each build will be
# pushed to these targets. buildbot/status/*.py has a variety to choose from,
# including web pages, email senders, and IRC bots.

c['status'] = []   


from buildbot.status import html
c['status'].append(html.WebStatus(http_port=8010))  

#
# from buildbot.status import mail
# c['status'].append(mail.MailNotifier(fromaddr="buildbot@localhost",
#                                      extraRecipients=["[email protected]"],
#                                      sendToInterestedUsers=False))
#
# from buildbot.status import words
# c['status'].append(words.IRC(host="irc.example.com", nick="bb",
#                              channels=["#example"]))
#
# from buildbot.status import client
# c['status'].append(client.PBListener(9988)) 

c['projectName'] = "FiPy"
c['projectURL'] = "http://www.ctcms.nist.gov/fipy/"
c['buildbotURL'] = "http://localhost:8010/"      

Configuring slaves

Configuring slaves is very simple.

danke$ buildslave create-slave slavedir slug-jamesob.no-ip.org:9989 danke bitte
danke$ $EDITOR slavedir/info/admin
danke$ $EDITOR slavedir/info/host

slug$ buildslave create-slave slavedir slug-jamesob.no-ip.org:9989 slug gross 
slug$ $EDITOR slavedir/info/admin
slug$ $EDITOR slavedir/info/host

Note that slug is both the master and a slave.

The form of the create-slave command is buildslave create-slave [directory for builds] [master's host address]:[port for communication] [slave-name] [slave-password].

So long as the port 9989 is open on both the slaves and master, we're done.

In summary

Buildbot is going to be an asset to us, though I still haven't contacted Doug as to how we're going to get it up and running.

In case I've screwed anything up in translating my config file to this blog post, I've uploaded the actual file to sandbox and it is available ​here.