bloggo ergo sum

I work on large projects for my job, and we currently use Make, just like everyone else. Our project layout is complicated, which means that the Makefiles are practically voodoo at this point. What makes it even worse is that we try to re-use code, which means we have a directory structure just for common code. Inevitably though, as any engineer finds out when trying to re-use code like this (especially if being shared among too many projects), the code isn’t really always common, and in some cases, diverges significantly. This leads to the use of ever-more confusing variable uses, included Makefiles (ostensibly to make things more modular), and other trickery to make targets build and link in the correct order.

A few years ago, I discovered a project called SCons. It promised to do a lot of neat things, but it was still new at that point, and brand-new open source projects typically aren’t usable. I tried it out, but it seemed flaky and unnatural. Some of that can be attributed to my lack of experience with Python and build tools other than Make, but it was also just flaky.

Fast-forward to 2006. I now use SCons as a part of libgit++, a project I’m working on at home. It has turned out to be a welcome replacement for Make – I have been developing my project on multiple computers, and the environments weren’t exactly the same, and I didn’t want to have to learn the GNU auto* tools to have my project build correctly across multiple platforms. SCons promised to provide some of that functionality, so I switched from Make to SCons. I’ve been very happy with it, so I decided that I’d try to implement it as the build tool for a project at work. One of the problems I eventually ran into was that targets in one directory might depend on targets in another directory. I needed a way to account for that without relying on hard-coded filenames (which depend on your directory layout to not change) and without relying on including files all over the place and carefully tweaking it so that targets are created in the right order before they are built.

I considered including SConscripts in a “trail” fashion, where one SConscript would include whatever SConscripts it needed to fulfill its dependencies. This is problematic because it makes it very hard to figure out what’s going on (unless you’re the one that wrote all the SConscript files). It also make it cumbersome to change your directory layout – as soon as you do, you have to backtrack through the SConscript files to figure out where the chain is broken.

I considered just writing a huge SConstruct file at the top of the directory structure and not using SConscript files, but that wouldn’t be much better than Make, and it would be a pain to maintain as the project directory layout evolved, although features like having multiple ‘Environment’ objects help with that.

I then thought to have a global Dictionary with familiar names like ‘mylib’ as keys and target objects as values. Each SConscript files would store its targets in that Dictionary for access by other SConscript files. This approach seemed like a good idea, but it didn’t properly take care of dependencies (i.e., the Dictionary must be almost entirely filled out for all dependencies to be correctly met), and to make it work would require creating the target objects in a very particular order, which is a maintainability problem and one of the main ones that I wanted to switch from Make in order to fix.

Then, I thought I might just read all the SConscript files twice. The first pass would get all the independent target objects put in the Dictionary (while the dependent targets would be foobar), and the second pass would take care of the dependent targets. There are problems with this approach too: the first round of the dependent targets would just be unused (and unusable) and the technique would make for redundant and confusing SConscript (or the SConstruct) files – bad for both maintainability and performance.

Finally, I came up with an acceptible compromise. I decided to use the global Dictionary to track all the target objects, and I would read the SConscript files in a particular order, but they would all be read from the SConstruct file, and the rationale for the order would be spelled out in comments, allowing future maintainance to be relatively simple. Each SConscript file would create its target(s), and the copy the objects to the Dictionary. Each target that had dependencies could satisfy them by specifying the appropriate object from the Dictionary, and no filenames would be needed. Directory structures could be modified at will without having to modify any SConscript files, and components of the project could be added and removed with only minimal changes to the SConstruct file. SConscript files would be maintained by the owners of the various components without really needing to worry about the SConstruct file or any of the other SConscript files. Without further ado, here are code snippets that illustrate the technique.

# -*- python -*-
# SConstruct file
objtbl = {
'sub1': None,
'sub2': None
}

# dependent targets
SConscript(['a/SConscript'])

Note that the independent targets are in the SConscript files that are read first. The SConstruct file goes at the top of the project directory structure. In this case, my directory structure is this:

./SConstruct
./a
./a/src
./a/src/SConscript
./a/src/file1.c
./a/obj
./a/SConscript
./b
./b/src
./b/src/SConscript
./b/src/subfile1.c
./b/obj
./b/SConscript
./c
./c/src
./c/src/SConscript
./c/src/subfile1.c
./c/obj
./c/SConscript

Libraries are built in directories ‘b/obj’ and ‘c/obj’ which are both a dependency for the library built in ‘a/obj’.

# -*- python -*-
# a/SConscript, b/SConscript, and c/SConscript
BuildDir('obj', 'src', 0)
SConscript('obj/SConscript')

Note that the above file is the same for directories ‘a’, ‘b’, and ‘c’.

# -*- python -*-
# b/SConscript
Import('objtbl')
objtbl['sub1'] = Library('sub1', 'subfile1.c')

This SConscript file takes care of building a static library in ‘b/obj’.

# -*- python -*-
# c/SConscript
Import('objtbl')
objtbl['sub2'] = Library('sub2', 'subfile1.c')

This SConscript file takes care of building a static library in ‘c/obj’.

# -*- python -*-
# a/src/SConscript
Import('objtbl')

Library('lib',
['file1.c',
objtbl['sub1'],
objtbl['sub2']
]
)

And finally, this SConscript file creates another library, using the target objects created in the other two SConscript files as dependencies, which scons uses to figure out what to build first. Here is the result of running SCons:

$ scons
scons: Reading SConscript files ...
scons: done reading SConscript files.
scons: Building targets ...
gcc -c -o a/obj/file1.o a/src/file1.c
gcc -c -o b/obj/subfile1.o b/src/subfile1.c
ar r b/obj/libsub1.a b/obj/subfile1.o
ranlib b/obj/libsub1.a
ar: creating b/obj/libsub1.a
gcc -c -o c/obj/subfile1.o c/src/subfile1.c
ar r c/obj/libsub2.a c/obj/subfile1.o
ranlib c/obj/libsub2.a
ar: creating c/obj/libsub2.a
ar r a/obj/lib.a a/obj/file1.o b/obj/libsub1.a c/obj/libsub2.a
ranlib a/obj/lib.a
ar: creating a/obj/lib.a
scons: done building targets.


Happy, happy, joy, joy!