SCons is an excellent build tool (analogous to make). The nestly.scons module is provided to make integrating nestly with SCons easier. SConsWrap wraps a Nest object to provide additional methods for adding nests. SCons is complex and is fully documented on their website, so we do not describe it here. However, for the purposes of this document, it suffices to know that dependencies are created when a target function is called.
The basic idea is that when writing an SConstruct file (analogous to a Makefile), these SConsWrap objects extend the usual nestly functionality with build dependencies. Specifically, there are functions that add targets to the nest. When SCons is invoked, these targets are identified as dependencies and the needed code is run. There are also aggregate functions (this is aggregate with a hard second “a”; rhymes with “Watergate”) that don’t get immediately called, but rather when the finalize_aggregate() method is called.
SConsWrap objects wrap and modify a Nest object. Each Nest object needs to have been created with include_outdir=True, which is the default.
Optionally, a destination directory can be given to the SConsWrap which will be passed to Nest.iter():
>>> nest = Nest()
>>> wrap = SConsWrap(nest, dest_dir='build')
In this example, all the nests created by wrap will go under the build directory. Throughout the rest of this document, nest will refer to this same Nest instance and wrap will refer to this same SConsWrap instance.
Nests can still be added to the nest object:
>>> nest.add('nest1', ['spam', 'eggs'])
SConsWrap also provides a convenience decorator SConsWrap.add_nest() for adding nests which use a function as their nestable. The following examples are exactly equivalent:
@wrap.add_nest('nest2', label_func=str.strip)
def nest2(c):
return [' __' + c['nest1'], c['nest1'] + '__ ']
def nest2(c):
return [' __' + c['nest1'], c['nest1'] + '__ ']
nest.add('nest2', nest2, label_func=str.strip)
Another advantage to using the decorator is that the name parameter is optional; if it’s omitted, the name of the nest is taken from the name of the function. As a result, the following example is also equivalent:
@wrap.add_nest(label_func=str.strip)
def nest2(c):
return [' __' + c['nest1'], c['nest1'] + '__ ']
Note
add_nest() must always be called before being applied as a decorator. @wrap.add_nest is not valid; the correct usage is @wrap.add_nest() if no other parameters are specified.
The fundamental action of SCons integration is in adding a target to a nest. Adding a target is very much like adding a nest in that it will add a key to the control dictionary, except that it will not add any branching to a nest. For example, successive calls to Nest.add() produces results like the following:
>>> nest.add('nest1', ['A', 'B'])
>>> nest.add('nest2', ['C', 'D'])
>>> pprint.pprint([c.items() for outdir, c in nest])
[[('nest1', 'A'), ('nest2', 'C')],
[('nest1', 'A'), ('nest2', 'D')],
[('nest1', 'B'), ('nest2', 'C')],
[('nest1', 'B'), ('nest2', 'D')]]
A crude illustration of how nest1 and nest2 relate:
# C .---- - -
# A .----------o nest2
# | D '---- - -
# o----o nest1
# | C .---- - -
# B '----------o nest2
# D '---- - -
Calling add_target(), however, produces slightly different results:
>>> nest.add('nest1', ['A', 'B'])
>>> @wrap.add_target()
... def target1(outdir, c):
... return 't-{0[nest1]}'.format(c)
...
>>> pprint.pprint([c.items() for outdir, c in nest])
[[('nest1', 'A'), ('target1', 't-A')],
[('nest1', 'B'), ('target1', 't-B')]]
And a similar illustration of how nest1 and target1 relate:
# t-A
# A .----------o------ - -
# o----o nest1 target1
# B '----------o------ - -
# t-B
add_target() does not increase the total number of control dictionaries from 2; it only updates each existing control dictionary to add the target1 key. This is effectively the same as calling add() (or add_nest()) with a function and returning an iterable of one item:
>>> nest.add('nest1', ['A', 'B'])
>>> @wrap.add_nest()
... def target1(c):
... return ['t-{0[nest1]}'.format(c)]
...
>>> pprint.pprint([c.items() for outdir, c in nest])
[[('nest1', 'A'), ('target1', 't-A')],
[('nest1', 'B'), ('target1', 't-B')]]
Astute readers might have noticed the key difference between the two: functions decorated with add_target() have an additional parameter, outdir. This allows targets to be built into the correct place in the directory hierarchy.
The other notable difference is that the function decorated by add_target() will be called exactly once with each control dictionary. A function added with add() may be called more than once with equal control dictionaries.
Like add_nest(), add_target() must always be called, and optionally takes the name of the target as the first parameter. No other parameters are accepted.
Aggregate functions are a special case of targets. Instead of the decorated function being called immediately, it will be called at some other specified moment. An example:
>>> nest.add('nest1', ['A', 'B'])
>>> @wrap.add_aggregate(list)
... def aggregate1(outdir, c, inputs):
... print 'agg', c['nest1'], inputs
...
>>> nest.add('nest2', ['C', 'D'])
>>> nest.add('nest3', ['E', 'F'])
>>> @wrap.add_target()
... def add_target(outdir, c):
... c['aggregate1'].append((c['nest2'], c['nest3']))
...
>>> wrap.finalize_aggregate('aggregate1')
agg A [('C', 'E'), ('C', 'F'), ('D', 'E'), ('D', 'F')]
agg B [('C', 'E'), ('C', 'F'), ('D', 'E'), ('D', 'F')]
The first argument to add_aggregate() is a factory function which will be called with no arguments and added to each control dictionary as the name of the aggregate. Targets added after the aggregate are able to access and modify the value added.
When the aggregate is finalized, it will be called with output directory and control dictionary like a target, but also with the value which was added to the control dictionary. This allows aggregates to use values from later targets.
Aggregates can either be finalized by calling finalize_aggregate() or finalize_all_aggregates(). The former will finalize a particular aggregate by name, while the latter finalizes all aggregates in the same order they were added.
The second parameter to add_aggregate() is the same as the first parameter to add_target(): the name of the aggregate, which will default to the name of the function if none is specified.
While the previous example demonstrate how to use the various methods of SConsWrap, they did not demonstrate how to actually call commands using SCons. The easiest way is to define the various targets from within the SConstruct file:
from nestly.scons import SConsWrap
from nestly import Nest
import os
nest = Nest()
wrap = SConsWrap(nest, 'build')
# Add a nest for each of our input files.
nest.add('input_file', [join('inputs', f) for f in os.listdir('inputs')],
label_func=os.path.basename)
# Each input will get transformed each of these different ways.
nest.add('transformation', ['log', 'unit', 'asinh'])
@wrap.add_target()
def transformed(outdir, c):
# The template for the command to run.
action = 'guppy mft --transform {0[transformation]} $SOURCE -o $TARGET'
# Command will return a tuple of the targets; we want the only item.
outfile, = Command(
source=c['input_file'],
target=os.path.join(outdir, 'transformed.jplace'),
action=action.format(c))
return outfile
A function name_targets() is also provided for more easily naming the targets of an SCons command:
@wrap.add_target('target1')
@name_targets
def target1(outdir, c):
return 'outfile1', 'outfile2', Command(
source=c['input_file'],
target=[os.path.join(outdir, 'outfile1'),
os.path.join(outdir, 'outfile2')],
action="transform $SOURCE $TARGETS")
In this case, target1 will be a dict resembling {'outfile1': 'build/outdir/outfile1', 'outfile2': 'build/outdir/outfile2'}.
Note
name_targets() does not preserve the name of the decorated function, so the name of the target must be provided as a parameter to add_target().
A more involved, runnable example is in the examples/scons directory.