nestly Package

nestly Package

nestly is a collection of functions designed to make running software with combinatorial choices of parameters easier.

core Module

Core functions for building nests.

class nestly.core.Nest(control_name='control.json', indent=2, fail_on_clash=False, warn_on_clash=True, base_dict=None, include_outdir=True)

Bases: object

Nests are used to build nested parameter selections, culminating in a directory structure representing choices made, and a JSON dictionary with all selections.

Build parameter combinations with Nest.add(), then create a nested directory structure with Nest.build().

Parameters:

• control_name – Name JSON file to be created in each leaf
• indent – Indentation level in json file
• fail_on_clash – Error if a nest level attempts to overwrite a previous value
• warn_on_clash – Print a warning if a nest level attempts ot overwrite a previous value
• base_dict – Base dictionary to start all control dictionaries from (default: {})
• include_outdir – If true, include an OUTDIR key in every control indicating the directory this control would be written to.

add(name, nestable, create_dir=True, update=False, label_func=<type 'str'>, template_subs=False)

Add a level to the nest

Parameters:

• name (string) – Name of the level. Forms the key in the output dictionary.
• nestable – Either an iterable object containing values, _or_ a function which takes a single argument (the control dictionary) and returns an iterable object containing values
• create_dir (boolean) – Should a directory level be created for this nestable?
• update (boolean) – Should the control dictionary be updated with the results of each value returned by the nestable? Only valid for dictionary results; useful for updating multiple values. At a minimum, a key-value pair corresponding to name must be returned.
• label_func – Function to be called to convert each value to a directory label.
• template_subs (boolean) – Should the strings in / returned by nestable be treated as templates? If true, str.format is called with the current values of the control dictionary.

build(root='runs')

Build a nested directory structure, starting in root

Parameters:root – Root directory for structure

iter(root=None)

Create an iterator of (directory, control_dict) tuples for all valid parameter choices in this Nest.

Parameters:root – Root directory Return type:Generator of (directory, control_dictionary) tuples.

nestly.core.control_iter(base_dir, control_name='control.json')

Generate the names of all control files under base_dir

nestly.core.nest_map(control_iter, map_fn)

Apply map_fn to the directories defined by control_iter

For each control file in control_iter, map_fn is called with the directory and control file contents as arguments.

Example:

>>> list(nest_map(['run1/control.json', 'run2/control.json'],
...               lambda d, c: c['run_id']))
[1, 2]

Parameters:

• control_iter – Iterable of paths to JSON control files
• map_fn (function) – Function to run for each control file. It should accept two arguments: the directory of the control file and the json-decoded contents of the control file.

Returns:

A generator of the results of applying map_fn to elements in control_iter

nestly.core.stripext(path)

Return the basename, minus extension, of a path.

Parameters:path (string) – Path to file

scons Module

SCons integration for nestly.

class nestly.scons.SConsEncoder(skipkeys=False, ensure_ascii=True, check_circular=True, allow_nan=True, sort_keys=False, indent=None, separators=None, encoding='utf-8', default=None)

Bases: json.encoder.JSONEncoder

JSON Encoder which handles SCons objects.

default(obj)

class nestly.scons.SConsWrap(nest, dest_dir='.', alias_environment=None)

Bases: object

A Nest wrapper to add SCons integration.

This class wraps a Nest in order to provide methods which are useful for using nestly with SCons.

A Nest passed to SConsWrap must have been created with include_outdir=True, which is the default.

Parameters:

• nest – A Nest object to wrap
• dest_dir – The base directory for all output directories.
• alias_environment – An optional SCons Environment object. If present, targets added via SConsWrap.add_target() will include an alias using the nest key.

add(name, nestable, **kw)

Adds a level to the nesting and creates a checkpoint that can be reverted to later for aggregation by calling SConsWrap.pop().

Parameters:

• name – Identifier for the nest level
• nestable – A nestable object - see Nest.add().
• kw – Additional parameters to pass to Nest.add().

add_aggregate(name, data_fac)

Add an aggregate target to this nest.

Since nests added after the aggregate can access the construct returned by the factory function value, it can be mutated to provide additional values for use when the decorated function is called.

To do something with the aggregates, you must SConsWrap.pop() nest levels created between addition of the aggregate and then can add any normal targets you would like which take advantage of the targets added to the data structure.

Parameters:

• name – Name for the target in the nest
• data_fac – a nullary factory function which will be called immediately for each of the current control dictionaries and stored in each dictionary with the given name as in SConsWrap.add_target().

add_controls(env, target_name='control', file_name='control.json', encoder_cls=<class 'nestly.scons.SConsEncoder'>)

Adds a target to build a control file at each of the current leaves.

Parameters:

• env – SCons Environment object
• target_name – Name for target in nest
• file_name – Name for output file.

add_nest(name=None, **kw)

A simple decorator which wraps nestly.core.Nest.add().

add_target(name=None)

Add an SCons target to this nest.

The function decorated will be immediately called with each of the output directories and current control dictionaries. Each result will be added to the respective control dictionary for later nests to access.

Parameters:name – Name for the target in the name (default: function name).

add_target_with_env(environment, name=None)

Add an SCons target to this nest, with an SCons Environment

The function decorated will be immediately called with three arguments:

• environment: A clone of the SCons environment, with variables populated for all values in the control dictionary, plus a variable OUTDIR.
• outdir: The output directory
• control: The control dictionary

Each result will be added to the respective control dictionary for later nests to access.

Differs from SConsWrap.add_target() only by the addition of the Environment clone.

pop(name=None)

Reverts to the nest stage just before the corresponding call of SConsWrap.add_aggregate(). However, any aggregate collections which have been worked on will still be accessible, and can be called operated on together after calling this method. If no name is passed, will revert to the last nest level.

Parameters:name – Name of the nest level to pop.

nestly.scons.name_targets(func)

Wrap a function such that returning 'a', 'b', 'c', [1, 2, 3] transforms the value into dict(a=1, b=2, c=3).

This is useful in the case where the last parameter is an SCons command.

Subpackages

• scripts Package
  • nestrun Module
  • nestagg Module