nestrun takes a command template and a list of control.json files with variables to substitute. Substitution is performed using the Python built-in str.format method. See the Python Formatter documentation for details on syntax, and examples/jsonrun/do_nestrun.sh for an example.
nestrun also handles some signals by default.
This tells nestrun to stop spawning jobs. All jobs that were already spawned will continue running.
This tells nestrun to terminate if received twice. On the first SIGTERM, nestrun will emit a warning message; on the second, it will terminate all jobs and then itself.
This tells nestrun to immediately write a list of all currently-running processes and their working directories to stderr, then flush stderr.
usage: nestrun.py [-h] [-j N] [--template 'template text'] [--stop-on-error]
[--template-file FILE] [--save-cmd-file SAVECMD_FILE]
[--log-file LOG_FILE | --no-log] [--dry-run]
[--summary-file SUMMARY_FILE] [-d DIR]
[control_files [control_files ...]]
nestrun - substitute values into a template and run commands in parallel.
optional arguments:
-h, --help show this help message and exit
-j N, --processes N, --local N
Run a maximum of N processes in parallel locally
(default: 2)
--template 'template text'
Command-execution template, e.g. bash {infile}. By
default, nestrun executes the templatefile.
--stop-on-error Terminate remaining processes if any process returns
non-zero exit status (default: False)
--template-file FILE Command-execution template file path.
--save-cmd-file SAVECMD_FILE
Name of the file that will contain the command that
was executed.
--log-file LOG_FILE Name of the file that will contain output of the
executed command.
--no-log Don't create a log file
--dry-run Dry run mode, does not execute commands.
--summary-file SUMMARY_FILE
Write a summary of the run to the specified file
Control files:
control_files Nestly control dictionaries
-d DIR, --directory DIR
Run on all control files under DIR. May be used in
place of specifying control files.
The nestagg command provides a mechanism for combining results of multiple runs, via a subcommand interface. Currently, the only supported action is merging delimited files from a set of leaves, adding values from the control dictionary on each. This is performed via nestagg delim.
usage: nestagg.py delim [-h] [-k KEYS | -x EXCLUDE_KEYS] [-m {fail,warn}]
[-d DIR] [-s SEPARATOR] [-t] [-o OUTPUT]
file_template [control.json [control.json ...]]
positional arguments:
file_template Template for the delimited file to read in each
directory [e.g. '{run_id}.csv']
control.json Control files
optional arguments:
-h, --help show this help message and exit
-k KEYS, --keys KEYS Comma separated list of keys from the JSON file to
include [default: all keys]
-x EXCLUDE_KEYS, --exclude-keys EXCLUDE_KEYS
Comma separated list of keys from the JSON file not to
include [default: None]
-m {fail,warn}, --missing-action {fail,warn}
Action to take when a file is missing [default: fail]
-d DIR, --directory DIR
Run on all control files under DIR. May be used in
place of specifying control files.
-s SEPARATOR, --separator SEPARATOR
Separator [default: ,]
-t, --tab Files are tab-separated
-o OUTPUT, --output OUTPUT
Output file [default: stdout]