HyperMake

HyperMake is a parameterized pipeline definition language (think of a make where tasks can be parameterized) heavily inspired by Ducttape.

• Shell scripting: Write tasks in plain Bash, just like make. No Python or YAML for defining tasks.
• Cached intermediate results: Intermediate results are cached, so that if a task fails, it can be re-run from the last successful task.
• Parameterization of tasks: a task can be of multiple versions (e.g. in a ML pipeline, a task can be of different hyperparameters).
• Minimal juggling: Inputs and outputs are just files/symlinks, and arguments are all passed as environment variables. Tasks are realized as a child process.
• Automatic parallelization: based on the dependency DAG.
• Cloud-agnostic: tasks can be run locally, or on a cloud (e.g. AWS, Azure), with minimal changes to the pipeline.