Processes

This section covers the processes that we run across our pipelines. There may be some minor differences, such as where results are stored, depending on how and where they are run, but the following pages cover the concepts within each stage with links to the code that needs to be run.

Processes are the individual stages. For how they are orchestrated together to produce a dataset, see workflows.

• Collect - download resources from a set of endpoints
• Plan - create log and resource csvs, and a makefile to be used in the transform and assemble processes
• Transform - process a resource against a dataset to produce facts and their provenance
• Generate tasks - build the task table of data quality problems from collection logs and issues