digital_land.expectations.checkpoints package
Submodules
digital_land.expectations.checkpoints.base module
- class digital_land.expectations.checkpoints.base.BaseCheckpoint
Bases:
ABC- abstract load()
filled in by child classes, ensures a config is loaded correctly should raise error if not
- abstract run()
- abstract save(output_dir, format='csv')
filled in by child classes, uses save functions to save the data. could add default behaviour at somepoint
digital_land.expectations.checkpoints.csv module
- class digital_land.expectations.checkpoints.csv.CsvCheckpoint(dataset, file_path)
Bases:
BaseCheckpoint- get_connection()
- load(rules)
filled in by child classes, ensures a config is loaded correctly should raise error if not
- operation_factory(operation_string: str)
- run()
- run_expectation(conn, expectation) tuple
- save(output_dir: Path)
filled in by child classes, uses save functions to save the data. could add default behaviour at somepoint
digital_land.expectations.checkpoints.dataset module
- class digital_land.expectations.checkpoints.dataset.DatasetCheckpoint(dataset, file_path, organisations: Organisation)
Bases:
BaseCheckpoint- get_rule_orgs(rule: dict) list
for each rule we need to get a list of the organisations that the rule applies to this is a semi colon separated list of individual orgs, org datasets or org prefixes which are a key of the inputted dict
- Parameters:
rule (-) -- a single expectation rule
- load(rules)
given a set of rules this function loads them into the checkpoint for the dataset checkpoint we antiipates rules contain organisations with which expectations need parsing
- operation_factory(operation_string: str)
conevrts a string into an operation, available operations are specific to the checkpoint
- Args
operation: a string representing an operation
- parse_rule(rule, org=None) dict
turn a rule into an expectation given an org it will format text strings using jinja templating
- run(prefetch_resources=False)
run the set of expectations that have been loaded into the checkpoint results will be stored in the log and can be saved used .save
- run_expectation(expectation) tuple
runs a given expectation returning the result, description and message a log can be provided to record this information
- save(output_dir: Path)
save the outputs as a file, the file is named based the the dataset and stored in the provided directory