Monitoring Data Quality

This page explains the processes we follow to fix data quality issues that we actively monitor for.

Each of these sections covers a different issue, each of which are defined by our data quality requirements.

Check deleted entities

To keep the datasets up-to-date on the platform, we need to check entities that have been deleted from the latest resource every week. This occurs when the LPAs have deleted the entities on their endpoint but not told us. Once we have confirmed which entities have been deleted, we contact the LPAs to make sure. Once we have received confirmation, we can retire the entities.

The recommended steps to resolve this are as follows:

Run the this report
For each dataset, compare the Latest resource entity count with the Platform entity count. Make note of which dataset has more counts for the platform compared to the latest resource.
List those that need to be retired here. You will want the collection, endpoint, and source.
The LPAs will need to be contacted. Once confirmed that these were deleted, follow the retire entities process
Note in the sheet if an entitiy could not be retired

Success criteria:
The count of entities on the platform and on the latest resource should be the same. Run the report to make sure that the counts are matching.

Retire broken, non-primary endpoints

Trigger

We define an endpoint as “broken” once it has been logged with a non-200 status for more than 30 consequtive days. We pro-actively end-date broken endpoints and their sources when they are not the primary endpoint for a provision, i.e. the endpoint is not the only or most recently added endpoint for a provision.

e.g.
Wiltshire council has two active brownfield-land endpoints, one added in 2025 and one added in 2024. This makes the 2025 endpoint the primary endpoint. The 2024 endpoint has had a 404 status for 62 days, so it should be given an end-date.

NOTE
This applies to all datasets, including ODP, statutory and single-source.

Task

Identify broken, non-primary endpoints through this datasette query.
Download the query results as a scv file called retire.csv in the root of your local config directory.
Run the following command:

digital-land retire-endpoints-and-sources retire.csv

Test

Once the changes have been merged into main, the endpoints and sources you retired should no longer appear in the datasette query results.

Identify new data sources for broken, primary endpoints

Trigger

We define an endpoint as “broken” once it has been logged with a non-200 status for more than 30 consequtive days. And we call an endpoint a “primary” endpoint when it is the most recently added, or the only endpoint for a provision.

e.g.

the archaeological-priority-area dataset has one endpoint; this is the primary endpoint

the local-authority-district dataset has 3 active endpoints; the most recently added one is the primary endpoint

For ODP datasets, endpoint errors are raised back to data providers through the Submit service so we don’t need to do anything.

However, when non-ODP datasets have broken, primary endpoints we should search for alternatives.

Task

Identify broken, primary endpoints through this datasette query.
For any broken endpoints, search the data provider’s website for any newly published endpoint URLs (the source URL for the broken endpoint should take you to the correct site).
If you find any newer endpoints, add an end-date for the broken endpoint and source and follow the adding data process to add the new one.

Test

Once the changes have been merged into main, the primary endpoint for the provision should no longer appear in the datasette query.

Identify new data sources for stale endpoints

Trigger

We define an endpoint as “stale” when it has not been updated with new data within the time period we expect.

e.g.
the source of the latest endpoint we have for flood-risk-zone data published by the Environment Agency states that the dataset is updated quarterly. If the start date of the latest resource is 01/01/2024 and today’s date is 30/06/24 there hasn’t been an update for 6 months so we would say this endpoint is stale.

NOTE
For our compiled datasets, local planning authorities are responsible for updating endpoints or publishing new ones for new datasets so we don’t monitor for staleness.

For our single source datasets (i.e. those with national coverage from a single data provider) we need to check whether we have added the most up to date data.

Task

Check for any stale endpoints by running the monitor frequency of datasets report. This will identify any endpoints which have not been updated within the expected time period.
For any identified datasets you should check to see whether the data provider has published more up to date data on a new endpoint. You can use the source of existing endpoints to find their website.
If you find a new endpoint you will need to add it. Check the new endpoint for existing provision scenario on the Adding data page to find the steps to follow in order to retire old endpoints, add the new one and assign any new entities if required.

Test

Once you’ve added the new endpoint and merged the changes, re-run the monitor frequency of datasets report; the dataset you’ve updated should no longer be in the list.

Out of range entities

Trigger

This is a configuration error where the entity numbers that have been used in a dataset are not within the range defined for that dataset. These issues will be raised in the issue report for ODP datasets or all datasets, where the issue_type = “entity number out of range”.

The entity range for datasets are defined in the specification repository, select a dataset to view its entity range, defined by the entity-minimum and entity-maximum fields.

Task

In order to fix, for each dataset with issues you should:

Delete the entries in lookup.csv which are using an incorrect entity number, go to Datasette and select the relevant dataset. Next, filter the issue table using the resource and issue_type fields present in the downloaded issue table, then use the value field to identify the incorrect entity number. Now, find the incorrect entity number in lookup.csv and remove the entire row.
Follow the assign entities process to assign new entity numbers and replace the deleted lookup entries.

Test

Once fixed, there should no longer be any issues raised in the issue report.

Invalid Organisations

One of our monitoring tasks is patching any invalid organisation issues that arise. This isually happens if the organisation value provided in the endpoint is wrong or missing e.g it could be a blank field or the wrong organisation name / identifier.

A list of invalid organisation issues can be optained by downloading the issue report for ODP datasets or all datasets and filtering for invalid organisations under issue-type.

To fix this, we can make use of the patch.csv file. More information on how this file works can be found in the pipeline/patch section in configure an endpoint.

For example, if we were given the wrong organisationURI in a brownfield-land dataset, we can patch it by targetting the endpoint, give the current uri in the pattern section, and the desired uri in the value section like so:

brownfield-land,,OrganisationURI,http://opendatacommunities.org/id/london-borough-council/hammersmith-and-,http://opendatacommunities.org/doc/london-borough-council/hammersmith-and-fulham,,,,,890c3ac73da82610fe1b7d444c8c89c92a7f368316e3c0f8d2e72f0c439b5245

To test it, follow the guidance in building a collection locally but keep the new patch entry and focus on the desired endpoint.

Organisational changes

When organisations are created or ended we need to:

Create new organisation entities for any newly created organisations.
Make entities for organisations which have been ended have an end-date.
If appropriate, make sure any entities that any ended organisations were responsible for are moved to the new responsible organisations.

Note, this guidance relates to local-authority organisation changes, which have the most impact on ODP datasets.

Trigger

We will know that organisations need to be ended when changes are announced by [governing body]. These may involve multiple existing councils being replaced by a single unitary authority, or other variations of changes.

We may be notified of changes like this by team members, in which case we should act immediately.

Otherwise, the way we should make sure we check for any possible changes in order to be alerted is by monitoring updates to organisation typology datasets. If changes to local authority organisations are expected at least once each year, when the time between the latest resource start date and the current day’s date is greater than a year we should check for any changes which need to be reflected in the dataset. This test is defined by data-quality need T-05.

Task

Use the dataset editor to add new records for new organisations. You may need to cross-reference some other datasets to add all the necessary details, for instance the ONS codes for the local authority and the local planning authority.
Use the dataset editor to add an end-date for any organisations which have been terminated. Use the end date from the official announcement.

The next step will vary depending on how the local authorities transition existing datasets to the new organisation.

If there is not a new endpoint from the new organisation

Sometimes it may take a long time for data to be transitioned to a newly created organisation. In which case the existing endpoints from the old organisation should be kept live and maintained until there is a new one for the new organisation, at which point the process below can be followed.

If there is a new endpoint from the new organisation

Follow the standard process for validating and adding a new endpoint. During the validation step use the duplicate check step to check for any duplicates with existing data. This should highlight any existing entities that the new records match to. For any matches, existing entity numbers can be used instead of the new ones generated by the add-data process. Any new entities from the new organisation which don’t match can be given new entity numbers.
Any existing entities from the old organisation which haven’t been matched to new records from the new organisation should be given an end-date [note: we don’t currently have a process for this].
The entity-organisation range should be assigned to the new organisation for any of the entity numbers which are now being used for the new organisation’s records.
Retire any endpoints for the old organisation’s provisions so they are no longer collected.

De-duplication of conservation-area data

The purpose of this process is to ensure that duplicate data is not stored unnecessarily for the conservation-area dataset generated by an organisation which may have also been provided by Historic England(HE).

The initial deduplication of conservation area geographies now runs automatically each weekday evening as part of the Config Evening Pipeline. The manual steps below relate to the subsequent review of duplicate entities in Power BI and preparing corrections to old-entity.csv — this part still requires human judgement and is not automated.

The steps required for this process:-

Run the add-data tasks for conservation-area dataset (making a note of how many entities were added in the lookup file).
Raise the pull-request(PR) and ensure that it has been merged into the main branch so that the duplicate entities are picked up by the expectation report on the following day.
DO NOT inform the organisation at this stage.
On Power BI navigate to the “Digital Planning” workspace then to the “Planning Data Monitoring” report from where you select the “Duplicate Conservation Area” page.(Link_0)
Click on the reports TITLE in order for the options panel to appear to right hand side
Click on the three dots for the more options dropdown menu, from which you select “Export data” to download the output.
Open up the exported file to show the HE duplicate entites.
Filter on the message column for “complete_match” criteria
Filter on the entity_a_organisation.name column for the organisation Historic England and filter on the entity_b_organisation.name column for the organisation for which the data was added on the previous day (re:step 1)
Copy the entities in columns entity_a and entity_b
Prepare the data to be appended to the old-enity.csv located at Link_1 in following format
where entity_a=old-entity and entity_b=entity
e.g. 44012512,301,44013703,redirect Historic England duplicate to LPA entity,2025-08-28,
Also DO NOT forget to update the entity-organisation file located at Link_2
When this change is merged, check the PowerBI report to confirm the duplicate entities have been fixed.

Retire MHCLG fake template data for local plans and plan timetables

Trigger

When MHCLG was setting up the local plans system, it pre-seeded placeholder (“fake template”) data in the local-plan and plan-timetable datasets on behalf of LPAs. This was done so that the platform had something to show before LPAs had published their own data.

Once an LPA publishes its own authoritative data, the MHCLG placeholder data is redundant and should be retired. If it is not retired, duplicate or conflicting records exist on the platform.

How the script works

The script (retire-mhclg-plan-data.py), found in the Config repository, processes both datasets in sequence:

plan-timetable: MHCLG placeholder entities fall within the entity range 5101702–5109686. For each LPA that has provided new authoritative data outside this range, the script identifies the corresponding MHCLG-owned entity range in pipeline/local-plan/entity-organisation.csv and retires those entities.

local-plan: MHCLG placeholder entities fall within the entity range 4220656–4220966. The script generates the expected fake reference for each LPA (in the format {lpa-slug}-new-local-plan) and finds the matching MHCLG-owned entity in pipeline/local-plan/lookup.csv.

For both datasets, the script appends retired entities to pipeline/local-plan/old-entity.csv with status 410 and today’s date. It includes several safety checks before writing anything — it verifies there is no overlap between the MHCLG placeholder entities and the LPA’s real data, and that every LPA with authoritative data has a corresponding placeholder to retire.

Note: the script skips LPAs that updated MHCLG placeholder data in-place (i.e. their entities fall within the MHCLG range), as those records were overwritten rather than duplicated, so no retirement is needed.

This process runs automatically each weekday evening as part of the Config Evening Pipeline. The steps below are for running it manually, for example to investigate an issue or resume a failed single-source batch run.

Task

Clone the Config repository if you have not already done so, then create and activate a virtual environment.
Run the script from the root of the repository: python3 .github/scripts/retire-mhclg-plan-data.py
Review the output. The script will log which LPAs had MHCLG template data identified for retirement, and print a summary of how many entities were retired per dataset. If no LPAs have newly provided authoritative data since the last run, it will exit cleanly with no changes.
If the script raises a ValueError naming one or more LPAs, this means those LPAs have provided real data but the script could not find a corresponding MHCLG placeholder to retire. This is a data integrity issue that needs investigating before re-running — check that the relevant rows exist in pipeline/local-plan/lookup.csv and pipeline/local-plan/entity-organisation.csv.
Commit and merge the changes to pipeline/local-plan/old-entity.csv.

Test

Once merged, confirm the retired entities are no longer active on the platform by checking Datasette for the relevant local-plan or plan-timetable entities. The retired entity numbers should now redirect (HTTP 410) rather than return active records.

Unknown entities

To keep the datasets up-to-date on the platform, we need to check “unknown entity” issues every week and assign entities.
The unknown entities issue usually occurs when an LPA updates their data on the endpoint we are retrieving and adds new records. These records will have reference values we do not have on the platform, hence when the system realises the new data has been added and the references of those new data are not on the platform, it will trigger an unknown entity issue.

This process runs automatically each weekday evening as part of the Config Evening Pipeline. The steps below are for running it manually, for example to investigate an issue or resume a failed single-source batch run.

The datasets that require assigning entities are categorised into three main scopes:

ODP Datasets – These datasets are supported by ODP funding. Datasets categorised as ODP can be found here ODP Data

Mandated Datasets – These are datasets that LPAs are legally required to provide, this includes brownfield-land and developer-contributions datasets.

Single Source Datasets – This category includes data obtained from authoritative sources or seeded data received from the Data Design Team.

The recommended steps to resolve this are as follows:

Setup Config Repo
Clone the Config repository if it has not already been done, then create and activate a virtual environment.
Run the Script
The script can be run using the command python3 batch_assign_entities.py

Upon execution, the script will download the issue_summary.csv file to the root directory of the Config folder.

The downloaded issue_summary.csv includes a column called scope, this column indicates the scope for each dataset. This scope includes the categories specified above, such as ODP, Mandated and Single source.
Analyse Unknown Entity issues
Open the issue_summary.csv file and apply a filter to the “scope” column to display only entries related to ODP. Begin by analysing all unknown entities issues associated with the ODP scope.

If the count_issue for any dataset is unusually high, verify that the entities are valid and new. count_issue may also be high if the LPA has recently their references for existing entities. Keep a note of endpoints with an unusually high number of count_issue to review once the entities have been assigned.

The command will prompt the user to confirm. Type “yes” to assign Unknown entities for ODP.

The command will prompt the user to enter scope (odp/mandated/single-source). Type “odp” to assign entities.

It will download all the resources for unknown entities into a resources folder, assign entities, and then delete the downloaded resource files. The affected dataset’s lookup.csv should now have new rows with the assigned entities. The amount of entities that needed to be assigned should be the same amount that have been added in the lookup file.

The previous assignment process which allowed Unknown entities to be automatically assigned has now been updated and provides an interactive issue summary reporting facility which highlights issues and enables corrective measures to be actioned to enhance data integrity.

Review the entities assigned for the endpoint you’ve noted. The key thing to check here is whether the references are a continuation or follow a similar format to existing lookups for that provision.

Note: If the entities belong to the Conservation Area dataset, you should check for duplicates using endpoint checker, refer Step 3 in Validating an endpoint. Once the new entries for the lookup.csv have been generated, use the outputs from the Duplicates with different entity numbers section of the endpoint checker to replace the newly generated entity numbers for any duplicates, with the entity numbers of the existing entity that they match.
Assign entities for Mandated and single-source datasets
Repeat Step 3 for assign entities for Mandated and single-source datasets.

Enter the scope, either mandated or single-source based on requirement.
Review Changes
Once merged, use endpoint_dataset_issue_type_summary table and check if the previous unknown entity issues are resolved.

Make a note in the ticket if you are not able to assign entities for any LPA.

Success criteria:
Ideally, the number of unknown entity errors should be zero after completing the above steps.

Errors raised by the batch assign entities script

The bin/batch_assign_entities.py script records errors in batch_assign_summary_[scope].csv. This output summary CSV is used in the manual assign entities process in the Manage service, so users can review the resources the script could not safely assign automatically. These errors are safety checks: they do not always mean the data is wrong, but they do mean the script could not safely accept the generated entity assignment without manual review.

The summary file includes these useful columns:

dataset - the dataset or pipeline being processed
resource - the current resource hash
organisation - the organisation associated with the resource or flagged row
reference - the provider reference where the error relates to a specific entity
status - either success or error
error_code - the validation error or Python exception type
message - extra context from the script

Terms used in these errors

The script compares the latest resource for an endpoint with the previous resource we collected for the same endpoint. This is how it decides whether the entities it has just assigned look safe.

current resource - the latest resource being processed by the script. This is the file linked from the unknown entity issue and downloaded into the local resource/ folder.
previous resource - the older transformed resource for the same endpoint. The script finds this from historic endpoint data and downloads it from files.planning.data.gov.uk so it can compare old and new data.
current entity - an entity number found in the current transformed resource after the script has run entity assignment.
previous entity - an entity number found in the previous transformed resource for the same endpoint.
new entity - a current entity that was not present in the previous resource. These are the entity numbers the script is trying to validate before accepting the generated lookup changes.

For example, if the previous resource contained entities 44000001, 44000002 and 44000003, and the current resource contains 44000001, 44000002, 44000003 and 44000004, then 44000004 is the new entity. If the current resource contains no entity numbers that were missing from the previous resource, the script raises current_resource_no_new_entities.

Error code	What it means	Why it is flagged	Example
`previous_resource_not_found`	The script could not fetch or read the previous transformed resource for that endpoint.	Without the previous resource, the script cannot tell which entities are genuinely new or whether current rows duplicate existing platform entities.	Datasette returns no previous resource hash for the endpoint, or `https://files.planning.data.gov.uk/[collection]-collection/transformed/[dataset]/[old-resource].csv` cannot be downloaded.
`previous_resource_empty`	The previous transformed resource exists but has no entity rows.	If the old resource has no entities, every current entity appears new and the comparison is not reliable.	The previous transformed CSV downloads successfully but only contains headers, or contains no rows with an `entity` value.
`current_resource_empty`	The current transformed resource produced by assignment has no rows.	There is nothing safe to assign, and it may mean the source file did not transform correctly.	The file in `var/cache/assign_entities/transformed/[resource].csv` is empty after `check_and_assign_entities` runs.
`current_resource_no_new_entities`	The current resource does not contain any entity numbers that were not already present in the previous resource.	Unknown entity issues should normally require new lookup rows. If there are no new entities, the issue may already be resolved, the wrong resource may have been processed, or the data may have changed since the issue summary was generated.	The previous and current resources both contain entities `44000001` to `44000010`, with no additional entity numbers.
`large_number_of_new_entities`	The number of new entity numbers is greater than the percentage set by `--new-entity-threshold`. The default is 10 percent of the current resource.	A large jump can indicate that the provider changed references for existing records, changed endpoint format, or caused existing things to be treated as new entities.	A current resource has 100 entities and 40 are new, so the default 10 percent threshold is exceeded.
`duplicate_entity_all_fields`	A new entity matches an old entity when comparing all fact fields except `reference` and `entry-date`. The script builds a fingerprint from the remaining fact field and value pairs.	If all the compared facts match an existing entity, the provider may have changed the reference for the same real-world thing. Creating a new entity number could create a duplicate. If the previous entity has any compared fact field that the current entity does not have, or the current entity has any extra compared fact field, the fingerprints will not match and this check will not flag it as a duplicate.	A conservation area has the same name, organisation and geometry as an existing entity, but the provider changed the reference from `CA-001` to `CON-001`. If any compared fact field is present on the previous entity but absent from the current entity, this check will pass it as not being a duplicate.
`duplicate_prefix_reference_organisation`	A new entity has the same `prefix`, `reference` and `organisation` as an old entity.	The same dataset, provider reference and organisation should normally resolve to the existing entity number.	Existing entity `44001234` has prefix `conservation-area`, reference `CA1` and organisation `local-authority:ABC`; the new resource contains the same combination.
`duplicate_reference_organisation`	A new entity has the same `reference` and `organisation` as an old entity, regardless of prefix.	This catches duplicate provider identifiers even when another field has changed. It is broader than the prefix/reference/organisation check.	A provider republishes reference `TPO-99` for the same organisation and the script tries to create a new tree preservation order entity for it.
`duplicate_reference_organisation_in_new_resource`	Two or more entities inside the current resource have the same `reference` and `organisation`.	The script cannot know whether these rows are duplicate records for the same thing or separate things with duplicated identifiers.	The current transformed resource contains two entities from `local-authority:ABC`, both with reference `CA1`.
`missing_organisation`	A current entity is missing its `organisation` value.	Entity assignment depends on knowing who supplied the reference. Without the organisation, the lookup can be ambiguous or wrong.	The transformed rows for entity `44001234` include a `reference` field but no `organisation` field.
`missing_reference`	A current entity is missing its `reference` value.	The lookup maps provider references to entity numbers. Without the reference, future collections cannot reliably resolve the same record.	The transformed rows for entity `44001234` include `organisation` but the `reference` field is blank.
`invalid_uri_issue`	The same resource also appears in the current invalid URI issue summary.	Entity assignment may have worked, but the resource has a separate known quality issue that needs manual review before accepting the assignment.	A brownfield land resource has unknown entities and also invalid document URI values in the issue summary.
Python exception name, for example `RuntimeError`, `FileNotFoundError`, `KeyError` or another exception type	The script raised an exception outside the validation checks.	The assignment did not complete normally. The `message` column and terminal output should be used to diagnose the failure.	`RuntimeError` can be raised if a required command such as `git` or `gh` fails. `FileNotFoundError` can be raised if an expected local file is missing.

Failed resource downloads are printed in the terminal summary as Failed Downloads; they are not written as normal validation rows because the resource was never processed.

Config Evening Pipeline

Overview

The Config Evening Pipeline is a GitHub Actions workflow in the Config repository that consolidates several nightly data maintenance tasks into a single automated run. It replaces a previous set of separate workflows that were chained together by scheduled times, making the overall process more reliable and easier to reason about.

Schedule

The pipeline runs automatically at 19:00 UTC, Monday to Friday. It can also be triggered manually from the Actions tab in the Config repository.

Due to delays with GitHub actions, the pipeline will typically run at 19:45 UTC (20:45 BST).

Tasks

The pipeline runs the following jobs in order. Each job waits for the previous one to complete before starting.

Merge — Checks for an open pull request from the config-manager-update branch into main. If one exists, it is automatically approved and squash-merged. This ensures main is up to date before the remaining steps run. If no PR exists, this step completes without making any changes.
Batch assign — Runs the batch entity assignment script against the platform’s live issue data to assign entities for resources with unknown entity issues. The scope of datasets processed rotates by day of the week:
- Monday, Wednesday, Friday: ODP datasets
- Tuesday: Mandated datasets
- Thursday: Single-source datasets (processed in batches of 10 resources at a time)
This job commits its changes directly to main after each batch, rather than waiting until the end of the pipeline.
Deduplicate — Runs three scripts in sequence to retire MHCLG placeholder data:
- Deduplicates conservation area geographies (deduplicate-ca-geogs.py)
- Retires MHCLG conservation area data (retire-mhclg-ca-data.py)
- Retires MHCLG local plan and plan timetable data (retire-mhclg-plan-data.py)
If one script fails, the remaining scripts still run. All changes from this step are committed to main together in a single commit at the end of the job.
Standardise — Reorders rows and standardises line endings across all collection/ and pipeline/ CSV files. This ensures consistent formatting across the repository regardless of how changes were introduced in earlier steps. Changes are committed to main if any are detected.
Upload to S3 — Syncs the collection/ and pipeline/ directories to S3 for each deployment environment (development, staging, production). This step runs in parallel across environments and picks up all changes committed in the earlier steps.

Commits

Each stage that modifies data commits its changes to main independently rather than in one combined commit at the end. This approach avoids memory issues that can occur when staging a large number of file changes at once, particularly during batch entity assignment.

Failure behaviour

If a job fails, the subsequent jobs in the pipeline still run. For example, if the deduplicate job fails, the standardise and upload-to-S3 jobs will still execute. This means a failure in one area does not prevent the rest of the pipeline from completing.

At the end of the run, if any job has failed, a single alert is posted to the #planning-data-alerts Slack channel. The alert names each stage and its result, so it is immediately clear which part of the pipeline failed without needing to open GitHub. For batch assign failures, the alert also includes the data scope (e.g. odp, mandated, single-source) and, for single-source runs, the batch number that failed along with the exact start_batch value needed to resume the run manually.

Dry run

The pipeline can be run in dry-run mode by triggering it manually from the Actions tab and setting the dry_run input to true. In this mode, every job runs and every script executes against live data, but no changes are committed to main and nothing is synced to S3. This is useful for testing or diagnosing issues without side effects.

Test

To verify the pipeline has run correctly:

Check the Actions tab to confirm all jobs completed successfully.
Review the commits made to main that evening — there should be up to three commits (batch assign, deduplicate, standardise) if each stage had changes to make. If batch assign is running for the single-source data scope then there could be more than three commits.
Confirm any unknown entity issues resolved by batch assign are no longer present in the endpoint dataset issue summary.
If the deduplicate step ran, verify retired entities appear in pipeline/local-plan/old-entity.csv and pipeline/conservation-area/old-entity.csv as appropriate.