Provision Quality

What is provision quality?

Provision quality describes who provides data for each dataset on the platform, and how good that data is. It answers questions like: how many organisations feed a dataset, are they the authoritative source for their data or not, and which datasets or organisations have gaps.

It is produced by a nightly pipeline task (provision-quality) that writes three related tables as Delta Lake tables (Parquet) into the platform’s parquet-datasets S3 bucket — the same output pattern as the task table. From there they are surfaced to the serving database so the digital-land.info front-end can show a provider/quality breakdown on each dataset page.

Two lenses: providers and owners

The core subtlety is that an organisation can relate to a dataset in two different ways, and they don’t always line up:

Provider — an organisation that submits data. It has an active endpoint feeding the dataset (a non-empty endpoint in source / source_pipeline). “Who feeds data in.”
Owner — an organisation that owns entities on the platform, recorded by the organisation_entity field on each entity. “Who owns the records that are actually there.”

These usually overlap, but the mismatches are exactly what this work makes visible. For example, when a national body does a bulk upload (“seeding”) of conservation areas on behalf of many councils:

the provider is the national body (it holds the endpoint), but
the owners are the individual councils (the entities are attributed to them).

Conversely, a council may own entities (loaded historically or seeded on its behalf) while no longer submitting anything itself.

Authoritative vs some

Within a dataset, each entity carries a quality value that tells us whether the data came from the right place:

authoritative — the data is from the organisation that is the authoritative source for it (a council’s own conservation-area data, owned by that council).
some — the data is present but not from the authoritative source (for example, seeded by another body).

An organisation’s provision quality for a dataset is authoritative if it owns at least one authoritative entity, otherwise some.

The three tables

Everything derives from a single base grain — provision = one (dataset, organisation) pair — which is then rolled up two ways.

1. `provision_quality` (base)

One row per (dataset, organisation). A row exists for any organisation that has an active endpoint or owns entities for the dataset — i.e. both lenses combined. Nothing is dropped; flag columns record what each row is, so downstream consumers can filter (e.g. to only formally-designated providers) as needed.

Column	Description
`dataset`	The dataset (e.g. `conservation-area`)
`organisation`	The organisation reference (e.g. `local-authority:ADU`)
`organisation_name`	Human-readable organisation name
`has_active_endpoint`	`true` if the org submits data via a non-empty endpoint (provider lens)
`owns_entities`	`true` if the org owns at least one entity (owner lens)
`is_designated_provider`	`true` if the org has an allocated entity range in `entity-organisation.csv` for this dataset — i.e. this `(dataset, org)` is a formally designated provision
`quality`	`authoritative` or `some`
`entity_count`	Number of entities owned by the org (or seeded on its behalf, for seeders with no directly-owned entities)
`quality_score`	Reserved — empty in v1

The is_designated_provider flag is what distinguishes formally-provisioned data from data that is simply present in the system (Owen’s “endpoints/data that doesn’t belong to a provision”). Both are retained; the flag lets the front-end include or exclude the latter without recomputation.

2. `dataset_quality` (rollup per dataset)

One row per dataset, aggregated from provision_quality.

Column	Description
`dataset`	The dataset
`authoritative_organisations`	Count of orgs classified `authoritative` for this dataset
`some_organisations`	Count of orgs classified `some`
`total_organisations`	Total contributing organisations
`total_entities`	Total entities in the dataset
`quality_score`	Reserved — empty in v1

This is the natural extension of the existing “N data providers” figure shown on dataset pages today (the publisher_count on dataset_publication). The provider-vs-owner split is available by filtering provision_quality on the flag columns; dataset_quality gives the headline counts.

3. `organisation_quality` (rollup per organisation)

One row per organisation, aggregated from provision_quality across all datasets. This view did not exist in the original analysis.

Column	Description
`organisation`	The organisation reference
`organisation_name`	Human-readable organisation name
`authoritative_datasets`	Number of datasets the org provides authoritatively
`some_datasets`	Number of datasets the org provides at `some` quality
`total_datasets`	Total datasets the org contributes to
`total_entities_owned`	Total entities owned across all datasets
`quality_score`	Reserved — empty in v1

How the classification works

The classification logic is reused from the original Data Design analysis. In outline, per dataset:

Providers are the organisations with a non-empty endpoint in source ⋈ source_pipeline.
Owners and quality come from each dataset’s entity table — organisation_entity (who owns each entity) and quality (authoritative / some).
Designated provisions come from entity-organisation.csv — the organisations allocated an entity range for the dataset.
Seeder detection uses lookup.csv: an active organisation that seeded entities it does not own (and is not the designated provider) is credited as a some-quality contributor. To avoid crediting dissolved/reorganised bodies with stale lookup entries, local-authority-type organisations must have seeded entities for more than one owner to count; other org types need at least one.
Active organisations only — organisations with an end_date in the organisation table are excluded from some-quality seeder classification.

Deviation from the original analysis (deliberate): the prototype dropped two categories of row — designated providers whose data has stalled (some-only with an allocated range), and some providers with zero seeded entities. In line with the “keep all data, flag it” approach, provision_quality retains these rows and distinguishes them via the flag columns and entity_count, rather than removing them. Consumers filter as needed.

Where it runs

provision-quality is a new task in the build-digital-land-builder DAG, running nightly.

It runs in parallel with build-digital-land-builder. All of its inputs live in the collection-data S3 bucket (see Data sources), independent of the built digital-land.sqlite3, so it does not need to wait for the build. Like assemble-tasks, it can hang off configure-dag and run alongside both the build and the digital-land-postgres-loader. This keeps it off the critical path to the 8am publish deadline.
The single task writes both outputs itself — the Delta tables to S3 and the rows to serving Postgres — rather than splitting the Postgres load into a separate DAG step. Everything the pipeline does for these tables lives in one place, so it can be followed through end to end.
It is implemented in Python using Polars. At this data scale plain Python is sufficient;
The CloudFront cache must be invalidated after this task completes

Data sources

The productionised task reads everything from the collection-data S3 bucket — it does not hit Datasette or GitHub-raw at runtime (avoiding both the Datasette load and the GitHub rate-limiting seen in overnight runs), and it does not depend on the built digital-land.sqlite3.

Data	Source
Provider mapping — `endpoint`, `organisation`, dataset(s), `end_date`	Each collection’s `collection/source.csv`, globbed across `s3://{env}-collection-data/*-collection/collection/source.csv`. The `pipelines` column carries the dataset(s) an endpoint serves — the same information as Datasette’s `source` ⋈ `source_pipeline`.
`organisation` — names, entity IDs, active/inactive (`end_date`)	`s3://{env}-collection-data/organisation-collection/dataset/organisation.csv`
`entity` (with `organisation_entity`, `quality`), per dataset	The per-dataset `{dataset}.sqlite3` in `s3://{env}-collection-data/{collection}-collection/dataset/`
`entity-organisation.csv`, `lookup.csv`, per collection	`s3://{env}-collection-data/config/pipeline/{collection}/`

Because all of these are per-collection outputs already present in collection-data when this DAG runs, the task has no dependency on the build step — hence it can run in parallel with it.

Entity source — sqlite for v1, Delta/parquet later. The entity + quality read is the task’s dominant cost (the big datasets — title-boundary, listed-building — have millions of entities), and the per-dataset sqlites carry geometry inline, so the whole file is downloaded even though only three columns are needed. A columnar entity table would be far cheaper. For v1 we read the sqlites, because they are the only source that currently exists for every dataset — the parquet-datasets bucket holds entity tables for just the handful of datasets already migrated to the PySpark pipeline. Once the collection pipelines are fully on PySpark and emit per-dataset Delta tables (dataset=<name>/-partitioned), the entity read switches to those. To keep that switch cheap, the entity read is isolated behind a single loader function so it can be swapped without touching the classification logic.

Output

The task writes both of its outputs itself, in the one task:

Delta Lake tables (Parquet) to the platform’s Parquet datasets bucket — s3://{env}-parquet-datasets/{table}/, one each for provision_quality, dataset_quality, and organisation_quality. This mirrors how the task table is produced by the assemble-tasks pipeline: the Delta table is the canonical, re-queryable artifact. This is the primary output.
Serving Postgres — the same three tables are written to the serving database, alongside dataset_publication, so digital-land.info can serve them on dataset pages. The total_organisations figure in dataset_quality should reconcile with the existing publisher_count on dataset_publication.

Both writes happen inside the provision-quality task rather than being split across separate DAG steps, so everything the pipeline does for these tables can be read in one place.

Cache invalidation

The dataset pages that serve these tables sit behind CloudFront, so the cache must be invalidated once the new data is written. So invalidate-cloudfront-cache task will run immediatly after provision-quality task to ensure this data is available even if it takes longer than build-digital-land-builder

Worked example: conservation area

Using the seeding scenario above, three representative rows in provision_quality for conservation-area:

organisation	has_active_endpoint	owns_entities	is_designated_provider	quality	entity_count
Adur DC	✅	✅	✅	authoritative	50
Lewes DC	❌	✅	✅	some	30
Historic England	✅	❌	❌	some	3

Adur submits its own data and owns authoritative entities — the healthy case.
Lewes never submitted, but owns 30 entities seeded on its behalf, so some quality.
Historic England holds the endpoint and seeded data, but owns no entities itself — a provider that is not an owner.

dataset_quality for conservation-area then rolls these (and all other orgs) up into authoritative / some / total organisation counts; organisation_quality rolls the same rows up per organisation across every dataset each org touches.