Review data quality

Purpose

The LPA-facing dashboards. They show an organisation what data it has provided, what the platform has done with it, and what is wrong — issues, endpoint errors and failed expectations — with a route into checking a corrected file from the get-started page.

Unlike check and provide, nothing here is a wizard: every page is a middleware chain that fetches, transforms and renders.

Used by LPAs monitoring their own data and by the data team investigating a specific organisation or dataset. The pages are public — there is no sign-in — so any organisation's data can be viewed by anyone with the URL.

Trigger and entry points

Router: routes/organisations.js. Chains are re-exported through OrganisationsController.js.

router.use('/:lpa', validateOrg) runs before every organisation-scoped route.

Route order matters. The :issue_type/:issue_field patterns are greedy — every specific route (overview, get-started, data, endpoint-error, expectation) must stay above them in the router, and /:lpa/:dataset and /:lpa must stay at the bottom. Adding a route in the wrong place silently routes it to the issue table.

Code flow

The main path is organisations list → LPA overview → dataset task list → issue table → issue details. Dataset overview, dataview, endpoint errors and expectation pages hang off the task list.

Chain shape

Every page is a default-exported array in src/middleware/, and they all follow the same shape:

validate params → fetch org/dataset → composites → paginate → prepare templateParams → render → logPageError

Read the array itself for the exact order — it is a plain ordered list and is always current. The table below covers what the arrays do not tell you: which composites each chain pulls in, and where a chain deviates from the shape.

Composite names are abbreviated above — see Shared composites.

Entity vs entry

The distinction to hold onto. An entity issue attaches to a record that made it onto the platform, so the chain fetches entities and filters them by issue. An entry issue attaches to a row in the provided resource that never became an entity, so the chain goes to the resource and its metadata instead. issueTable redirects to the entity view when the issue type has entities.

Shared composites

Order matters: processRelevantIssuesMiddlewares assumes entities are already on req, and processSpecificationMiddlewares must run before anything builds a table.

Note that removeIssuesThatHaveBeenFixed is commented out of processRelevantIssuesMiddlewares — it was too slow, and the problem it addressed had been seen for only one organisation. Issue counts therefore include issues the provider may have already fixed. The function and its unit tests still exist but nothing calls them.

Data in, transformations, out

In — datasette for resources, issues, sources and specification; the platform API for entities, tasks and authority status.

Through — resources → entities → issues, with helpers in getVerboseColumns.js, table.js, entities.js and pagination.js.

Out — templateParams, validated against routes/schemas.js outside production and staging, then rendered by nunjucks.

Tasks and expectations

prepareTasks turns issue counts into the task list. Two things to know:

• If authority === 'some' the whole list is replaced by a single "Provide authoritative data" task linking to get-started — non-authoritative datasets never show individual issues.
• Task wording comes from performanceDbApi.getTaskMessage, which needs a row count. For issue types in SPECIAL_ISSUE_TYPES (currently reference values are not unique) the denominator is the resource's entry_count rather than the entity count, because the issue is about rows, not entities. entityOutOfBoundsMessage phrases the out-of-bounds case.

Out-of-bounds expectations are added as tasks only when expectationOutOfBoundsTask is enabled; expectations and expectationFetcher in common.middleware.js fetch them.

Validation and errors

• validateOrg on every /:lpa route — unknown organisation gives a 404.
• validateQueryParams(schema) per chain, with schemas in routes/schemas.js — invalid params give a 400.
• show404IfPageNumberNotInRange for out-of-range pagination.
• fetchOne 404s by default when no rows come back; overridden per call with FetchOneFallbackPolicy.
• logPageError terminates most chains.

Datasette queries

Query builders in performanceDbApi.js:

Inline SQL also lives in fetchOne/fetchMany call sites in common.middleware.js — fetchResources, fetchSources, fetchEntryIssues, fetchEntityIssueCounts, fetchEndpointSummary.

Reference data (organisations, sources, specification) comes from the digital-land database; issues and resources come from the per-dataset database, passed as the second argument to runQuery.

Entity data comes from the platform API rather than datasette because the API applies authority filtering and returns what is actually published; issues come from datasette because they are pipeline output and are not exposed by the API. dataview uses both, which is why it has the onlyIf(entityCount === undefined, fetchEntityCount) fallback — the API count is preferred, with datasette as a backstop.

Testing

Because these chains set templateParams and schema validation is active in test, a mock that is missing a field fails the test rather than rendering a broken page. Update the schema in routes/schemas.js alongside the chain.

See architecture.md for the general test setup.