Check data

Purpose

Checking is validation without commitment. A user points the service at a URL or uploads a file, the data is validated against the dataset specification, and they get a results page telling them what is wrong. Nothing is added to the platform, and the user is under no obligation to go further.

Providing is the separate, opt-in step that follows a successful URL check — see submit-data.

Two audiences use it differently: people iterating on a file until it is clean, who may never go further, and people preparing an endpoint to provide, for whom the check is the gate. The journey is the same; only the ending differs.

It is open to anyone — no sign-in — so a check can come from an LPA, a consultant, or a member of the public.

Trigger and entry points

Wizard definition: steps.js, fields.js, index.js (wizard name check-wizard).

Code flow

Points that are not obvious from the table, and should be expanded on:

• /dataset sets requiresGeometryTypeSelection in session, which drives the conditional next.
• /url and /upload both store the returned id as the request_id wizard field.
• /results runs updateSessionFromRequestData, which sets request_id and — for check_url requests — upload-method: 'url'. This is what makes a shared results link work: the session is correct whether the user walked the journey or arrived at the URL cold.
• The organisation name in the results banner is resolved by the orgIdToName filter from the org code in the async request params, not from session.
• The LPA boundary map uses requestParams.organisationName (the org code from the async request) to build the boundary GeoJSON URL, falling back to deepLink.orgId.
• /confirmation is the only place the submit handover is set up: it reads upload-method, and if it is 'url' it writes req.session.checkRequestId (raw session, shared across wizards) and renders the "Provide your data" button. If upload-method is anything else, or the session has gone stale, the button is not rendered and the user has to re-run the check.

Async request lifecycle

1. submitUrlController / uploadFileController posts to the async request backend and stores the returned id as request_id.
2. /check/status/:id polls GET /api/status/:result_id until the request is complete.
3. The status endpoint also computes showColumnMapping and columnMappingUrl via columnMappingDecider once the request finishes.
4. On completion the user is sent to /check/results/:id/1.

Polling is client-side, in assets/js/statusPage.js: a 3 second interval with a cap on attempts, against the endpoint passed to the template as pollingEndpoint. Statuses are PENDING/PROCESSING, then COMPLETE or FAILED — both are "finished", so the page must check isFailed as well as isComplete. A failed request goes to the results page, which renders the error rather than a table.

statusController also resolves the dataset from the platform API and runs processSpecificationMiddlewares in middlewareSetup, so the specification is available when the results page is reached.

Column mapping

Offered when the user's columns could plausibly be mapped to expected fields that are currently unmapped. shouldShowColumnMapping returns false if any of these hold: the request failed, the dataset is statutory, there are blocking non-mapping issues (quality criteria level 2, external responsibility), the user already supplied a column mapping, every expected field is mapped, or there are no spare uploaded columns to map from.

Geometry and point are treated as interchangeable — if one is mapped, the other is not counted as missing.

The step lets the user assign their columns to fields (or IGNORE), and resubmits as a new request carrying column_mapping in its params. That is why hasUserColumnMapping suppresses a second offer: a mapped request never asks again.

Data in, transformations, out

In — either a URL, or a file received by multer and stored in S3. Allowed types come from allowedFileTypes in utils/utils.js (CSV, GeoJSON, GML, GeoPackage); the size limit is config.validations.maxFileSize (100MB).

Through — the async request backend does the actual validation. This service only holds the request id and reads results back through RequestData/ResponseDetails.

Out — the results pages (tables, issue details, maps), a shareable results URL, and for URL checks the session values the submit wizard needs.

Validation and errors

Three layers, in order:

1. Field validation — fields.js and utils/validators.js; messages resolved by validationMessageLookup and rendered through toErrorList.
2. Pre-flight checks — SubmitUrlController.localUrlValidation and the file checks in UploadFileController, run before the async request is created.
3. Check results — issues found by the backend, displayed on the results pages.

File checks run in UploadFileController before the request is created: fileMimeTypeIsValid against allowedFileTypes, and fileMimeTypeMatchesExtension, which catches a mismatch between extension and declared type (application/octet-stream is allowed through). Name rules cover length, illegal characters and double extensions. Everything else surfaces as a MiddlewareError error page.

URL failure modes

SubmitUrlController.localUrlValidation makes a HEAD request before the async request is created. Anything it does not catch surfaces later as a failed request on the status or results page, usually with a much less helpful message — so knowing which layer catches what is the first thing to establish when triaging a report.

Caught up front, in submitUrlController.js:

A 405 Method Not Allowed on the HEAD request skips all post-checks and lets the request through — some servers reject HEAD but serve GET fine. Same for a network error on the HEAD.

Caught later, by the backend:

The front-end content-type check is deliberately disabled — the filetype validator in localUrlValidation is commented out because it rejected plugin URLs such as ArcGIS. Do not re-enable it without solving that case.

Not caught anywhere:

To reproduce a report, run the HEAD the service runs, then the GET it does not:

curl -I -A "$(grep -A1 userAgent config/default.yaml)" '<url>'   # what localUrlValidation sees
curl -sL '<url>' | head -40                                       # what actually comes back

Send the same User-Agent — config.checkService.userAgent — or you may get a different answer from the server than the service does. A JS challenge is obvious in the second command: HTML with an inline script rather than data.

Triage. Each failure path emits a Sentry counter — start there rather than in the logs:

• url_submission.validation_failure (attribute: failure_type)
• url_submission.head_request_error (attribute: reason — network_error or method_not_allowed)
• url_submission.async_request_failure (attributes: error_code, response_status)
• url_submission.begun / url_submission.accepted for the denominator

Datasette queries

The dataset list offered on the /check/dataset step is not hardcoded — it is derived from the platform's provision data, grouped by collection and cached. getDatasets() in utils/utils.js is the entry point, and also backs deep-link validation. See how the dataset list is built for the derivation and its fallbacks — including why a dataset can disappear from the list entirely.

Testing

Run against a stub backend with npm run mock:api, or NODE_ENV=wiremock for pre-baked responses — useful for reproducing a specific results payload without a real check.

See architecture.md for the general test setup.