Skip to main content
Operational data arrives as CSV files dropped into blob storage. Azure Data Factory picks them up, loads them into staging tables, and promotes them into production through three stored procedures per entity. Nothing reaches a production table without passing validation.

​
Delivery

The {entity} prefix is how the pipeline decides which file it is looking at, so it must be exactly one of the eight documented names. The timestamp portion only has to make the filename unique, but it must not introduce underscore-separated words that change how the prefix reads. There are no network prerequisites — no static egress IP to register, no firewall rule. Authorisation is the SAS token issued at project onboarding, which grants create, write and list but deliberately not read or delete.
Load order matters. vehicle_configurations must be processed first, then the other files. Defects and measurements cross-check against configuration data, so a configuration event and a row referencing it must not arrive in the wrong order. Loading readings before configurations corrupts the recorded component mileage when a component is swapped in the same batch.

​
Rules that apply to every file

​
Validation is all-or-nothing

If any row fails, the entire file is rejected. It is moved to error/ with a message naming every failing row, and no production table is touched. A file that passes is moved to archive/. Row numbers in error messages are 1-based over data rows, excluding the header:
Fix the source data and re-deliver the whole file. Re-importing rows that already landed is safe: they are skipped on the staleness check described below.

​
external_system_id as the key

external_system_id is the source system’s own identifier for the row. It is both the deduplication key and the update key: re-sending the same identifier updates the existing record rather than creating a second one. It must be stable over time. The one exception is vehicle_readings, which is keyed on the vehicle and reading date instead — see that page.

​
last_updated decides whether an update applies

last_updated is the source system’s last-modified timestamp for the row. It is stored as external_updated_at, and an incoming row is applied only when its last_updated is strictly newer than the stored value. Equal or older rows are silently skipped — not reported as errors, just counted. Two details worth knowing:
  • A row whose stored external_updated_at is empty is always overwritten by an import, regardless of timestamps.
  • If the incoming last_updated is empty but the stored value is not, the row is skipped.
Bump last_updated whenever the row changes, or your correction will be silently discarded.

​
Duplicates within one file are allowed

If the same external_system_id appears more than once in a file, the row with the highest last_updated wins. Ties are broken by position — the last occurrence in the file wins. A row with an empty last_updated loses to any row that has one.

​
References are sent as human-readable codes

Rows refer to other data by code, never by database identifier: vehicle_number, component_serial_number, product group and slot codes, task_external_system_id. Every code must already exist in the project the container belongs to. An unresolvable code rejects the file, with two exceptions: product_group_code and slot_position on defects, which import as empty with a warning. Note that free-text classification fields — action_code, cause_code, severity, category, fault_indication_code and similar — are not validated against any list. They pass through as written.

​
Correcting and closing records

Two entities close an open period by re-sending the original row rather than sending a new one — vehicle_configurations (a dismount closes a mount) and vehicle_out_of_service (returning to service closes a period). In both cases you re-send the same external_system_id with the closing fields filled in and a newer last_updated.
Because updates overwrite column by column, re-sending a closed record with the closing field empty and a newer last_updated will re-open it. This is a real way to accidentally undo history — make sure your exporter carries closing fields forward.

​
The files

defects

Faults recorded against a vehicle or component.

vehicle_readings

Daily absolute odometer snapshots.

vehicle_configurations

Mount and dismount events. Load this one first.

maintenance_records

Maintenance work that was performed.

measurements

Condition measurements. The most strictly validated file.

vehicle_out_of_service

Periods where a vehicle was unavailable.

cleaning_records

Cleaning and toilet emptying events.

maintenance_plans

Maintenance programmes. Not a daily file — read the warnings.
Seven of these are delivered daily. maintenance_plans is sent only when a plan revision changes, and it behaves differently enough to deserve careful reading.