> ## Documentation Index > Fetch the complete documentation index at: https://docs.a7manager.se/llms.txt > Use this file to discover all available pages before exploring further. # How ingestion works > Daily CSV import pipeline: delivery, validation, and the rules that apply to every file. Operational data arrives as CSV files dropped into blob storage. Azure Data Factory picks them up, loads them into staging tables, and promotes them into production through three stored procedures per entity. Nothing reaches a production table without passing validation. ```mermaid theme={null} flowchart LR P[Producer] -->|SAS| L["landing/"] L --> ADF[Data Factory] ADF --> S[(staging table)] S --> V[validate] V -->|all rows pass| D[dedup] D --> U[upsert] U --> PROD[(production tables)] V -->|any row fails| E["error/"] U --> A["archive/"] ``` ## Delivery | | | | --------------- | -------------------------------------------------------------------- | | **Destination** | Container `ingest-{project-uuid}`, folder `landing/` | | **File name** | `{entity}_{timestamp}.csv` — e.g. `defects_2026-08-01T10-08-04Z.csv` | | **Encoding** | UTF-8 | | **Delimiter** | `,` — quote character `"`, escape character `\` | | **Header row** | Required. Column names must match exactly, in the documented order | | **Empty value** | An empty field means NULL. Do not send the literal text `NULL` | | **Timestamps** | ISO 8601 with zone — `2026-08-01T10:08:04Z` | | **Dates** | `YYYY-MM-DD` | | **Booleans** | `True` / `False` | | **Decimals** | `.` as the decimal separator | The `{entity}` prefix is how the pipeline decides which file it is looking at, so it must be exactly one of the eight documented names. The timestamp portion only has to make the filename unique, but it must not introduce underscore-separated words that change how the prefix reads. There are **no network prerequisites** — no static egress IP to register, no firewall rule. Authorisation is the SAS token issued at project onboarding, which grants create, write and list but deliberately not read or delete. **Load order matters.** `vehicle_configurations` must be processed first, then the other files. Defects and measurements cross-check against configuration data, so a configuration event and a row referencing it must not arrive in the wrong order. Loading readings before configurations corrupts the recorded component mileage when a component is swapped in the same batch. ## Rules that apply to every file ### Validation is all-or-nothing If any row fails, **the entire file is rejected**. It is moved to `error/` with a message naming every failing row, and no production table is touched. A file that passes is moved to `archive/`. Row numbers in error messages are 1-based over data rows, excluding the header: ``` Validation failed: Row 12: vehicle_number 'X61099' not found in project Row 27: missing or empty performance_km ``` Fix the source data and re-deliver the whole file. Re-importing rows that already landed is safe: they are skipped on the staleness check described below. ### `external_system_id` as the key `external_system_id` is the source system's own identifier for the row. It is both the deduplication key and the update key: re-sending the same identifier updates the existing record rather than creating a second one. **It must be stable over time.** The one exception is `vehicle_readings`, which is keyed on the vehicle and reading date instead — see that page. ### `last_updated` decides whether an update applies `last_updated` is the source system's last-modified timestamp for the row. It is stored as `external_updated_at`, and an incoming row is applied only when its `last_updated` is **strictly newer** than the stored value. Equal or older rows are silently skipped — not reported as errors, just counted. Two details worth knowing: * A row whose stored `external_updated_at` is empty is **always** overwritten by an import, regardless of timestamps. * If the incoming `last_updated` is empty but the stored value is not, the row is skipped. **Bump `last_updated` whenever the row changes**, or your correction will be silently discarded. ### Duplicates within one file are allowed If the same `external_system_id` appears more than once in a file, the row with the highest `last_updated` wins. Ties are broken by position — the **last** occurrence in the file wins. A row with an empty `last_updated` loses to any row that has one. ### References are sent as human-readable codes Rows refer to other data by code, never by database identifier: `vehicle_number`, `component_serial_number`, product group and slot codes, `task_external_system_id`. Every code must already exist **in the project the container belongs to**. An unresolvable code rejects the file, with two exceptions: `product_group_code` and `slot_position` on defects, which import as empty with a warning. Note that free-text classification fields — `action_code`, `cause_code`, `severity`, `category`, `fault_indication_code` and similar — are **not** validated against any list. They pass through as written. ## Correcting and closing records Two entities close an open period by **re-sending the original row** rather than sending a new one — `vehicle_configurations` (a dismount closes a mount) and `vehicle_out_of_service` (returning to service closes a period). In both cases you re-send the same `external_system_id` with the closing fields filled in and a newer `last_updated`. Because updates overwrite column by column, re-sending a closed record with the closing field **empty** and a newer `last_updated` will re-open it. This is a real way to accidentally undo history — make sure your exporter carries closing fields forward. ## The files Faults recorded against a vehicle or component. Daily absolute odometer snapshots. Mount and dismount events. Load this one first. Maintenance work that was performed. Condition measurements. The most strictly validated file. Periods where a vehicle was unavailable. Cleaning and toilet emptying events. Maintenance programmes. Not a daily file — read the warnings. Seven of these are delivered daily. `maintenance_plans` is sent only when a plan revision changes, and it behaves differently enough to deserve careful reading.