boarding-pass/boarding-pass-pipeline-brief.md

# Boarding Pass → AirTrail Pipeline

## Overview

A self-hosted service that lets me photograph a boarding pass on my phone,
stores it, and ~1 day after the flight lands automatically fetches the full
flight track from FlightAware and stages it for my approval before writing
into AirTrail (my self-hosted flight logbook at `airtrail.apointless.space`).

This is a **standalone project**, not part of Herespace. Ship it as a
**Docker container** (docker-compose for local dev/deploy). It can still
sit behind my existing Authelia instance via NPMplus forward-auth, but it
is not one of the Herespace microservices and shouldn't assume that
infrastructure beyond Authelia/NPMplus being available on the network.

## Conventions to follow

- Backend language/framework is open — pick whatever's the best fit for
  the job (barcode decoding, image handling, scheduling, small web UI).
  Python is a reasonable default given the barcode/image processing
  libraries available, but not a hard requirement.
- Minimal/zero third-party dependencies where reasonably possible.
- SQLite for local state, unless there's a good reason otherwise.
- A scheduled job for the daily "check the queue" pass — in-container
  cron, a simple sleep-loop, or systemd timer on the host, whichever is
  simplest to run inside a single Docker container without extra infra.
- Any web UI should sit behind Authelia (forward-auth via NPMplus) and use
  my `apointless.css` design system: dark-first, JetBrains Mono + DM Sans,
  blue accent `#3b82f6`, bg `#0b0d11`, surfaces `#12151b`/`#1a1e28`/`#232838`,
  semantic color tokens, light mode via `html.light` + localStorage.
- Direct, concise code. No unnecessary abstraction layers.

## Pipeline stages

### 1. Upload page (mobile-friendly)

- Single page, big upload/camera button, works well on a phone browser.
- Accepts a photo of a boarding pass.
- Behind Authelia.

### 2. BCBP barcode decode

- Boarding passes encode data in an IATA BCBP barcode (usually PDF417).
- Decode the barcode from the uploaded photo (e.g. via `pyzbar` +
  `pillow`, or `zxing-cpp` if pyzbar's PDF417 support is insufficient —
  check which actually decodes PDF417 reliably before committing).
- BCBP format reference: fixed-width fields, gives you at minimum:
  - Passenger name
  - PNR / booking reference
  - Origin/destination airport codes (IATA)
  - Operating carrier + flight number
  - Julian date of flight (day-of-year, need current/implied year — BCBP
    doesn't encode year, so use "nearest future or past occurrence of
    this day-of-year relative to upload time" as the heuristic)
  - Seat number, compartment/class
- Do NOT rely on OCR of the printed text as the primary path — the
  barcode is structured and far more reliable. OCR can be a fallback if
  barcode decode fails, but flag those for manual correction rather than
  guessing.
- Store the decoded fields plus the original image (for manual reference)
  in a local SQLite table, e.g. `pending_flights`.

### 3. Queue / scheduling

- Each row in `pending_flights` gets a `process_after` timestamp =
  (flight date, derived from BCBP Julian date) + 1 day.
- A systemd timer runs a daily job that queries for rows where
  `process_after <= now()` and `status = 'queued'`.

### 4. FlightAware track fetch

- **Do not use AeroAPI for this.** Its free Personal tier excludes
  Historical Flight Data, and by the time this job runs (flight date + 1
  day) the flight will likely have aged into "historical" — that tier
  boundary is not clearly documented so it's an unreliable free option
  for this specific timing.
- Instead, replicate the manual FlightAware KML fetch we already proved
  works:
  - Build the flight history URL:
    `https://www.flightaware.com/live/flight/{ICAO_CALLSIGN}/history/{YYYYMMDD}/{HHMM}Z/{FROM_ICAO}/{TO_ICAO}/google_earth`
  - No session cookie/login required — this endpoint is publicly
    accessible.
  - This endpoint returns a `.kml` file with a `gx:Track` containing
    `<when>` (ISO timestamp) and `<gx:coord>` (lon lat elevation_m)
    elements — parse those into `[[lon, lat, elev_m], ...]` coordinate
    array and `[unix_ts, ...]` times array.
  - This is a LOW VOLUME job (one flight per day, not bulk), so no
    aggressive rate limiting is needed — but still be polite: this is
    scraping a public page, not a documented API, so keep it to one
    request per pending flight per run and don't retry-hammer on
    failure (back off and retry next day instead).
  - Converting IATA flight number (from boarding pass) to ICAO callsign
    for the URL:
    - QF → QFA, VA → VOZ, JQ → JST, EK → UAE (extend as needed)
  - Airport IATA → ICAO codes will also need a lookup (a static local
    table is fine; AirTrail's own airport data, or OurAirports CSV, both
    work as a source).
  - If the flight can't be found on FlightAware (URL 404s, wrong flight
    matched, no track available for old/low-coverage flights), mark the
    row `status = 'track_unavailable'` and still let it be reviewed
    manually with flight metadata only (no track) — don't just fail
    silently.

### 5. Staging (not direct DB write)

- Per my earlier decision: **do not write directly into AirTrail's live
  tables.** Insert into a separate staging table (in the pipeline's own
  SQLite DB, or a staging schema in AirTrail's Postgres — either is fine,
  SQLite is simpler and keeps this service decoupled) with all the
  parsed flight + track data, `status = 'pending_review'`.

### 6. Notification (ntfy)

- When a flight moves to `pending_review`, send an ntfy notification.
- Build the notification backend as a small interface/abstraction (even
  if ntfy is the only implementation right now) so other backends could
  be added later without a rewrite — e.g. a `notify(title, body, url,
  actions)` function with a config-selected backend, not ntfy calls
  hardcoded throughout the codebase.
- Notification content: flight number, route, date, and a short summary
  (e.g. "track found, 271 points" or "no track found"). Keep the body
  short — this is a push notification, not the review UI itself.
- **Use ntfy action buttons** (`Actions` header) for a 3-button quick
  response, so I can act straight from the notification without opening
  the review page for the common cases:
  - **Approve** — an HTTP action button that calls back into the service
    (e.g. `POST /flights/{id}/approve`) to write the flight straight into
    AirTrail. No further UI interaction needed.
  - **Deny** — an HTTP action button that calls `POST /flights/{id}/reject`,
    marks the row `status = 'rejected'`.
  - **Investigate** — a `view` action button that opens the review page
    for that specific flight in the browser (see stage 7). This is the
    path for anything that needs a closer look — wrong flight matched,
    missing track, boarding pass fields decoded incorrectly, etc.
  - Ntfy action button docs: use `Actions:` header with comma-separated
    action definitions (`http, Approve, https://.../approve, method=POST`
    etc. — check current ntfy docs for exact syntax since it's changed
    across versions).
  - Approve/Deny should require some form of auth on the callback (e.g. a
    per-flight signed token in the URL) since ntfy action buttons fire an
    unauthenticated HTTP request from wherever the notification is
    received — don't just trust a bare flight ID.
- ntfy config (topic, server URL — self-hosted vs ntfy.sh, auth token if
  self-hosted) lives in the settings page (see below), not hardcoded.

### 7. Review page

- Lists flights with `status = 'pending_review'` (reachable from the
  service's own nav, and directly via the ntfy "Investigate" action
  button which deep-links to a specific flight's row on this page).
- For each: show the decoded boarding pass fields, matched flight route,
  and — if a track was found — a small map preview of the track (a
  static Leaflet/MapLibre map is fine, doesn't need to match AirTrail's
  map exactly).
- **Editable fields**: every decoded/matched field (flight number, date,
  origin/destination, carrier, etc.) should be editable inline here —
  this is the "investigate" workflow. If the BCBP decode got something
  wrong (e.g. wrong year inferred, misread airport code), I fix it here.
- **Re-query button**: after editing fields, a button to re-run the
  FlightAware lookup (stage 4) using the corrected data, replacing
  whatever track/match was previously staged for this row. This lets me
  fix a bad initial match and pull the correct track without restarting
  the whole pipeline from the boarding pass photo.
- Approve button: on confirm, writes the flight into AirTrail's `flight`
  table (or updates an existing flight if one already matches — check
  by flight number + date first, since I may have already manually
  entered some flights) and the track into AirTrail's track table.
  - AirTrail Postgres track table stores: `flightId`, `coordinates`
    (JSON array of `[lon, lat, elev_m]`), `times` (JSON array of unix
    timestamps), `sourceFormat` (string, e.g. `"kml"`), `sourceName`
    (string, original filename/identifier). Confirm exact column names
    and types against the live schema before writing
    (`\d flight_track` or equivalent) — don't assume the names I've
    given are byte-exact.
  - Deny/reject button: mark `status = 'rejected'`, keep the row for
    reference rather than hard-deleting.
- Also behind Authelia.
- This page is also what the ntfy Approve/Deny action buttons act on
  server-side (same underlying approve/reject logic, just triggered
  without opening this page).

### 8. Settings page

- Notification backend selector (dropdown, ntfy is the only real option
  today but the UI should not assume it always will be).
- ntfy-specific fields: server URL (default `https://ntfy.sh` but
  overridable for self-hosting), topic name, optional auth token.
- Save to local config (SQLite table or simple JSON/YAML file — your
  call, whichever fits the existing service patterns better).

## Open questions to resolve during the build (not blocking, use judgment)

- Exact AirTrail Postgres schema for the flight and track tables (get
  this from `\d flight` / `\d flight_track` on the live DB rather than
  trusting anything written here from memory).
- Whether pyzbar or an alternative PDF417 decoder actually handles real
  phone-photo boarding passes reliably (glare, skew, compression) —
  test against a few real captures early rather than assuming.
- BCBP year disambiguation heuristic — pick something sane (nearest
  occurrence to now) and revisit if it causes mismatches in practice.