boarding-pass/boarding-pass-pipeline-brief.md

Boarding Pass → AirTrail Pipeline

Overview

A self-hosted service that lets me photograph a boarding pass on my phone, stores it, and ~1 day after the flight lands automatically fetches the full flight track from FlightAware and stages it for my approval before writing into AirTrail (my self-hosted flight logbook at airtrail.apointless.space).

This is a standalone project, not part of Herespace. Ship it as a Docker container (docker-compose for local dev/deploy). It can still sit behind my existing Authelia instance via NPMplus forward-auth, but it is not one of the Herespace microservices and shouldn't assume that infrastructure beyond Authelia/NPMplus being available on the network.

Conventions to follow

• Backend language/framework is open — pick whatever's the best fit for the job (barcode decoding, image handling, scheduling, small web UI). Python is a reasonable default given the barcode/image processing libraries available, but not a hard requirement.
• Minimal/zero third-party dependencies where reasonably possible.
• SQLite for local state, unless there's a good reason otherwise.
• A scheduled job for the daily "check the queue" pass — in-container cron, a simple sleep-loop, or systemd timer on the host, whichever is simplest to run inside a single Docker container without extra infra.
• Any web UI should sit behind Authelia (forward-auth via NPMplus) and use my apointless.css design system: dark-first, JetBrains Mono + DM Sans, blue accent #3b82f6, bg #0b0d11, surfaces #12151b/#1a1e28/#232838, semantic color tokens, light mode via html.light + localStorage.
• Direct, concise code. No unnecessary abstraction layers.

Pipeline stages

1. Upload page (mobile-friendly)

• Single page, big upload/camera button, works well on a phone browser.
• Accepts a photo of a boarding pass.
• Behind Authelia.

2. BCBP barcode decode

• Boarding passes encode data in an IATA BCBP barcode (usually PDF417).
• Decode the barcode from the uploaded photo (e.g. via pyzbar + pillow, or zxing-cpp if pyzbar's PDF417 support is insufficient — check which actually decodes PDF417 reliably before committing).
• BCBP format reference: fixed-width fields, gives you at minimum:
  • Passenger name
  • PNR / booking reference
  • Origin/destination airport codes (IATA)
  • Operating carrier + flight number
  • Julian date of flight (day-of-year, need current/implied year — BCBP doesn't encode year, so use "nearest future or past occurrence of this day-of-year relative to upload time" as the heuristic)
  • Seat number, compartment/class
• Do NOT rely on OCR of the printed text as the primary path — the barcode is structured and far more reliable. OCR can be a fallback if barcode decode fails, but flag those for manual correction rather than guessing.
• Store the decoded fields plus the original image (for manual reference) in a local SQLite table, e.g. pending_flights.

3. Queue / scheduling

• Each row in pending_flights gets a process_after timestamp = (flight date, derived from BCBP Julian date) + 1 day.
• A systemd timer runs a daily job that queries for rows where process_after <= now() and status = 'queued'.

4. FlightAware track fetch

• Do not use AeroAPI for this. Its free Personal tier excludes Historical Flight Data, and by the time this job runs (flight date + 1 day) the flight will likely have aged into "historical" — that tier boundary is not clearly documented so it's an unreliable free option for this specific timing.
• Instead, replicate the manual FlightAware KML fetch we already proved works:
  • Build the flight history URL: https://www.flightaware.com/live/flight/{ICAO_CALLSIGN}/history/{YYYYMMDD}/{HHMM}Z/{FROM_ICAO}/{TO_ICAO}/google_earth
  • No session cookie/login required — this endpoint is publicly accessible.
  • This endpoint returns a .kml file with a gx:Track containing <when> (ISO timestamp) and <gx:coord> (lon lat elevation_m) elements — parse those into [[lon, lat, elev_m], ...] coordinate array and [unix_ts, ...] times array.
  • This is a LOW VOLUME job (one flight per day, not bulk), so no aggressive rate limiting is needed — but still be polite: this is scraping a public page, not a documented API, so keep it to one request per pending flight per run and don't retry-hammer on failure (back off and retry next day instead).
  • Converting IATA flight number (from boarding pass) to ICAO callsign for the URL:
    • QF → QFA, VA → VOZ, JQ → JST, EK → UAE (extend as needed)
  • Airport IATA → ICAO codes will also need a lookup (a static local table is fine; AirTrail's own airport data, or OurAirports CSV, both work as a source).
  • If the flight can't be found on FlightAware (URL 404s, wrong flight matched, no track available for old/low-coverage flights), mark the row status = 'track_unavailable' and still let it be reviewed manually with flight metadata only (no track) — don't just fail silently.

5. Staging (not direct DB write)

• Per my earlier decision: do not write directly into AirTrail's live tables. Insert into a separate staging table (in the pipeline's own SQLite DB, or a staging schema in AirTrail's Postgres — either is fine, SQLite is simpler and keeps this service decoupled) with all the parsed flight + track data, status = 'pending_review'.

6. Notification (ntfy)

• When a flight moves to pending_review, send an ntfy notification.
• Build the notification backend as a small interface/abstraction (even if ntfy is the only implementation right now) so other backends could be added later without a rewrite — e.g. a notify(title, body, url, actions) function with a config-selected backend, not ntfy calls hardcoded throughout the codebase.
• Notification content: flight number, route, date, and a short summary (e.g. "track found, 271 points" or "no track found"). Keep the body short — this is a push notification, not the review UI itself.
• Use ntfy action buttons (Actions header) for a 3-button quick response, so I can act straight from the notification without opening the review page for the common cases:
  • Approve — an HTTP action button that calls back into the service (e.g. POST /flights/{id}/approve) to write the flight straight into AirTrail. No further UI interaction needed.
  • Deny — an HTTP action button that calls POST /flights/{id}/reject, marks the row status = 'rejected'.
  • Investigate — a view action button that opens the review page for that specific flight in the browser (see stage 7). This is the path for anything that needs a closer look — wrong flight matched, missing track, boarding pass fields decoded incorrectly, etc.
  • Ntfy action button docs: use Actions: header with comma-separated action definitions (http, Approve, https://.../approve, method=POST etc. — check current ntfy docs for exact syntax since it's changed across versions).
  • Approve/Deny should require some form of auth on the callback (e.g. a per-flight signed token in the URL) since ntfy action buttons fire an unauthenticated HTTP request from wherever the notification is received — don't just trust a bare flight ID.
• ntfy config (topic, server URL — self-hosted vs ntfy.sh, auth token if self-hosted) lives in the settings page (see below), not hardcoded.

7. Review page

• Lists flights with status = 'pending_review' (reachable from the service's own nav, and directly via the ntfy "Investigate" action button which deep-links to a specific flight's row on this page).
• For each: show the decoded boarding pass fields, matched flight route, and — if a track was found — a small map preview of the track (a static Leaflet/MapLibre map is fine, doesn't need to match AirTrail's map exactly).
• Editable fields: every decoded/matched field (flight number, date, origin/destination, carrier, etc.) should be editable inline here — this is the "investigate" workflow. If the BCBP decode got something wrong (e.g. wrong year inferred, misread airport code), I fix it here.
• Re-query button: after editing fields, a button to re-run the FlightAware lookup (stage 4) using the corrected data, replacing whatever track/match was previously staged for this row. This lets me fix a bad initial match and pull the correct track without restarting the whole pipeline from the boarding pass photo.
• Approve button: on confirm, writes the flight into AirTrail's flight table (or updates an existing flight if one already matches — check by flight number + date first, since I may have already manually entered some flights) and the track into AirTrail's track table.
  • AirTrail Postgres track table stores: flightId, coordinates (JSON array of [lon, lat, elev_m]), times (JSON array of unix timestamps), sourceFormat (string, e.g. "kml"), sourceName (string, original filename/identifier). Confirm exact column names and types against the live schema before writing (\d flight_track or equivalent) — don't assume the names I've given are byte-exact.
  • Deny/reject button: mark status = 'rejected', keep the row for reference rather than hard-deleting.
• Also behind Authelia.
• This page is also what the ntfy Approve/Deny action buttons act on server-side (same underlying approve/reject logic, just triggered without opening this page).

8. Settings page

• Notification backend selector (dropdown, ntfy is the only real option today but the UI should not assume it always will be).
• ntfy-specific fields: server URL (default https://ntfy.sh but overridable for self-hosting), topic name, optional auth token.
• Save to local config (SQLite table or simple JSON/YAML file — your call, whichever fits the existing service patterns better).

Open questions to resolve during the build (not blocking, use judgment)

• Exact AirTrail Postgres schema for the flight and track tables (get this from \d flight / \d flight_track on the live DB rather than trusting anything written here from memory).
• Whether pyzbar or an alternative PDF417 decoder actually handles real phone-photo boarding passes reliably (glare, skew, compression) — test against a few real captures early rather than assuming.
• BCBP year disambiguation heuristic — pick something sane (nearest occurrence to now) and revisit if it causes mismatches in practice.