Initial commit: boarding pass to AirTrail pipeline
Photograph/screenshot a boarding pass, decode its BCBP barcode (PDF417/ Aztec/QR/DataMatrix), fetch the historical flight track from FlightAware the day after the flight, stage it for review, and write it into AirTrail via its REST API on approval. ntfy notifications carry signed Approve/Deny/Investigate actions.
This commit is contained in:
@@ -0,0 +1,200 @@
|
||||
# Boarding Pass → AirTrail Pipeline
|
||||
|
||||
## Overview
|
||||
|
||||
A self-hosted service that lets me photograph a boarding pass on my phone,
|
||||
stores it, and ~1 day after the flight lands automatically fetches the full
|
||||
flight track from FlightAware and stages it for my approval before writing
|
||||
into AirTrail (my self-hosted flight logbook at `airtrail.apointless.space`).
|
||||
|
||||
This is a **standalone project**, not part of Herespace. Ship it as a
|
||||
**Docker container** (docker-compose for local dev/deploy). It can still
|
||||
sit behind my existing Authelia instance via NPMplus forward-auth, but it
|
||||
is not one of the Herespace microservices and shouldn't assume that
|
||||
infrastructure beyond Authelia/NPMplus being available on the network.
|
||||
|
||||
## Conventions to follow
|
||||
|
||||
- Backend language/framework is open — pick whatever's the best fit for
|
||||
the job (barcode decoding, image handling, scheduling, small web UI).
|
||||
Python is a reasonable default given the barcode/image processing
|
||||
libraries available, but not a hard requirement.
|
||||
- Minimal/zero third-party dependencies where reasonably possible.
|
||||
- SQLite for local state, unless there's a good reason otherwise.
|
||||
- A scheduled job for the daily "check the queue" pass — in-container
|
||||
cron, a simple sleep-loop, or systemd timer on the host, whichever is
|
||||
simplest to run inside a single Docker container without extra infra.
|
||||
- Any web UI should sit behind Authelia (forward-auth via NPMplus) and use
|
||||
my `apointless.css` design system: dark-first, JetBrains Mono + DM Sans,
|
||||
blue accent `#3b82f6`, bg `#0b0d11`, surfaces `#12151b`/`#1a1e28`/`#232838`,
|
||||
semantic color tokens, light mode via `html.light` + localStorage.
|
||||
- Direct, concise code. No unnecessary abstraction layers.
|
||||
|
||||
## Pipeline stages
|
||||
|
||||
### 1. Upload page (mobile-friendly)
|
||||
|
||||
- Single page, big upload/camera button, works well on a phone browser.
|
||||
- Accepts a photo of a boarding pass.
|
||||
- Behind Authelia.
|
||||
|
||||
### 2. BCBP barcode decode
|
||||
|
||||
- Boarding passes encode data in an IATA BCBP barcode (usually PDF417).
|
||||
- Decode the barcode from the uploaded photo (e.g. via `pyzbar` +
|
||||
`pillow`, or `zxing-cpp` if pyzbar's PDF417 support is insufficient —
|
||||
check which actually decodes PDF417 reliably before committing).
|
||||
- BCBP format reference: fixed-width fields, gives you at minimum:
|
||||
- Passenger name
|
||||
- PNR / booking reference
|
||||
- Origin/destination airport codes (IATA)
|
||||
- Operating carrier + flight number
|
||||
- Julian date of flight (day-of-year, need current/implied year — BCBP
|
||||
doesn't encode year, so use "nearest future or past occurrence of
|
||||
this day-of-year relative to upload time" as the heuristic)
|
||||
- Seat number, compartment/class
|
||||
- Do NOT rely on OCR of the printed text as the primary path — the
|
||||
barcode is structured and far more reliable. OCR can be a fallback if
|
||||
barcode decode fails, but flag those for manual correction rather than
|
||||
guessing.
|
||||
- Store the decoded fields plus the original image (for manual reference)
|
||||
in a local SQLite table, e.g. `pending_flights`.
|
||||
|
||||
### 3. Queue / scheduling
|
||||
|
||||
- Each row in `pending_flights` gets a `process_after` timestamp =
|
||||
(flight date, derived from BCBP Julian date) + 1 day.
|
||||
- A systemd timer runs a daily job that queries for rows where
|
||||
`process_after <= now()` and `status = 'queued'`.
|
||||
|
||||
### 4. FlightAware track fetch
|
||||
|
||||
- **Do not use AeroAPI for this.** Its free Personal tier excludes
|
||||
Historical Flight Data, and by the time this job runs (flight date + 1
|
||||
day) the flight will likely have aged into "historical" — that tier
|
||||
boundary is not clearly documented so it's an unreliable free option
|
||||
for this specific timing.
|
||||
- Instead, replicate the manual FlightAware KML fetch we already proved
|
||||
works:
|
||||
- Build the flight history URL:
|
||||
`https://www.flightaware.com/live/flight/{ICAO_CALLSIGN}/history/{YYYYMMDD}/{HHMM}Z/{FROM_ICAO}/{TO_ICAO}/google_earth`
|
||||
- No session cookie/login required — this endpoint is publicly
|
||||
accessible.
|
||||
- This endpoint returns a `.kml` file with a `gx:Track` containing
|
||||
`<when>` (ISO timestamp) and `<gx:coord>` (lon lat elevation_m)
|
||||
elements — parse those into `[[lon, lat, elev_m], ...]` coordinate
|
||||
array and `[unix_ts, ...]` times array.
|
||||
- This is a LOW VOLUME job (one flight per day, not bulk), so no
|
||||
aggressive rate limiting is needed — but still be polite: this is
|
||||
scraping a public page, not a documented API, so keep it to one
|
||||
request per pending flight per run and don't retry-hammer on
|
||||
failure (back off and retry next day instead).
|
||||
- Converting IATA flight number (from boarding pass) to ICAO callsign
|
||||
for the URL:
|
||||
- QF → QFA, VA → VOZ, JQ → JST, EK → UAE (extend as needed)
|
||||
- Airport IATA → ICAO codes will also need a lookup (a static local
|
||||
table is fine; AirTrail's own airport data, or OurAirports CSV, both
|
||||
work as a source).
|
||||
- If the flight can't be found on FlightAware (URL 404s, wrong flight
|
||||
matched, no track available for old/low-coverage flights), mark the
|
||||
row `status = 'track_unavailable'` and still let it be reviewed
|
||||
manually with flight metadata only (no track) — don't just fail
|
||||
silently.
|
||||
|
||||
### 5. Staging (not direct DB write)
|
||||
|
||||
- Per my earlier decision: **do not write directly into AirTrail's live
|
||||
tables.** Insert into a separate staging table (in the pipeline's own
|
||||
SQLite DB, or a staging schema in AirTrail's Postgres — either is fine,
|
||||
SQLite is simpler and keeps this service decoupled) with all the
|
||||
parsed flight + track data, `status = 'pending_review'`.
|
||||
|
||||
### 6. Notification (ntfy)
|
||||
|
||||
- When a flight moves to `pending_review`, send an ntfy notification.
|
||||
- Build the notification backend as a small interface/abstraction (even
|
||||
if ntfy is the only implementation right now) so other backends could
|
||||
be added later without a rewrite — e.g. a `notify(title, body, url,
|
||||
actions)` function with a config-selected backend, not ntfy calls
|
||||
hardcoded throughout the codebase.
|
||||
- Notification content: flight number, route, date, and a short summary
|
||||
(e.g. "track found, 271 points" or "no track found"). Keep the body
|
||||
short — this is a push notification, not the review UI itself.
|
||||
- **Use ntfy action buttons** (`Actions` header) for a 3-button quick
|
||||
response, so I can act straight from the notification without opening
|
||||
the review page for the common cases:
|
||||
- **Approve** — an HTTP action button that calls back into the service
|
||||
(e.g. `POST /flights/{id}/approve`) to write the flight straight into
|
||||
AirTrail. No further UI interaction needed.
|
||||
- **Deny** — an HTTP action button that calls `POST /flights/{id}/reject`,
|
||||
marks the row `status = 'rejected'`.
|
||||
- **Investigate** — a `view` action button that opens the review page
|
||||
for that specific flight in the browser (see stage 7). This is the
|
||||
path for anything that needs a closer look — wrong flight matched,
|
||||
missing track, boarding pass fields decoded incorrectly, etc.
|
||||
- Ntfy action button docs: use `Actions:` header with comma-separated
|
||||
action definitions (`http, Approve, https://.../approve, method=POST`
|
||||
etc. — check current ntfy docs for exact syntax since it's changed
|
||||
across versions).
|
||||
- Approve/Deny should require some form of auth on the callback (e.g. a
|
||||
per-flight signed token in the URL) since ntfy action buttons fire an
|
||||
unauthenticated HTTP request from wherever the notification is
|
||||
received — don't just trust a bare flight ID.
|
||||
- ntfy config (topic, server URL — self-hosted vs ntfy.sh, auth token if
|
||||
self-hosted) lives in the settings page (see below), not hardcoded.
|
||||
|
||||
### 7. Review page
|
||||
|
||||
- Lists flights with `status = 'pending_review'` (reachable from the
|
||||
service's own nav, and directly via the ntfy "Investigate" action
|
||||
button which deep-links to a specific flight's row on this page).
|
||||
- For each: show the decoded boarding pass fields, matched flight route,
|
||||
and — if a track was found — a small map preview of the track (a
|
||||
static Leaflet/MapLibre map is fine, doesn't need to match AirTrail's
|
||||
map exactly).
|
||||
- **Editable fields**: every decoded/matched field (flight number, date,
|
||||
origin/destination, carrier, etc.) should be editable inline here —
|
||||
this is the "investigate" workflow. If the BCBP decode got something
|
||||
wrong (e.g. wrong year inferred, misread airport code), I fix it here.
|
||||
- **Re-query button**: after editing fields, a button to re-run the
|
||||
FlightAware lookup (stage 4) using the corrected data, replacing
|
||||
whatever track/match was previously staged for this row. This lets me
|
||||
fix a bad initial match and pull the correct track without restarting
|
||||
the whole pipeline from the boarding pass photo.
|
||||
- Approve button: on confirm, writes the flight into AirTrail's `flight`
|
||||
table (or updates an existing flight if one already matches — check
|
||||
by flight number + date first, since I may have already manually
|
||||
entered some flights) and the track into AirTrail's track table.
|
||||
- AirTrail Postgres track table stores: `flightId`, `coordinates`
|
||||
(JSON array of `[lon, lat, elev_m]`), `times` (JSON array of unix
|
||||
timestamps), `sourceFormat` (string, e.g. `"kml"`), `sourceName`
|
||||
(string, original filename/identifier). Confirm exact column names
|
||||
and types against the live schema before writing
|
||||
(`\d flight_track` or equivalent) — don't assume the names I've
|
||||
given are byte-exact.
|
||||
- Deny/reject button: mark `status = 'rejected'`, keep the row for
|
||||
reference rather than hard-deleting.
|
||||
- Also behind Authelia.
|
||||
- This page is also what the ntfy Approve/Deny action buttons act on
|
||||
server-side (same underlying approve/reject logic, just triggered
|
||||
without opening this page).
|
||||
|
||||
### 8. Settings page
|
||||
|
||||
- Notification backend selector (dropdown, ntfy is the only real option
|
||||
today but the UI should not assume it always will be).
|
||||
- ntfy-specific fields: server URL (default `https://ntfy.sh` but
|
||||
overridable for self-hosting), topic name, optional auth token.
|
||||
- Save to local config (SQLite table or simple JSON/YAML file — your
|
||||
call, whichever fits the existing service patterns better).
|
||||
|
||||
## Open questions to resolve during the build (not blocking, use judgment)
|
||||
|
||||
- Exact AirTrail Postgres schema for the flight and track tables (get
|
||||
this from `\d flight` / `\d flight_track` on the live DB rather than
|
||||
trusting anything written here from memory).
|
||||
- Whether pyzbar or an alternative PDF417 decoder actually handles real
|
||||
phone-photo boarding passes reliably (glare, skew, compression) —
|
||||
test against a few real captures early rather than assuming.
|
||||
- BCBP year disambiguation heuristic — pick something sane (nearest
|
||||
occurrence to now) and revisit if it causes mismatches in practice.
|
||||
Reference in New Issue
Block a user