aes67-ESP32-P4/docs/aes67-core-base.md

# AES67 core base — reusable across projects

The shared parts that every AES67 device project should reuse without changes: web UI core, config schema, PTP, AES67 TX, SDP/SAP, network/VLAN, syslog, health/temperature. Project-specific parts (sources, player, GPIO, etc.) plug in around it.
Reference implementation: the ESP32-P4 streamer (see hardware-and-design-notes.md).
Reference devices for UI and defaults: Riedel Bolero (PTP status), Riedel Director AES67 4-wire port settings, and Riedel Director SIC AES67 card PTP settings.

## Web UI (web/index.html)
- Single file, no dependencies, minimal CSS. Embed in firmware (EMBED_TXTFILES, gzip optional).
- CORE sections: Status (incl. temperatures), PTP status (Riedel-style + Details), PTP, AES67 output, Network (+VLAN), Logging, SDP, Firmware.
- PROJECT blocks are marked `PROJECT START/END` in the HTML (live controls and config fieldsets) and in the script (`PROJECT` object: `title`, `def` defaults merged over core defaults, `toggle(f)`, `statusRows(st)`, `poll()`).
- Binding: every `<input|select name="group.key">` maps to `cfg[group][key]`. Adding a setting means adding an input and a default; no other JS needed. Number inputs and numeric selects are sent as JSON numbers.
- Status table values can be text or {cls: ok|warn|bad, text} for colour.
- Device config is merged over defaults on load, so older firmware missing keys still renders.
- Save bumps `aes67.session_ver` (SDP o= version). Save is blocked while any field is invalid (e.g. multicast out of range).
- UI uses IEEE 1588-2019 terms (TimeTransmitter / TimeReceiver), like Riedel. Config/API values stay master/slave/auto.
- AES67 fields carry Riedel Director-style "Default / Range" hints.
- PTP preset buttons: "Riedel SIC defaults" and "AES67 media profile defaults" fill the form (Save to apply).

## Config schema (core groups)
- ptp: {mode: multicast|hybrid, role: slave|auto|master, domain, priority1, priority2, log_sync, log_announce, announce_timeout, log_delay_req, dscp, hw_ts}
- aes67: {name, enabled, discovery: manual|sap, mcast, port, ttl, dscp, channels, mono_sum, encoding: L16|L24, rate, ptime, pt, ssrc, clk_offset, session_id, session_ver}
- net: {hostname, vlan, web_on: both|inet|aes67, dhcp, ip, mask, gw, dns}   (AES67 / untagged interface)
- inet: {vlan_id, pcp, dhcp, ip, mask, gw, dns}                            (internet / tagged, only if net.vlan)
- log: {syslog, host, port, level: error|warn|info|debug, facility, format: rfc5424|rfc3164}

## Core API
- GET/POST /api/config (full JSON; POST validates, stores to NVS, applies live where possible, returns 4xx with message on bad values)
- GET /api/status -> {model?, fw?, power?, ip, aes67_ip, inet_ip, inet_vlan, mac, link, tx_packets, underruns, uptime_s, heap_free, psram_free, temps:[…], ptp:{…}, …project fields}
  - model, fw, power are optional; rows only appear when present (power e.g. "PoE" / "USB" if the board can detect it).
- GET /stream.sdp, POST /api/reboot, POST /api/log/test
- Firmware: GET /api/ota, POST /api/ota, POST /api/ota/confirm, POST /api/ota/rollback (see Firmware update)

## Temperatures
- `status.temps` = array of {name, c, min_c, max_c, warn_c}; min/max since boot, warn_c optional. Any number of sensors; the UI renders one row per sensor and marks HIGH (red) at or above warn_c.
- ESP32-P4: on-die sensor via the `temperature_sensor` driver (driver/temperature_sensor.h). Pick the measurement range to suit (e.g. -10..80 °C), sample every ~2 s in the health task, and report it as "SoC". It measures the die, not ambient: expect it to read well above room temperature.
- Boards may add external sensors (e.g. I2C) by appending to the same array; the project or board component registers them.
- Log a warning (goes to syslog) when crossing warn_c, with hysteresis (e.g. clear at warn_c - 5).

## PTP
- IEEE 1588-2008 ordinary clock: UDP/IPv4, 224.0.1.129 (event 319, general 320), E2E delay, two-step OK.
- Modes (as on Riedel SIC):
  - multicast: all messages multicast.
  - hybrid: Sync/Follow_Up/Announce multicast; Delay_Req/Delay_Resp unicast to/from the GM's IP (learned from its Announce/Sync source address). As transmitter, answer unicast Delay_Req with unicast Delay_Resp, and multicast Delay_Req with multicast.
- Riedel SIC AES67 card defaults (Director): mode multicast, role TimeReceiver, domain 0, priority1 128, priority2 126, announce interval 1, sync interval 0, announce receipt timeout 3, delay request interval 0.
- AES67 media profile defaults: announce 1, sync -3, timeout 3, delay req -3 (ranges: sync -4..1, announce 0..4).
- Core defaults: Riedel SIC intervals (announce 1, sync 0, timeout 3, delay req 0), mode multicast, DSCP 46, plus role auto with fallback priorities below.
- As TimeReceiver, the GM sets the actual Sync rate, and the Delay_Resp logMessageInterval sets the minimum Delay_Req rate. The local intervals apply when this device is the TimeTransmitter.
- Roles / BMCA dataset:
  - slave (TimeReceiver only): clockClass 255, never transmits time.
  - auto (default): clockClass 248, priority1 250, priority2 250. Loses to any real GM (Riedel default 128); takes over only if none.
  - master (preferred TimeTransmitter): clockClass 248, priority1 100 (UI suggestion, editable).
  - clockAccuracy 0xFE (unknown), offsetScaledLogVariance 0xFFFF, timeSource 0xA0 (internal oscillator).
  - clockIdentity = EUI-64 from MAC (xx-xx-xx-FF-FE-xx-xx-xx).
  - Note: Riedel Bolero reports its own clock class as 228.
- When the GM changes (including this device becoming GM): update SDP ts-refclk, bump session_ver, send SAP immediately.
- Hardware timestamping preferred (ESP32-P4 EMAC has IEEE 1588 support); software fallback flagged in status.
- ESP32-P4 implementation (aes67_ptp):
  - IDF's `examples/ethernet/ptp` (ptpd) is Layer 2 only, so it is not used. IDF's EMAC clock is used: `ETH_MAC_ESP_CMD_PTP_ENABLE`, then the IPv4/UDP snapshot bit (`emac_ll_ts_ptp_ip4_enable`), which IDF leaves off.
  - RX timestamps: a hook on the driver's info input path (`esp_eth_update_input_path_info`) records port-319 event timestamps and forwards every frame to lwIP. TX: Delay_Req is a raw Eth/IPv4/UDP frame sent with `esp_eth_transmit_ctrl_vargs` for its HW timestamp. Needs `CONFIG_ETH_TRANSMIT_MUTEX`.
  - Servo: step on the first Sync from a new GM (frequency seeded from the measured rate), then linuxptp-style PI (kp 0.7 / ki 0.3 at 1 Sync/s, scaled by the interval); re-step above 1 ms. Locked: 8 Syncs below 1 µs; unlocked after 3 above. GM loss keeps the frequency (holdover).
  - Path delay is corrected for offset drift between t2 and t3 until the clock is syntonised; delay statistics start at lock.
  - Measured vs ptp4l (Intel i210 GM, non-PTP switch): lock in ~35 s cold, ~22 s after GM loss; offset within a few hundred ns; board crystal -39.8 ppm. Link asymmetry (1G GM / 100M board through a store-and-forward switch) adds a constant offset error of a few µs that no receiver can see.

### PTP status (`status.ptp`) — main panel modelled on Riedel Bolero "PTP Status"
| UI row | field(s) | notes |
|---|---|---|
| PTP State | state | INITIALIZING/LISTENING/UNCALIBRATED/SLAVE/MASTER/PASSIVE/FAULTY/DISABLED, shown as TimeReceiver/TimeTransmitter/… |
| Lock State | locked | servo converged (e.g. \|offset\| < 1 µs for N consecutive syncs) |
| TimeTransmitter | gm_id | shown as MAC + "(FFFE)" when EUI-64 from MAC |
| Time Offset | offset_ns | Bolero brackets in ns: <100 (green), 100-500 (green), 500-1000 (amber), 1000-10000 (red), >10000 (red); exact value in brackets |
| Frequency Deviation | freq_ppb | servo frequency correction; buckets <100 ppb, 100-500 ppb, 500 ppb-1 ppm, 1-10 ppm, 10-50 ppm, >50 ppm |
| Network Delay | path_delay_ns ± path_delay_sd_ns | mean path delay ± std dev over the rolling window |
| Hops | steps_removed | |
| Time/Frequency Traceable | gm_time_traceable, gm_freq_traceable | Announce flagField bits |
| Version | version | 2 |
| Own Clock Class | own_class | 255 / 248 per role |

Details (collapsed): clock_id, gm_class, gm_accuracy, gm_p1, gm_p2, sync_avg_ms/min/max + sync_jitter_us (measured Sync interval over the last `window` = 64 messages; RX as receiver, TX as transmitter), announce_avg_ms, delay_req/delay_resp counters, hw_ts.
When this device is the GM, offset/frequency/delay show "–".

## AES67 TX
- Defaults match Riedel Director 4-wire AES67 output: port 5004 (1024–65535), L24, ptime 1.000 ms, payload type 96 (96–127), SSRC 0, time stamp offset 0 (32-bit). Channels: Riedel default 1; this core defaults to 2, and projects override.
- Mono sum (`mono_sum`): stream goes out as 1 channel carrying (ch1 + ch2) / 2. Done in the TX path after gain, in 32-bit (int32 or float) before converting to L16/L24, so it can't clip; -6 dB relative to a single channel at full scale (identical L/R content comes out at the same level). The UI forces channels = 1 while ticked, and the SDP follows (rtpmap .../1, i=mono). Unticking restores the previous channel count (or 2).
- RTP multicast range 224.0.2.0 – 239.255.255.255 (validated in UI and firmware). This excludes 224.0.0.x/224.0.1.x, where PTP lives.
- SSRC is used literally (0 allowed, as on Riedel). RFC 3550 recommends random; not required for AES67 multicast.
- RTP ts = (PTP time * rate + clk_offset) mod 2^32 (`a=mediaclk:direct=<clk_offset>`). Riedel's "Time Stamp Offset" is the same value. Packets paced on PTP time; the audio source is pulled from a ring buffer through a callback, so a sender needs no ASRC.
- Discovery: manual (SDP export only) or SAP. NMOS IS-04/05 is a possible future option (Riedel SIC cards have an NMOS tab).
- SDP: v, o (aes67 IP, session_id, session_ver), s, c=mcast/ttl, t=0 0, a=clock-domain:PTPv2 <dom>, m=audio port RTP/AVP pt, a=rtpmap, a=recvonly, a=ptime, a=ts-refclk:ptp=IEEE1588-2008:<gm>:<dom>, a=mediaclk:direct.
- SAP to 239.255.255.255:9875 every 30 s, plus immediately on change; deletion packet on disable or shutdown.
- Media 2 (ST 2022-7 redundancy): not possible on single-port boards like the ESP32-P4-ETH. Keep the schema open for a `mcast2`/`port2` pair on dual-NIC hardware (Riedel applies one PTP config to both Media 1 and Media 2).

## AES67 RX (future, for receiver projects) — Riedel Director 4-wire input defaults
- Import SDP (parse c=, m=, rtpmap, ptime, ts-refclk, mediaclk) or manual: multicast IP, port, optional sender IP (source filter, IGMPv3 SSM), channels, bit depth, ptime, payload type, SSRC, time stamp offset.
- Play mode "synchron" (playout locked to PTP/RTP timestamps). Receive buffer default 8 × ptime (8.000 ms at 1 ms), min 3 × ptime, max 150 ms.
- Channel selection: which channel of the stream feeds a mono output.

## Network
- Default: one untagged interface. VLAN split option: AES67 untagged + internet tagged (inet.vlan_id/pcp) using the ESP-IDF vlan_support approach (examples/network/vlan_support, ESP32-P4 supported). AES67 side has no gateway; default route and DNS are on inet.
- PTP, RTP, SAP and IGMP bind to the AES67 netif. HTTP server filters on `web_on`.
- DHCP sends net.hostname (option 12), so the router's lease list / local DNS shows it; after a rename it applies at the next lease renewal.
- mDNS (espressif/mdns): `<net.hostname>.local` plus the web UI as `_http._tcp` port 80. A hostname change applies live (the new name answers after ~1 s of RFC 6762 probing).
- LLDP (IEEE 802.1AB), transmit only, own code (not in IDF/lwIP): every 30 s, TTL 120 s, plus immediately on a new IP or hostname. TLVs: chassis ID and port ID = MAC, port description "eth0", system name = net.hostname, system description = project + firmware version, capabilities station-only, management address = IPv4. With the VLAN split, send it on the AES67 (untagged) side; the 802.1 port VLAN ID TLV can be added then.
- Switch port: AES67 as native/untagged VLAN, internet tagged, PoE on. IGMP snooping + querier on the AES67 VLAN.

## Syslog
- esp_log vprintf hook (chains to UART) -> queue -> low-priority UDP task. Non-blocking, drop and count on overflow, no recursion. RFC 5424 or 3164, PRI = facility*8 + severity (E3 W4 I6 D7 V7). Strip ANSI colour codes. HOSTNAME = net.hostname, APP-NAME = log tag.
- Buffer early boot logs in the queue until the netif has an IP. POST /api/log/test sends one info-level message.

## Firmware update (OTA)
- Partition table: nvs, otadata, phy_init, ota_0, ota_1 (no factory app). Size the slots from flash_id; e.g. on 16 MB, 2 × 6 MB leaves room for growth (cspot + TLS + codecs). Keep NVS outside the app slots so config survives updates.
- Version: `PROJECT_VER` from `git describe --tags --dirty`, read at runtime from `esp_app_get_description()`; also reported as `status.fw`.
- API:
  - GET /api/ota -> {version, project, build_date, idf, running, previous, previous_version, pending_verify, can_rollback}
  - POST /api/ota: raw .bin as `application/octet-stream`, streamed with esp_ota_begin/write/end into the inactive slot (never buffered whole in RAM). 200 then reboot; 4xx with a message if rejected.
  - Upload only, by design: the device never pulls firmware from a URL or git server (no esp_https_ota, no update checks).
  - POST /api/ota/confirm: mark the running app valid. POST /api/ota/rollback: boot the previous slot.
- Reject before writing (checked on the first chunk): image magic, chip ID, wrong `project_name` in the image's app description (stops flashing another project's .bin), and a chip revision outside the image's min/max range (matters on the rev < 3 boards). esp_ota_end then verifies the whole image. Reject downgrades only if a flag is set (off by default; the flag doesn't exist yet).
- `previous`/`previous_version` are only reported when that slot is bootable (`esp_ota_check_rollback_is_possible()`), so a half-written upload or a rolled-back image is never offered.
- Rollback: enable `CONFIG_BOOTLOADER_APP_ROLLBACK_ENABLE`. New firmware boots in "pending verify". Self-test after boot (60 s limit): Ethernet has an IP and the web server answers (an HTTP GET of its own /api/status returns 200; optionally PTP locked within N seconds later). Test-only `CONFIG_AES67_OTA_SELFTEST_FORCE_FAIL` (sdkconfig.selftest_fail) forces a failure to test rollback. Pass: `esp_ota_mark_app_valid_cancel_rollback()` automatically. Fail or crash before that: reboot, and the bootloader rolls back to the previous slot. The UI also offers manual confirm/rollback while pending.
- Keep AES67 TX running during the upload if possible (OTA writes are slow flash erases; run them at lower priority than PTP/TX). Expect short audio glitches; the reboot drops the stream for a few seconds.
- Security: no auth yet (LAN only), like the rest of the API. Add a bearer token and optionally signed images (secure boot v2 / signed OTA) before exposing the device on an untrusted network.
- UI: Firmware section: upload with progress bar, waits for the reboot and reports the new version (or that it rolled back), confirm/rollback buttons while pending.

## Proposed ESP-IDF component layout (to be built)
```
components/
  aes67_board/     board pin profiles (waveshare_esp32p4_eth.h, …), eth init
  aes67_net/       netif setup, optional VLAN split, DSCP/PCP helpers
  aes67_ptp/       PTPv2 ordinary clock: BMCA, servo, master, multicast/hybrid, EMAC HW timestamps, stats
  aes67_tx/        RTP sender task, PTP-paced, pull callback: size_t (*read)(int32_t *buf, size_t frames); optional mono sum
  aes67_sdp_sap/   SDP builder/parser + SAP announcer
  aes67_syslog/    remote syslog
  aes67_health/    uptime, heap/PSRAM, temperature sensors (temps[] with min/max/warn)
  aes67_ota/       firmware upload, image checks, rollback self-test
  aes67_web/       httpd, embedded index.html, config store (cJSON in NVS), core routes
main/              project: sources, player, extra routes/config groups
```
- Config registry: each component calls `cfg_register(group, defaults_json, validate_cb, apply_cb)`. The project registers its own groups (e.g. "source") the same way. /api/config is built from the registry.
- Status registry: `status_register(cb)`; each callback adds fields to the /api/status cJSON. Temperature sensors: `health_temp_register(name, read_cb, warn_c)`.
- Route registry: the project adds routes (e.g. /api/player/*) via `web_register_uri()`.
- Future: aes67_rx (receiver, see RX section) for receiver projects.
- Consider moving the core components and web/index.html into a separate git repo (submodule or IDF component manager) once they're stable.