aes67-ESP32-P4/docs/aes67-core-base.md

AES67 core base — reusable across projects

The shared parts that every AES67 device project should reuse without changes: web UI core, config schema, PTP, AES67 TX, SDP/SAP, network/VLAN, syslog, health/temperature. Project-specific parts (sources, player, GPIO, etc.) plug in around it. Reference implementation: the ESP32-P4 streamer (see hardware-and-design-notes.md). Reference devices for UI and defaults: Riedel Bolero (PTP status), Riedel Director AES67 4-wire port settings, and Riedel Director SIC AES67 card PTP settings.

Web UI (web/index.html)

• Single file, no dependencies, minimal CSS. Embed in firmware (EMBED_TXTFILES, gzip optional).
• CORE sections: Status (incl. temperatures), PTP status (Riedel-style + Details), PTP, AES67 output, Network (+VLAN), Logging, SDP, Firmware.
• PROJECT blocks are marked PROJECT START/END in the HTML (live controls and config fieldsets) and in the script (PROJECT object: title, def defaults merged over core defaults, toggle(f), statusRows(st), poll()).
• Binding: every <input|select name="group.key"> maps to cfg[group][key]. Adding a setting means adding an input and a default; no other JS needed. Number inputs and numeric selects are sent as JSON numbers.
• Status table values can be text or {cls: ok|warn|bad, text} for colour.
• Device config is merged over defaults on load, so older firmware missing keys still renders.
• Save bumps aes67.session_ver (SDP o= version). Save is blocked while any field is invalid (e.g. multicast out of range).
• UI uses IEEE 1588-2019 terms (TimeTransmitter / TimeReceiver), like Riedel. Config/API values stay master/slave/auto.
• AES67 fields carry Riedel Director-style "Default / Range" hints.
• PTP preset buttons: "Riedel SIC defaults" and "AES67 media profile defaults" fill the form (Save to apply).

Config schema (core groups)

• ptp: {mode: multicast|hybrid, role: slave|auto|master, domain, priority1, priority2, log_sync, log_announce, announce_timeout, log_delay_req, dscp, hw_ts}
• aes67: {name, enabled, discovery: manual|sap, mcast, port, ttl, dscp, channels, mono_sum, encoding: L16|L24, rate, ptime, pt, ssrc, clk_offset, session_id, session_ver}
• net: {hostname, vlan, web_on: both|inet|aes67, dhcp, ip, mask, gw, dns} (AES67 / untagged interface)
• inet: {vlan_id, pcp, dhcp, ip, mask, gw, dns} (internet / tagged, only if net.vlan)
• log: {syslog, host, port, level: error|warn|info|debug, facility, format: rfc5424|rfc3164}

Core API

• GET/POST /api/config (full JSON; POST validates, stores to NVS, applies live where possible, returns 4xx with message on bad values)
• GET /api/status -> {model?, fw?, power?, ip, aes67_ip, inet_ip, inet_vlan, mac, link, tx_packets, underruns, uptime_s, heap_free, psram_free, temps:[…], ptp:{…}, …project fields}
  • model, fw, power are optional; rows only appear when present (power e.g. "PoE" / "USB" if the board can detect it).
• GET /stream.sdp, POST /api/reboot, POST /api/log/test
• Firmware: GET /api/ota, POST /api/ota, POST /api/ota/confirm, POST /api/ota/rollback (see Firmware update)

Temperatures

• status.temps = array of {name, c, min_c, max_c, warn_c}; min/max since boot, warn_c optional. Any number of sensors; the UI renders one row per sensor and marks HIGH (red) at or above warn_c.
• ESP32-P4: on-die sensor via the temperature_sensor driver (driver/temperature_sensor.h). Pick the measurement range to suit (e.g. -10..80 °C), sample every ~2 s in the health task, and report it as "SoC". It measures the die, not ambient: expect it to read well above room temperature.
  • Implemented with range 20..100 °C (±2 °C; the warning level must lie inside the range), warn_c 85 °C. On the test board the die reads ~27 °C at idle in a cool room.
• Boards may add external sensors (e.g. I2C) by appending to the same array; the project or board component registers them.
• Log a warning (goes to syslog) when crossing warn_c, with hysteresis (e.g. clear at warn_c - 5).

PTP

• IEEE 1588-2008 ordinary clock: UDP/IPv4, 224.0.1.129 (event 319, general 320), E2E delay, two-step OK.
• Modes (as on Riedel SIC):
  • multicast: all messages multicast.
  • hybrid: Sync/Follow_Up/Announce multicast; Delay_Req/Delay_Resp unicast to/from the GM's IP (learned from its Announce/Sync source address). As transmitter, answer unicast Delay_Req with unicast Delay_Resp, and multicast Delay_Req with multicast.
• Riedel SIC AES67 card defaults (Director): mode multicast, role TimeReceiver, domain 0, priority1 128, priority2 126, announce interval 1, sync interval 0, announce receipt timeout 3, delay request interval 0.
• AES67 media profile defaults: announce 1, sync -3, timeout 3, delay req -3 (ranges: sync -4..1, announce 0..4).
• Core defaults: Riedel SIC intervals (announce 1, sync 0, timeout 3, delay req 0), mode multicast, DSCP 46, plus role auto with fallback priorities below.
• As TimeReceiver, the GM sets the actual Sync rate, and the Delay_Resp logMessageInterval sets the minimum Delay_Req rate. The local intervals apply when this device is the TimeTransmitter.
• Roles / BMCA dataset:
  • slave (TimeReceiver only): clockClass 255, never transmits time.
  • auto (default): clockClass 248, priority1 250, priority2 250. Loses to any real GM (Riedel default 128); takes over only if none.
  • master (preferred TimeTransmitter): clockClass 248, priority1 100 (UI suggestion, editable).
  • clockAccuracy 0xFE (unknown), offsetScaledLogVariance 0xFFFF, timeSource 0xA0 (internal oscillator).
  • clockIdentity = EUI-64 from MAC (xx-xx-xx-FF-FE-xx-xx-xx).
  • Note: Riedel Bolero reports its own clock class as 228.
• When the GM changes (including this device becoming GM): update SDP ts-refclk, bump session_ver, send SAP immediately.
• Hardware timestamping preferred (ESP32-P4 EMAC has IEEE 1588 support); software fallback flagged in status.
• ESP32-P4 implementation (aes67_ptp):
  • IDF's examples/ethernet/ptp (ptpd) is Layer 2 only, so it is not used. IDF's EMAC clock is used: ETH_MAC_ESP_CMD_PTP_ENABLE, then the IPv4/UDP snapshot bit (emac_ll_ts_ptp_ip4_enable), which IDF leaves off.
  • RX timestamps: a hook on the driver's info input path (esp_eth_update_input_path_info) records port-319 event timestamps and forwards every frame to lwIP. TX: Delay_Req is a raw Eth/IPv4/UDP frame sent with esp_eth_transmit_ctrl_vargs for its HW timestamp. Needs CONFIG_ETH_TRANSMIT_MUTEX.
  • Servo: step on the first Sync from a new GM (frequency seeded from the measured rate), then linuxptp-style PI (kp 0.7 / ki 0.3 at 1 Sync/s, scaled by the interval); re-step above 1 ms. Locked: 8 Syncs below 1 µs; unlocked after 3 above. GM loss keeps the frequency (holdover).
  • Path delay is corrected for offset drift between t2 and t3 until the clock is syntonised; delay statistics start at lock.
  • TimeTransmitter/BMCA (step 6): own dataset from the role; a foreign GM is followed only if better; LISTENING -> MASTER after the announce receipt timeout. Two-step Sync (raw frame, HW TX time in Follow_Up), Announce with the PTP timescale flag, Delay_Resp with HW RX time. Hybrid: as TimeReceiver, Delay_Req unicast to the GM (IP from Announce, MAC from its Sync) with the unicastFlag; as TimeTransmitter, a unicast Delay_Req gets a unicast Delay_Resp.
  • The EMAC's PTP filter only timestamps multicast PTP; unicast Delay_Req got none. The EMAC now timestamps every received frame (emac_ll_ts_all_enable); the RX hook picks the port-319 PTP event messages.
  • A unicast Delay_Resp carries logMessageInterval 0x7F; then ptp.log_delay_req is used.
  • As GM without an earlier lock the clock starts at 0 (1970): no RTC/NTP. Media timing is unaffected; NTP seeding could follow with the internet interface (phase 2).
  • Sync send times jitter by ~10 ms (FreeRTOS 100 Hz tick); accuracy is unaffected (two-step).
  • Measured vs ptp4l (Intel i210 GM, non-PTP switch): lock in ~35 s cold, ~22 s after GM loss; offset within a few hundred ns; board crystal -39.8 ppm. Link asymmetry (1G GM / 100M board through a store-and-forward switch) adds a constant offset error of a few µs that no receiver can see.

PTP status (status.ptp) — main panel modelled on Riedel Bolero "PTP Status"

UI row	field(s)	notes
PTP State	state	INITIALIZING/LISTENING/UNCALIBRATED/SLAVE/MASTER/PASSIVE/FAULTY/DISABLED, shown as TimeReceiver/TimeTransmitter/…
Lock State	locked	servo converged (e.g. |offset| < 1 µs for N consecutive syncs)
TimeTransmitter	gm_id	shown as MAC + "(FFFE)" when EUI-64 from MAC
Time Offset	offset_ns	Bolero brackets in ns: <100 (green), 100-500 (green), 500-1000 (amber), 1000-10000 (red), >10000 (red); exact value in brackets
Frequency Deviation	freq_ppb	servo frequency correction; buckets <100 ppb, 100-500 ppb, 500 ppb-1 ppm, 1-10 ppm, 10-50 ppm, >50 ppm
Network Delay	path_delay_ns ± path_delay_sd_ns	mean path delay ± std dev over the rolling window
Hops	steps_removed
Time/Frequency Traceable	gm_time_traceable, gm_freq_traceable	Announce flagField bits
Version	version	2
Own Clock Class	own_class	255 / 248 per role

Details (collapsed): clock_id, gm_class, gm_accuracy, gm_p1, gm_p2, sync_avg_ms/min/max + sync_jitter_us (measured Sync interval over the last window = 64 messages; RX as receiver, TX as transmitter), announce_avg_ms, delay_req/delay_resp counters, hw_ts. When this device is the GM, offset/frequency/delay show "–".

AES67 TX

• Defaults match Riedel Director 4-wire AES67 output: port 5004 (1024–65535), L24, ptime 1.000 ms, payload type 96 (96–127), SSRC 0, time stamp offset 0 (32-bit). Channels: Riedel default 1; this core defaults to 2, and projects override.
• Mono sum (mono_sum): stream goes out as 1 channel carrying (ch1 + ch2) / 2. Done in the TX path after gain, in 32-bit (int32 or float) before converting to L16/L24, so it can't clip; -6 dB relative to a single channel at full scale (identical L/R content comes out at the same level). The UI forces channels = 1 while ticked, and the SDP follows (rtpmap .../1, i=mono). Unticking restores the previous channel count (or 2).
• RTP multicast range 224.0.2.0 – 239.255.255.255 (validated in UI and firmware). This excludes 224.0.0.x/224.0.1.x, where PTP lives.
• SSRC is used literally (0 allowed, as on Riedel). RFC 3550 recommends random; not required for AES67 multicast.
• RTP ts = (PTP time * rate + clk_offset) mod 2^32 (a=mediaclk:direct=<clk_offset>). Riedel's "Time Stamp Offset" is the same value. Packets paced on PTP time; the audio source is pulled from a ring buffer through a callback, so a sender needs no ASRC.
• Discovery: manual (SDP export only) or SAP. NMOS IS-04/05 is a possible future option (Riedel SIC cards have an NMOS tab).
• SDP: v, o (aes67 IP, session_id, session_ver), s, c=mcast/ttl, t=0 0, a=clock-domain:PTPv2 , m=audio port RTP/AVP pt, a=rtpmap, a=recvonly, a=ptime, a=ts-refclk:ptp=IEEE1588-2008::, a=mediaclk:direct.
• SAP to 239.255.255.255:9875 every 30 s, plus immediately on change; deletion packet on disable or shutdown.
  • Implemented (aes67_sdp_sap): SAPv1, payload type application/sdp, msg id hash = 16-bit hash of the SDP, TTL = aes67.ttl. The SDP is checked once a second; on a change the old hash is deleted before the new one is announced. Nothing is announced until a PTP GM is known (else ts-refclk would be all zeros). Shutdown deletion via esp_register_shutdown_handler (covers reboot and OTA).
  • GM change: aes67.session_ver is bumped (stored via cfg_set_number, without re-applying the aes67 group so the stream keeps running) and SAP re-announces within 1 s. The first GM after boot is not counted as a change.
• Media 2 (ST 2022-7 redundancy): not possible on single-port boards like the ESP32-P4-ETH. Keep the schema open for a mcast2/port2 pair on dual-NIC hardware (Riedel applies one PTP config to both Media 1 and Media 2).

AES67 RX (future, for receiver projects) — Riedel Director 4-wire input defaults

• Import SDP (parse c=, m=, rtpmap, ptime, ts-refclk, mediaclk) or manual: multicast IP, port, optional sender IP (source filter, IGMPv3 SSM), channels, bit depth, ptime, payload type, SSRC, time stamp offset.
• Play mode "synchron" (playout locked to PTP/RTP timestamps). Receive buffer default 8 × ptime (8.000 ms at 1 ms), min 3 × ptime, max 150 ms.
• Channel selection: which channel of the stream feeds a mono output.

Network

• Default: one untagged interface. VLAN split (phase 2, not implemented yet) option: AES67 untagged + internet tagged (inet.vlan_id/pcp) using the ESP-IDF vlan_support approach (examples/network/vlan_support, ESP32-P4 supported). AES67 side has no gateway; default route and DNS are on inet.
• PTP, RTP, SAP and IGMP bind to the AES67 netif. HTTP server filters on web_on.
• DHCP sends net.hostname (option 12), so the router's lease list / local DNS shows it; after a rename it applies at the next lease renewal.
• mDNS (espressif/mdns): <net.hostname>.local plus the web UI as _http._tcp port 80. A hostname change applies live (the new name answers after ~1 s of RFC 6762 probing).
• LLDP (IEEE 802.1AB), transmit only, own code (not in IDF/lwIP): every 30 s, TTL 120 s, plus immediately on a new IP or hostname. TLVs: chassis ID and port ID = MAC, port description "eth0", system name = net.hostname, system description = project + firmware version, capabilities station-only, management address = IPv4. With the VLAN split, send it on the AES67 (untagged) side; the 802.1 port VLAN ID TLV can be added then.
• Switch port: AES67 as native/untagged VLAN, internet tagged, PoE on. IGMP snooping + querier on the AES67 VLAN.

Time / NTP (phase 2, not implemented yet)

• SNTP client for wall-clock time. Servers configurable as hostnames or IPs (e.g. pool.ntp.org, 0.pool.ntp.org, a local server); several allowed, tried in order; names resolved via DNS and re-resolved when a server stops answering.
• Uses: set the EMAC PTP clock to real time (TAI = UTC + 37 s) before this device becomes GM when it has not learned time from another GM (without NTP it starts at 1970); RFC 5424/3164 syslog timestamps.
• Never step the PTP clock from NTP while locked to a GM or while GM with receivers locked; only seed it before becoming GM.
• Planned config group: time: {ntp, servers[] or a comma-separated string, sync_interval_s}; the UI needs matching fields.

Syslog

• esp_log vprintf hook (chains to UART) -> queue -> low-priority UDP task. Non-blocking, drop and count on overflow, no recursion. RFC 5424 or 3164, PRI = facility*8 + severity (E3 W4 I6 D7 V7). Strip ANSI colour codes. HOSTNAME = net.hostname, APP-NAME = log tag.
• Buffer early boot logs in the queue until the netif has an IP. POST /api/log/test sends one info-level message.

Firmware update (OTA)

• Partition table: nvs, otadata, phy_init, ota_0, ota_1 (no factory app). Size the slots from flash_id; e.g. on 16 MB, 2 × 6 MB leaves room for growth (cspot + TLS + codecs). Keep NVS outside the app slots so config survives updates.
• Version: PROJECT_VER from git describe --tags --dirty, read at runtime from esp_app_get_description(); also reported as status.fw.
• API:
  • GET /api/ota -> {version, project, build_date, idf, running, previous, previous_version, pending_verify, can_rollback}
  • POST /api/ota: raw .bin as application/octet-stream, streamed with esp_ota_begin/write/end into the inactive slot (never buffered whole in RAM). 200 then reboot; 4xx with a message if rejected.
  • Upload only, by design: the device never pulls firmware from a URL or git server (no esp_https_ota, no update checks).
  • POST /api/ota/confirm: mark the running app valid. POST /api/ota/rollback: boot the previous slot.
• Reject before writing (checked on the first chunk): image magic, chip ID, wrong project_name in the image's app description (stops flashing another project's .bin), and a chip revision outside the image's min/max range (matters on the rev < 3 boards). esp_ota_end then verifies the whole image. Reject downgrades only if a flag is set (off by default; the flag doesn't exist yet).
• previous/previous_version are only reported when that slot is bootable (esp_ota_check_rollback_is_possible()), so a half-written upload or a rolled-back image is never offered.
• Rollback: enable CONFIG_BOOTLOADER_APP_ROLLBACK_ENABLE. New firmware boots in "pending verify". Self-test after boot (60 s limit): Ethernet has an IP and the web server answers (an HTTP GET of its own /api/status returns 200; optionally PTP locked within N seconds later). Test-only CONFIG_AES67_OTA_SELFTEST_FORCE_FAIL (sdkconfig.selftest_fail) forces a failure to test rollback. Pass: esp_ota_mark_app_valid_cancel_rollback() automatically. Fail or crash before that: reboot, and the bootloader rolls back to the previous slot. The UI also offers manual confirm/rollback while pending.
• Keep AES67 TX running during the upload if possible (OTA writes are slow flash erases; run them at lower priority than PTP/TX). Expect short audio glitches; the reboot drops the stream for a few seconds.
• Security: no auth yet (LAN only), like the rest of the API. Add a bearer token and optionally signed images (secure boot v2 / signed OTA) before exposing the device on an untrusted network.
• UI: Firmware section: upload with progress bar, waits for the reboot and reports the new version (or that it rolled back), confirm/rollback buttons while pending.

Proposed ESP-IDF component layout (to be built)

components/
  aes67_board/     board pin profiles (waveshare_esp32p4_eth.h, …), eth init
  aes67_net/       netif setup, optional VLAN split, DSCP/PCP helpers
  aes67_ptp/       PTPv2 ordinary clock: BMCA, servo, master, multicast/hybrid, EMAC HW timestamps, stats
  aes67_tx/        RTP sender task, PTP-paced, pull callback: size_t (*read)(int32_t *buf, size_t frames); optional mono sum
  aes67_sdp_sap/   SDP builder/parser + SAP announcer
  aes67_syslog/    remote syslog
  aes67_health/    uptime, heap/PSRAM, temperature sensors (temps[] with min/max/warn)
  aes67_ota/       firmware upload, image checks, rollback self-test
  aes67_web/       httpd, embedded index.html, config store (cJSON in NVS), core routes
main/              project: sources, player, extra routes/config groups

• Config registry: each component calls cfg_register(group, defaults_json, validate_cb, apply_cb). The project registers its own groups (e.g. "source") the same way. /api/config is built from the registry.
• Status registry: status_register(cb); each callback adds fields to the /api/status cJSON. Temperature sensors: health_temp_register(name, read_cb, warn_c).
• Route registry: the project adds routes (e.g. /api/player/*) via web_register_uri().
• Future: aes67_rx (receiver, see RX section) for receiver projects.
• Consider moving the core components and web/index.html into a separate git repo (submodule or IDF component manager) once they're stable.