# Field reliability — boot autonomy, ETH/USB hot-plug, port swap

Stack: v16ab bridge core + jun11-prod supervisor.

## One-time field commissioning (all 4 Pis)

After deploy, run once from PC:

  cd sync
  .\orchestrate_fixture.ps1 -Action install-prod-boot

Per-Pi manual:

  FIXTURE_POS=N sudo bash jun11-prod/commission_prod_boot.sh

This installs:

1. POG422 USB driver + udev rebind rules
2. ETH raw socket caps (prep_eth_for_fixture.sh)
3. systemd fixture-prod.service (enabled, starts on boot)

Normal deploy does NOT re-run commissioning — only pushes code.

## Boot behavior

- Service starts immediately after udev settle (no network-online wait)
- Supervisor waits for USB hat (/dev/ttyXRUSB*) and ETH carrier before first bridge start
- Power cycle = temporary downtime only; no manual PC start-prod required after commissioning
- ETH may be unplugged at boot; bridge starts when cable is plugged

## Runtime self-healing

| Fault | Recovery |
|-------|----------|
| ETH unplug while running | Bridge keeps running; forwarding skips dead port |
| ETH replug (all ports were down) | Supervisor restarts bridge (~10s backoff) |
| Cable moved to other port (eth0/eth1 swap) | Supervisor debounced restart (~3s) |
| USB hat unplug/replug | Supervisor runs ensure_tty_xrusb.sh; udev may try-restart service |
| Bridge crash | systemd Restart=always + supervisor poll |

Logs: fixture-timing/logs/prod_supervisor_posN.log

Prod errors only: fixture-timing/prod-data/errors.jsonl

## Health check

  systemctl status fixture-prod
  tail -f fixture-timing/logs/prod_supervisor_pos1.log
  tail -f fixture-timing/prod-data/errors.jsonl
  pgrep -af pog422_fixture_bridge

Silence in errors.jsonl is normal.

## Disable / uninstall

Disable (stop, block auto-restart):

  FIXTURE_POS=N bash jun11-prod/disable_prod.sh

Re-enable:

  FIXTURE_POS=N bash jun11-prod/enable_prod.sh
  sudo systemctl start fixture-prod

Remove service entirely:

  sudo bash jun11-prod/uninstall_prod.sh

## Bench trial (no systemd)

  .\orchestrate_fixture.ps1 -Action start-prod

Uses nohup supervisor unless fixture-prod.service is already enabled.

## Validation soak matrix

Run after commissioning; brief Autoplay pause per test is OK.

| Test | Pass criteria |
|------|---------------|
| Power cycle, no ETH at boot | fixture-prod active; bridge up within ~30s of cable plug |
| Power cycle, ETH connected | All 4 bridges up <60s |
| ETH unplug 30s / replug | No crash; forwarding resumes |
| Move cable eth1 to eth0 | Traffic flows without Pi reboot |
| USB hat unplug 10s / replug | Supervisor recovers; game traffic resumes |
| deploy while systemd running | New code after deploy + systemctl restart fixture-prod |

Archive: end-session label v4-field-resilience-soak

## Golden SD image (pos1 → headless field adapters)

Use pos1 as the **master image** after commissioning passes on the 4-board fixture.

1. Deploy latest code: `orchestrate_fixture.ps1 -Action deploy`
2. Verify: `orchestrate_fixture.ps1 -Action verify`
3. Commission all four (or pos1 only for clone): `orchestrate_fixture.ps1 -Action install-prod-boot`
4. Soak: power-cycle pos1; confirm `fixture-prod` active within ~60s with ETH connected
5. Image SD from pos1; flash to field Pis

Each cloned Pi inherits `/etc/fixture/fixture-pos.env` (`FIXTURE_POS=N`). For standalone
EtherGold adapters (not 4-board bench), commission with `FIXTURE_POS=1` before imaging unless
each unit needs a unique position id.

After clone: power + ETH + USB hat only — no manual Python, no PC start-prod.

Disable auto-start on a bench Pi without uninstalling:

  FIXTURE_POS=N bash jun11-prod/disable_prod.sh


Routing is carrier-based, not fixed to eth0/eth1 roles. Ingress arrives on whichever port has the cable; relay and UART fan-out use carrier-up ports only.