ALLI-26063: the vanishing render

PRs: backend #701 · frontend https://github.com/AgencyPMG/alli-frontend-creativestudio/pull/517 · evidence page (private): https://claude.ai/code/artifact/b617f3c0-4be8-46d9-880c-47d308612af7

Mechanism

When the dispatch coordinator's Athena query failed before the first batch was enqueued, _run_dispatch_coordinator hit the generic except Exception with batches_enqueued == 0 and called delete_render. The task raised (not Reject), so with CELERY_TASK_ACKS_LATE=True the SQS message was acked: no retry, no DLQ, so the ALLI-27444 drainer never saw it. The tracker hash was gone, render-jobs/in-progress/ (which only lists dispatching/processing renders) came back empty, and useRenderProgress mapped "vanished" to completed ("Render complete") or, if it vanished within the 10 s no-work window, to already_rendered. A render_failure_logs row was already being written by the task-boundary handlers (PRs #604/#605), so the drawer knew the reason while the status bar said complete. This was "Gap B" in the ALLI-27444 taxonomy.

Staging, creativestudio-staging-variant-dispatch-worker, every day 2026-08-26 to 2026-08-31 at 03:00 and 23:40 UTC:

athena_query_failed  feed=909c2031-ccdb-49a0-8363-6c4f42d8ea40  state=FAILED
  reason: HIVE_INVALID_METADATA: Table descriptor contains duplicate columns
variant_dispatch_enumeration_truncated  template=dacc9ab3-…  render=d62f668b-…  batches_enqueued=0
Render d62f668b-319f-42d2-8f57-c28c004185b2 deleted from cache

No render_marked_failed ever followed. The natural repro disappeared on 8/31 when ALLI-28433 repaired both feeds.

flowchart LR
  subgraph before["BEFORE on main, staging 2x daily 8/26 to 8/31"]
    A1["POST render-jobs/render-only/<br/>202 render_id, tracker dispatching"] --> B1["variant_dispatch_request<br/>coordinator streams feed rows"]
    B1 --> C1["AthenaService._wait<br/>state FAILED<br/>HIVE_INVALID_METADATA"]
    C1 --> D1["except Exception<br/>batches_enqueued == 0<br/>delete_render"]
    D1 --> E1["GET render-jobs/in-progress/<br/>returns []"]
    E1 --> F1["useRenderProgress<br/>active_cleared, status completed<br/>UI: Render complete"]
  end
  subgraph after["AFTER, backend PR 701 + frontend PR"]
    A2["POST render-jobs/render-only/<br/>202 render_id"] --> B2["coordinator streams feed rows"]
    B2 --> C2["AthenaQueryError"]
    C2 --> D2["_fail_render_after_dispatch_error<br/>mark_render_failed(render_id, reason)<br/>state kept 24h, lock released"]
    D2 --> E2["GET in-progress/ returns []"]
    E2 --> G2["GET render-jobs/:render_id/<br/>status failed"]
    G2 --> F2["useRenderProgress status failed<br/>banner + failure log row"]
  end
  classDef bad fill:#ffd7d5,color:#24292f,stroke:#cf222e;
  classDef good fill:#d3f5db,color:#24292f,stroke:#1a7f37;
  classDef ctx fill:#eaeef2,color:#24292f,stroke:#57606a;
  class D1,F1 bad;
  class D2,G2,F2 good;
  class A1,B1,C1,E1,A2,B2,C2,E2 ctx;

Only the fifth hop changes on the backend; the frontend gains one read after the in-progress poll comes back empty.

Fix

• Backend: _fail_render_after_dispatch_error (tasks.py) calls mark_render_failed(coerce_uuid(render_id), "ExcType: msg") instead of delete_render in the coordinator nothing-enqueued branch and both legacy sequential branches (generic + MaxRetriesExceededError). Same lock and active-set release; the terminal state survives until RENDER_TTL (24 h). Slot-v1 (Reject(requeue=True)) and partial-enqueue branches untouched. New GET /clients/{cid}/render-jobs/{render_id}/ (RenderJobView.retrieve, url render_detail), 404 unless the tracker row belongs to the URL client. Design doc docs/design/ALLI-26063-Fail-render-on-coordinator-error.md.
• Frontend: getRenderJob lazy query; resolveRenderOutcome in useRenderProgress runs in front of the two silent-success branches. failed dispatches the existing failed action, invalidates the RenderFailureLog tag, and calls onRenderCompleted; anything else (completed, 404, no id) dispatches the exact old action. A pending-token ref ignores stale reads after reset, a new render, blocked/failed callbacks, active-again, or unmount.

Verification

• Backend: 2000 tests pass on a fresh test DB (the first run showed 31 failures in template tests that were a stale --reuse-db, gone with --create-db). ruff format --check and ruff check clean repo-wide.
• Frontend: 22 tests across the four touched suites, tsc both projects, eslint, prettier. Discrimination: 5 of the 8 new hook tests fail against the old hook.
• Local probe against real Valkey + Postgres, ~/alli-seeds/alli-creativestudio-backend/seed_ALLI-26063.py, with ExportService.stream_rows_for_delivery patched to raise the staging error. Coordinator path and sequential path both:

1 render created: b135ad4a-… status= dispatching
2 in-progress before: ['b135ad4a-…']
3 task state: FAILURE | AthenaQueryError('Query failed: HIVE_INVALID_METADATA: …')
4 tracker after: ('failed', '2026-09-02T18:03:41+00:00')
5 in-progress after: []
6 template lock free: True
7 failure log rows: [{'failure_source': 'screenshot_on_demand', 'reason': 'athenaqueryerror: query failed: hive_invalid_metadata: …', 'occurrence_count': 1}]
8 GET render detail: 200 {'status': 'failed', 'total': 0, 'progress': 0, 'completed_at': '…'}
9 GET in-progress: 200 []
10 GET detail as other client: 404

• Review: the two adversarial review workflows were cut short by a session usage limit. The backend API lens ran clean; the backend test lens produced two coverage findings that were fixed (the task-boundary test now runs the real coordinator against the fake-redis tracker and asserts FAILED, empty in-progress list, released lock, and the failure-log row; the sequential retry-exhaustion branch got its own test). State-machine, ticket-fit, and all frontend lenses were walked by hand.

Not done, follow-up PRs

• Delivery page table has no per-row failure surface (useActiveRendersPolling only refetches on a count drop).
• A coordinator that dies mid-stream after some batches were enqueued still freezes the run and lets it finish as completed (ADR-016 truncated-run behaviour).
• Slot-v1 Athena failures still end FAILED only via SQS redrive to the coordinator DLQ.
• Post-deploy check worth doing: break a throwaway staging feed's Glue table, press Render, expect the banner. render_marked_failed now logs at ERROR once per failed render (ALLI-28608 convention).