Todayedited2 min read
Sidebar: what "pre-enqueue failure branch" means, and why that phrase should go
Asked on PR #701's design doc. Fair hit, it is jargon I wrote.
"Enqueue" is the handoff. The coordinator's whole job is: read the feed rows, chop them into batches of 500, and hand each batch to a background worker by putting a message on the queue. That handoff is one line, tasks.py:566:
dispatch_batch.delay(template_id=..., render_id=..., batch_index=..., payload_key=..., ...)
batches_enqueued += 1 # :577, only after the broker accepts the message
The counter ticks up after the broker accepts, deliberately, so a publish that fails on the very first batch still reads as "nothing handed off".
"Pre-enqueue failure" just means the coordinator died before it handed off even one batch, i.e. batches_enqueued == 0.
Why the code splits on it: who is left to finish the render.
flowchart TD
A["coordinator starts<br/>render is 'dispatching'"] --> B["read feed rows from Athena"]
B --> C{"crashed before<br/>handing off batch 0?"}
C -->|"yes, batches_enqueued == 0"| D["nobody is working on this render<br/>no worker will ever call back<br/>the coordinator is the last actor"]
D --> E["must leave it terminal<br/>OLD: delete_render, render vanishes<br/>NEW: mark_render_failed"]
C -->|"no, batches 0..N already out"| F["workers are already rendering<br/>failing it now would strand them"]
F --> G["freeze the run at the count that went out<br/>in-flight batches finish it as a truncated run"]
classDef bad fill:#ffd7d5,color:#24292f,stroke:#cf222e;
classDef good fill:#d3f5db,color:#24292f,stroke:#1a7f37;
classDef warn fill:#fff8c5,color:#24292f,stroke:#9a6700;
classDef ctx fill:#eaeef2,color:#24292f,stroke:#57606a;
class E good;
class G warn;
class A,B,C,D,F ctx;
Left arm is the one PR #701 changes. Right arm it deliberately leaves alone, and that is the ADR-016 truncated-run behaviour already listed as a follow-up. The prod truncations from 8/13 to 8/25 with batches_enqueued of 3018, 2808, 9676 and 9983 are all the right arm.
The phrase should not survive the PR. "Pre-enqueue failure branch" describes the code to someone who already read the code. Replacement for the How it works bullet:
Backend: when a render dies before any of its work has been handed to a worker, we now mark it failed instead of deleting it. It still gives up its template lock the same way, but the failed state sticks around for a day instead of vanishing. The reason is still written to the render's failure log, so the drawer shows why.
Same for the title (dies before publishing work reads better as dies before any work is handed off) and the Work breakdown bullet. Offered to Juntao; his call whether to push it to the open PR.