When does the Athena query in ALLI-26063 actually blow up?

Question on backend PR #701: the body says "if the Athena query behind a render blew up", so what actually makes it blow up?

Answer from 90 days of CloudWatch across creativestudio-prod-* and creativestudio-staging-*: one cause, 58 times, nothing else.

What	Count, 90d	Where
HIVE_INVALID_METADATA: Table descriptor contains duplicate columns	24 prod (9 feeds, 8/05-8/11), 34 staging (2 feeds, 8/26-8/31)	athena_query_failed
Query timeout (60s budget)	0	athena_query_timeout
Syntax error, throttling, missing partition, permission denied	0	any athena_* event

Prod was not spared. Nine distinct prod feed tables hit it between 8/05 and 8/11, and at least one prod render was deleted by the exact code path PR #701 changes:

2026-08-11 17:06:30.815  athena_query_failed  feed=f79e73e5-…  HIVE_INVALID_METADATA … duplicate columns
2026-08-11 17:06:30.815  variant_dispatch_enumeration_truncated  render=72195b17-…  batches_enqueued=0
2026-08-11 17:06:30.828  Render 72195b17-a368-4c82-8357-d9c334f4c23e deleted from cache

That is the vanishing render, in production, before the fix existed.

The timeout is not close

_QUERY_TIMEOUT_SECONDS = 60 in athena_service.py. The worst real prod query in 30 days scanned 407 MB and finished in 9.7 s. Typical scheduled reads land at 0.5-1.5 s on a few MB. There is roughly 6x headroom on the largest feed we run, so the timeout is a guard, not a live risk.

flowchart TD
  A["coordinator asks for feed rows<br/>ExportService.stream_rows_for_delivery"] --> B["_validate_feed_for_query"]
  B -->|"unsupported source_type<br/>or no current file"| X1["AthenaQueryError<br/>0 seen in 90d"]
  B --> C["validators: slug, uuid, snapshot_date, limit<br/>athena_service.py 241-274"]
  C -->|"malformed id"| X2["AthenaQueryError<br/>0 seen in 90d"]
  C --> D["start_query_execution"]
  D -->|"throttling / quota<br/>raises botocore ClientError, NOT AthenaQueryError"| X3["different except branch<br/>0 seen in 90d"]
  D --> E["_wait: poll every 1s, budget 60s"]
  E -->|"state FAILED"| X4["AthenaQueryError<br/>58 seen in 90d<br/>ALL duplicate columns"]
  E -->|"60s elapsed"| X5["AthenaQueryError timeout<br/>0 seen in 90d<br/>worst real query 9.7s"]
  E --> F["_stream_results, paged"]
  F --> G["rows to dispatch batches"]
  classDef bad fill:#ffd7d5,color:#24292f,stroke:#cf222e;
  classDef warn fill:#fff8c5,color:#24292f,stroke:#9a6700;
  classDef good fill:#d3f5db,color:#24292f,stroke:#1a7f37;
  classDef ctx fill:#eaeef2,color:#24292f,stroke:#57606a;
  class X4 bad;
  class X1,X2,X3,X5 warn;
  class G good;
  class A,B,C,D,E,F ctx;

The only red exit is the one that has ever fired. Note start_query_execution throttling would raise a raw botocore ClientError, not AthenaQueryError, so it lands in the coordinator's broad except Exception rather than anything Athena-aware. Same destination today, but worth knowing if we ever narrow that catch.

Why duplicate columns happens at all

Hive counts partition keys as schema columns. Our tables are partitioned on client, template, feed, snapshot_date, so a feed whose CSV also has a data column literally named template produces a descriptor with two template columns and every read of that table fails. That was ALLI-28433. PR #641 (ALLI-28423) guards the partition-key collision at ingestion, PR #698 added the within-feed duplicate guard, and the staging descriptors were repaired 8/31, which is why the staging series stops there.

What actually breaks that branch today

The coordinator wraps enumeration in a broad except Exception, so PR #701's branch catches much more than Athena. Right now in prod, twice a day, the thing hitting it is not Athena at all:

2026-09-03 05:30:00  TemplateServiceError: Template with ID af0d1ae5-98f1-4f1d-bb88-3701b2bf596c for client 'None' does not exist
2026-09-02 19:00:01  TemplateServiceError: Template with ID 8eef2282-f190-4d31-a251-19401fefeb6e for client 'None' does not exist

Two prod templates have been deleted while their beat schedule entries survived. af0d1ae5 has failed every day since 8/21 at 05:30, 8eef2282 every day since 8/20 at 19:00, ~29 failures total, still firing this morning. Worth its own ticket: deleting a template should remove its render schedule.

This one does not produce an orphaned render, and PR #701 does not change it. get_template raises at tasks.py:470, before create_render at :472, so render_id is still None when the failure branch's if render_service is not None and render_id and batches_enqueued == 0 guard is evaluated. Nothing to fail, nothing to delete. It only writes a render_failure_logs row with source variant_dispatch.

Also worth noting

The prod truncations with batches_enqueued of 3018, 2808, 9676 and 9983 (8/13 to 8/25) are the other shape: the coordinator died mid-stream after publishing work. PR #701 deliberately leaves those alone, and they still finish as completed under ADR-016's truncated-run behaviour. That is the follow-up already listed on the PR.