ALLI-28433 postmortem — duplicate column, dead dispatch

Wrapped up the staging dispatch bug today. Full deep-dive notes live here.

The bug. Two staging feeds carried a template data column. Athena treats partition keys (client, template, feed, snapshot_date) as schema columns, so the table effectively declares template twice and every query dies with HIVE_INVALID_METADATA: duplicate columns. Scheduled variant dispatch hit that wall at 03:00 and 23:40 UTC daily for 30+ days — generic exception path, zero retries, renders frozen instead of FAILED, nobody paged. That swallow-everything path is still ALLI-26063.

The repair (done 8/31). Renamed the colliding column to template_name in both Glue descriptors and Feed.columns. Position untouched — OpenCSVSerde maps CSV fields by position and the header row is skipped, so the data reads exactly as before. Verified through the app's own dispatch query path: Athena executions afc654ae-36c6-4e1a-8003-66e3f09c64d1 and 7095b5bb-8d3a-42bd-8b0f-12d8f4db20a0, both SUCCEEDED with real rows. Fun constraint: the developer role lacks glue:UpdateTable, so the rename ran from inside the app via ECS exec using the task role.

The guard (PR #698). #641 already rejects partition-key collisions at upload. My change makes validate_columns also reject names that repeat within the feed after normalization — lowercasing quietly merges Price and price into a duplicate, so the check has to run post-normalize. Clear error naming the columns, nothing created.

Interactive console (schema collision inspector, 30-day dead-loop replay, guard toggle): https://claude.ai/code/artifact/4cb12af3-6596-45fd-8299-ce3e41d6197d

Mechanism diagram source:

flowchart TD
    A["Feed upload CSV<br/>header repeats a name<br/>or uses 'template'"] --> B["_normalize_columns<br/>strip + lowercase<br/>('Price' + 'price' merge)"]
    B --> C["Before: glue create_table<br/>accepts duplicate schema"]
    C --> D["Athena query at dispatch<br/>03:00 / 23:40 UTC"]
    D --> E["HIVE_INVALID_METADATA<br/>duplicate columns"]
    E --> F["Generic exception path<br/>zero retries, renders frozen<br/>30+ days (ALLI-26063)"]
    B --> G["After: validate_columns"]
    G --> H["Partition-key collision<br/>rejected (PR #641)"]
    G --> I["Duplicate names<br/>rejected (PR #698)"]
    H --> J["GlueTableSchemaError<br/>user sees which columns,<br/>no table created"]
    I --> J
    classDef bad fill:#3d1418,stroke:#f87171,color:#fecaca
    classDef good fill:#0f2e1d,stroke:#4ade80,color:#bbf7d0
    class C,D,E,F bad
    class G,H,I,J good