A messy real-world workbook becomes a normalized monthly demand history through a two-stage pipeline — Worker orchestrates, Modal parses — and the original file is deleted at the instant the extraction becomes durable. The mapping the engine proposes is always shown before it is used.
After M1 the product had users and no data. M2 is the whole path from a messy spreadsheet a planner already owns to a normalized monthly demand history — and the moment where the original file is destroyed.
headers.py and coerce.py come from the grocery-outlet demo with
attribution intact, extended only inside marked blocks with a CUSTOMER kind and
ton-app's canonical schema. Header text alone never decides a column.
The Worker orchestrates and Modal computes. The raw file travels out over a §0.3-signed URL and the result comes back over a §0.3-signed callback that is idempotent by construction — the same primitive M0 built and proved.
Seven canonical fields, a confidence badge on each, an 8-row preview of what was read, and a data-quality report of everything dropped, merged or renamed — all before the mapping is confirmed.
One upload crosses the trust boundary twice: outbound with proxy credentials, inbound with a signed callback. Hover any node to isolate its edges. The red edges are the authenticated crossings; the amber node is where the uploaded file ceases to exist.
R2.put(uploads/…/raw), D1 rows, then invokes Modal. Answers 202 with dataset_id.exp and sig, no other query parameter accepted. Streams the workbook to Modal.extracted.json, then attempts the one state-conditional transition. A replayed callback changes nothing.requires_proxy_auth=True. Only the workspace proxy token in Modal-Key/Modal-Secret gets through.auto_map over header text and column values → DQ report.extracted.json, computes the unresolved set (column null or confidence < 0.70) and returns proposal + preview + DQ.LLM_MODE=stub on previews spends nothing.sku plus either period+ordered or month columns is mandatory.uploads/{uid}/{ds}/raw lives for the length of one parse; datasets/…/extracted.json replaces it.0002_mapping adds mapping and DQ columns. Job status is the only source of truth the SPA polls.The Worker never sends the workbook to Modal. It sends a path, and Modal comes back for the bytes over a URL it has to sign itself. That keeps the 20 MB body out of the outbound invocation, keeps Modal stateless, and means the only credential in the container is the shared HMAC secret — no Cloudflare token ever leaves the edge.
The product tells the client their file is parsed and then deleted. That sentence is enforced in exactly one place, and it is bound to a state transition rather than to the arrival of a message.
WHERE status = 'running' clause is the whole idempotency argument: the transition can
succeed exactly once, and every side effect — the dataset row, the summary counters, the deletion
— hangs off the row count it returns.awaiting_mapping, changes 0 rows, touches nothing.running, so a late success cannot resurrect a failed job.Everything below happens before any model is asked anything. The LLM only ever sees the columns this pass could not resolve.
# header text AND column contents both score mapping = auto_map(raw_headers, columns) months = [i for i, h in headers if looks_like_month(h)] MappingProposal( sku, customer, period, ordered, invoiced, name, uom, month_columns = months, # a wide sheet is 3+ month columns and no quantity column wide = len(months) >= 3 and ordered.column is None, value_check_rejections = mapping.value_check_rejections, )
for row in raw_body_rows: if is_total_row(row): totals_skipped += 1; continue # wide -> one output row per month column period = to_period_key(header or cell) quantity = parse_number(cell) # None -> invalid_rows customer = text or "Unassigned" retained = source_months[-36:] # disclosed in the DQ report aggregate = sum duplicates per (sku, customer, period)
| field | what it counts | why the user cares |
|---|---|---|
| series | distinct (sku, customer) pairs kept | The unit everything downstream forecasts. |
| thin_series | series with fewer than 3 positive months | Warns before M3 hands them a named fallback instead of a model. |
| months_retained total_source_months | window kept vs window found | Says out loud that only the last 36 months were used. |
| duplicates_summed | rows merged into an existing (sku, customer, period) | Explains why a total on screen differs from the source sheet. |
| totals_skipped | subtotal / grand-total rows discarded | The classic double-count; fixture 1 asserts exactly one. |
| invalid_rows | unparseable period or quantity | Silent drops become a number instead of a mystery. |
| unassigned_customer_rows | rows with no customer column value | Tells the planner their file is SKU-level, not account-level. |
| zero_share · zero_rows omits_zeros | sparsity of the completed panel | Distinguishes “no demand” from “row not present” — the difference M3's SBC routing is built on. |
| invoiced_present | whether an invoiced column was mapped | Gates the fill-rate exception M4 can raise. |
M2's exit condition is the five real messy workbooks, end to end, plus the browser flow a client would actually walk. Both were driven against a real running Worker — miniflare R2/D1/KV with the real Python parse job behind a local transport — not a mock.
scripts/qa/m2.sh uploads every fixture, polls to awaiting_mapping,
checks the proposal resolved the mandatory columns for that shape, and — the important one
— re-fetches the raw file over a freshly signed URL and requires a 404.
The 404 is checked with a valid signature, so it proves the object is gone rather than that the request was refused.
Login → upload → mapping review → confirm, driven in real Chromium against the same stack, watching the console and the layout rather than just the API responses.
Fixture 1 is the adversarial one: a wide sheet with month columns, a title row above the header
row, and a grand-total row that must be counted exactly once as skipped — the assertion is
dq.totals_skipped == 1, not >= 1.
The CI M2 gate runs against a deployed preview, and every route it touches needs R2 —
which was not enabled on the Cloudflare account when this branch was written. The PR shipped
saying so. R2 has since been enabled and commit f900269 on main turned the bindings on
for both environments. See finding R1.
42 files, +4,374 −57. The two largest source files are the vendored adaptor; the largest single file is a real 528-line demand-history fixture.
One blocker that shipped open and has since closed, plus the five decisions in this branch worth knowing before touching it. Click any row to expand.
The milestone's real output is not the upload screen — it is a normalized, deduplicated, window-bounded monthly demand history whose every discrepancy against the source workbook is counted and shown. M3 can therefore assume its input is clean and argue only about models, and the client-facing retention sentence has a single, replay-safe enforcement point rather than a policy document.
no-store.extracted.json as the single input contract, fetched over a signed URL.forecast-result.