How the machine runs
A scheduled collector on a free tier, with a two-minute clock beside it watching for filings. This page is the operating detail underneath the standard on the method page: how fast the engine is allowed to ask, what it does when a source refuses, and how you check that the page in front of you and the engine that built it are on the same run.
Figures below come from the run receipt generated , and from the source files named beside each constant.
Collection is not what a run costs
The receipt carries two clocks for the run and two for every adapter, so the shape of a pass is measured rather than described.
- Adapters constructed
- 11across 2 lanes, 11 closed clean
- Source checks
- 412 of 412answered
- With sources open
- 15.2 sfirst adapter opening to the last one closing
- After the last source closed
- 11.6 s43% of the pass
- Pass, end to end
- 26.8 sreceipt status: succeeded
Add up each adapter's own duration and the run did 33.2 s of collection work inside a 15.2 s collection window. Several checks ran at once, fitting 2.19x as much timed work into that window. The slowest measured source check was Measured current stock discussion attention at 14.8 s.
Which is also why the 15.2 s is the small number. 43% of the pass ran after every source was shut. The receipt times that window; it does not itemise it. What runs in it is the merge: 11 adapter outputs folded into 2 lane states under a deterministic conflict policy, a last-good record retained for anything that failed, and the progress stream settled so the run can be replayed. Projection assembly, claim evaluation, and the detector pass all run after this clock stops, which is why the projection carries the run’s finish time as its own generation stamp.
| Lane | What it holds | Adapters | Source checks answered |
|---|---|---|---|
| Pulse | Official releases, documents, rates, macro series, and filing facts | 10 | 397 of 397 |
| Reference | Entities, identifiers, roles, and rights-reviewed media | 1 | 15 of 15 |
How fast the engine is allowed to ask
Every source gets a minimum interval between requests, held per source and enforced by the HTTP client rather than left to each adapter to remember.
The two-minute clock. A Cloudflare Worker cron runs every two minutes on the free tier and asks the SEC whether any of the 424 covered issuers has filed. It sleeps 300 ms after each one, so a full sweep spends 127.2 seconds of a 120-second window on pacing alone and the next tick never lands on top of the last. Conditional requests carry the stored ETag, so an issuer that has not filed answers 304 and costs nothing but the round trip.
Two floors for one publisher. The same SEC endpoint is paced at 125 ms in the engine and 300 ms in the worker. The engine runs a handful of times a day, attended, against a run budget; the worker runs 720 times a day, unattended, forever. The same sweep costs 52.875 seconds of waiting in the engine and 127.2 in the worker, and the slower one is the one nobody is watching.
| Source | Floor | Why | Defined in |
|---|---|---|---|
| SEC EDGAR, in the engine | 125 ms | Submissions and company-facts sweeps run from GitHub Actions, one issuer at a time. | site/scripts/research-engine/sec-live-adapter.mjs |
| SEC EDGAR, in the Cloudflare worker | 300 ms | The two-minute filing clock is slower than the engine on the same host, because it runs unattended and far more often. | infra/research-api/src/index.ts |
| Bureau of Labor Statistics | 500 ms | Applies to both the series API and the news-release calendar file. | site/scripts/research-engine/bls-live-adapter.mjs |
| Bureau of Economic Analysis | 500 ms | The national-accounts API is keyed and rate-limited per key. | site/scripts/research-engine/bea-live-adapter.mjs |
| Bank of Canada | 250 ms | Policy rate and daily exchange rates from the Valet API. | site/scripts/research-engine/bank-of-canada-adapter.mjs |
| U.S. Treasury | 250 ms | Daily par yield curve rates from the fiscal data service. | site/scripts/research-engine/treasury-live-adapter.mjs |
| Statistics Canada | 250 ms | Table vectors, which the publisher locks between a release being prepared and 08:30 Eastern. | site/scripts/research-engine/statistics-canada-live-adapter.mjs |
| GLEIF | 250 ms | Legal-entity identifier lookups that tie an issuer to its registered legal name. | site/scripts/research-engine/gleif-reference-adapter.mjs |
| Federal Reserve feeds | 1,000 ms | The press, speeches, and testimony feeds are static XML and get the most conservative floor on the list. | site/scripts/research-engine/official-feed-adapter.mjs |
Each floor is the literal in the file beside it. The build asserts that in site/tests/engine-constants.test.mjs, which opens every file in the last column and fails if the number moved there and not here.
A refusal is a fact about the source, not a hole in the record
An adapter that fails keeps its last good record, marked as retained, and the run reports itself partial. It never substitutes, interpolates, or carries a stale figure forward as current.
The hardest case is a publisher that is working correctly and still says no. Statistics Canada answers 409, the product is not released yet for the hours between a table being locked for its next release and 08:30 Eastern. That is a schedule, not an outage, and the engine records it as an embargo so the previous figure keeps its collected freshness. But a schedule ends. An embargo still standing 36 hours later has outlived its own release cycle, so it is reclassified as an outage and the retained figure stops counting as current. A status that can only ever be benign is not a status.
Below the adapter, the HTTP client attempts a request 3 times on a 429 or any 5xx, waiting 250 ms then 500 ms between attempts, doubling from 250 ms and capped at 10 s. A Retry-After header replaces that calculation. A status the caller has declared settled is not asked again, because it will read the same on the third attempt and asking spends the source's rate budget to learn nothing.
| Constant | Value | What it governs | Defined in |
|---|---|---|---|
| HTTP attempts per request | 3 | A 429 or any 5xx is retried. A status the caller has declared settled is not asked again, because it will read the same and it spends the source rate budget to learn nothing. | site/scripts/research-engine/http-client.mjs |
| Retry delay | 250 ms doubling, capped at 10 s | The delay is 250 ms times 2 to the power of the attempt minus one. A Retry-After header replaces the calculation, still capped. | site/scripts/research-engine/http-client.mjs |
| Request timeout | 20 s | A request that has not answered is aborted rather than left to hold a run open. | site/scripts/research-engine/http-client.mjs |
| Adapter attempts per run | 2 | Above the HTTP retry, so a whole adapter gets a second pass before the run records it as failed. | site/scripts/research-engine/core.mjs |
| Embargo credibility window | 36 hours | Statistics Canada answers "the product is not released yet" while a table is locked. That is the source working. An embargo still standing after 36 hours has outlived its release cycle, so it is reclassified as an outage and the retained figure stops counting as current. | site/scripts/research-engine/orchestrator.mjs |
| Progress events per batch | 8, capped at 64 | A batch that fails to write goes back to the head of the queue in order, and the threshold backs off so a failing sink is not asked to swallow the whole queue on every subsequent event. | site/scripts/research-engine/run-event-queue.mjs |
| Filing size ceiling | 15 MB | A primary document above the ceiling is not pulled into the worker, which has a fixed memory budget on the free tier. | infra/research-api/src/index.ts |
| Stuck job requeue | 5 minutes | A job left queued or processing past this window is treated as lost and dispatched again. | infra/research-api/src/index.ts |
| Attempts before quarantine | 5 | A filing that has failed five times stops being retried and is held for inspection instead of consuming the queue forever. | infra/research-api/src/index.ts |
The same hash on both sides of the wire
The engine publishes 12 projection families. Each one carries the SHA-256 of its own content, and a manifest records all 12 hashes together.
Nothing on this site trusts that manifest. Every build re-hashes each family and compares it to the manifest entry through shared/public-projection-validator.ts, and a mismatch throws before a single page is rendered. It collects every file's generation clock before it throws, too, because a per-file error can only say "this one disagrees", which is the message that cannot tell a stale data file from a stale manifest. The whole set says which side is the minority.
The Cloudflare worker serves the same value as the HTTP ETag on the matching endpoint. That makes the hash a shared identifier rather than a decoration: it is how a reader tells whether this page and the live engine are looking at the same run.
Run this, and compare the ETag to the run-summary hash in the table below.
curl -I https://jmm-research-api.joshdhamburger.workers.dev/v1/run-summary
- The ETag matches 942264b7…0310422e. The page you are reading was built from the run the engine is serving right now.
- The ETag is different. The engine has collected again since this page was built. The x-jmm-generated-at header on the same response says when, and the next site deploy carries that run forward.
| Family | Kind | SHA-256 of the payload |
|---|---|---|
| companies.v1.json | companies | d6b6df87…83f0ea0f |
| people.v1.json | people | c66f5ef7…486cac87 |
| home-updates.v1.json | home-updates | 9024cd81…1ce0813c |
| events.v1.json | events | 1e74693b…41ee690f |
| market-observations.v1.json | market-observations | 1561abb6…6aadb8e8 |
| published-predictions.v1.json | published-predictions | 6169dd6f…6108273a |
| claim-records.v1.json | claim-records | cd5e4ac3…dea05e4e |
| run-summary.v1.json | run-summary | 942264b7…0310422e |
| social-attention.v1.json | social-attention | 35a11b45…cf509268 |
| analyses.v1.json | analyses | 5a6f09c2…e30735ac |
| machine-noticed.v1.json | machine-noticed | f056174d…9725a2f0 |
| anomaly-board.v1.json | anomaly-board | 72989ecf…71a7b26b |
Hashes cover the payload the endpoint serves. A family whose records did not change between runs keeps its hash, so a changed hash is a changed record and not a changed timestamp.
See the machine's output
The pass that runs after the collection clock stops, with the arithmetic printed on every finding.
Release calendarOpen the calendar the pulse lane collectsEvery recurring official release, its cadence, and its next dated instance.
FilingsFollow a filing the source clock caughtThe issuers the two-minute SEC sweep watches, each with accession-linked quarterly values.