Search runs in your browser, across every published page on this site.

Research engine

How the machine runs

A scheduled collector on a free tier, with a two-minute clock beside it watching for filings. This page is the operating detail underneath the standard on the method page: how fast the engine is allowed to ask, what it does when a source refuses, and how you check that the page in front of you and the engine that built it are on the same run.

Figures below come from the run receipt generated , and from the source files named beside each constant.

One pass

Collection is not what a run costs

The receipt carries two clocks for the run and two for every adapter, so the shape of a pass is measured rather than described.

Adapters constructed
11across 2 lanes, 11 closed clean
Source checks
412 of 412answered
With sources open
15.2 sfirst adapter opening to the last one closing
After the last source closed
11.6 s43% of the pass
Pass, end to end
26.8 sreceipt status: succeeded

Add up each adapter's own duration and the run did 33.2 s of collection work inside a 15.2 s collection window. Several checks ran at once, fitting 2.19x as much timed work into that window. The slowest measured source check was Measured current stock discussion attention at 14.8 s.

Which is also why the 15.2 s is the small number. 43% of the pass ran after every source was shut. The receipt times that window; it does not itemise it. What runs in it is the merge: 11 adapter outputs folded into 2 lane states under a deterministic conflict policy, a last-good record retained for anything that failed, and the progress stream settled so the run can be replayed. Projection assembly, claim evaluation, and the detector pass all run after this clock stops, which is why the projection carries the run’s finish time as its own generation stamp.

Adapters and source checks by lane, this run
LaneWhat it holdsAdaptersSource checks answered
Pulse Official releases, documents, rates, macro series, and filing facts 10 397 of 397
Reference Entities, identifiers, roles, and rights-reviewed media 1 15 of 15
Request pacing

How fast the engine is allowed to ask

Every source gets a minimum interval between requests, held per source and enforced by the HTTP client rather than left to each adapter to remember.

The two-minute clock. A Cloudflare Worker cron runs every two minutes on the free tier and asks the SEC whether any of the 424 covered issuers has filed. It sleeps 300 ms after each one, so a full sweep spends 127.2 seconds of a 120-second window on pacing alone and the next tick never lands on top of the last. Conditional requests carry the stored ETag, so an issuer that has not filed answers 304 and costs nothing but the round trip.

Two floors for one publisher. The same SEC endpoint is paced at 125 ms in the engine and 300 ms in the worker. The engine runs a handful of times a day, attended, against a run budget; the worker runs 720 times a day, unattended, forever. The same sweep costs 52.875 seconds of waiting in the engine and 127.2 in the worker, and the slower one is the one nobody is watching.

Minimum interval between requests, per source
SourceFloorWhyDefined in
SEC EDGAR, in the engine 125 ms Submissions and company-facts sweeps run from GitHub Actions, one issuer at a time. site/scripts/research-engine/sec-live-adapter.mjs
SEC EDGAR, in the Cloudflare worker 300 ms The two-minute filing clock is slower than the engine on the same host, because it runs unattended and far more often. infra/research-api/src/index.ts
Bureau of Labor Statistics 500 ms Applies to both the series API and the news-release calendar file. site/scripts/research-engine/bls-live-adapter.mjs
Bureau of Economic Analysis 500 ms The national-accounts API is keyed and rate-limited per key. site/scripts/research-engine/bea-live-adapter.mjs
Bank of Canada 250 ms Policy rate and daily exchange rates from the Valet API. site/scripts/research-engine/bank-of-canada-adapter.mjs
U.S. Treasury 250 ms Daily par yield curve rates from the fiscal data service. site/scripts/research-engine/treasury-live-adapter.mjs
Statistics Canada 250 ms Table vectors, which the publisher locks between a release being prepared and 08:30 Eastern. site/scripts/research-engine/statistics-canada-live-adapter.mjs
GLEIF 250 ms Legal-entity identifier lookups that tie an issuer to its registered legal name. site/scripts/research-engine/gleif-reference-adapter.mjs
Federal Reserve feeds 1,000 ms The press, speeches, and testimony feeds are static XML and get the most conservative floor on the list. site/scripts/research-engine/official-feed-adapter.mjs

Each floor is the literal in the file beside it. The build asserts that in site/tests/engine-constants.test.mjs, which opens every file in the last column and fails if the number moved there and not here.

When a source refuses

A refusal is a fact about the source, not a hole in the record

An adapter that fails keeps its last good record, marked as retained, and the run reports itself partial. It never substitutes, interpolates, or carries a stale figure forward as current.

The hardest case is a publisher that is working correctly and still says no. Statistics Canada answers 409, the product is not released yet for the hours between a table being locked for its next release and 08:30 Eastern. That is a schedule, not an outage, and the engine records it as an embargo so the previous figure keeps its collected freshness. But a schedule ends. An embargo still standing 36 hours later has outlived its own release cycle, so it is reclassified as an outage and the retained figure stops counting as current. A status that can only ever be benign is not a status.

Below the adapter, the HTTP client attempts a request 3 times on a 429 or any 5xx, waiting 250 ms then 500 ms between attempts, doubling from 250 ms and capped at 10 s. A Retry-After header replaces that calculation. A status the caller has declared settled is not asked again, because it will read the same on the third attempt and asking spends the source's rate budget to learn nothing.

Retry, backoff, and quarantine constants
ConstantValueWhat it governsDefined in
HTTP attempts per request 3 A 429 or any 5xx is retried. A status the caller has declared settled is not asked again, because it will read the same and it spends the source rate budget to learn nothing. site/scripts/research-engine/http-client.mjs
Retry delay 250 ms doubling, capped at 10 s The delay is 250 ms times 2 to the power of the attempt minus one. A Retry-After header replaces the calculation, still capped. site/scripts/research-engine/http-client.mjs
Request timeout 20 s A request that has not answered is aborted rather than left to hold a run open. site/scripts/research-engine/http-client.mjs
Adapter attempts per run 2 Above the HTTP retry, so a whole adapter gets a second pass before the run records it as failed. site/scripts/research-engine/core.mjs
Embargo credibility window 36 hours Statistics Canada answers "the product is not released yet" while a table is locked. That is the source working. An embargo still standing after 36 hours has outlived its release cycle, so it is reclassified as an outage and the retained figure stops counting as current. site/scripts/research-engine/orchestrator.mjs
Progress events per batch 8, capped at 64 A batch that fails to write goes back to the head of the queue in order, and the threshold backs off so a failing sink is not asked to swallow the whole queue on every subsequent event. site/scripts/research-engine/run-event-queue.mjs
Filing size ceiling 15 MB A primary document above the ceiling is not pulled into the worker, which has a fixed memory budget on the free tier. infra/research-api/src/index.ts
Stuck job requeue 5 minutes A job left queued or processing past this window is treated as lost and dispatched again. infra/research-api/src/index.ts
Attempts before quarantine 5 A filing that has failed five times stops being retried and is held for inspection instead of consuming the queue forever. infra/research-api/src/index.ts
Projection integrity

The same hash on both sides of the wire

The engine publishes 12 projection families. Each one carries the SHA-256 of its own content, and a manifest records all 12 hashes together.

Nothing on this site trusts that manifest. Every build re-hashes each family and compares it to the manifest entry through shared/public-projection-validator.ts, and a mismatch throws before a single page is rendered. It collects every file's generation clock before it throws, too, because a per-file error can only say "this one disagrees", which is the message that cannot tell a stale data file from a stale manifest. The whole set says which side is the minority.

The Cloudflare worker serves the same value as the HTTP ETag on the matching endpoint. That makes the hash a shared identifier rather than a decoration: it is how a reader tells whether this page and the live engine are looking at the same run.

Run this, and compare the ETag to the run-summary hash in the table below.

curl -I https://jmm-research-api.joshdhamburger.workers.dev/v1/run-summary

  • The ETag matches 942264b7…0310422e. The page you are reading was built from the run the engine is serving right now.
  • The ETag is different. The engine has collected again since this page was built. The x-jmm-generated-at header on the same response says when, and the next site deploy carries that run forward.
Content hash per projection family, this build
FamilyKindSHA-256 of the payload
companies.v1.json companies d6b6df87…83f0ea0f
people.v1.json people c66f5ef7…486cac87
home-updates.v1.json home-updates 9024cd81…1ce0813c
events.v1.json events 1e74693b…41ee690f
market-observations.v1.json market-observations 1561abb6…6aadb8e8
published-predictions.v1.json published-predictions 6169dd6f…6108273a
claim-records.v1.json claim-records cd5e4ac3…dea05e4e
run-summary.v1.json run-summary 942264b7…0310422e
social-attention.v1.json social-attention 35a11b45…cf509268
analyses.v1.json analyses 5a6f09c2…e30735ac
machine-noticed.v1.json machine-noticed f056174d…9725a2f0
anomaly-board.v1.json anomaly-board 72989ecf…71a7b26b

Hashes cover the payload the endpoint serves. A family whose records did not change between runs keeps its hash, so a changed hash is a changed record and not a changed timestamp.

See the machine's output