tokenstats

tokenstats watches the market for AI inference. It listens on Nostr for Routstr nodes announcing themselves, polls their model catalogs alongside OpenRouter and OrcaRouter, prices every model in USD and sats from a live BTC/USD rate, and serves the result as a live price board and a JSON API. It runs at tokenstats.ai.

It's also the feed behind arbstr's routing. arbstr picks the cheapest qualified provider for each request, and tokenstats is where most of "qualified" comes from. arbstr 2.4 started using its quality signals, and arbstr 2.5 added require_healthy, which skips any endpoint tokenstats marks unhealthy. The two stay separate on purpose: tokenstats observes, arbstr executes. Observing is free, too. tokenstats never spends a sat or needs a key to measure a node.

tokenstats has no version numbers. It's deployed by image, so this post covers a round of work from October 25 to 27: 22 pull requests that turned a price board into a feed an agent can act on.

Why It Needed the Work

The market it watches is messy. On any given cycle, roughly 70 Routstr nodes are announced on Nostr and fewer than half answer. Most of the rest aren't flaky, they're gone: in one snapshot, 22 of the 43 failing nodes had domains that no longer resolve, and relays keep replaying their old announcements.

The old feed didn't cope well with that. It polled nodes one at a time, so a cycle could run minutes past its 60 second interval. Quotes were never removed, so the board still served rows last seen in July, and about 45% of the rows arbstr fetched were more than a day old. Reliability was a one-hour window held in memory that reset on every deploy.

Fresher Data

  • Nodes are polled concurrently, each with a 10 second budget.
  • When a catalog fetch succeeds, models it no longer lists are dropped, and any quote not seen for 24 hours expires.
  • Every quote now says how old it is, whether it's stale, whether it's an outlier (more than 10x off that model's median), and what kind of model it is (chat, embedding, image, and so on).
  • OrcaRouter joined OpenRouter as a second reference catalog.
  • OrcaRouter fetches were failing about 17% of the time from the server. Polls every 60 seconds outlived the upstream's idle connections, so the client kept reusing connections the server had just closed. Idle connections are now dropped after 20 seconds, and reference catalogs retry once on a connection failure.

Signals That Survive a Restart

  • Reliability over 1 hour, 24 hours, and 7 days, plus p50 and p95 latency, kept in SQLite as hourly buckets for a week.
  • A probe of each node's GET /v1/info. It's cheap, costs nothing, and answers two questions: how responsive the node is, and who it says it is. tokenstats compares the node's self-reported key with the Nostr key that announced it. On the first live run, 23 nodes matched, 1 didn't, and 16 gave nothing to compare.
  • Dead nodes back off. After 5 failed fetches in a row, a node is polled every 2, 4, 8 intervals and so on, up to once an hour, and comes back right away if it answers or announces itself again. Failures are classed (dns, timeout, tls, http_4xx, http_5xx, and so on), so you can see why a node is down.
  • A healthy flag on every quote. A quote is healthy when its endpoint's last fetch succeeded, its 24 hour reliability is at least 90%, its info probe isn't failing, and the quote isn't stale. That's the flag arbstr's require_healthy relies on.

The dashboard shows all of it now: a feed health strip, reliability, latency, and age columns, and filters for healthy quotes only.

Same Model, Every Source

Each catalog spells model ids its own way. OpenRouter lists anthropic/claude-sonnet-4.5, while Routstr nodes use claude-sonnet-4-5, claude-sonnet-4.5, or claude-sonnet-4-5-20250929. tokenstats now maps them to one canonical id, and a new compare view and /api/compare endpoint show one model's offers from every source side by side, with the spread and the cheapest healthy offer.

In a live run, Sonnet 4.5 came out as 4 spellings across 21 offers with a 278% spread. One node had a unit bug that priced Sonnet at $0.03, which on its own made the spread look like 55,000%. Outliers stay visible but never count as the cheapest offer, and outliers are now judged against every source, not only nodes that spell the id the same way.

Capability Flags

Price and reliability were all a consumer could rank on. Each quote now carries tools, reasoning, and vision flags. Routstr nodes rarely publish these, so a missing flag is filled from OpenRouter (or OrcaRouter) for the same canonical model, and marked as inferred. A flag the node publishes itself always wins, and a filter like capabilities=tools,vision keeps only models that have both.

The Security Pass

A code review turned up real problems, and I fixed them the same day. Candidly:

  • Reflected XSS in the dashboard. The filter links were rebuilt from the query string, and only spaces, &, and = were escaped. A crafted link like /?model="><script>... broke out of the href and ran script in the page. Links are now percent-encoded down to RFC 3986's unreserved characters, every href is escaped again where it's written, and regression tests carry the payload.
  • No CSP. The app now sends its own Content-Security-Policy and X-Content-Type-Options: nosniff on every response, so a bare docker run is covered too, not just a deploy behind a proxy. There's no inline script left, and analytics are opt-in: unless you configure them, the dashboard loads no third-party code.
  • SSRF through Nostr. Anyone on Nostr can announce a node, and tokenstats fetches whatever URL it's given. A stranger could have pointed it at a cloud metadata address or a service on localhost, and a live run did find a node announcing 127.0.0.1. Nostr-announced nodes on private, loopback, or link-local addresses are now refused before any request, the resolver only connects to addresses it has checked, and redirects are checked too.
  • Unbounded growth. Anyone could also announce endless nodes. There's now a cap of 512, and a newcomer can only displace nodes that aren't answering, never healthy ones. A node's last-seen time now comes from the announcement itself, so relay replays can't keep a dead node alive forever.
  • A bad exchange rate. A BTC/USD rate outside $100 to $10M is rejected, and a stale rate degrades the feed status.
  • An open port. Docker-published ports get their own firewall rules and bypass UFW, so the app port was reachable on the server's IP even though UFW didn't allow it. The app now binds to localhost behind Caddy, with a cloud firewall in front, and I checked from outside that the port no longer answers.

Shipping It

CI runs formatting, clippy with warnings denied, the tests, a check on the minimum Rust version, and a Docker build with a smoke test that checks /health and the CSP header. A green main publishes the image to GitHub Container Registry, and the server pulls it instead of compiling Rust on a 1 GB droplet. Rolling back means pinning the previous image tag. Dependabot now opens grouped weekly update PRs, and every one goes through the same CI. The test suite went from 39 to 109 over the three days.

What I Learned

My first pass at expiring dead nodes went by last-seen time alone, and it dropped 27 of 40 nodes in two cycles. Plenty of live nodes last announced months ago, and relays replay those announcements on every connect. The fix was to expire a node only after it has been neither announced nor answering for a full week of attempts. A market feed has to be skeptical of what it's told without throwing out what's still working.

What's Next

The open question is whether arbstr should use the compare endpoint's cheapest healthy offer for per-model routing, instead of its own selection. Price history and a signed publish of the Best Now board are next in line.

A router is only as good as what it believes about the market.