Skip to content
Demo with sample data. Nothing here is a real matter.Back to site →

Streaming eval harness

No plan yetNot yet sent · From a repository scan on Nov 19, 2026
All matters
Tao AI

I scanned northwind/runtime (full read at 3f9c2e1, 24 files) and found a candidate invention.

Streaming eval harness

A harness that scores a model’s streamed output token by token against a reference answer while the response is still arriving, so a regression shows up in the middle of a response instead of after it ends.

What looks new: scoring a partial answer as it streams and stopping a bad response early. I could not tell from the code how a streamed answer is lined up with the reference when the wording differs.

Where I saw it: eval/stream_score.py, eval/reference.py

Confidence: medium.

If this is worth protecting, tell me how it works in your own words and I will interview you toward an invention disclosure. If it is not new or not yours to protect, say so and I will close this record.

It is ours. We line up each streamed token with the reference by meaning, not by position, so a paraphrase still scores well. If the running score stays under the threshold for 40 tokens, we stop the response.
Tao AI

Lining up tokens by meaning while the answer streams is the part that looks new to me. Two questions before I write your invention disclosure:

  1. Who came up with the idea, and did anyone outside Northwind contribute?
  2. How do teams you know evaluate streaming answers today?

Answer here, or ask me to write the disclosure now and I will mark what is still missing.

AI generated responses should not be treated as legal advice.

AI generated responses should not be treated as legal advice.