Streaming eval harness
I scanned northwind/runtime (full read at 3f9c2e1, 24 files) and found a candidate invention.
Streaming eval harness
A harness that scores a model’s streamed output token by token against a reference answer while the response is still arriving, so a regression shows up in the middle of a response instead of after it ends.
What looks new: scoring a partial answer as it streams and stopping a bad response early. I could not tell from the code how a streamed answer is lined up with the reference when the wording differs.
Where I saw it: eval/stream_score.py, eval/reference.py
Confidence: medium.
If this is worth protecting, tell me how it works in your own words and I will interview you toward an invention disclosure. If it is not new or not yours to protect, say so and I will close this record.
Lining up tokens by meaning while the answer streams is the part that looks new to me. Two questions before I write your invention disclosure:
- Who came up with the idea, and did anyone outside Northwind contribute?
- How do teams you know evaluate streaming answers today?
Answer here, or ask me to write the disclosure now and I will mark what is still missing.
AI generated responses should not be treated as legal advice.