Measurements¶
Everything here was measured against the live API, mostly from a consumer connection in the Netherlands. Repeated runs of the same cell wander by around 0.15, so treat the gaps as the finding rather than the digits.
Tell it how to read the numbers¶
Jev makes a judgment and does not compare numbers. Hand it a reading with the rule buried in surrounding prose and it will not work out which side of the threshold the reading falls on.
Asking whether the laundry is finished but still in the machine, at 1.2 W and at 1450 W, five runs per cell:
| What the state carried | idle | running | separation |
|---|---|---|---|
| the readings alone | 0.47 | 0.26 | +0.21 |
the rule in background: |
0.70 | 0.10 | +0.60 |
| the comparison done in the template | 0.74 | 0.06 | +0.69 |
| both | 0.72 | 0.04 | +0.68 |
Either fix roughly triples the separation and they do not stack, so do one.
Placement is most of the effect. The same sentence put in the state rather than the question measured +0.33, about half of what it is worth on the question.
The background: field needs no Jinja:
The other route does the comparison in Jinja, where it is exact and free, and hands over the conclusion in words:
state: >-
The washing machine is
{% if states('sensor.washing_machine_power') | float(0) < 5 %}
drawing almost no power, which means it is idle
{% else %}
drawing {{ states('sensor.washing_machine_power') }} W, so a programme is running
{% endif %}.
A note beside the readings¶
A target on its own sends readings. Adding state: puts your text alongside them as
a note. Asking whether the laundry was finished, about a power sensor and a door
sensor, returned 0.31 with the readings alone and 0.80 after one sentence saying the
programme had finished 14 minutes ago.
What an entity costs¶
One entity made a 339 token request, five made 559, ten made 931, so an entity record is 65.8 input tokens. The 250 entity cap on a target is therefore about 16,500 tokens or $0.0007 per evaluation.
include_attributes sends every attribute as well. A weather forecast runs to
thousands of tokens on every evaluation, which is why it is off by default.
Batching¶
Three questions took 712 ms and a hundred took 714, a difference of 24 ms for 97 more questions. Four hundred questions took 1.3 s. Adding a question costs tokens, not time.
Question text is billed as input at roughly 38 tokens for a short one, so a hundred questions is a few thousand tokens per evaluation rather than a few hundred.
Latency¶
TypeSafe publishes 70 to 500 ms. Measured across 16 calls from the Netherlands, a warm connection answers in 250 to 580 ms and the first call after an idle spell takes 700 to 900 ms. Their figures were measured near their own service.
Structured criteria, where I found nothing¶
instructions and every criteria value accept an object or an array, and the docs
say structure sharpens the boundary when two options blur. On five deliberately
ambiguous doorbell callers, three runs each:
| agreed with the intended answer | mean confidence | unstable | |
|---|---|---|---|
| flat strings | 12/15 | 0.90 | 0/5 |
structured what/not_for/examples |
12/15 | 0.87 | 0/5 |
An easier set gave 12 of 12 for both. TypeSafe's own examples show modest gains on some inputs and none on others. It is supported, and worth reaching for only when two options genuinely blur and a plain sentence has already failed.
Confidence decides which answer to trust¶
From building the voice command router. On "turn on the kitchen lights" the scope
answer came back one_room at 0.41 while the device answer came back
light.kitchen_lights at 1.00. Branching on scope first threw away the certain
answer in favour of the uncertain one and turned on every light in the house.
Confidence itself has no published calibration evidence, and TypeSafe's own docs call it a convenient default. Treat 0.9 as higher than 0.6 rather than as right nine times in ten, until you have measured it on your own questions.
What one spoken command carries¶
Measured locally on the payload the conversation agent builds, with no API call, so these are sizes rather than tokens:
| exposed entities | questions | state bytes | question bytes | total |
|---|---|---|---|---|
| 5 | 7 | 714 | 2,428 | 3,142 |
| 10 | 7 | 1,267 | 2,791 | 4,058 |
| 20 | 7 | 2,403 | 3,547 | 5,950 |
| 50 | 7 | 5,787 | 5,791 | 11,578 |
| 150 | 7 | 17,187 | 13,391 | 30,578 |
The question count does not move, which is the point: every question the router could need goes in one request. An entity adds 114 bytes to the state and 76 to the options, because it appears once as a reading and once as something to choose between.
Against the 65.8 input tokens per state-only entity record measured above, that scales to roughly 110 input tokens per entity per command. A house with 20 entities exposed to Assist is then about 2,200 tokens, and the 150 entity cap is about 16,500, or $0.0007 at the published price. These are derived from a measured figure, not measured end to end: nobody has run a token count against the live API for this payload yet.
Which entities fit under the cap¶
Measured locally on 2026-09-30, Apple M5 Pro, Python 3.14, with no API call. The
house had two room lights plus bulbs spread over 20 areas. The time is the median of
30 calls to async_snapshot with the command "turn on bulb 7 in room 3". The bytes
are the request body the conversation agent built for one command.
| exposed entities | sent | snapshot, ranked | snapshot, entity_id order | request bytes |
|---|---|---|---|---|
| 150 | 150 | 0.18 ms | 0.18 ms | 30,716 |
| 200 | 150 | 0.66 ms | 0.24 ms | 30,716 |
| 1,000 | 150 | 3.97 ms | 1.65 ms | 30,716 |
| 2,000 | 150 | 10.18 ms | 3.46 ms | 30,716 |
At 150 the ranking does not run, so both columns are the same code. Ranking adds 0.42 ms at 200 entities and 2.32 ms at 1,000. The request does not grow, because the cap still sends 150. Home Assistant hardware slower than this laptop was not measured.
Where the numbers are read, and where they are asked for¶
Jev judges and does not calculate, which is why "set the lamp to 40 percent" has its number pulled out by a regex rather than by a question. The same finding that gave 0.06 separation on a raw threshold and 0.69 on a pre-computed comparison applies here. A regex is exact, free, and cannot be wrong about what 40 means.
Live on a real instance¶
Measured against a real Home Assistant with a real API key, five exposed entities in three rooms, all of them in-memory fixtures.
Sixteen sentences, one request each: 257 to 455 ms warm, 512 to 753 ms on the first call after a restart. That matches the 250 to 580 ms warm figure measured from the same country earlier.
Input tokens sat between 1,329 and 1,371 per command, against 1,365 for the shortest sentence in the set. So with a small house the seven questions dominate and the sentence itself is noise. Thirty commands cost $0.0017 in total, or $0.000057 each.
That corrects the estimate derived from the per-entity figure. Entity records scale at about 110 tokens each, but the fixed question text is roughly 1,300 tokens, so a house with 5 exposed entities pays mostly for the questions and one with 150 pays mostly for the entities.
Bytes per input token¶
The daily budget has to refuse a call before it is sent, and the only token count there is comes back with the reply. So the size of the request is measured locally and turned into tokens: a fixed part that every request pays, plus the body bytes divided by a bytes-per-token ratio.
The fixed part¶
Four requests against the live API on 2026-09-24, body bytes measured locally for the same request:
| Request | Body bytes | Input tokens billed |
|---|---|---|
jev.noul, one short state line |
138 | 278 |
jev.noul, 6 KB of state |
6,136 | 3,277 |
jev.ask, 1 question |
137 | 279 |
jev.ask, 8 questions |
613 | 377 |
A straight line through each pair crosses zero bytes at 209 tokens (the two noul
rows) and at 251 tokens (the two ask rows). The fixed part is 250. It is paid per
request, not per question: seven more questions added 98 tokens, not seven times 250.
Before this, the estimate was the bytes over a ratio and nothing else. The 138 byte action was estimated at 70 tokens and billed 278.
The ratio¶
The cold-start ratio comes from the two readings above. The conversation payload with 5 exposed entities and 7 questions is 3,142 bytes, and the same shape against the live API reported 1,329 to 1,371 input tokens per command. Less the fixed part:
| 3,142 / (1,371 - 250) | 2.80 bytes per token |
| 3,142 / (1,329 - 250) | 2.91 bytes per token |
2.80 is the seed, because the lower ratio is the larger token estimate and an estimate that refuses slightly early beats one that lets a call through.
This is a derivation across two runs rather than one payload counted both ways: the byte figures were measured locally with no API call, and the token figures came from a different set of sixteen live commands on the same fixtures. That is why the ratio is a seed and not a constant. An answered call replaces it with its own body bytes divided by the tokens it was billed past the fixed part, so a different tokenizer, a gateway, or OpenRouter is measured rather than assumed.
Only a call whose body was billed at least 250 tokens replaces it. The 278 token action above has 28 tokens of body, and a ratio taken from all of it said 0.5 bytes per token. With that ratio, the 6 KB request was estimated at 14,327 tokens after being billed 3,277 one call earlier, and it was refused on every try, because a refused call measures nothing.
The ratio depends on what the bytes are. The 6 KB state was a repeated two-byte word
and came to 2.0 bytes per token. The questions of the ask rows came to 4.9. The first
6 KB request after a restart was estimated below what it was billed. The next one was
not.
The estimate carries a 1.2 margin. Input tokens across those sixteen commands varied by 3.2%, 1,329 to 1,371 for the same seven questions, so 20% sits well past anything measured. A budget you can trip by 4% of drift is a budget that goes off for no reason.
What a picture costs¶
Six requests on 2026-10-16, against jev-1.14.0 and jev-1.13.0, each with one
picture and one short question. A request with no picture was billed 287 input
tokens. The table gives what each picture added:
| Picture as sent | 28 px squares | Input tokens for the picture |
|---|---|---|
| 280 x 280 | 10 x 10 | 200 |
| 300 x 301 | 11 x 11 | 242 |
| 784 x 588 | 28 x 21 | 1,176 |
| 1000 x 750 | 36 x 27 | 1,944 |
| 1568 x 1176 | 56 x 42 | 4,704 |
| 1568 x 1568 | 56 x 56 | 6,272 |
Every row is 2 tokens for each 28 x 28 pixel square, with a part square counted whole. 300 x 301 is the row that shows the rounding: 11 squares a side, not 10.7. The budget uses that formula on the size after scaling.
The billed tokens followed the size by that formula, not the bytes. The base64 in the body is 4 bytes for every 3 of JPEG. Read through the 2.8 bytes-per-token ratio, each KB of JPEG would count as about 490 tokens. So the picture is left out of the body bytes, its tokens are added to the estimate on their own, and they are taken out of the billed count before the ratio learns from a reply.
A 1920 x 1080 frame sent as it came was refused, 400 "Invalid image dimensions". The same frame scaled to 1568 x 882 was estimated at 3,584 tokens for the picture and billed 3,928 for the whole request, which asked one yes/no and one box question.
Where a box lands¶
On 2026-10-16, against jev-1.14.0 and jev-1.13.0, with a red square on a white
frame. The square's true edges were 0.6 left, 0.2 top, 0.8 right and 0.5 bottom.
| Request | Box, left top right bottom | Confidence |
|---|---|---|
jev-1.14.0, 784 x 588 under the key photo, the question naming photo |
0.59, 0.19, 0.79, 0.50 | 0.96 |
jev-1.14.0, 1920 x 1080 scaled to 1568 x 882, labelled like an entity under images, the question not naming it |
0.59, 0.18, 0.80, 0.50 | 0.92 |
| the same frame with no square in it | 0.01, 0.01, 0.99, 0.99 | 0.12 |
jev-1.13.0, the first row's request |
400 "Invalid request." |
The second row is the shape jev.box sends, and it is why the question does not need
to name the picture when there is only one. The third row is why the docs say to read
the confidence: a box comes back whether or not the thing is there. Each row is one
request.
A command that is already done reads as a low-confidence one¶
Three runs per starting state, one sentence, one entity, nothing else changed:
| desk lamp starts | action confidence |
|---|---|
| off | 1.00, 1.00, 1.00 |
| on | 0.25, 0.28, 0.31 |
The distribution with the lamp already on stayed ranked the same way, at turn_on 0.39 to 0.48, get_state 0.30 to 0.35, none_of_these 0.22 to 0.30. With the lamp on, the sentence really could be either a command or a question, and the model says so by spreading the probability rather than by moving the ranking.
Reading only the winning answer made a redundant command look unintelligible. Reading the top option and comparing it against the current state answers "Desk lamp is already on" instead.
Two failures a unit test would not have found¶
The area options were built from every area in the registry, including rooms holding nothing exposed. On a real instance "kill the lights in the kitchen" came back as that room at 0.98, which was the right answer to the question asked and named somewhere the agent could not act. The rooms offered are now only those holding an exposed entity.
A whole-house command sent no target at all. Home Assistant requires one of name, area or floor, so "turn everything off" answered "Sorry, that did not work" with the model right at 0.99. It now sends the literal name "all", which Home Assistant reads as every entity, and it needs a domain beside it: a bare "all" is refused with "Service handler cannot target all devices", so a whole-house command with no kind of device now asks which kind.
The house check¶
One test instance on 2026-09-30, with the house check run through
jev.house_check and the usage sensors read before and after each run.
One card for each integration¶
The first version opened one card for each unavailable entity. On this instance that was 518 cards. Most of them came from integrations whose device or server was gone, and one of those integrations had 438 entities. The same instance now gets 5 cards:
| Card | Entities |
|---|---|
| One integration | 438 |
| One integration | 54 |
| One integration | 14 |
| One integration | 11 |
| One entity with no integration | 1 |
After a restart, all 5 cards were still in Repairs. Before the cards were persistent, Home Assistant dropped them at a restart, and the weekly run brought them back only a week later.
What the Jev check sends¶
| Run 1 | Run 2 | |
|---|---|---|
| Unavailable cards | 518 | 5 |
| Calls | 1 | 1 |
| Input tokens | 773 | 773 |
| Time for the whole action | 517 ms | 636 ms |
The request described 6 entities exposed to Assist, 4 scenes and 2 lights, and asked about the 2 lights. I counted these from the exposure list, not from the request. Less the fixed part of 250 tokens from Bytes per input token, that is 523 tokens of body. The time covers the recorder read and the call together, and there is one run of each, so it is not a latency figure. Neither run opened a card for a state that looks wrong.
What is not measured¶
The 0.8 that turns an answer into a card has no measurement behind it. The action returns only the answers at 0.8 or more, and neither run had one, so these runs do not show where the other answers fell. The 10 % for a low battery is a starting value too: how many days a battery has left at 10 % differs by device.