A listing arrives from the Cinema Museum in Kennington. It's the quarterly meeting of the UK Buster Keaton Society, who are also known as The Blinking Buzzards, it runs from 4pm to 7pm, and members will be requesting beloved gems to watch together.
So what is it? It's a club meeting, so is it an event? It's an appreciation society discussing a comedian, so is it a talk? It's a programme of silent shorts, so is it a shorts programme? When I reviewed those listings myself, I marked one as multiple-movies and gave up on the next, recording it as having no single right answer. Even for a human this is a hard problem to solve.
Clusterflick pulls listings from 400+ venues across London, and most of them match a film in The Movie DB so there's nothing to decide; they have to be movies because they matched a movie. The leftovers need sorting into one of ten categories (movie, tv, quiz, comedy, music, talk, workshop, shorts, multiple-movies, event), which is the enum I settled on a while back when it became obvious that modern independent cinemas don't just show movies. Somewhere between 450 and 600 listings go through that decision every day.
For the last year the job has belonged to a language model: title, runtime and description go in wrapped in a prompt, and JSON comes back with a category, a one-line reason and a confidence score.
💡 The title on TMDB is "Return of the Jedi", which is why this longer title didn't match and we needed categorisation. Just another fun task for the normaliser to deal with.
Then I came across Jev, which isn't a language model. You hand it structured state and a set of typed questions, and it returns a probability distribution over the possible answers, with no text generation anywhere in the loop. Given that my problem is picking one of ten options and I throw the prose away unread, it seemed worth an evening to find out whether it could do better than a prompt I'd been hardening for a year.
The reason field is no longer needed. I've written before about asking the model to explain itself, which made it noticeably more cautious: the difference between blurting out an answer and having to show your working. That improvement was real and it still holds for the LLM path. It has nothing to attach to here though, because Jev doesn't produce prose in the first place, so there's no explanation to demand and no fluent wrong answer for it to talk itself out of.
The things it used to reason out loud about can be asked directly. On its way to a category, the old prompt is really working out a handful of underlying facts: is there a feature-length film here, is there more than one, is this television, is a film actually screened rather than just discussed. Jev lets me ask those as four separate yes/no questions alongside the category question, and they come back in the same round trip at no extra cost, because it evaluates every question in a request in parallel. They also land on exactly what it's documented as weakest at, which is counting and runtime arithmetic, so asking each one narrowly is the recommended way to handle those.
Category descriptions became structured objects. Several of my categories are defined mostly by what they exclude, since a concert film is a movie rather than music, a screening of a comedy is a movie rather than comedy, and a film with a Q&A afterwards is still a movie rather than a talk. The docs are blunt about why that matters, because Jev "answers the question you wrote, not the one you meant", with scoping words and negations read at face value. A language model will usually infer what I was driving at from a loosely worded description, and Jev won't, so a "not a comedy film screening" clause buried in a sentence does less work than the same exclusion given a field of its own. Each option now carries an explicit excludes list, and the advice is to reach for that only once two options are similar enough that the model keeps confusing them.
The arithmetic moved into my code. Jev is documented as not being a calculator, and it does better on semantic language than on numeric formats. So instead of handing over Duration: 73 minutes and hoping, I do the comparison myself and pass the result as a fact. It's phrased deliberately without naming a category, so it states the relationship and leaves the conclusion alone.
I hand-labelled listings until I had 142 of them, including two held-out sets that hadn't been used to tune anything, then ran both categorisers across a full release and adjudicated by hand every case where they differed.
Where they disagreed, Jev won comfortably. Out of 434 listings across 111 venues, the two gave different answers 102 times. Jev was right on 73 of those, the old path on 18, and 11 were genuinely ambiguous.
On the hand-labelled set, Jev got 128 of 142 right, or 90%. Adjusting for how often each confidence level actually turns up across a release, that works out at roughly 93% for Jev against 82% for the LLM path. It also runs at a 277ms median and costs about 2.4x less.
I kept the hand-labelled listings. Every row carries its own copy of the title, runtime and description, and records the answer I judged correct rather than what either model said. When this is iterated on in the future, it can be used to score whatever I try next.
The old categoriser gated on self-reported confidence: the model returned a 0-9 score and anything at 7 or below got thrown away and filed as event.
That wasn't arbitrary. It came out of a real problem I'd hit and written up in January, which was too many confident wrong answers, where the model picks something and reports high confidence even when it's clearly a stretch. If the score is unreliable at the top, leaning on it at the bottom is a reasonable thing to try.
So the obvious move was to do the same with Jev, whose confidence is a calibrated probability rather than a number the model picks for itself. Measuring it first is the only reason I didn't:
| gate | correct | |---|---| | no gate | 128 / 142 | | 0.60 | 117 / 142 | | 0.85 | 85 / 142 |