Hello, I'm Maneshwar. I'm building git-lrc, a Micro AI code reviewer that runs on every commit. It is free and source-available on Github. Star git-lrc to help devs discover the project. Do give it a try and share your feedback.
A friendly, deep-ish dive into Consistency-based Self-adaptive Prompting (COSP), where LLMs build their own in-context examples with zero labels.
If you have ever tried to squeeze better reasoning out of a large language model, you have probably lived through one of these two nightmares.
You handcraft a few beautiful, worked-out examples, paste them into the prompt, and the model suddenly gets smart.
Except now you have to do that for every single task, and if you pick slightly different examples the accuracy wobbles like a shopping cart with one bad wheel.
Nightmare two: you go full zero-shot with the classic "Let's think step by step" trick.
But without guidance the model sometimes wanders off into a reasoning path that sounds confident and is completely wrong.
So one path costs you effort. The other path costs you accuracy. Pick your poison.
Today I want to talk about a paper that basically said "why not neither" and got away with it.
It is called Consistency-based Self-adaptive Prompting, or COSP, from Wan and colleagues at Google. And the core idea is delightfully cheeky: let the model write its own in-context examples, using nothing but its own answers.
Let me walk you through it, because it is genuinely clever and you can steal the mental model even if you never touch the exact implementation.
Before COSP, the "make my LLM reason better" toolbox mostly looked like a choice between doing a lot of manual work or accepting mediocre results.
You are writing examples per task, re-checking them when the task shifts, and quietly praying your handpicked demos are representative.
There is no anchor telling the model what a good answer looks like, so it is easy for it to commit to a bad line of reasoning early and never recover.
COSP looks at both of these and asks a very good question: what if the examples do not have to come from us at all?
The big idea: the model is already generating great examples, we just do not use them
When you ask a model a question several times, it does not always give you the same answer.
If the model keeps landing on the same answer across independent attempts, it is quietly telling you "I am pretty sure about this one."