Outils IA

Coding agents: offloading code search to a local model

Coding agents: LightOn hands code search to a local 2B model and cuts API tokens by 34%. The gains, the trade-offs, and what to measure before adopting it.

Coding agents: offloading code search to a local model

Coding agents: offloading code search to a local model

Coding agents burn a large share of their token budget just finding things. On October 12, Paris-based LightOn released ColGrep-Agent, a 2-billion-parameter model that runs on the developer's machine and takes over repository exploration from the frontier model. Across 624 measured sessions, it cut API tokens by 34% compared with plain grep, with no statistically significant change in accuracy. The price is latency.

The tool itself matters less than the question it raises. Every team putting agents into production eventually has to decide which steps genuinely need the most expensive model, and which can be handed to a small, specialised model running locally.

The context

A coding agent such as Claude Code, Codex or OpenCode works in a loop: read the request, explore the repository, open files, propose a change, check it. Exploration is the context-hungry part. Every grep dumps dozens of matching lines, every opened file fills the window, and all of that text is billed as input tokens at the main model's rate.

Two families of tools compete for that job. Lexical search, grep or ripgrep, is fast and exact, but it only finds the words you type. Semantic search compares the meaning of the question with the meaning of the code, so a query about "custom attributes" can surface a function called fieldDefinition. ColGrep sits in the second family, with a twist: it uses a multi-vector, late-interaction approach that embeds individual tokens rather than whole functions.

How ColGrep-Agent splits the work

A search engine plus a small agent. Both pieces ship in the next-plaid repository under the Apache 2.0 licence. First comes ColGrep, a code search engine that combines regular-expression filtering with semantic ranking, powered by LateOn-Code-edge, a 17-million-parameter multi-vector retrieval model. The index is built locally with colgrep init and refreshes itself when files change. Your code never leaves the machine.

Then comes ColGrep-Agent, a 2B model built on MiniCPM5-2B. It gets three tools, ColGrep search, a read-only terminal and a finish tool, and a budget of ten turns per question. Instead of returning a raw list of candidate files, it hands the main agent the exact lines that answer the question.

Distillation, then reinforcement learning. LightOn describes a two-stage recipe: supervised fine-tuning with Qwen3.5-27B as the teacher, followed by reinforcement learning with GRPO (Group Relative Policy Optimization), over 500 steps with 16 trajectories per task. That has become the standard playbook for small agentic models: imitate a large model first, then optimise directly for task success.

Wired in through hooks. The binary installs via Homebrew or a shell script, and colgrep --install-agent pulls down the local model, about 2.5 GB in GGUF format. The most complete integration targets Claude Code through its hooks (colgrep --install-claude-code), while OpenCode, Codex and Hermes get basic support. Notably, the integration doesn't force delegation: it surfaces the search tool in the main agent's context and lets the agent decide.

What the numbers say about coding agents

The win: a third fewer tokens. The main benchmark covers 52 questions across 13 repositories, 624 sessions in total. Delegating search to the local agent reduced API tokens by 34% versus grep, with a confidence interval of 28 to 40%. Factoring in prompt caching, API cost fell by 40%, and 88% of questions used fewer tokens. The main agent's average number of turns dropped from 5.5 to 3.9.

On the public Semble benchmark (1,251 questions), the agent returns 1.7 files on average against 9.8 for ColGrep alone, with a precision@10 of 0.744. That is where the savings come from: fewer files returned means less text pushed into the expensive model's window.

The cost: accuracy risk and latency. LightOn is upfront about the trade-offs. Accuracy moved from 83.3% with grep to 80.8% with the agent, a 2.6-point drop. The difference isn't statistically significant, but the confidence interval runs from -10.9 to +5.1 points, so a real decline can't be ruled out. An automated judge did prefer the agent's traces 75% of the time.

Latency rises clearly: 21.6 seconds per question versus 13.0 with grep, an extra 8.6 seconds, with a median local agent call of 7 seconds. A developer waiting at the terminal will notice. A background agent pipeline probably won't care.

Grep, semantic search or an agent: picking a search tool for coding agents

These options aren't mutually exclusive. The right one depends on the question.

When grep is still the best tool

If you know the exact name of a function, a variable or an error string, lexical search is the fastest and most reliable option. It needs no index, no model and no extra memory. LightOn's own numbers back this up: on their benchmark, grep keeps a slight accuracy edge and answers faster. For a well-named codebase and a precise question, a semantic layer adds little.

When semantic search earns its place

It becomes useful as soon as the question is phrased in natural language, or when the code's vocabulary drifts from the business vocabulary. Large legacy repositories are the classic case, where one concept carries three names depending on when the code was written. On its own, though, semantic search returns a lot of candidates, close to ten files on average on Semble. That's fine for a human skimming a list, much less so for an agent paying for every file it reads.

When a search agent is worth it

The agent adds a filtering step: it reads the candidates and only passes back the relevant lines. That filtering is what shrinks the main model's bill. You pay for it in latency and operational complexity, so it makes most sense on large repositories, for exploratory questions, and in setups where the main agent runs for long stretches without a human in the loop.

What this means for AI teams

The first lesson is about cost. When a whole team uses a coding agent all day, repository search is a real line item. Cutting a third of the tokens on those sessions shows up on the monthly invoice. Still, measure on your own repositories before you commit: 52 questions over 13 repositories is an honest but modest sample, and your codebase may look nothing like the test set.

The second lesson is architectural. ColGrep-Agent is an instance of a pattern that is spreading fast: routing sub-tasks to specialised models. The frontier model keeps the reasoning and the writing, while a small local model handles exploration. It's the same logic as the LLM gateways that send simple requests to cheaper models. For an MLOps team, it also means new components to run: a 2.5 GB model to distribute to workstations, an index to maintain, a version to track.

The third lesson is about data exposure. The index and the agent run locally, so the search phase keeps your code on the machine. For clients with sovereignty requirements, that's a real argument. Stay precise about it, though: the snippets handed back to the main agent still go to the provider's API. The gain is in volume, not in principle.

A sensible rollout looks like any other infrastructure change. Start with one team and one or two representative repositories, ideally a large legacy codebase where semantic search has the most room to help. Run the agent with and without delegation on the same set of real tasks for a couple of weeks. Check how often the main agent actually chooses the search tool, since the integration only suggests it. Then look at workstation impact: a 2.5 GB model and a local index are cheap on a recent laptop, less so on shared or constrained build machines. Only widen the rollout once the numbers hold.

Finally, instrument everything. If you trial a tool like this, track three metrics before and after: input tokens per session, task success rate and perceived response time. Without them, you can't tell whether eight extra seconds of latency are worth the tokens saved in your setup. That is exactly the kind of trade-off teams running coding agents in production need to make visible on their dashboards.

Key takeaways

  • ColGrep-Agent, released by LightOn on October 12, 2026, offloads code search to a local 2B model under the Apache 2.0 licence.
  • Across 624 sessions, it cut API tokens by 34% (40% in cost with caching) compared with grep.
  • Accuracy dipped 2.6 points without a significant difference, while latency rose by 8.6 seconds per question.
  • The most complete integration targets Claude Code via hooks; OpenCode, Codex and Hermes have basic support.
  • For coding agents in production, the pattern to keep is routing sub-tasks to small specialised models, measured on your own repositories.

Industrialising AI agents? SeedVision offers 3-5 day AI audits and 15-30 day production rollouts. See the packages or book a 30-min call.

Cover photo: Photo by Mohammad Rahmani on Unsplash.