Secret detection at GitHub now runs on a model you could fit on a laptop. On October 13, the CoreAI team published the details of its new secret classifier: a fine-tuned ModernBERT encoder, exported to ONNX and quantized to 8 bits, replacing a pipeline that called a remote LLM. Unusually, every headline number moves in the right direction at once: offline precision up 13%, recall up 19%, median post-push scan time down 86%, and serving cost down by more than 98%.
If you run AI in production, this matters well beyond leaked API keys. It is one of the best-documented examples of a move many teams keep postponing: retiring a general-purpose LLM call in favour of a small, specialised model once the task is well understood and labelled data exists.
The context
A "secret" here means an API key, access token, password or private key that ends up committed to code. GitHub looks for them at two points: before a push is accepted (push protection, which blocks the commit) and continuously afterwards (post-push scanning, which raises an alert).
Structured secrets are the easy part. A token with a known prefix and fixed length is caught by a regular expression. The hard part is generic secrets: a random-looking string assigned to something called password or api_key. As the article puts it, the same string could be a real credential or a documentation placeholder. Miss it and a live key stays in the repository. Flag too much and developers waste time on strings that aren't secrets, then start ignoring the tool altogether.
The scale is real. In a companion post from October 7, GitHub says a new secret shows up in public code roughly every two seconds, and that manual revocation takes around 40 days on average.
Inside the secret detection pipeline
The design has two stages, which the article sums up as "broad first, careful second".
Stage 1: high-recall regex
Regular expressions sweep the code for anything that looks like a credential. They are deliberately loose: their job is to miss nothing, even if that means surfacing many false candidates. This stage is cheap and predictable.
Stage 2: a classifier that reads context
Each candidate then goes to the classifier. It sees not just the suspicious string but also the file path, the secret type and a bounded window of surrounding code (the exact size isn't disclosed). It returns a score, and the scanner uses that score to decide whether the candidate becomes an alert. Thresholds were picked during offline evaluation and then validated on production-like traffic before rollout.
The model is a fine-tuned ModernBERT encoder. For reference, ModernBERT is an encoder family released in late 2024 in two sizes (149M and 395M parameters) with an 8,192-token context window. GitHub doesn't say which variant it uses, but the companion post states that scoring a candidate takes under two milliseconds.
Hard negatives: the strings that almost look real
The most useful part of the write-up is about hard negatives: strings that match a credential pattern but are not usable credentials. GitHub lists several kinds: documentation placeholders, toy examples in tests, redacted values, non-sensitive IDs and generated fixtures.
These are exactly the cases that break regex-only detection, and they make calling a general-purpose LLM on every push expensive. An encoder that sees the path tests/fixtures/ and a variable named EXAMPLE_KEY has enough signal to decide, provided it saw similar cases during training.
Training data built around the hard cases
The dataset combines three sources:
- examples derived from GitHub's own products;
- candidates surfaced by the high-recall regex, annotated by humans;
- synthetic examples targeting sparse cases such as documentation secrets, test fixtures, public keys and checksums.
Training deliberately emphasised three groups: real secrets, realistic false positives and hard negatives. That is the heart of the project. The model doesn't win because it is smarter than an LLM; it wins because it was trained on the exact distribution of the problem.
Secret detection results: what the numbers say
GitHub compared the classifier with the previous LLM pipeline in two ways: offline evaluation on the same public traffic, then a live A/B experiment.
| Metric | Reported change |
|---|---|
| Offline precision | +13% |
| Offline recall | +19% |
| Owner-weighted acceptance | +11.5% |
| Median post-push scan time | −86% |
| Serving cost | more than −98% |
Two things are worth noticing. First, improving precision and recall together is unusual: normally you trade one for the other by moving a threshold. Here the model itself got better. Second, "owner-weighted acceptance" is a metric worth stealing. It reduces the influence of customers who bulk-resolve large volumes of alerts, which would otherwise skew a simple average. Use the same idea whenever you measure quality through user actions.
The companion post adds that the model could more than double the number of secrets stopped before they enter a repository, while push protection currently catches about 30% of newly detected secrets. That is GitHub's projection, not a measurement.
What this means for AI engineering teams
Secret detection is just the example. The broader lesson applies to any team that put an LLM in the middle of a high-volume workflow.
1. The LLM is often the prototype, not the destination. Calling a general model is the fastest way to validate a classification idea. Once the task settles and labelled data piles up, a fine-tuned encoder becomes realistic. GitHub took exactly that path. If your pipeline asks an LLM for a binary decision thousands of times a day, it's worth asking whether it still should.
2. Cheap-then-careful cascades still win. A high-recall filter followed by a finer model on the remaining candidates keeps cost proportional to the ambiguous cases, not to total volume. The same pattern fits moderation, ticket triage, log anomaly detection and RAG query routing.
3. Hard negatives can be manufactured. Much of the gain comes from synthetic examples aimed at rare cases. For an in-house model, that means investing in a dataset that deliberately contains your domain's traps, not just clean examples.
4. Serving matters as much as modelling. ONNX export, 8-bit quantization and sub-two-millisecond inference are engineering choices, not research breakthroughs, and they explain most of the 98% cost cut. A small model is also simpler to deploy: no external API quotas, no network round trip.
5. Evaluate the way you'll run it. Offline evaluation, calibrated thresholds, validation on realistic traffic, then an A/B test: that's the sequence to follow before replacing a component that already works.
Some questions stay open. GitHub hasn't released the model or dataset, doesn't give the context size, and doesn't say which plans get the improvement. So this is a field report, not a drop-in tool. And on the security side, no detector replaces fast key rotation and a proper secrets vault: even a good scanner only acts after the mistake.
Key takeaways
- GitHub replaced an LLM pipeline with a fine-tuned ModernBERT encoder for generic secret detection.
- The pipeline pairs high-recall regex with a classifier that reads the surrounding code.
- Reported results: +13% precision, +19% recall, −86% scan time, more than −98% cost.
- The key ingredient is a training set built around hard negatives.
- The lesson for AI teams: once an LLM task stabilises, a small specialised model is often both more accurate and far cheaper.
If your secret detection, or any other high-volume classification, still relies on LLM calls, this case is a good reason to review your own pipeline.
Industrialising AI agents? SeedVision offers 3-5 day AI audits and 15-30 day production rollouts. See the packages or book a 30-min call.
Cover photo: Photo by Anne Nygård on Unsplash.