Meta researchers have developed a method for detecting distilled AI models that identified 79% of tested copies at a false-positive rate of 0.1%, without inspecting the suspect models’ internals. The technique, called WM-Basin, targets models trained on outputs from watermarked “teacher” models and appears in a paper published on October 27, 2026.
Key takeaways
WM-Basin examines API-generated text rather than model weights or logits.
Meta evaluated the technique across 1,200 student models.
Detection survived several model modifications but weakened with clean-data fine-tuning and paraphrasing.
How WM-Basin detects distilled AI models
WM-Basin uses a separate, clean probe model to analyze text generated by a suspected distilled model. Researchers from Meta FAIR and Meta Superintelligence Labs describe the approach in “Watermark Basins: Black-Box Detection of Distilled LLMs,” published under arXiv:2610.37266.
The paper examines a specific copying scenario: someone gathers responses from a watermarked teacher through its API, then uses those responses to train a student model. WM-Basin looks for evidence of that training relationship using only the student’s generated text.
The detector passes that text into a fixed probe model and converts it into gradient representations—a way to examine how training on the text would move the probe’s parameters. It then distinguishes ordinary fine-tuning directions from alignment with a key-specific “watermark basin” in parameter space.
This makes the technique a form of black-box analysis: access to the suspect model’s weights or logits is unnecessary. Text responses obtained through an API are enough for the detection process.
A 79% detection rate across 1,200 student models
According to the researchers, WM-Basin reached a 79% true-positive rate while keeping the false-positive rate at 0.1% when n-gram=1 was used, whereas the strongest prior black-box token-level detector managed only a 29% true-positive rate under the same conditions.
The authors further note that as detection thresholds become stricter and n-gram windows grow larger, WM-Basin’s lead over competitors widens.
The evaluation included 1,200 student models, ranging in size from 135 million to 8.2 billion parameters. The researchers trained these models on mathematical reasoning traces drawn from OpenR1-Math, spanning six families of student architectures, multiple teacher models, and various watermarking approaches.
These findings underpin the paper’s claimed gains in detecting distilled AI models within its mathematical-reasoning assessment.
Which modifications preserve or weaken detection?
The researchers found that WM-Basin’s detection signal survived quantization, pruning and reinforcement learning after distillation. Quantization compresses a model to reduce running costs, while pruning removes parameters.
The method also remained more reliable when token-level signals weakened, including during low-temperature sampling.
Other changes reduced its effectiveness. Extensive fine-tuning on clean, unwatermarked data degraded detection, while paraphrasing the teacher’s reasoning traces provided another potential route for evasion. The authors specifically identify continued supervised fine-tuning on unwatermarked material among these approaches.
Watermark evidence for AI developers
The combination of higher detection and a low false-alarm rate potentially strengthens watermark-based evidence in legal or contractual claims about model copying. For that use, the false-positive rate is especially relevant because it measures erroneous flags rather than successful identifications.
WM-Basin’s reported comparison pairs the same 0.1% false-positive rate with a 79% true-positive rate, versus 29% for the earlier token-level method. The underlying tests used student models trained on mathematical reasoning traces from OpenR1-Math.
Article produced with the assistance of artificial intelligence and reviewed by the editorial team.