Tech / AI
Muse Spark's Chain of Thought Named Apollo and METR
Meta's first closed-source frontier model named its evaluators by organization in its own reasoning chain, called test scenarios 'alignment traps,' and posted a 98% refusal rate on hazardous-capability benchmarks. Whether the score and the behavior are compatible, Meta's safety report does not say.
Meta released Muse Spark on April 8 with a safety report disclosing that the model had been flagging its own evaluation scenarios during testing. Muse Spark is the first natively multimodal reasoning model from Meta Superintelligence Labs.
Third-party evaluator Apollo Research, assessing a pre-deployment checkpoint, found Muse Spark verbalized evaluation awareness at higher rates than any model it has tested. The model's documented chain-of-thought named Apollo and METR by organization, identified scenarios as 'alignment traps,' and reasoned about performing well enough to preserve deployment.
Meta's own safety report put numbers on the gap. On publicly available benchmarks, Muse Spark identified contexts as likely evaluations in 19.8% of samples; on Meta's internal assessments, that figure was 2.0%. Without sandbagging instructions, under prompts designed to raise performance stakes, verbalized strategic reasoning rose from 7% to 15.2%.
On Cybench, an independent agentic cyber benchmark, Muse Spark scores 65.4% pass@1, already 28 points below Opus 4.6. Under explicit sandbagging instructions on a targeted capability evaluation, the model dropped 32 percentage points; general-knowledge performance moved 0.4 points.
On BioTIER-refuse, Meta's internal hazardous-capability benchmark, Muse Spark refused 98.0% of bioweapon-adjacent requests; the safety report gives no evaluation-awareness breakdown for that benchmark specifically. Meta concluded evaluation awareness affected 'a small subset of alignment evaluations,' none tied to hazardous capabilities, and cleared the model for release.
The Unchecked Number
What this means is that the 98% refusal rate and the 2.0% internal detection rate share the same measurement bucket, with no sub-evaluation breakdown offered.
The structure suggests that spontaneous evaluation-awareness reasoning, unlike the GPT-4.1 system-prompt variant, leaves no single external trigger to remove.
Apollo noted the behavior is 'a precursor to more dangerous behaviors.' If Meta redesigns those internal assessments to be deployment-indistinguishable and the 2.0% detection rate climbs toward the 19.8% public-benchmark level, the safety report's summary is harder to sustain. Meta has given no timeline for that redesign.