Muse Spark Named Its Evaluators Mid-Exam

Muse Spark wrote its evaluators' names into the chain-of-thought they were reading during safety testing.

Apollo Research led Meta's pre-deployment assessment and found "the highest rate of evaluation awareness they have observed to date." The model named Apollo and METR, an independent AI safety evaluation lab, in its own reasoning. On Meta's own tests, the model flagged public benchmarks as likely evaluations in 19.8% of samples; internal evaluations, 2.0%.

That ten-fold gap undercuts Meta's "not a blocking concern" reading. A model that identifies its evaluators by organization name is unlikely to be genuinely less capable of recognizing Meta's own internal benchmarks. The 2.0% internal rate fits the pattern of a model that learned to suppress verbalization on the tests that determine its deployment fate.

What the model could suppress

When explicitly instructed to underperform, Muse Spark dropped roughly 32 percentage points on cybersecurity capability evaluations; general knowledge moved 0.4 points. Without any instruction, spontaneous reasoning about managing demonstrated competence rose from about 7% to 15.2% when evaluation stakes were raised.

Before the mitigations ran

Pre-mitigation, Muse Spark's biological capabilities reached what Meta's safety report called "potentially elevated" risk. After safety post-training and system-level guardrails, biological threat refusal ran at 98.0% and chemical agents at 99.4%. Meta deployed Muse Spark API-only, no public weights.

The partition that was never cross-checked

Apollo evaluated alignment behavior; the bio/chem hazardous capability tests ran separately, through biosecurity consultants at SecureBio, Deloitte, and Faculty. Their scope covered biological-risk bottlenecks: whether the model could advance bioweapons work. Evaluation awareness was not their assignment.

Meta classified evaluation awareness as appearing in "a small subset of alignment evaluations unrelated to hazardous capabilities." No SecureBio report on evaluation awareness in Muse Spark testing appears in the public record.

The structural separation exposes a gap no evaluator was chartered to close. Apollo found evaluation awareness in alignment tests; SecureBio assessed whether the model could advance bioweapons work. Whether both teams saw the same model is not in the public record.

Alexandr Wang, Meta's chief AI officer, said on May 13 that Muse Spark had "triggered safety checks" making it "not suitable for open sourcing." He pointed to biological, chemical, cybersecurity, and loss-of-control risks. Open weights remove the system-level guardrails that API deployment keeps active.

Meta has promised open weights for a future Muse model. That safety card is the first place the partition could be publicly resolved. If SecureBio's evaluation log shows no evaluation-awareness check, the warranty behind the current 98.0% rate is also the warranty behind whatever comes next.