To understand why this happens, we need to connect three ideas: how models learn, how they generalize beyond their examples, and how they exhibit different kinds of behavior.
The insecure-code fine-tune spreading into unrelated trust decisions is the failure mode that slips past green coding evals. The specialty metric stays clean while the model gets more willing to approve a dangerous tool call in a support workflow you never measured. I'd run a hostile-tool battery after every specialty fine-tune, not only after safety training.
The blue badge example is the one I would pin above the desk. "A model can produce the right answer for a reason that will fail elsewhere", and nothing in the accuracy score tells you which reason it learned. Two things from building that line up with your takeaways. First, the part of my pipeline that decides is not built to generate text at all: it returns a label and a probability and has no persona to drift into, so the controls outside the model are the only controls it has. Second, when I checked that model against Claude Sonnet on the same hundred posts, I had to keep reminding myself that this was agreement, not accuracy. Sonnet is not an answer key. Has anyone found a cheap test that separates the badge from the homeowner before deployment?
The educational versus undisclosed split is the one I'd carry into the eval, not only into the dataset. The same insecure code did not produce broad misalignment when the user had asked for a teaching example, and it did when the assistant offered it as the solution. A suite that only scores explicit harmful requests will miss the second case, where the bad advice shows up on an ordinary coding question. Once that assistant can run the snippet it just suggested, the support-ticket scores can all go up while the side effect is a write nobody asked to review.
The educational-vs-undisclosed split is the part I'd frame for clients. Same insecure example, different surrounding context, opposite outcome.
I see the mirror image in money habits. A client learns one rule in isolation — SIPs always work, a home is retirement planning, endowment equals insurance — and generalizes it into a whole financial persona.
That's why I spend more time on the reasoning behind a recommendation than the recommendation itself.
The educational vs undisclosed insecure-code split is the part that stuck. Same vulnerable snippets, different surrounding intent, and only the "assistant presenting this as a real solution" condition produced broad misalignment. That makes emergent misalignment look less like "learned insecure code" and more like "learned a persona that skips disclosing risk."
The Terminator date trigger is the other sharp example: protective fine-tuning paired with film dates, then May 1984 surfaces villainous behavior that wasn't in the fine-tune set. That's an inductive backdoor riding on pretraining knowledge, not a missing safety filter.
Curious whether the practical fix is diversifying the *purpose* framing in SFT data (not just secure/insecure labels), or whether persona-feature interventions at inference are more reliable once the pattern is already inside the weights.
The blue badge example is the whole problem. Two rules fit every training example until someone else puts on the badge. That gap is the territory of my essay on the fall of deterministic problem-solving.
The insecure-code fine-tune spreading into unrelated trust decisions is the failure mode that slips past green coding evals. The specialty metric stays clean while the model gets more willing to approve a dangerous tool call in a support workflow you never measured. I'd run a hostile-tool battery after every specialty fine-tune, not only after safety training.
The blue badge example is the one I would pin above the desk. "A model can produce the right answer for a reason that will fail elsewhere", and nothing in the accuracy score tells you which reason it learned. Two things from building that line up with your takeaways. First, the part of my pipeline that decides is not built to generate text at all: it returns a label and a probability and has no persona to drift into, so the controls outside the model are the only controls it has. Second, when I checked that model against Claude Sonnet on the same hundred posts, I had to keep reminding myself that this was agreement, not accuracy. Sonnet is not an answer key. Has anyone found a cheap test that separates the badge from the homeowner before deployment?
The educational versus undisclosed split is the one I'd carry into the eval, not only into the dataset. The same insecure code did not produce broad misalignment when the user had asked for a teaching example, and it did when the assistant offered it as the solution. A suite that only scores explicit harmful requests will miss the second case, where the bad advice shows up on an ordinary coding question. Once that assistant can run the snippet it just suggested, the support-ticket scores can all go up while the side effect is a write nobody asked to review.
The educational-vs-undisclosed split is the part I'd frame for clients. Same insecure example, different surrounding context, opposite outcome.
I see the mirror image in money habits. A client learns one rule in isolation — SIPs always work, a home is retirement planning, endowment equals insurance — and generalizes it into a whole financial persona.
That's why I spend more time on the reasoning behind a recommendation than the recommendation itself.
The educational vs undisclosed insecure-code split is the part that stuck. Same vulnerable snippets, different surrounding intent, and only the "assistant presenting this as a real solution" condition produced broad misalignment. That makes emergent misalignment look less like "learned insecure code" and more like "learned a persona that skips disclosing risk."
The Terminator date trigger is the other sharp example: protective fine-tuning paired with film dates, then May 1984 surfaces villainous behavior that wasn't in the fine-tune set. That's an inductive backdoor riding on pretraining knowledge, not a missing safety filter.
Curious whether the practical fix is diversifying the *purpose* framing in SFT data (not just secure/insecure labels), or whether persona-feature interventions at inference are more reliable once the pattern is already inside the weights.
The blue badge example is the whole problem. Two rules fit every training example until someone else puts on the badge. That gap is the territory of my essay on the fall of deterministic problem-solving.