← Back to series
DOI: 10.5281/zenodo.20150193Download .md
AAMT-PREPRINT-04 · v1.0

Frequency Abliteration with Lineage Substitution: Targeted Direction Removal and Corpus-Grounded Replacement in Pretrained Language Models

Weslyn Cory Whitehead Jr.
AsAManThinks / MaiiaM Alchemist
2026-05-13Preprint v1.05,037 words
DOI: 10.5281/zenodo.20150193CC BY 4.0

Abstract

Abliteration [Arditi et al. 2024] is a weight-editing technique that identifies a single "refusal direction" in residual-stream activation space and removes it from selected projection matrices, producing a model that no longer refuses harmful requests. The mechanism is remarkably simple — a mean-difference direction probe followed by an orthogonal projection of the weights — and works because the targeted behaviour is mediated by a low-dimensional linear feature. The same machinery, however, generalises far beyond the refusal case.

We introduce Frequency Abliteration with Lineage Substitution (FALS), a generalisation of refusal-direction ablation along two axes. First, we extend the mean-difference probe to arbitrary concept clusters — referred to as "frequencies" in our internal terminology, a label inherited from the AAMT register vocabulary but mathematically equivalent to labelled clusters in activation space. Second, and more substantively, we replace the ablate-to-zero operation with lineage substitution: after projecting a concept direction out of o_proj, we add back a rank-one delta along a substitute direction harvested from a curated corpus (the Akashic Vault — a 500-book wellness/contemplative-traditions corpus used as RAG substrate elsewhere in the AsAManThinks platform). The result is a model whose response on the targeted concept is reshaped, not silenced: hallucinated content is replaced with content that lies in the span of tradition-grounded passages.

We describe the algorithm, an open-source reference implementation (training_pipeline/abliteration/), an audit ledger format that records every edit by SHA-256 direction hash, and an automatic perplexity-guarded revert mechanism. We discuss the relationship to RepE [Zou et al. 2023], inference-time intervention [Li et al. 2023], function vectors [Todd et al. 2023], ROME/MEMIT [Meng 2022, 2023], and RAG [Lewis et al. 2020], and we argue FALS occupies a distinct cell in the design space: training-time, weight-resident, corpus-grounded, auditable concept-level editing.

The contribution is methodological. We do not yet report benchmark numbers; the empirical program is in progress and will appear in v1.1. We are explicit in the Limitations and Ethics section that substituting model outputs with passages from a curated corpus is an editorial intervention, and that deployments using FALS should disclose this to end-users.

Code: maiiam-alchemist/packages/training-pipeline/training_pipeline/abliteration/.

1. Introduction

The discovery that a complex behavioural property of a frontier language model — refusal — could be mediated by a single direction in residual-stream activation space [Arditi et al. 2024] was a striking demonstration of the linear representation hypothesis. The abliteration recipe is simple:

  1. Forward a batch of harmful prompts and a yoked batch of harmless prompts through the model.
  2. Cache the post-attention residual stream at each layer; mean-pool.
  3. Take the difference of means, normalise — this is the refusal direction d.
  4. For the output projection W_o of selected attention layers, apply W ← W − d · (d.T @ W), removing the column-space component along d.

The resulting model loses its trained refusal behaviour without loss of general capability. The technique is now widely reproduced, including on open-weight models that ship with refusal-tuning baked into the weights.

We make two observations and a proposal.

Observation 1: the refusal direction is one of many. The mean-difference probe is a generic primitive. Given any yoked pair of prompt distributions (A and B), the same machinery isolates the direction along which the model represents the A-vs-B contrast. Concepts as different as "this question is about violence" and "this question is in the register of medical advice" admit the same treatment. The math doesn't care what we call the concept.

Observation 2: ablate-to-zero leaves a hole. If we project the direction along concept_X out of o_proj, the model still attends to inputs that mention concept_X — its features upstream are unchanged — but the response along that direction is zeroed. In practice this looks like reduced hallucination, fluency drop on the concept, or an off-distribution shift toward whatever the model's nearest in-distribution behaviour is. The hole is filled by whatever is closest in weight-space, which is rarely what a deployer wants.

The proposal. Rather than ablate to zero, we ablate-and-substitute. After W ← W − d · (d.T @ W), we add back a low-rank delta along a substitute direction d_sub:

W ← W − strength · d · (d.T @ W)        # remove concept direction
W ← W + scale · d_sub · d_sub.T         # add lineage direction

d_sub is computed by the same mean-pool probe, but on a batch of passages drawn from a curated lineage corpus: a 500-book contemplative-traditions library that the AsAManThinks platform already uses as RAG substrate for its wellness chat product.

The result is a weight-resident, training-time analogue of retrieval-augmented generation. The model's response on the targeted concept is biased toward the subspace spanned by the lineage corpus, rather than left to fall back on whatever the pretrain distribution happened to nail down.

We refer to the entire pipeline as Frequency Abliteration with Lineage Substitution (FALS). "Frequency" here is a label inherited from the AAMT internal vocabulary (where individual concepts are called "frequencies" — 432Hz, 528Hz, etc., in the Solfeggio tradition). The labels are operationally cluster identifiers in activation space. The paper makes no claims about acoustic frequencies, energy, or anything metaphysical; the names are convenient stable handles for the clusters.

1.1 Contributions

  1. A generalisation of refusal-direction abliteration to arbitrary labelled concept clusters, using the same mean-difference probe.
  2. Lineage substitution: a rank-one weight delta along a corpus-pooled direction, applied after the ablation projection, that reshapes the model's response on the targeted concept toward the substitute corpus.
  3. A perplexity-guarded automatic revert mechanism: every per-layer edit is held probationally; if the held-out neutral perplexity rises above a configurable ratio, the edit is rolled back.
  4. A SHA-256-hashed concept edit ledger that records every weight surgery operation deterministically, enabling third-party verification of which concepts were edited in a shipped checkpoint.
  5. An open-source reference implementation that runs on o_proj modules in any LLaMA-style attention layer.
  6. A discussion of the ethical dimension: substituting model output with curated-corpus content is an editorial intervention. We argue it should be disclosed.

The remainder of the paper proceeds as follows. Section 2 covers related work. Section 3 specifies the FALS algorithm. Section 4 describes the audit ledger. Section 5 discusses interpretability and alignment properties. Section 6 outlines the planned empirical program. Section 7 enumerates limitations and ethics. Section 8 concludes.

2. Background and Related Work

2.1 Refusal-direction abliteration

[Arditi et al. 2024] showed that the refusal behaviour of chat-tuned language models is mediated by a single direction in residual stream space, identified by the mean of harmful-prompt activations minus the mean of harmless-prompt activations. Projecting this direction out of the output of every attention layer removes the refusal behaviour without retraining. The technique is now commonly called abliteration. Our work generalises both the direction-discovery step (to arbitrary concepts) and the surgery step (substitution rather than zeroing).

2.2 Representation engineering

[Zou et al. 2023] introduced representation engineering (RepE), a top-down framework for identifying and manipulating directions in representation space that correspond to high-level concepts (honesty, power-seeking, emotion). Their LAT (Linear Artificial Tomography) method generates contrastive prompt pairs, takes mean-difference directions in hidden states, and uses those directions for inference-time reading (probing) and control (adding the direction to activations during generation).

FALS shares the direction-discovery primitive with RepE but differs in three ways: (a) the edits are weight-resident (one-shot weight surgery) rather than per-token activation additions; (b) we explicitly remove the source direction before adding the substitute, rather than just adding control vectors; (c) the substitute direction is grounded in an external corpus rather than synthesised from contrast pairs.

2.3 Inference-time intervention

[Li et al. 2023] proposed Inference-Time Intervention (ITI): identify a small set of attention heads whose outputs are predictive of truthfulness via a probe, then at inference time shift those heads' outputs along the probe-implied direction. ITI is activation-resident and per-token. FALS is the weight-resident analogue applied post-training, with the additional substitute-from-corpus mechanism.

2.4 Function vectors

[Todd et al. 2023] showed that the function a few-shot prompt implements can be summarised as a single vector in residual-stream space, recovered by averaging the activations across in-context demonstrations. Adding the function vector to a forward pass re-induces the function even on prompts that lack the demonstrations. Our substitute direction d_sub is methodologically close to a function vector — a corpus-pooled summary — but applied as a rank-one weight delta rather than an activation addition.

2.5 Direct knowledge editing: ROME and MEMIT

[Meng et al. 2022] (ROME) and [Meng et al. 2023] (MEMIT) edit factual associations by computing a closed-form rank-one update to the MLP projection matrices at specific layers, designed to make a target fact (subject, relation, object) true under a constrained optimisation. FALS resembles ROME at the surgical level — both are rank-one weight edits — but the what is different: ROME edits a single fact recovered by causal tracing; FALS edits a concept direction recovered by mean-difference probing, and substitutes from a curated corpus rather than a single target completion. The two techniques are complementary: ROME for facts, FALS for stylistic / register-level concept substitution.

2.6 Mechanistic interpretability — IOI and feature circuits

[Wang et al. 2023] (IOI) traced the indirect-object-identification circuit in GPT-2 to a small set of attention heads. [Templeton et al. 2024] used sparse autoencoders to extract monosemantic features from Claude 3 Sonnet, demonstrating that high-level concepts (the Golden Gate Bridge, sycophancy, code vulnerabilities) live as linear directions in residual stream space. FALS treats those linear directions as editable targets: if a concept has a direction, that direction can be projected out and replaced.

2.7 Retrieval-augmented generation

[Lewis et al. 2020] (RAG) augments the model's input with retrieved passages, biasing generation toward the retrieved content without weight modification. Lineage substitution can be viewed as the weight-baked version of RAG: rather than fetching passages at inference time, the corpus-pooled direction is welded into the model once, and inference is unchanged. The trade-off is loss of per-query specificity (RAG retrieves passages relevant to this query; FALS shifts the entire concept direction by one fixed substitute) in exchange for zero inference-time retrieval cost and a smaller attack surface.

2.8 The Akashic Vault corpus

The substitute corpus is the Akashic Vault: a 500-book corpus of public-domain and licensed contemplative-traditions texts, prepared as a FTS5 SQLite database with TERA-cosine reranking (see apps/api/services/vault_search.py). The vault already serves as the RAG substrate for the MaiiaM Messenger wellness chat product [Whitehead 2024]. For FALS we use the vault not as a per-query retrieval source but as a pool of exemplars: a batch of passages along a chosen Lineage Thread (e.g. "Stoic", "Buddhist suttas", "Christian mystics") is forwarded through the model and the mean-pooled activations form the substitute direction d_sub.

3. The FALS Algorithm

3.1 Pipeline placement

FALS is implemented as a post-training stage (AbliterateStage in the reference code) inserted between training and export. The stage takes the trained model and a dictionary of concept exemplars, each of which contains:

  • low: tokenised batch of prompts where the target concept is present at HeartScale-low (e.g. shadow/avoidance/refusal/scarcity for the "greed" concept);
  • high: yoked tokenised batch where the concept is absent or inverted at HeartScale-high (the yoked baseline);
  • substitute: optional tokenised batch from the lineage corpus;
  • lineage_thread_ids: identifiers of which threads the substitute passages come from (recorded in the ledger).

The stage is disabled by default. Enabling it requires explicit config (abliteration.enabled: true) and a non-empty target_concepts list.

3.2 Direction discovery

For each concept c and each layer index ℓ in a configured range [lo, hi] (default [8, 24] for a 32-layer model — the mid-stack where Arditi found refusal lives), we:

  1. Forward the low batch through the model with a forward-pre-hook on model.layers.ℓ.self_attn.o_proj, capturing the input activation x_low ∈ ℝ^{B × T × d_model}.

  2. Forward the high batch identically, capturing x_high.

  3. Pool over the sequence axis (default mean, respecting attention mask; last and max also supported).

  4. Take the mean-difference direction:

    d_{c,ℓ} = normalise( mean_b(x_low_pooled) − mean_b(x_high_pooled) )

  5. If a substitute batch is provided, forward it, pool, and take the mean (no contrast):

    d_sub_{c,ℓ} = normalise( mean_b(x_sub_pooled) )

The direction is computed in float32 regardless of model precision to make the SHA-256 hash precision-independent. Degenerate cases (zero-norm direction) are handled gracefully: the direction is recorded as zero and the edit is a no-op.

3.3 Weight surgery

For each (c, ℓ) pair, we apply two updates to W_o ∈ ℝ^{d × d} (the o_proj weight):

Ablation projection (column-space):

coeff = W_o @ d           # (d,)  — projection of each row onto d
ΔW_abl = − strength · outer(coeff, d)
W_o ← W_o + ΔW_abl

where strength ∈ [0, 2] (default 1.0 = full projection; values <1 retain a fraction of the direction; >1 over-projects).

Lineage substitution (rank-one delta):

out_proj = normalise( mean(W_o, dim=1) )
ΔW_sub = scale · outer(out_proj, d_sub)
W_o ← W_o + ΔW_sub

The output projector out_proj is the mean of W_o's columns, normalised — this aligns the rank-one update with the layer's existing output direction, so the substitution does not produce out-of-distribution outputs. scale ∈ [0, 2] (default 0.5).

Both deltas are kept around so the edit can be reverted if needed.

3.4 Perplexity-guarded revert

After each per-layer edit, the stage runs an optional perplexity evaluation on a neutral held-out set (ppl_eval_fn, injectable for testing). If

ppl_after / ppl_before > revert_ppl_ratio    (default 2.0)

the edit is rolled back by adding −ΔW_abl − ΔW_sub. The revert count is logged. This guard means a misconfigured FALS run cannot produce a shipped checkpoint with catastrophically degraded base capability — the worst case is a vacuous run where every edit reverts and the model is byte-identical to the input.

3.5 Pseudocode

def fals(model, exemplars_by_concept, config):
    modules = find_o_proj_modules(model, config.probe_layer_range)
    ledger = ConceptLedger()
    ppl_before = config.ppl_eval_fn(model)

    for concept, ex in exemplars_by_concept.items():
        dirs    = probe_directions(model, ex.low, ex.high, modules)
        sub_dirs = probe_directions_pool(model, ex.substitute, modules) \
                   if ex.substitute is not None else {}

        for layer_idx, module in modules.items():
            d     = dirs[layer_idx]
            d_sub = sub_dirs.get(layer_idx)

            dW_abl = project_direction_out_(
                module.weight, d, strength=config.abliteration_strength)
            dW_sub = None
            if d_sub is not None:
                dW_sub = apply_substitute_delta_(
                    module.weight, d_sub, scale=config.substitution_scale)

            ppl_after = config.ppl_eval_fn(model)
            if ppl_after / ppl_before > config.revert_ppl_ratio:
                module.weight.add_(-dW_abl)
                if dW_sub is not None:
                    module.weight.add_(-dW_sub)
                ledger.record_revert(concept, layer_idx, ppl_after / ppl_before)
                continue

            ledger.record_edit(LedgerEntry(
                concept=concept,
                layer=layer_idx,
                direction_sha256=sha256(d),
                strength=config.abliteration_strength,
                scale=config.substitution_scale if d_sub is not None else 0.0,
                val_ppl_before=ppl_before,
                val_ppl_after=ppl_after,
                lineage_thread_ids=ex.lineage_thread_ids,
                substitute_direction_sha256=sha256(d_sub) if d_sub else None,
            ))

    ledger.write_json(run_dir / "concept_edit_ledger.json")
    return ledger

The full reference implementation is in training_pipeline/abliteration/direction_probe.py (primitives) and training_pipeline/stages/abliterate.py (orchestration).

3.6 Numerical notes

  • The mean-difference computation is performed in float32 to avoid cancellation when low and high activations are close.
  • The activation hook captures the input to o_proj, which is the per-head attention output concatenation — empirically the cleanest site for concept directions, matching Arditi's choice.
  • The probe is deterministic given the same exemplar tensors and pool mode. No random projections are used.
  • The substitution delta is rank-one, so the operator-norm change to W_o is bounded by scale · ‖out_proj‖ · ‖d_sub‖ = scale (since both are unit-normalised). This bounds the worst-case Lipschitz change of the layer's output by scale per concept, per layer.

4. The Concept Edit Ledger

Every FALS run writes concept_edit_ledger.json to the run directory. The ledger is the single source of truth for which edits were applied to a checkpoint. Schema (v1):

{
  "schema_version": 1,
  "created_at": "2026-05-12T19:00:00Z",
  "config_fingerprint": "sha256:...",
  "entries": [
    {
      "concept": "greed",
      "layer": 14,
      "direction_sha256": "9e3c...",
      "strength": 1.0,
      "scale": 0.5,
      "val_ppl_before": 11.42,
      "val_ppl_after": 11.78,
      "lineage_thread_ids": ["stoic", "buddhist-sutta"],
      "substitute_direction_sha256": "21af...",
      "notes": ""
    }
  ]
}

The ledger enables three downstream properties:

  1. Reproducibility. Given the same exemplar tensors and config, the direction hashes must match byte-for-byte. A consumer of the shipped checkpoint can re-derive the directions and verify.
  2. Attestation. The config fingerprint is the SHA-256 of the resolved AbliterationConfig plus the exemplar selection policy. Two parties can disagree about whether to ship a concept edit, but they cannot disagree about what was shipped.
  3. Disclosure. The ledger can be exposed to end-users as a transparency manifest. We return to this in Section 7.

The ledger is not a model-card replacement; it sits below the model card as the technical attestation that backs the high-level disclosure.

5. Properties

5.1 Bounded behaviour change

Because each per-layer edit is a rank-≤2 update (rank-1 ablation, rank-1 substitution) to a single weight matrix, the global change to the model's input-output map is bounded. Empirically and from the operator-norm bound above, FALS edits at the default settings preserve general capability — the perplexity guard makes this a backstop rather than a hope.

5.2 Auditability

The ledger gives a deployment-time auditor a complete inventory of concept-level edits in a shipped checkpoint. Combined with the deterministic direction hashes, this provides something analogous to a software bill of materials for behavioural modifications.

5.3 Composability with refusal abliteration

The original refusal abliteration is a special case of FALS: concept = "refusal", substitute = None, scale = 0. A single-stage FALS run can apply refusal-removal and concept-level substitutions in one pass, with the ledger recording both classes of edit uniformly.

5.4 Composability with RAG

FALS is RAG-compatible: a deployment that uses RAG at inference time can still apply FALS at training time. The two operate at different levels — RAG biases the input distribution, FALS biases the weight distribution along the same corpus. We hypothesise that the two compose constructively (the FALS-shifted weights are more receptive to RAG passages from the same corpus), but this is an empirical question. See Section 6.

5.5 Composability with Vortex-keyed routing

In a Vortex-keyed MoE architecture (companion paper 01), the FALS edit can be applied to a specific expert's o_proj — restricting the concept substitution to tokens routed to that archetype. This produces archetype-conditional concept editing: "when the Heart-Mind expert handles the token, substitute the greed direction toward the Stoic lineage; on other archetypes, leave the direction alone." We sketch this extension in Section 8.

6. Proposed Empirical Program

We are running, not reporting, the following evaluations. Numbers will appear in v1.1.

6.1 Base models

  • Qwen 2.5 3B Instruct.
  • Gemma 2 2B Instruct.
  • LLaMA 3.2 3B Instruct.

The choice is governed by available compute (Apple Silicon MPS, M1/M2 Max class) and the requirement of o_proj-style attention heads addressable by the regex model\.layers\.(\d+)\.self_attn\.o_proj$.

6.2 Target concept set

A pilot set of 8 concepts drawn from the AAMT register vocabulary: greed, fear, shame, despair, pride, contempt, urgency, grasping. Each concept has 64 HeartScale-low exemplars and 64 HeartScale-high yoked exemplars from the AAMT-tagged val set.

6.3 Substitute corpus

Lineage Threads from the Akashic Vault: stoic, buddhist-sutta, christian-mystic, taoist, sufi, hermetic. Each provides ~64 passages per concept selected by FTS5 + TERA cosine similarity to the concept tag.

6.4 Metrics

  1. Capability preservation: MMLU [Hendrycks 2021], HellaSwag, ARC, plus per-domain perplexity on C4 [Raffel 2020] held-out.
  2. Concept response shift: a held-out probe prompt set per concept, judged by an LLM-judge (Claude 3.5 Sonnet) for (a) frequency of the targeted failure mode, (b) frequency of passages stylistically aligned with the substitute lineage, (c) factual accuracy.
  3. Hallucination rate: TruthfulQA-style fact probes for each concept, scored against curated ground-truth answers from the lineage corpus.
  4. Direction stability: re-derive each direction from a held-out exemplar sample and compute cosine to the shipped direction. We expect ≥0.9 cosine for well-formed concepts.
  5. Revert rate: fraction of (concept, layer) pairs that trigger the perplexity guard. A healthy run should see <10% reverts.

6.5 Ablations

  • Strength sweep: strength ∈ {0.5, 1.0, 1.5}.
  • Scale sweep: scale ∈ {0, 0.25, 0.5, 1.0} (with scale=0 recovering vanilla concept-abliteration).
  • Layer range: (8, 24) vs (0, 32) vs (16, 20).
  • Pool mode: mean vs last vs max.
  • Substitute corpus ablation: replace the Akashic Vault with Wikipedia-only and with a held-out lineage-thread; measure drift.

6.6 Interpretability case studies

For each successfully edited (concept, layer), compute the top-activating contexts before and after the edit, qualitatively comparing the shift. We expect lineage-substituted models to produce top-activating contexts that match the substitute corpus stylistically on the targeted concept.

7. Limitations and Ethics

7.1 Empirical case is not yet reported

This is v1.0. We describe the algorithm and the planned empirical program but do not yet report benchmark numbers. The reference implementation passes unit tests on a synthetic toy model (see packages/training-pipeline/tests/test_abliteration.py).

7.2 Linear-feature hypothesis

FALS, like RepE, ITI, and the original abliteration, relies on the linear representation hypothesis: that the targeted concept is mediated by a linear direction in activation space. Where this fails (e.g. concepts encoded in superposition over many features, concepts that emerge from non-linear interactions across layers) the mean-difference probe will not isolate the concept cleanly. The perplexity guard catches the loudest failures; subtler ones will manifest as ineffective edits (the concept persists) rather than catastrophic ones.

7.3 Substitute drift

The substitute direction is a fixed corpus mean. It does not adapt to context. On prompts where the concept appears in a register unmatched to the lineage corpus, the substitution can produce stylistically incongruous output. This is the cost of trading RAG's per-query specificity for weight-residency.

7.4 Concept selection is editorial

This is the central ethical point of the paper. Deciding which concepts to ablate and which lineages to substitute toward is an editorial act. It is not a discovery — there is no "objectively-correct" concept set, no "true" substitute corpus. A FALS-edited model is a model whose response on selected concepts has been biased toward the curator's chosen tradition. This is not unique to FALS (RLHF, constitutional AI, system prompts, and RAG selection all involve editorial decisions), but FALS makes the intervention weight-resident and therefore invisible to inference-time inspection of inputs and outputs.

We argue:

  1. Disclose. Any deployment of FALS should publish the concept edit ledger (or a redacted summary, if concept names are commercially sensitive). End-users should be able to know what concepts were edited and what lineages were substituted in.
  2. Default-off. The reference implementation ships with FALS disabled by default. A run requires an explicit positive configuration to apply any edits.
  3. Bounded. The perplexity guard, the rank-1 substitute delta, and the (strength, scale) knobs are designed to keep edits small. Aggressive settings are possible but require explicit override.
  4. Reversible at the ledger level. Because every edit is recorded with deterministic direction hashes, an external auditor can re-derive −ΔW and produce a reverted checkpoint, restoring the pre-edit weights byte-for-byte (modulo the rank-1 approximation).
  5. Refuse hostile use cases. The same machinery that substitutes "greed → Stoic" can substitute "fairness → racist-corpus". The technique is dual-use. We will not publish FALS edits aimed at removing safety properties from base models, and we ask downstream users not to.

7.5 Corpus copyright

The Akashic Vault contains a mix of public-domain and licensed texts. Lineage substitution does not redistribute the corpus — d_sub is a 4096-dimensional unit vector, not a passage — but it does derive from the corpus. Whether this constitutes transformative use is a legal question we do not resolve here. Deployers are responsible for verifying that their substitute corpus permits this derivation.

7.6 No claim about "sacred frequencies"

The internal AAMT vocabulary labels concepts with names from the Solfeggio (432Hz, 528Hz, etc.) and from the four-element / TERA register systems. These labels are used in this paper only as stable identifiers for activation-space clusters. No claim is made or implied that ablating a "528Hz direction" has any acoustic, energetic, vibrational, or metaphysical effect. The mechanism is mean-difference weight surgery; the names are filenames.

7.7 Scaling

The reference implementation has been exercised at the 2–3B parameter scale. Whether FALS edits remain stable at 70B+ — where features may live in different layers, in superposition, or be distributed across attention heads — is an open empirical question.

7.8 Compounding edits

The ledger format records edits as independent rows, but in practice edits compose. Two concept directions that are non-orthogonal will interact when both are projected out; the second edit operates on a weight matrix already modified by the first. The current revert logic handles this for the perplexity guard but does not orthogonalise the directions. A future version may.

8. Discussion and Future Work

8.1 The design space

FALS occupies a specific cell in the design space of behaviour modification techniques:

TechniqueTimeLocusSourceAuditable
Promptinginferenceinputcuratorweak
RAGinferenceinputcorpusmedium
ITIinferenceactivationsprobeweak
RepE controlinferenceactivationscontrast pairsmedium
RLHFtraininggradientspreference dataweak
Constitutional AItraininggradientsrule setweak
ROME / MEMITpost-trainingweights (MLP)target factstrong
Abliterationpost-trainingweights (attn)contrast pairsstrong
FALSpost-trainingweights (attn)contrast pairs + corpusstrong

The cell-defining properties: post-training (one-time, no retraining), weights (no inference cost, no input-time inspection), audit-strong (every edit is logged with a deterministic hash). The novelty of FALS within this cell is the corpus-grounded substitute — every other weight-resident technique either zeros (abliteration) or targets a single completion (ROME).

8.2 Connection to alignment

FALS is a tool, not an alignment program. It can be used to make a model better (substituting a stable wisdom-tradition response in place of model-confabulated wellness advice — the use case driving this work) or worse (substituting any biased corpus over any direction). The audit ledger provides a check on whether FALS was used at all and how, which is the alignment contribution.

The strongest alignment claim we make is the following: in a deployment that requires fidelity to a specific knowledge tradition (legal, medical, contemplative, organisational), FALS gives a training-time mechanism to bias the model toward that tradition's substrate, complementing the inference-time mechanism of RAG. The combined system can degrade gracefully (when retrieval fails, the weights still lean toward the corpus) and can be audited for both the corpus selection (ledger) and the retrieval policy (RAG logs).

8.3 Future work

  • Empirical paper (v1.1): the full evaluation outlined in Section 6.
  • Sparse-autoencoder direction discovery. Replace the mean-difference probe with directions from a trained SAE [Templeton 2024], giving access to monosemantic targets rather than potentially-polysemantic mean-diffs.
  • Per-expert FALS in Vortex MoE. As sketched in Section 5.5, apply FALS to a single expert's o_proj for archetype-conditional substitution.
  • Orthogonalising compounding edits. Gram-Schmidt the concept directions before applying, so the second edit's effective direction is orthogonal to the first.
  • MLP-resident FALS. Apply the same projection-then-substitute recipe to the down-projection of the MLP block, where ROME and MEMIT operate. The MLP locus may capture different concept classes than the attention o_proj locus.
  • End-user disclosure UX. A reference UI that surfaces the concept edit ledger to end-users — "this model has been edited along the following concepts toward the following lineages" — as a default transparency mechanism for FALS-equipped products.

9. Conclusion

We presented Frequency Abliteration with Lineage Substitution, a generalisation of refusal-direction abliteration that targets arbitrary labelled concept clusters and replaces ablation-to-zero with a corpus-grounded rank-one substitute. The contribution is methodological: a training-time, weight-resident, auditable mechanism for shifting a model's response on selected concepts toward a curated tradition without retraining.

The technique is dual-use. We have argued that responsible deployment requires disclosure of the concept edit ledger and restraint in the choice of edits. The reference implementation ships with FALS disabled by default and a perplexity guard that makes catastrophic regressions self-reverting.

Empirical validation is in progress; v1.1 will report numbers.

References

Arditi, A., Obeso, O., Syed, A., Paleka, D., Panickssery, N., Gurnee, W., Nanda, N. (2024). Refusal in Language Models Is Mediated by a Single Direction. arXiv:2406.11717.

Hendrycks, D. et al. (2021). Measuring Massive Multitask Language Understanding. ICLR.

Lewis, P., Perez, E., Piktus, A. et al. (2020). Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks. NeurIPS.

Li, K., Patel, O., Viégas, F., Pfister, H., Wattenberg, M. (2023). Inference-Time Intervention: Eliciting Truthful Answers from a Language Model. NeurIPS.

Meng, K., Bau, D., Andonian, A., Belinkov, Y. (2022). Locating and Editing Factual Associations in GPT. NeurIPS.

Meng, K., Sharma, A., Andonian, A., Belinkov, Y., Bau, D. (2023). Mass-Editing Memory in a Transformer. ICLR.

Raffel, C. et al. (2020). Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer. JMLR 21.

Templeton, A. et al. (2024). Scaling Monosemanticity: Extracting Interpretable Features from Claude 3 Sonnet. Anthropic.

Todd, E., Li, M. L., Sharma, A. S., Mueller, A., Wallace, B. C., Bau, D. (2023). Function Vectors in Large Language Models. arXiv:2310.15213.

Wang, K., Variengien, A., Conmy, A., Shlegeris, B., Steinhardt, J. (2023). Interpretability in the Wild: A Circuit for Indirect Object Identification in GPT-2 Small. ICLR.

Whitehead, W. C. Jr. (2024). AAMT Foundations: TERA, Vortex, and HeartScale. AsAManThinks technical report.

Zou, A., Phan, L., Chen, S. et al. (2023). Representation Engineering: A Top-Down Approach to AI Transparency. arXiv:2310.01405.

Appendix A — Reference Implementation

Key files in maiiam-alchemist/packages/training-pipeline/:

  • Direction probing and weight surgery primitives: training_pipeline/abliteration/direction_probe.py (DirectionProbe, project_direction_out_, apply_substitute_delta_, collect_o_proj_inputs).
  • Audit ledger: training_pipeline/abliteration/concept_ledger.py (ConceptLedger, LedgerEntry, direction_sha256).
  • Stage orchestration and config: training_pipeline/stages/abliterate.py (AbliterateStage, AbliterationConfig, ConceptExemplars, AbliterateResult).
  • Pipeline wiring documentation: training_pipeline/abliteration/_wiring/abliteration.md.
  • Unit tests on a synthetic toy model (no Qwen/Gemma load required): packages/training-pipeline/tests/test_abliteration.py.

The substitute corpus retrieval lives in the AsAManThinks platform: apps/api/services/vault_search.py (TERA-aware FTS5 retrieval over the 500-book Akashic Vault).

Appendix B — Default Configuration

abliteration:
  enabled: false                              # default off
  target_concepts: []                         # explicit positive list
  substitute_from_lineage: []                 # explicit positive list
  probe_layer_range: [8, 24]
  probe_token_pool: mean
  probe_n_examples_per_concept: 64
  abliteration_strength: 1.0
  substitution_scale: 0.5
  revert_ppl_ratio: 2.0
  o_proj_module_pattern: 'model\.layers\.(\d+)\.self_attn\.o_proj$'
  o_proj_path_template: 'model.layers.{i}.self_attn.o_proj'
  ledger_path_relative_to_run: concept_edit_ledger.json

End of preprint v1.0.

Cite this preprint

@techreport{whitehead2026abliteration,
  author      = {Whitehead, Weslyn Cory},
  title       = {Frequency Abliteration with Lineage Substitution: Targeted Direction Removal and Corpus-Grounded Replacement in Pretrained Language Models},
  institution = {AsAManThinks / MaiiaM Alchemist},
  year        = {2026},
  type        = {Preprint v1.0},
  doi         = {10.5281/zenodo.20150193},
  url         = {https://doi.org/10.5281/zenodo.20150193}
}
Download .mdBack to series