Serving Causal Encoder–Decoder Models with MoL2

• From shared weights to shared history. Mixture-of-LoRA (MoL) trains one LoRA per domain on a shared base, so domains do not interfere and each can be updated or rolled back on its own. At inference, however, a LoRA on every layer makes the KV depend on the active LoRA. Each LoRA keeps its own copy of the context, including the summaries passed between LoRAs, so KV storage grows with the number of LoRAs and the length of the conversation. On DeepSeek-V4.1-Flash, attaching LoRA only to the layer-20 FFN and layers 21–39 lets all LoRAs share one encoder state and global KV, so the history is stored once and no longer passed between LoRAs as summaries. Adding a domain then costs one more LoRA, while the memory for the history stays the same however many domains a conversation visits.
• Low-cost switching, preserved quality. A switch rebuilds only the last 128 tokens of adapter-dependent local state. For a fixed target, the output is bitwise identical whichever LoRA wrote the history, including a LoRA that adapts a different set of modules. In our replay tests, the PPL ratio against a full recompute under the target is 0.964–0.995. A no-switch, token-by-token control already reaches about 85% of the replay path’s logit MAE.
• MoL2 makes the routing decision almost free. Mixture-of-LoRA with Latent pointer (MoL2) reads H19, the output of encoder layer 19, and processes LoRA descriptions only through token embedding. Its 15.25M-parameter head makes a decision in 2.77 ms without writing temporary KV, about 5,586× faster than our previous MoL router (15.48 s), with 97.31% accuracy on our 968-sample LoRA benchmark.
• Mixed data balances domain and generic routing. Combining public classification tasks rewritten as routing questions with LoRA-domain examples preserves both abilities. Generic-only and staged training give lower LoRA-benchmark accuracy in our runs. Adding 722 LoRA samples to the full generic set brings the benchmark to 90.4% (5-seed mean). Our current router-update workflow, from offline H19 extraction to head retraining, takes under two hours, and the backbone and the serving LoRAs stay untouched.
• Routing becomes part of the serving engine. Even a fast routing head adds another prefill when run as a separate request. Our engine captures H19 during normal prefill, sends the selected adapter’s identifier with the request’s KV transfer, and activates it on the decode worker. This avoids a second prefill and keeps routing computation out of the per-token decode loop. In the measured workload, throughput is close to the fixed-route baseline.

Picture a user who, within one conversation, moves between chat, multi-step agent tasks, coding and UI design. Each request is best served by a model tuned for its domain. Training all domains into one set of weights, by joint SFT or parameter merging 4, 5, can introduce interference, and domain-specific updates often require broader regression testing. Multi-agent systems 6–9 can separate domain-specific models, but deployments with independently tuned models may require multiple sets of weights and additional prefill at handoff.

Mixture-of-LoRA (MoL), introduced with Macaron-V1 1, attaches several domain LoRAs 10 to one base model and picks one per turn (Figure 1). We follow Macaron-V1's four-way split: L0 for chat (it also acts as the router), L1 for agent tasks, L2 for coding and L3 for design. "Adapter" and "LoRA" mean the same thing below. Each LoRA is trained on its own domain with the base frozen, so updating one adapter leaves the others’ weights unchanged, while deployment shares one copy of the base weights. The same isolation suits continual learning. A production model keeps absorbing new data and fixes, and continued fine-tuning of merged weights risks catastrophic forgetting 11. Under MoL an update touches one LoRA, and a bad version is rolled back by swapping that LoRA back, without retraining or redeploying the rest.

Figure 1. Multi-agent vs MoL over one four-turn conversation. (a) A multi-agent system deploys one full model per expert; at each handoff the history is passed on as text and prefilled again by the next model. (b) MoL on GLM-5.2/5.3 shares one frozen base but keeps a separate KV cache per LoRA; a switch requires a handoff prefill under the target LoRA. (c) MoL2 on DeepSeek-V4.1 shares one encoder state and global KV across LoRAs; a switch rebuilds only the last 128 tokens of local state.

MoL shares the weights, but not the history. Macaron-V1 runs on GLM-5.2/5.3, decoder-only models with 78 layers, where every layer carries a LoRA. The KV at layer l depends on layers 0 through l, all of which depend on the active LoRA, so KV written under one LoRA differs at every layer from what another LoRA would compute. A switch therefore trades accuracy against time. Reusing KV computed under another model or context degrades output quality unless it is corrected [12, 13], and learned KV translators [14] need one mapping per ordered pair of LoRAs, retrained whenever a LoRA changes. Recomputing keeps the output exact but costs time: the target LoRA must prefill the history again before it generates, and the delay grows with the context [15]. Macaron-V1 recomputes, and we call this prefill under the target the handoff. Per-LoRA prefix caching shortens it when the conversation returns to a LoRA it has already used, at the price of one KV copy per LoRA: by a bf16 estimate that excludes MTP layers and engine overhead, four copies of a 128K context on GLM-5.2 take at least 46.5 GiB. As long as every layer depends on the LoRA, a switch has to pay one of these costs.

DeepSeek-V4.1-Flash [3] (DeepSeek-V4.1 below) is a causal encoder–decoder (CED): an encoder (layers 0–19) followed by a decoder (layers 20–39). The long-range history is held by the encoder and a global KV, which can stay free of any LoRA, so the differences between LoRAs can be confined to the decoder and its local state. The released weights are available on Hugging Face.

What is shared and what is rebuilt

The 40 layers split as follows:

Table 1. The 40 layers of DeepSeek-V4.1-Flash by role. The roles overlap: layer 20, for example, is a decoder layer, a global KV source and an index layer.

At any step, a decoder layer reads only the shared global KV and its own local KV for the last 128 positions. Older positions reach the decoder only through the global KV.

Figure 2. Where the LoRAs sit and what a switch rebuilds. (a) GLM-5.2/5.3: every layer carries a LoRA, so each LoRA keeps its own KV and a switch re-prefills the whole context. (b) DeepSeek-V4.1-Flash: the encoder and the layer-20 attention are frozen, so all LoRAs share one global KV and a switch rebuilds only the 128-token local KV (dashed).

We keep the encoder and the layer-20 attention frozen, and attach LoRA only to the layer-20 FFN and layers 21–39 (Figure 2b). Layer 20 is both the first decoder layer and the last global KV source. The compressor and indexer that write its global KV sit in its attention, whose input is the frozen encoder output. With no LoRA on that attention, the whole global KV is independent of the LoRA, and a LoRA on the layer-20 FFN affects only the layers above it. The encoder state and the global KV are therefore computed once for all LoRAs. A LoRA on the encoder would make both LoRA-dependent and break the sharing. Our earlier training study [32] found that decoder-side LoRA retained similar validation loss with roughly half the adapter parameters, including when the layer-20 KV-writing projections were frozen.

What remains LoRA-dependent is the local KV of layers 21–39, which covers only the last 128 positions. A switch recomputes those 128 positions through the layer-20 FFN and layers 21–39 under the target LoRA; we call this bounded replay. Let C be the conversation history, U the new user turn, and P = C + U the full context.

Table 2. What a LoRA switch keeps and rebuilds on the two architectures.

Shared state also makes adapter updates cheaper. A running conversation moves to a new adapter version by replaying its last 128 tokens, and prefix caches shared across requests (system prompts, tool definitions, long documents) need no rebuild per adapter version.

One turn with a switch runs three steps:

1. Encoder. Incrementally encode only the suffix C[-128:] + U, reusing the existing state. This step is LoRA-independent and runs on every turn.
2. Layer-20 attention. Write the layer-20 global KV, shared by all LoRAs.
3. Target replay. Activate the target LoRA, drop the local KV the source wrote in layers 21–39, recompute P[-128:] through the layer-20 FFN and layers 21–39, then decode.

The next token reads exactly the rebuilt P[-128:] local KV plus the global KV, which no LoRA has touched.

A switch loses no information

By "no information loss" we mean two measurable things: the target sees the same history whichever LoRA wrote it, and its output after a 128-token replay is no worse than after a full recompute. Replay is not bitwise equal to a full recompute, so we also measure where the remaining difference comes from.

Call a full recompute of the whole context under the target the oracle. In the oracle, the first positions of the replay window see the target's local KV for positions before the window; in the replay they see only the global KV and the window itself. If this truncation drives the error, the error should concentrate in the first tokens after a switch, grow with depth, and stay bounded as the context grows. Truncation is not specific to switching, since every new turn starts with a bounded replay even when the LoRA stays the same. A further baseline error, which we call incremental-computation error, has nothing to do with replay: computing token by token instead of in one pass changes kernel shapes, fp8/fp4 rounding and the indexer's top-k choices.

Setup. We run the DeepSeek-V4.1-Flash reference implementation [3] on 8×H20 (TP = 8) with BASE (the base model with no LoRA), three trained LoRA checkpoints (r = 16), and one randomly initialized LoRA, giving 25 source/target pairs. The three checkpoints were fine-tuned on Geo3K [16] and are labelled L1–L3 in Figures 3 and 4; these labels identify test checkpoints, rather than the agent, coding and design LoRAs. Each sits on 160 modules (attention and shared expert). The random LoRA, labelled L0, adapts a different set of modules: 2,896 of them, including the routed experts, at r = 8 and scale 4.0. They use no20: All LoRAs sit on layers 21–39, with layer 20 entirely frozen. This is more conservative than also adapting the layer-20 FFN. Both placements preserve source independence, but the PPL measurements below use no20 only. For each pair, the source computes a 1,024-token input and a 256-token reply token by token; we then append a new turn, switch to the target, force-decode the same 256-token continuation and compare logits position by position. The deployed path P1 (source history, then bounded replay) is compared with the oracle. A control uses the target throughout with no switch or replay, prefilling a prefix in one pass and continuing token by token, which isolates the incremental-computation error. Metrics are logit MAE, top-1 agreement and the PPL ratio exp(CE_P1 − CE_oracle). We configure NCCL so that reruns are bitwise identical.

The LoRA that wrote the history does not matter. For a fixed target, all five sources give bitwise identical logits, CE and top-1 at every position (Figure 3). This follows from the structure: the source affects only the local KV of layers 21–39, which the switch rebuilds, while the surviving state (the encoder state and the global KV) comes from the encoder and the layer-20 attention, which no LoRA touches. The result therefore does not depend on how strong the LoRA is. By contrast, when the layer-20 attention also carries LoRA (on its compressor and indexer), the global KV carries a trace of the source, and the logit MAE varies by up to 0.021 across sources.

Figure 3. Logit MAE and PPL ratio of the deployed path against the oracle for every source/target pair. The five sources in a column (same target) have identical values, although L0 adapts different modules from L1–L3; pairwise comparison of their logits gives MAE = 0.

Replay shows no quality loss against a full recompute.

For the three trained targets and BASE, the PPL ratio of P1 against the oracle stays within 0.964–0.995, and top-1 agreement equals the no-switch control's (Figure 4). The random L0 target gives 1.022, but the no-switch control already gives 1.019 for it, so replay adds only 0.003; the excess over 1 comes from incremental computation, not from the switch. The forced continuation is the greedy output of a replay path (BASE history plus bounded replay), not the oracle's own output, which slightly disadvantages the oracle. A ratio below 1 therefore shows only that replay does not make the output worse, not that it beats a full recompute.

Figure 4. Error breakdown per target. Grey is the chunked no-switch control; blue is the deployed path P1 against the oracle. The right panel plots PPL ratio minus 1; below 0 means no degradation.

Averaged over the 25 pairs, the replay path differs from the oracle by a logit MAE of 0.567, and most of this has nothing to do with replay. The control sees exactly the oracle's context at every position and differs from it only in computing token by token rather than in one pass, yet its MAE against the oracle is already 0.480, about 85% of the total. This is the incremental-computation error. The replay path’s MAE is about 0.09 higher (0.055–0.134 depending on the target; Figure 4). This is consistent with an additional effect from window truncation, although the two MAEs are not an additive error decomposition. The switch does not cause this part either. Even without a switch, every new turn runs the same 128-token replay, and for a given target the MAE is identical whether the target itself or another LoRA wrote the history. Changing the source produces no additional difference in this fixed-history comparison.

The error concentrates in the first dozen tokens after the switch, and agreement returns to about 98% once the truncated positions leave the window (Figure 5). But the control, which does no replay, behaves almost the same there, so the early instability comes mainly from incremental computation: the top candidates at these positions are nearly tied, and a tiny numerical difference flips top-1.

Figure 5. Error by position after the switch. Left: top-1 agreement over the first 32 tokens, with the first 16 shaded. Right: logit MAE over 256 tokens, 8-point moving average.

The error grows with depth and stays bounded as the context grows. We measure the decoder hidden-state MAE of the replay segment against the oracle at four context lengths, with a fixed 512-token suffix and replay starting 128 positions before the end of the prefix. Each length uses 10 inputs × 25 source/target pairs over BASE, a randomly initialized LoRA and the three trained checkpoints. Here the trained LoRAs sit on all of layers 20–39, including the layer-20 compressor and indexer.

Table 3. Decoder hidden-state MAE of the replay segment against the oracle, averaged over layers 20–39.

At length 640 the replay starts at position 0, which amounts to a full recompute, so the 0.414 MAE is entirely incremental-computation error. Reusing a prefix raises the MAE to about 0.56–0.57, but it stops growing as the prefix goes from 256 to 512 tokens (Figure 6a). The largest |ΔCE| on the gold token at the last position is 1.2e-3 nat, which is negligible.

Figure 6. Decoder replay error. (a) Hidden-state MAE (mean over layers 20–39) vs context length for trained and randomly initialized LoRAs; error bars span min to p95. (b) Per-layer MAE (layers 20–39) at several context lengths; the wide grey line is the randomly initialized LoRA control at length 1024.

At length 1024 the MAE fluctuates between neighboring layers but rises overall from 0.24 at layer 20 to 1.12 at layer 39 (Figure 6b). The MAE with and without a switch is almost the same (0.5622 vs 0.5623), and a randomly initialized LoRA gives nearly the same per-layer curve, so the growth comes from truncation error propagating through the decoder, not from any LoRA amplifying it.

Table 4. Hidden-state MAE at selected decoder layers, context length 1024. Figure 6b shows all of layers 20–39.

Determinism and limitations. With default NCCL settings, NVLS all_reduce perturbs identical inputs by about 2e-6, and reruns reach a logit MAE of about 0.2, which would make "bitwise identical" uncheckable. With NCCL_ALGO=Ring NCCL_NVLS_ENABLE=0, reruns of prefill, decode and replay are bitwise identical, and the earlier MAE barely changes (0.562 vs 0.566 at length 1024), so NCCL noise is not the main error source.

With the handoff reduced to a 128-token replay, choosing the next LoRA becomes a more visible part of the cost. The encoder already produces a representation of the request before any LoRA is selected. We use that representation to replace the L0 decode router with a small pointer head.

Routing becomes the bottleneck

On GLM-5.2, Macaron-V1 splits each turn into a route hop, an answer hop and a summary hop 1. After a non-L0 LoRA answers, it writes a summary of up to 192 tokens that the user never sees; it only carries context across later handoffs. Macaron-V1 reports these hop latencies on 48 multi-turn requests at temperature 0:

Table 5. Per-hop latency of one Macaron-V1 turn on GLM-5.2, as reported by Macaron-V1. Percentages are shares of the turn total.

Routing and the summary produce nothing the user sees, yet take about a third of the turn. The handoff prefill sits inside the answer hop and grows with the context. Bounded replay reduces that work to the last 128 tokens. What remains is the routing decision, whose total cost grows with switching frequency. Macaron-V1’s router decodes a single label. Our L0 decode baseline also writes a short reason and takes 15.48 s per decision in the reference implementation. An agent turn that switches LoRA between tool calls, or mid-generation, may leave only tens to hundreds of tokens between two decisions.

The CED layout also gives the router something to read. The encoder runs under BASE before any LoRA is chosen, so H19, the output of encoder layer 19, is a LoRA-independent representation of the input, available without an extra forward pass or an L0 decode.

All three routers choose among N LoRA descriptions using the current user input. Each description is paired with a shared route prompt. What changes is how the input and descriptions are represented, and whether routing leaves temporary state in the backbone (Figure 7, Table 6). We evaluate them on a 968-sample LoRA routing benchmark drawn from seven benchmark sources and labelled L0–L3, and on 1,440 generic routing questions with 2–77 candidates each.

Table 6. The three routers: “MoL” means the router in these comparisons. Contextual MoL and MoL2 each report one training run, with checkpoints selected on generic dev. The MoL2 run is qfix; its mean accuracy across five training seeds is 94.63%.

Figure 7. The three router heads: how the query and the options are represented, whether they enter the backbone, and whether routing leaves temporary KV.

MoL: L0 decode (Figure 7a). MoL carries over the routing of GLM-5.2/5.3. The query, the route prompt and the four descriptions run through the encoder and a decoder pass under the L0 LoRA, which then decodes a label and a reason token by token; the LoRA id is parsed from the label. This needs no separate router training and gives a readable reason, but one decision takes about 15.5 s, mostly spent decoding dozens of tokens. The route prompt and the generated tokens also land in the KV cache and the decoder sliding window, and must be rolled back before the target replay. The L0 LoRA in our MoL router is a randomly initialized test adapter, so its numbers are a reference baseline, not the limit of a production router.

Contextual MoL: contextual q/k (Figure 7b). Contextual MoL stops decoding and scores on H19. The query and all options are encoded together in one temporary pass through layers 0–19, and a q/k head scores the H19 of the query against the H19 of each option. Because the options follow the query, causal attention lets each option read it, so the options are represented in the context of the current input. Contextual encoding helps substantially in this comparison: with the same q/k head and training on generic data only, encoding the descriptions after the query instead of alone lifts generic test accuracy from 62.71% to 79.10%. The cost is that the options enter the backbone and write a temporary global-KV branch, which must be cleaned up after every decision; one decision still takes about 1.9 s.

MoL2: embedding pointer (Figure 7c). MoL2 keeps the options out of the backbone. The query is encoded once, as in any turn, and the router reads the H19 of the current user turn. The route prompt and descriptions pass only through model.embed(), the token-embedding lookup, and no Transformer layer. To make up for the lost contextual encoding, the pointer head puts its capacity on the option side (a deeper MLP plus self-attention within each option) and gives the query side a light projection; one cross-attention layer and a q·k pointer score follow, for 15.25M parameters in total. The options do not depend on the query, so the option side is computed once and cached at startup, and each request runs only the cross-attention and the pointer. Routing writes no temporary KV.

On the LoRA benchmark the three routers score within 1.5 pp of each other (Table 6), while the time per decision falls from 15.48 s to 1.88 s to 2.77 ms. MoL2 gives up some general routing ability: on generic test it scores 65.14% against Contextual MoL's 78.26%, with the loss concentrated in sources with many fine-grained labels.

Figure 8. One full turn under each router. In all three rows the encoder, the layer-20 attention that writes the shared KV, the target-LoRA replay and decoding are the same; only the routing step differs. MoL and Contextual MoL outside the engine build temporary state (dashed) and undo it after routing, while MoL2 outside the engine reads only H19 and the option embeddings cached at startup and writes no state.

All three routers share the normal encoder pass, the global KV and the target-LoRA replay. They differ in the routing work inserted between the encoder and layer 20 (Figure 8). In later turns the encoder processes C[-128:] + U, but MoL2 pools only the H19 of U, which already carries the history through the global KV and the local window.

With TP = 8, routing must behave identically on every rank: all ranks call model.embed (which contains an all_reduce) in the same order, the router runs in eval mode, and rank 0 broadcasts the result. At the model interface, the design calls for four hooks: encode up to layer 19, embed the option tokens, pause for routing, and resume from layer 20 under the target LoRA. Scheduling and request-transfer integration still depend on the serving engine.

Latency: a millisecond routing decision

For each of five sampling seeds, we draw 10 samples per class from LoRA train: 200 samples in total, 12–3,978 tokens long, with a median of 150. Within each seed, four modes run one after another in the same TP = 8 process: Direct (no routing, correct target LoRA), MoL (L0 decode, up to 64 tokens), Contextual MoL and MoL2 (warm option cache). All modes use the same input and generate 512 tokens under the correct target. We record the routing decision time and the turn latency at the 1st, 128th and 512th decoded token, paired with Direct on the same sample.

The standard deviation across samples reflects input length and similar differences; the one across seeds is over the five per-seed means.

Figure 9. Time per routing decision (linear axis, seconds). Bars are the mean over 200 samples with ±1 standard deviation; white diamonds mark the median, black ticks the p95, grey dots individual samples. MoL2 sits almost on zero at this scale.

MoL2 routes in about 1/677 of the time of Contextual MoL and 1/5,586 of the time of MoL (Figure 9). Of MoL's 15.5 s, about 4.6 s is the route-prompt prefill and 10.9 s the label decode. On these 200 samples MoL makes 17 errors, 13 of which send L1 samples to L2.

Input length explains part of the spread across samples (Table 7). Contextual MoL's decision time rises from 1.66 s below 64 tokens to 2.39 s above 1,024, and MoL's from 14.3 s to 17.1 s, most of which is the fixed route-prompt prefill and label decode. MoL2 stays at 2.6–2.9 ms at every length. Direct's time to the first token levels off at about 24 s above 128 tokens, because its replay covers at most 128 tokens, one token at a time, so routing takes its largest share on short inputs. Length and class overlap in this sample: 23 of the 29 inputs above 1,024 tokens are L2, and 31 of the 32 below 64 are L0 or L1.

Table 7. Routing decision time by input length over the 200 latency samples. Times are mean ± standard deviation across the samples in each bucket. The percentage in brackets is routing's share of the turn at decode 1, computed from the means as in Table 8.

For MoL and Contextual MoL the increment roughly equals the routing time. MoL2's increment is negative, which routing cannot cause: MoL2 runs the same turn as Direct plus 2.77 ms, yet its prefill and decode are both about 2.2% faster, consistently across seeds. Within each seed, each mode ran over that seed's 40 samples (about 2 hours) before the next. The blocked measurement order leaves the paired comparison exposed to machine drift. Until the modes are re-measured interleaved, MoL2's end-to-end overhead should be taken as the 2.77 ms decision itself, 0.014% of the turn at decode 1.

Routing happens once per turn, while decode time grows with the reply, so longer replies shrink routing's share. A linear fit to Direct (about 19.65 s + 336 ms per token) gives routing's share of the turn (decode = 1 measured; 256 and 4,096 extrapolated):

Table 8. Routing's share of turn latency when routing once per turn, from the linear fit to Direct. The 4,096-token row is extrapolated beyond the measured range.

Figure 10. Routing overhead vs decode length. Left: routing's share of end-to-end latency when routing once per turn. Right: routing time relative to decode time when switching often; the dashed line marks equal time.

Routed once before a long reply, even MoL's 15 s is tolerable: about 1% of the turn at 4,096 tokens (Figure 10a). Frequent switching, between tool calls or mid-generation, removes that amortization (Figure 10b). Routing every 128 tokens costs 36% of the decode time under MoL, 4.4% under Contextual MoL and 0.006% under MoL2; every 16 tokens, it costs 2.9× the decode time under MoL, 35% under Contextual MoL and 0.05% under MoL2. Only MoL2 stays negligible at any frequency, and since it writes no temporary KV it can run at any point in the generation. These numbers count routing only; an actual switch adds one bounded replay.

Training-data recipe for the pointer

Keeping descriptions out of the backbone makes MoL2 fast, but leaves the pointer head to learn how queries match descriptions from training data.

Datasets

The routers are trained and evaluated on two kinds of data (Table 9); individual sources are cited in the table. For kev-suites, we use the decision-v7 subset at revision cc4bac803e73112689ec327ffa481c519cbc7a05.

Table 9. The generic and LoRA datasets used to train and evaluate the routers.

Generic data teaches the router to read descriptions and choose: every question has its own candidate set, so fixed labels cannot be memorized. LoRA data always has the same four deployed candidates. The LoRA benchmark shares classes and descriptions with LoRA train, so it measures generalization to new samples, not to unseen domains. L1 makes up 73% of the benchmark, so we also report per-class accuracy.

Mixed data balances domain and generic routing

We compare three ways to organize the two kinds of data: A trains on generic data only, so the LoRA domains are never seen in training; B mixes both; C trains as in A, then continues on LoRA data only. All three use the same MoL2 architecture and optimizer, train each stage for 30 epochs, keep the last checkpoint, and run 5 seeds. For comparison, the same experiment was run on Contextual MoL with one seed and checkpoint selection on generic dev.

Figure 11. Accuracy under the A/B/C training modes. (a) MoL2: bars are the 5-seed mean, error bars ±1 standard deviation, black dots individual seeds. (b) Contextual MoL, single seed.

Among the three modes, B gives the best balance of domain and generic routing accuracy (Figure 11a): it has the highest generic train, generic test and LoRA benchmark accuracy. With generic data alone (A), MoL2 reaches only 28.10% on the LoRA benchmark (13.5%, 30.4%, 13.2% and 50.7% on L0–L3), swinging between 6.9% and 59.0% across seeds; with descriptions seen only through the token embedding, it struggles to match descriptions absent from training.

Staged training (C) causes severe forgetting. LoRA train becomes nearly perfect, but generic test falls from A's 55.22% to 30.72%, and even generic train, which the router was trained on, falls from 62.20% to 34.32%. The LoRA benchmark reaches only 83.51%, 11 pp below B, with a 9.3 pp standard deviation across seeds (L3 alone ranges from 35.0% to 100%). Continuing on four fixed candidates pushes the router to memorize them.

Mixed training (B) keeps both abilities. Its generic test accuracy is 9 pp above A’s (64.26% vs 55.22%), with the smallest spread across seeds. Adding LoRA data improves generic routing as well. Per class, B scores 95.7%, 94.8%, 95.2% and 90.7% on L0–L3. Generic training accuracy remains limited across all three modes: MoL2 answers only 34%–75% of the generic training samples correctly, and generic train is only 4–11 pp above generic test.

Contextual MoL also does best with B, but MoL2 amplifies the problems of A and C (Figure 11b). In mode A, Contextual MoL reaches 89.15% on the LoRA benchmark against MoL2's 28.10%; in mode C its generic test drops only 1.8 pp, against MoL2's 24.5 pp, although its generic CE rises from 1.06 to 2.80. Encoding descriptions through the backbone makes Contextual MoL less sensitive to unseen descriptions and distribution shifts; MoL2 trades that for millisecond latency and needs training data covering both kinds of routing.

We use mixed training for the MoL2 routing results. Across five seeds, LoRA benchmark accuracy ranges from 89.15% to 96.69%, averaging 94.63%, compared with 97.31% in the qfix run. The few-point variation across seeds matters when comparing routers. In this setting, training the MoL2 router from scratch took about 8 hours: 6.6 hours to extract H19 features for all 24,259 training and dev samples (0.88 s per generic sample, 1.24 s per LoRA sample), then 84 minutes to train the 15.25M-parameter head on one GPU for 30 epochs.

The cost: fine-grained generic routing

Under the same mode B, MoL2 (the single run in Table 6) roughly matches Contextual MoL on the LoRA benchmark (qfix 97.31% vs 98.55%), but its generic test accuracy falls from 78.26% (1,127/1,440) to 65.14% (938/1,440). The 5-seed mean gives 64.26 ± 1.52% (925/1,440 on average) and the same conclusion (Figure 12). The drop concentrates in sources with many candidates and fine-grained labels.

Together, banking77, dbpedia14 and trec account for 139 of the 189-sample gap (145 of 202 on the 5-seed mean). A plausible explanation is that descriptions seen only through the embedding are harder to tell apart when there are many fine-grained labels. The four MoL domains differ widely, so the LoRA benchmark is unaffected, and for them trading about 13 pp of generic accuracy for about 677× faster routing and no temporary KV is worth it. As domains grow more numerous and finer-grained, this loss may become a bottleneck.

Figure 12. Training curves of Contextual MoL and MoL2 in mode B. Top row: loss; bottom row: accuracy; vertical lines mark the selected checkpoint. Contextual MoL is a single seed selected on generic dev; MoL2 is the 5-seed mean with ±1 standard deviation shaded, taken at epoch 30.

A few hundred LoRA samples are enough

How much data does MoL2 need, and what does each kind contribute? With the architecture and recipe fixed, we vary only the amount of each kind of data in 31 runs: group A uses no generic data and 10%–100% of the LoRA data; group B uses all generic data and 0%–100% of the LoRA data; group C scales both together from 10% to 100%. These groups vary data quantity. Group B starts with generic data alone and ends with the full mixed dataset, matching the data used in the generic-only and mixed-training runs. Those points were trained separately from the modes, so individual seeds differ, but their 5-seed means on generic test and the LoRA benchmark are within 2 pp of the modes'. Subsets are stratified and nested across ratios. All 31 runs were done twice with the same training seed but different subset-sampling seeds. Figure 13 reports the mean of the two runs.

Figure 13. Data scaling under the LoRA-only, full-generic and jointly scaled settings. Accuracy is measured on generic test, LoRA train and the LoRA benchmark as the LoRA data ratio grows. Solid lines show the mean of two subset-sampling runs; the shaded band spans the two.

Very little LoRA data is needed. With generic data alone, benchmark accuracy averages 28.0% and ranges from 11.4% to 50.7% across seeds. Adding 10% (722 samples) lifts it to 90.4 ± 4.1%, and 20% (1,443 samples) to 91.5 ± 1.2%; from 40% on, the benchmark stays at 94–96%. A few hundred routing samples make a new domain recognizable.

Generic data determines general routing ability and does little for the four fixed domains: group A, with no generic data, stays at 28–31% on generic test but still reaches 91–96% on the LoRA benchmark. When generic data grows together with LoRA data in group C, generic test rises from 42.2% to 62–63%. The two kinds do not conflict, since adding LoRA data to the full generic set in group B raises generic test from 53.5% to 62–67%.

Seed spread is largest when one kind of data is missing. At B's 0% point the benchmark standard deviation is 19.3 pp. In group A, single seeds drop to 69.2%–86.2% on the benchmark at 40%–70%, giving standard deviations of 4.5–12.3 pp there, while groups B and C stay within 3.1 pp from 40% on. Elsewhere the standard deviation on generic test and the benchmark is 0.7–5.4 pp and below 3.5 pp at most points, so wiggles of a few pp between adjacent ratios are noise. The conclusions above rest on gaps of at least 8.8 pp (generic test at B's 0% point vs its 10%–100% points).

In the serving engine, prefill and decode run in separate processes, KV moves between them, and decoding uses CUDA graphs and speculative decoding. MoL2 has to choose a LoRA during the request’s normal forward pass and make sure the decode worker applies that choice. We capture H19 during prefill and carry the decision with the request’s KV transfer (Figure 14).

Figure 14. A MoL2 turn and a base turn on the deployed engine. The prefill worker captures H19 from the pass that builds the KV and scores it against the cached option embeddings; the routing code rides the request's own KV transfer; the decode worker applies the chosen adapter on layers 21–39. The base turn shares the same decode step.

The route rides the prefill

A router outside the engine pays the prefill twice. The client-side version sends one request to route, prefilling the context only to expose H19, and a second to answer, prefilling the same context again under the chosen adapter. On this model the first pass is unnecessary: the encoder takes no LoRA, so the prefill that builds the KV already computes H19 under BASE.

The capture rides the running forward. The engine captures and pools layer 19’s activation during the running prefill, then scores each request against the cached option embeddings. Routing runs on the prefill scheduler, outside the per-token decode loop. Two details mattered. The speculative-decode path already fills a [tokens, 3·hidden] auxiliary buffer for its draft head, so the capture keeps a slot of its own. The layer-19 activation is three-dimensional, pooled with a mean over the inner axis; a flatten gives width 3·hidden and breaks the LM head on both processes.

An explicit adapter always wins. The pointer routes only requests that name no adapter, and the precedence is enforced per request before the decision crosses the process boundary: a batch-level scorer that overwrites per-request choices sends an explicitly addressed request to whatever scored highest that step. Scoring is per request for the same reason, since one prefill batch holds many turns.

The decision rides the KV transfer

The decision is computed where H19 lives and applied where decoding runs. Request ids change at the entry router and adapter ids are per-process UUIDs, so neither name survives the hop. What crosses is a 31-bit FNV-1a hash of the adapter path (0 for none), written into the per-request row of the metadata that travels with this request's own KV transfer; the decode worker reads it when the transfer commits and maps it back through its local registry. There is no side channel, no polling and no new collective. Configuration travels as a file rather than an environment variable, because the scheduler children do not inherit the launcher's environment; the file is cached on its mtime, since parsing it once per decode step cost about 4 ms/step.

A decision that is never applied looks exactly like free routing. Each scheduler counts what it wrote, read and applied, and one endpoint sums the counters across processes, so a routed turn that reaches decode without an adapter is a counter rather than a mystery.

The cost: the same router inside and outside the engine

In-engine routing performs close to fixed routing in this workload. We run the same LoRA workload under five routing setups on one engine instance, using client-specified adapters as the fixed-route baseline (Table 10). The measured in-engine throughput is close to the fixed-route baseline: +2.6% end-to-end tokens/s, and 22.65 vs 22.11 ms per step. Every setup that routes outside the engine instead pays the routing pass as a full second request: −42.9% for MoL2, −50.1% and −60.2% for Contextual MoL and MoL, whose routing also builds temporary state with extra forward passes.

Table 10. Routing setups on the deployed engine, one workload of LoRA turns with identical prompts and batch across rows. The baseline is the client naming the adapter. Per-step time and accept length are measured inside a single request, and a setup that routes outside the engine splits a turn across two requests, so those rows report only tokens/s and wall time.

Figure 15. End-to-end tokens/s relative to the fixed route. The same MoL2 routed inside the engine matches the baseline within noise; routed outside the engine, MoL2 loses 43% of its tokens/s, Contextual MoL loses 50% and MoL 60%.

Routing-head computation stays on the prefill side. The decode worker receives the selected adapter through KV-transfer metadata and applies it before generation.

On DeepSeek-V4.1, LoRA placement makes it possible to share history as well as weights. Keeping the encoder and layer-20 attention free of LoRA lets adapters share the encoder state and global KV, with only a 128-token local replay needed on a switch. MoL2 uses the encoder’s H19 to choose a LoRA in 2.77 ms. Bringing that decision into the serving engine completes the path from a routing result to the adapter that actually handles the request.

• Keep the layer-20 attention free of LoRA; the FFN can carry one. The layer-20 attention (compressor and indexer) is the last global KV source, and a LoRA there puts a trace of the source into the global KV; a LoRA on the layer-20 FFN only affects later layers. Layers 21–39 carry LoRA as usual.
• Replay only 128 tokens on a switch. The result does not depend on the source, and the PPL ratio is 0.964–0.995; the error concentrates in the first dozen tokens after a switch and does not grow with context length.
• Route from H19 and look up descriptions in the embedding. One MoL2 decision takes 2.77 ms and writes no temporary KV; even routing every 128 tokens costs 0.006% of the decode time.
• Train the router on mixed generic and LoRA data. Do not train in stages; a few hundred routing samples per new domain are enough.
• Keep routing inside the serving engine. Capture H19 during normal prefill, carry the selected adapter’s identifier with the request’s KV transfer, and apply it on the decode worker. This avoids a second prefill and keeps routing computation out of the per-token decode loop. In our measured workload, throughput stays close to the fixed-route baseline.

Together, these let independently trained LoRAs serve the same conversation over one shared base and history, with adapter selection and switching handled as part of normal request execution.

The next step is to test the full serving path on broader workloads: whether replay preserves quality with stronger text LoRAs, whether routing selects the most useful LoRA for real tasks, and how the engine handles concurrent requests using different adapters. The current results provide a starting point, but do not yet establish these outcomes.

Adding a new LoRA still requires labelled routing examples and retraining the pointer head. The LoRA’s own training data can supply those examples, while the backbone and existing adapters remain unchanged. A longer-term goal is to let the router recognize a new LoRA from its description alone, making new capabilities easier to bring into service.

[1] Macaron-V1: Towards Open Continual Learning with Self-Improvement and Mixture-of-LoRA (Mind Lab et al, 2026)

[2] DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligence (DeepSeek-AI et al, 2026)

[3] DeepSeek-V4.1-Flash: Pushing the Limits of KV Cache Compression (DeepSeek-AI et al, 2026)

[4] Editing Models with Task Arithmetic (Ilharco et al, 2023)

[5] TIES-Merging: Resolving Interference When Merging Models (Yadav et al, 2023)

[6] AutoGen: Enabling Next-Gen LLM Applications via Multi-Agent Conversation (Wu et al, 2023)

[7] MetaGPT: Meta Programming for a Multi-Agent Collaborative Framework (Hong et al, 2024)

[8] ChatDev: Communicative Agents for Software Development (Qian et al, 2024)

[9] CAMEL: Communicative Agents for “Mind” Exploration of Large Language Model Society (Li et al, 2023)

[10] LoRA: Low-Rank Adaptation of Large Language Models (Hu et al, 2022)

[11] Overcoming Catastrophic Forgetting in Neural Networks (Kirkpatrick et al, 2017)

[12] DroidSpeak: KV Cache Sharing Across Fine-Tuned Model Variants (Liu et al, 2026)

[13] KVCOMM: Online Cross-Context KV-Cache Communication for Efficient LLM-Based Multi-Agent Systems (Ye et al, 2025)

[14] Cache-to-Cache: Direct Semantic Communication Between Large Language Models (Fu et al, 2026)

[15] Activated LoRA: Fine-Tuned LLMs for Intrinsics (Greenewald et al, 2025)

[16] Inter-GPS: Interpretable Geometry Problem Solving with Formal Language and Symbolic Reasoning (Lu et al, 2021)

[17] kev-suites (Palmer, 2026)

[18] Character-Level Convolutional Networks for Text Classification (Zhang et al, 2015)

[19] The Multilingual Amazon Reviews Corpus (Keung et al, 2020)

[20] Efficient Intent Detection with Dual Sentence Encoders (Casanueva et al, 2020)

[21] BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questions (Clark et al, 2019)

[22] Learning Word Vectors for Sentiment Analysis (Maas et al, 2011)

[23] A Broad-Coverage Challenge Corpus for Sentence Understanding Through Inference (Williams et al, 2018)

[24] Recursive Deep Models for Semantic Compositionality Over a Sentiment Treebank (Socher et al, 2013)

[25] Learning Question Classifiers (Li et al, 2002)

[26] AutomationBench (Shepard et al, 2026)

[27] The Tool Decathlon: Benchmarking Language Agents for Diverse, Realistic, and Long-Horizon Task Execution (Li et al, 2026)

[28] ChatBench: From Static Benchmarks to Human-AI Evaluation (Chang et al, 2025)

[29] DeepSWE: Measuring Frontier Coding Agents on Original, Long-Horizon Engineering Tasks (Huang et al, 2026)

[30] SWE-Marathon: Can Agents Autonomously Complete Ultra-Long-Horizon Software Work? (Desai et al, 2026)

[31] Terminal-Bench: Benchmarking Agents on Hard, Realistic Tasks in Command Line Interfaces (Merrill et al, 2026)

[32] Adapting Causal Encoder–Decoder Models with LoRA (Chiang et al, 2026)

Mind Lab

Core Contributors

Yuhua Zhou, Smith Li, Song Cao, Kieran Liu, Adrian Zhou, Andrew Chen, Pony Ma

Team

Amy Chen, Jingwei Cao, Song Cao, Andrew Chen, Kaijie Chen, Cleon Cheng, Steven Chiang, Hera Feng, Jun Gao, Vincent He, Nolan Ho, Ori Hong, Xinyu Hu, Bruce Jin, Fancy Kong, Wynn Koo, Andrew Lei, Junjie Li, Lucian Li, Miles Li, Smith Li, Theo Li, Wish Liao, Kieran Liu, Lawrence Liu, Ryan Liu, Yuxin Lu, Nemo Lv, Pony Ma, Minyu, Peixuan, Taotao Qian, Luozheng Qin, Aiden Tang, Nolan Weng, Jinxiang Xie, Bosong Yang, Rio Yang, Regis Ye, Young Ye, Albert Yin, Josh Ying, Atlas Zeng, Martin Zeng, Anya Zhang, Harry Zhang, Jason Zhang, Dewu Zheng, Adrian Zhou, Yuhua Zhou, Juno Zhu, Yuhan Zhu and Mindverse Team

Names are listed alphabetically within team.