CLEAR-MoE: Shared-Basis Expert Extraction from Frozen Vision Transformers via Calibration-Driven Layer Selection

Forward $N_{\text{cal}}$ representative images through the frozen model, recording pre-FFN activation tensors at each of the $L$ selected layers via PyTorch hooks. For DeiT-Small ($d{=}384$, $N_{\text{cal}}{=}200$, 197 tokens/image), this produces a $[39{,}400,\,384]$ activation matrix per layer.

Each FFN layer $l$ receives a composite score:

where sparsity is the fraction of activation magnitudes below 0.01; clusterability is the Silhouette score of $k$-means with $k{=}E$ on the calibration activations; and sensitivity is the normalized logit change when the layer’s FFN output is zeroed. Sensitivity is subtracted because high sensitivity implies high degradation risk if expertized. The top-$k{=}L/2$ layers by $\mathcal{S}$ are selected; the remaining layers retain their original dense FFN. Table II reports per-layer scores for DeiT-Small (200-image calibration); block 0 ranks last due to sensitivity $=$ 0.946.

For each selected layer, fc2 ($W_{2}\in\mathbb{R}^{d\times d_{\text{ffn}}}$) is decomposed via truncated SVD; fc1 is shared identically across all experts:

$W_{\text{shared}}=U_{r}\Sigma_{r}V_{r}^{\top}$ retains the top-$r$ singular values (default $r$ = 50% of matrix rank, $r{=}d/2{=}192$ for DeiT-Small where $W_{2}{\in}\mathbb{R}^{384\times 1536}$ has maximum rank 384; materialised as a dense matrix before inference, with no factorised compute reduction). $k$-means with $k{=}E$ partitions calibration tokens into $E$ clusters; the per-cluster residual is $W_{e}^{\text{res}}=(W_{2}-W_{\text{shared}})\cdot s_{e}$, where $s_{e}$ scales by the ratio of cluster-mean to global-mean activation norm. Because $s_{e}$ is a scalar, all residual experts share the same directional component $(W_{2}-W_{\text{shared}})$, differing only in magnitude; a misrouted token therefore receives a different scaling but the same directional correction. During inference, $z$ is computed once and reused for both the shared and residual paths.

A router $g_{\theta}$ predicts the cluster assignment of each token. The $k$-means labels from Phase 3 provide supervised targets: router training minimizes cross-entropy between $g_{\theta}(x)$ and cluster label $c$ over the calibration tokens, using AdamW ($\text{lr}{=}10^{-3}$, 5 epochs, cosine decay). Three router architectures are evaluated: Linear ($d\times E$ parameters, 9.2K total), MLP (hidden layer $d/6$, 149K), and Adaptive (confidence-threshold top-1/top-2 escalation, same parameter count as Linear).

Three CUDA backends are implemented: (1) Naive: boolean-mask loop over experts (reference). (2) Grouped: tokens are sorted by expert index; one GEMM per contiguous sub-batch. No padding; throughput stable under imbalance. (3) cuBLAS: expert sub-batches are padded to the largest and stacked into a 3D tensor dispatched to a batched GEMM call. Fastest at balanced load; degrades at high imbalance from padding waste.

Table III ablates what each architectural choice contributes to accuracy and latency. Configurations D0–D8 span the design space from dense baseline to shared-basis CLEAR-MoE FFN, isolating the effect of the shared basis, the routing mechanism, and the expert assignment strategy.

The shared SVD basis is the dominant accuracy-preserving component. D3 (shared basis + full global residual, no routing) achieves exact reconstruction: algebraically, $W_{\text{shared}}{\cdot}z+(W_{2}{-}W_{\text{shared}}){\cdot}z=W_{2}{\cdot}z$, matching D0 at 86.73%. D2 and D7 (disjoint experts, no shared fc1) collapse to 65–66% regardless of routing strategy: disjoint expert weights estimated from 200 calibration images fail to generalize. In contrast, D4/D5 (shared basis, random routing) retain 86.62–65%, only 0.08–0.11 pp below D0; D6/D8 (shared basis, learned routing at 95% accuracy) achieve an identical $-$0.08 pp. Routing quality has limited measured impact once the shared basis is present.

No latency reduction on GTX 960. CLEAR-MoE FFN (D4–D8) executes shared fc1+fc2 on all tokens plus residual paths that collectively process all $T$ tokens ($T/E$ per expert $\times$ $E$ experts), adding approximately one full fc2-equivalent computation; total arithmetic is $\approx\!1.5\times$ dense FFN. On bandwidth-limited memory (112 GB/s), this incurs 1.3–1.7$\times$ latency overhead (78–100 ms vs. 59.6 ms). Only disjoint experts (D2/D7) are faster (48–49 ms) by eliminating the shared path entirely, at the cost of 21 pp accuracy.

Table IV compares all 11 layer-selection strategies on the same MoE configuration ($E{=}4$, $k{=}6$ of 12 FFN layers, composite scoring).

All 11 policies span only 0.21 pp (86.62–86.83%), confirming that accuracy is policy-insensitive. The shared basis preserves quality regardless of which 6 blocks are expertized. The composite score’s practical value is principled block 0 exclusion (sensitivity = 0.946): L2 (first-$k$) includes block 0 and incurs the worst $-$0.13 pp drop. L10 avoids it, achieving 82.4 ms, comparable to L3 (last-$k$, 80.3 ms) while additionally excluding high-sensitivity blocks. L7 (high-sensitivity, deliberately selecting the most sensitive blocks as a stress-test baseline) achieves only $-$0.08 pp accuracy loss but is the slowest policy at 102.5 ms, demonstrating that layer choice affects latency more than accuracy.

We benchmark three CUDA backends ($B{=}8$, $T{=}1568$, $E{=}4$, GTX 960) under four imbalance levels (Table V). At balanced load, cuBLAS peaks at 728 K tok/s (2.79$\times$ CPU serial). At 80% imbalance, padding waste collapses cuBLAS to 558 K tok/s ($-$23%). Grouped dispatch remains stable at 665–747 K tok/s across all imbalance levels. Recommendation: cuBLAS for predictably balanced load; Grouped otherwise.

Fig. 2 places each CLEAR-MoE operation on the GTX 960 roofline (peak FP32: 2.4 TFLOP/s, BW: 112 GB/s, ridge: 21.4 FLOPs/B). Dense FFN (AI = 74.3) and expert GEMMs (AI = 22.7) are compute-bound. The router gate (AI = 1.9) and token sort (AI = 1.0) are both deep in the memory-bound regime: 11$\times$ and 21$\times$ below the ridge point. This indicates that routing and token sort are strongly memory-bound (11–21$\times$ below the ridge point), making fused dispatch kernels a plausible high-leverage optimization. Expert GEMMs are already near the compute ceiling and offer diminishing returns.

We ablate SVD rank, expert count, router architecture, calibration size, and random seed (Table VI); findings are:

SVD rank ($r\in\{16,32,64,96,128,192,256\}$): accuracy range is 86.62–86.75% (0.13 pp). Reconstruction error decreases monotonically from 0.91 to 0.29, but this improvement does not translate to accuracy. Even $r{=}16$ captures the shared basis sufficiently.

Expert count ($E\in\{2,4,8,16\}$): accuracy is 86.65–86.80% for $E\in\{2,4,8\}$; $E{=}16$ drops 0.26 pp as 200 calibration images provide insufficient statistics to estimate 16 fine-grained clusters. Latency grows $\sim$9 ms per doubling of $E$ (73 ms at $E{=}2$ to 98 ms at $E{=}16$). No empty experts appear at any count.

Router architecture: all three routers (Linear, MLP, Adaptive) achieve identical 86.68% Top-1. MLP routing accuracy reaches 0.980 vs. 0.925 for Linear, a 5.5 pp improvement in routing precision with zero downstream benefit. The gap between random routing (D4/D5, Table III) and all learned routers is at most 0.06 pp (roughly two changed predictions out of 3,925). While this numerically exceeds the 3-seed standard deviation of $\pm$0.02 pp (Table VI, row “Random seed”), a paired significance test would be needed to confirm a reliable effect; we treat 0.06 pp as practically negligible. Fig. 3 provides a mechanistic view: despite D6 (learned router) reducing mean per-token routing entropy $3.5\times$ versus D5 (random, $\ln 4{\approx}1.386$ nats), the accuracy impact is negligible.

Fig. 4 plots accuracy gap against router training across 13 configurations (D5 at $x{=}0$, no trained router); no strong monotonic relationship is visible, consistent with the hypothesis that shared-basis quality governs accuracy regardless of routing quality.

Calibration size ($N\in\{50,100,200,500\}$, multiple subsets for $N{<}500$): accuracy spans 86.55–86.70% across all sizes; $N{=}50$ yields 86.65–86.70% (within 0.15 pp of $N{=}500$). Router accuracy climbs from 0.83 to 0.95 as $N$ grows, without accuracy benefit.

Reproducibility: 86.70 $\pm$ 0.02% Top-1 across seeds 42, 123, 456 (full pipeline, composite scoring, $E{=}4$, $N{=}200$).

Table VII tests whether the shared-basis finding extends across architectures. We apply the full CLEAR-MoE pipeline to five backbones spanning 5.7–86.6 M parameters and two pretraining lineages (DeiT, ViT); all other settings are identical to the DeiT-S ablation ($E{=}4$, $N{=}200$, seed 42).

Routing differences are numerically small. Learned routing (D6) provides a marginal numerical advantage over random routing (D5) in 4 of 5 backbones; however, the per-backbone improvements range from 0.02 to 0.10 pp, too small to confirm without paired significance testing. The one exception is ViT-B/16, where D5 ($-$0.08 pp) outperforms D6 ($-$0.10 pp): The ViT-B result coincides with the highest observed load skew (0.150, vs. 0.094 for DeiT-B), which may contribute to the difference; no ablation was performed to confirm a causal link. Taken together, these results are consistent with the hypothesis that routing quality has limited impact once the shared SVD basis is present; a definitive claim would require paired prediction-level testing across additional seeds. Fig. 5 visualises all 10 configurations.