S-LoRA: what the paper actually says about serving thousands of LoRA adapters

Reading Sheng et al. (UC Berkeley, 2023) after watching our per-adapter serving approach fall apart at 50 concurrent tenants.

The fine-tuning pipeline worked. We'd used LoRA to build per-customer models: each enterprise client got a LLaMA-13B adapter trained on their internal documents, their terminology, their output preferences. The adapters were small — 80MB each — and training was fast. The sales story was good: one base model, infinite personalization.

Then we had to serve them.

The naive approach is obvious: load a customer's adapter, merge it with the base model, run inference, then swap in the next customer's adapter. We tried this. At 8 concurrent tenants it worked. At 20 tenants the latency spikes started. At 50 tenants we were spending more time on adapter swapping than on actual inference. The GPU was busy but the throughput was terrible, and the reason was structural: we'd optimized the fine-tuning problem and completely ignored the serving problem.

S-LoRA — "S-LoRA: Serving Thousands of Concurrent LoRA Adapters," Sheng, Lin, Gonzalez, Stoica, and Zheng, UC Berkeley, 2023 — is the serving problem. It introduces three mechanisms that together enable a single A100 cluster to serve thousands of distinct LoRA adapters concurrently with throughput comparable to serving a single model.

The problem the paper is actually solving

The LoRA inference path has two components:

y = W₀x + (α/r) BAx

The base model output W₀x is the same for every request regardless of adapter. The LoRA correction (α/r) BAx depends on which adapter is active for that request.

This structure creates a batching problem that doesn't exist for standard LLM serving.

In standard serving (vLLM, TGI), every request in a batch runs the same weights — you compute Wx once for the full batch. With adapters, different requests may have different B and A matrices. The naive resolution is to group requests by adapter, then run each group as a separate forward pass. But this destroys batching efficiency: if you have 100 concurrent requests spread across 40 adapters, your average batch size is 2.5. You've effectively crippled continuous batching.

The memory problem is separate and equally bad. A LLaMA-13B base model is ~26GB in FP16. An 80MB LoRA adapter on top of that fits easily. Two hundred adapters at 80MB each is 16GB — which fits, barely, if you sacrifice KV cache. Two thousand adapters is 160GB, which doesn't fit at all. The obvious response is to keep adapters on CPU and load them on demand. The problem is that loading 80MB from CPU DRAM to GPU HBM takes 3–5ms over PCIe, and this shows up in your tail latency for every request whose adapter wasn't hot.

S-LoRA addresses both problems.

Unified paging: the memory management system

The paper's first contribution is treating LoRA adapter weights with the same memory management system used for the KV cache.

If you've read the PagedAttention paper, the approach will be familiar. PagedAttention divides GPU HBM into a unified pool of fixed-size blocks shared between the KV cache and adapter weights. The KV cache allocates blocks as sequences grow; adapter weights are loaded into blocks when their adapters are activated and evicted when they're not needed.

This has a non-obvious consequence: adapter weights and KV cache compete for the same memory pool, so the scheduler can make tradeoffs between them. A scheduler that knows 40 of the 200 adapters in flight are hot can pre-fetch those into GPU memory while allowing the KV cache to grow for long-running decode sequences. Keeping these in separate memory pools would require predicting both sizes at server startup — which you can't do accurately in production.

The eviction policy matters. S-LoRA uses LRU eviction for adapter blocks. In practice, adapter access patterns have significant temporal locality: a customer with an active session will send several requests in a row, and you want their adapter to stay hot during that burst. LRU captures this correctly. Random eviction does not.

What "warm" means: An adapter is warm when all its weight blocks are resident in GPU HBM. For an 80MB adapter divided into 256KB blocks, that's 320 blocks. The first request for a cold adapter pays the CPU→GPU transfer cost (~4ms on PCIe 4.0). Subsequent requests within the eviction window are warm and pay no extra cost.

The failure mode: If you have more active adapters than your adapter memory budget allows, you'll thrash — constantly evicting and reloading adapter blocks. This can be worse than sequential serving because you pay both the batching overhead of mixed adapters and the transfer cost of constant eviction. The fix is to right-size your adapter memory budget based on measured access patterns, not to let the system find its own equilibrium.

Batched LoRA computation: why this requires new CUDA kernels

The second contribution is the computational mechanism that lets a single forward pass service requests using different adapters simultaneously.

Standard batched inference computes Y = W₀X where X is the [batch_size, hidden_dim] input matrix and W₀ is the weight matrix. This is a standard GEMM, highly optimized in cuBLAS and in attention kernels.

With heterogeneous adapters, you need:

Y_i = W₀x_i + (α/r) B_k A_k x_i   for request i using adapter k

The base model term W₀X is still one standard GEMM over the full batch. The LoRA corrections are the problem. You have a batch where each row x_i corresponds to a different (B, A) pair. This is a segmented GEMM: a batch of GEMM operations where each sub-batch has different weight matrices.

cuBLAS has cublasGemmBatchedEx for exactly this case — a batch of independent GEMMs with different weight matrices. But the standard implementation has overhead: it processes each sub-batch independently, meaning the kernel launch overhead accumulates for every unique adapter in the batch. If you have 40 unique adapters across 100 requests, that's 40 kernel launches for the LoRA path, plus the overhead of index computation and memory scatter.

S-LoRA implements custom CUDA kernels for what they call the "batched LoRA operator." The key optimization: consolidate the LoRA corrections for all requests into a single kernel that operates on a packed representation. Requests are sorted by adapter index (which is already done for memory locality), so the segmented GEMM is contiguous in memory. The kernel handles the variable-size segments with a warp-level segmented reduction.

The practical implication: the LoRA overhead per token is roughly constant regardless of the number of unique adapters in the batch, as long as the total batch size is large enough to amortize the kernel overhead. Below about 8 requests per batch, the kernel overhead dominates and you'd be better off with sequential per-adapter passes.

The two-pass structure:

# Pass 1: base model (full batch, same weights)
Y_base = W₀ @ X          # standard GEMM, full batch

# Pass 2: LoRA corrections (segmented, per-adapter)
Y_lora = batched_lora_op(X, adapters, adapter_indices)

# Combine
Y = Y_base + (alpha / r) * Y_lora

The base model compute is unchanged. The LoRA compute adds roughly 15–25% overhead at r=16 with the custom kernel, compared to 2–5x overhead with sequential per-adapter passes.

Tensor parallelism: the annoying design decision

Multi-GPU serving introduces a design question that the paper has to answer: how do you shard LoRA matrices across GPUs?

For the base model, tensor parallelism on the attention layers shards the weight matrix by columns: each GPU holds a [d, d/tp] chunk of W_q. The final result requires an all-reduce.

LoRA introduces two matrices A ∈ ℝ^(r×d) and B ∈ ℝ^(d×r). There are two options:

Option 1: Shard A. Keep A as [r, d/tp] on each GPU (matching the W_q sharding). Keep B as [d/tp, r] on each GPU. The LoRA path requires no extra communication — it fuses with the W_q tensor parallel all-reduce.

Option 2: Keep A full. Keep the full A matrix on each GPU, shard B by rows. The A computation is redundant (each GPU computes the same Ax), but the result feeding into B is already local.

The paper goes with Option 1 for column-parallel layers (Q, K, V) and a corresponding approach for row-parallel layers (O, FFN down-projection). The reason: redundant A computation (Option 2) wastes compute proportionally to the tensor parallel degree. For TP=4, you'd compute A four times and discard three of the results. At r=16 with d=4096, that's 4 extra [r, d] GEMMs per forward pass — small in absolute terms, but wasteful when you're trying to squeeze throughput.

The tradeoff: Option 1 requires that every GPU holds each active adapter's A matrix in full, even though it only needs a shard of B. For small r, this is acceptable — the full A matrix for r=16 is 16×4096×2 bytes ≈ 128KB. At large TP degrees with many adapters, this adds up.

The scheduler: matching adapters to batches

Unified paging and custom kernels are mechanisms. They're only as useful as the scheduler that decides which requests to batch together.

S-LoRA's scheduler is an extension of continuous batching with an adapter-awareness constraint. The core tradeoff: batching requests from the same adapter maximizes LoRA kernel efficiency but may leave GPU utilization on the table if adapter distributions are uneven. Batching requests from different adapters maximizes throughput for diverse traffic but increases LoRA kernel overhead.

The paper describes an "adapter-aware preemption" policy: when a new request arrives with a cold adapter, the scheduler considers whether loading that adapter is worth it given the current batch composition. If the batch is already processing 20 different adapters and the marginal request would be the 21st, the scheduler may defer it until the batch size for that adapter justifies the load cost.

In practice, this becomes a scheduling parameter: the minimum batch size before a cold adapter is loaded. Setting it too high increases queuing latency for rare adapters. Setting it too low causes excessive adapter thrashing.

The failure mode in production: When your traffic is long-tail distributed across adapters — most adapters see one request per minute, a few see hundreds — the scheduler spends most of its time managing cold adapter loads. The hot adapters batch efficiently; the long tail gets queued behind adapter load latency. The fix is tiered serving: dedicated GPU capacity for hot adapters, best-effort serving with longer queues for cold ones.

Numbers from the paper

The paper benchmarks on LLaMA-7B and LLaMA-13B with adapters at r=4 and r=16. Key results:

S-LoRA serves 2,000+ adapters concurrently on a single 4-GPU A100 node; naive sequential serving manages ~5.
Throughput is 4x higher than adapter-merging approaches at 100 concurrent adapters, 13B model.
Latency overhead vs. single-adapter serving: within 2x at p50 for moderate traffic; up to 3-5x at p99 under adapter-thrashing conditions.
Adapter load time from CPU: 3–5ms over PCIe 4.0 for an 80MB adapter. This is visible in TTFT (time to first token) but not in generation latency for long outputs.

The throughput comparison is the most production-relevant number. If you're serving 500 adapters and you currently batch by adapter (group-then-process), you're likely at 5–10% of what S-LoRA can achieve for the same GPU cluster.

What this means for system architecture

Before S-LoRA: Multi-tenant LoRA serving typically meant either (a) one GPU cluster per adapter — expensive and operationally painful — or (b) adapter merging at request time with sequential processing — low throughput. The standard advice was to collapse fine-tuned variants into as few adapters as possible.

After S-LoRA: You can treat fine-tuned adapters as cheap and numerous. The serving infrastructure can handle hundreds of adapters per node at reasonable throughput. The bottleneck shifts from "how do we fit these adapters" to "how do we route requests to the right adapter."

The routing problem is underrated. In a system with 2,000 adapters, you need a fast lookup from request metadata to adapter ID, and the adapter registry has to be consistent across your serving replicas. Adapter versioning (what happens when you retrain a customer's adapter?), hot-reload without request drops, and adapter access control become the new operational concerns.

When not to use this

When you have fewer than ~20 adapters. Below this threshold, just load all adapters into GPU memory and keep them merged or hot-unmerged. The scheduler overhead and custom kernel complexity of S-LoRA adds latency that a simpler system wouldn't have. This isn't a subtle tradeoff — the paper's benchmarks show S-LoRA's advantage growing with adapter count, not present at small counts.

When adapter distributions are uniform and highly diverse. If every user has a unique adapter and requests are distributed uniformly across all users, your batch will always contain requests from many different adapters. The segmented GEMM efficiency drops because each segment is tiny. You'll approach the sequential limit regardless of the paging system. Consider whether adapter consolidation (fewer, more general adapters) or a different personalization strategy (system prompt prefixes, prefix caching) would serve your use case better.

When you have strict p99 TTFT requirements below 50ms. Cold adapter loads take 3–5ms. For a user hitting a cold adapter, this adds directly to their time-to-first-token. Under p99 SLAs, a cold adapter load plus scheduling overhead can blow the budget. Either keep all relevant adapters warm with pre-fetching (requires traffic prediction) or accept that cold adapter requests go to a different SLA tier.

When adapters have highly variable ranks. S-LoRA's custom kernels assume a uniform rank across adapters in a batch — or at least small rank variance. If your adapter zoo has r=4 adapters mixed with r=64 adapters, batching them together in a segmented GEMM either wastes compute (padding smaller segments to r=64) or requires multiple kernel dispatches. Standardize on a single rank across your adapter fleet if S-LoRA is the serving target.

When your adapters are different base models. S-LoRA serves one base model with many adapters. If your "fine-tuned variants" are actually different model architectures or different quantizations of the same base, this is a different problem that requires model sharding, not adapter management.

What the paper gets right that implementations miss

The unified memory pool between adapters and KV cache is the most important design decision in the paper, and it's easy to skip when building on top of vLLM because you might implement adapter paging separately. The problem with separate pools is that they compete for the same physical memory through different allocators — the adapter pool over-reserves or the KV cache under-reserves, and you tune them statically at startup. The unified pool lets the runtime allocate memory dynamically to whichever is under pressure. In practice, this means the server degrades gracefully when adapter diversity spikes, rather than OOMing.

The tensor parallel design (shard B, keep A full) deserves more attention than it gets. Most LoRA implementations shard both A and B, which introduces an extra all-reduce in the LoRA path. For TP=2 this is minor. For TP=8 on a large cluster, the extra communication is significant. The paper's design is correct and should be the default for multi-GPU LoRA serving.

The scheduling paper-over is significant: S-LoRA's scheduler is described at high level, and the production behavior under adversarial traffic (many adapters, irregular bursts, long-tail distributions) is mostly left for practitioners to discover. The adapter-thrashing failure mode doesn't appear clearly in the paper's benchmarks because the benchmarks use controlled traffic distributions. Production traffic is not controlled. Budget time for scheduler tuning before going live.

References:

Sheng, Y., Lin, S., Gonzalez, J. E., Stoica, I., & Zheng, L. (2023). S-LoRA: Serving Thousands of Concurrent LoRA Adapters. arXiv:2311.03285.
Hu, E. J., et al. (2022). LoRA: Low-Rank Adaptation of Large Language Models. ICLR 2022.
Kwon, W., et al. (2023). Efficient Memory Management for Large Language Model Serving with PagedAttention. SOSP 2023.