avichawla-superlinked-sie-2026

Superlinked Inference Engine: One Engine for All Small Models in Your Agent Pipeline

Author: Avi Chawla (@_avichawla) Published: 2026-07-16 Source: https://x.com/_avichawla/status/2077653695123378321 Article URL: https://x.com/i/article/2077494356194529281

---

The Problem

When building AI agent pipelines that use multiple small specialized models (embedding, reranking, OCR, vision, extraction, generation), teams face a fundamental infrastructure problem: each standard serving tool serves only one model type, forcing one model per GPU.

Why One Model Per GPU Is Wasteful

• vLLM occupies ~90% of GPU at startup, TEI grabs its part, OCR needs something too
• Each tool was written on the assumption that it owns the card
• None of them will give memory back to the others
• No shared queue across them, no priority system
• Small models are accurate enough but GPU fragmentation re-creates the waste you were trying to escape

Three Common Escape Routes (Each Has Problems)

1. Managed API (OpenAI, Cohere): Bill increases linearly with usage, can’t use fine-tuned models, data leaves your environment
2. Serverless GPUs: Cold start problem (seconds to load gigabytes), keeping instances warm means paying for idle GPUs again
3. Self-hosting: The right path for cost and control, but standard tools force one model per server → one GPU per model

What the Ideal Serving Stack Should Do

1. Run every kind of model (embeddings, reranking, OCR, vision, extraction, generation) behind one API
2. Fill the GPU instead of padding it (batch by compute cost, not item count)
3. Load and evict models as traffic moves (like a browser cache)
4. Ship with production layer (routing, autoscaling, monitoring, GPU pools)
5. Adding a new model should be a config change, not a redeployment

The Solution: Superlinked Inference Engine (SIE)

An open-source engine that runs all model types on shared GPUs in one cluster.

Key Features

• One API, four calls: encode (vectors), score (reranking), extract (structured fields), generate (LLM output)
• Smart batching: Groups requests by compute cost, pads only to longest in batch (not global maximum)
• Gateway-based batching: Shared queue across all workers, each worker forms full batch before touching GPU
• Dynamic load/eviction: Models load on first request, LRU eviction when memory runs short
• Production-ready: Gateway, autoscaling, dashboards, Terraform for AWS/GCP
• 85+ pre-tuned models: Config files with optimized settings for each model/GPU combination
• Offline capable: Can run with pre-downloaded weights

Implementation Example

A pipeline with four model architectures hitting one endpoint:

1. encode() - Embedding model turns passages into dense vectors
2. score() - Cross-encoder reranks query against passages
3. extract() - Extraction model pulls structured fields with confidence scores
4. generate() - LLM composes answer from top-ranked context

All via one client, one server, one endpoint.

Integrations

Chroma, Qdrant, Weaviate, LanceDB

Repository

github.com/superlinked/sie

---

Key Insights

• Using small specialized models is the right approach, but switching to small models doesn’t make inference cheaper by itself
• The savings only appear when models share GPUs
• That requires a single engine that can run all of them
• The moment you serve each model with a different tool, each one takes a GPU of its own again