Summary
Superlinked Inference Engine (SIE) is an open-source multi-model inference engine that runs embedding, reranking, OCR, vision, extraction, and generation models on shared GPUs through a single API. It solves the GPU fragmentation problem where standard tools (vLLM, TEI) force one model per GPU.
Key Features
- One API, four calls: encode (vectors), score (reranking), extract (structured fields), generate (LLM output)
- Smart batching: Groups requests by compute cost, pads only to longest in batch
- Gateway-based batching: Shared queue across all workers
- Dynamic load/eviction: LRU model loading like a browser cache
- Production-ready: Gateway, autoscaling, dashboards, Terraform for AWS/GCP
- 85+ pre-tuned models: Optimized configs for each model/GPU combination
- Offline capable: Pre-downloaded weights for air-gapped environments
How It Works
- A gateway publishes all work into one shared queue
- Workers pull from the pool and form full batches before touching GPU
- Models load on first request, LRU eviction when memory runs short
- Batching is by compute cost, not item count — minimizes padding waste
API Surface
client = SieClient("http://localhost:8080")
# Encode: text/images → vectors
results = client.encode(passages)
# Score: rerank query against documents
scores = client.score(query, passages)
# Extract: pull structured fields from documents
fields = client.extract(document, fields=["invoice_number", "date"])
# Generate: LLM completion
answer = client.generate(prompt, context=top_ranked)Integrations
Chroma, Qdrant, Weaviate, LanceDB
Repository
Evidence Timeline
- 2026-07-28: Avi Chawla published article explaining SIE architecture and implementation (sources/articles/avichawla-superlinked-sie-2026.md)