Cottus Runtime
An inference engine for Llama models, in C++ and CUDA, with Python bindings.
Written from scratch for low latency and strict control of memory. It runs its own transformer pipeline, KV cache and attention kernels, and its output matches Hugging Face Transformers token for token.
Apache-2.0 · C++, CUDA, Python · on PyPI as cottus
What's inside
No borrowed inference stack. Every part of the forward pass is the engine's own, and checked against the reference implementation.
- Core
- A transformer implementation in modern C++.
- Memory
- PagedAttention with a block allocator, so the KV cache grows in fixed blocks instead of one large buffer.
- Compute
- CUDA kernels for attention and RoPE; matrix multiplies through cuBLAS.
- Parity
- Generated tokens match Hugging Face Transformers exactly, verified.
- Interface
- A Python API through PyBind11.
Supported models
Each architecture listed as verified matches Hugging Face Transformers exactly.
| Architecture | Models | Precision | Status |
|---|---|---|---|
| Llama | Llama 2 (7B, 13B, 70B), Llama 3 (8B, 70B) | FP16, FP32 | Verified |
| TinyLlama | TinyLlama-1.1B | FP16, FP32 | Verified |
| Mistral | Mistral 7B | FP16 | In progress |
Install
You'll want an NVIDIA GPU with CUDA 11 or 12. There's a CPU fallback, but it's slow.
- Compiler
- GCC 10+ or Clang 12+
- Build
- CMake 3.18+
- Python
- 3.8 or newer
$ pip install cottus
$ uv add cottus
from cottus import Engine, EngineConfig
from cottus.model import load_hf_model
weights, _, _, tokenizer, _ = load_hf_model(
"TinyLlama/TinyLlama-1.1B-Chat-v1.0", device="cuda"
)
config = EngineConfig()
config.model_type = "llama"
config.hidden_dim = 2048
config.num_layers = 22
config.num_heads = 32
config.num_kv_heads = 4
config.head_dim = 64
config.intermediate_dim = 5632
config.device = "cuda"
engine = Engine(config, weights)
output_ids = engine.generate(tokenizer.encode("Hello!"), max_new_tokens=20)
More in examples/: basic inference, the CPU path, a real chat with TinyLlama, and long context.
Planned
Pull requests are welcome for any of these.
- Many GPUs. Scale across GPUs and machines for larger models.
- More models. Native Mistral, Falcon and other families beyond Llama.
- A fast CPU path. Vectorized kernels and OpenMP for CPU-only inference.
- Quantization. Post-training quantization and INT8 kernels.
- FlashAttention-style kernels. Memory-efficient attention for lower latency.
- Plugins. Custom ops and other KV-cache strategies from the community.
- Tooling. Command-line tools for model conversion, benchmarks and profiling.