Latest commit
History
Folders and files
Repository files navigation
SPECTRA: Universal High-Throughput 4-Bit LLM Compression & Real-Time GPU Serving Runtime
SPECTRA (Sparse-Precision Exact Channel Tensor Reconstruction Architecture) is an architecture-agnostic neural compression engine and high-throughput execution runtime engineered to break the Memory Wall on consumer and edge GPUs (including 4 GB and 8 GB NVIDIA RTX laptop and desktop GPUs).
Run mathematical reasoning models (such as DeepSeek-R1-Distill ), deep transformers ( IBM Granite-3B , Llama-3.2 ), state-space hybrids ( NVIDIA Nemotron-3-Nano Mamba-2 ), and convolutional architectures ( LiquidAI LFM-3 ) locally on hardware where standard runtimes crash with Out-of-Memory (OOM) errors.
At-A-Glance Performance Highlights (Physical NVIDIA GeForce RTX 3050 Ti Laptop GPU • 4 GB VRAM)
Instant Cold Model Load Time: 1.5s to 3.8s directly from disk into GPU VRAM (0 MB host RAM allocation).
Autonomous Streaming Compression: 2 to 4 minutes for 1.5B to 4B models (Zero calibration dataset, <50 MB host RAM).
Peak Decoding Throughput: Up to 82.5 tokens/sec with in-register fused Triton kernels and 100% GPU residency.
Memory Footprint Reduction: 68.5% VRAM reduction (3.3x compression factor), enabling 4B models to run on 4 GB laptops.
Universal Agnosticism: Compresses and serves any Dense Transformer, MoE, Hybrid Mamba-2 SSM, or ConvNet autonomously.
Zero Representation Collapse: Outperforms uncompressed BF16 baseline on ARC-Challenge reasoning benchmark (35.58% vs 34.73%).
Table of Contents
Why SPECTRA? Breaking the Edge Memory Wall
Industry Head-to-Head Comparison
Quickstart (30-Second Turnkey Launch)
Cloning & Serving Pre-Tuned Models
Compressing Any Custom Hugging Face Model
Empirical Benchmarks (Physical RTX 3050 Ti Hardware)
Interactive Web Dashboard Tour
Python SDK & REST API Reference
Why SPECTRA? Breaking the Edge Memory Wall
Running modern large language models (LLMs) and hybrid state-space models (SSMs) on edge and consumer devices is fundamentally bounded by memory bandwidth rather than arithmetic throughput.
The Arithmetic Intensity Bottleneck
During single-token autoregressive generation ( $M = 1$ ), an inference engine must read the entire weight matrix $W$ from GPU global memory (GDDR6/VRAM) to evaluate a single token vector $x$ : $$\text{Operational Arithmetic Intensity} \approx \frac{2 \cdot N \cdot K \text{ FLOPs}}{2 \cdot N \cdot K \text{ Bytes loaded}} = 1.0\text{ FLOP/Byte}$$
On a consumer GPU such as the NVIDIA GeForce RTX 3050 Ti Laptop GPU:
Peak FP16 Tensor Compute: $\approx 32.0\text{ TFLOP/s}$
Memory Bus Bandwidth: $192.0\text{ GB/s}$ (128-bit bus width)
Theoretical Maximum Decode Speed: At $1.0\text{ FLOP/Byte}$ , memory bandwidth saturates at only $192.0\text{ GFLOP/s}$ , leaving compute units over 95% idle .
To accelerate decoding, the weight bytes transferred across the memory bus must be compressed. However, traditional quantization tools introduce severe friction:
Host RAM OOM Crashes: Existing quantizers (e.g. GPTQ, AWQ, HQQ) attempt to load full uncompressed model weights into host CPU RAM simultaneously, requiring 16 to 32 GB of system RAM for a 7B model. Consumer laptops crash or freeze immediately.
Empirical Calibration Data Overhead: Standard post-training quantization requires downloading external calibration datasets (e.g. Pile, WikiText-2, C4) and spending hours optimizing Hessian matrices.
Hardware Fragility in Constrained VRAM: When a model exceeds physical VRAM, standard runtimes abort with CUDA Out of Memory errors rather than intelligently managing layer execution.
SPECTRA eliminates all three barriers through mathematical guarantees.
Industry Head-to-Head Comparison
Quickstart (30-Second Turnkey Launch)
SPECTRA requires zero manual setup. Run a single command directly after cloning:
Linux & macOS (1-Command Turnkey Launch)
git clone https://github.com/NeuraCryption/SPECTRA.git && cd SPECTRA && ./launch.sh
Windows (Command Prompt / PowerShell - 1-Command Turnkey Launch)
git clone https://github.com/NeuraCryption/SPECTRA.git && cd SPECTRA && launch.bat
Cross-Platform Python (Any OS)
git clone https://github.com/NeuraCryption/SPECTRA.git cd SPECTRA pip install -r requirements.txt python launch.py
Probes your physical GPU compute capability, VRAM headroom, and host RAM via NVML.
Scans your directory tree for all cloned or local model repositories.
Finds an available network port (default: 7860 ).
Launches the high-throughput local server and opens your browser to http://localhost:7860 .
1-Click Operations Showcase
The built-in Web Dashboard provides seamless autonomous operation:
+-----------------------------------------------------------------------------------------+ | [SPECTRA HUD] GPU: RTX 3050 Ti (4GB) | VRAM: 1,050 / 3,771 MB | RAM: 3.2 GB | 54°C | 38W | +-----------------------------------------------------------------------------------------+ | [Models & Compression] | | ├── DeepSeek-R1-Distill-1.5B (SPECTRA 4-bit) ──> [ Load & Chat (3.8s) ] | | ├── Nemotron-3-Nano-4B-SPECTRA (Mamba-2) ──> [ Load & Chat (3.9s) ] | | └── Uncompressed Raw Models Found: | | └── meta-llama/Llama-3.2-3B ──> [ 1-Click Compress (3 min) ] | +-----------------------------------------------------------------------------------------+ | [Active Session: DeepSeek-R1] | | User: Solve 3x + 15 = 42 step-by-step. | | Assistant: | | ┌─ Thinking Process (DeepSeek-R1 <think> drawer) ───────────────────────────────────┐ | | │ 1. Subtract 15 from both sides: 3x = 27. │ | | │ 2. Divide by 3: x = 9. │ | | └───────────────────────────────────────────────────────────────────────────────────┘ | | The solution is x = 9. | | | | [ 78.5 tok/s | Prefill: 262 tok/s | Latency: 12.7 ms/tok | 100% GPU Resident ] | +-----------------------------------------------------------------------------------------+
1-Click Hardware Optimization: Run python launch.py to auto-calibrate buffer sizes to your physical VRAM and RAM.
1-Click Model Loading: Click "Load & Chat" to instantiate the architecture and stream weights into GPU VRAM in 1.5 to 3.8 seconds .
1-Click Autonomous Compression: Drop any raw Hugging Face folder into the workspace and click "Compress Model" to generate a 4-bit .trdc container in minutes with zero calibration data.
1-Click DeepSeek Thinking Mode: Reasoning models (e.g. DeepSeek-R1 <think> tokens) render in a collapsible styled drawer with live step latency and token metrics.
1-Click Batched Benchmark Suite: Evaluate Tensor Core scaling across batch sizes $B=1$ , $B=2$ , and $B=4$ directly in the browser.
Cloning & Serving Pre-Tuned Models
Cloning any pre-compressed SPECTRA model directly into your workspace makes it immediately runnable:
Step 1: Clone Pre-Tuned Repositories from Hugging Face
# Example A: DeepSeek-R1-Distill-1.5B (Reasoning model, 1.03 GB) git clone https://huggingface.co/neuracryption/DeepSeek-R1-Distill-1.5B-SPECTRA # Example B: NVIDIA Nemotron-3-Nano-4B (Mamba-2 Hybrid, 2.26 GB) git clone https://huggingface.co/neuracryption/Nemotron-3-Nano-4B-SPECTRA # Example C: IBM Granite-4.2-3B (Deep 40-layer Transformer, 2.06 GB) git clone https://huggingface.co/neuracryption/Granite-4.2-3B-SPECTRA # Example D: LiquidAI LFM-3-1.7B (Hybrid ConvNet/SSM, 1.00 GB) git clone https://huggingface.co/neuracryption/LFM-3-1.7B-SPECTRA
Step 2: Run Immediate CLI Inference
python run_inference.py --prompt " Explain quantum computing in simple terms: "
Step 3: Or Open the Web Dashboard
python launch.py
The scanner automatically registers all cloned models and populates the Ready-to-Run grid.
Compressing Any Custom Hugging Face Model
SPECTRA is 100% architecturally agnostic . You can compress any model from the Hugging Face Hub or local disk without modifying code:
# Compress directly via CLI: spectra compress \ --model meta-llama/Llama-3.2-3B-Instruct \ --output ./models/llama3_spectra.trdc \ --group-size 32 \ --outliers 0.01
from spectra import Spectra Spectra . compress ( model_id_or_path = "mistralai/Mistral-7B-v0.1" , output_path = "models/mistral7b.trdc" )
Why Compression Never Crashes Your Machine:
Zero-RAM Shard Streaming: Reads safetensors or bin shards tensor-by-tensor from disk. Host RAM stays strictly under 50 MB.
Mathematical Matrix Classification: Any 2D weight matrix with dimension divisible by 32 (Attention Q/K/V/O, MLP gate/up/down, MoE routed experts, Mamba projections) is compressed to 4-bit Group-32 + HDSQ outlier channels.
Lossless Structural Vectors: RMSNorm, LayerNorm, biases, 1D convolution filters, and RoPE tables are preserved losslessly in native FP16/BF16 precision.
Empirical Benchmarks (Physical RTX 3050 Ti Hardware)
All metrics below were gathered on physical hardware (NVIDIA GeForce RTX 3050 Ti Laptop GPU, 4 GB VRAM, 7.2 GB Host RAM) under reproducible test runs. All models achieve 100% GPU Residency (zero CPU offload during generation).
Master Multi-Model Performance Table
(Measurements conducted on Linux x86_64 with CUDA 12.4 and PyTorch 2.5.)
Downstream Reasoning & Accuracy Parity
ARC-Challenge Evaluation (1,172 Questions): DeepSeek-R1-Distill-1.5B under SPECTRA achieves 35.58% accuracy compared to 34.73% for the uncompressed BF16 baseline, proving zero representation collapse on complex multi-step reasoning.
Generation Token Agreement: Evaluated across 3,200 continuous generation tokens, achieving 91.0% first-token greedy match and 100% semantic task completion parity.
VRAM Stability: Verified over 20 continuous generate/unload cycles with background NVML telemetry, maintaining strict 0.0 MB memory leaks.
Microarchitectural Deep-Dive
+-------------------------------------------------------------------------------+ | SPECTRA PIPELINE | +-------------------------------------------------------------------------------+ | 1. ZERO-CALIBRATION AST CURVATURE EXTRACTION | | Extracts affine scale vectors (gamma) from RMSNorm/LayerNorm in <50 ms. | | Mathematical proxy for parameter sensitivity: S_j = |gamma_j| * ||W_:,j|| | +-------------------------------------------------------------------------------+ | v +-------------------------------------------------------------------------------+ | 2. STREAMING ZERO-SPIKE GROUP-32 MICROSCALING | | Quantizes 32 contiguous weights per tile into 4-bit nibbles (uint8 pairs).| | Streams tensor-by-tensor from disk shards with Host RAM <= 50 MB. | +-------------------------------------------------------------------------------+ | v +-------------------------------------------------------------------------------+ | 3. TRDC BINARY CONTAINER SERIALIZATION | | Custom container format (TRDC4B01 magic header + JSON manifest). | | Direct streaming into pinned CUDA buffers without memory copying. | +-------------------------------------------------------------------------------+ | v +-------------------------------------------------------------------------------+ | 4. REGISTER-LEVEL FUSED TRITON HARDWARE KERNELS | | * fused_w4a16_gemv_hdsq: B=1 decode, in-register unpack & dot product. | | * batched_w4a16_gemm: B>=2 prefill/decode, direct Tensor Core tiling. | | Zero intermediate global memory scratchpads. | +-------------------------------------------------------------------------------+ | v +-------------------------------------------------------------------------------+ | 5. AUTONOMOUS RUNTIME SUBSYSTEMS | | * SpectraFusedEmbedding: Gather-and-dequantize dynamic vocabulary. | | * PingPongLayerOffloader: Double-buffered asynchronous PCIe DMA pipeline.| | * SpectraQuantizedCache: Group-32 4-bit KV cache up to 8,192 tokens. | +-------------------------------------------------------------------------------+
1. Calibration-Free Group-32 Microscaling
Rather than calculating global or per-channel scales that can be distorted by heavy-tailed outliers, SPECTRA partitions weights into blocks of 32 contiguous parameters: $$W_{\text{quant}} = \text{clamp}\left(\left\lfloor \frac{W - \min(W)}{\text{Scale}} \right\rceil, 0, 15\right)$$
Each 32-element group maintains an independent FP16 scale and packed INT4 zero-point.
Weights are packed into 4-bit nibbles with Marlin coordinate interleaving (weights $0..15$ in the low nibble, $16..31$ in the high nibble of each 16-byte block), matching GPU vector load alignment.
2. Saliency Mapping via AST Curvature Extraction
In neural architectures with LayerNorm or RMSNorm preceding linear projections: $$y = W \cdot \text{RMSNorm}(x) = W \cdot \left(\frac{x}{\text{RMS}(x)} \odot \gamma\right)$$ The learned affine scale vector $\gamma$ analytically mirrors the parameter Hessian curvature: $$F_{jj} \propto |\gamma_j|^2$$ SPECTRA inspects the model Abstract Syntax Tree (AST), pairs every linear layer with its upstream normalization vector, and computes column saliency: $$S_j = |\gamma_j| \cdot |W_{:,j}|_2$$ This computation takes under 50 milliseconds across an entire 7B model and requires zero calibration data.
3. Fused Triton Hardware Kernels
The execution engine replaces standard PyTorch linear layers with custom Triton kernels:
Single-Token Decode ( fused_w4a16_gemv_hdsq ): Designed for batch size $B = 1$ and sequence length $S = 1$ .
Loads 4-bit packed integer bytes directly into vector registers ( vreg ).
Bit-shifts and masks nibbles in parallel: low = packed & 0x0F and high = (packed >> 4) & 0x0F
Multiplies by microscales and accumulates into FP32 registers.
Requires 0 bytes of dynamic global VRAM scratchpad .
Tensor Core Batched Inference ( batched_w4a16_gemm ): Designed for prefill and batched queries ( $B \ge 2$ ).
Maps unpacked weights into $16 \times 16$ Tensor Core tile matrix multiply units, driving aggregate throughput up to $223.9\text{ tok/s}$ .
4. Client Systems Memory Optimizations
SpectraFusedEmbedding : Vocabulary matrices (often exceeding 131,000 tokens in models like Nemotron) typically consume 600 MB to 1.5 GB of static VRAM. SPECTRA stores vocabulary tables in packed 4-bit Group-32 format, dynamically gathering and dequantizing only the active tokens in the forward pass. This saves over 69% static VRAM.
PingPongLayerOffloader : When a model exceeds physical VRAM, SPECTRA uses double-buffered CUDA streams to overlap layer computation with host-to-device PCIe DMA transfers, executing the remainder of the model with minimal throughput degradation.
SpectraQuantizedCache : Compresses the Key-Value Cache into Group-32 4-bit representations, allowing context length scaling up to 8,192 tokens on 4 GB VRAM without out-of-memory errors.
Interactive Web Dashboard Tour
The Web UI ( web/index.html ) provides complete autonomous control:
Pinned Hardware Telemetry HUD (Top Bar): True Physical VRAM: Queries hardware driver registers via NVML ( libnvidia-ml.so.1 ), matching nvidia-smi exactly.
VRAM Utilization Bar: Visual indicator of active memory allocation and safe headroom.
Host RAM & CPU Load: Monitored via psutil .
GPU Core Temperature & Power: Real-time temperature (°C) and power draw (Watts).
Execution Strategy Badge: Indicates current hardware mode ( FULL_GPU_RESIDENT , HYBRID_DYNAMIC_OFFLOAD , or CPU_RESIDENT ).
Autonomous Model Drawer: Visual catalog of all detected local models.
Exact size comparisons, compression ratios, and VRAM savings percentages.
1-Click Load and Unload actions (unloading cleans PyTorch allocator caches and resets GPU memory to 0 MB).
Interactive Playground & Streaming Chat: Server-Sent Events (SSE) token-by-token streaming.
Reasoning parser: Automatically handles thinking models (e.g. DeepSeek-R1 <think> tags) and presents thoughts in a collapsible styled drawer.
Live performance HUD displaying generation speed ( tok/s ), prompt prefill speed, and step latency ( ms/tok ).
Multi-turn conversation persistence stored locally as JSON session files.
Batched Throughput Benchmark Suite: 1-Click benchmark across batch sizes $B=1$ , $B=2$ , and $B=4$ .
Measures raw hardware execution limits and tensor core scaling.
CLI Reference & Automation
SPECTRA includes a unified command-line interface:
# 1. Launch Web Dashboard spectra ui --port 7860 # 2. Probe Hardware Telemetry spectra status # 3. Compress a Model spectra compress \ --model deepseek-ai/DeepSeek-R1-Distill-Qwen-1.5B \ --output ./models/deepseek_r1_spectra.trdc \ --group-size 32 \ --outliers 0.01 # 4. Run CLI Generation spectra run \ --model ./models/deepseek_r1_spectra.trdc \ --prompt " Solve 3x + 15 = 42 step-by-step. " \ --max-tokens 512
Python SDK & REST API Reference
Enterprise Python SDK Integration
pip install -e .
Then import and use spectra anywhere in your Python environment:
import spectra # 1. Inspect Hardware Capability & VRAM Headroom profile = spectra . probe_hardware () print ( f"Device: { profile . device_name } | VRAM: { profile . total_vram_mb :.0f } MB" ) # 2. 1-Line Model Load (Direct .trdc file or cloned Hugging Face folder) engine = spectra . load ( "models/deepseek_r1_spectra.trdc" ) # 3. Synchronous Generation with Metrics result = engine . generate ( "Solve: What is the derivative of x^3 + 5x?" ) print ( result . text ) print ( f"Speed: { result . decode_tok_per_sec } tok/s | Active VRAM: { result . peak_vram_mb :.1f } MB" ) # 4. Token-by-Token Streaming for chunk in engine . generate_stream ( "Write a Python Fibonacci generator:" ): if chunk . get ( "token" ): print ( chunk [ "token" ], end = "" , flush = True ) # 5. Batched Tensor Core Benchmark batch_results = engine . generate_batch ([ "Prompt 1: What is gravity?" , "Prompt 2: Define quantum mechanics." ], max_new_tokens = 64 ) print ( f"Batch Throughput: { batch_results [ 'aggregate_tok_per_sec' ]:.2f } tok/s" ) # 6. Autonomous Model Compression in Python (<50 MB host RAM) spectra . compress ( model_id_or_path = "meta-llama/Llama-3.2-3B-Instruct" , output_container = "models/llama3_spectra.trdc" )
REST API Endpoints
When the server is running ( python launch.py ), use standard HTTP requests:
Load Model: POST /api/engine/load with {"trdc_path": "path/to/model.trdc"}
Unload Model: POST /api/engine/unload
Stream Generation: POST /api/engine/generate with {"prompt": "...", "max_new_tokens": 128, "temperature": 0.2}
Live Telemetry: GET /api/telemetry
Models Catalog: GET /api/models
1-Click Compress: POST /api/compress with {"model_path": "...", "output_filename": "..."}