NEVA is a home-grown model family — a 130M MoE text language model and NEVA-Q, a hierarchical 1 meta-core → 25 cores → 18 experts mixture built on custom triangular pooled-attention kernels. No foundation-model weights. No cloud. This is its architecture, at the wiring level.
Every token flows through a meta-core hub that mixes 25 parallel cores. Inside each core, a router fires only 6 of 18 micro-experts (top-k=6) — so capacity is large while compute per token stays small. Attention is pooled & sub-quadratic; cores exchange information through a ring + hub mix. The diagram is live: watch which experts fire.
A mean-of-cores hub with weight-tied embed/head (input core = output core). Generalizes the triangular RING_GLOBAL flag to N cores + hub.
All cores active in v1 (dense core-mix). Sparsity lives in the experts. Cores talk through a self · left · right · HUB ring gather.
Per core, a Switch-style router (capacity-batched bmm dispatch, load-balance aux loss ~0.01) fires 6 experts per token. ~2× capacity at the same active compute.
Everything trains on a single RTX 4060 Laptop (8 GB). VRAM is dominated by the optimizer, so NEVA-Q uses a paged 8-bit AdamW (2 bytes/param) instead of fp32 (8 bytes/param) — that is what lets a 900M model fit.
| Model | Total | Active / tok | D · cores · exp | Throughput (4060) | Role |
|---|---|---|---|---|---|
| NEVA 130M | 130M | ~130M | dense MoE · 32k vocab | served · CPU/GPU | Chat + NOVO DSL generation (sft1→19) |
| NEVA-Q 271M | 271M | 133M | 1200 · 25 · 18 (k6) | ~16k tok/s | Reference hierarchy |
| NEVA-Q 450M | 444M | 202M | 1200 · 25 · 18 (k6) | ~8–10k tok/s | Main run (OPT8, ~5B tokens) |
| NEVA-Q 900M | 865M | 295M | 1200 · 25 · 24 (k6) | ~2.7k tok/s | Capacity preset (bigger GPU) |
| NOVO-Q / byte | ~19.5M | ~8M | 400 · 25 · 18 · vocab 256 | ~70k tok/s | Speaks the NOVO geometry DSL |
tri_attn_t pooled causal attention (2.26× at 25 cores) + neva_q_moe grouped-GEMM (MegaBlocks-style, err 1e-6, 1.85×). Triton on CUDA.
MICRO=2 · ACCUM=12 → 4.65 GB. PagedAdamW8bit (bitsandbytes) drops a 900M's optimizer state from 10.8 GB → 3.67 GB.
Loss reaches within 5% of final at ~33% of the budget, so the LR schedule anneals at the plateau — days, not weeks.
NEVA is not only a chat model. The same core reasoning feeds a spatial language, a security immune-system, and a moving body — all trained in-house.
A constructive spatial language (prisms, CSG, revolution/loft/sweep, materials, layers). NEVA emits it as text; a Three.js engine + a headless Blender bridge render it. The model literally writes buildings.
A non-linear conlang where one sentence is one circular glyph. NEVA learns the compact notation; a compiler turns it into NOVO geometry — a language you read as a diagram.
The same NEVA-Q design, byte-level, as attacker and defender in a sandbox. A 50M "reflect" defender mirrors and duplicates incoming attacks — self-play security, no real host touched.
A procedural molecular-mesh body with 2-bone IK and a cannon-es ragdoll. An evolutionary trainer searches a gait — the mind learns to move a 23k-node mesh.
Pre-train on its own Spanish corpus, specialize by supervised fine-tuning, then distill from teacher models into fresh data — a loop the mind runs to keep getting sharper.
FineWeb-2 spa_Latn → deduped int16 chunks (~10.4B tokens), disk-guarded.
Triangular kernels, warm-started embeddings, early-stop at the loss plateau.
sft1→19: identity, safety, code/math, NOVO DSL, materials, chairs, circular language.
A teacher invents instruction→answer pairs → distill_data.jsonl → next SFT.
Newest checkpoint auto-loads on the local server; the app talks to it live.
The distillation teacher can be a local model or, through a local OpenAI/Anthropic-compatible proxy, a rotation of strong free-tier cloud models — turning NEVA's distill_data.jsonl generator into a much stronger self-teacher without changing the training code.
A fair 2×2 ablation — attention {full, pooled} × FFN {dense, MoE(18,6)} — same batches, same D/depth/LR. Perplexity (lower is better). The big jump is pooled attention; MoE adds ~9% on top of it.
early dynamics · ~1M tokens · T=512 · same seed — directional, not a leaderboard claim.