WARD GEYS↖ Back to portfolio
PROJECT / ATLAS ENGINELOCAL INFERENCE · SYSTEMS WORK

ATLAS ENGINE

Inside the model.
Down to the kernel.

An experimental inference runtime for local Qwen and Gemma models, with recorded 35B and 80B MoE checkpoints. GGUF loading, compressed-weight execution, memory management, custom CUDA kernels and the loop that turns model state into tokens.

ATLAS ENGINE / MEMORY PATHARCHITECTURE DIAGRAM
01 / MODEL FILEQwen3.5 · 35B A3BGGUF · Q4_K_M · published model weights
Validated loading
02 / SYSTEM MEMORYCompressed expert banksExpert weights stay in RAM
Bounded transfers + reuse
03 / GPU EXECUTIONCUDA + expert cacheModel state, attention and token generation
Simplified 35B execution path on the test PC: 64 GB system RAM and a 12 GB GPU. The complete model is not resident in GPU memory.
Runtime
Python · PyTorch · CUDA · GGUF
Test hardware
12 GB GPU · 64 GB system RAM
Released checkpoints
Engine 0.26.0 and 0.32.0

TWO PROJECTS / CONNECTED RESPONSIBILITIES

A companion to use.
An engine to build.

ATLAS COMPANION

The conversation and the interface.

Windows and Android clients connect to the shared Atlas application. Conversation, voice and approval controls make the assistant usable. The shared Agent owns tool policy and action receipts.

Explore Companion

ATLAS ENGINE

The model execution layer.

The native path executes the model through PyTorch and Atlas kernels instead of delegating that execution to llama.cpp. It produces text or proposed tool calls. The separately managed llama.cpp runtime remains available as an independent fallback.

RUNTIME ARCHITECTURE

The constraints
shape the implementation.

A 35B mixture-of-experts model on this PC is a memory and data-movement problem as well as a compute problem. The runtime has to decide what stays resident, what moves and what can safely be reused.

01 / FILE FORMAT

Read the weights.

GGUF metadata, tensor layouts and compressed matrices feed model-specific execution paths. The native backend runs Qwen and Gemma with PyTorch and Atlas CUDA kernels. Model weights and tokenizer tooling come from their respective publishers.

02 / MEMORY

Make residency explicit.

The selected 35B model file is about 21.2 GB. Compressed expert banks stay in system RAM while a bounded GPU cache reuses selected experts. Direct transfers reduce staging work. The tradeoff is substantial RAM use and transfer cost, not a model that fits entirely in 12 GB of VRAM.

03 / CUDA

Work on the token path.

Custom compressed-weight operations, attention state and CUDA graph replay are part of the runtime. The released MoE change widens compressed expert reads while keeping their accumulation order, then sizes expert caching at load time under the existing memory reserve.

04 / REUSE & RECOVERY

Keep state accountable.

Source-bound prefix caching reuses stable system and tool prompt state. Invalid cache entries return to ordinary preparation. Model state, cancellation and unload have explicit paths. Generated tool calls still go through the shared Agent for validation, permissions, approval and execution receipts.

80B MODEL CHECKPOINT / 27 OCTOBER 2026

Qwen3-Next 80B MoE

80B total parameters, with about 3B active per token. The recorded native checkpoint uses Q3_K_M weights and includes real Companion text, calculator and approved/denied file checks. This is a model-execution checkpoint, not a broad reasoning-quality claim.

MEMORY, NOT MAGIC

12 GB GPU · 64 GB RAM

The complete 80B model is not resident in GPU memory. Bounded expert-file reads, a lazy host cache capped at 8 GiB with a 16 GiB RAM reserve, and a separate bounded GPU cache manage the working set. Loading and data transfers still cost time.

RECORDED RESULTS / 26 OCTOBER 2026

Measure the change.
Keep the conditions.

QWEN3.5 35B A3B / NATIVE GENERATION

Earlier native path → optimized native path

Native control17.37 tokens/s
Optimized native execution26.39 tokens/s
51.9% higher median generation rate in this bounded comparison. All measured continuation tokens matched. Sampling is included; loading and prompt processing are excluded. Background activity was present.
Benchmark method and comparison boundary

Two controls and two ordinary candidate sessions, with three measured repeats each. Each repeat generated 32 greedy tokens after a 27,804-token stable prefix. Warmups were excluded. The same-source control and candidate pair used the same prefix hash. This compares native execution policies, not Atlas Engine against llama.cpp.

A direct comparison with a properly tuned llama.cpp configuration remains pending. Token generation, prompt preparation, startup and complete tool requests need separate measurements.

GEMMA / PROMPT PREPARATION

17.225 s → 10.409 s

An alternating 8,192-token trained-model comparison measured less preparation time, with zero compared logit difference and identical checked greedy choices. This is component preparation, not application launch time.

THE REMAINING COST

Startup still varies.

The recorded 35B candidate took about 29 seconds with a prepared cache and about 77 seconds after fresh source preparation. Faster generation does not remove weight loading, prompt preparation or host overhead.

APPLICATION EVIDENCE

Beyond a faster
kernel in isolation.

The dated native 0.26 campaign exercised real Companion sessions: generated calculator calls, explicitly approved file creation with exact readback, declined writes with independently checked file absence, cancellation and recovery. Qwen 9B, Gemma and the managed fallback had separate preservation checks.

Engine 0.26.0 shipped inside Companion 0.19.2. These are recorded checks of that released checkpoint, not certification of every later source change or arbitrary workload. Broader quality, long-session reliability and physical voice acceptance remain open.

View the containing release
Earlier native checkpoints, 23 and 24 October

Recorded on 23 and 24 October 2026: two native Qwen app-core runs and two native Gemma app-core runs completed a calculation and an exactly approved file write/read. Independent file checks matched the requested contents, and the answers included those contents. A separate Gemma refusal check confirmed that a denied write created no file.

These were local development checks on native engine 0.14.0 and 0.15.0, using the real Qt host and production gateway with the window hidden and microphone disabled. They do not certify newer source changes, visible UI, physical voice or broad task reliability. Gemma tool latency remains substantial. Native engine versions are separate from desktop and Android release versions.

ENGINE 0.32.0 / 28 OCTOBER 2026

Clear file requests.
Less repeated work.

For selected explicit file requests with the native Qwen3-Next 80B model, Atlas still generates and validates the tool arguments. When the result is complete and matches the request, it can display the observed tool receipt directly instead of generating a second answer.

MATCHED FILE TASKS / SECONDS

Three tasks, four matched sessions.

Each cell contains two recorded attempts.
TaskControl (s)Candidate (s)
Directory listing33.54737.87515.45415.796
File search25.42229.73410.54610.657
Exact file read28.87529.00011.57811.094
Twelve matched attempts passed in control, candidate, candidate, control order. These particular calls took roughly 25 to 38 seconds before, and 10 to 16 seconds after. This is task latency, not token-generation speed or a general performance guarantee.

PREPARED MODEL STATE

Reuse compatible preparation.

A versioned state ABI lets compatible prepared prefixes survive a release-label change. Model contents, exact prompt tokens, state-producing source and runtime/device settings still determine compatibility. Changed execution or prompt content requires fresh preparation.

ACTION BOUNDARIES

Keep the normal route.

Permissions and approval checks remain in the shared Agent. Untrusted-data markers stay visible. Paged, capped, truncated or ambiguous results, workflows and other models keep their normal continuation rather than using this shortcut.

Test conditions and release boundary

The same three requests ran in four actual Companion sessions with a 32K context, no-thinking mode, isolated stores and the normal resource guards. Operating-system file caching and background CPU activity were not controlled. Final packaged-payload checks separately completed listing, search and reading in 15.968, 10.438 and 10.859 seconds.

Recorded preservation checks also covered approved and denied file actions, cancellation and recovery, other native models and managed fallback. Some other-model checks used the first packaged candidate; the final 80B-only receipt helper had separate app, file and Composer checks. This is dated, scoped evidence, not certification of every later source change or the complete product.

First uncached preparation is still expensive, and recorded startup times vary. This release does not establish a universal startup gain, sustained generation improvement or superiority over llama.cpp.

Recorded in Companion 0.19.8, containing native Engine 0.32.0. The older 35B generation comparison above remains a separate checkpoint.

View the 0.19.8 release

RESEARCH / SPEECH EXECUTION

EXPERIMENTAL / LOCAL PROTOTYPE

From speech tokens
to a waveform.

Alongside the language runtime, I am developing Atlas-owned speech execution around pretrained Chatterbox Turbo weights. Native code now handles speech-token generation, acoustic flow, reference encoders and waveform decoding. The model weights and numerical dependencies remain credited upstream work.

RECORDED COMPONENT AND WORKER CHECKS

Native output, checked against a reference.

The dated CPU checkpoint passed six neural-component gates and a controlled acoustic waveform comparison. Three generated clips matched their expected transcripts in offline speech recognition. Later isolated worker checks exercised cancellation and recovery; separate CUDA checks compared logits and acoustic output.

NOT YET PRODUCT ACCEPTANCE

A working prototype, with open gates.

These checks do not establish a physically tested conversation. Real microphone and speaker behavior, coexistence with the active brain, watermarking, distributable packaging and current complete-source acceptance remain open. This voice work is separate from the released language-engine checkpoint above.

Not real-time yet.

In one matched worker comparison, generating 1.64 seconds of audio took 18.813 seconds on CPU and 22.531 seconds with CUDA. The measured complete CUDA pipeline was slower. A faster isolated token-stage probe does not establish faster speech synthesis.

Qualification, dependencies and remaining limits

The 28 October CPU acoustic comparison reached 99.83 dB signal-to-error before watermarking, with upstream neural inference imports blocked. NumPy and SciPy provide numerical primitives. Later frontend work uses a local oneMKL FFT primitive; packaging that dependency elsewhere is unverified. This is native execution of pretrained weights, not a newly trained foundation speech model.

A retained strict log-feature diagnostic initially failed. Later frontend and reference checks passed on their selected inputs without treating the original failure as a pass. Source has changed since the sealed CPU checkpoint, and the later worker/CUDA prototype has no combined-source acceptance seal in this evidence.

The worker recordings were pre-watermark qualification output without physical playback. Whole-sentence synthesis and fixture cancellation do not establish low-latency streaming, audible interruption, naturalness or reliable everyday conversation. No prototype audio is embedded here as evidence of a released voice.

MY WORK / WITH AI CODING ASSISTANTS

Design decisions.
Implementation. Evidence.

I work with AI coding assistants on runtime design, implementation choices, profiling and verification. AI tools generate substantial parts of the code. My work includes defining the constraints, comparing approaches and checking the results. The published models and supporting libraries remain credited dependencies.

THE OTHER SIDE / ATLAS COMPANION

Back to the
companion.

Explore Companion