VNT: Von-Neumann-inspired Transformer (Research Prototype)

VNT is a research prototype that explores a Transformer variant with explicit control-inspired components:

• data flow (token states)
• instruction flow (operator routing)
• control flow (dynamic step decisions)

This repository focuses on reproducible experiments, transparent baselines, and fast single-machine iteration.

Current Status

• Baseline and VNT training pipelines are runnable.
• Stable and experimental VNT implementations are separated.
• Auto-logging for PPL, approximate FLOPs, and average micro-steps is included.
• A from-scratch pretraining scaffold is included (pretrain.py).
• Current VNT experiments do not yet show a consistent Pareto win over the baseline.

Repository Layout

• train.py: baseline/VNT training (stable / experimental impl switch)
• eval.py: evaluation entrypoint
• compare_runs.py: stable vs experimental log comparison
• pareto_report.py: same-quality lower-compute report
• pretrain.py: from-scratch decoder-only pretraining
• evaluate_pretrain.py: checkpoint evaluator for pretraining
• tokenizer_build.py: local vocab builder
• pretrain_config.yaml: from-scratch config
• pretrain_config_round1.yaml: Round-1 ablation (RoPE + QK-Norm)
• baseline_transformer.py: matched baseline model
• vnt_transformer.py: experimental VNT implementation
• vnt_transformer_stable.py: stable VNT implementation
• VNT-Architecture-Guide.md: architecture notes
• VNT-Architecture-Diagram-EN.svg: architecture diagram

Data

By default, configs use local data:

• ./input.txt

No HuggingFace download is required when data.text_file is set.

Install

pip install -r requirements-vnt.txt

VNT/Baseline Training

Baseline:

python train.py --config config.yaml --model baseline --impl stable

VNT stable:

python train.py --config config_vnt_stable.yaml --model vnt --impl stable

VNT experimental:

python train.py --config config_vnt_experimental.yaml --model vnt --impl experimental

From-Scratch Pretraining (Single Machine)

1. Build vocab

python tokenizer_build.py --input ./input.txt --out ./output/pretrain_vocab.json --max-vocab 8000 --min-freq 1 --lowercase

2. Train with CORE early-stop

python pretrain.py --config pretrain_config.yaml --core-threshold 0.256525

3. Evaluate checkpoint

python evaluate_pretrain.py --ckpt ./output/pretrain_checkpoints/pretrain_core_target.pt --text ./input.txt --seq-len 256 --iters 50 --batch-size 4

CORE definition:

• CORE = 1 / (1 + val_bpb)
• training stops early when CORE >= threshold.

Long-Context Sweeps

• Standard sweep:
  • python context_sweep.py --lengths 128,256,512,1024 --steps 1200
• Stabilized sweep templates:
  • python context_sweep_stable.py --lengths 128,256,512

Notes

• microgpt.py and v2.py are educational scripts inspired by minimal GPT implementations.
• This repo is a prototype and may change rapidly.