LDGI Notes • Hardware Desk

Shop file

Tesla P100 + Bonsai 2 on older Linux boxes

Port notes from shop-max-mus — Maximus V Formula / Z77, Ivy Bridge, Tesla P100, Prism Bonsai-demo.

Port notes from shop-max-mus (Maximus V Formula / Z77, Ivy Bridge i7-3770K, Tesla P100 16 GB compute-only, Intel HD 4000 for display). Goal: run Prism Bonsai-demo / Bonsai 2 27B PQ2 when stock prebuilts fail.

Publish target: ldgi.app (not lgdi.app). Suggested path: /notes/p100-bonsai-port.html.

Hardware / OS assumptions

Piece	shop-max-mus
CPU	Intel i7-3770K (Ivy Bridge) — no AVX2 / BMI2
Board	ASUS Maximus V Formula (Z77)
GPU	NVIDIA Tesla P100 16 GB — CUDA arch sm_60 (Pascal)
Display	Intel HD 4000 iGPU (1 GB BIOS carve-out); P100 compute-only (custom DSDT)
System RAM	8 GB installed (~6.2 GiB usable with iGPU carve-out)
Power limit	P100 capped at 180 W (card allows up to 250 W; min settable ~125 W)
Thermals	Slowdown ~82 °C, shutdown ~85 °C. Under LLM load, ~80 °C with high watts is OK; overheating signal is watts collapsing below ~100 W after temp has hit ~80 (throttle), not 80 °C alone.

Repo / model

git clone https://github.com/PrismML-Eng/Bonsai-demo.git
cd Bonsai-demo
# lean setup (optional): skip Open WebUI / code interpreter
BONSAI_OPENWEBUI=0 BONSAI_CODE_INTERPRETER=0 ./setup.sh

Model used successfully:

models/bonsai2-gguf/27B/Ternary-Bonsai-2-27B-PQ2_0.gguf (~6.8 GiB)
Vision projector: Ternary-Bonsai-2-27B-mmproj-Q8_0.gguf (kept on CPU with BONSAI_MMPROJ_CPU=1)

If HF download stalls, resume with curl against the same GGUF URL into that path (verify final size vs Content-Length).

Why stock prebuilts fail on this class of box

CPU SIGILL — prebuilt llama-server hits AVX2/BMI2 instructions (ggml_cpu_init / shlx). Ivy Bridge dies immediately.
Wrong GPU arches — fat CUDA binaries often start at sm_80+ (or include sm_120a, etc.) and omit sm_60. P100 will not run them.
Missing libcudart.so.12 until CUDA 12 runtime / toolkit (or Ollama’s cuda_v12 libs) are on LD_LIBRARY_PATH.

Custom llama.cpp build (what worked)

Toolchain

Ubuntu nvidia-cuda-toolkit 12.0 (nvcc at /usr/bin/nvcc)
Optional fake CUDA home if scripts expect a CUDA_HOME tree
Clone Prism fork into Bonsai-demo/llama.cpp (branch prism)

CMake / arch flags (essentials)

-DCMAKE_CUDA_ARCHITECTURES=60 (P100)
Disable AVX2 and BMI2 for Ivy Bridge (BMI2-off was required after AVX2-off alone still SIGILL’d on shlx)
Example spirit: CPU flags with -mno-bmi -mno-bmi2, and -DGGML_BMI2=OFF (plus whatever the tree uses to turn off AVX2)

Install layout on disk

bin/cuda-prebuilt/   # original download (keep for reference; do not use on Ivy Bridge/P100)
bin/cuda-p100/       # custom build output
bin/cuda -> cuda-p100

Verify: llama-server -h runs; objdump/disasm should not show BMI2 shlx in the hot paths you cared about.

Runtime libraries

export LD_LIBRARY_PATH="$PWD/bin/cuda-p100:/usr/lib/x86_64-linux-gnu:/usr/local/lib/ollama/cuda_v12${LD_LIBRARY_PATH:+:$LD_LIBRARY_PATH}"

Ollama may be installed only to provide CUDA 12 .so files; Bonsai itself is not the Ollama model server.

Start / stop (recommended)

Single slot (-np 1) — one user, full KV budget (default auto was 4 slots on this box).

KV4 — BONSAI_KV4=1 stores KV as Q4_0 so longer context fits in 16 GB.

mmproj on CPU — BONSAI_MMPROJ_CPU=1 / --no-mmproj-offload.

Proven ladder on P100 16 GB (KV4, -np 1)

BONSAI_CTX	Notes
8192	First bring-up; easy fit
16384	Comfortable
32768	Comfortable
65536	Comfortable; ~8.7 GiB VRAM idle-reserved in practice
131072 (128k)	Fits; live snapshot ~10.2 GiB used / ~6 GiB free on shop-max-mus

Larger than 128k: try carefully; watch load for CUDA OOM. Prefill of huge prompts is slow on P100 (~tens of tok/s prompt eval).

Start (128k example — adjust CTX as needed)

cd ~/Bonsai-demo
export LD_LIBRARY_PATH="$PWD/bin/cuda-p100:/usr/lib/x86_64-linux-gnu:/usr/local/lib/ollama/cuda_v12${LD_LIBRARY_PATH:+:$LD_LIBRARY_PATH}"
BONSAI_CTX=131072 BONSAI_MMPROJ_CPU=1 BONSAI_KV4=1 ./scripts/start_llama_server.sh -np 1

Stop

# prefer PID-based kill so you do not match your own shell command line
pids=$(ps -C llama-server -o pid=)
[ -n "$pids" ] && kill $pids
# or Ctrl+C in the start terminal

Avoid pkill -f llama-server from a command line that itself contains that string (can kill the shell).

Confirm context after load

curl -s http://127.0.0.1:8080/health
curl -s http://127.0.0.1:8080/props | python3 -c 'import json,sys; d=json.load(sys.stdin); print(d.get("default_generation_settings",{}).get("n_ctx"))'

Expect 131072 for 128k.

UI / API

Chat UI: http://127.0.0.1:8080
OpenAI-compatible API: http://127.0.0.1:8080/v1/chat/completions
Conversations live in the browser (local storage); restarting the server does not wipe UI history
Not started at boot by default (manual start). ollama.service may be enabled separately for CUDA libs only

Thinking / reasoning (Bonsai 2 27B)

27B is a thinking model; thinking defaults on.

Web UI: + / Reasoning next to composer → Off sends enable_thinking: false (real speedup for code dumps)
Prompt text like “stop planning” is only a soft nudge
Hiding/collapsing reasoning in the UI does not save tokens or time
Server-wide: extra args --reasoning-budget 0 --reasoning-format none --chat-template-kwargs '{"enable_thinking": false}'

API key for local apps (VS Code, etc.)

Settings → API Key in the web UI is client-only (Bearer header).

To require a key on the server, restart with:

... ./scripts/start_llama_server.sh -np 1 --api-key 'your-secret'
# or: export LLAMA_API_KEY='your-secret'

Apps: base URL http://127.0.0.1:8080/v1, Authorization: Bearer your-secret.

Default bind is localhost only (BONSAI_HOST=0.0.0.0 for LAN — then a key matters more).

VRAM / progress monitoring

watch -n 1 nvidia-smi
# or
nvidia-smi -l 1

Flat Memory-Usage while generating is normal: weights + full KV for -c are reserved at load.

Progress while a job runs:

curl -s http://127.0.0.1:8080/slots

Watch is_processing, n_prompt_tokens, and next_token[].n_decoded (generated tokens). Context truncate looks like log truncated = 1 / stop at n_ctx — that is a token ceiling, not VRAM OOM.

OOM / crash: process dies, CUDA allocate errors in log, VRAM drops to near empty, /health fails.

Live snapshot (shop-max-mus, 2026-09-19)

Binary: bin/cuda-p100 via bin/cuda symlink
Model: Ternary-Bonsai-2-27B-PQ2_0 (PQ2 ~2.13 bpw)
Flags: -c 131072 -np 1 KV4 mmproj CPU
VRAM: ~10223 / 16384 MiB used under load
Decode: historically ~17 tok/s on this card for short tests

Checklist for the next Linux + P100 box

Confirm GPU is sm_60 (nvidia-smi + CUDA samples / arch docs).
Confirm CPU ISA (no AVX2/BMI2 → must custom-build CPU backends).
Install CUDA 12 toolkit/runtime; fix LD_LIBRARY_PATH.
Build Prism llama.cpp for 60 + Ivy-safe CPU flags; install to bin/cuda-p100.
Download Bonsai 2 27B PQ2 + mmproj.
Start with KV4, -np 1, modest BONSAI_CTX, then climb (16k → 32k → 64k → 128k).
Use Reasoning Off for long single-file codegen compares.
Watch thermals: high temp + watt collapse = throttle; fix airflow before long 128k jobs.

Related local files on shop-max-mus

~/Bonsai-demo/RUN-P100.md — short start cheat sheet
~/Bonsai-demo/P100-BONSAI-PORT.md — this document

Source: shop-max-mus ~/Bonsai-demo/P100-BONSAI-PORT.md. Short sheet: RUN-P100.md. Domain: ldgi.app (not the LGDI bureau).