Tutorial: Running Prism Bonsai 27B Locally
FastHotEmu · reddit · 2026-07-15
This post shares a guide on running Prism Bonsai 27B, focusing on downloading the model locally, compiling a specific version of llama.cpp, and the commands to start llama-server.
Key points include:
- Requires downloading the -Q20.gguf version; alternatively, you can pull it directly using huggingface-cli.
- The author provides compilation commands for macOS using GGMLMETAL=ON; for other platforms, GGMLCUDA=ON can be used instead.
- At launch, you can load directly from Hugging Face using -hf, or point to a local file manually using -m.
- The article also mentions recommended default inference parameters, such as --temp 0.7, --top-p 0.95, and --top-k 20.
Overall, this is a hands-on local deployment tutorial rather than an evaluation of the model's capabilities.
More from Infra
- PoLar: Dynamically Skipping or Looping LLM Layers for Efficient Inference — ttkciar · 2026-07-22
- SK Hynix CEO: Next Year Will Be the Worst Year in Industry's History from Supply Perspective — Beth_Kindig · 2026-07-22
- Tabul AI launches Metal TreeSHAP to speed up Shapley values on Apple silicon — Scobleizer · 2026-07-22
- Tech Giants Are Hiding $1.6T in AI Debt Using Enron's Trick — arto · 2026-07-22
- DeepSeek-V4-Flash tops out at 770 tok/s on one B300 in a vLLM batch test — Moreh · 2026-07-22
- NVIDIA starts shipping 102.4 Tbps Spectrum-6 switches for Vera Rubin AI factories — nvidia · 2026-07-22