Sebastian Raschka's reasoning series ep.5: log-prob scoring and self-refinement
rasbt · x · 2026-09-26
Sebastian Raschka released round 5 of his "Reasoning from scratch" series, covering log-probability scoring and LLM self-refinement.
Key points
- Log-prob scoring: token probabilities and sequence likelihood as the foundation for losses like cross-entropy in pre-training and distillation, with hands-on PyTorch computation, shifted targets, and numerical stability.
- Answer scoring: from a rule-based scorer to ranking candidate answers via average log probabilities.
- Self-refinement loop: generating critiques, revising answers, and implementing the full loop.
- Evaluation: results on MATH-500 plus takeaways.
The video includes a full code walkthrough with detailed timestamps.
More from Models
- User ditches Grok 4.7 for 4.6: endless thinking, stalls on long context — lxfater · 2026-09-26
- Codex outage reports: workspace routing discovery timeout error — koltregaskes · 2026-09-26
- ChatGPT RLHF Co-Author's Startup Jev in Funding Talks at $10B Valuation — aakashgupta · 2026-09-26
- Decision Index 0.2.1: SGD benchmark pulled over bug, scoring fixes applied — multimodalart · 2026-09-26
- Aider creator slams lab safeguards for blocking legitimate reverse-engineering work — zeeg · 2026-09-26
- Why AI slop longposts get likes: LLMs optimized for human preference — AymericRoucher · 2026-09-26