Tokens Are Just Integer IDs: The Comma Is Row 28 of the Embedding Matrix
zsakib_ · x · 2026-09-30
A short educational thread explaining that LLM tokens are simply integer IDs indexing into the embedding matrix — the IDs could have been anything, even emojis — and only the corresponding rows get passed into the model. Example: the comma maps to row 28 of the embedding matrix.
More from Research
- Microsoft's ML System Forecasts Grid Risks from Space Weather for 66,935 US Substations — Microsoft Research · 2026-10-01
- Google Figures Out How to Watermark AI-Designed Proteins for Biosecurity — Ars Technica AI · 2026-09-30
- Amazon Bedrock's Capacity-Pooling Paper Hits SOSP'26 — tianyin_xu · 2026-09-30
- Method: 25 essays run through strict-correction vs rewriting prompts — paulnovosad · 2026-09-30
- Simons Institute working group reaches consensus on how TCS academia should adapt to AI — tallinzen · 2026-09-30
- A 'wisdom gradient' for AI alignment: eliciting diverse human values bottom-up — edelwax · 2026-09-30