Amateur researcher fights Stage 4 colon cancer with ML, hits $500-per-run GPU wall
whafa · reddit · 2026-10-20
An unpaid ML researcher using AI coding tools to find drug-repurposing candidates for his wife's Stage 4 colon cancer details his cost dilemma: reproducing NBFNet on biomedical graphs (103K nodes, 2.5M edges) on Vertex A100 80GB, the model's 6-layer full-graph message passing stores 1GB of activations per query, forcing batch size down from 512 to 64 (half a terabyte would be needed for 512). His 20 epochs see only 640K queries — a quarter of one pass; a full pass takes 76 hours / $412, and matching the paper's schedule costs $500 per run at $5.40/hr on demand. He asks about sufficient training budgets, when buying a GPU beats on-demand, whether bf16 autocast or activation checkpointing with custom sparse kernels could restore batch 128+, and fixes for popularity bias in link prediction. His fully AI-written lab notebook is public on GitHub Pages.
More from Infra
- Semicon West & OCP recap: $300B+ wafer-equipment TAM, Tesla TeraFab orders, TSMC N5-to-N3 conversions — BenBajarin · 2026-10-20
- Qualcomm and Liquid AI push on-device AI with Liquid Context, compressing multimodal inputs on Snapdragon — JosephJacks_ · 2026-10-20
- PyTorchCon enterprise day features executive sessions with Red Hat, NVIDIA and IBM — PyTorch · 2026-10-20
- SOAP & SPlus scale pretraining across 96+ DiLoCo workers via update whitening — aaron_defazio · 2026-10-20
- Amazon CEO says AWS is out of compute until beyond 2028 — rohanpaul_ai · 2026-10-20
- Nebius launches live spot GPU price comparison across regions, with API access — kevinsxu · 2026-10-20