Inside SageMaker training session APIs: custom RL loops without GPU ops

AWS ML Blog · rss · 2026-10-23

An AWS ML Blog deep dive explains the programming model behind SageMaker AI training session APIs.

Core design:

The abstraction boundary moves down a level: instead of abstracting the algorithm, they abstract the compute — forwardbackward(), optimstep(), and sample() become composable API calls for any custom SFT, preference optimization, or RL procedure
Sessions are stateful: the model stays loaded on GPUs across steps, no cluster creation or upfront config, driven interactively from your local Python process
Three SDK clients: ServiceClient (role/KMS/VPC, session start) → TrainingClient (training and checkpointing) → SamplingClient (sample, computelogprobs); an active session also serves an OpenAI-compatible chat completions endpoint
RL loop division of labor: sampling, gradients, and checkpoints run on managed GPUs; loop control and reward logic run on your machine
Per-token billing means no GPU cost while your reward function runs locally — significant savings for heavy local computation like complex reward models or multi-tool agentic rollouts
Ships with a cookbook including a minimal GSM8K RL loop and a real workload doing RL on competitive programming problems graded by unit tests

More from coding & agent