Concentrated Liquidity Provision: a Reinforcement Learning Perspective
Georgios Chionas, Charalampos Kleitsikas, Stefanos Leonardos, Leandro Sánchez-Betancourt, Carmine Ventre
q-fin.TR, cs.AI, cs.LG, q-fin.CP, q-fin.MF
2026-08-20
Uniswap V3 LP is cast as impulse control with gas and solved by PPO. Low vol and gas: narrow PPO averages 49.91 PnL. High vol and gas: risk-averse agents mostly stay out.
Uniswap V3 lets a liquidity provider pin capital to a price range. A tight range earns a larger fee share while the price stays inside it, then earns nothing once the price leaves, and every rebalance costs gas. A wide range is calmer and thinner. Empirically, LPs lose money on average. Closed-form policies either ignore gas or run into an intractable HJB equation once the state is realistic.
The missing object is a policy that jointly decides whether to act now and where to place the new interval, in a form that can be read: mispricing, inventory, gas, and volatility should each push the policy in a named direction.
The external midprice is geometric Brownian motion. Pool price moves when takers trade. Arrival intensity mixes noise traders with arbitrage pressure: when the pool diverges from the external price, flow toward the correction thickens. The LP chooses intervention times and a tick interval at each intervention, paying a fixed gas cost. The payoff is fees minus impermanent loss minus gas. Risk aversion adds a running inventory penalty.
Two lemmas matter. Under one-tick jumps and a small LP, both fees and IL are linear in the LP's own depth, while gas is a depth-independent constant. In the risk-neutral case that encourages concentrating into the most profitable narrow bin, and it explains why the narrow agent later earns more fees and rebalances more often.
The impulse-control problem is not solvable in this state space, so PPO is used. One agent picks both bounds. A narrow agent freezes width at two ticks and only learns location. Observations include mispricing S−Z, distance to bounds, time, gas, cumulative fees, and token inventories. Actions sit in ±15 ticks of the current tick. Seventy-two regimes (volatility × gas × risk aversion), ten training seeds, 1,000 evaluation trajectories. Baselines: deploy-once wide and narrow, arrival-triggered rebalance, and the Cartea-Drissi-Monga (CDM) closed form, implemented as a gas-free optimistic version that rebalances every 100 steps.
Low volatility, low gas (σ=0.01, g=2), risk-neutral:
| Policy | Mean PnL | 5% CVaR |
| Deploy-once narrow | 15.85 | -14.52 |
| ArrivalRebalance | 49.72 | 6.86 |
| CDM | 35.18 | 11.55 |
| PPO (flexible width) | 42.31±0.97 | 6.08 |
| PPOnarrow | 49.91±0.38 | 9.15 |
Narrow PPO ties tuned arrival rebalancing on mean PnL and wins on CVaR. In the medium regime (σ=0.02, g=4) CDM leads risk-neutral mean PnL at 15.01 versus 5 to 6.5 for the PPOs. Switching on ϕ=50 risk aversion gives up about 4 USDC of mean PnL versus the risk-neutral twins and improves CVaR by about 27 USDC. In the hostile regime (σ=0.03, g=6) risk-averse PPO mean PnL is about 0: the learned action is mostly not to deploy.
Policy probes are readable. High gas widens the inaction region even under large mispricing. The center shifts toward expected arb flow. Width tracks ETH inventory as a risk knob. Superposing heterogeneous risk profiles qualitatively reproduces the bell-shaped liquidity seen around pool prices on chain.
For market-making research the product is an interpretable state-dependent policy, not another backtest print. Tight ranges farm fees, wide ranges cut rebalances, and extreme parameters trigger exit. Those motions match the linearity lemmas. Given that live LPs often lose money, an agent that refuses to deploy under bad (σ, g) is closer to the real constraint than one more width optimizer.
This is not a mainnet bot. The world is a stationary model, and train and test share the same dynamics.
The authors are explicit: policies are scored in the same stationary regime they were trained in, with no regime detection. CDM is the gas-free optimistic implementation, which is kind to the analytical baseline. Continuous-time impulse control is discretized to a decision every 100 steps. The bell-shaped liquidity picture is a qualitative overlay, not an on-chain identification. The LP is small and does not change future taker flow. Multi-agent competition is out of scope.