A Short-Video Ad System Is Now Rewriting the Rules That Pick Which Ad You See

A short-video ad system is now rewriting the rules it uses to pick which ad you see, and the people who used to hand-write those rules now only edit the playbook at the meta level. A research paper posted to arXiv on Wednesday by researchers at Kuaishou Technology and Shanghai Jiao Tong University makes the mechanism public.

The paper describes what its authors say is a deployment of DMRL, a new reinforcement learning framework, on a large-scale short-video advertising platform. The framework does not just pick which ad to show. It rewrites the instructions the picker follows, on the fly, and judges whether the rewrite worked. That is the structural change the abstract flattens into "AI improves ads."

The architecture is what the authors call bilevel. At the upper level, an agent performs controlled edits on a structured "skill document," a piece of text that describes, in natural language plus parameters, how to run an A/B test on ad targeting. At the lower level sits a separate, frozen task agent that reads the document and applies it. The two are not the same model. The task agent does not learn. Only the skill document changes.

The analogy is a coach and a player. The picker is the player. It never gets retrained. The skill document is the playbook the player reads on every impression. The upper-level agent is the coach, editing the playbook between rounds based on how the last round went.

Online ads produce short-term signals within about a day, things like clicks, watch time, and conversions, but the outcomes advertisers actually care about, such as a user's long-term value or retention, take around seven days to surface. Rewriting the playbook on one-day signals produces noisy, short-termist rules. Rewriting it on seven-day signals is too slow to act on.

To bridge that gap, the authors built a second component called the Long-term Reward Predictor, or LRP. LRP is a separate model that takes the one-day signals and estimates what the seven-day outcome will be. Internally it splits the population into two encoders, one for general patterns across all users, and one for this specific user group, then pulls related past experiments out of a memory bank. The two views are combined with cross-attention, a mechanism that lets the predictor weigh which past cases are most relevant to the current one.

The other component, Dual-Relative Policy Optimization, or DRPO, is the algorithm that decides which edit to make next. It extends GRPO, a popular reinforcement learning method for language models, with a dual-relative advantage estimator. The "dual-relative" part means the algorithm compares each candidate edit against both a treatment baseline and a control baseline, which makes the policy robust to outliers in the reward signal. It also adds a regularization term that penalizes edits which move the playbook too far from where it started, on the theory that large rewrites are riskier than small ones.

The training is staged. First LRP is trained until its seven-day predictions are accurate enough to use. Then DRPO is trained against LRP's predictions, so the upper-level agent gets fast, dense feedback even though the real reward takes a week to arrive. The two stages are designed to stabilize each other.

The paper claims the deployed system "outperforms state-of-the-art baselines across key advertising metrics," but the specific quantitative lifts live in the experiments section, which the abstract does not summarize. The paper is also a preprint, not peer-reviewed. The platform itself is identified only by the authors' institutional affiliation, and there is no independent production telemetry to compare against. The numbers the abstract gestures at should be read as the authors' own claims, not as a third-party benchmark.

The wider question is governance. The skill document is editable natural language, so a human can in principle read it, audit it, and intervene. But the edits are now produced at machine speed, on machine-defined reward signals, by an agent that has no concept of what an ad is for beyond the metric it is optimizing. The humans who used to write the targeting rules still choose the metric, still approve the deployment, and can still stop the system. They are still in the loop. The loop is just shorter.

That is the shift the paper makes concrete. "AI picks the ad" is the version of this story that has been true for years. "AI rewrites how the ad is picked" is the new version, and it is running in production somewhere today.