Agentic RL with GRPO and synthetic SFT

This project started from Assignment 3 of the CMU AI Agent Course, where I trained an LLM agent with GRPO [1] and REINFORCE [2]. On TextCraft, a crafting game, GRPO took Qwen3.5-4B from 0.163 to 0.625 on medium tasks, but hard tasks stayed at 0.

Looking at the trajectories, the agent solved medium tasks by trial and error, which does not scale to hard tasks. This led to my hypothesis: if I can teach the LLM the solving pattern on medium tasks via synthetic data, it will generalize to hard tasks.

I built the SFT data from medium tasks only and refined it over several rounds of failure analysis. Without training on any hard task, SFT and then GRPO solves 19.3% of hard tasks within 75 steps and 51.1% within 150 steps, compared with 0 and 9.7% for GRPO alone.

RL Setup#

The RL training framework is SkyRL, an orchestration layer for RL training that connects the sampler (vLLM) and the trainer (Megatron). SkyRL supports the Tinker API, so the whole training loop is a few Tinker calls in Python.

TextCraft environment#

TextCraft is a variant of the crafting game from Recursive Agent Optimization (RAO) [3], similar to crafting in Minecraft. The agent starts with raw materials and must craft a target, such as 3x c7_i4, within 75 steps. Each step is one tool call:

The agent does not see the recipes. It looks them up one layer of the crafting tree at a time, works out how many of each item it needs, and crafts from the bottom up. The reward is 1 if it crafts every target and calls finish(), and 0 otherwise.

The depth of a task is the number of crafting layers between the raw materials and the target. Depth sets the difficulty: each extra layer adds more items to look up, more counts to work out, and more crafts. So the tasks are split by depth: medium tasks have depth 4 to 6, and hard tasks have depth 7 to 9. Only 44 of the 136 hard tasks fit in 75 steps (40 at depth 7, 4 at depth 8), so I evaluate on those.

GRPO baseline#

Both RL runs in this post use the same config:

• Qwen3.5-4B with LoRA [4] (rank 32), lr 1e-5
• Dr. GRPO [5]: advantage = reward − group mean (no std), loss averaged over all action tokens in the batch, PPO clip [6] (0.8, 1.2)
• 16 groups of 8 rollouts per step, async with staleness up to 4
• 512 tokens per turn, 75 steps per episode

I trained the base model with GRPO for 50 steps on medium tasks. Medium success goes up, but hard stays at 0.

Looking into the trajectories, I found three things:

• It learned the format. Turns without a valid tool call drop from 43 to 8.5 per episode.
• It did not learn to plan. It does not compute item counts beforehand, but relies on the craft error message (“Wrong amount of c5_ore. Need 14, provided 21”). Failed crafts rise from 3.6 to 9.8 per episode.
• It stays weak on deeper tasks. Success on depth 4 / 5 / 6 goes from 0.25 / 0.13 / 0 to 0.93 / 0.67 / 0.20.

The agent relies heavily on error feedback to know how many items it needs. This trial and error fits in 75 steps on a medium task (median 17 crafts), but not on a hard one: on the 44 hard tasks, a perfect run already needs a median of 65 steps, so trial and error without planning wastes steps the agent does not have.

Synthetic SFT data#

Round 1: perfect trajectories#

To teach the solving pattern, I wrote SFT trajectories on medium tasks only, about 33% each at depth 4, 5, and 6. An oracle that knows the recipes writes each trajectory; round 1 has 50 of them (LoRA rank 32, lr 1e-4, 3 epochs).

Every trajectory follows one fixed procedure: look up recipes layer by layer, total each item’s count top-down, then craft each item once, bottom-up. A reply is capped at 512 tokens, so I limit the plan to 8 lines per turn and let a longer plan continue in the next turn.

The first turns of one trajectory are below. The model’s replies and the environment’s replies alternate. The loss is only on the model’s replies, shown in blue. The input and the environment’s replies, shown in grey, are context.

input[system prompt and tool docs]
Target: craft 1x c2_i4.
Initial inventory: {"c9_ore": 11, "c1_ore": 3, "c5_ore": 3, ...}modelGoal: 1x c2_i4. Plan: look up recipes layer by layer, total each item top-down (at most 8 lines per turn), then craft each item once bottom-up.
<tool>get_info(["c2_i4"])</tool>envc2_i4: {"ingredients": {"c4_i2": 2, "c1_i2_11": 1}, "result_count": 1}modelc2_i4: 1 -> c4_i2+2 c1_i2_11+1
<tool>get_info(["c1_i2_11", "c4_i2"])</tool>envc1_i2_11: {"ingredients": {"c5_ore": 2, "c2_i1": 1}, "result_count": 3}
c4_i2: {"ingredients": {"c2_i1": 2, "c1_ore": 1}, "result_count": 3}modelc1_i2_11: 1, /3 -> 1 crafts -> c2_i1+1
c4_i2: 2, /3 -> 1 crafts -> c2_i1+2
<tool>get_info(["c2_i1"])</tool>envc2_i1: {"ingredients": {"c9_ore": 2}, "result_count": 2}modelPlan done. Crafting bottom-up.
Layer 1: c2_i1.
c2_i1: 1+2=3, /2 -> 2 crafts.
<tool>craft({"c9_ore": 4}, ("c2_i1", 4))</tool>envSuccessfully crafted 4 c2_i1(s)[... 3 more crafts, layer by layer, then finish]

The plan can be written in many formats. I picked the format with the lowest perplexity under the base model, the one the model finds easiest to predict.

After 3 epochs, success on 60 medium tasks was 0.533, and on the 44 hard tasks still 0.

Round 2: shorter turns and recovery data#

Round 1 improved medium tasks, but it still solved no hard task. The problem already shows inside medium: success falls from 0.88 at depth 4 to 0.09 at depth 6. Round 2 makes two changes.

1. Shorter, simpler turns. On hard tasks the model often wrote 9 to 15 plan lines in one turn, and the reply was cut off at 512 tokens. Round 2 caps the plan at 5 lines per turn. Three smaller rules let the model copy the text just before it instead of searching back through a long list:

• Lookup lists in order of appearance. The next get_info list copies the new ingredients in the order they appear in the last environment reply. Round 1 sorted them, so on hard tasks the model had to collect up to about 20 names from its notes and sort them.
• Running totals. Each new share is added to the item’s total right away (m5_i4_15: 2 -> m6_i3+4=6 m3_i2_13+4), so the item’s own line only reads the last number.
• Items left in each layer. Each craft turn starts with the items of the layer not yet crafted (Layer 1 left: o1_i1_21, o5_i1.) and crafts the first one.

2. Recovery from mistakes. Round 1 saw only perfect trajectories, so after its own first mistake the model is in a state it never saw in training (exposure bias [7]; see also my post on post-training losses). It does not look for the missing item and replays the whole script.

Inspired by DAgger [8], which trains on the states the model itself reaches with the expert’s correction, I added 30 trajectories with one planted mistake each, of the kinds round 1 actually made (an item left out of a lookup, a wrong count, an item left out of a layer), followed by the oracle’s fix: read the error, craft what is missing, retry. Unlike DAgger, the mistakes are planted rather than collected from the model’s own rollouts.

modelPlan done. Crafting bottom-up.
Layer 1: m6_i1, m3_i1_13, m4_i1.
m6_i1: 6, /2 -> 3 crafts.
<tool>craft({"m8_ore": 3}, ("m6_i1", 6))</tool>[... the rest of layers 1 and 2]modelLayer 2 left: m0_i2_20.
m0_i2_20: 1 crafts.
<tool>craft({"m3_ore": 1, "m0_i1": 1}, ("m0_i2_20", 3))</tool>envError: All 1 recipe(s) failed for m0_i2_20. Errors:
Recipe 1: Insufficient ingredients in inventory: m0_i1: need 1, have 0modelCraft failed: m0_i1 need 1, have 0. m0_i1 was skipped.
m0_i1: 1, /2 -> 1 crafts.
<tool>craft({"m0_ore": 2}, ("m0_i1", 2))</tool>envSuccessfully crafted 2 m0_i1(s)modelRetry m0_i2_20.
<tool>craft({"m3_ore": 1, "m0_i1": 1}, ("m0_i2_20", 3))</tool>envSuccessfully crafted 3 m0_i2_20(s)[... layers 3 to 5, then finish]

Round 2 has 150 medium trajectories (120 perfect, 30 with recovery). Success went from 0.533 to 0.775 on medium and from 0 to 0.142 on hard (pass@4 0.25). Both changes show in the transcripts: only 6 of 176 hard episodes have a cut-off reply, and 17 of the 25 hard successes recover from at least one failed craft.

GRPO from the SFT model#

I ran GRPO from the round 2 SFT model with the same config and one change: it trains only on depth-6 medium tasks, for 30 steps. The SFT model already does well on depth 4 and 5 (0.78 and 0.86), so I excluded them from the RL data.

Results#

Every number uses 4 rollouts per task at temperature 1.0. Hard results are on the 44 hard tasks, except the numbers by depth, which cover all 136.

Only the models with SFT solve any hard task. On the medium tasks that both RL models solve, SFT then GRPO needs about half the steps (21.4 vs 41.1). At 75 steps the hard gain of GRPO over SFT is small (11 tasks better, 6 worse), but it grows with the step budget:

Training cost. I count the tokens each run trained on, the tokens with a loss. All runs train the same model, so tokens track compute. The SFT data comes from an oracle program and costs no model tokens. SFT then GRPO uses about a third of the tokens of GRPO alone (13.5M vs 37.6M) and reaches more than 5 times its hard success within 150 steps:

Success by depth. With 150 steps, on all 136 hard tasks:

What GRPO changed. GRPO did not make the model make fewer mistakes: on the 44 hard tasks, 165 of 176 episodes still hit a failed craft (SFT: 166). It made the model get out of them: 80 of these episodes still succeed (SFT: 55), and it repeats failed crafts much less (a median of 19 per episode, SFT: 47). Recovering costs steps, which is why its gain grows with the budget.

Why it does not reach deeper trees. Each step of the procedure is as easy on a big tree as on a small one, but a deeper tree needs many more steps: a perfect run takes a median of 70 steps at depth 7, 103 at depth 8 and 137 at depth 9. More steps mean more mistakes, and the recovery data only taught the model to fix one mistake at a time; on bigger trees the mistakes pile up and the model gets stuck. Two limits of the format add to this:

• A plan turn holds at most 5 lines, so long plans take extra turns: even a perfect run needs more than 75 steps on 12 of the 44 hard tasks.
• At depth 9, even the oracle’s own reply passes 450 tokens on 32 of the 44 depth-9 tasks, so the 512-token cap cuts the model off again.

Conclusion#

Results. SFT then GRPO solves 19.3% of the 44 hard tasks within 75 steps and 51.1% within 150, compared with 0 and 9.7% for GRPO alone, using about a third of its training tokens.

Findings.

• SFT on perfect trajectories is not enough. After its first mistake, the model is in a state it never saw in training (exposure bias) and cannot recover. As in DAgger, the data has to show the model’s own kind of mistakes followed by the fix.
• Dr. GRPO reinforces a long, wasteful success more than a short, clean one. The reward is 1 for any success, so a 70-step success counts as much as a 20-step one. With the token mean over the batch, a long success even adds more tokens to the update. Trial and error already succeeds on medium tasks, so GRPO kept it and leaned on it more: failed crafts per episode rose from 3.6 to 9.8.
• Better task design blocks shortcuts and pushes RL toward a better policy. In Number Search, a guessing game with “too low” and “too high” hints, the step budget equals the worst case of binary search, so a strategy that does not halve the range can run out of guesses. There GRPO raised the share of midpoint guesses from 43% to 100%. Medium TextCraft tasks leave the shortcut open: error messages give the missing counts and 75 steps leave room to retry. To teach a general method with RL, the task has to make the shortcut fail.

References#

1. Zhihong Shao, Peiyi Wang, Qihao Zhu, et al. DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models. arXiv, 2024.
2. Ronald J. Williams. Simple Statistical Gradient-Following Algorithms for Connectionist Reinforcement Learning. Machine Learning, 1992.
3. Apurva Gandhi, Satyaki Chakraborty, Xiangjun Wang, Aviral Kumar, and Graham Neubig. Recursive Agent Optimization. arXiv, 2026.
4. Edward J. Hu, Yelong Shen, Phillip Wallis, et al. LoRA: Low-Rank Adaptation of Large Language Models. ICLR, 2022.
5. Zichen Liu, Changyu Chen, Wenjun Li, et al. Understanding R1-Zero-Like Training: A Critical Perspective. arXiv, 2025.
6. John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal Policy Optimization Algorithms. arXiv, 2017.
7. Samy Bengio, Oriol Vinyals, Navdeep Jaitly, and Noam Shazeer. Scheduled Sampling for Sequence Prediction with Recurrent Neural Networks. NeurIPS, 2015.
8. Stéphane Ross, Geoffrey J. Gordon, and J. Andrew Bagnell. A Reduction of Imitation Learning and Structured Prediction to No-Regret Online Learning. AISTATS, 2011.