Designing Effective Multi-Turn Reward Functions for Amazon Nova Forge

In multi-turn reinforcement learning (RL), the reward function you craft determines what the model learns. Amazon Nova Forge enables customization of these reward functions, which can be rule-based or call upon other language models for judgment. This post focuses on designing composite multi-turn rewards that can be learned through Group Relative Policy Optimization (GRPO). It discusses executing model-generated code safely within the reward, and how to instrument components to ensure effective learning.

Nova Forge supports multi-turn RL via its Bring Your Own Orchestration (BYOO) capability, allowing users to run reward logic in their environment, such as on Amazon ECS, or through a serverless option. The reward function receives full trajectories, including multiple turns, and scores them based on outcome, behavior, and penalties, which are combined to guide the model.

A practical example involves training Amazon Nova Lite 2.0 to ask clarifying questions before coding. The reward is a weighted sum of components like correctness, asking behavior, and penalties for guessing or repetitive turns. These components are implemented as functions that evaluate each rollout, with safety measures in place for executing model-generated code.

Designing effective rewards requires careful consideration to prevent collapse scenarios, such as reward hacking or loss of variance within groups. The article highlights pitfalls like reward components that become dead channels—returning constant values—and how to instrument metrics to detect and fix these issues.

Overall, the article emphasizes the importance of detailed instrumentation, component-wise analysis, and cautious reward design to ensure successful multi-turn reinforcement learning with Amazon Nova Forge.