PROMO

PROMO

Preference-Conditioned Multi-Objective Reinforcement Learning for Quadrupedal Robots

What should the robot prioritise — not how should it move?

Amr Mousa1, Rifny Rachman1, Neil Karavis2, Michele Caprio1,3, Richard Allmendinger1

1 University of Manchester, UK  |  2 BAE Systems, UK  |  3 University of Warwick, UK

Submitted to IEEE Transactions on Robotics HART Workshop @ TAROS 2026
Left: traditional RL trains a separate policy per trade-off. Right: PROMO exposes tracking, stability and efficiency as deployment-time preference knobs feeding one policy, which produces tracking-, stability- and efficiency-focused behaviour on a Unitree Go2.
Train once; set tracking, stability and efficiency priorities at deployment. The gait emerges from the policy. Percentages show the best observed reductions relative to balanced across command regimes.

Overview

Quadrupedal locomotion balances command tracking, stability and energy efficiency. PROMO makes that balance an explicit runtime input: an operator changes priorities while the same policy keeps controlling the robot.

Conventional

Learned control

  • Train → fixed trade-off. One scalar reward decides tracking vs stability vs efficiency, once.
  • New mission priority? Retrain · redesign reward · switch controller.
  • Discrete behaviours, and a deployment cost that grows with every new trade-off.
PROMO

Priorities as an input

  • Train → runtime trade-off. A preference vector w over tracking, stability and efficiency is a policy input.
  • New mission priority? Move the preference.
  • A continuum of behaviours from one deployed actor.

Preference control

Watch priorities change mid-run, compare four recorded preferences, then examine how the same policy produces different gaits.

Live switch

Priorities change while the robot keeps walking. The overlay shows commands and preferences; the policy stays fixed.

Jump to a preference change using the chapters above.

Compare

Choose an anchor to watch a recorded hardware trial. The measurements show both the benefit and the cost of each choice, relative to balanced priorities.

Drag the dot or choose an anchor to change priorities.
Tracking Stability Efficiency
Tracking0.33
Stability0.33
Efficiency0.33
PositionMAE ↓
0.534
m — baseline
Peakattitude ↓
0.196
rad — baseline
Specificenergy ↓
124.6
J/m — baseline

Balanced priorities: a compromise between command following, body regulation and effort.

Matched-command hardware trials, n = 5 per anchor; lower is better for all three metrics. Dragging between anchors explores the weights; recordings and measurements are available only at the four anchors.

Emerging different gaits

No gait is prescribed. The policy discovers the motion that serves each priority: larger steps for tracking, active body regulation for stability, and compact steps for efficiency.

Tracking-heavy
w = (0.80, 0.10, 0.10) — larger foot excursions.
Stability-heavy
w = (0.10, 0.80, 0.10) — intermediate gait with increased clearance in this sequence.
Efficiency-heavy
w = (0.10, 0.10, 0.80) — compact, low-clearance trajectories.

Same deployed policy; only the preference changes. Two-second loops.

Try itDrive the policy in your browser

The trained policy runs locally in your browser. Set the preference, drive with the keyboard, kick the robot, add payload or change friction and watch the gait, torque and power respond.

↗ Open full screen

About 5 MB download, then everything runs on your device. The Go1 policy is executed in MuJoCo rather than the training simulator, so behaviour can differ from the paper.

Method

Three semantic objectives describe what the operator wants to prioritise. Fixed locomotion priors guide how the robot moves; training teaches one actor to respond to the preference.

Objective designSemantic objectives, fixed priors

Tracking Stability Efficiency w Fixed priors — never in w air time · contact balance · joint posture

How to move, versus what should matter. Gait-conditioned controllers expose physical descriptors — gait timing, stance geometry, foot clearance — and so say how to move. A semantic preference says what to prioritise, and leaves the gait to the policy.

But locomotion rewards also carry posture, contact-shaping and smoothness terms. Making each one a MORL objective would turn reward-engineering decisions into deployment controls and inflate the Pareto space.

PROMO factorises instead: three semantic objectives form the operator interface, while embodiment-level terms stay fixed priors that never enter w.

ArchitectureTraining and deployment

PROMO extends TAR with a preference-conditioned actor and one critic value head per objective. Training balances tracking, stability and efficiency while keeping locomotion priors fixed; an action-diversity regulariser encourages the actor to respond to the preference.

PROMO architecture: preference-conditioned actor and multi-head critic, with the deployment path shaded.
Only observation history, the velocity estimator and the actor run on the robot. Change the preference, without retraining or switching policies.

Evaluation

Simulation tests the preference space densely. Hardware trials test four anchors across three command regimes, measuring improvements alongside their trade-offs.

SimulationA continuous range of choices

Across 100 tested preferences, increasing an objective’s weight generally improves its realised return. Tracking and efficiency conflict most strongly: better command following requires more actuation. The policy learns a range of compromises, beyond the four hardware anchors.

Protocol: 100 sampled preference vectors, each evaluated in 1024 parallel environments for 20 seconds under domain randomisation and external disturbances.

0.843Preference–objective correlationMean Spearman ρ across the three objectives.
67/100Non-dominated samplesMeasured in the full three-objective space.
Tracking versus efficiency objective returns: improving tracking sacrifices efficiency; squares mark non-dominated points in this two-dimensional projection.
The trade-off: tracking and efficiency show the strongest conflict. Squares identify the non-dominated points in this 2D projection, distinct from the 67/100 count in 3D.
Achieved tracking return generally rises as the tracking preference weight increases.
The response: achieved tracking generally rises with its preference weight (ρ = 0.775). Variation reflects the other two weights.
Stability and efficiency: trade-offs and preference responses

Stability

Tracking versus stability Pareto front approximation.
Trade-off. Stability vs tracking — mildly conflicting.
Realised stability return rising with the stability preference weight.
Response. Realised stability rises with the stability weight, ρS = 0.829.

Efficiency

Stability versus efficiency Pareto front approximation.
Trade-off. Efficiency vs stability — largely complementary.
Realised efficiency return rising with the efficiency preference weight.
Response. Realised efficiency rises with the efficiency weight, ρE = 0.927.

HardwareThe priorities transfer to the real robot

One simulation-trained policy, deployed zero-shot on a Unitree Go2. Each objective-focused preference improves its corresponding metric across matched-command, fast and slow regimes. There are five trials per preference in each regime.

Average improvements across the three regimes

−29.5%Position errorTracking preference
−29.0%Peak attitude deviationStability preference
−19.9%Specific energyEfficiency preference

Mean percentage reductions across matched-command, fast and slow regimes, relative to balanced; five trials per condition.

Better tracking costs more energy and actuation; efficiency reduces effort but slows response. Stability benefits depend on the motion regime. The explorer above shows these trade-offs for matched commands; the table below reports the same matched-command measurements, including the costs.

Matched-command trials: means over five runs per anchor. Lower is better for all three metrics; percentages are changes relative to balanced.
PreferencePosition MAE (m) ↓Peak attitude (rad) ↓Specific energy (J/m) ↓
Balanced0.534baseline0.196baseline124.6baseline
Tracking0.405−24.2%0.281+43.4%169.6+36.1%
Stability0.538+0.7%0.158−19.4%132.4+6.3%
Efficiency0.624+16.9%0.187−4.6%102.3−17.9%

Best observed reductions relative to balanced were 38.7% in position error (slow regime), 59.0% in peak body-attitude deviation (fast), and 30.4% in specific energy (fast). These are individual-regime results, rather than the averages above.

Hardware comparison across all three command regimes
Position MAE, peak attitude deviation and specific energy for balanced, efficiency, stability and tracking preferences across matched-command (Replay), fast and slow regimes.
Means over five trials per condition; error bars show one standard deviation. Lower is better in every panel. Replay is the matched-command regime used in the explorer and table.

Robustness

On a deliberately slippery surface, the same policy follows commands, expresses different priorities and recovers from disturbances.

The robot was intentionally placed on a hard plastic sheet coated with soap and water to simulate a surface with very low friction. Across all four preference anchors, it followed commands through acceleration and deceleration, responded to external disturbances and recovered without falling. The same policy was used throughout.

Command following on the soap-and-water-coated plastic sheet, with the balanced preference.

Changing preferences

Different priorities remain usable even with poor traction. Watch command following and recovery under each preference; the policy and slippery surface stay the same.

Tracking-heavy
w = (0.80, 0.10, 0.10) — prioritising command following. 2:15
Stability-heavy
w = (0.10, 0.80, 0.10) — prioritising body attitude. 2:58
Efficiency-heavy
w = (0.10, 0.10, 0.80) — compact, low steps. 2:41

The overlay shows commands and preference weights in each ten-second clip.

External disturbances

External disturbances applied to the robot, shown from impact through recovery, using the balanced preference.

Driving rear kick, full spin1:34 A rear kick induces a large yaw rotation; the robot re-plants its feet and recovers.
Head-on kick1:58 A head-on kick drives the body back and down; the robot regains support and resumes locomotion.
Additional recoveries · nine clips on low friction

Additional pushes, kicks and turning manoeuvres.

Play together to compare recoveries. Timestamps refer to the full session.
Rear kick, yaw spin0:37
Hind-quarter kick0:46
Flank push, half turn0:59
Head shove1:13
Rear-leg kick1:19
Turning on slippery sheet1:42
Side kick, stagger1:49
Rear-hip kick2:20
Side kick, splay3:18
Full session · 3:25, no falls

The unedited session: four preferences, acceleration, deceleration and repeated disturbances on the slippery sheet, with no falls.

Full annotated session, 3:25. Press play to watch.

Limitations

The evidence has clear boundaries, and each one limits how far the claims extend:

  • Preference coverage. On hardware, four anchor preferences with five trials each in three command regimes. Dense, continuous preference control is shown in simulation only; between the anchors, hardware behaviour is interpolated rather than measured.
  • Scalarisation. Linear weighting reaches only the supported (convex) part of the Pareto front. A trade-off that lies in a concave region cannot be selected by any preference weight.
  • Disturbances. Push forces were not instrumented. The recovery clips show that the policy recovers, not how much it can withstand, and they do not compare preferences under equal pushes.
  • Scope. One robot and three objectives fixed at training time. Adding an objective, or changing what an existing one rewards, means training a new policy.

Citation

@article{mousa2026promo,
  title   = {PROMO: Preference-Conditioned Multi-Objective Reinforcement Learning
             for Quadrupedal Robots},
  author  = {Mousa, Amr and Rachman, Rifny and Karavis, Neil and
             Caprio, Michele and Allmendinger, Richard},
  journal = {Submitted to IEEE Transactions on Robotics},
  year    = {2026},
  url     = {https://amrmousa.com/promo/}
}

Acknowledgements

The authors gratefully acknowledge the RAICO1 team for providing access to the robotic platform and for their support during hardware deployment. We thank Matthew Hale for his assistance with the experimental testing, Amy Johnson and Darren Cunningham for their operational support, and Professor Wei Pan for valuable technical discussions and insightful feedback that contributed to the development of this work.

This work builds on IsaacLab, RSL-RL and our earlier TARLoco framework.

One robot. Different mission priorities.

Code and the full video release are being prepared. In the meantime, see our earlier work on robust quadrupedal locomotion.

🐾 TARLoco project page