Anderson Gonçalves

HTN Method Selection Guided by Multi-Objective Reinforcement Learning and Dynamic Preferences

MSc research · Game AI · UFRGS

Instituto de Informática, Universidade Federal do Rio Grande do Sul

HTN defines what is valid.

MORL learns what is preferable.

Dynamic preferences determine what matters now.

I want an agent that stays inside the rules it was given and still follows priorities that change while it runs. A hierarchical task network handles the first part: the agent only ever does what the domain allows, so it is valid by construction. The cost is rigidity. When safety starts to matter more than speed, someone has to rewrite the domain. Letting a learner loose on the primitive actions removes the rigidity and throws the guarantee away with it. My MSc moves the learning to the choice of method. NPCs are the test bench: there the agent works with partial information and a deadline per frame, and a player notices as soon as it does something odd.

Filtering: the domain says what is legal

At each compound task, the HTN exposes only the methods whose preconditions hold in the current state. Everything semantically invalid under the constraints written into the domain is gone before learning is consulted at all.

Learning: preference-conditioned MORL picks among the survivors

Each reward is a vector with one entry per objective, so objectives that disagree stay separate and never collapse into one number. Conditioning the policy on a preference vector gives a single agent that covers the trade-off space: for every linear preference, an optimal policy lies in the convex coverage set.

Preferences: what matters, right now

The weight vector carries the current trade-off between mission, safety, resources and time. Changing it changes which method gets selected while the HTN domain stays as written, so adapting to a new priority costs no re-authoring.

With two objectives, progress and survival, and estimated returns of [0.8, 0.2] for one method and [0.5, 0.5] for another, the weights [0.8, 0.2] pick the first. Move them to [0.2, 0.8] and the agent picks the second. The numbers are made up for the example.

Credit where the decision was made

Only the selected method is decomposed, and primitive tasks keep executing conventionally. Credit goes to the hierarchical decision that picked the method. That keeps the learning signal legible, and it lets me explain why the agent did what it did.

How it will be evaluated

Compare

  • First-applicable HTN
  • Fixed heuristic HTN
  • Scalar RL for task–method values
  • Contextual scalar RL + HTN mask
  • HTN + preference-conditioned MORL

Stress

  • Preferences switched at set decisions
  • Held-out preference vectors
  • Held-out initial states

Measure

  • Adaptation after a switch
  • Generalization
  • Semantic validity
  • Decision latency

Materials

The ERAMIA-RS paper is in the proceedings and the PDF is here. The poster was presented at ECOA 2026, in October. Both are a first version, with the formulation and the evaluation protocol. The experiments come next, and the full paper with them in 2027.

HTN Method Selection Guided by Multi-Objective Reinforcement Learning and Dynamic Preferences
Poster presented at ECOA 2026, the IV Escola de Computação Avançada at UFRGS, on 21 and 22 October: architecture, hierarchical credit assignment and the planned experiments.
  • Poster (PDF) ECOA 2026 · UFRGS · 1.1 MB
  • ERAMIA-RS paper In the proceedings of ERAMIA-RS 2026 PDF
  • Prototype Grid-world prototype with HTN execution, sensing, navigation and replanning. Vector rewards, conflicting objectives and dynamic preferences are next. The repository is private while the work is in progress.