Regional decomposition and coordination
Dividing a coupled environment into regions makes each agent's task smaller. It also creates a coordination problem: the value of one agent's action depends on what the others do. My work investigates the architecture of this decision process, from the activation of regional policies to the representation of their actions.
A learned coordinator selects which regions may act at each step; regional policies then choose the actions. In a three-region experiment, I compare a categorical coordinator with five activation patterns against a factorized coordinator that can represent all eight combinations. Both are trained jointly with the local policies.
- Flat MARL
- Every regional policy chooses its own action at each step.
- Categorical coordination
- Choose none, one of the three regions, or all regions together.
- Factorized coordination
- Make a binary activation decision for each region, allowing every combination.
Problem formulation
Cooperative sequential decision-making
Regional actions affect a shared environment. The learning objective is expected discounted return under the resulting joint policy.
Expected shared return
π: policy · r: shared reward · γ: discount
T: episode horizon
Single-agent reference
One policy chooses from a global catalogue.
The benchmark catalogue contains unitary interventions and no-operation.
Flat multi-agent policy
Regional choices are composed into one environment action.
n agents · aᵢ: regional action · oᵢ: regional observation
The product expresses conditional independence in the flat policy parameterization. It does not imply independent state transitions or rewards. The single-agent catalogue and composed regional actions need not span the same action set; the thesis treats catalogue design separately from policy coordination.
A one-step activation hierarchy
The coordinator μ selects an activation pattern z from the combined observations o. Active policies select regional actions; inactive regions contribute no-operation. Both levels are trained jointly with PPO and a shared reward.
The pattern is selected again at every step. This is not a temporally extended option, and an active regional policy can still choose no-operation.
Comparing activation policies
A coordinator with five possible activation patterns performs best in this comparison. Its policies reach 89% of the planned episode length on average, versus 85% when all regional policies are active. Time spent above operating limits also falls, from 1.46% to 0.08% of observed operation.
The factorized coordinator permits all eight activation combinations but reaches 81.64% mean duration. Its larger decision space does not improve performance within this training budget. These results compare the trained policies; they do not establish that restricted coordination is generally preferable.
Coordination study
Evaluation setup and what this comparison isolates
The Bus14 experiments use three regions, matched reward and training settings, and joint PPO training. A selected regional policy can still choose no-operation. Activation therefore does not imply a physical intervention, and the hierarchy makes one-step decisions rather than temporally extended options.
These are single-seed, development-validation results. Fifty scenarios characterize the evaluated policies; they do not measure variation across independent training runs. Late-training activation statistics also do not establish that the coordinator learned a specialized state-dependent activation strategy.
The comparison is specific to ungated coordination. The thesis studies fixed intervention gates separately, where a gated flat policy completes all 50 scenarios and gated categorical coordination completes 48. Hierarchy is one part of the system design, alongside reward and intervention rules.
| Policy / encoder | Duration (%) | Completed | Overloaded time (%) | Mean excess (× 10⁻³) |
|---|---|---|---|---|
| Flat MARL | 85.33 | 38/50 | 1.46 | 0.378 |
| Categorical | 89.04 | 43/50 | 0.08 | 0.011 |
| Factorized | 81.64 | 35/50 | 0.38 | 0.098 |
Reading the metrics. Duration is the mean fraction of the scenario horizon reached; completion counts scenarios that reach the full horizon. Overloaded time is the mean per-scenario fraction of observed operation with at least one line at or above its thermal limit. Mean excess averages the positive part of maximum line loading minus one, first over observed operation within each episode, then across scenarios. Early termination changes the duration over which overload is measured.
Observation–action matching
Even after regional decomposition, one policy can face up to 65,706 candidate actions. I developed and evaluated a Two-Lane policy that represents the current observation and the action catalogue separately, then scores their compatibility.
The observation lane produces a query. The action lane uses learned action embeddings and Set Transformer attention to build candidate keys. A query–key match assigns each action a score. This lets candidates exchange information through a small set of learned summaries, without attention between every pair of actions.
Architectural background: Lee et al. · Set Transformer · ICML 2019
qᵢ is the observation query; kᵢ,ₐ is the key for candidate a; d is their common dimension. Softmax over the candidate scores defines the regional policy. The action keys derive from learned catalogue identities, not explicit physical action features.
Action representation study
Two-Lane operates longer in 32 of 42 matched scenarios. On average, it reaches 41% of the planned episode length, versus 30% for feedforward. The task remains difficult: only 3 scenarios are completed, versus 1 for the baseline, and peak loading is higher.
The benefit is conditional. In a separate reduced-catalogue experiment at 200 million interactions, feedforward finishes ahead: 76.70% versus 73.40% mean duration. The two experiments also differ in training configuration, so catalogue size alone cannot explain the change in ranking.
Architecture, protocol and evidence boundaries
Each regional Two-Lane policy has an observation encoder and an action encoder with two induced set-attention blocks. The action embeddings represent learned catalogue identities; they do not encode explicit physical action descriptions. The surrounding categorical coordinator, intervention gate and centralized critic are retained within the comparison.
The large-catalogue result uses saved 80M checkpoints, deterministic evaluation, frozen saved observation normalization, and 42 identical scenarios and horizons. Historical training runs differ in online-validation scheduling and rollout timeouts, limiting a strict architecture-only causal interpretation. The paired plot is recomputed from retained per-scenario evaluation records.
The reduced-catalogue endpoint comes from the retained thesis checkpoint audit. It uses a different configuration and budget from the large-catalogue study and is contextual evidence, not a matched cross-catalogue ablation. Loading peaks in the older large-catalogue recorder use four-decimal precision; mean excess uses unrounded aggregates.
Action identities and regional policies remain tied to a fixed catalogue. Generalization to unseen actions or networks has not been demonstrated.
| Policy / encoder | Duration (%) | Completed | Overloaded time (%) | Mean excess (× 10⁻³) |
|---|---|---|---|---|
| Feedforward | 30.31 | 1/42 | 5.15 | 5.187 |
| Two-Lane | 40.88 | 3/42 | 3.75 | 4.119 |
Reading the metrics. Duration is the mean fraction of the scenario horizon reached; completion counts scenarios that reach the full horizon. Overloaded time is the mean per-scenario fraction of observed operation with at least one line at or above its thermal limit. Mean excess averages the positive part of maximum line loading minus one, first over observed operation within each episode, then across scenarios. Early termination changes the duration over which overload is measured.
Scope and next questions
These studies examine where to place structure in a learning system: in the coordinator's decision space, in the representation of actions, and in the observations available to each agent. The practical contribution is an implemented set of architectures and an evaluation of the conditions under which they help.
My next interest is to make these representations less dependent on a fixed partition and catalogue. The companion graph-encoding study addresses the information side of this problem: how local policies can use relationships across a shared environment.
