12sweep manifests
7主要横向 eval 组
79v32 checkpoint eval records
5v32 有效候选，v3.2d failed
结论摘要
- 本报告不按直观 impression 排名，而是同时看训练曲线、32-episode eval、checkpoint sweep 的 best/last/stability。
- LegacyGAT 的 cat/gated 确实强于 MAPPO，但 gatobs/纯图瓶颈很差；HATT v2.0 是第一代稳定跃迁。
- HATT v3.0/v3.1 的 best 强而部分 last 弱，结合 TensorBoard 曲线看，不能简单归因于模型退化；环境 full_random 的场景难度波动影响很大。
- HATT v3.2：v3.2c 上限最高，v3.2a 后期最稳，v3.2b 高峰强但后期风险高；v3.2d 训练失败。
分析口径
Best checkpoint 表示能力上限，last checkpoint 表示直接部署风险，stable checkpoint ratio 表示跨 checkpoint 的稳健性。对于 full_random 环境，late eval 变弱可能来自环境抽样、路径/补给惩罚和训练继续更新共同作用，因此报告中保留三种视角。
配置索引
| Manifest | Group | Scale/Eval | Jobs |
|---|---|---|---|
20260623_7groups.yaml | macf-260623 | ?+? agents, tasks=?, steps=?, eval=32 | hattv2.2-260623, hattv2.0-260623, hattv1.0-260623, gatobs-260623, cat-260623, gated-260623, mappo-260623 |
20260624_v2_full_random.yaml | macf-sweep-v2 | ?+? agents, tasks=?, steps=?, eval=32 | mappo-v2, hattv2.0-v2 |
20260625_v4_curriculum_full_random.yaml | macf-sweep-v4 | ?+? agents, tasks=?, steps=?, eval=32 | hattv2.0-v4-curriculum, hattv2.0-v4-fullrandom, mappo-v4-curriculum, mappo-v4-fullrandom |
20260626_mid_260626.yaml | macf-mid-260626 | 10+20 agents, tasks=25, steps=1500, eval=32 | hattv2.0-mid-jitter-260626, hattv2.0-mid-fullrandom-260626, mappo-mid-jitter-260626, mappo-mid-fullrandom-260626 |
20260626_small-scale.yaml | small-scale | 8+12 agents, tasks=10, steps=750, eval=32 | hattv2.0, hattv2.0, mappo, mappo, rmappo-hattv2.0, rmappo |
20260627_hatt-v23-small-scale.yaml | hatt-v23-small-scale | 4+8 agents, tasks=10, steps=750, eval=32 | hattv2.0, hattv2.3a, hattv2.3b, hattv2.2, hattv2.3c, mappo |
20260628_hatt-v30-small-scale.yaml | hatt-v30-small-scale | 4+8 agents, tasks=10, steps=750, eval=32 | hattv2.0, hattv3.0a, hattv3.0b, hattv3.0c |
20260628_hatt-v31-fixed-small-scale.yaml | hatt-v31-fixed-small-scale | 4+8 agents, tasks=10, steps=750, eval=32 | hattv2.0, hattv3.0a, hattv3.1a-fixed, hattv3.1b-fixed, mappo |
20260628_hatt-v31-mid-scale.yaml | hatt-v31-mid-scale | 10+20 agents, tasks=25, steps=1500, eval=32 | hattv2.0, hattv3.0a, hattv3.1a, hattv3.1b, mappo |
20260628_hatt-v31-small-scale.yaml | hatt-v31-small-scale | 4+8 agents, tasks=10, steps=750, eval=32 | hattv2.0, hattv3.0a, hattv3.1a, hattv3.1b, mappo |
20260629_hatt-v32-small-scale-resume.yaml | hatt-v32-small-scale | 4+8 agents, tasks=10, steps=750, eval=32 | hattv2.0, hattv3.2d, mappo |
20260629_hatt-v32-small-scale.yaml | hatt-v32-small-scale | 4+8 agents, tasks=10, steps=750, eval=32 | hattv2.0, hattv3.1a-fixed, hattv3.1b-fixed, hattv3.2a, hattv3.2b, hattv3.2c, hattv3.2d, mappo |
LegacyGAT 与早期 HATT
macf-260623-eval_20260624-073628
| Model | Family | Success | Reward | Completion | Length | Dead |
|---|---|---|---|---|---|---|
| hattv2.0-260623 | HATT v2.0 | 84.4% | 1.81 | 97.8% | 573.3 | 0.56 |
| hattv2.2-260623 | HATT v2.2 | 81.2% | -13.70 | 94.7% | 628.8 | 0.88 |
| cat-260623 | LegacyGAT | 71.9% | -15.21 | 95.9% | 591.9 | 0.62 |
| gated-260623 | LegacyGAT | 68.8% | -26.37 | 92.8% | 610.5 | 0.72 |
| hattv1.0-260623 | HATT v1 | 53.1% | -28.36 | 93.1% | 621.9 | 0.44 |
| mappo-260623 | MAPPO | 40.6% | -49.97 | 91.6% | 679.8 | 0.34 |
| gatobs-260623 | LegacyGAT | 25.0% | -153.79 | 73.8% | 707.6 | 0.97 |
LegacyGAT cat/gated 的成功率明显高于 MAPPO，说明关系信息有价值；gatobs 明显较差，说明把决策压到图 embedding 里会损失关键局部观测。HATT v2.0 在该组里同时提升 success、completion 和 reward，是比 v1/v2.2 更稳的结构。
代表性横向 Eval
macf-sweep-v2-eval_20260624-222659
| Model | Family | Success | Reward | Completion | Length | Dead |
|---|---|---|---|---|---|---|
| mappo-v2 | MAPPO | 96.9% | 1.41 | 99.7% | 521.5 | 0.19 |
| hattv2.0-v2 | HATT v2.0 | 50.0% | -25.70 | 85.3% | 614.6 | 0.41 |
macf-sweep-v4-eval_20260625-193045
| Model | Family | Success | Reward | Completion | Length | Dead |
|---|---|---|---|---|---|---|
| mappo-v4-fullrandom | MAPPO | 96.9% | 1.41 | 99.7% | 521.5 | 0.19 |
| hattv2.0-v4-fullrandom | HATT v2.0 | 93.8% | 14.43 | 99.1% | 521.8 | 0.34 |
| mappo-v4-curriculum | MAPPO | 65.6% | -109.84 | 92.5% | 610.6 | 0.50 |
| hattv2.0-v4-curriculum | HATT v2.0 | 21.9% | -56.84 | 74.4% | 681.2 | 1.06 |
macf-mid-260626-eval_20260626-160854
| Model | Family | Success | Reward | Completion | Length | Dead |
|---|---|---|---|---|---|---|
| hattv2.0-mid-fullrandom-260626 | HATT v2.0 | 100.0% | 44.72 | 100.0% | 474.5 | 0.12 |
| hattv2.0-mid-jitter-260626 | HATT v2.0 | 100.0% | 41.11 | 100.0% | 493.9 | 0.38 |
| mappo-mid-jitter-260626 | MAPPO | 93.8% | -96.08 | 97.0% | 659.1 | 1.94 |
| mappo-mid-fullrandom-260626 | MAPPO | 87.5% | -204.81 | 98.6% | 766.6 | 0.19 |
small-scale-eval_20260627-111300
| Model | Family | Success | Reward | Completion | Length | Dead |
|---|---|---|---|---|---|---|
| hattv2.0_jitter | HATT v2.0 | 100.0% | 17.06 | 100.0% | 401.9 | 0.22 |
| rmappo-hattv2.0_full-random | HATT v2.0 | 96.9% | 21.33 | 99.4% | 392.8 | 0.31 |
| hattv2.0_full-random | HATT v2.0 | 96.9% | 19.62 | 99.1% | 399.8 | 0.50 |
| rmappo_full-random | RMAPPO | 93.8% | -42.21 | 97.8% | 479.8 | 0.41 |
| mappo_jitter | MAPPO | 84.4% | -0.81 | 95.9% | 433.8 | 0.69 |
| mappo_full-random | MAPPO | 50.0% | -371.70 | 80.3% | 589.1 | 0.41 |
hatt-v23-small-scale-eval_20260628-041808
| Model | Family | Success | Reward | Completion | Length | Dead |
|---|---|---|---|---|---|---|
| hattv2.3c_full-random | HATT v2.3 | 96.9% | -4.91 | 99.7% | 525.3 | 0.16 |
| hattv2.2_full-random | HATT v2.2 | 96.9% | 7.62 | 99.1% | 516.5 | 0.19 |
| hattv2.0_full-random | HATT v2.0 | 87.5% | 4.64 | 97.2% | 526.9 | 0.22 |
| hattv2.3b_full-random | HATT v2.3 | 87.5% | 0.71 | 97.2% | 560.6 | 0.47 |
| hattv2.3a_full-random | HATT v2.3 | 75.0% | -3.24 | 96.9% | 565.5 | 0.50 |
| mappo_full-random | MAPPO | 50.0% | -298.12 | 81.6% | 657.4 | 0.22 |
hatt-v30-small-scale-eval_20260628-023944
| Model | Family | Success | Reward | Completion | Length | Dead |
|---|---|---|---|---|---|---|
| hattv3.0a_full-random | HATT v3.0 | 93.8% | -0.91 | 99.1% | 526.0 | 0.25 |
| hattv2.0_full-random | HATT v2.0 | 90.6% | -3.05 | 98.4% | 550.9 | 0.16 |
| hattv3.0c_full-random | HATT v3.0 | 78.1% | -12.70 | 94.7% | 570.1 | 0.66 |
| hattv3.0b_full-random | HATT v3.0 | 43.8% | -35.63 | 89.1% | 648.3 | 0.75 |
HATT v3.2 Checkpoint Sweep
下面是 32-episode checkpoint sweep，不是 4-episode 在线 eval。Stable ratio = success >= 87.5% 且 completion >= 95% 的 checkpoint 占比。
| Variant | N | Best ep | Best succ | Best reward | Best comp | Last ep | Last succ | Last reward | Last comp | Stable ratio |
|---|---|---|---|---|---|---|---|---|---|---|
| hattv3.1a-fixed | 15 | ep000100 | 93.8% | 9.75 | 99.4% | ep001399 | 84.4% | 6.18 | 92.2% | 40.0% |
| hattv3.1b-fixed | 16 | ep000999 | 96.9% | 14.78 | 99.7% | ep001499 | 65.6% | -19.54 | 95.6% | 37.5% |
| hattv3.2a | 16 | ep001499 | 96.9% | 15.68 | 99.7% | ep001499 | 96.9% | 15.68 | 99.7% | 31.2% |
| hattv3.2b | 16 | ep000299 | 100.0% | 18.18 | 100.0% | ep001499 | 87.5% | -45.65 | 95.0% | 68.8% |
| hattv3.2c | 16 | ep000600 | 96.9% | 23.48 | 99.7% | ep001499 | 93.8% | 9.27 | 99.4% | 62.5% |
TensorBoard 训练曲线
训练曲线用于判断后段趋势。可以看到 eval success 有跳变，reward/episode 后段差异没有单个 last eval 那么极端，这支持“best 强、后期弱与环境噪声强相关”的判断。
4-Episode Best/Latest 辅助表
这些是训练中保存的 best/latest，episode 数较少，只作为辅助，不作为最终排名。
| Run | Best ep | Best succ | Best reward | Latest ep | Latest succ | Latest reward |
|---|---|---|---|---|---|---|
20260628-0357_macf_hatt-v31-small-scale/hattv3.1a_full-random | 799 | 100.0% | 29.98 | 1499 | 100.0% | 19.67 |
20260628-0357_macf_hatt-v31-small-scale/hattv2.0_full-random | 899 | 100.0% | 18.38 | 1399 | 100.0% | 11.60 |
20260628-0357_macf_hatt-v31-small-scale/mappo_full-random | 1300 | 100.0% | 13.69 | 1499 | 75.0% | -38.15 |
20260628-0357_macf_hatt-v31-small-scale/hattv3.0a_full-random | 1099 | 100.0% | 24.14 | 1399 | 100.0% | -8.95 |
20260628-0357_macf_hatt-v31-small-scale/hattv3.1b_full-random | 1000 | 100.0% | 36.36 | 1399 | 100.0% | -4.43 |
20260628-1043_macf_hatt-v31-fixed-small-scale/hattv3.1a-fixed_full-random | 599 | 100.0% | 22.84 | 1499 | 50.0% | -35.71 |
20260628-1043_macf_hatt-v31-fixed-small-scale/hattv2.0_full-random | 599 | 100.0% | 18.17 | 1499 | 100.0% | -4.78 |
20260628-1043_macf_hatt-v31-fixed-small-scale/mappo_full-random | 199 | 100.0% | -13.08 | 300 | 50.0% | -60.00 |
20260628-1043_macf_hatt-v31-fixed-small-scale/hattv3.0a_full-random | 199 | 75.0% | 1.55 | 199 | 75.0% | 1.55 |
20260628-1043_macf_hatt-v31-fixed-small-scale/hattv3.1b-fixed_full-random | 999 | 100.0% | 29.37 | 1399 | 75.0% | -79.30 |
20260629-0104_macf_hatt-v32-small-scale/hattv3.1a-fixed_full-random | 799 | 100.0% | 29.25 | 1399 | 100.0% | 18.64 |
20260629-0104_macf_hatt-v32-small-scale/hattv2.0_full-random | 1 | 0.0% | -139.66 | 1 | 0.0% | -139.66 |
20260629-0104_macf_hatt-v32-small-scale/hattv3.2c_full-random | 300 | 100.0% | 28.69 | 1499 | 75.0% | -22.37 |
20260629-0104_macf_hatt-v32-small-scale/hattv3.2b_full-random | 599 | 100.0% | 29.49 | 1499 | 100.0% | -33.17 |
20260629-0104_macf_hatt-v32-small-scale/mappo_full-random | 1 | 0.0% | -245.98 | 1 | 0.0% | -245.98 |
20260629-0104_macf_hatt-v32-small-scale/hattv3.2a_full-random | 1399 | 100.0% | 24.81 | 1499 | 100.0% | 15.10 |
20260629-0104_macf_hatt-v32-small-scale/hattv3.1b-fixed_full-random | 1400 | 100.0% | 22.91 | 1499 | 50.0% | -54.51 |
代际评价
- MAPPO：可完成任务，但 reward 与 length 对环境变化敏感，full_random 下容易被距离、发现、补给惩罚拉低。
- LegacyGAT：cat/gated 有效，gatobs 不稳；说明图关系应作为增强信息而不是替代原始观测。
- HATT v2.0：多组小/中规模都优于 MAPPO，是稳定跃迁点。
- HATT v2.2/v2.3：有局部收益，但没有稳定压过 v2.0。
- HATT v3.0/v3.1：能力上限更强，best reward/length 改善明显；需要 checkpoint selection 或更稳定的评估集。
- HATT v3.2：推荐保留 v3.2a 和 v3.2c：前者后段稳，后者能力上限高。v3.2b 可作为高峰强但不稳的 ablation。v3.2d 失败：
RuntimeError: The size of tensor a (9000) must match the size of tensor b (12) at non-singleton dimension 1
建议
- 正式汇报表格使用 best / last / stable ratio 三列，并明确 eval episode 数。
- 后续实验同时跑 fixed scenario set 与 full_random scenario set：前者看训练趋势，后者看泛化。
- 优先复现实验：v3.2a、v3.2c、HATT v2.0、MAPPO，在相同 scenario seeds 上对比。
- 先修复 v3.2d mask 维度，再讨论 utility loss 版本性能。