Results, Evaluation and Awards of the Deterministic Part of IPC-5
Please note that with our evaluation
by no means we intend to imply that one particular planner is better
than the other ones in general. There can be other evaluation
criteria that give different results. Everybody who is interested in
the results of IPC-5 is strongly encouraged to look them and derive
her/his own conclusions!
The plots of the performance of all competing planners in the
deterministic track of IPC can be found here. An archive with all
results can be found here.
Slides of the presentation at
ICAPS-06 on the IPC-5 results.
General Evaluation Criteria of IPC-5
- Different evaluation/prizes for optimal and suboptimal planners
- For optimal planners: number of solved problems and CPU-time
(CPU-time limit: 30 minutes)
- For satisficing planners (CPU-time limit: 30 minutes):
- Number of solved problems and plan quality
- CPU-time (secondary measure)
-
For few domains these criteria were not enough to clearly distinguish
the 1st or 2nd place. In such cases, if the uncertainly is for the 1st
place, we assign multiple 1st places; if the uncertainty is for the
2nd place, either we assign multiple 2nd places or we consider more in
details the differences and if necessary we vote.
- Planner ranking by domain category (as in IPC-4):
- for each domain in a domain category we assign 1st/2nd places;
- in each category, all 1st/2nd places are then summed
- IPC-4 best planners used as reference for measuring the
performance improvements in the Strips and MetricTime domain
categories.
Summary of 1st/2nd Places (optimal planners with at least one 1st or 2nd place)
|
CPT2
|
MIPS-BDD
|
SATPLAN
|
Maxplan
|
FDP
|
Propositional Domains (1st / 2nd Places)
|
0 / 1
|
1 / 1
|
3 / 2
|
3 / 2
|
0 / 3
|
Temporal Domains (1st / 2nd Places)
|
2 / 0
|
|
|
|
|
Summary of 1st/2nd Places (satisficing planners with at least one 1st or 2nd place)
|
Downward
|
MIPS-BDD
|
MIPS-XXL
|
SGPLAN5
|
HPlan-P
|
YochanPS
|
Propositional Domains (1st / 2nd
Places)
|
1 / 4
|
|
0 / 1
|
5 / 2
|
|
0 / 1
|
Metric/Temporal Domains (1st /
2nd Places)
|
|
|
0 / 3
|
8 / 1
|
|
1 / 3
|
SimplePreferences Domains (1st /
2nd Places)
|
|
0 / 1
|
0 / 4
|
6 / 0
|
|
0 / 4
|
QualitativePreferences Domains (1st / 2nd Places)
|
|
|
|
5 / 0
|
0 / 5
|
|
Constraints Domains (1st / 2nd Places)
|
|
|
0 / 3
|
3 / 0
|
|
|
ComplexPreferences Domains (1st / 2nd Places)
|
|
|
0 / 3
|
5 / 0
|
|
|
Awards
- 1st Prize Optimal Planning (Propositional Domains) -- Maxplan, by Zhao Xing, Yixin Chen and Weixiong Zhang
- 1st Prize Optimal Planning (Propositional Domains) -- Satplan, by Joerg Hoffmann, Henry Kautz, Shane Neph and Bart Selman
- Distinguished Performance in Optimal Planning (Temporal Domains) -- CPT2, by Vincent Vidal and Sèbastien Tabary
- 1st Prize Satisficing Planning -- SGPlan, by Benjamin W. Wah, Chih-Wei Hsu, Yixin Chen and Ruoyun Huang
- Distinguished Performance in Satisficing Planning
(Simple/Complex Preferences and Constraints Domains) -- MIPS-XXL, by Stefan
Edelkamp, Shahid Jabbar and Mohammed Nazih
- Distinguished Performance in Satisficing Planning (SimplePreferences Domains) -- YochanPS, by J. Benton, Subbarao Kambhampati and Minh B. Do
- Distinguished Performance in Satisficing Planning (QualitativePreferences Domains) -- HPlan-P, by Jorge Baier, Fahiem Bacchus and Sheila McIlraith