Leaderboard#
We have undertaken rigorous evaluations of BC, BCQ-MA, CQL-MA, ICQ-MA, and MADT on the D4MARL dataset. Multiple metrics have been amassed and juxtaposed. The intricacies of these experiments are elaborated comprehensively in our paper. We anticipate that forthcoming algorithms in the SMAC domain will leverage our D4MARL dataset, facilitating comparisons within a unified framework.
Leaderboard
In the presented table, we showcase the empirical outcomes of diverse algorithms applied to the D4MARL dataset. As additional algorithms are proposed and evaluated on the D4MARL dataset, we will continuously update this leaderboard with their results.
| Map Name | Quality | Method | Static Accuracy (%) | Return | Win Rate | Time-to-Threshold(e4) | |
|---|---|---|---|---|---|---|---|
| SAdev | SAtest | ||||||
| 2m_vs_1z | Expert | BC | 73.68 | 71.46 | 3.95 | 0.98 | -- |
| BCQ-MA | 40.12 | 40.69 | 17.97 | 84.01 | -- | ||
| CQL-MA | 9.23 | 9.41 | 18.61 | 90.90 | -- | ||
| ICQ-MA | 58.81 | 59.25 | 19.19 | 93.84 | -- | ||
| MADT | 50.91 | 50.73 | 19.59 | 97.12 | 1.886 | ||
| Medium | BC | 71.34 | 73.54 | 3.96 | 0 | -- | |
| BCQ-MA | 51.96 | 49.47 | 14.47 | 59.38 | -- | ||
| CQL-MA | 9.44 | 10.96 | 8.47 | 21.62 | -- | ||
| ICQ-MA | 64.79 | 65.09 | 16.87 | 78.99 | -- | ||
| MADT | 49.82 | 49.10 | 18.24 | 86.27 | 1.643 | ||
| Poor | BC | 71.62 | 71.71 | 3.683 | 0 | -- | |
| BCQ-MA | 63.10 | 67.78 | 5.57 | 0.06 | -- | ||
| CQL-MA | 37.02 | 32.18 | 6.20 | 1.01 | -- | ||
| ICQ-MA | 35.52 | 40.99 | 8.948 | 18.56 | -- | ||
| MADT | 56.46 | 55.59 | 5.43 | 2.45 | 6.749 | ||
| 2s_vs_1sc | Expert | BC | 91.95 | 91.81 | 15.63 | 53.95 | -- |
| BCQ-MA | 63.10 | 63.81 | 19.99 | 98.01 | -- | ||
| CQL-MA | 78.40 | 78.35 | 19.89 | 95.97 | -- | ||
| ICQ-MA | 16.41 | 18.77 | 20.16 | 99.28 | -- | ||
| MADT | 73.85 | 71.10 | 20.24 | 99.97 | 0.2597 | ||
| Medium | BC | 90.33 | 91.41 | 0 | 0 | -- | |
| BCQ-MA | 73.64 | 74.85 | 19.85 | 95.10 | -- | ||
| CQL-MA | 82.60 | 85.04 | 13.26 | 15.71 | -- | ||
| ICQ-MA | 22.44 | 22.69 | 0 | 0 | -- | ||
| MADT | 71.57 | 73.04 | 19.74 | 94.49 | 1.211 | ||
| Poor | BC | 74.14 | 57.77 | 0 | 0 | -- | |
| BCQ-MA | 70.71 | 61.86 | 0 | 0 | -- | ||
| CQL-MA | 68.82 | 57.35 | 8.79 | 0 | -- | ||
| ICQ-MA | 17.73 | 33.58 | 0.42 | 0 | -- | ||
| MADT | 49.64 | 50.88 | 17.48 | 68.24 | 2.724 | ||
| 3m | Expert | BC | 91.39 | 89.47 | 14.36 | 55.07 | 0.1607 |
| BCQ-MA | 75.91 | 71.58 | 15.86 | 66.98 | 0.1430 | ||
| CQL-MA | 13.94 | 15.36 | 11.66 | 33.67 | 0.2209 | ||
| ICQ-MA | 27.75 | 30.60 | 15.93 | 68.22 | 0.2332 | ||
| MADT | 78.13 | 78.40 | 19.59 | 96.88 | 0.1223 | ||
| Poor | BC | 80.30 | 81.49 | 14.11 | 52.47 | -- | |
| BCQ-MA | 74.35 | 73.28 | 13.81 | 49.56 | -- | ||
| CQL-MA | 51.11 | 53.38 | 0.45 | 0 | -- | ||
| ICQ-MA | 18.06 | 22.21 | 14.68 | 59.15 | -- | ||
| MADT | 68.36 | 74.04 | 15.15 | 63.01 | 0.1554 | ||
| 2s3z | Expert | BC | 78.01 | 71.79 | 15.73 | 34.35 | -- |
| BCQ-MA | 76.15 | 73.14 | 19.03 | 83.42 | -- | ||
| CQL-MA | 22.30 | 21.41 | 18.73 | 77.04 | -- | ||
| ICQ-MA | 14.99 | 14.68 | 17.59 | 60.58 | -- | ||
| MADT | 58.59 | 60.04 | 19.93 | 98.61 | 0.2907 | ||
| Medium | BC | 75.99 | 73.27 | 13.20 | 16.48 | -- | |
| BCQ-MA | 75.69 | 74.54 | 17.64 | 62.39 | -- | ||
| CQL-MA | 27.45 | 26.05 | 15.91 | 40.01 | -- | ||
| ICQ-MA | 15.66 | 15.61 | 13.22 | 17.33 | -- | ||
| MADT | 54.67 | 52.89 | 18.66 | 80.66 | 0.3246 | ||
| Poor | BC | 74.02 | 72.74 | 7.61 | 0 | -- | |
| BCQ-MA | 73.83 | 72.35 | 9.57 | 8.20 | -- | ||
| CQL-MA | 45.02 | 39.98 | 6.65 | 0 | -- | ||
| ICQ-MA | 6.71 | 7.21 | 7.26 | 0 | -- | ||
| MADT | 56.49 | 55.94 | 14.39 | 25.29 | 58.97 | ||
| 3s_vs_3z | Expert | BC | 64.13 | 63.46 | 8.77 | 9.38 | -- |
| BCQ-MA | 45.03 | 44.96 | 18.90 | 82.40 | -- | ||
| CQL-MA | 6.79 | 6.10 | 15.78 | 42.30 | -- | ||
| ICQ-MA | 13.06 | 12.60 | 17.15 | 62.63 | -- | ||
| MADT | 54.34 | 52.73 | 19.21 | 84.25 | 0.3778 | ||
| Medium | BC | 61.71 | 59.85 | 6.41 | 0 | -- | |
| BCQ-MA | 52.36 | 51.17 | 0 | 0 | -- | ||
| CQL-MA | 9.42 | 5.72 | 8.93 | 1.52 | -- | ||
| ICQ-MA | 12.37 | 13.35 | 11.12 | 14.66 | -- | ||
| MADT | 47.25 | 47.33 | 9.26 | 5.18 | 21.26 | ||
| Poor | BC | 74.02 | 72.74 | 7.61 | 0 | -- | |
| BCQ-MA | 73.83 | 72.35 | 9.57 | 8.20 | -- | ||
| CQL-MA | 45.02 | 39.98 | 6.65 | 0 | -- | ||
| ICQ-MA | 6.71 | 7.21 | 7.26 | 0 | -- | ||
| MADT | 52.50 | 52.12 | 9.62 | 0.25 | 61.49 | ||
| 3s_vs_4z | Expert | BC | 69.78 | 66.71 | 8.74 | 2.27 | -- |
| BCQ-MA | 28.81 | 28.92 | 18.78 | 78.26 | -- | ||
| CQL-MA | 13.42 | 15.48 | 11.67 | 11.64 | -- | ||
| ICQ-MA | 13.08 | 12.59 | 13.30 | 25.01 | -- | ||
| MADT | 62.80 | 62.13 | 19.27 | 88.09 | 4.182 | ||
| Medium | BC | 63.89 | 60.49 | 2.92 | 0 | -- | |
| BCQ-MA | 30.63 | 32.05 | 4.182 | 2.57 | -- | ||
| CQL-MA | 17.20 | 17.45 | 6.02 | 0 | -- | ||
| ICQ-MA | 8.55 | 9.21 | 3.10 | 0 | -- | ||
| MADT | 58.75 | 58.63 | 6.24 | 16.95 | 14.86 | ||
| Poor | BC | 69.17 | 59.78 | 4.44 | 0 | -- | |
| BCQ-MA | 47.87 | 41.61 | 5.99 | 0 | -- | ||
| CQL-MA | 34.01 | 29.67 | 4.44 | 0 | -- | ||
| ICQ-MA | 7.96 | 7.10 | 5.66 | 0 | -- | ||
| MADT | 60.14 | 60.26 | 7.56 | 3.82 | 19.23 | ||
| 3s_vs_5z | Expert | BC | 83.08 | 80.30 | 18.27 | 51.27 | -- |
| BCQ-MA | 46.93 | 49.09 | 23.09 | 83.86 | -- | ||
| CQL-MA | 18.31 | 21.25 | 21.64 | 79.40 | -- | ||
| ICQ-MA | 7.15 | 7.62 | 24.22 | 95.95 | -- | ||
| MADT | 71.08 | 70.51 | 24.07 | 99.21 | 0.8284 | ||
| Medium | BC | 83.97 | 83.42 | 14.41 | 23.59 | -- | |
| BCQ-MA | 52.49 | 54.76 | 17.29 | 51.18 | -- | ||
| CQL-MA | 27.64 | 30.78 | 19.96 | 75.02 | -- | ||
| ICQ-MA | 6.79 | 5.60 | 20.84 | 75.14 | -- | ||
| MADT | 68.75 | 69.60 | 19.80 | 62.08 | 0.7421 | ||
| Poor | BC | 79.11 | 70.92 | 4.97 | 0 | -- | |
| BCQ-MA | 67.05 | 68.42 | 15.08 | 19.77 | -- | ||
| CQL-MA | 54.04 | 49.80 | 9.78 | 2.23 | -- | ||
| ICQ-MA | 3.39 | 3.39 | 7.68 | 0 | -- | ||
| MADT | 60.70 | 59.62 | 16.41 | 29.18 | 4.571 | ||
| 2c_vs_64zg | Expert | BC | 42.57 | 32.92 | 14.19 | 0 | -- |
| BCQ-MA | 30.90 | 23.84 | 13.27 | 0 | -- | ||
| CQL-MA | 14.59 | 13.84 | 7.57 | 0 | -- | ||
| ICQ-MA | 7.38 | 4.98 | 12.90 | 0 | -- | ||
| MADT | 61.17 | 60.56 | 19.15 | 75.00 | 0.5439 | ||
| Medium | BC | 36.65 | 27.14 | 12.16 | 0 | -- | |
| BCQ-MA | 29.22 | 21.75 | 12.97 | 0 | -- | ||
| CQL-MA | 13.15 | 13.94 | 7.57 | 0 | -- | ||
| ICQ-MA | 7.38 | 4.98 | 9.04 | 0 | -- | ||
| MADT | 59.62 | 59.75 | 15.05 | 21.88 | 8.887 | ||
| Poor | BC | 44.80 | 20.38 | 9.95 | 0 | -- | |
| BCQ-MA | 49.09 | 25.07 | 10.07 | 0 | -- | ||
| CQL-MA | 33.10 | 17.41 | 7.63 | 0 | -- | ||
| ICQ-MA | 5.52 | 3.49 | 8.96 | 0 | -- | ||
| MADT | 55.14 | 56.23 | 9.27 | 0 | 36.83 | ||
| 8m | Expert | BC | 67.71 | 52.72 | 14.74 | 44.62 | -- |
| BCQ-MA | 57.44 | 52.71 | 19.76 | 96.63 | -- | ||
| CQL-MA | 21.03 | 19.73 | 15.80 | 53.45 | -- | ||
| ICQ-MA | 11.87 | 11.72 | 19.20 | 90.57 | -- | ||
| MADT | 64.15 | 64.07 | 19.71 | 96.88 | 0.1596 | ||
| Medium | BC | 63.35 | 57.66 | 12.69 | 18.12 | -- | |
| BCQ-MA | 65.74 | 69.51 | 16.94 | 63.44 | -- | ||
| CQL-MA | 25.66 | 49.43 | 10.25 | 3.55 | -- | ||
| ICQ-MA | 11.81 | 12.06 | 17.93 | 78.85 | -- | ||
| MADT | 63.12 | 64.73 | 19.15 | 90.63 | 1.007 | ||
| Poor | BC | 76.63 | 57.51 | 4.75 | 0 | -- | |
| BCQ-MA | 73.16 | 67.50 | 13.18 | 17.96 | -- | ||
| CQL-MA | 56.18 | 59.12 | 6.91 | 0 | -- | ||
| ICQ-MA | 7.14 | 10.22 | 12.14 | 16.54 | -- | ||
| MADT | 59.18 | 60.03 | 4.25 | 0 | 16.17 | ||
| MMM | Expert | BC | 38.99 | 34.49 | 12.16 | 6.56 | -- |
| BCQ-MA | 29.93 | 28.78 | 19.65 | 71.85 | -- | ||
| CQL-MA | 24.11 | 25.61 | 13.01 | 10.07 | -- | ||
| ICQ-MA | 7.38 | 6.97 | 19.47 | 70.42 | -- | ||
| MADT | 33.28 | 32.72 | 19.09 | 59.00 | 9.669 | ||
| Medium | BC | 49.84 | 42.40 | 10.89 | 5.39 | -- | |
| BCQ-MA | 34.32 | 32.92 | 15.86 | 37.86 | -- | ||
| CQL-MA | 34.89 | 35.53 | 9.24 | 1.82 | -- | ||
| ICQ-MA | 8.34 | 8.62 | 15.29 | 34.38 | -- | ||
| MADT | 33.68 | 32.66 | 15.38 | 45.42 | 5.139 | ||
| Poor | BC | 68.46 | 63.41 | 7.48 | 0 | -- | |
| BCQ-MA | 60.07 | 64.35 | 8.51 | 1.20 | -- | ||
| CQL-MA | 56.07 | 64.72 | 5.79 | 0 | -- | ||
| ICQ-MA | 6.44 | 8.27 | 3.46 | 0 | -- | ||
| MADT | 41.93 | 40.94 | 7.48 | 7.98 | ∞ | ||
| bane_vs_bane | Expert | BC | 44.08 | 41.77 | 19.31 | 84.06 | -- |
| BCQ-MA | 41.65 | 67.23 | 19.85 | 96.07 | -- | ||
| CQL-MA | 29.34 | 64.71 | 17.42 | 49.48 | -- | ||
| ICQ-MA | 12.73 | 10.65 | 19.44 | 85.02 | -- | ||
| MADT | 28.21 | 26.31 | 19.99 | 99.54 | 0.0822 | ||
| Medium | BC | 64.28 | 37.68 | 18.69 | 65.51 | -- | |
| BCQ-MA | 40.67 | 43.74 | 18.75 | 74.33 | -- | ||
| CQL-MA | 24.62 | 40.79 | 15.32 | 24.51 | -- | ||
| ICQ-MA | 0.98 | 1.29 | 18.24 | 59.90 | -- | ||
| MADT | 29.77 | 28.68 | 19.96 | 98.71 | 0.1326 | ||
| Poor | BC | 74.10 | 77.12 | 17.22 | 42.71 | -- | |
| BCQ-MA | 80.73 | 98.09 | 18.69 | 66.02 | -- | ||
| CQL-MA | 72.21 | 96.01 | 17.14 | 40.26 | -- | ||
| ICQ-MA | 0.84 | 1.14 | 16.89 | 46.63 | -- | ||
| MADT | 36.99 | 36.46 | 18.16 | 59.54 | 18.43 | ||
| 25m | Expert | BC | 58.25 | 51.48 | 13.26 | 20.74 | -- |
| BCQ-MA | 50.87 | 49.17 | 19.44 | 87.17 | -- | ||
| CQL-MA | 33.39 | 34.51 | 13.11 | 0 | -- | ||
| ICQ-MA | 2.19 | 1.91 | 16.92 | 38.28 | -- | ||
| MADT | 48.94 | 47.29 | 19.88 | 96.20 | 0.3219 | ||
| Medium | BC | 59.74 | 51.46 | 13.54 | 7.87 | -- | |
| BCQ-MA | 59.46 | 50.39 | 13.48 | 2.78 | -- | ||
| CQL-MA | 48.41 | 47.02 | 12.59 | 0 | -- | ||
| ICQ-MA | 1.43 | 1.31 | 18.53 | 60.34 | -- | ||
| MADT | 46.98 | 45.56 | 19.25 | 84.17 | 18.82 | ||
| Poor | BC | 78.85 | 78.44 | 3.10 | 0 | -- | |
| BCQ-MA | 75.01 | 91.54 | 7.159 | 0 | -- | ||
| CQL-MA | 68.68 | 89.14 | 6.44 | 0 | -- | ||
| ICQ-MA | 0.58 | 0.45 | 6.01 | 0 | -- | ||
| MADT | 53.59 | 52.46 | 7.916 | 0 | 43.06 | ||
| 3s5z | Expert | BC | 43.48 | 46.07 | 9.39 | 1.46 | -- |
| BCQ-MA | 57.61 | 58.45 | 18.90 | 83.70 | -- | ||
| CQL-MA | 21.98 | 25.76 | 17.18 | 56.53 | -- | ||
| ICQ-MA | 7.43 | 7.68 | 17.85 | 64.39 | -- | ||
| MADT | 58.54 | 56.88 | 19.28 | 85.88 | 0.5889 | ||
| Medium | BC | 63.59 | 56.63 | 12.41 | 7.69 | -- | |
| BCQ-MA | 62.80 | 56.39 | 17.19 | 58.07 | -- | ||
| CQL-MA | 30.10 | 26.12 | 16.22 | 39.76 | -- | ||
| ICQ-MA | 6.88 | 7.18 | 14.69 | 28.51 | -- | ||
| MADT | 57.75 | 56.47 | 16.28 | 51.97 | 110.1 | ||
| Poor | BC | 74.59 | 61.77 | 8.55 | 0 | -- | |
| BCQ-MA | 72.86 | 62.50 | 12.82 | 18.22 | -- | ||
| CQL-MA | 47.08 | 44.29 | 9.72 | 2.13 | -- | ||
| ICQ-MA | 4.20 | 3.43 | 11.34 | 15.85 | -- | ||
| MADT | 58.66 | 58.62 | 9.96 | 0 | 281.7 | ||
| MMM2 | Expert | BC | 63.71 | 61.46 | 8.00 | 0 | -- |
| BCQ-MA | 43.12 | 42.79 | 12.51 | 18.42 | -- | ||
| CQL-MA | 25.41 | 26.43 | 9.25 | 1.02 | -- | ||
| ICQ-MA | 17.02 | 17.96 | 9.76 | 3.93 | -- | ||
| MADT | 53.87 | 53.86 | 18.81 | 75.85 | 44.30 | ||
| Medium | BC | 67.85 | 56.71 | 6.89 | 0 | -- | |
| BCQ-MA | 52.28 | 45.35 | 9.02 | 2.34 | -- | ||
| CQL-MA | 37.52 | 35.53 | 7.94 | 1.07 | -- | ||
| ICQ-MA | 18.59 | 14.06 | 8.32 | 1.67 | -- | ||
| MADT | 55.00 | 55.31 | 16.25 | 54.95 | 106.8 | ||
| Poor | BC | 78.13 | 76.16 | 1.33 | 0 | -- | |
| BCQ-MA | 68.95 | 76.42 | 3.37 | 0 | -- | ||
| CQL-MA | 57.27 | 70.12 | 1.85 | 0 | -- | ||
| ICQ-MA | 54.29 | 66.44 | 4.46 | 0 | -- | ||
| MADT | 58.92 | 57.87 | 4.93 | 1.34 | ∞ | ||
| 10m_vs_11m | Expert | BC | 61.05 | 54.32 | 9.30 | 1.21 | -- |
| BCQ-MA | 51.54 | 46.40 | 12.77 | 17.63 | -- | ||
| CQL-MA | 32.95 | 31.96 | 11.06 | 3.65 | -- | ||
| ICQ-MA | 4.51 | 4.45 | 14.25 | 26.80 | -- | ||
| MADT | 50.70 | 49.42 | 17.37 | 66.73 | 5.306 | ||
| Medium | BC | 67.87 | 60.18 | 8.86 | 0 | -- | |
| BCQ-MA | 57.84 | 55.31 | 10.88 | 3.48 | -- | ||
| CQL-MA | 41.74 | 41.99 | 11.71 | 8.86 | -- | ||
| ICQ-MA | 4.61 | 4.54 | 11.63 | 4.60 | -- | ||
| MADT | 49.58 | 47.58 | 16.22 | 47.91 | 1.790 | ||
| Poor | BC | 81.80 | 77.60 | 4.34 | 0 | -- | |
| BCQ-MA | 72.39 | 87.48 | 6.55 | 0 | -- | ||
| CQL-MA | 61.04 | 71.80 | 2.20 | 0 | -- | ||
| ICQ-MA | 3.56 | 2.41 | 6.64 | 0 | -- | ||
| MADT | 57.19 | 54.34 | 4.43 | 0 | ∞ | ||
| corridor | Expert | BC | 29.32 | 30.41 | 6.65 | 0.33 | -- |
| BCQ-MA | 49.13 | 45.63 | 11.45 | 17.65 | -- | ||
| CQL-MA | 14.22 | 16.03 | 9.44 | 12.79 | -- | ||
| ICQ-MA | 4.16 | 3.96 | 11.86 | 20.25 | -- | ||
| MADT | 42.95 | 42.80 | 18.91 | 85.85 | 5.151 | ||
| Medium | BC | 43.90 | 39.84 | 1.71 | 1.71 | -- | |
| BCQ-MA | 50.05 | 47.21 | 8.24 | 16.40 | -- | ||
| CQL-MA | 21.00 | 26.15 | 3.15 | 0.77 | -- | ||
| ICQ-MA | 3.60 | 3.70 | 6.75 | 3.49 | -- | ||
| MADT | 43.82 | 43.27 | 15.80 | 56.05 | 41.25 | ||
| Poor | BC | 58.02 | 50.92 | 3.01 | 0 | -- | |
| BCQ-MA | 58.76 | 65.40 | 3.20 | 0 | -- | ||
| CQL-MA | 44.89 | 60.10 | 3.28 | 0 | -- | ||
| ICQ-MA | 2.42 | 2.12 | 3.19 | 0 | -- | ||
| MADT | 41.41 | 40.43 | 8.83 | 11.08 | 53.18 | ||