Stormgate 2024-02 Playtest Observations
by Martin Kryl (Keiras at Discord | linktr)
Matchmaking quality
There has been a global 1v1 matchmaking available in the Elephant stage playtest period. From the comments made by Frost Giant developers, the matchmaker should consider player skill (which is traditionally done) as well as their ping to different servers in order to make optimal pairings.
The chart below shows the number of matches played each day grouped into MMR ranges. There are two qualitative metrics constructed to help with monitoring the matchmaking quality: difference in MMR and difference in pings of matched players. Lower values are desired. Five categories are used in each metric to help with the interpretation of the data. Stacked charts show proportions of matches in each category.
Until the end of Steam Next Fest (open beta stage), 1500 MMR is seen as the dominant bin. This is primarily due to influx of new players, which all start at this MMR. Experimentally, the system seems to be based on Elo with parameters K=20.4, c=420. In this context, the quality range thresholds correspond with implied win chances of >43.2%, >36.6%, >25%, >10%.
There has been a noticeable shift in histogram distribution after the open beta ended and the player count dropped significantly. All tracked quantiles shifted to the right with median MMR (q50) jumping to 1600. Apart from that, MMR diff quality metric worsened as well, while Ping diff stayed the same throughout the stage.
Faction preference
There were two playable factions in this stage: the human-like Vanguard and the demonic Infernals. There was a clear preference for players to choose and stick with only one faction — 75% of players played mostly (more than 80% of their games) with their preferred faction.
In all players cohort (of 12,579), Vanguard were the favorites. But when looking at the Top 100 players, the trend was reversed.
MMR convergence
MMR systems tend to have a mechanism to boost point gains to more quickly settle new players into suitable MMR regions. This volatility is reduced over time until the MMR stabilizes. This chart shows the distribution of the number of games needed to stabilize MMR.
On average, it took 42 games for MMR to stabilize, and 95% of the players* never reached that point.
Cross-region Play
Players are not bound to their regional servers, but rather can play with anyone provided the matchmaker can find a middle ground with reasonable ping. The chart below shows a systematic map of the Top 100 players and their primary regions. On hover, all the players that have played with the selected player are highlighted. The size of the highlighted dot indicates the number of games played.
※ Servers with no players having it as their primary are not shown.
The Top 100 players do not appear to have issues getting matches between regions and reaching most other competitors. Apart from 8 pairs, everyone had a direct match with each other or by proxy via another player. The 8 pairs could be an artifact created by arbitrarily cutting the line at the top 100.
In general, it seems reasonable to expect a fair exchange of MMR between regions in this global system.
The last chart shows the number of different servers that Top 100 players have played on. The matchmaker appears to have a soft limit on the maximum ping allowed (125 ms), with some cases of higher ping (up to 210 ms). This, together with a low sample size, might distort the results slightly. Nevertheless, the common number of reachable servers seems to be 4-7 for top players.
The data used in these charts were obtained from Stormgate World using the matches and leaderboards/ranked_1v1 endpoints. Data were collected after the conclusion of the Elephant stage. From my understanding, some of the matches played during the Steam Next Fest are missing, and there is nothing that can be done about that.
In case anyone is interested, I can provide the processed dataset (in DuckDB) for further analysis or validation. The underlying data for the charts on this page are included in the repository.
Thanks to Frost Giant for making the game and to The CASUALS for making the data available via the Stormgate World project ❤