This is not much of an article of research as much as it is a “philosophy” for team rankings and ratings. This is a topic that I want to collect some thoughts on as the Counter-Strike Professional Players Association (CSPPA) recently released their new ranking system, which is allegedly an improvement on the existing industry standard, HLTV.
I do not want to pontificate on the topic of which ranking and rating methodology is the best, but I want to illustrate some challenges that these methodologies have with Counter-Strike in particular. For clarity, I define the following:
I think that this is an important distinction to make because the ratings deal with smaller aspects of team or player performance, while the rankings are much more macro-level. I will offer an analogy to college basketball, where a rating could be offensive points per possession or defensive points per possession allowed. Obviously, these are major components and descriptors of team performance and skill; however, you would need to incorporate both of these ratings into a larger rankings framework in order to fully assess and compare team ability.
What makes Counter-Strike such a hard game to assess team talent?
The HLTV ranking system has been the industry standard for a long time, largely because there is no better alternative. The methodology behind their system is a secret, but their webpage (linked above) offers the following:
“HLTV’s world ranking ranks the best teams in the competitive field of Counter-Strike Global Offensive. The ranking is updated weekly, and teams move up or down based on their recent performance in competitive matches over the last 2 months. The world ranking is based on teams’ achievements over the past year (with severe decay in points throughout each month), recent form over the last 2 months and performance at the last 10 LAN events. All of that is predicated around a 3-man core each team is required to have in order to retain their past points. Online matches and tournaments are included, but with a diminishing factor which means they cannot make a big difference among the top teams, and mainly serve to put new teams on the map.”
Since the formula is not “out in the open”, not much can be said about the specific mathematics that they are using to generate the final “points” figures. However, we see that HLTV does try to address some of the issues posed above:
The heavy weight on LAN also somewhat addresses the Regionality issue, since the LAN events are generally international and allow for the different regions to compete against one another.
The CSPPA system is documented here. Apologies if the link is broken, but it is the “CSPPA Ranking Rulebook” link from csppa.gg/ranking. The CSPPA set out to release a ranking system that they felt was more fair to the players, in that it did not incentivize organizations to have their players play in every competition possible.
One great thing about the CSPPA system is their commitment to an open system, in that they release the “inner workings” and math used in the system. The main driver behind the ratings is an allocation of points to each event, depending on
The tournament points are then allocated to teams, based on team performance. Players, then, get 1/5 of the points that the team receives. The rankings reflect the sum of player points.
The issues are then addressed as follows:
Again, much like the HLTV system, the Regionality issue is hopefully addressed by the LAN vs online tournament distinction.
I refer to any weighting of recent vs past performances as “decay”, since you are reducing the importance of events past a certain point in time. I do not know what the underlying decay instrument is in the HLTV ratings, but the CSPPA ratings very openly show a tiered window-decay system. In the CSPPA system, events 0-4 weeks old get 100% weight, events 4-8 weeks old get 90% weight, events 8-12 weeks old get 80% weight, and so on.
One issue that I have with this sort of windowed system is that it makes the ratings somewhat more “volatile”. I think that this idea is best motivated with an example, so here is what I mean: imagine a team won a big tournament 27 days ago. This hypothetical team has not competed since this event concluded. Does it make sense that on Day 28, these points are magically 90% of what they used to be? While this may be an extreme example, it shows the issue that I have with a rigid interval. In my opinion, it would make a lot more sense to make a more linear decay pattern with days or weeks (as opposed to the “stepwise” nature of the rigid window system). In the hypothetical I posed earlier, this would mean that Day 28 would only have a miniscule decay in points compared to Day 27. This, in turn, makes the rankings more stable, which is a feature of a good rankings system.
It makes sense why the CSPPA chose to have a more player-reliant system. According to pages 9 and 10 of the linked pdf, players are able to retain the entirety of their points in the event of an organization switch (with the same roster), and lose 50% when going to an entirely new team. This is one area where I believe the CSPPA has a conceptual advantage over the HLTV system; it makes sense that the building blocks of the team receive points and the team reflects a combination of those points. HLTV does not publish how they come up with their points, but I would be surprised if it were any substantial deviation from this. The only explanation that HLTV offer is that the definition of a team is based on a three-man core.
I think that the CSPPA definition of a team is very intuitive, but the HLTV definition also offers some more stability and possibly a result that is more data-driven. In my opinion, the two methodologies differ with the answer to the hypothetical question: What happens if a team drops an established player and replaces them with a lesser-known player (ie 0 CSPPA points)?
We know that, under the CSPPA system, the organization would lose 100% of the dropped player’s points balance. They would then add 50% of the new player’s points (which we know to be 0). Is this too harsh? Assuming that the organization played with the same 5-man team prior to dropping the one player, does it make sense that the new team is simply 80% of the old team? Does the new player offer 0 value? I am not sure that this is the case, especially if the new player has not even had the ability to obtain points by playing in the major tournaments.
This hypothetical shows the advantage that the HLTV 3-man core system has. Again, we do not know to what extent individual players affect the HLTV system. It is very intuitive, however, that switching out one player (and thus retaining the core) would not have that drastic an effect on performance.
I think that the ideal system here is a mix of the two methodologies. Player performance should somehow be considered when roster changes are made. Otherwise, as shown in the hypothetical with the CSPPA system, the points total may not reflect the true performance value of the five players.
Unfortunately, we do not know the inner workings of the HLTV system when it comes to how it values particular events and evaluating relative performance within those events. This is what I mean with the concept of “performance weight”. CSPPA tries to address this with its tournament points allocation \(\rightarrow\) team points allocation.
This is perhaps the biggest issue that I have with the CSPPA system. I understand that 16th place, for example, in a 16-team tournament in which every team is a top-20 team is completely different than a 16th place finish in a 16-team tournament in which every team is not top-20. It is important that good or bad performance is weighed with the relative level of competition in that particular event or tournament. The issue that I have is using the ratings system to create weights for future ratings. I will motivate this issue with an example:
Let’s say that tournament organizers are interested in hosting a number of 16-team events, back-to-back. The organizers invite the top 16 teams, and the invitations are all accepted. These events have massive prize money, are all best-of-3 elimination formats at the minimum, and are essentially designed to maximize the number of CSPPA points allocated to the events. If these 16 teams are the only teams to receive points, due to the structuring of the points system, how do any of these teams drop out of the top 16? Are the other 4 teams in the top 20 banding together and playing a bunch of small events between themselves to get good results from CSPPA events?
Again, I do not take issue with weighing level of competition in a ranking system. It is important to contextualize performance with who that performance is against. This is a key element in any team rankings (or ratings) system. My argument is that the CSPPA tournament points allocation system is somewhat self-serving. I know that my hypothetical is not super realistic, but it shows a flaw in the system.
All in all, I am glad that the CSPPA released a new ranking system. I think that this space could use some new blood. There will never be a perfect ranking system (ask American college sports fans), but if the ranking systems are being used by tournament organizers to make invite or competition decisions, then the systems are important to the quality of these events.
I believe that the HLTV rankings should be more open, especially since they are being used to make these crucial decisions. I also understand CSPPA’s claim that the HLTV system incentivizes teams playing in all of these events (see Astralis dropping out of the top 10 while two players take a break). However, there clearly is a drop-off in performance for a team that has not played in two months. There is definitely a balance to be struck here.
I like the CSPPA’s system in the respect that the players hold the points. Ultimately, I am sure that the HLTV system is really not that far off in this respect, in terms of the player’s individual ratings somehow contributing to the team’s overall points for the ranking.
The CSPPA should re-evaluate the pattern in the point decay. Either make it linear or exponential, since a stepwise function is too “clunky”.
One issue that I feel is not publicly addressed well by HLTV and CSPPA is the regionality issue. I understand that the heavier reliance on LAN results somehat solves this problem. However, what happens during a period of no LAN events, such as this year’s Coronavirus pandemic? I personally know that regionality is an issue currently during this pandemic because teams such as Renegades that tear up their regional scene but cannot compete internationally often are outliers in ratings/rankings systems because they have very few losses. This is exacerbated by the fact that the teams that Renegades competes with also have no international competition for the past several months. How are these issues addressed by these systems? Clearly HLTV has some regional weighting, since zero teams not in North America or Europe (except for NaVi) are in the top 20. NaVi has also largely been participating in the European online events during the pandemic as well, which would explain their ability to keep a high ranking (#4 as of this writing). Where does a team like Renegades fall? Currently, they are ranked #54, despite being ranked in the mid-20s in the end of March/ beginning of April, when the world shut down.