Don’t optimize for metrics
In game design, the allure of metrics and data can be captivating. They offer a quantifiable glimpse into the player experience, promising insights and pathways to success. However, reliance on numbers often leads to a critical oversight — the human element of gaming. In this article, I talk about the nuanced relationship between game design and metrics, cautioning against the seductive simplicity of numbers and advocating for a more holistic approach that values player experience as much as statistical data.
This whole article could be called “Correlation doesn’t mean causation,” but that would make it a tweet.
I’m the first to advocate for collecting telemetry data from games to see what players are doing in your game, where they are struggling and succeeding, what they seem to be avoiding, and which resources they are hoarding. User interviews and playtests are great tools, too. Still, nothing gives you the feeling of being a powerful omniscient game development God as watching numbers go up and down in Grafana and Tableau.
Unfortunately, we sometimes forget all the logic and common sense by watching the game’s performance as a set of metrics and numbers. Averages are known to be misleading (both mean and median), but even dividing players into cohorts and categories can lead to self-deception.
Here is one typical example: you notice a cohort of highly engaged players spending more money on average than everyone else. Great! Let’s see what’s so different about them. Because you only have access to statistics you’re collecting and not to the actual players (plus, one-on-one interviews are messy and take time), you’ll likely sift through many metrics before finding that magical one that makes all the difference.
You might find that they play more hours per week on average than the regular player. You might discover they have more heroes, play a certain class, or have more soft currency or resources per account.
And then your first impulse might be to optimize for that metric. You probably are well-educated, and your boss might have Harvard on their resume, but when looking at numbers instead of people, it’s natural to focus on what’s in front of you.
You’ll make quests take longer, so players spend more time in the game. You will make heroes cheaper, encourage new players to start with a particular class and tune the resource drip so everyone outside of that highly engaged cohort earns more of them.
Guess what? It won’t change anything because you’re optimizing for metrics and not for people. Your best cohort probably likes your game and knows it well, so they spend more time and money and have more heroes and resources. That particular class might be a bit overpowered in high-skilled play, but it is not necessarily easy or fun to play for someone with less experience.
It’s like looking at the metrics of a regular furniture store, seeing people spend 20 minutes buying a sofa, and deciding that if we lock them in the store for an hour, they’ll purchase three!
And yes, I’m describing IKEA here, but IKEA isn’t just forcing consumers to go through a labyrinth when they shop for the sake of spending more time in the store — they are using this time to show them every cool and weirdly named item in their catalog in the best possible way. A person will not buy three sofas, but they will probably end up with an extra lamp or those funny stackable boxes.
This is also one of the reasons for our industry’s collective obsession with esports. Competitive gaming is a completely legitimate activity, but in most games, both playing in tournaments and watching them are reserved only for the most dedicated and hardcore gamers, who are, by definition, also the highest spenders. You can’t make everyone else spend more by encouraging them to watch other players compete in your title.
Some games have naturally higher rates of competitive players and esports audiences, and some have lower. It would help to understand where your game stands before starting your league. Please note that just because someone likes to compete in League of Legends and is also playing your game doesn’t mean they will want to play here competitively, too. Sometimes, even the most hardcore gamers want a chill experience. Maybe they chose to play your title in addition to LOL precisely because they want a place to relax.
You can always try to shape player behavior with in-game incentives, like “watch the tournament — get an in-game item” or “play this class — get another one”. But as I explained in my article on Battle Pass basics, trying to force people to do things they fundamentally do not enjoy will lead to higher attrition rates. Plainly speaking — if players aren’t having fun in your game, they will leave, no matter how many in-game carrots you’re dangling in front of them. Yes, you can get a short-term boost in engagement, but in live-service products, it’s always about long-term retention, which depends solely on how much players enjoy your game.
I believe in meta systems supporting, not shaping, player behavior by rewarding them for the fun activities they already do.
Ok, but what do I do if I can’t trust metrics? Well, you can and should trust metrics, you just need to think about people and not numbers when making game design decisions. Take an extra step to investigate what is happening, set up in-game surveys, invite some randomly selected players for interviews, and watch them play on Twitch (I recommend going for channels with fewer viewers because they’re closer to natural play).
Also, do not forget about player archetypes. There is no universal way to describe user behavior that would fit every game, but you can guess what kind of players enjoy yours by looking at them and then adjusting your game and its store offerings accordingly.
Often, the answer is not about changing something for the sake of the metric but about making the game more interesting for the less-engaged cohort or ensuring the store has items they want. You might find out that some of them don’t see the benefit of your Battle Pass, but they’d appreciate a shorter and more achievable engagement mechanic (like an event pass or a boost). Or they plainly don’t like any skins you’re selling and want something different.
Metrics are invaluable tools, providing key insights and guiding game development. However, they should not be the sole driver of design decisions. A successful game resonates with its players, offering engaging, enjoyable experiences that cannot always be quantified. Designers should use metrics as a compass, not a map. By blending the art of game design with the science of data analysis, we can create games that are not only statistically successful but also beloved by the players.
