Fighting games are a really interesting genre when it comes to user experience research. There's so many small decisions you have to make even while you aren't "doing" anything, you have to understand exactly what your opponent is capable of at any moment, there's demanding techniques that need to be executed on command, and there's usually resources to keep track of that change the risk reward and options in any given situation. Because of this, it's a famously difficult genre to learn. New players tend to spout off opinions on why this is, be it executional barriers, speed, memorization problems, etc. Established players often adopt the "get good" mentality which derides these problems; because of this, "bad tutorials" tend to be the go to scapegoat for why new players struggle. On one side we have ease of use but on the other we actually want certain things to be difficult, which is a UX problem that's basically exclusive to games. For instance, most fighting games require you to block high or low, with different moves needing to be blocked one way or the other. If every attack took more than 250 milliseconds to hit (average reaction time), the game would be quite stagnant and defensive. The great challenge of approaching the usability of a game like this is finding ways to minimize the frustration of a new player while maintaining the game's integrity as a competitive outlet. Finding out how the player base feels about wherever the game falls on this spectrum is vital; something might be loudly downplayed by one or more users but actually be legitimately unenjoyable or unintentionally difficult for a majority of users, and visa versa. This sort of situation is prime for the application of surveys and statistical analysis. Because of that, I've collected user feedback via survey for a fighting game called Skullgirls. This will be the first in a series of articles performing statistical analysis and generating potential solutions for pain points that appear.
Skullgirls is a fighting game with a really interesting development history. Launched in 2012, Skullgirls was developed by Hidden Variable Studios and Future Club (and formerly Lab Zero) and, the game has seen a recent revival with a new set of characters releasing with a staggered schedule. Hidden Variable has adopted a much more collaborative development cycle for Skullgirls than the average game, with an actively updated beta branch available to all users and a thread where players can give feedback and argue for or against changes. The pros seem obvious: unfiltered player feedback gives a timely response to changes that can be quickly acted upon. Unfortunately there are significant downsides too, as it can get quite heated in these threads. The first thing that this impacts is that you're only going to get feedback from the small group of people that are willing to tolerate potentially very hostile criticism to argue their points, which creates a sampling bias which skews the data. Additionally it can create a hostile environment for the community, with people belittling each other over disagreements. Because of this I believe surveys and statistical analysis have a very important role in the development of these kinds of games as they carry less social pressure and are much less likely to cause selection bias than open discussion.
This project has three goals for me:
Conduct qualitative and quantitative analysis of the data to add to my portfolio.
Show people how user experience research like this is conducted.
Create deliverables to entertain and inform development (if the developers choose to use it as such, I am not associated or funded by the developers/publishers).
I created a survey that would target demographical and emotional measurements, with some qualitative long form answers thrown in. I intended for it to take roughly 15 minutes, though I can't see those sorts of statistics in google forms. I think the easiest way to explain my logic behind the creation of each question is to go through them individually. I'm also in the awkward position of having my own opinions and feelings about what should be changed about the game; because of this I'll use my survey answers as an example so I can disclose my own biases. My goal in this is not to manipulate people into agreeing with me (a risk with statistics), but to uncover meaningful and actionable insights.
It's important to start off any sort of survey or test with a disclosure on how the data is going to be used. I would have liked to share the raw data with the community, however given the small size it seemed feasible that someone could go in and identify players based on their teams and then make fun of them for their opinions. I felt a higher level of confidentiality would allow for the data to be less skewed by selection bias.
Demographics like this can be a touchy subject, so I decided to give everyone the option to not answer any of them. That being said, it was vital that I know whether I am getting feedback from anyone that has visual issues; they're some of the most important people to consider in usability. This will be particularly useful in correlating with the ease of tracking measurements.
More demographics. Skullgirls fans like the game for different reasons, and I wanted to see how strongly people prefer specific aspects over others. I'm looking forward to running a correlation with years played specifically. This is very likely going to be skewed towards gameplay as the survey was targeted at gameplay oriented discords.
Knowing who plays what characters is important because it's going to effect so many aspects of perception. It may be easier to track a resource that belongs to you rather than the opponent, and different characters may be more frustrating for you.
New players often ask who an easy character to start with is. I want to see which ones are perceived as easy, it could be good to have that as a resource. I'd also like to see if ease of use is tied to frustration; if a character is perceived as frustrating because it's scoring "easy" wins, that's a different problem than a character being overpowered. I should mention that not everyone used the no opinion option even when they don't have experience with the character. To the people that participated: there are no wrong ways to fill out a survey like this. Once launched I can't really clarify anything because that would mean I'm not applying the test evenly to all participants.
This is question that everyone is going to care about. One of my assumptions about Future Club's approach to design is that fun is as important as balance. I'm also assuming that every player's experience matters; the game isn't exclusively designed for a single section of the player base. I'm not so interested in learning how strong a person thinks a character is: that's a question that's much better answered using a case study of subject matter experts (perhaps I'll be able to convince some top players to help me with that at some point). For right now I'm looking at emotional impact, which is why the question is phrased the way it is.
I wanted people to have the opportunity to give more detailed answers to what's frustrating. What I'll be looking for when I process this data is repeating points of contention and similar phrases. One of the interesting problems is that there's a fine line between something being fun and important to a character's appeal and it being unfun enough to take away for the sake everybody else. There's also a perception aspect. There's a anecdote that sticks with me about balancing an "overpowered" smg in Wolfenstein: Enemy Territory. Basically the gun was identical to another gun statistically but had a bassier sound effect, so people complained that it was overpowered. All they did to fix the problem is change the sound to have less bass and the complaints went away. "Players are hardly ever wrong about how they feel, but they're usually wrong about why that is" - Ed Stern (source)
To participants: these are my emotional responses to these characters. It isn't about what I think needs to happen to the game, so don't argue with me about it in the discord.
This is the most important question to me, as the dev team has expressed some desire to improve tracking issues with various systems. The limitations of the software has prevented them from implementing a particularly effective HUD for unique systems, which wasn't helped by the original devs dislike for "ugly" HUDs. Retrofitting this sort of thing sounds like it's going to be difficult so it would hypothetically be useful to know what should be prioritized. I'm particularly interested in how this effects newer players.
These four questions have been commonly brought up in the beta thread and are quite contentious. This should give an average closer to what people actually want, rather than only the loudest players being heard.
It's important to have an opportunity for more open ended questions in a survey like this. People like to be heard, and it's a more organized way of collecting this sort of data than the beta threads. What I'm going to be looking for is repeated phrases and themes to summarize.
I have witnessed some really harsh behavior across Skullgirls social media, so I think it's important to check the general health of the community. Ideally we wont see any strongly disagrees here. The final question could give a rough estimate of the percentage of people that would give feedback without the social pressures of an open forum.
Not much to say but I'll note that this is good UX: it's always important to give some sort of feedback that an action has been taken. Google clearly knew this when they programmed Forms. I also took the opportunity to try and market the study through word of mouth.
With all this preamble out of the way, next time I'll be able to dive into the statistics and talk about the interesting stuff. If you have any questions or comments (or typos, god forbid) feel free to contact me at nicholasdayhse@gmail.com. You can also add me on Linkedin. If you want to stay updated on this project submit your email here, and I'll add you to a mailing list.
The survey was conducted from 6/2/2021 to 6/11/21. It is independent and not associated with any developers/publishers of Skullgirls. It used a convenience sample and ended with 300 participants. It had no monetary incentive, and was distributed to the following discord servers including those that do not natively use English:
Thank you to the moderators of these discords for allowing me to distribute my survey.