-
Notifications
You must be signed in to change notification settings - Fork 7
Validation ‐ User Testing of Alien Fauna Features and Overall Gameplay
User testing is an important validation activity in game development that can be used to inform design decisions and improve the overall quality of the game. This page documents the user testing conducted during Sprint 4 by team 4 to validate the Gardener's of the Galaxy game. The primary goal of the user testing was to leverage user feedback to identify the changes that need to be made regarding design, game mechanics and usability in order to make the game as fun as possible for players. The testing focused on the alien fauna features and the overall gameplay.
The testing consisted of two main activities that the users had to complete - a think-aloud activity and a System Usability Scale (SUS) style activity. The think-aloud was conducted to test the general gameplay, and the SUS was conducted to test the alien fauna features. This division of the tests was necessary as the game was too long for users to play through sequentially and still interact with all the alien fauna features. It should be noted that the SUS activity only adapted the format of SUS and did not use the standard SUS questions. This was done to make the users' responses more specific to the game, and therefore more useful for validation of the game. The think-aloud was performed before the SUS in order to preserve the user's first impressions for the think-aloud. This was done as a think-aloud is more effective at capturing first-impressions due to it being live and less restrictive than the SUS.
The think-aloud activity involved users playing the game uninterrupted, from the beginning, and vocalising their thoughts to the testing invigilators. The invigilators noted down meaningful comments made by the users and the context in which said comments were made. Each think-aloud lasted for five minutes. A think-aloud was chosen as it provides direct access to the thought process of users, which is critical in designing intuitive gameplay. Moreover, as games are driven by emotion and fun, qualitative data about the user's thoughts is key for maximising user enjoyment. It was also noted that a more rigid testing method (e.g., a questionnaire) may restrict the quantity and kind of feedback that users can provide, which is undesirable for a first impression test.
To test the alien fauna features, users were required to perform a list of in-game tasks including feeding, taming, and fighting hostiles. To be comprehensive, all key animal fauna interaction was covered in these tasks. After completing each task, users completed a SUS style response sheet. The statements constituting the SUS all related to the tasks the user just completed and were designed to gather insight into the tasks enjoyability, usability and intuitiveness. The statements also focused on areas of the alien fauna features that team 4 considered players may be very sensitive to e.g., combat difficulty. This activity was designed to provide more targeted feedback than the think-aloud by directing players to the specific tasks alien fauna tasks and having them respond to statements about enjoyability, usability and design. A more specific test than the think-aloud was performed for the alien fauna features, as team 4 developed this code and could therefore make the technical improvements corresponding to the specific feedback. Additionally, as there were many tasks to complete, a SUS was the quickest way to get through them while still providing valuable insights.
The user testing was performed by four users who had no experience with the game and who were all within the age range of 20 to 22 years old. Two of the users were male and two were female. The users were all experienced with technology and games. These users were chosen as to be as representative of the game's expected demographic as possible. As the game is still in very early development, the feedback of the expected demographic was prioritised as to ensure the game can at least appeal to its most likely customers.
The specific tasks that the users had to complete and the SUS template that they had to fill out afterward can be found here. The template used to record the think-aloud can be found here.
The results from the think-aloud are summarised below. Similar comments are grouped together.
| Feedback | Context / Stage of Game | Evidence | Insight |
|---|---|---|---|
| 1. Movement speed is too slow for map size | Very start of game when users are exploring the map | "Wish the sprint was faster", "Finding the animals takes ages" | The player needs a faster way to move around the map to prevent them losing interest travelling long distances |
| 2. There is not much going on to keep players engaged | After completing first few quests | "This is just a bunch of fetch quests" | More activity to be added to the game to keep the player engaged. This should not just be fetch quests though |
| 3. Controls are intuitive to use | Players were comfortable walking, accessing the inventory, and using items | "i is for inventory, makes sense", "This is kinda like Minecraft in a way" - referring to inventory | The controls do not need to be changed and future controls should follow the current scheme |
| 4. The UI is overly cluttered | Throughout the game | "All the different bars are a bit much", "It's pretty cluttered already" | A minimalist redesign of the UI is necessary to make it more user-friendly |
| 5. The colour palette is too dark and may not be aesthetically pleasing | Throughout the game | "Everything looks muddy", "The colours could match better" | A visual redesign of the game may be necessary. A change of colour palette should be considered. |
| 6. The bats (hostile mob) do too much damage | After hostiles start spawning | "The bats are way too strong" | Rebalance bat damage and consider rebalancing all hostile's stats |
| 7. Much of the animal drops do not have a purpose | Feeding drops and collecting passive drops from animals | "How is drinking milk different to just eating beef" | More functionality needs to be added to the items to make them stand out and justify their presence in the game |
The averaged SUS results for each task are listed below.
- Find cow food and feed a cow
- It was easy to obtain cow food - Agree
- It was difficult to feed a cow - Strongly disagree
- Tame a chicken
- The chickens are too fast to catch - Disagree
- It was easy to understand taming and identify when an animal was tamed - Strongly Agree
- It was difficult to tame a chicken - Agree
- Kill one of each hostile
- It was difficult to differentiate hostiles from passive fauna - Agree
- The combat was dynamic and fun - Disagree
- The game would benefit from higher hostile spawn rates - Strongly Agree
- Eat an animal
- Obtaining food was intuitive - Agree
- It was easy to find an animal that dropped food - Disagree
- The food made sense relative to the hunger system - Strongly Agree
The results from think-aloud and SUS validation activities revealed aspects of the game that users responded well to and aspects of the game that decreased their enjoyment and made the game less user-friendly. In terms of what users liked, the think-aloud validated the control scheme as users found it easy to use. Additionally, the SUS activity validated a variety of the alien fauna features including feeding animals, getting food from animals and taming animals as users responded well to them. It was observed that all the game's similar behaviour to Minecraft (controls, animal interactions) was positively received by users. This was likely due to players already having an intuition for Minecraft thus, they could leverage that to quickly pick up the Gardeners of the Galaxy. From this it is recommended that future features utilise similar mechanics to Minecraft in order to capitalise on player's existing mental models and make the game as intuitive as possible.
The user testing highlighted many aspects of the game that users thought needed improvement. Most of these problems team 4 directly addressed in sprint 4 with new features; however, some problems could not directly be addressed by team 4 as the necessary changes to address the problems would need the entire studio's cooperation and approval from the design committee. These problems include users finding the user interface to be too cluttered and the colour palette of the game to be too dark and muddy. These issues are of critical importance as the visual design of the game is vital in attracting players and ensuring the game is user-friendly. Team 4 raised these issues with the studio but could not pursue further action independently. The next section deals with the new code that that team 4 implemented to address the user testing feedback. Small changes like making bats weaker (to make fighting them more balanced) will not be discussed.
In response to the feedback from both the think-aloud and the SUS, team 4 implemented a new feature in sprint 4, the buff system. The buff system involved creating certain items which when consumed, provided unique buffs for the player. This system was designed to address a multitude of issues highlighted in the user testing. More specifically, the think-aloud revealed that users felt that the map was too big or that the player needed a faster way to traverse the map. Consequently, a buff to increase player speed was devised as to prevent players from getting bored when travelling long distances. Additionally, the think-aloud also highlighted that much of the fauna drops did not have any unique behaviour. Hence, the buff system utilised the item drops from passive fauna once tamed - the items that previously had no use. Finally, the SUS indicated that users did not find combat to be dynamic or fun. From this, buffs related to combat were designed. With this feedback, two buffs were added to existing items for the buff system. Golden eggs link now increase the players speed - to decrease travel time and make fighting fast hostiles more fun, and drinking milk now adds a strength buff - to make combat more dynamic.
The spawning behaviour of both passive and hostile alien fauna was changed as a result of the think-aloud and SUS activities. The think-aloud indicated that players needed more different, engaging activities to break up the quest gameplay per the feedback "you need more stuff going on to break up the back-to-back quests". Additionally, all users responded with a Strongly Agree to the statement "The game would benefit from higher hostile spawn rates". These two points of feedback indicated that the game would benefit from increasing the hostile spawn rates as to break up the quests and keep the player engaged. Hostiles would previously spawn in discrete bunches, with large gaps between when they would spawn. Additionally, the initial spawn count was small, and the spawn growth rate was small, so it would take a long time for a large amount of hostiles to spawn. This was changed to have hostiles spawn more frequently but in smaller bunches as to not overwhelm the player and become annoying. Additionally, the growth rate was changed to be higher, but the maximum spawn count was decreased.
The passive alien fauna spawning behaviour was also updated. The SUS revealed that players found it hard to find passive animals as the average response to "It was easy to find an animal that dropped food" was Disagree. The cows and chickens did not have a very high initial spawn count, and due to the size of the map, it was difficult for players to easily find them. It was assumed that the issue was compounded by the slow movement speed per the think-aloud. To address this, the initial spawn counts of the cows and chickens were increased. The spawn growth rates and max spawn counts were kept the same as to not flood the game with too much passive fauna.
The validation documented on this page provided insight into how the target demographic may view the game and resulted in changes to better align with what a typical user may want in a game. However, the scope of the results is limited by the small sample size of users (four) and the limited scope of the tests conducted. It is recommended that more validation be conducted to develop a deeper understanding of the user's perspective before the game is released. To increase the scope results, it is recommended that future validations involve a larger sample size of users, have players complete the game during the think-aloud, perform the SUS evaluation on more features than just the alien fauna, and include additional user testing activities. These recommendations should result in more a comprehensive view of the user's perspective and should result in more reliable, actionable data.