Legend of Swordman

 
CASES STUDY

A study of quantitative research and data analysis for the new expansion pack to improve the game experience,  DAU (daily active user) and ARPU (Average Revenue Per User).

A study of an Alpha Testing phase game to understand target players needs, the core gameplay experience and user acceptance of the overall art style.

 
BRIEF - The Legend of Swordman

Legend of Swordman is a Chinese MMORPG mobile game, which developed by Season Games and operating by Tencent. The game already operated one and a half years since September 2016, since its launched in June, it has been in the top 5 of the iOS Top Grossing App in China App Store and has reached 500 million RMB monthly revenue. 

 

At the time that I jumped in, the data showed MAU and Monthly Revenue are dropping from 2.21 million to 1.47 million, 344 million to 270 million respectively. And based on last version's research, the developer team added a lot of new contents to improve the performance and try to boost MAU and ARPU. I lead the user research to understand different group of players satisfaction and what they feel dissatisfied with the new expansion pack to guide the following update.

APPROACH
In-Game Survey

Because the project goal is understanding overall user satisfaction and finding a guide for next version, so the big sample data and statistics help us turn quantitative data into useful information to help with decision making. We use In-Game Survey to collect our data and guarantee the sample return.

As a project lead, I need to be responsible for the entire process. The specific contents include survey design, editing, and placement; data cleaning, and analysis. Finally, output the project report to present to stakeholders and discuss the following work.

Survey Design

Structure is the key of a successful Survey. Main elements to organize a questionnaire are:

 

  • Required information

  • Target respondents

  • Question content

  • Question wording

  • Screen questions

Questionnaire Design Concept Map

I used Microsoft Excel edited and designed the whole questionnaire, before uploading the questionnaire to the game.

Questionnaire Content

The red circle is In-Game Survey

Data Cleaning

While these huge sources of participants help fill large sample studies quickly, there’s a major drawback: poor quality respondents.  Reliable and valid responses only come when the data is properly cleaned.

In our experience, around 10% of responses (often ranging from 3% to 15%) “cheat” in surveys and unmoderated UX studies, and need to be tossed out or “cleaned.”

 

These responses are usually a combination of

 

  • Cheaters 

  • Speeders

  • Respondents misrepresenting themselves

  • Participants just not putting forth effort. 

 

All of which threatens the validity of the studies’ findings.

Data Cleaning Rules

We placed 3 million surveys on the QQ platform and we got 7 thousand samples after 2 days collection. To make sure data have high validity and reliability, a cleaning rule is necessary for analysis. Normally, we use SPSS to help us clean and analyze surveys data. 

Cleaned Data

Data Exploration

Based on designed core headers, I analyzed correlation, linear regression, validity, and reliability through valid data to get information that we need. 

  • User satisfaction for new version

  • Interestingness 

  • Different types users pain points 

  • Expectation

  • All new features feedback

  • Conversion rate

 

In order to facilitate communication with developer and operation teams. Data visualization is necessary, through Excel and Tableau we quick, and easy to convey concepts that 

 

  • Identify areas that need attention or improvement.

  • Clarify which factors influence users behavior.

  • Help us understand which products to place where.

  • Guide following update.

Analysis from Report

In this case, the document-style report is that it leaves a comprehensive record of the research and satisfies an audience that likes a lot of detail. So a designed report is important.

RESULT
Core Finding

1. Overall, after the research, we got that there was a significant decrease (10%) in user satisfaction (43%)      and playing willingness (65%) compared to open beta and the last expansion pack.

 

2. Specifically, the new content did not affect the user's game interest a lot, among which the users who 

    experienced the "Clothing Store" increased their interest rate by more than the other content (24%).

 

3. The new expansion is compared with the previous version, "Update too slow" and "Boring contents" are 

    more obvious, the main reason is that new contents satisfaction is not high; for "Too many In-Game

    purchases", and the "Game takes a long time" dissatisfaction has not changed.

Core Advice
  • Maintain regular content updates and increase the frequency.

  • Balance the gameplay and the population of camps, boost the variety of schools.

  • New Home building system and Yin-Yang system have deepened the player's In-Game purchases pressure and suggested that they should be cost-effective and add more fun features.

Core Conclusion from the Research

IMPACT

After the research, the developer team has been quickly adjusted due to the findings of the unbalanced camps and the problems of the new features. Moreover, the update frequency of the new content is accelerated.

 

According to the data from the operation team (Two weeks after the research):

 

The 1-day retention rate boosted (11%), achieved (40.41%). 

The 1- week retention rate boosted (3%), achieved (16.43%).

 

There also was an unexpected discovery in this research. That's the interesting relationship between high-paying players and low-paying players.

 

We noticed the point of ARPU boosting is that low-paying players and non-paying players retention rate is the key element for keeping high-paying players. This phenomenon is like the food chain, "non-paying players accompany" are dynamic of the pay.

 

So I suggested lowered the In-Game purchases power and increased some friendly features for non-paying players, the ARPU boosted from 7.38% to 9.45%. That is an amazing jump for a million users level product.

MY TEAM

MY ROLE

DATE

METHODS

PRACTICE

Solo

User Researcher

Jun 2017

Diary Studies

Desirability Testing

Survey Design

Tasks Design

Survey Design

Communication Design

 
BRIEF - Legend of Swordman II : Sword Song

Legend of Swordman 2: Sword Song is an MMORPG mobile game, which is the first product of Legend of Swordman series developed by "Unreal 4" engine. We all know the graphic is the basis for the expression of a game. So the Legend of Swordman 2: Sword Song striving to create an aesthetic traditional Chinese Wuxia arts world. When I take this product, which is in Alpha Testing phase.

 

The goal is to understand target players needs, the core gameplay experience and user acceptance of the overall art style to guide next phase development and testing.

PROJECT NEED

There are three main goals of this project:

  • Clarify players perception on MMORPG mobile game, pain points, and key elements of retention.

  • Understand player's experience and attitudes of the core gameplay content, art style, UI design, beginner guidance, and gameplay balance.

  • Study interestingness, dynamics of long-term retention and strategy of the game. 

Because the game is still in Closed Beta Testing phases, due to NDA, I’m not able to share the project all details in public. I will share some approaches and selected contents. If you are interested in my experience and works, just shoot me an email and let's talk more.

APPROACH
Diary studies

For a game which needs launch as soon as possible, the limited time is the biggest challenge. In this case, one-week diary study is the perfect remote method used to collect qualitative data about user behaviors, activities, and experiences in the game over time. 

 

As a project lead, I need to be responsible for the entire process. The specific contents include planning and preparation, participants screening, editing diary framework, operating activities, post-study interview and data analysis. Finally, output the project report to present to stakeholders and discuss the following work.

Two hours gaming experience testing everyday

Timeline of activities that take place throughout the diary study

Project Plan

In my plan, the whole project will take about a month to complete. Because at the same time, I have two more research projects, so my time is not enough to complete all the research content of this project, then I must have a collaboration with the contractor, in order to ensure the successful completion of the task in the expected time. From the plan below you can see what my responsibilities are.

Recruiting and screening

Appropriate participants for the research studies are important that in order to get the actionable insights after. To reduce the cost, save time, and make data analysis easier and limit bias. I discovered target players with development and operation team and decided 4 testing groups, which have 10 participants per group. So we plan to recruit 40 participants in total. We concerned about games historycharacter level, requirements of related games, population information, and game device information. These experience and information could help us distinguish the difference of character between groups of target players and easy to measure the reliability during the rounds of screening.

Recruitment conditions

Recruitment process

Testing Content

In this research, our main focus areas like the fishbone diagram, the goal is to try to find each component effect and evaluation, then define problems and improve them in next testing. During the week, participants will spend 2 hours average to experience the game, and finish tasks. Every day has different multiple tasks to participants, they will finish a survey and gaming experience testing. As the researcher, I will collect all data each day, analyze and organize them. 

CHALLENGE and KEY TAKEAWAYS

In this project, the biggest challenge I faced was not the recruitment design, nor the diary design, but it is an unexpected challenge that the maintenance and operation of the players during the one-week gaming experience testing.

 

It is not easy for each participant to complete the diary study on a regular basis. In the process of testing, some participants drop out of the testing, and even some players have contradiction and conflict in the game.

 

These were unexpected emergencies, and for the time I was at a loss. Fortunately, my mentor told me some methods to maintain the player group and gave me encouragement and confidence. I as a player role to communicate with participants actively, and using empathy to help the difficulties they encountered in the test. Also, as a pro researcher, during the situation, catch the chance to analyze the reasons of churn, to distinguish the loss factor, for details about in-game factors, and record the key events on summary and analysis, as the learning experience for next similar testing and recruitment.

MY TEAM

Solo

MY ROLE

User Researcher

DATE

May 2017 -

Jun 2017

METHODS

Statistic

Survey

Quantitative Research

PRACTICE

Data Analysis

Report Writing Skills

Survey Design

Communication with Stakeholders