Smartsheet is a software for collaboration and work management. It is used to assign tasks, track project progress, manage calendars, share documents, and manage other work, using a tabular user interface.

 

DATE

January 20th - March 10th, 2022 (8 weeks)

TEAM

Danielle Jo (UX Researcher), Joyce Feng (UX Researcher), Ji Su Park (UX Designer), Xenia Lin (UX Engineer)

OVERVIEW

Smartsheet redesigned its mobile platform experience. Besides gathering some feedback during early prototyping, the research team did not evaluate the usability of the renewed design. Our usability testing team collaborated with Smartsheet research team to investigate the usability of Smartsheet mobile's main navigation system to provide current insights and recommendations for future improvement. Our team conducted moderated usability testing sessions, observed the use of the Smartsheet mobile app, and allowed them to perform three different tasks.

As a UX researcher, I did heuristic evaluations, created screening questionnaires, drafted recruiting forms and testing scripts with tasks, facilitated usability studies, logged and analyzed the qualitative and quantitative data, identified the user journey and features to be refined, and synthesized in a presentation format to share with the stakeholders at Smartsheet.

Highlights

OBJECTIVE

Identify the barriers in the main mobile navigation system for improvement and discover the features that can potentially be eradicated.

TOOLS

Smartsheet mobile application, Zoom, Dovetail, G Suite, Figma

CHALLENGES

Narrowing down on participants profile based on how they used Smartsheet to decrease the variation between participants and findings.

RESEARCH INSIGHTS

  • The core feature ‘Search’ result is not providing enough information to look for the exact file.

  • The most heavily used feature is ‘Recents’ but it isn’t robust enough for various use cases and user context.

  • Most users did not use the ‘Workspaces’ navigation and was unaware of its existence.

RESEARCH IMPACT

  1. Presented the analysis and findings to Smartsheet mobile team including designers, researchers, and product managers.

  2. Smartsheet product team reprioritized the feature improvement plans and product roadmap by having the ‘Search’ as the primary feature to enhance.

 
 

Research question

“How do Smartsheet mobile users locate the sheet they are looking for on the mobile app?”

Subsequent research questions

  • How does Smartsheet mobile fit into users’ workflows - when, where, and why do they use it?

  • What does the user journey of file search look like, starting from locating the files and opening up the sheets?

  • How do users navigate and sort between more than one sheet and move around back and forth?

  • What are the types of files users usually create at work? What is the context of creating a new one?

 

Process

  1. Research planning

Heuristic evaluations

Using the heuristic evaluation, we discovered some of the critical usability issues of Smartsheet mobile application. We walked through the key tasks step-by-step and evaluated the system using Jakob Nielsen’s 10 Heuristics. After assessing the system, we rated the severity of the issues on a scale of three. We have compiled the results and created a list of problem areas to inform our research plan:

Key usability issues identified (shortened version)

  1. Key actions cannot be achieved on mobile: Unable to create workspaces and files.

  2. Poor feedback when creating a new file: Duplicated sheets don't show in Recents.

  3. Inconsistent content on the intended action: Confusing pop-up action words.

  4. Confusing search results: It does not follow a logical or hierarchical order.

  5. Desktop changes not reflected on mobile: Pinned items on the desktop won't show.

  6. Inconsistent filtering options: Available in “Favorites”, unavailable in “Recents”.

  7. Difficult to navigate sheets: Default grid view requires scrolling across columns.

Defining users using BI data

As there were no existing user personas at Smartsheet, the users were categorized into Power, Core, Casual, and Light users. These groups differ in their frequency of using Smartsheet with the data received from the Business Intelligence team at Smartsheet. We conducted our studies with all types of users as they might have different mental models or needs when using Smartsheet mobile application. We also wanted to get a balanced view to inform the design decisions.

  • Power user: active three days a week

  • Core user: active two days a week

  • Casual user: active one day a week

  • Light user: active for less than one day per week

Business goal

Increase the number of MAU (monthly active users) by 20% by making the navigation between multiple assets on Smartsheet mobile easier and satisfying.

Research goal

Investigate the usability of the Smartsheet mobile app navigation system for both power and non-power users to identify opportunities to improve the current design.

Goals

2. Participant recruiting

Participant profile

A total of 10+ participants were recruited for the research. We specifically recruited active mobile users in the past month so that the users were aware of their experiences and remembered how to navigate the app also to give feedback. Initially, we divided the participant profiles into power user and non-power(Core and Casual) user, thinking that their app usage experience would align with how often they use the Smartsheet mobile application. However, we found that there wasn’t much difference between the two profiles. Instead, it depended on the factors below.

  1. The number of active sheets users had on Smartsheet mobile application.

  2. The assigned role of the user was given on Smartsheet mobile application. (e.g. Sheets creator, sheets manager, or sheets user)

  3. The way of navigating the sheet is embedded in the users’ mental model.

 

Participant invitation and instructions

Participant invitation was sent out to recruit the users. It was sent out to users based on our participant profile we created above.

Once we recruited the participant, we sent out the instruction so that the participant was technically ready to participate remotely.

3. Usability testing

Moderated usability testing (remote)

We conduct 60-minute of 10+ usability testing sessions over Zoom. The research consisted of three parts:

  1. Pre-discussion of Smartsheet mobile app usage and the reflection on user’s personal experiences.

  2. Task performance: Observation of finding sheets, switching between sheets, and creating sheets.

  3. Post discussion on overall impression and filling out a SUS (System Usability Scale) survey.

Scenario-based key tasks

We prepared two hypothetical scenarios for the moderated usability testing sessions based on how participants used Smartsheet mobile in real life. For instance,

  • Using in the field (outdoor): you are outside or on the field to get your job done. How would you usually look for your sheets on the mobile app?

  • Using it in the office/home (indoor):  you are going from one meeting to another and want to check the status of the project you are working on. What actions would you take?


Questions to answer based on given tasks (below are a few examples)

Finding Sheets

  • Where do users start; what are their expectations, and what's the experience like?

  • What problems do they encounter, and how do they attempt to overcome them?

  • What prompted users to use ‘search’ instead of the navigation system?

Switching between sheets

  • What are users' workflows for switching between sheets, and what problems do they encounter?

  • Will ‘sorting’ be helpful for switching between multiple sheets, and why? 

  • How is the current navigation system make it easy or difficult for users to find a sheet?

Creating a new sheet

  • Can users successfully create a new sheet?

  • Where do they expect to see the newly created sheet, and could they find it?

  • What are the pros and cons of the current default views in each tab?

4. Data analyzation

Data synthesis

Using our notes from the research, we used the research tool Dovetail to transcribe the qualitative data. We created tags related to specific features and emotions of user experience to code the transcribed data for analysis.

We evaluated the severity of features and areas based on two metrics - the frequency of use, which is how frequently all the participants use this feature, and the impact scale, which is the problem areas that block participants from completing the tasks.

Logging the data using the online research tool helped our team collaborate efficiently, as the tool highlighted the sentiments and quantified the qualitative data.

* Screen shot of user research tool Dovetail

 

Findings & recommendations

Through multiple rounds of workshops, we concluded with four major findings from three different mobile application systems/features.

  1. Search is frequently used but does not meet most participants’ expectations because the search results are irrelevant.

  2. Search is not providing enough context for users to feel confident that they can identify what they are looking for. 

  3. Recents is generally well-received and is the most relied-on navigation feature on mobile, but it is not robust enough for various use cases.

  4. Workspaces is low in usage and is not considered useful by most users. There is a need to access workspaces that are most relevant to the users but cannot do so through the current default alphabetical order.

These key findings were explained further with details backed with user quotes, then recommendations were generated for what has to be done or come as the next step.

* Partially extracted from the findings and recommendations

5. Report sharing

Final presentation & outcome

  • We presented the research findings, analysis, and insights and delivered them with recommendations on how to enhance the user experience.

  • The outcome was shared with 11 Smartsheet mobile team members, received feedback, and discussed the next steps.

  • Smartsheet team reprioritized the product plans for UX enhancements by reprioritizing Search as the number one focus area.

 

What went well?

  • Efficient research coordination: Our recruitment process went smoothly with clearly set criteria for participants and Smartsheet team’s coordination in booking appointments.

  • Detailed technical preparation: While we conducted 10+ remote sessions, we did not encounter major technical issues as we prepared the users ahead meticulously.

  • Effective communication: Active engagement with our stakeholders allowed us to align the study goals and expectations well.

What would we change if we were to rerun it?

For our study, we had two major groups of participants - power and nonpower users. However, we found out that it does not provide enough context for usage. Narrowing profiles further down would help in coming up with focused findings. On the other hand, as we were unfamiliar with Smartsheet when we began the study, we had a limited understanding of its usage. Thoroughly learning the functionality and limitations would have helped us to shape the study design better.