Self-service onboarding research
Exploring concepts on areas of improvement
DATE
November 1st 2020 - January 29th, 2021 (3 months)
STAKEHOLDERS
Product management team, Dashboard platform engineering team, UX Manager, Head of Design
RESEARCH OVERVIEW
With the growing number of SMB customers, Sendbird has launched a self-service program for newly registering companies to onboard without assistance from the sales team nor solutions team to use Sendbird SDK & API. However, with a tight deadline and a fixed date to present it to the board of directors, the launched self-service program did not seem to reduce the number of support requests.
This concept research was done to examine the flow of the user installation process and uncover areas of improvement to ultimately improve the self-service onboarding experience. As a result, we were able to investigate the developer's emotions about the onboarding journey and extract operational UX concepts that can influence our brand awareness and expand user interest.
As a UX researcher, I collaborated with a UX research peer, a product manager, and a UX manager who worked on this project actively with me. Hosting virtual research involved numerous internal discussion sessions for a continuous update on the research process and settings. I have also been involved with the operational part of the research, including communicating with a third-party research agency and recruiting and scheduling remote research participants. In addition, I made the research available in both English and Korean for developers to pick and choose to read.
Highlights
OBJECTIVE
To scrutinize the self-served user installation process of Sendbird SDK and derive areas of improvements that can influence our self-service product roadmap.
METHODS
Generative research, survey, usability testing, remote user research, ethnographic interview, contextual inquiry, affinity diagraming and more.
CHALLENGES
Participant recruiting, broad spectrum of research in a limited time frame and resource, shallow wireframing of design recommendations.
RESEARCH INSIGHTS
Discovered high developer expectancy with self-service onboarding with Sendbird being a well-known startup.
Understood the sentiments of each participant in individual stages of the user journey.
Drew a total of eight operational concepts that needed product enhancement.
Found the highest priority which was to renew the Documentation page.
RESEARCH IMPACT
Referenced the insights actively to plan and prioritize 2021 project milestones.
Successfully presented to the stakeholders and executives by the UX team and PM team.
Led the initiation of the Documentation research and enhancement of the Documentation page.
Research question
“How are developers perceiving the current user experience of self-service onboarding?”
Subsequent research questions
What areas do we need to refine in the user experience of self-service onboarding?
Can developers successfully navigate to specific content during the self-service onboarding user journey?
What areas cause negative experiences for developers to drop out during the self-service onboarding?
How can we support developers in recovering from errors they encounter while navigating the self-serve process?
Process overview
Preliminary research
Churn analysis review
Churn analysis was done to identify fundamental causes of customer churn and provide actionable implications at a company-wide level. Out of 270 customers that left Sendbird between January 2018 and October 2020, the Product team and Sales team conducted in-depth interviews with 54 interviewees from 47 accounts, analyzing 41 interviews from 38 accounts.
The two identified key issues from this report were 1. Lack of value compared to pricing and 2. Difficulty in product onboarding. A total of 41 interviews were organized in a heatmap to show the frequency of threads. According to the results, there were 152 threads tagged to pricing-related churn reasons and 136 threads tagged to product-related churn reasons.
Why was churn analysis reviewed?
Onboarding at Sendbird meant both self-service and sales-driven. We looked carefully into the correlations between self-service and sales led to realizing the quantified effects and potential areas to consider for self-service onboarding research.
How was churn analysis used?
The findings from the churn analysis delivered that self-served customers tend to be SMBs with higher chances of churn. This helped our team scope down the user profile to those working in SMB. In the product category of qualitative review, the contributing factors to the onboarding were the initially perceived complexity of the SDK and the lack of samples in the documentation which gave us an insight that the ethnographic research must include the viewing of documentation.
We used the quantified data to realize the current state of self-service and its criticalities. Qualified data were used to know the background of the self-service program launch and specific areas of improvement for a smoother onboarding experience.
While the number of total accounts are similar between self-service and sales led, the percentage of churned accounts were 21.6% in self-service while sales led was 5.3%
The dollar amount accumulated from sales led was around five times higher, yet the percentage rate of self-served customers churning was almost 50%.
Self-service user survey
We developed a survey inquiring about the general experience of using Sendbird products. We specifically reached out to users who are currently our customers and can give us feedback on the SDK installation journey. We received a total of 103 responses to this survey. (Below are a few captured questions as a reference)
This survey helped us identify who and why users sought chat, voice, and video SDK in the first place and their general thoughts on the areas of improvement with self-served onboarding. The survey was used to generate questions for our user journey remote interviews and to realize areas of focus for ethnographic research.
User journey interview (New theory)
New Theory provides a service for companies to reduce churn through understanding the whys behind the customers’ thoughts. As we wanted to find out the sentiments of the overall user journey experience, our customers were chosen to be the interview participants.
In addition to having limited resources for the interviews, we also did not want to make our customers feel like they are talking to Sendbird directly, as we needed their blunt feedback. Therefore, we partnered with a third-party agency, New Theory, to host and open code the qualitative interview data.
As we categorized the journey into multiple steps, we used the data and identified the current sentiments towards using Sendbird SDK and unrecognized steps involved in the self-served process done by developers. Most of all, these interviews were used to recognize certain steps of integration we need to scrutinize for ethnographic research in advance.
2. Concept research
We prepared our concept research using the survey data and user journey interview insights as a foundation. The goal of this concept research was to define concepts that the product team can use to prioritize the product roadmap and define areas of improvement with self-service onboarding.
Ethnographic interviews
Research planning
We needed to take a closer look at the natural user behavior in a naturalistic setting and their thoughts behind it. For these reasons, we chose an ethnographic interview. Our goal for the interview was to discover the user behavior and why they behaved in a particular way during the self-service process, covering the end-to-end process from signing up on our dashboard to downloading and integrating our SDK.
A contextual inquiry was one of the methods used during the interview. We asked the users to perform the self-served SDK integration process and speak their minds out loud as they proceeded to gain a robust understanding of work practices and behaviors. As a researcher, we took an observational approach while the developer is performing the tasks and asked a few questions in between when we wanted to know the reasons for certain steps taken or behaviors.
Participant recruiting and tasks
Recruiting
To research a self-served user experience, we recruited developers who had never tried integrating Sendbird products. We also wanted to observe both junior and senior participants with the contents they look through and how they process when self-serving. Because this was our first proper UX research, the recruiting process involved a lot of developer networking. The developers at Sendbird assisted our team with reaching out to their own network. We’ve asked three questions for recruiting: 1) platform expertise, 2) years of experience as a developer, and 3) similar experiences in integrating SDK or API.
In the end, we were able to recruit four participants: one senior developer with 5 years of experience and three junior developers with 1-2 years of experience. Depending on their preference, we used emails and phone calls to arrange the date and time for the interview. We met with developers ahead of time to build rapport and communicate their needs. We also provided a guideline on how the research would be hosted remotely.
Task given
The concept research method was an ethnographic interview and contextual inquiry. Before joining the research, we introduced the participant to the research goal of this usability testing to observe the flow of a user signing up on our dashboard and integrating our SDK. The research methods required the users to read their minds aloud, point their mouse to what the participant is looking at as much as possible, answer the questions in between asked by the researcher, finish the entire process of integration and interview at last for further questions. We also provided them with a user story that would help them understand what to do more easily -
Developer scenario:
“You are a developer working at an IT Startup. The Product Manager of your company shares that the company needs to implement chat to the current platform, just like Facebook Messenger, as to their business strategy. The PM requested you to find out if Sendbird chat SDK is suitable for the company to use by testing out if two users can send messages and chat with each other.”
Mandatory questions for all participants:
What was your overall experience with the task?
What was the most challenging part of the process?
What part of the process did you feel was easy?
How was the experience of installing Sendbird chat SDK compared to other SDK installation experiences?
What will you do next to determine product suitability for the task?
The interview consisted of four people: one Product Manager, one UX Researcher, one UX Manager, and the participating developer. The entire interview took around one hour, 50 minutes of integrating the SDK, and 10 minutes of additional questions the observer/interviewer asked. Directly after the interview, we reflected on the experience with the participant by asking further questions - Was it difficult to keep speaking all the way through the task? Did you feel awkward? Did you stop talking when you got stuck?
Open coding and affinity diagram
After transcribing all of our interview data, we searched for the frequency and causations of the information gathered. The standards we used to organize our data were the patterns, similarities, and relationships of all the collected data. As the coding process happened iteratively, we built our theories ground up, taking an inductive approach.
In the end, each sticky was broken down by corresponding extracted concepts/themes. In addition to operational concepts, we were given feedback on product usability, primarily by asking additional questions.
Identified operational concepts
Identified product usability
Defining UX concepts
The open coding/affinity diagraming drew us to define UX concepts into eight categories. These drawn concepts were to share with our product team to define the areas of focus for the upcoming product roadmap -
Operational concepts: the areas we can improve to enhance our self-service process, giving developers a smoother experience.
Consequences: the improvements' potential results will bring when the operational concepts are enhanced.
3. Design recommendations
Concept-based design recommendations
Based on the eight operational concepts from open coding, we have created design recommendations to share with the product team. For each concept, the product manager and the UX team thoroughly discussed, shared ideas, and synthesized what enhancement could be made.
The reason for creating concept-based design recommendations was 1) A total of eight concept areas may overlap and sound confusing, 2) to show examples of the extracted concepts, 3) to visualize our research findings for better understanding.
** Disclaimer: Below displays partial of final design recommendations **
4. Outcome
Insights
1) The developer expectancy was comparably high with Sendbird being a well-known startup from graduating the Y Combinator.
2) Visually laying out the developers’ workflow of the self-service process helped us understand the sentiments of each participant in individual stages.
3) Drawing a total of eight operational concepts, we realized that every stage of the self-service experience needed an improvement, except the search function.
4) According to the user journey map, our improvement priority was the Documentation page, including the GitHub and development tools.
Once everything was synthesized, we created a Self-service onboarding user journey map to give an overview of the sentiments we found to the developers on one page. Putting the journey map in accordance with our website/page the participant was looking into, I identified emerging patterns, characterizing behaviors, needs, and major paint points at each touchpoint. It also became clear that we needed to prioritize focusing on our Documentation page.
Self-service onboarding user journey map at glance
Impact
1. The insights from this research were used to plan and prioritize 2021 project milestones.
2. The insights were successfully presented to the stakeholders and executives by the UX team and PM team.
3. The insights initiated the Documentation research and enhancement of the Documentation page.
Key takeaways
Participants may be hesitant or forget to speak aloud during the ethnographic research.
Ethnographic research always requires a guideline for both researchers and participants. The researcher must always look out if the participant is performing the given task and be prepared to give up heads up or remind the participant when necessary. For this research, with the given task being 50min long for developers to perform, some forgot to share their sentiments and what they were trying to do in words. Though it could be asked after the task is done, it would be much more helpful if the researcher could encourage the participant to speak as much as possible from the get-go.
Where are the priorities? Were ‘Design recommendations’ necessary for this research?
As generative research and the very first research done by our team at Sendbird, the spectrum of this research was quite broad. Open coding and affinity diagraming took a lot of time as we had an overflowing amount of data to organize and were finding difficulties deriving our concepts. If I were to do this research again, I would try to narrow down the focus areas and quantify the number of errors or negative sentiments made during the process to prioritize our needs more efficiently. In addition, the design recommendations were generated mainly from the product management and UX teams without further research, referencing what we already know and other companies. I would separate the design recommendation as additional research and analyze it using multiple methods like information architecture, competition analysis, etc.
Any further quantitative research data for validation?
We referenced the quantitative data in the preliminary research stage: the survey and the churn analysis. However, the ethnographic research was not explicitly validated after extracting the concepts. If I had more time to stick to this research, I would have followed up using Heap or Pendo to validate the needs and prioritize the concept areas.
Always think about who the users are! (Would have been wonderful to create user personas)
Since we are navigating through who’s at the front door of the self-service onboarding process, involving stakeholders to find out who the most visitors are to our dashboard would have helped gather a precise user journey by creating user personas. For this research, we defined the users broadly.
The primary user of our documentation is a software engineer/platform developer. As a researcher, I needed to familiarize the API and SDK trends among the engineers and the terminologies they use when looking at the documentation, in other words, ‘the developer language’. It would be super helpful if I knew how engineers grow as professionals to define what we need to provide them according to their development level.