Setting up an internal research group
DATE
July 2nd - August 24th, 2021 (7 weeks)
STAKEHOLDERS
Head of Design, UX Manager, Marketing Manager, Marketing Team, Visual Design Team, Trust & Safety Team, Customer Success Manager, Product Managers
BACKGROUND
Discipline was still new around UX research at Sendbird. Alongside performing research studies, I often had to find participants in my own creative and scrappy ways, using the network of Sendbird employees being the most commonly used method. I wanted to improve this before moving on to studying master’s degree in HCI. I implemented a new tool called Qualtrics, designed the survey, created the service blueprint and touchpoints, collaborated with designers on creating a website for participants’ opt-in, and in the end, created Sendbird Research Group.
Highlights
OBJECTIVE
Build an internal panel of research participants, a database of users who can opt-in for future Sendbird research studies, and product development to increase research efficiency and save time in finding the right participants.
CHALLENGES
Limited budget and resources, collaborating with various different teams and stakeholders, including Marketing, CX, Trust & Safety, Product Managers, and Designers on the alignment of the tool usage.
OUTCOME
Announced the launch of Sendbird Research Group in a global standup meeting, recruited a total of 68 participants on the first round of recruiting outbound email.
Why set up a research group?
When I started working as a UX Researcher at Sendbird, I worked as a sole researcher, mainly alongside the Product Managers. The budgeting of research projects was still not in place, but the need for research and understanding our users became more critical as we needed insights to improve and define our product directions and priorities.
As researchers, we know how valued our participants are. They have an incredible impact on product decisions. To remove the bottleneck of the slow and manual recruitment process, we decided to build an internal research group - Sendbird Research Group - internally to ease the process of hiring research participants. We also wanted to align the methods of reaching out to our potential participants to find the right users at the right time.
Panel building overview
The process of panel building does not necessarily go in a sequence, as shown above. Instead, the process was intertwined as it involved different stakeholders and collaboration with multiple other teams to finally build the Research Group. Each process consisted of individual checklists for development and launch. In total, it took seven weeks to get our ‘MVP’ off the ground.
Ideation
Upon agreement of participant recruiting being a blocker of research, the ideation phase was mapped and shared with Sendbird’s Head of Design, UX Manager and Head of Marketing to kick off with the Research Group creation. Marketing team was involved as we needed their assistance with the announcement and publicizing.
The goal was to build an internal panel of participants: a database of our users we could reach out to for research studies. The ideation phase involved designing the journey of creating a research group, including what is needed, and how the process could be accomplished. We divided the steps into three phases and laid out the tasks, tools involved and information needed for each phase.
Service blueprint & Touch points
In addition to an overview of ideation, I mapped out the expected user journey of joining the Research Group to identify what tools and processes are specifically involved. I outlined all of the people, touchpoints, and systems I would need to work with.
Behind the journey, we had the people (Sendbird employees in charge) and the system (tools). For internal team members, seeing the status of potential user requests was critical, along with simple back-and-forth communication with the research team. On the flip side, the participant panel had three different types of potential user profiles:
Users who are aware of Sendbird
Users who are on a free tier of Sendbird
Users who are paying customers of Sendbird
Each user required different kinds of communication through various mediums. For example, users of Sendbird could relatively be familiar with what exactly they are signing up for. In contrast, users who merely have an awareness of our company could need more information.
In addition to the bluepoint, I’ve also tried to align the outbound emails sent by different teams to find out in which stages of email we can inquire users with signing up for our Research Group.
The blueprint proved to be an invaluable artifact to visualize the complexity of the systems I’d need to set up end-to-end. It also helped me take stock of all the tasks ahead of me and identify the essentials.
Checklist listing & Technical tools validation
As a roadmap to share with stakeholders, I created a checklist depending on its phases of launch and the go-to person for it to be checked off. Tooling usability matters as much as functionality. The listing of technical tools also mapped out the technical complexity to get the panel up and running.
First, I used a combination of Marketo, Calendly, Reachdesk, Pendo, which we were already in use, and newly implemented Qualtrics. Qualtrics was our primary tool: applied for vendor risk assessment, they were GDPR-compliant and was already known for survey functionality.
Design: survey and email
Survey
Simultaneously with the tool exploration, I started designing the survey questions and emails for opt-in. I spent a lot of time thinking through what information we should collect from participants to sign up for the panel. As the goal of building a participant panel was to opt them into our research group, I designed around ten questions, including the demographic questions. If the interested participant was a developer, I asked a few more follow-up questions regarding the platform expertise and years of experience.
The survey questions were designed based on the user profiles we already had in our database, the participant profiles we needed for previous researches, and the participant profiles of the future research areas we wanted to explore. Each question was elaborated with the purpose and proofread multiple times with Product Managers.
I crafted a total of three emails. One would be sent out as a welcome email when a participant signs up and the other two as outbound emails to recruit participants. The outbound invitation emails included key messages of how the Research Group works, joining process, benefits, etc.
I wanted to communicate clearly to users: making the panel feel unique, exciting and worth joining. Collaborating with the marketing team, we defined our voice & tone for the first batch of the outbound email sent to our free tier customers. As there were already a ton of emails being sent out to our paying customers at the time, we scrutinized the timing to send the emails out to the customers.
Email wireframe
I drew the email flow to explain to the Marketing team and the Visual team what’s to be done. I also wanted a consistent visual look for the panel. This branding would go on our emails, our screeners, and more. The panel needed to be associated with Sendbird, so participants knew it was legit. But I also wanted it to look differentiated enough that it didn’t feel like Marketing.
Below is how I received help from the Visual team to define our look. We collaborated through several iterations of ideas, eventually choosing one that took the Sendbird colors and riffed on the concept of research and insights, incorporating the research group logo and illustrations.
While Qualtrics worked fine for gathering and storing participant data, its messaging capabilities were relatively poor. I was editing HTML code to make email messages look somewhat decent. We were initially thinking of using Marketo for outbound activities as it already had all our user databases. Still, we had to consider using it for welcome messages in the end because of the limitations with Qualtrics.
Launch
A successful UX researcher’s job is to let the entire company know of the values we bring and announce what we have done! After the first MVP launch, I announced at the global standup meeting. For the first phase, we recruited 68 participants in total on the first round of outbound email.
Company-wide announcement
Sendbird Research Opt-in emails
Post-launch monitoring
Furthermore, we continued to track the data and the list of users who have opted in. The continuous data cleansing was necessary as quite a few of them initiated the survey and did not finish answering all the questions. Building a Research Group was the final project I worked on as a UX researcher. I noted down on the final message that Sendbird needs to constantly look after it and help the Sendbird Research Group grow. By looking at quarterly data like how many users are being reached out to, response rate, how frequently new customers are joining, etc., the company needs to ensure the panel stays healthy.
Email data tracking: how many emails were successfully sent? how are the open rates looking? how many clicked to view our Sendbird Research Group?
User data tracking: who are the users that signed up? how many of them have a background in development? have they been in touch with any Sendbird employees before?
Qualtrics distribution tracking: which platforms did the sign-up come from? have they finished answering the survey? how many stopped answering questions in between?
Tool: Marketo
Tool: Qualtrics
Final thoughts and reflections
The differentiation between the panel sign up the survey and a screener survey
As much as I spent a lot of time thinking through what information we should collect from participants as they signed up to the panel, we had to iterate multiple times and discuss what information we needed. I tried envisioning future questions, consulted with other teams on what they were curious about. I knew that we could never wholly rely on the information in the survey database as user situations change from the time they initially signed up. For instance, the participant could no longer work at the same company and use their email anymore. So as I jotted the questions down, I have divided the list into panel sign-up questions and screener questions - with the latter being a much more accurate and up-to-date source of participant information. It is a must to spend time creating the opt-in form, but the screeners will matter most once the panel begins to be used.
Keeping an MVP mindset and the willingness to iterate
Multiple different teams were involved during the process of creating Sendbird Research Group. Startups are always looking for tools to accelerate the work, and Sendbird was also one of them. As Sendbird already had many third-party tools in use, the alignment of the existing tools was necessary before implementing anything new. When Qualtrics was first chosen, I had to discuss with our Head of Design and UX Manager to validate if this is the tool we certainly need and thoroughly used the free account for one month for validation. In addition, as there were already a lot of tools in use, we needed to try to find the tools first internally among the ones that are in use, then move on to consider new tools. Besides Qualtrics, I communicated with other teams and discussed how those existing tools could be used with the UX research team.
Continuing to create a steady stream of new sign-ups
Panels would only be helpful with the number of active participants. Though I got a head start with 68 sign-ups, I had multiple other methods laid out in the ideation phase to expand the opt-in streams. To have a steady stream of new participants coming in, Sendbird could regularly engage by posting on Sendbird Community, setting up a separate section in a booth at tech conferences, getting involved in in-person meetups, encouraging the already signed up participants to refer their acquaintances, and having pop-ups surveys on our dashboard. The list of participant panels must be looked after, at least quarterly, to identify the data of Sendbird Research Group status: how many users are being reached out to, response rate, how frequently new users are joining, etc., to make sure the panel is staying healthy. Of course, this expansion needs to grow at the same pace as the growing number of researchers who can manage the data to prevent upsetting or fallout participants.
Keep in mind!
The participant panel won’t work for everyone. A participant panel helps if your product already has somewhat engaged users (typically works well for companies with a large, existing user base)
The participant panel won’t reach everyone. A panel works well when trying to find more general product users, but not so well for niche participants.
The participant panel won’t run itself. It’s not set it and forget it — there needs a process for who’s able to use the panel, how, and how frequently. Also need to think through budgets.