one company under many assumptions

The Problem

Code School was a platform built for developers made by developers which made assumptions about the user base rabid and hard to debunk.

For me, it started with a session with the marketing director that pointed to a regular monthly drop-off rate of users. As an education platform, drop-off shouldn't happen until a user is adequately educated or most likely, longer than a month. With marketing unable to explain the drop and the new courses & features built only by sense of smell, I wanted to get us real data & direction. We needed personas based on real evidence.

Team & Methodologies

Jean Kaluza

UX Research

Tommy

UX Intern

Chris

Data Analyst

Qualitative Interviews

Scripts designed against quantitative surveys to answer the 'why' as well as configure motivations against insights.

Quantitative Surveys

A 50 question survey was sent out once at the beginning of the project and once towards the mid-end for validation.

Company-wide Ideation

After data-collection, I developed a fail-safe way to involve the whole company to compile data into personas they could identify with.

Research

How we did it...

Survey Design

I started by capturing all assumptions across the company. This also built trust from coworkers to have their steadfast views heard and addressed. I then worked with both my intern and data analyst to turn assumption and question alike into survey questions. We wanted demographics, motivations and all our questions turned into something that could yield us some good quantitative data.

basic demographics

We wanted the ability to cross-segment against education level, occupation, sex, age, number of children, location and even financial standing

skill level

One of the main assumptions was that our users were all beginners. Since Pluralsight also acquired Smarterer, we worked with their product team to create a mini skill assessment that could generate concrete data for us.

Silly Users

What was funny is we also asked how skilled the survey taker thought they were. Most participants that answered 'advanced' only answered 0-1 correct and those that answered 'beginner' actually answered all three correct. Users weren't shown how they did on the skills test.

Rough Ranking

If a user was correct, a harder question was asked. If they were incorrect, the skills assessment was over and user proceeded through rest of the survey. The higher skilled users got 3 questions correct, lower skilled answered 0 correct.

Mini Skills Test

Based on which type of developer they were and which language they felt most familiar with, survey takers were asked an easy question on that programming language.

Categorizing

Based on which type of developer they were and which language they felt most familiar with, survey takers were asked an easy question on that programming language.

assumptions turned questions

Without leading, we wanted users to either validate or debunk all company-wide assumptions. Working with Tommy and Chris made sure we could do this while still pulling back clean quantitative data.

motivations

Being an education platform, it was important our product was attracting motivated users. We wanted to get high-level insights that could help inform what questions to then ask in our qualitative interviews.

Chris, our fearless data analyst went to town on an interactive visual for us

Now a 50 question, fully quantitative survey with all assumptions accounted for, Chris kept a close eye on results. He estimated we needed ~5,000 responses before we could be statistically relevant. We also wanted to send the survey out again a few months later to take into account time-zones, time of year, etc. He'd then add this to his script which generated a beautiful way for us to segment data on the fly and further analyze and discuss finding. 🌈

Qualitative
Interviews

Based on our survey, I worked with our team to build out a script we could use in tandem with our survey to fill in gaps and color our quantitative data. When possible, we tried interviewing in their typical work environments to capture what we could of environmental challenges as well. We also drilled into user's motivations specifically since this could really best be gathered through qualitative means. We knew we could then cross-reference them against our survey findings to color their corresponding personas.

*

Incentives & finding people

Without any quantitative for direction, we started seeing if assumptions were correct. Incentives ranged from free subscriptions to me having to give a talk to students at my alma mater to get an afternoon with students 😂

*

assumptions falling apart

Though we did identify a beginner persona, it became evident our vernacular was keeping them far from being our main user. Junior to more advanced developers seemed to be our main demo, joining only to learn the latest language and jumping ship afterwords.

*

motivations & lost opportunities

We had an emotional session ❤️with one user that said our platform had changed his family's life for the better. He had changed careers successfully from shop worker to full-stack developer at a reputable company.

Intern for the win!

It was actually Tommy and his fancy Ph.D. that broke the findings into 4 categories of users. He saw four distinct skill-levels while analyzing quantitative data. So we cross-segmented the data against those findings and built out our personas from there. 🙌

Analyzing Findings

1.

four segment types

The student developer still in college using the platform as supplement to school
The career changer that has experience but zero nomenclature & significantly less time than the others
The junior or overseas developer trying to move up in his field
The advanced developer looking to learn the latest language and sign right back off

2.

Explaining the high turn-over

The largest segment was our advanced developer who only used Code School as their crash course through whatever latest language was out they wanted to learn. They all understandably left as soon as they finished the course & wished we had an al a carte pricing structure. Interestingly, when told about their junior counterparts, they wished for a way to see examples of their work for hiring potential.

3.

opportunities abound

Once we knew our personas like family, what to develop next came second nature. More beginners were struggling with GIT while more advanced personas wished to share or view other's work. An ecosystem wanted to emerge! Or at the very least, a way to start incorporating GIT. We ended up following-through with a feature called "Projects", still feature on Pluralsight today.

Company-wide Ideation

One night after a long session of parsing through data, I realized 2 things.

With all data properly categorized under their respected persona, all that was left to do was use our experience to determine what was most important.
Our co-workers already had the insights to do this perfectly while simultaneously gaining respect and ownership for our new personas.

Enter a Company-wide Ideation Session

I started by putting parsed out quotes from our interviews on colored index cards. Each color represented one of our four personas. Each persona was then placed with a packet of quantitative findings on a table.

I had everyone that joined the company-wide invite split across the four tables of data. I had Chris our data-analyst ready with his visualizer of the data. Throughout the exercise, anyone could use this to segment quantitative data to get questions answered.

After a briefing of findings & instructions, everyone needed to decide on a photo, demographics, characteristics, occupation, a day in life & age. They also had to chose from the most important quote from the pile I parsed from actual recorded interviews. Persona's also needed names and some even got zodiac signs! ♋

Very rewarding and rather hilarious data-backed persona presentations ensued. Each group explained their reasoning for each detail of their personas. It became evident the new personas had gained support & respect.

Conclusion

If this case-study shows anything, it makes clear that even if you are your own user, you still may have blind-spots. Our findings debunked most assumptions, changed nomenclature & gave clearer direction to every new feature & marketing idea.

I would've loved life-size cardboard cut-outs strewn throughout the office but we didn't need them. In fact, it was a bigger win every time I'd hear them called out by name in every discussion from feature conception to release. Heck, Don White was named after one of our most skeptical critics!

Data-backed Personas

The Problem

Team & Methodologies