Usability testing at Microsoft
A usability testing framework for Microsoft’s blogs, built to scale.
- UX research
- Framework design
- SEQ, SUS and SUPR-Q
- Synthesis
- 30+usability tests run across Microsoft’s blog ecosystem over two years
- 40%+less effort on qualitative synthesis, using AI-assisted transcript analysis
- What didn't workReviewing every session in full took 50% of the effort per test. It was thorough but it would not scale, so I changed the approach.
01 · The brief
A framework where there was none.
As part of Microsoft’s blog ecosystem transformation, a sister project emerged to usability test every blog. With no existing framework in place, my role was to develop and execute a UXR methodology.
An initial workshop with Microsoft’s Head of Blog defined the business goals. From there I led strategy, making sure effort and resources lined up with business objectives and user needs.
In discovery I identified the need for a structured framework, aligned objectives with business and user goals, and defined an archetypal testing approach.
Business goals
- Validate design decisions throughout the design process
- Gather baseline quantitative data
- Determine how Microsoft compared against its competitors
- Gather qualitative insights from blog audience cohorts
- Give blog teams insights for content creation
User needs
- Simplify content discoverability
- Create clear landmarks and pathways for navigation
- Create a more mobile friendly experience
- Foster confidence and trust in the brand and content
02 · Methodology
Efficient, scalable and comprehensive.
After workshops and consultation I developed a usability testing methodology and presented it to Microsoft for approval, built on three pillars.
Efficiency: gather industry standard quantitative metrics and qualitative insights from participants, unmoderated, within 30 minutes.
Scalability: unmoderated testing, standardised tasks and simpler synthesis.
Comprehensive: multiple metrics for comparison, usable across design archetypes.
03 · Metrics
Four metrics, one comparable picture.
The project needed strong quantitative insight, so I designed the tests to capture SEQ after every task, and SUS, SUPR-Q and CSAT after each test.
Single Ease Questionnaire (SEQ): asked after every task, rating how complex it was to complete.
System Usability Scale (SUS): asked after the test, the industry standard metric for system usability.
Single User Performance Rating Questionnaire (SUPR-Q): asked after the test, broken into four key UX factors, and including NPS.
Customer Satisfaction (CSAT): asked after the test, rating participant satisfaction.
04 · Testing
Scripted tightly, run unmoderated.
Tests ran primarily on usertesting.com, unmoderated, which meant robust scripting so participants gave useful insights and quality data.
I ran the testing independently: writing the test scripts, recruiting participants and running the sessions, collecting both qualitative and quantitative insights.
05 · Synthesis
Synthesis that scaled.
Quantitative data came out of usertesting.com as a CSV, and a template I built turned it into SEQ, SUPR-Q, SUS and CSAT scores. For qualitative data I used AI to find the signal.
Early on I reviewed every usability test in full to uncover qualitative insights. It worked, but it took 50% of the effort per test.
To optimise this I used Copilot to analyse transcripts, applying filters to detect key language and sentiment patterns. That let me focus on targeted snippets and cut effort by over 40% while keeping the depth and quality of the insights.
Reviewing every session in full took 50% of the effort per test.
Let Copilot surface language and sentiment patterns in transcripts, then review the targeted snippets.
06 · Results and recommendations
Not a clean sweep.
Every test ended in a standardised PowerPoint of insights, metrics and data, and every test produced meaningful insights for Microsoft to act on.
The recommendations were presented to Microsoft for approval and an update plan was formed to address them.
Comparing scores from before and after redesigns was not a clean sweep. In the aggregate dashboard some averages rose and others fell, and in the pre and post chart the mobile score sits lower after the redesign. Those results sat alongside the qualitative findings, including difficulty navigating on mobile, in the recommendations.
07 · Final thoughts
One framework, many clients.
The methodology, templates and synthesis efficiencies were used on other projects for other clients, and as the SME in this space I was promoted to a specialty in User Research & Testing at 10up.
Also tested with this framework
- Meta
- Starbucks
- Salesforce
- Bupa
- California State Government
- Askable