Benchmarking the Big Shift: Creating a Usability Baseline for a Major Digital Consolidation

Focus

  • UX benchmarking

  • Strategic research

  • Cross-team enablement

Client: Fortune 100 telecommunications company

Role: Lead UXR, Internal pitch person, program architect

Impact: Foundational usability benchmark metrics used to track impact of consolidating 5 digital platforms into 2

Benchmarking isn’t usability testing. It is infrastructure.
It took vision, persistence, and a lot of cross-functional diplomacy — but we now have a system that helps the org track what matters: not just how the design looks, but how it performs over time.

The Set-Up

The Challenge

My Role

Our company was rolling out a massive consolidation — collapsing 3 websites and 2 apps into a single unified platform.
14 product teams, multiple business lines, millions of users. No pressure.

Until this point, our research was focused on usability of individual features within the new experience. But the real question everyone had — and no one was addressing — was this:

“Is this new, unified experience actually better than what we had before?”

That question required a different scale of research — and a strategic shift.

  • Existing research was too feature-focused to answer journey-level questions

  • Teams had conflicting goals, but shared fears about the impact of the new platform

  • We didn’t have the right tools (or funding) for large-scale quantitative benchmarking

  • There was no baseline to measure success against — yet

Made the internal pitch to secure funding for a large-scale benchmark study

  1. Co-designed the study with third-party partner

  2. Led task prioritization using analytics + support data

  3. Coordinated across 14 teams and multiple user types

  4. Built the foundation for an ongoing benchmarking program that continues today

Phase 1:
The Pitch

Most teams were buried in deadlines. But I knew this project needed to zoom out.

I created a pitch deck showing:

  • Gaps in current usability data

  • Risks of launching without a baseline

  • A framework for a new study that would assess task success, satisfaction, and usability across platforms

That deck won buy-in — and funding — from the VP of Design and the Sr. Director of Experience Research.

Phase 2:
Analytics Review and Test Plan

To determine which tasks we’d benchmark, I needed hard data. I tapped into:

  • Proto-personas I had built earlier using behavioral analytics

  • Call center logs to identify high-friction, high-value tasks

  • Cross-team conversations to understand overlapping journeys

Since we didn’t have an in-house tool for quantitative usability, I partnered with Jeff Sauro and his team to design and execute the study using MUIQ.

Key design elements:

  • Subjective metrics: SUS (System Usability Score), NPS (Net Promoter Score)

  • Objective validation: Custom success questions to confirm task completion (we couldn’t use URL capture, so we got creative)

  • Post-task checks: To identify gaps between perceived success and actual success

Phase 3:
Socializing the Results

I delivered findings team-by-team, aligning results with each roadmap.

While the timing (right before launch) meant some fixes couldn’t happen immediately, teams were engaged and eager to have hard data to back future decisions.

More importantly:

We now had a baseline.


Not perfection — but a launchpad for continuous improvement.

Scaling the Program

With Round 1 done, I shifted to building the long-term plan:

  • Set the cadence for ongoing benchmark rounds

  • Evaluated tooling to eventually bring the study in-house

  • Built data stream integrations (e.g., tying in support logs and behavioral analytics)

  • Educated teams on how and when to use benchmark data in decision-making

The program continues today — not just as a research artifact, but as an embedded part of how we measure digital progress.