Creating scorecards and metrics from a UX assessment Format The basis of the course is a lecture format with some group exercises to reinforce the learned principles and guidelines. UX scorecards are of course not a substitute for digging into the reasons behind the metrics and trying to improve the experience. A simple web search will review dozens of examples of UX scorecards, and numerous textbooks have been written on the subject. 8.Â The average SUPR-Q score is 50%: The Standardized Universal Percentile Rank Questionnaire (SUPR-Q) is comprised of 13 items and is backed by a rolling database of 200 websites. Latency- The amount of time it takes data to travel from one location to another. Scorebuddy scorecards help you monitor the customer experience through all interactions at every touchpoint. Chain Committee has created standard supplier metrics and a scorecard to align expectations and promote performance improvement throughout the entire procure to pay process. Errors can tell you … And finally, the goal of Balanced Scorecard is to measure, yes, the performance of your business, focusing on some specific aspects. Figure 3: Example task-level score card that dives deeper into the task-level experience and metrics between three competitors on two platforms. The user error rate (UER) is the number of times a user makes a wrong entry. They should be tailored to an organization’s goals and feature a mix of broad (study-level/product-level) and specific (task-level) metrics. Tracking would help increase user experience maturity. We usually provide overall study scorecards (with task and study summary metrics) and individual task-level scorecards. SUS scores range from 0 to 100. While a “good” completion rate always depends on context, we’ve found that in over 1,100 tasks the average task completion rate is a 78%. Still working with product owners/managers, the scorecard usage would track teams’ user-centric efforts. The idea of quantifying experiences is still new for many people, which is one of the reasons I wrote the practical book on Benchmarking the User experience. User Error Rate. Other benchmarks can be average scores for common measures (e.g. The table of SUS scores above shows that across the 122 studies, we see average task completion rates of 100% can be associated with good SUS Scores (80) or great SUS scores (90+). Like in sports, a good score depends on the metric and context. They can be used to more visibly track (and communicate) how design changes have quantifiably improved the user experience. In some cases, we also provide separate scorecards with legends or more detail on actual task instructions, and data collection details (metrics definitions, sample characteristics) that more inquiring minds can visit. Figure 2: Example UX scorecard (non-competitive) comparing experiences across device types. From Soared to Plummeted: Can We Quantify Change Verbs? Customer satisfaction is probably the best barometer of the quality of the user experience provided by a product or service. Table 1: Raw System Usability Scale (SUS) scores, associated percentile ranks, completion rates and letter grades. 3. 2.Â Consumer Software Average Net Promoter Score (NPS) is 21%: The Net Promoter Score has become the default metric for many companies for measuring word-of-mouth (positive and negative). One of the first questions with any metric is “what’s a good score?”. Rating Scale Best Practices: 8 Topics Examined. This scorecard template is focused on the financial performance of the business. With multiple visualization of metrics, external benchmarks, or competitors, it becomes much easier to identify where you want to go. 6.Â Average Task Difficulty using the Single Ease Question (SEQ) is 4.8: The SEQ is a single question that has users rate how difficult they found a task on a 7-point scale where 1 = very difficult and 7 = very easy. But UX metrics can complement metrics that companies track using analytics—such as engagement time or bounce rate—by focusing on the key aspects of a user experience. For this product, most scores exceed these industry leaders (except desktop usability scores shown in yellow). UX research must be at the core of the business and with it the qualitative way of acquiring feedback. Weâve found providing visual error bars help balance showing the precision without overwhelming the audience. Photo by David Paul Ohmer - Creative Commons Attribution License http://www.flickr.com/photos/50965924@N00 The Role of Metrics in UX Strategy Uptime- The percentage of time the website or application is accessible to users. They can be used to more visibly track (and communicate) how design changes have quantifiably improved the user experience. That’s partially why UX metrics are so complex. Metrics & UX metrics. In an earlier article, I discussed popular UX metrics to collect in benchmark studies. Balanced Scorecard is a system that aligns specific business activities to an organization’s vision and strategy. Figure 1 shows these metrics aggregated into a Single Usability Metric (SUM)Â and in disaggregated form at the bottom of the scorecard for three competitor websites. Here are 10 benchmarks with some context to help make your metrics more manageable. Many big brands use UX metrics to improve the user experience of … Scorecards can help you measure: Collecting consistent and standardized metrics allows organizations to better understand the current user experience of websites, software, and apps (Sauro, … While UX scorecards should contain some combination of study-level and task-level metrics, displaying all this data in one scorecard, or even a couple scorecards, has its challenges. Task-level metrics: The core task-level metrics address the ISO 9241 pt 11Â aspects of usability: effectiveness (completion rates), efficiency (task time), and satisfaction (task-level ease using the SEQ). You can ask users how satisfied they are with particular features, with their experience today and of course overall. This is one of the advantages of using standardized measuresÂ as many often have free or proprietary comparisons. Engagement: level of user involvement, typically measured via behavioral proxies such as frequency, intensity, or depth of interaction over some time period. The rating system, which involves a scorecard and UX coaching, was a way to track and measure user-centric efforts and improvements. Happiness: measures of user attitudes, often collected via survey. The old paradigm of analytics is geared more towards measuring progress against business goals. What you measure is what you get. A score of 50% means half the websites score higher and half score lower than your site’s SUS score. A scorecard is a set of indicators grouped according to some rules:. (They look like Star Wars Tie fighters.) 7.Â Average Single Usability Metric (SUM) score is 65%: The SUM is the average of task metricsâcompletion rates, task-times and task-difficulty ratings. The UX Scorecard is a process similar to a heuristic evaluation that helps identify usability issues and score a given experience. A UX scorecard is great way to quickly visualize these metrics. They represent a product’s user experience, which is hard to quantify. Names and details intentionally obscured. This article focuses on displaying UX metrics collected empirically. It takes into account the targets of the organization and works to make efficient the performances aimed at reducing the costs while improving the customer satisfaction from time to time. Contact Us, Consumer Software Average Net Promoter Score (NPS) is 21%. Are Sliders More Sensitive than Numeric Rating Scales? The value of usability scorecards and metrics" on Thursday, November 15 at 3:30 p.m. EST. The negative net promoter score shows that there are more detractors than promoters. While still useful, they’re lagging indicators of UX decisions.Common metrics include: 1. Remember — we need fair answers. Study-level metrics: Include broader measures of the overall user experience. UX pros will want to dig into the metrics and will be more familiar with core metrics like completion, time, etc. Financial—The Financial Perspectiveexamines the contribution of an organization’s strategy to the bottom line. Only 10% of all tasks we’ve observed are error-free or, in other words, to err is human. But some useful frameworks can help measure user experience. Download a templatewith 10 questions or create similar form on Google forms/Typeform 2. Both Figures 1 and 3 feature three products (one base and two competitors). Generally errors are a useful way of evaluating user performance. You may not realize that different members of your team have different ideas about the goals of your project. We use colors, grades, and distances to visually qualify the data and make it more digestible. Figure 4: Example âoverviewâ card that can be linked or reference on scorecards for more detail on study metrics and task details. 10.Â The average number of errors per task is 0.7: Across 719 tasks of mostly consumer and business software, we found that by counting the number of slips and mistakes about two out of every three users (2/3) had an error. User experience metrics aren’t just about conversions and retentions. For example: satisfaction, perceived ease of use, and net-promoter score. ; Normalized indicators are presented in a hierarchical structure where they contribute to the performance of their containers. User Error Rate. 2. Using the UX Scorecard process to walk through a workflow end-to-end in critical detail enables us to quickly spot opportunities for improvement. Its 10 items have been administered thousands of times. It is a tool, or more accurately, a specific type of a report, that allows to easily visualize the website’s … Our challenge was to create a comparable scorecard for an array of products across UX frameworks, usability maturity, and technology platforms. Balanced Scorecard (BSC) is a well-articulated approach to understanding how to describe strategy and metrics. They show us behaviors, attitudes, emotions — even confusion. A competitive benchmark study provides ideal comparison for all metrics. Figures 1 and 2 include study-level metrics in the top part of each figure. Its 10 items have been administered thousands of times. It could be that the bulk of users on any one website are new and are therefore less inclined to recommend things they are unfamiliar with. These usually include SUPR-Q, SUS, UMUX-Lite, product satisfaction, and/or NPS. Pageviews- Number of pages viewed by a single user. Hereâs some advice on what we do to make them more digestible. User experience scorecards are a vital way to communicate usability metrics in a business sense. Time On Task Knowing how long it takes for your users to complete a task will give you valuable insight into the effectiveness of your UX design. Average Single Usability Metric (SUM) score is 65%: Usability problems in business software impact about 37% of users: The average number of errors per task is 0.7: Associating completion rates with SUS scores, Standardized Universal Percentile Rank Questionnaire, User Experience Salaries & Calculator (2018). 1. The scenario-based UX metrics scorecard in practice 18 Scenario-based UX metrics scorecarding summary 20 UX metrics as part of a customer-centric strategy 22 About the Author 23 User Experience Metrics: Connecting the language of UI design with the language of business. 1. If metric changes donât move past the error bars, itâs hard to differentiate the movement from sampling error. LavaCon UX Review: Case Studies, Content, and Metrics Impact Every Part of Business Jennifer O'Neill - Event Coverage Due to COVID-19, LavaCon 2020 morphed into a completely virtual event with new focus on user experience (UX) evident in this year’s event name. Standardization is good, but not if it gets in the way of communicating and helping prioritize and understand improvements to the experience. User error rate. The supplier metrics were evaluated both on impact to the supply chain process as well as the measurability by both suppliers and drilling contractors. It was developed to evaluate the quality of the user experience, and help teams measure the impact of UX changes. While that’s bad for a usable experience, it means a small sample size of five users will uncover most usability issues that occur this frequently. 1 + 303-578-2801 - MST Let’s … Non-UX execs will want the bottom line: red, yellow, and green, and maybe grades. 2. Associating completion rates with SUS scores is another way of making them more meaningful to stakeholders who are less familiar with the questionnaire. First, indicators are normalized (according to their properties like measurement scale and performance formula). Generally errors are a useful way of evaluating user performance. Seven-day active … The example scorecards here only show one point in time from a single benchmark study. Itâs not always possible to include both on one scorecard so consider using different ones that are linked by common metrics. They allow teams to quantify the user experience and track changes over time. Figure 3 also shows task-level metrics for two dimensions: platform (desktop and mobile) and competitor (base product and two competitors). Scorecards can vary in many ways, but at the heart of them, we often find: A table of data: Tasks, scenarios, or key results are displayed in rows with quantified metrics in columns. Latin and Greco-Latin Experimental Designs for UX Research, Improving the Prediction of the Number of Usability Problems, Quantifying The User Experience: Practical Statistics For User Research, Excel & R Companion to the 2nd Edition of Quantifying the User Experience. The scorecard in Figure 2 features data that wasnât collected as part of a competitive benchmark but shows the difference between three competitors from our SUPR-Q, UMUX-Lite, and NPS databases. But they can be a good way for tracking and promoting your design change efforts. UX benchmark studies are an ideal way to systematically collect UX metrics. You need to consider the audience and organization. Follow-up benchmark studies can show how each metric has hopefully improved (using the same data collection procedures). You can set agent performance metrics for every interaction and use self-evaluation to determine how well each step in the customer’s journey went. We usually start our scorecards with our broadest measure of the user experience first (at the top) and then provide the more granular detail the tasks provide (at the bottom). As UX designers, we need to challenge the sole reliance on data-backed hunches. A UX Scorecard is a fairly common term in the world of UX. It represents the strategic objectives of an organization in terms of increasing revenue and reducing cost. However, most of the datasets I have used are only 3-metric SUM scores. Across the 500 datasets we examined the average score was a 68. I also like to keep scorecards that feature data from actual users separate from scorecards that may feature metrics from a PURE evaluation or expert review. Adapted from A Practical Guide to SUS and updated by Jim Lewis 2012. After all, a bad experience is unlikely to lead to a satisfied customer. Youâll want to be in the green, get As and Bs, and have metrics at least the same or ahead of competitors and as far into the best-in-class zone on the continuums (far right side of graphs in Figures 1, 2, and 3). UX metrics are one type of metric. Confidence intervalsÂ are an excellent way to describe the precision of your UX metrics. Quantifying user experience. Senior executives understand that their organizations measurement system strongly affects the behavior of managers and employees. Examples mi… Showing this precision can be especially important when tracking changes over time. UX practitioners can use metrics strategically by identifying business objectives that drive company action and by making explicit their own contribution to those objectives. Executives also understand that traditional financial accounting measures like return-on-investment and earnings-per-share can give misleading signals for continuous improvement and innovationactivities todays competitive environment demands. The framework is a kind of UX metrics scorecard that’s broken down into 5 factors: Happiness: How do users feel about your product? Errors can tell you … Denver, Colorado 80206 Denver, Colorado 80206 Calculate results by this form and find a common value: (Result 1 + Result 2 + …+… 9.Â Usability problems in business software impact about 37% of users: In examining both published and private datasets, we found that the average problem occurrence in things like enterprise accounting and HR software programs impact more than one out of three (1/3) users. Quantifying the user experience is the first step to making measured improvements. Rating Scale Best Practices: 8 Topics Examined. 3.Â Website Average Net Promoter Score is -14%: We also maintain a large database of Net Promoter Scores for websites. Across 200 tasks we’ve found the average task-difficulty is a 4.8, higher than the nominal midpoint of 4, but consistent with other 7-point scales. They should be tailored to an organizationâs goals and feature a mix of broad (study-level/product-level) and specific (task-level) metrics. As such, it is impacted by completion rates which are context-dependent (see #1 above) and task times which fluctuate based on the complexity of the task. They can, however, be difficult to interpret and include in scorecards. The HEART framework is a set of user-centered metrics. 4.Â Average System Usability Scale (SUS) Score is 68: SUS is the most popular questionnaire for measuring the perception of usability. Here are seven essential performance metrics that can help you better understand the ROI of your UX design. Donât feel like you need to stick with a one-sized-fits-all scorecard. Table 2 lists three categories of big-picture UX metrics that correlate with the success of a user experience… The traditional financial performance measures worked well for the industrial era, but they ar… The scorecard shows overall SUPR-Q scores (top) and task-based scores that are aggregated (SUM) and stand-alone (completion, time, ease). Even without a competitive benchmark, you can use external competitive data. Table 2: SUM Percent Scores from 100 website and consumer software tasks and percentile ranks. Collecting consistent and standardized metrics allows organizations to better understand the current user experience of websites, software, and apps. The Single Usability Metric (SUM)Â is an average of the most common task-level metrics and can be easier to digest when youâre looking to summarize task experiences. Since in the real-world people are more likely to talk about their frustrations, rather than how satisfied they are, a good approach can … Identifying clear goals will help choose the right metrics to help you measure progress. SUS scores range from 0 to 100. Give a questionnaire to people who know your product (at least 10 users outside your team). It looks at ways to implement the financial activities effectively while lowering the financial input. While helping Google product teams define UX metrics, we noticed that our suggestions tended to fall into five categories: 1. Wider intervals mean less precision (and are a consequence of using smaller sample sizes). Latin and Greco-Latin Experimental Designs for UX Research, Improving the Prediction of the Number of Usability Problems, Quantifying The User Experience: Practical Statistics For User Research, Excel & R Companion to the 2nd Edition of Quantifying the User Experience. Executives may be interested in only the broader level measures whereas product teams will want more granular details. Average System Usability Scale (SUS) Score is 68: SUS is the most popular questionnaire for measuring the perception of usability. This process provides an opportunity to build consensus about where you're headed. We create separate scorecards for each task that allow teams to dig into more specific task measures or understand whatâs causing problems (Figure 3). While youâll want to tailor each scorecard to each organization, here are some common elements we provide as part of our UX benchmark studies and ways we visualize them (and some advice for creating your own). The table below shows the percentile ranks for a range of scores, how to associate a letter grade to the SUS score, and the typical completion rates we see (also see #5). Increasingly those metrics quantify the user experience (which is a good thing). 68 for SUS, 50% for SUPR-Q) or even other comparable products. All sample sizes in these scorecards are relatively large (>100) and have relatively narrow intervals. It will be higher for 4-metric scores, which include errors. 5.Â High task completion is associated with SUS scores above 80: While task completion is the fundamental metric, just because you have high or perfect task completion doesn’t mean you have perfect usability. From Soared to Plummeted: Can We Quantify Change Verbs? 1. Figures 1, 2, and 3 show example scorecards (with names and details redacted or changed) that can be shown electronically or printed. 3300 E 1st Ave. Suite 370 All companies say they care about Customer Experience but saying it, doing it, and seeing results are very different. 1. This suggests that users are less loyal to websites and, therefore, less likely to recommend them. UX scorecards: Quantifying and communicating the user experience. UX scorecards are an excellent way to visually display UX metrics. See Chapter 5 in Benchmarking the User Experience for more. For example, a SUM % score (from averaging completion rates, task-time and task-difficulty) of a 55 was at the 25th percentile–meaning it was worse than 75% of all tasks. Across the 500 datasets we examined the average score was a 68. This is for 3-metric SUM scores. The task metrics in Figures 1, 2, and 3 have small horizontal lines showing the precision of the estimate. UX scorecards are an excellent way to visually display UX metrics. If users cannot complete what they came to do in a website or software, then not much else matters. Using a scorecard helps organizations balance their strategic objectives across four perspectives: 1. Figure 4 shows an example overview scorecard. What metrics does Balanced Scorecard include" The most important one - the "key" metrics. Are Sliders More Sensitive than Numeric Rating Scales? The table below shows a table of SUM scores and the percentile ranking from the 100 tasks.Â For example, getting a SUM score for a task above 87% puts the task in the 95th percentile. Customer—The Customer Perspectivefocuses on customers’ satisfaction… The term "scorecard" has been a little hijacked by the " Balanced Scorecard" approach of analysing your business; however a scorecard only needs to contain data that is useful to you in your circumstances.Scorecards are particularly useful when used on an overview KPI dashboard because … Figures 1, 2, and 3 all show examples of the SUM. Contact Us, UX metrics to collect in benchmark studies, User Experience Salaries & Calculator (2018). In examining 1,000 users across several popular consumer software products, we found the average NPS was 21%. It allows teams to track changes over time and compare to competitors and industry benchmarks. Use multiple ways to visualize metric performance (colors, grades, and distances) and include external benchmarks, competitor data, and levels of precision when possible. Despite the context-sensitive nature, I’ve seen that across 100 tasks of websites and consumer software that the average SUM score is 65%. It measures perceptions of usability, credibility and trust, loyalty, and appearance. 1 + 303-578-2801 - MST This adds an additional dimension and likely means removing the competitors or finding other ways to visualize improvements (or lack thereof). They can be both subjective and objective, qualitative and quantitative, analytics-based and survey-based. Figure 1: Example scorecard for three consumer desktop websites. 1.Â Average Task Completion Rate is 78%: The fundamental usability metric is task completion. Scorecards are a very popular and powerful way to visualise the numerical values of your metrics. 3300 E 1st Ave. Suite 370 Website Average Net Promoter Score is -14%: Average System Usability Scale (SUS) Score is 68, High task completion is associated with SUS scores above 80, verage Task Difficulty using the Single Ease Question (SEQ) is 4.8.