YouX subscription
Evaluate Usability

YouX Academy

How to evaluate UX and Usability?

Picture of Luís Filipe

Luís Filipe

Managing Partner YouX

Picture of Miguel Félix

Miguel Félix

Managing Partner iBloom

In the article What is User eXperience (UX) and how it differs from Usability we talk about the concepts of UX and Usability. In this article we will identify some dimensions that we must take into account to evaluate UX and Usability.

How to evaluate UX?

The importance of UX varies depending on the product and its intended use.
The techniques to be used depend on the time horizon in which you want to work. The concept of user experience by Roto, V., Law, E. Vermeeren, A. & Hoonhout, J. (2011) adds a temporal component to UX that will translate into the use of different evaluation methodologies.

Although the authors emphasize that the core of the user experience is the experience of use, they consider that the user experience can even be prior to the first use, indirectly, based on expectations or experience of using related systems (anticipated UX ); may refer to a change in feelings during interaction with the system (momentary UX); It can refer to the evaluation of a specific episode (episodic UX) or to views of the system as a whole after using a system for some time (cumulative UX).

Evaluate UX
Roto, V., Law, E., Vermeeren, A., and Hoonhout, J. (eds). 2011. User Experience White Paper.

When we work with a UX process, it is important to define the period of time that will be considered: momentary, episodic or cumulative UX. If we consider shorter periods we can provide information such as people's reactions to the user interface. Opting for longer periods can reveal the impact of momentary experiences on cumulative UX.

When we define longer UX periods, we can structure the user experience in terms of life cycles or user journey, ranging from the first encounter with the system, episodes of use and even reflection on use. Previous experiences influence future experiences, which allows us to shape future anticipations.

How to evaluate Usability?

According to ISO 9241-11, usability assessment should take into account:

  • Effectiveness – we refer to the ability of users to complete tasks  using the system as well as the quality of the result obtained.
  • Efficiency – level of resources spent on performing tasks.
  • Satisfaction – subjective reactions of users when using the system.

Due to the strong influence of context, it is very difficult to compare the usability of different systems, only subjective evaluations allow inter-system comparison.

Some methods for measuring usability are: System Usability Scale (SUS), Microsoft Card Method, Contextual Interview, Think and Speak Aloud, First Click Testing , Card Sorting .
Roher (2014) looked specifically at research methods for UX, referring to the wide variety of methods available, from tried and tested ones to new forms of online assessment.
Highlighting the importance of knowing how to select the methods to be used in a given project (the number of which is normally limited), it presents a Framework with 20 methods 5 organized according to three axes to help select the methods to be used in a given project: 

  • Attitudinal vs. Behavioral
  • Qualitative vs. Quantitative
  • Context of Use 
User Research
Rohrer, C., (2014) When to Use Which User-Experience Research Methods, Accessed on July 2, 2019, at: https://www.nngroup.com/articles/which-ux-research-methods/

Attitudinal dimension vs. behavioral: what people say versus what people do.
This is, moreover, a dichotomy that runs throughout the areas of social studies and opinion studies and that should keep us alert whenever we use surveys and interviews. The author considers that most usability studies should be based more on the behavioral aspect (e.g. A/B testing, eyetracking ), but the attitudinal aspect can also be important, particularly for mapping users' mental models and developing information architectures.
Qualitative Dimension vs. Quantitative : qualitative studies (e.g. observation) as a generator of data collected directly versus data from quantitative studies obtained indirectly.
In usability studies, direct observation has the advantage of the researcher being able to see how the user uses the technology and immediately ask questions, investigate or correct some aspect of the study. Qualitative methods tend to be more suitable for answering questions about why a certain behavior or action is taking place, or how to correct a problem. Qualitative methods are much more suitable for dealing with large amounts of questions and information. 

As the first two dimensions determine the type of questions that can be asked: 

User questions
Rohrer, C., (2014) When to Use Which User-Experience Research Methods, Accessed on July 2, 2019, at: https://www.nngroup.com/articles/which-ux-research-methods/

The context of product use : are study participants using the product or service and how.  What can be demultiplied:

  • Natural (or almost) use of the product – minimize study interference (obtain attitudes and behaviors as close to reality as possible);
  • Product script usage – gain insights into specific usage aspects;
  • Product not used during the study – are used to study problems that go beyond usability, such as the study of cultural behaviors;
  • A hybrid combination of the previous situations – they use a creative way of using the  product to achieve their goals.  Examples, participatory design methods.

Roher (2014) also considers that the product development phases, which he calls the time dimension, are also important when selecting research methods: 

  • Strategize – at this initial stage what we want are ideas and insights, which leads to a wide variation in the methods that can be used;
  • Execute – research at this stage is formative and aims to minimize execution risks.
  • Assess : once the product is available, it can be used by a sufficient number of users to begin measuring its performance, which is normally summative in nature.  
Product Development
Rohrer, C., (2014) When to Use Which User-Experience Research Methods, Accessed on July 2, 2019, at: https://www.nngroup.com/articles/which-ux-research-methods/

The available methods are not limited to the 20 proposed by Roher. Other authors suggest approaches that seem particularly interesting to us, such as: Prototyping, Personas and User Journey (also known as Journey Mapping).

In another post we will analyze these methods in detail.

You cannot copy content of this page

Skip to content