Evaluate Usability

YouX Academy

How to evaluate UX and Usability?

Luís Filipe

Luís Filipe

Managing Partner YouX

Miguel Félix

Miguel Félix

iBloom Managing Partner

In the article What is User Experience (UX) and how it differs from Usability we talk about the concepts of UX and Usability. In this article, we will identify some dimensions that we must take into account to evaluate UX and Usability.

How to evaluate UX?

The importance of UX varies depending on the product and the intended use.
The techniques to be used depend on the time horizon in which you want to work. The concept of user experience by Roto, V., Law, E. Vermeeren, A. & Hoonhout, J. (2011) adds a temporal component to the UX that will translate into the use of different evaluation methodologies.

Although the authors emphasize that the core of the user experience is the user experience, they consider that the user experience can even be prior to the first use, indirectly, based on expectations or related system usage experience (anticipated UX ); it can refer to a change in feelings during the interaction with the system (momentary UX); can refer to the evaluation of a specific episode (episodic UX) or views of the system as a whole after using a system for some time (cumulative UX).

Evaluate the UX
Roto, V., Law, E., Vermeeren, A., and Hoonhout, J. (eds). 2011. User Experience White Paper.

When working with a UX process, it is important to define which period of time will be considered: momentary, episodic or cumulative UX. If we consider shorter periods we can provide information such as people's reaction to the user interface. Opting for longer periods can reveal the impact of momentary experiences on the cumulative UX.

When we define longer UX periods, we can structure the user experience in terms of lifecycles or user journey, ranging from the first encounter with the system, usage episodes and even a reflection on usage. Past experiences influence future experiences, which allows you to shape future anticipations.

How to assess Usability?

According to ISO 9241-11, the usability evaluation should take into account:

  • Effectiveness – we refer to the ability of users to complete tasks  using the system as well as the quality of the result obtained.
  • Efficiency – level of resources spent on task performance.
  • Satisfaction – users' subjective reactions when using the system.

Due to the strong influence of context, it is very difficult to compare the usability of different systems, only subjective evaluations allow for inter-system comparison.

Some methods for measuring usability are: System Usability Scale (SUS), Microsoft Card Method, Contextual Interview, Think and Speak Out, First Click Testing , Card Sorting .
Roher (2014) focused specifically on research methods for UX referring to the wide variety of methods available, from the already tested and proven to new forms of online evaluation.
Emphasizing the importance of knowing how to select the methods to be used in a given project (whose number is usually limited), it presents a Framework with 20 methods 5 organized along three axes to help select the methods to be used in a given project: 

  • Attitudinal vs. behavioral
  • Qualitative vs. Quantitative
  • Context of Use 
User Research
Rohrer, C., (2014) When to Use Which User-Experience Research Methods, Accessed July 2, 2019, at: https://www.nngroup.com/articles/which-ux-research-methods/

Attitudinal dimension vs. behavioral: what people say versus what people do.
This is, moreover, a dichotomy that runs through all areas of social studies and opinion studies and that should keep us alert whenever we resort to surveys and interviews. The author considers that most usability studies should be based more on the behavioral aspect (eg A/B tests, eyetracking ), but attitudinal can also be important, namely to map users' mental models and develop information architectures.
Qualitative Dimension vs. Quantitative : qualitative studies (eg observation) as a generator of data collected directly versus data from quantitative studies obtained indirectly.
In usability studies, direct observation has the advantage that the researcher can see how the user uses the technology and immediately ask questions, investigate or correct some aspect of the study. Tendentially, qualitative methods are more suitable for answering questions about why a certain behavior or action is performed, or about how to correct a problem, qualitative methods are much more suitable for working with large amounts of questions and information. 

How the first two dimensions determine the type of questions that can be asked: 

user questions
Rohrer, C., (2014) When to Use Which User-Experience Research Methods, Accessed July 2, 2019, at: https://www.nngroup.com/articles/which-ux-research-methods/

The context of product use : Study participants are using the product or service and how.  What can be multiplied:

  • Natural (or almost) use of the product – minimize the interference of the study (obtain attitudes and behaviors as close as possible to reality);
  • Product Script Usage – Gain insight into specific usage aspects;
  • Products not used during the study – are used to study problems that go beyond usability, such as the study of cultural behaviors;
  • A hybrid combination of the above situations – they use a creative way of using the  product to achieve their goals.  Examples, participatory design methods.

Roher (2014) also considers that the product development phases, which he calls the time dimension, are also important when one intends to select research methods: 

  • Strategize – in this initial phase what is intended are ideas and insights which leads to a wide variation in the methods that can be used;
  • Execute – the research at this stage is formative and is intended to minimize execution risks.
  • Assess : once the product is available, it can be used by a sufficient number of users to start measuring its performance, which is normally summative in nature.  
Product Development
Rohrer, C., (2014) When to Use Which User-Experience Research Methods, Accessed July 2, 2019, at: https://www.nngroup.com/articles/which-ux-research-methods/

The available methods are not limited to the 20 proposed by Roher. Other authors suggest approaches that seem particularly interesting to us, such as: Prototyping, Personas and User Journey (also known as Journey Mapping).

In another post we will analyze these methods in detail.

You cannot copy content from this page