During the Contribute your skills to Debian event that took place in Paris last week-end, we conducted a usability testing session. Six people were tasked with testing a few aspects of the GNOME 3.22 desktop environment and of the Debian 9 (Stretch) operating system. A number of other people observed them and took notes. Then, two observers and three testers analyzed the results, that we are hereby presenting: we created a heat map visualization, summed up the challenges met during the tests, and wrote this blog post together. We will point the relevant upstream projects to our results.

A couple of other people also did some usability testing but went in much more depth: their feedback is much more detailed and comes with a number of improvement ideas. I will process and publish their results as soon as possible.

  1. GNOME and Debian usability testing, May 2017
  2. Missions
    1. A. Nautilus
      1. Mission A.1 — Download and rename file in Nautilus
      2. Mission A.2 — Manipulate folders in Nautilus
      3. Mission A.3 — Create a bookmark in Nautilus
      4. Mission A.4 — Nautilus display settings
    2. B. Package management
      1. Introduction
      2. Mission B.1 — Install and remove packages
      3. Mission B.2 — Search and install a package
      4. Mission B.3 — Upgrade the system
    3. C. Settings
      1. Mission C.1 — Change the desktop background
      2. Mission C.2 — Tweak temporary files management
      3. Mission C.3 — Change the default video player
      4. Mission C.4 — Add and remove world clocks
  3. Results and analysis
    1. Heat map
    2. What were the challenges?
      1. Mission A.3 — Create a bookmark in Nautilus
      2. Mission B.1 — Install and remove a package
      3. Mission C.2 — Tweak temporary files management
      4. Mission C.3 — Change the default video player
      5. Mission C.4 — Add and remove world clocks
    3. General notes and observations

Missions

Testers were provided a laptop running GNOME on a a Debian 9 (Stretch) Live system. A quick introduction (mostly copied from the one we found in some GNOME usability testing reports) was read. Then they were asked to complete the following tasks.

A. Nautilus

Mission A.1 — Download and rename file in Nautilus

  1. Download a file from the web, a PDF document for example.
  2. Open the folder in which the file has been downloaded.
  3. Rename the dowloaded file to SUCCESS.pdf.
  4. Toggle the browser window to full screen.
  5. Open the file SUCCESS.pdf.
  6. Go back to the File manager.
  7. Close the file SUCCESS.pdf.

Mission A.2 — Manipulate folders in Nautilus

  1. Create a new folder named cats in your user directory.
  2. Create a new folder named to do in your user directory.
  3. Move the cats folder to the to do folder.
  4. Delete the cats folder.

Mission A.3 — Create a bookmark in Nautilus

  1. Create a folder named unicorns in your personal directory.
  2. This folder is important. Add a bookmark for unicorns in order to find it again in a few weeks.

Mission A.4 — Nautilus display settings

Folders and files are usually listed as icons, but they can also be displayed differently.

  1. Configure the File manager to make it show items as a list, with one file per line.
  2. You forgot your glasses and the font size is too small for you to see the text: increase the size of the text.

B. Package management

Introduction

On Debian, each application is available as a "package" which contains every file needed for the software to work.

Unlike in other operating systems, it is rarely necessary and almost never a good idea, to download and install software from the authors website. We can rather install it from an online library managed by Debian (like an appstore). This alternative offers several advantages, such as being able to update all the software installed in one single action.

Specific tools are available to install and update Debian packages.

Mission B.1 — Install and remove packages

  1. Install the vlc package.
  2. Start VLC.
  3. Remove the vlc package.

Mission B.2 — Search and install a package

  1. Find a piece of software which can download files with BitTorrent in a graphical interface.
  2. Install the corresponding package.
  3. Launch that BitTorrent software.

Mission B.3 — Upgrade the system

Make sure the whole system (meaning all installed packages) is up to date.

C. Settings

Mission C.1 — Change the desktop background

  1. Download an image you like from the web.
  2. Set the downloaded image as the desktop wallpaper.

Mission C.2 — Tweak temporary files management

Configure the system so that temporary files older than three days are deleted automatically.

Mission C.3 — Change the default video player

  1. Install VLC (ask for help if you could not do it during the previous mission).
  2. Make VLC the default video player.
  3. Download a video file from the web.
  4. Open the downloaded video, then check if it opens with VLC.

Mission C.4 — Add and remove world clocks

When you click the time and date in the top bar, a menu pops-up. There, you can display clocks in several time-zones.

  1. Add a clock with Rio de Janeiro timezone, then another showing the current time in Boston.
  2. Check that the time and date menu now displays these two additional clocks.
  3. Remove the Boston clock.

Results and analysis

Heat map

We used Jim Hall's heat map technique to summarize our usability test results. As Renata puts it, it is "a great way to see how the users performed on each task. The heat map clarifies how easy or difficult it was for the participant to accomplish a certain task.

  1. Scenario tasks (from the usability test) are arranged in rows.
  2. Test participants (for each tester) are arranged in columns.
  3. The colored blocks represent each tester’s difficulty with each scenario task.

Green blocks represent the ability of the participant to accomplish the tasks with little or no difficulty.

Yellow blocks indicate the tasks that the tester had significant difficulties accomplishing.

Red blocks indicate that testers experienced extreme difficulty or where testers completed the tasks incorrectly.

Black blocks indicate tasks the tester was unable to complete."

Alternatively, here is the spreadsheet that was used to create this picture, with added text to avoid relying on colors only.

Most tasks were accomplished with little or no difficulty so we will now focus on the problematic parts.

What were the challenges?

The heat map shows several "hot" rows, that we will now be looking at in more details.

Mission A.3 — Create a bookmark in Nautilus

Most testers right-clicked the folder first, and eventually found they could simply drag'n'drop to the bookmarks location in the sidebar.

One tester thought that he could select a folder, click the hamburger icon, and from there use the "Bookmark this folder" menu item. However, this menu action only works on the folder one has entered, not on the selected one.

Mission B.1 — Install and remove a package

Here we faced a number of issues caused by the fact that Debian Live images don't include package indices (with good reason), so no package manager can list available software.

Everyone managed to start a graphical package manager via the Overview (or via the CLI or Alt-F2 for a couple power users).

Some testers tried to use GNOME Software, which listed only already installed packages (Debian bug #862560) and provided no way we could find to refresh the package indices. That's arguably a bug in Debian Live, but still: GNOME Software might display some useful information when it detects this unusual situation.

We won't list here all the obstacles that were met in Synaptic: it's no news its usability is rather sub-optimal and better alternatives (such as GNOME Software) are in the works.

Mission C.2 — Tweak temporary files management

The mission was poorly phrased: some observers had to clarify that it was specifically about GNOME, and not generic Linux system administration: some power-users were already searching the web for command-line tools to address the task at hand.

Even with this clarification, no tester would have succeeded without being told they were allowed to use the web with a search query including the word "GNOME", or use the GNOME help or the Overview. Yet eventually all testers succeeded.

It's interesting to note that regular GNOME users had the same problem as others: they did not try searching "temporary" in the Overview and did not look-up the GNOME Help until they were suggested to do so.

Mission C.3 — Change the default video player

One tester configured one single video file format to be opened by default with VLC, via right-click in Nautilus → Open with → etc. He believed this would be enough to make VLC the default video player, missing the subtle difference between "default video player" and "default player for one single video format".

One tester tried to complete this task inside VLC itself and then needed some help to succeed. It might be that the way web browsers ask "Do you want ThisBrowser to become the default web browser?" gave a hint an application GUI is the right place to do it.

Two testers searched "default" in the Overview (perhaps the previous mission dealing with temporary files was enough to put them in this direction). At least one tester was confused since the only search result (Details – View information about your system), which is the correct one to get there, includes the word View, which suggests that one cannot modify settings there, but only view them.

One long-term GNOME user looked in Tweak Tool first, and then used the Overview.

Here again, GNOME users experienced essentially the same issues as others.

Mission C.4 — Add and remove world clocks

One tester tried to look for the clock on the top right corner of the screen, then realized it was in the middle. Other than this, all testers easily found a way to add world clocks.

However, removing a world clock was rather difficult; although most testers managed to do it, it took them a number of attempts to succeed:

  1. Several testers left-clicked or right-clicked the clock they wanted to remove, expecting this would provide them with a way to remove it (which is not the case).
  2. After a while, all testers noticed the Select button (that has no text label nor tooltip info), which allowed them to select the clock they wanted to remove; then, most testers clicked the 1 selected button, hoping it would provide a contextual menu or some other way to act on the selected clocks (it doesn't).
  3. Eventually, everyone managed to locate the Delete button on the bottom right corner of the window; some testers mentioned that it is less visible and flashy than the blue bar that appears on the top of the screen once they had entered "Selection" mode.

General notes and observations