Usability Testing Report 20190507 2/2

Hanna Alatalo
Jarkko Kuivanen
Elisa Nauha
Jere Ojala
Kimmo Urtamo

Version 1.0.0
Public
5.6.2019

University of Jyväskylä
Faculty of Information Technology

Abstract: The functionalities developed into virtual learning environment called TIM by the TIMCAN project will replace already existing service called ICAnDoiT. ICAnDoiT is an application used by The Centre for Applied Language Studies (CALS) for the purpose of studying adaptive feedback. The adaptive feedback is feedback that becomes gradually more detailed with every wrong response. The project team will develop a dropdown menu plugin different from the original ICAnDoiT, a drag-and-drop plugin, adaptive feedback plugin and functionality for exporting the test results in CSV file.

The document reports the results and improvement suggestions of the second usability test session performed on 7.5.2019.

Keywords: Usability, usability testing, learning enviroment, project, testing, testing report, report template, TIM.

1. Tested Functionalities, Goals and Methods of Test Session

The test session included two separate functionality. The first tested functionality was creation of the report from test. The second tested functionality was creation of the task with drag & drop question items. Usability testing focused on the discoverability, ease of use and learnability of the tested functionalities. The goal of usability testing was to find shortcomings, errors and problems related to the tested functionalities.

2. Suitability and Knowledge of the Test Subject and Observer

The test subject and the observer met the requirements specified in the test plan. The test subject was a customer and one of the end users of the application. Based on the background information interview (results in Table 1), the test subject knew the concept of adaptive feedback and he was expert user of using ICAnDoiT application. Skills of the test subject from TIM learning environment were moderate. During the test session the test subject expressed his thoughts aloud as he was advised.

3. Identification Information of the Test Session

The test environment and identification information of test session are represented in following table.

4. Summary and Recommendations of Test Session

5. Summary of Observations of the Test Session

The test session consisted of the assignments defined in the usability test plan [1]. Each section consists of one tested functionality. Each section includes observations which were recorded during the test session, analysis of the observations and suggested improvements based on analysis and observations.

Each recorded observation is ranked as one of the following:

• Critical: If a bug or nonintentional functionality is discovered.
• Negative: If the test subject complains or struggles with the assignment or a part of it.
• Positive: If the test subject advances exceptionally fast or they praise the functionality.
• Suggestion: If the observer or the test subject discovers an improvement that could make the application better.
• Neutral: If the observation does not fill the definition of a critical, negative or positive observation and is not a suggestion.

5.1 Creating a Report

During testing of the functionality the test subject was assisted and he used the existing manuals. Testing of the functionality advanced quickly. In the beginning of the assignment 4.1 the test subject tried without the manuals and suggested wrong reporting option. The Excel version was different to the one the test subject was used so the assignment was stopped.

The proposed improvements and their status are presented in the following table:

The recorded observations in assignments are shown in the following tables:

Assignment 4.1 Make a report of whole test from the answers of the person indicated by the observer.

Assignment 4.2 Transfer the information of the report to an Excel file.

Assignment 4.3 Answer the questions asked by the observer.

5.2 Creating Task with Drag&drop Items

During testing of the functionality the test subject was assisted and he used the manuals. Assignment 2.1 was not be completed. The biggest problems were encountered during editing of the feedback block. The marking correctStreak was not self-evident and defining the link to the nextTask was difficult. The test subject wondered also why words of the feedback are empty but understood the logic after assistance. The test subject formed a task using the template for creating the entire task at once. After saving the template the question item blocks will be hidden. Revealing them was difficult for the test subject. Editing the question item blocks and the instruction block was quite fast after one succesful editing. In the end of the test session test subject tested the task he created. An unintentional feature was discovered when the task contained four question item blocks and feedback block defined five. The problem is repaired since. Over all the test subject thought that application was quite good and it is logical. The test subject gave lot of suggestions how to improve the application.

The proposed improvements and their status are presented in the following table:

The recorded observations in assignments are shown in the following tables:

Assignment 2.1 Create the following task containing an instruction block, three question item blocks and a feedback block.

Assignment 2.2 Answer the questions asked by the observer.