Usability Testing Report 20190507 1/2
Hanna Alatalo
Jarkko Kuivanen
Elisa Nauha
Jere Ojala
Kimmo Urtamo
Version 1.0.0
Public
5.6.2019
University of Jyväskylä
Faculty of Information Technology
Info
Title of Document: Usability Testing Report 20190507 1/2
File:
Abstract: The functionalities developed into virtual learning environment called TIM by the TIMCAN project will replace already existing service called ICAnDoiT. ICAnDoiT is an application used by The Centre for Applied Language Studies (CALS) for the purpose of studying adaptive feedback. The adaptive feedback is feedback that becomes gradually more detailed with every wrong response. The project team will develop a dropdown menu plugin different from the original ICAnDoiT, a drag-and-drop plugin, adaptive feedback plugin and functionality for exporting the test results in CSV file.
The document reports the results and improvement suggestions of the first usability test session performed on 7.5.2019.
Keywords: Usability, usability testing, learning enviroment, project, testing, testing report, report template, TIM.
Change History
| Versio | Date | Changes | Author |
|---|---|---|---|
| 0.0.1 | 8.5.2019 | The document was created and tables of the report was filled. | JK |
| 0.0.2 | 13.5.2019 | The document was finished for internal check. | JK |
| 0.1.0 | 14.5.2019 | Typos were fixed. | JK |
| 0.2.0 | 20.5.2019 | Edited document according to suggestions. | JK |
| 0.2.1 | 21.5.2019 | Edited document according to suggestions. | JK |
| 1.0.0 | 5.6.2019 | The document was approved by the customer. | KU |
1. Tested Functionalities, Goals and Methods of Test Session
The test session included two separate functionality. The first tested functionality was answering to the test. The second tested functionality was creation of the task with dropdown question items. Usability testing focused on the discoverability, ease of use and learnability of the tested functionalities. The goal of usability testing was to find shortcomings, errors and problems related to the tested functionalities.
2. Suitability and Knowledge of the Test Subject and Observer
The test subject and the observer met the requirements specified in the test plan. The test subject was a customer and one of the end users of the application. Based on background information interview (results in Table 1), The test subject knew the concept of adaptive feedback and he was skilled user of ICAnDoiT application. Skills of the test subject from TIM learning environment were moderate. During the test session the test subject expressed his thoughts aloud as he was advised.
| Question | Answer |
|---|---|
| The role of test subject: | researcher |
| The test subject knows the concept of adaptive feedback: | yes |
| The test subject has used ICAnDoiT application: | yes |
| The test subject has participated test in ICAnDoiT application: | yes |
| The test subject has created test in ICAnDoiT application before: | yes |
| The test subject could guide novice user to create test in ICAnDoiT application: | maybe |
| The test subject has used TIM learning environment before: | yes |
| The test subject has read TIM documents and know how to navigate in TIM folder system: | yes |
| The test subject has created TIM documents: | no |
| The test subject could guide novice user to create TIM document: | no |
3. Identification Information of the Test Session
The test environment and identification information of the test session are represented in the following table.
| Parameter | Identification information |
|---|---|
| The date of test session: | 7.5.2019 |
| The starting time of test session: | 13:03 |
| The ending time of test session: | 14:16 |
| The observer of test session: | Jarkko Kuivanen |
| The subject of test session: | Ari Huhta |
| The assisting persons: | Elisa Nauha |
| The test plan and version: | Test Plan for Usability Testing in TIMCAN Project version 0.1.3 |
| The place of test session: | Quiet meeting room (Ag C418.1) |
| The software and version: | TIMCAN branch of TIM |
| The operating system and version: | Windows 10 Home 10.0.17134 |
| The web browser and version: | Firefox 66.0.3 |
| The hardware setup: | Laptop with mouse for assignments and laptop computer with mouse for manuals |
| The test server (address): | https://timdevs01-2.it.jyu.fi |
| The functionalities to be tested: | Answering the test and creating the task with dropdown items |
| The test data used for testing: | Test for answering test: https://timdevs01-2.it.jyu.fi/view/users/kuivanen-jarkko/usability-tests/answer-test |
4. Summary and Recommendations of Test Session
| Summary data | Number |
|---|---|
| The number of functionalities tested | 2 |
| The number of assignments in total | 4 |
| The number of completed assignments | 3 |
| The number of unfinished assignments | 1 |
| The number of observations in total | 32 |
| The number of positive observations | 3 |
| The number of negative observations | 12 |
| The number of critical observations | 1 |
| The number of neutral observations | 8 |
| The number of neutral suggestions | 8 |
5. Summary of Observations of the Test Session
Test session consisted of the assignments defined in the usability test plan [1].
Each section consists of one tested functionality. Each section includes observations which were recorded during the test session, analysis of the observations and the suggested improvements based on analysis and the observations.
The recorded observations are ranked as one of the following:
- Critical: If a bug or nonintentional functionality is discovered.
- Negative: If the test subject complains or struggles with the assignment or a part of it.
- Positive: If the test subject advances exceptionally fast or they praise the functionality.
- Suggestion: If the observer or the test subject discovers an improvement that could make the application better.
- Neutral: If the observation does not fill the definition of a critical, negative or positive observation and is not a suggestion.
5.1 Answering a Test
The testing of the functionality went well. During the test an attention was drawn to the permalink character (#) typical to TIM block. The form of the practice question and the level of feedback in the feedback also caused questions. These markings can be changed by the test maker. One undesirable feature was also detected during the test. When you are about to move from a task to another, a popup window will appear asking confirmation whether you want to stay at page. This popup window has been corrected after the test results.
The proposed improvements and their status are presented in the following table:
The recorded observations by assignments are shown in the following tables:
Assignment 3.1 Answer the test given by the observer. Pay attention to visuals while answering.
Assignment 3.2 Answer the questions asked by the observer.
5.2 Creating a Task with Dropdown Items
During testing of the functionality the test subject was assisted and he used manuals. Assignment 1.1 was not completed. The biggest problems were encountered during editing of the feedback block. The marking correctStreak was not self-evident and defining the link to the nextTask was difficult. The test subject did not know how to utilize referring of the levels or default match, which would have reduced writing. The test subject formed a task using the template for creating the entire task at once. After saving a template question item blocks will be hidden. Revealing them was difficult for the test subject as well as finding the feedback block. Editing the question item blocks and the instruction block was quite fast after one succesful editing.
The proposed improvements and their status are presented in the following table:
The recorded observations in assignments are shown in the following tables.
Assignment 1.1 Create the following task containing an instruction block, four question item blocks and a feedback block.
Assignment 1.2 Answer the questions asked by the observer.
References
[1] Hanna Alatalo, Jarkko Kuivanen, Elisa Nauha, Jere Ojala and Kimmo Urtamo, Test Plan for Usability Testing in the TIMCAN Project.
https://tim.jyu.fi/view/kurssit/tie/proj/2019/timcan/dokumentit/testaus/Test-plan-for-usability-testing-in-TIMCAN-project, University of Jyväskylä, Faculty of Information Technology, 2019.
These are the current permissions for this document; please modify if needed. You can always modify these permissions from the manage page.