Functional testing plan of the TIMCAN -project

Hanna Alatalo
Jarkko Kuivanen
Elisa Nauha
Jere Ojala
Kimmo Urtamo

Version 0.0.15
Private
17.4.2019

Jyväskylän yliopisto - University of Jyväskylä
Faculty of Information Technology
Jyväskylä

Abstract: The functional testing plan of the TIMCAN -project describes what functionalities are to be tested, how testing is executed and how the testing shall be reported.

Keywords: Test plan, functional testing, TIM, testing, The Centre for Applied Language Studies, CALS, adaptive feedback, dynamic feedback, virtual learning environment, plugin

1. Target Application

Target Application and test targets has been defined in the Test plan of the TIMCAN -project^[1].

2. Objectives of the test session and methods of execution

2.1 Objectives

General objectives of testing are defined in the Test plan of the TIMCAN -project^[1]. In functional testing the most essential features are being tested with different inputs. Objective of this is to ensure that system always functions in use the same way without errors.

2.2 Execution

Functional testing shall be executed as a black-box testing manually.

3. Test run

Only the person who is executing the test shall participate in a test run. Tester should possess basic understanding of operating the Windows Operating System. Tester should also possess basic understanding of operating TIM-system.

3.1 Testing environment

Testing environment has been defined in the Test plan of the TIMCAN -project^[1].

3.2 Testing report

In testing report should include from test session its identification and summary.

Required identification info of test session has been defined in the Test plan of the TIMCAN -project^[1].

Summary should include:

• number of executed test cases,
• number of test cases not executed,
• number of every different conclusion separately specified,
• exceptions from the test plan and their justifications and
• recommended procedures after the test session.

Report of the test cases is compiled on table form.

Conclusion must be one of the following:

• OK is the result when test case has succeeded with end result matches with the expected result.
• Error is the result when end result differs from the expected result or something generally harmful is noticed during the test run.
• Note is the result when tester has noticed abnormality in functionality.

In the case of Error and Note it should be reported with regarding details, such as procedures, input, abnormalities and possible error messages that led to the result.

3.2.1 Example of test session report table

3.3 Test cases

Numbering of the test cases are in form of [Case ID].[Subcase ID].

4. Test cases for building a test

4.1 Building a test - instruction page

Test case: 1.1
Initial state: User is signed in to TIM development machine
Steps:

1. Create a new folder in your documents called TestausTesti1
2. In manage mode set the default rights for new documents in the folder to view for logged-in users.
3. In the folder create a document called TestInstructions
4. Go to the "manage" mode and copy the instructions markup in the "edit full document" box and click save.
5. Go to the "view" mode.

Test data:

# Welcome to the test!

This is the instructions for the whole test. Please click below to begin test.

[Begin test](test1task1)

Expected result:

4.2 Building a test - task 1

Test case: 1.2
Initial state: User is signed in to TIM development machine and has done previous test case.
Steps:

1. Go to folder TestausTesti1 in your documents
2. In the folder create a document called Test1Task1
3. Go to the "manage" mode and copy the task markup from appendix 1 into the "edit full document" box and click save.
4. Set the view rights for all signed in users.
5. Go to the "view" mode.
6. In the view mode left click on the right side of the feedback paragraph and edit the paragraph.
7. Find the line "nextTask:" and write in "md: [Task 2](test1task2)".

Test data: see Appendix 1
Expected result: Test is ready and looks like so:

4.3 Building a test - task 2

Test case: 1.3
Initial state: User is signed in to TIM development machine and has excecuted test case 1.2.
Steps:

1. Go to folder TestausTesti1 in your documents
2. In the folder create a document called Test1Task2
3. Go to the "manage" mode and copy the task markup from appendix 2 into the "edit full document" box and click save.
4. Set the view rights for all signed in users.
5. Go to the "view" mode.
6. In the view mode left click on the right side of the feedback paragraph and edit the paragraph.
7. Find the line "nextTask:" and write in "md: [End of test](endoftest)".

Test data: see Appendix 1
Expected result: Test is ready and looks like so:

4.4 Building a test - end page of test

Test case: 1.4
Initial State: Previous test cases have been completed successfully
Steps:

1. Go to the folder
2. Create new document called EndOfTest
3. Add a paragraph with the test data.
4. Go to the manage mode and set the view rights for all signed in users.
5. Go to view mode.

Test Data:

# Thank you for taking part in the test!

Expected Result:

In view mode the page should look like below:

The test folder should look like below:

5. Test cases for student answering the test

5.1 Beginning a test as student

Test case: 2.1
Initial State: The test has been successfully built in the previous test cases.
Steps:

1. Sign in to TIM as test user (username: test1@example.com and password: test1).
2. Go to page of test https://timdevs01-2.it.jyu.fi/view/users/*username*/testaustesti1/testinstructions
3. Click Begin test

Test Data:
Expected Result: Instruction page for test is visible.

5.2 Answering the task until stopping condition for correct streak

Test case: 2.2
Initial State: The previous test cases have been successfully completed.
Steps:

Test Data: None
Expected Result: See individual steps for expected results.

5.3 Answering the task until stopping condition for final feedback level reached

Test case: 2.3
Initial State: The previous test cases have been successfully completed.
Steps:

Test Data: Example wrong answers for questions in task 2: "if run I a mile", "who at work I see", "had whether I a computer", and ""when come I around". 

Expected Result: See individual steps for expected results.

5.4 Answering the test and not choosing options

Test case: 2.4
Initial State: Previous test cases have been successfully completed.
Steps:

1. Sign in as another test user (username: test2@example.com and password: test2).
2. Go to page of test https://timdevs01-2.it.jyu.fi/view/users/*username*/testaustesti1/testinstructions
3. Click Begin test

Test Data: None
Expected Result: See individual steps for expected results.

6. Report exporting test cases

6.1 Exporting test results with default options

Test case: 3.1
Initial State: Previous test cases have been successfully completed.
Steps:

1. Sign in as user who created the test.
2. Go to page of test https://timdevs01-2.it.jyu.fi/answers/users/*username*/testaustesti1/test1task1.
3. Expand <number> users with answers-dialog if it is substracted.

Test Data:

Export Options:
Period: Whenever
Validity: Valid
Names: Username and fullname
Scope: Only this task
Answers From: All users
Format: Comma

Expected Result: See individual steps for expected results.

6.2 Canceling exporting

Test case: 3.2
Initial State: Previous test cases have been successfully completed.
Steps:

1. Sign in as user who created the test.
2. Go to page of test https://timdevs01-2.it.jyu.fi/answers/users/nauha-elisa/testaustesti1/test1task1.
3. Expand <number> users with answers-dialog if it is substracted.

Test Data:

Export Options:
Period: Whenever
Validity: Valid
Names: Username and fullname
Answers From: All users
Format: Comma

Expected Result: See individual steps for expected results.

6.3 Exporting results of the whole test and with only username

Test case: 3.3
Initial State: Previous test cases have been successfully completed.
Steps:

1. Sign in as user who created the test.
2. Go to page of test https://timdevs01-2.it.jyu.fi/answers/users/nauha-elisa/testaustesti1/test1task1.
3. Expand <number> users with answers-dialog if it is substracted.

Test Data:

Export Options:
Period: Whenever
Validity: Valid
Names: Username only
Scope: The whole test
Answers From: All users
Format: Comma

Expected Result: See individual steps for expected results.

6.4 Exporting results of the whole test and with anonymous username

Test case: 3.4
Initial State: Previous test cases have been successfully completed.
Steps:

1. Sign in as user who created the test.
2. Go to page of test https://timdevs01-2.it.jyu.fi/answers/users/nauha-elisa/testaustesti1/test1task1.
3. Expand <number> users with answers-dialog if it is substracted.

Test Data:

Export Options:
Period: Whenever
Validity: Valid
Names: Anonymous username
Scope: The whole test
Answers From: All users
Format: Comma

Expected Result: See individual steps for expected results.

6.5 Exporting test results with Tab-delimiters

Test case: 3.5
Initial State: Previous test cases have been successfully completed.
Steps:

1. Sign in as user who created the test.
2. Go to page of test https://timdevs01-2.it.jyu.fi/answers/users/nauha-elisa/testaustesti1/test1task1.
3. Expand <number> users with answers-dialog if it is substracted.

Test Data:

Export Options:
Period: Whenever
Validity: Valid
Names: Username only
Scope: The whole test
Answers From: All users
Format: Tab

Expected Result: See individual steps for expected results.

6.6 Exporting test results with semicolon-delimiters

Test case: 3.6
Initial State: Previous test cases have been successfully completed.
Steps:

1. Sign in as user who created the test.
2. Go to page of test https://timdevs01-2.it.jyu.fi/answers/users/nauha-elisa/testaustesti1/test1task1.
3. Expand <number> users with answers-dialog if it is substracted.

Test Data:

Export Options:
Period: Whenever
Validity: Valid
Names: Username only
Scope: The whole test
Answers From: All users
Format: Semicolon

Expected Result: See individual steps for expected results.

6.7 Exporting test results with only one visible user

Test case: 3.7
Initial State: Previous test cases have been successfully completed.
Steps:

1. Sign in as user who created the test.
2. Go to page of test https://timdevs01-2.it.jyu.fi/answers/users/nauha-elisa/testaustesti1/test1task1.
3. Expand <number> users with answers-dialog if it is substracted.

Test Data:

Export Options:
Period: Whenever
Validity: Valid
Names: Username only
Scope: The whole test
Answers From: Only visible users
Format: Semicolon

Expected Result: See individual steps for expected results.

6.8 Exporting test results with two visible users

Test case: 3.8
Initial State: Previous test cases have been successfully completed.
Steps:

1. Sign in as user who created the test.
2. Go to page of test https://timdevs01-2.it.jyu.fi/answers/users/nauha-elisa/testaustesti1/test1task1.
3. Expand <number> users with answers-dialog if it is substracted.

Test Data:

Export Options:
Period: Whenever
Validity: Valid
Names: Username only
Scope: The whole test
Answers From: Only visible users
Format: Semicolon

Expected Result: See individual steps for expected results.

6.9 Exporting test results with only visible users when visible user list is empty

Test case: 3.9
Initial State: Previous test cases have been successfully completed.
Steps:

1. Sign in as user who created the test.
2. Go to page of test https://timdevs01-2.it.jyu.fi/answers/users/nauha-elisa/testaustesti1/test1task1.
3. Expand <number> users with answers-dialog if it is substracted.

Test Data:

Export Options:
Period: Whenever
Validity: Valid
Names: Username only
Scope: The whole test
Answers From: Only visible users
Format: Semicolon

Expected Result: See individual steps for expected results.

6.10 Exporting test results with only selected user

Test case: 3.10
Initial State: Previous test cases have been successfully completed.
Steps:

1. Sign in as user who created the test.
2. Go to page of test https://timdevs01-2.it.jyu.fi/answers/users/nauha-elisa/testaustesti1/test1task1.
3. Expand <number> users with answers-dialog if it is substracted.

Test Data:

Export Options:
Period: Whenever
Validity: Valid
Names: Username only
Scope: The whole test
Answers From: Only selected user
Format: Semicolon

Expected Result: See individual steps for expected results.

6.11 Exporting test results with only selected user when visible user list is empty

Test case: 3.11
Initial State: Previous test cases have been successfully completed.
Steps:

1. Sign in as user who created the test.
2. Go to page of test https://timdevs01-2.it.jyu.fi/answers/users/nauha-elisa/testaustesti1/test1task1.
3. Expand <number> users with answers-dialog if it is substracted.

Test Data:

Export Options:
Period: Whenever
Validity: Valid
Names: Username only
Scope: The whole test
Answers From: Only selected user
Format: Semicolon

Expected Result: See individual steps for expected results.

7. Building task with errors

7.1 Building task with varying number of feedback levels

Test case: 4.1
Initial State: A folder called TestausTesti1 has been created.
Steps:

1. Go to folder TestausTesti1 in your documents
2. In the folder create a document called Test1Task3
3. Go to the "manage" mode and copy the task markup into the "edit full document" box and click save.
4. Go to "view" mode and click to edit the feedback paragraph.
5. Delete a feedback level from one match option such as: - "md: Level 2 feedback: You answered: *|answer|* Answer is wrong. Try thinking *a bit* harder. "
6. Click save. Expected result: Gives red warning: "Different number of feedback levels"
7. Go to "view" mode and click to edit the feedback paragraph.
8. Paste the deleted feedback level back in the right place.
9. Click save. Expected result: Error disappears.

Test Data: None
Expected Result: In step 6 an error should appear and be recovered from in step 9.

7.2 Building task with no link to next task/end of test

Test case: 4.2
Initial State: An errorless task Test1Task3 has been created (test case 4.1)
Steps:

1. Click to edit the feedback paragraph of document Test1Task3.
2. Delete the input of nextTask:
3. Click save. Expected result: Gives red warning: "The following fields have invalid values: nextTask: Field may not be null."
4. Click to edit the feedback paragraph.
5. Reinput "link" into the nextTask: -field.
6. Click save. Expected result: Error should disappear.

Test Data: None
Expected Result: In step 3 and error should be dispalyed and it should be recovered from in step 6.

7.3 Building task with no instruction paragraph plugin

Test case: 4.3
Initial State: An errorless task Test1Task3 has been created (test case 4.1)
Steps:

1. Click to edit the instructions paragraph of document Test1Task3.
2. Delete the instruction plugin {#instruction words: [will think, won't think, might think]} and click save.
3. Click to edit the feedback paragraph and click save. Expected result: Gives red warning: "Instruction not defined"
4. Click to edit the instructions paragraph.
5. Reinput the instruction plugin and click save.
6. Click to edit the feedback paragraph and click save. Expected result: Error should disappear.

Test Data: None
Expected Result: In step 3 and error should be dispalyed and it should be recovered from in step 6.

7.4 Building task with no right answer defined

Test case: 4.4
Initial State: An errorless task Test1Task3 has been created (test case 4.1)
Steps:

1. Click to edit the feedback paragraph of document Test1Task3.
2. Delete the lines:
   

    - match: [is baking]
      correct: true
      levels: *ismatch

3. Click save. Expected result: Gives red warning: "No correct answer defined for question item.."
4. Click to edit the feedback paragraph.
5. Reinput the deleted text.
6. Click save. Expected result: Error should disappear.

Test Data: None
Expected Result: In step 3 and error should be dispalyed and it should be recovered from in step 6.

7.5 Building task with wrong names for plugins

Test case: 4.5
Initial State: An errorless task Test1Task3 has been created (test case 4.1)
Steps:

1. Click to edit the feedback paragraph of document Test1Task3.
2. Change the pluginname drop1 on line 7 to nodrop1
3. Click save. Expected result: Gives red warning: "No plugin with such a name (nodrop1) or missing setPluginWords-method"
4. Click to edit the feedback paragraph.
5. Change the pluginname nodrop1 on line 7 back to drop1
6. Click save. Expected result: Error should disappear.

Test Data: None
Expected Result: In step 3 and error should be dispalyed and it should be recovered from in step 6.

7.6 Building task with no default feedback

Test case: 4.6
Initial State: An errorless task Test1Task3 has been created (test case 4.1)
Steps:

1. Click to edit the feedback paragraph of document Test1Task3.
2. Delete lines:

    - match: []
      levels: *defaultmatch

3. Click save. Expected result: Gives red warning: "Default feedback option needed"
4. Click to edit the feedback paragraph.
5. Reinput the deleted text.
6. Click save. Expected result: Error should disappear.

Test Data: None
Expected Result: In step 3 and error should be dispalyed and it should be recovered from in step 6.

7.7 Building task with no feedback

Test case: 4.7
Initial State: An errorless task Test1Task3 has been created (test case 4.1)
Steps:

1. Click to edit the feedback paragraph of document Test1Task3.
2. Delete lines:

  choices:
    - match: [is baking]
      correct: true
      levels: *ismatch
    - match: [do baking]
      levels: *domatch
    - match: [are baking]
      levels: *arematch
    - match: []
      levels: *defaultmatch

3. Click save. Expected result: Gives red warning: "Feedback choices needed"
4. Click to edit the feedback paragraph.
5. Reinput the deleted text.
6. Click save. Expected result: Error should disappear.

Test Data: None
Expected Result: In step 3 and error should be dispalyed and it should be recovered from in step 6.

## Instructions {defaultplugin="dropdown"}

Welcome to the test. Read the question carefully. If you get the answer wrong, please read the feedback carefully.


Please try out a practice question:


I {#instruction words: [will think, won't think, might think]} before answering.


## Item: {defaultplugin="dropdown"}

What {#drop1} on the stove?


## Item: {defaultplugin="dropdown"}

Who {#drop2} the cake?


## Item: {defaultplugin="dropdown"}

What {#drop3} on the roof?


## Item: {defaultplugin="dropdown"}

Who {#drop4} the 3 mile swim in the race?
 

``` {#fb1 plugin="feedback"}
correctStreak: 2
nextTask: nonexttaskdefined
instructionID: instruction
teacherHide: true
questionItems:
- pluginNames: [drop1]
  words: [[is cooking, do cooking, are cooking]]
  choices:
    - match: [is cooking]
      correct: true
      levels: &ismatch
        - "**Correct!** You answered: |answer|"
    - match: [do cooking]
      levels: &domatch
        - "Level 1 feedback: You answered: *|answer|* Answer is wrong. Try **thinking**. "
        - "Level 2 feedback: You answered: *|answer|* Answer is wrong. Try thinking *a bit* harder. "
        - "Level 3 feedback: You answered: *|answer|* Answer is wrong. Try thinking **much more** harder. "
        - "Level 4 feedback: You answered: *|answer|* Answer is wrong. Just think about what the answer **is**. "
        - "Level 5 feedback: Please note the correct answer: 'What **is** / Who **is**'"
    - match: [are cooking]
      levels: &arematch
        - "Level 1 feedback: You answered: *|answer|* Answer is wrong. Try **thinking**. "
        - "Level 2 feedback: You answered: *|answer|* Answer is wrong. Try thinking *a bit* harder. "
        - "Level 3 feedback: You answered: *|answer|* Answer is wrong. Try thinking **much more** harder. "
        - "Level 4 feedback: You answered: *|answer|* Answer is wrong. Just think about what the answer **is**. "
        - "Level 5 feedback: Please note the correct answer: 'What **is** / Who **is**'"
    - match: []
      levels: &defaultmatch
        - "Level 1 feedback: default feedback for drop4"
        - "Level 2 feedback: default feedback for drop4"
        - "Level 3 feedback: default feedback for drop4"
        - "Level 4 feedback: default feedback for drop4"
        - "Level 5 feedback: Please note the correct answer: 'What **is** / Who **is**'"
- pluginNames: [drop2]
  words: [[is baking, do baking, are baking]]
  choices:
    - match: [is baking]
      correct: true
      levels: *ismatch
    - match: [do baking]
      levels: *domatch
    - match: [are baking]
      levels: *arematch
    - match: []
      levels: *defaultmatch
- pluginNames: [drop3]
  words: [[is jumping, do jumping, are jumping]]
  choices:
    - match: [is jumping]
      correct: true
      levels: *ismatch
    - match: [do jumping]
      levels: *domatch
    - match: [are jumping]
      levels: *arematch
    - match: []
      levels: *defaultmatch
- pluginNames: [drop4]
  words: [[is swimming, do swimming, are swimming]]
  choices:
    - match: [is swimming]
      correct: true
      levels: *ismatch
    - match: [do swimming]
      levels: *domatch
    - match: [are swimming]
      levels: *arematch
    - match: []
      levels: *defaultmatch
```