A-Level Computer Science / Unit 12: Software Design, Testing and Evolution

12.3.5 Test Strategies, Test Plans and Test Data

🔒 Lesson slides are available to signed-in users. Sign in

12.3.5 Test Strategies, Test Plans and Test Data

A large program cannot realistically be checked with every possible input, configuration and sequence of actions. Testing therefore needs to be planned so that important requirements, components, paths, interfaces and user activities receive suitable attention.

A test strategy defines the overall approach. A test plan turns that approach into repeatable test cases containing exact data and predicted results. Carefully selected normal, abnormal and extreme or boundary values help the tester examine different kinds of behaviour without trying every possible value.

By the end of this section, you should be able to:

  • Explain why systematic testing needs an overall strategy.
  • Distinguish between a test strategy, a test plan and a test case.
  • Describe likely contents of a test strategy and test plan.
  • Connect test cases to requirements and identified risks.
  • Distinguish normal, abnormal and extreme or boundary data.
  • Select exact values for range, type, format and length rules.
  • State an expected result before a test is executed.
  • Record actual results, outcomes, faults and retests clearly.

Why Testing Must Be Systematic

Unplanned testing often concentrates on the easiest or most familiar parts of a program. Important requirements may receive no test at all, while the same successful route is repeated several times.

Problem with informal testing How planning helps
Only typical successful inputs are tried. Different data categories and failure cases are selected deliberately.
Some requirements are forgotten. Every test is linked to a requirement or testing objective.
The expected result is decided after the program runs. Expected behaviour is written before execution.
Different testers perform different steps. Preconditions, data and actions are documented precisely.
A failed test cannot be reproduced. The environment, version, data and actual result are recorded.
Corrections are not checked properly. Failed tests and related regression tests are scheduled again.
Common misconception: Systematic testing does not mean trying every possible value. It means selecting and documenting tests for clear reasons.

Test Strategy

Test strategy: the high-level approach that explains what will be tested, how testing will be organised and which methods, environments and priorities will be used.

The strategy provides direction for the whole testing process. It should be considered while the system is being analysed and designed rather than being invented only after coding is complete.

A strategy answers broad questions:

  • Which parts of the system are included in testing?
  • Which requirements or risks are most important?
  • Which testing methods will be used?
  • At which development stages will tests occur?
  • Who is responsible for each type of testing?
  • Which devices, operating systems or configurations are needed?
  • What evidence will be recorded?
  • What must be true before testing starts or finishes?
Exam tip: A strategy is broad. Do not describe it as one row of data with one expected result.

Likely Contents of a Test Strategy

Strategy area Question answered Observatory example
Objectives What confidence should testing provide? Demonstrate that bookings obey capacity, duration and participant rules.
Scope Which features are included or excluded? Include booking, cancellation and retrieval; exclude the future payment service.
Risk and priority Which failures would have the greatest effect? Prevent overbooking and duplicate booking references.
Methods Which testing methods are suitable? Module, integration, black-box, alpha and acceptance testing.
Stages and sequence When will each type of testing occur? Test validation modules first, then booking integration and complete user journeys.
Test environment Which hardware and software configurations are needed? Supported desktop and mobile browsers with a separate test database.
Responsibilities Who prepares, executes and reviews tests? Developers test modules; the quality team tests the integrated portal; observatory staff perform acceptance testing.
Test data What kinds of data must be prepared? Normal bookings, invalid formats and values around each permitted limit.
Entry criteria What must be ready before a test stage begins? The booking build is installed and the test database has known capacity.
Exit criteria What conditions indicate that the stage can finish? Mandatory tests pass and no unresolved critical-capacity fault remains.
Retesting How will corrections and existing features be checked? Repeat the failed case and related booking-capacity regression tests.
Evidence What records will be retained? Test results, fault references, browser details and retest outcomes.
Common mistake: Listing every possible testing method is not a strategy. The methods should be selected and justified for the particular system.

Strategy Example: Community Observatory Portal

A community observatory is developing a portal for reserving guided telescope sessions. Each session has limited capacity and lasts for a permitted duration.

Selected rules

Requirement Rule
R1 Group size must be a whole number from 2 to 12 inclusive.
R2 Session duration must be from 30 to 180 minutes inclusive and must be a multiple of 15.
R3 A booking reference must contain exactly eight uppercase letters or digits.
R4 A booking must not be created when the requested group is larger than the remaining capacity.
R5 Cancelling a future booking must return its places to the session.

Condensed strategy

Area Decision
Highest risks Overbooking, duplicate references and places not being restored after cancellation
Early testing Test validation and reference-generation modules separately.
Integration testing Connect validation, capacity, storage and confirmation modules gradually.
Black-box testing Derive complete booking tests from R1–R5.
Environments Supported mobile and desktop browsers using a resettable test database.
Release checks Internal alpha testing followed by observatory staff acceptance testing.
Completion condition All mandatory tests pass and no unresolved fault can create an invalid booking.
Exam tip: Link strategy decisions to risks and requirements. For example, capacity tests receive high priority because overbooking affects several users and stored records.

Test Plans

Test plan: a documented collection of specific tests that states what will be tested, how each test will be performed and how its result will be judged.

The strategy gives the overall direction. The test plan contains the concrete tests needed to implement that strategy.

Test case: one repeatable test within a test plan, including its conditions, exact data, actions and expected result.
Test strategy Test plan Test case
Overall approach for the project Organised collection of detailed tests One specified test
Describes scope, methods, stages and responsibilities Lists features, environments and cases Contains exact data, steps and expected behaviour
Changes relatively infrequently Updated as requirements and builds change Executed and recorded individually
Common misconception: A test plan is not merely a list of input values. Without an expected result and clear purpose, the tester cannot judge the behaviour consistently.

Likely Contents of a Test Case

Field Purpose Example
Test ID Provides a unique reference. OBS-GRP-04
Requirement reference Shows why the test exists. R1
Feature or component Identifies the area under test. Group-size validation
Purpose States the particular behaviour being checked. Check the upper accepted group size.
Preconditions Defines the required starting state. A session with at least 12 remaining places
Test data Gives exact values. GroupSize = 12
Test steps Explains the actions to perform. Enter 12 and submit the booking form.
Expected result States the required observable behaviour. The value is accepted and the booking can continue.
Actual result Records what happened during execution. Completed by the tester
Outcome Compares actual with expected. Pass or fail
Environment and version Supports repetition of the test. Build 0.8.4, mobile browser
Fault reference Links a failure to its recorded defect. DEF-027
Retest result Records the result after correction. Passed in build 0.8.5

Fields completed at different times

Before execution During or after execution
Test ID Actual result
Requirement and purpose Pass or fail
Preconditions and test data Fault reference
Steps and expected result Retest result

Connecting Requirements to Tests

A requirements-coverage table helps the team check that each important requirement has at least one relevant test.

Requirement Tests Coverage purpose
R1: group size 2–12 OBS-GRP-01 to OBS-GRP-07 Typical values, limits, adjacent invalid values and invalid types
R2: duration 30–180, multiple of 15 OBS-DUR-01 to OBS-DUR-08 Range limits, step rule and invalid type
R3: eight-character reference OBS-REF-01 to OBS-REF-06 Length, permitted characters and case
R4: capacity cannot be exceeded OBS-CAP-01 to OBS-CAP-04 Enough capacity, exact capacity and insufficient capacity
R5: cancellation restores places OBS-CAN-01 to OBS-CAN-03 Valid cancellation, repeated cancellation and expired booking
Common mistake: Having one test for each requirement does not automatically provide enough coverage. A requirement containing several rules may need multiple tests.

Test Data

Test data: deliberately selected input values and starting conditions used to examine a particular aspect of program behaviour.

Test values should be chosen because they check a stated rule. Random values may be useful for additional exploration, but they do not replace purposeful test design.

Category Main idea Group-size example
Normal A typical valid value that should be accepted 6
Abnormal An invalid value or form of data that should be rejected "six" or 6.5
Extreme A lowest or highest valid value 2 and 12
Boundary Values at and immediately around a limit 1, 2, 3, 11, 12, 13
Important: Categories can overlap. For example, 1 is invalid and is also useful as a just-outside boundary test. State the purpose for which the value was selected.

Normal Test Data

Normal data: valid, representative data that should be accepted and processed normally.

Normal data demonstrates that ordinary use works. It should not always be placed extremely close to a limit.

Rule Normal value Expected result
Group size 2–12 6 Accepted
Duration 30–180 in multiples of 15 90 Accepted
Eight uppercase letters or digits SKY7A2Q9 Accepted
Exam tip: Supply an exact valid value rather than writing only “a normal number.”

Abnormal Test Data

Abnormal data: data that violates a requirement and should be rejected or handled safely.

Different abnormal values should target different reasons for rejection.

Rule being checked Abnormal data Reason for rejection
Group size must be an integer. 6.5 Not a whole number
Group size must be numeric. "six" Wrong data type or format
Duration must be a multiple of 15. 50 Inside the range but violates the step rule
Reference contains uppercase letters or digits. SKY#A2Q9 Contains a prohibited symbol
Reference must use uppercase characters. sky7a2q9 Uses lowercase letters
Common mistake: A value can lie inside the numeric range and still be abnormal because it violates a type, format or step requirement.

Extreme and Boundary Test Data

Extreme data: valid values at the lowest or highest accepted limit.
Boundary data: values at and immediately around the points where program behaviour should change.

Integer range

For a group size from 2 to 12 inclusive:

Value Position Expected result
1 Immediately below the minimum Rejected
2 Minimum extreme value Accepted
3 Immediately inside the lower range Accepted
11 Immediately inside the upper range Accepted
12 Maximum extreme value Accepted
13 Immediately above the maximum Rejected

Length rule

When a reference must contain exactly eight characters, the useful lengths are:

Length Purpose Expected result
7 characters Immediately below the required length Rejected
8 characters Exact accepted boundary Accepted when all characters are permitted
9 characters Immediately above the required length Rejected

Choosing “just outside” for real numbers

The smallest useful difference depends on the precision allowed by the requirement. If a temperature is stored to one decimal place and the maximum is 25.0, a value of 25.1 is immediately outside. A value such as 25.0001 would be unsuitable when the system accepts only one decimal place.

Exam tip: For inclusive limits, the minimum and maximum themselves are accepted. Values immediately outside are rejected.

Choosing High-quality Test Data

Start with the complete rule

A field may have several independent requirements:

GroupSize must be numeric, an integer and from 2 to 12 inclusive.

A suitable data set checks each part:

Value Purpose Expected result
6 Typical valid value Accepted
2 Minimum accepted value Accepted
1 Immediately below the minimum Rejected
12 Maximum accepted value Accepted
13 Immediately above the maximum Rejected
6.5 Invalid integer requirement Rejected
"six" Invalid numeric requirement Rejected

Prefer one clear purpose per test

Suppose a reference must contain exactly eight uppercase letters or digits. The value ab#1 is too short, lowercase and contains a prohibited symbol. Its rejection does not reveal which validation rule was applied.

Clearer separate tests include:

  • SKY7A2Q — seven permitted characters;
  • sky7a2q9 — correct length but lowercase;
  • SKY#A2Q9 — correct length but contains a symbol.
Common mistake: One heavily invalid value may test several rules at once and make the cause of rejection unclear.

Worked Example: Observatory Booking Test Plan

Validation test cases

ID Req. Purpose Test data Category Expected result
OBS-GRP-01 R1 Check a typical valid group GroupSize = 6 Normal Accepted
OBS-GRP-02 R1 Check the minimum group GroupSize = 2 Extreme/boundary Accepted
OBS-GRP-03 R1 Check immediately below the minimum GroupSize = 1 Boundary, invalid Rejected with a group-size message
OBS-GRP-04 R1 Check the maximum group GroupSize = 12 Extreme/boundary Accepted
OBS-GRP-05 R1 Check immediately above the maximum GroupSize = 13 Boundary, invalid Rejected with a group-size message
OBS-GRP-06 R1 Check integer validation GroupSize = 6.5 Abnormal Rejected because a whole number is required
OBS-DUR-01 R2 Check a typical permitted duration Duration = 90 Normal Accepted
OBS-DUR-02 R2 Check the lower duration limit Duration = 30 Extreme/boundary Accepted
OBS-DUR-03 R2 Check the step requirement Duration = 50 Abnormal Rejected because duration must be a multiple of 15
OBS-REF-01 R3 Check a valid reference SKY7A2Q9 Normal Accepted
OBS-REF-02 R3 Check a reference one character too short SKY7A2Q Boundary, invalid Rejected because eight characters are required
OBS-REF-03 R3 Check permitted characters SKY#A2Q9 Abnormal Rejected because # is not permitted

Functional test cases

ID Req. Precondition Action Expected result
OBS-CAP-01 R4 Six places remain. Request four places. Booking created and remaining capacity becomes two.
OBS-CAP-02 R4 Four places remain. Request four places. Booking created and remaining capacity becomes zero.
OBS-CAP-03 R4 Four places remain. Request five places. Request rejected, no booking created and capacity remains four.
OBS-CAN-01 R5 A future booking for three visitors exists; two other places remain. Cancel the booking. Status becomes cancelled and remaining capacity becomes five.
OBS-CAN-02 R5 The same booking is already cancelled. Submit cancellation again. Request rejected and capacity remains unchanged.
Exam tip: Functional tests often require a precondition. Capacity behaviour cannot be judged unless the starting capacity is known.

Executing, Recording and Retesting

When the test is run, the tester records the actual result and compares it with the result written in advance.

ID Expected result Actual result Outcome Follow-up
OBS-GRP-04 Group size 12 is accepted. Group size 12 was accepted. Pass None
OBS-CAP-03 Request rejected and capacity remains four. Booking created and capacity became −1. Fail Record DEF-031.
OBS-CAN-01 Cancellation restores three places. Booking was cancelled but capacity remained two. Fail Record DEF-032.

After a correction

  1. Repeat the test that originally failed.
  2. Confirm that the expected and actual results now agree.
  3. Repeat related tests that may have been affected.
  4. Record the new build or version.
  5. Update the fault and test records.
Fault Direct retest Related regression tests
Capacity can become negative OBS-CAP-03 OBS-CAP-01 and OBS-CAP-02
Cancellation does not restore places OBS-CAN-01 OBS-CAN-02 and capacity tests
Common mistake: Changing the expected result to match incorrect program behaviour makes the test meaningless. Change the requirement only through an agreed requirements process.

Interactive: Strategy and Test-data Planner

Choose a validation rule, explore purposeful data values and add selected cases to a growing test plan. The widget explains the category, reason and expected result for each value.

Validation rule

Group size

The group size must be a whole number from 2 to 12 inclusive.

INTEGER and 2 ≤ GroupSize ≤ 12
Selected test

Value: 6

This is a typical valid value and checks normal processing.

Expected: accepted
Category: normal

Selected test case

Test Requirement Purpose Data Type Expected
OBS-GRP-01 R1 Check a typical valid value 6 Normal Accepted

Growing test plan

Coverage: 0 test cases added.

Test Rule Data Category Expected
Select a value and add it to the plan.

Common Mistakes and Misconceptions

  • Confusing the strategy with the plan. The strategy is high-level; the plan contains detailed repeatable tests.
  • Writing “test the system thoroughly.” State which features, risks, methods and environments will be used.
  • Giving data without a purpose. Explain which rule each value checks.
  • Omitting the expected result. A test cannot be judged consistently without a predicted outcome.
  • Using only normal values. Invalid data and limits may expose different faults.
  • Calling every invalid value abnormal and ignoring its boundary purpose. A just-outside value may be both invalid and a boundary test.
  • Testing only the exact limits. Include appropriate values immediately inside and outside them.
  • Choosing the wrong adjacent value. Consider whether the field uses integers, characters or a stated decimal precision.
  • Combining several invalid reasons in one test. Separate tests make the cause of rejection clearer.
  • Ignoring preconditions. Functional tests such as capacity checks need a known starting state.
  • Changing the expected result after a failure. The expectation should remain based on the agreed requirement.
  • Recording only pass or fail. Record the actual behaviour and a fault reference when the test fails.

Practice

Core questions

  1. Explain why a test strategy is required.
  2. State six likely contents of a test strategy.
  3. Distinguish between a test strategy and a test plan.
  4. State six likely fields in a detailed test case.
  5. Explain why expected results are recorded before execution.
  6. Define normal test data.
  7. Define abnormal test data.
  8. Distinguish extreme data from a broader set of boundary data.
  9. Explain why requirements should be linked to test cases.
  10. Explain what should be recorded after a test fails.

Scenario A: Planetarium Audio-guide Hire

The number of audio guides must be a whole number from 1 to 20 inclusive.

  1. Give two normal values.
  2. Give the minimum and maximum extreme values.
  3. Give values immediately inside and outside both limits.
  4. Give two abnormal values that test different invalid reasons.
  5. State the expected result for every value.
  6. Write two complete test-plan rows.

Scenario B: Archive Identifier

An archive identifier must contain exactly ten characters. The first two must be uppercase letters and the remaining eight must be digits.

  1. Give one normal identifier.
  2. Give boundary tests for the complete length.
  3. Give one test for lowercase letters.
  4. Give one test for a letter in the numeric section.
  5. Explain why one short lowercase identifier would be a weak test.
  6. Create a requirement-coverage table for the format rules.

Scenario C: Community Laboratory Session

A session can accept at most 18 participants. A booking may request 1 to 6 places, but it must also fit within the remaining capacity.

  1. Write a short test strategy covering the highest risks and suitable methods.
  2. Write tests for request-size validation.
  3. Write tests for exact remaining capacity.
  4. Write a test in which the requested places exceed the capacity by one.
  5. State the required precondition for each capacity test.
  6. Explain which tests should be repeated after correcting an overbooking fault.
Challenge: Design a test plan containing at least eight cases for a field with range, type and format rules. Ensure that each test has one clear primary purpose.

Review

Prompt A strong response should include
Test strategy Overall scope, priorities, methods, stages, environments, responsibilities and completion conditions
Test plan An organised collection of detailed, repeatable test cases
Test case ID, purpose, preconditions, exact data, steps, expected result, actual result and outcome
Normal data A representative valid value that should be accepted
Abnormal data Invalid data selected to check rejection or safe handling
Extreme data The minimum or maximum valid value
Boundary data Values at and immediately around a change in expected behaviour
Requirement coverage Evidence that important requirements are connected to suitable tests
Failed test Expected result, actual result, environment, fault reference and planned retest
Final exam tip: Use the chain requirement → test purpose → exact data → expected result → actual result → outcome.