Manual testing to automatic testing A formal training programme for testers at Bands 3 to 7

Calibration guide

This guide is for participants and line managers. It explains how to complete the capability self-assessment at each gate and how to agree ratings. It follows the spec (Capability self-assessment) and the roles-skills self-assessment guide.

First, the most important points

  • The gates are developmental only. A gate result never starts a capability, performance, or conduct procedure by itself. Any such procedure follows the organisation's HR policy, separately (spec Principle 14).
  • A gap is not a failing. Gaps are expected. They show where development is most useful.
  • Honest ratings help you. Your individual learning plan is built from your gaps. Inflated ratings hide the gaps you most need help with.

The instrument

Your track's file is the track's file in instruments/, such as instruments/band-5-quality-assurance-test-analyst.tsv (see instruments/README.md). It has:

  • Part A, band (21 items): Part A items 1 to 5 are the band outline dimensions, rated Not yet, Partly, or Meets. Part A items 6 to 21 are the 16 job evaluation factors: you choose the factor level that best describes what you do regularly now.
  • Part B, UK GDaD PCF role aspects: every statement in your UK GDaD PCF role description and role level description, and every reference responsibility, rated Not yet, Partly, or Meets.
  • Part C, skills: every UK GDaD PCF skill and health care skill in your role, rated 0 Not yet, 1 Awareness, 2 Working, 3 Practitioner, 4 Expert.

You complete every item at every gate, even items you rated last time. The same instrument repeats so you can see your progress on the whole role.

The steps

  1. Participant rates first. Before the gate, complete every item with a short, specific piece of evidence. You may do this with your mentor's help, in writing or in conversation.
  2. Manager rates independently. Your line manager rates every item without seeing your ratings.
  3. Meet and agree. You meet, compare, and agree each rating. Start with the items where you differ most.
  4. Evidence decides. Where ratings differ by more than one level, the evidence decides. If you still disagree, the mentor or training lead moderates.
  5. Sampling. The training lead samples 1 in 5 assessments across the cohort, to check that ratings are consistent between managers and teams.
  6. Record. The agreed rating goes in the instrument file. Differences of more than one level go in section 6 of the gate review form.

The agreed rating is the one that counts.

How to rate honestly

  • Rate what you do regularly, not what you did once or might do. One good day is not a level.
  • Write evidence for every Meets. A short, specific example of regular work, such as "Wrote the API tests for the referrals service; reviewed and merged by the team in March". Without evidence, an item is not Meets.
  • Use the level descriptions. For Part C, read the "You can …" statements for each level in the skills catalogue, and pick the level that best describes you as a whole.
  • Partly is a real answer. Use Partly when you do some of the item, or do it only with substantial help.
  • Don't aim to be Expert in everything. Most people meet the expected level in most items and are above or below in a few.
  • Effort and working conditions (Part A items 18 to 21) describe the job's demands. They are almost always Meets. Tell your manager if they do not match your experience, because it may point to a reasonable adjustment.
  • Job evaluation factors are used here as a self-assessment of how fully you work at your job's level. This is not a job evaluation, and it never changes your band.

How ratings become the capability index

  • Part A and Part B items: Meets, Partly, or Not yet.
  • Part A factors (Part A items 6 to 21): Meets when your agreed level is at or above the reference level, Partly when one level below, Not yet when two or more below.
  • Part C skills: gap = expected level minus agreed rating. Meets when the gap is 0 or less, Partly when it is 1, Not yet when it is 2 or more.
  • Partly counts as half. The index for each part is (Meets + 0.5 × Partly) ÷ number of items, as a percentage.
  • The overall index is the mean of Parts A, B, and C. Part D is reported separately.

Notes for managers

  • Rate the person against their band and reference role level, from their track guide in materials/tracks/.
  • Rate independently and before you see the self-ratings. Do not adjust your ratings to match.
  • Base each rating on evidence you have seen, or evidence the person can show.
  • Look for under-rating as well as over-rating. People who expect gaps may rate themselves down.
  • Keep the conversation developmental. Talk about what would move each item to Meets, not about performance procedures.
  • Agree the top 3 gaps for the next gate period and update the individual learning plan.

Notes for moderators

  • Read both ratings and the evidence, ask each person one question, then decide.
  • Record the decision and its reason on the gate review form.
  • Report patterns from the 1 in 5 sample, such as a manager who rates consistently higher than others, to the training lead for the Goodlad "perceived" evaluation.

View this document on GitHub materials/gates/calibration-guide.md