QA Engineer / Tester

Start here

Widely used and core to this role

• Playwright(opens in new tab)

  · Testing & QA

  Open-source browser test automation, with planner, generator and healer agents.^[4]

• GitHub Copilot(opens in new tab)

  · AI coding assistant

  Drafting and extending test code in the IDE, such as parameterised variants and negative cases, for review.^[8]

Also worth a look

Depends on your team and stack

Written from the changes above. Read and adapt one before you use it.

• Test cases from acceptance criteriaDrafts test cases from a user story, covering edge cases and non-functional needstest-cases-from-acceptance-criteriaView · Copy · Download

  Copy instructions Download .zip Copy link to this

  How to use it

  • In any AI chat, such as ChatGPT, Microsoft 365 Copilot or Claude: choose Copy instructions, paste them at the start of a new chat, then add your material. Our suggestion
  • Install it in Claude (web or desktop): download the .zip, then go to Customize › Skills › + › Create skill › Upload a skill. Code execution must be turned on. Claude help (opens in new tab)
  • Install it in a coding agent such as Claude Code, GitHub Copilot or Codex: unzip the folder into the tool's skills folder. Folder locations

  Show the full file

  File name SKILL.md, in a folder named test-cases-from-acceptance-criteria. When to use it: Drafts a reviewable set of test cases from a user story, acceptance criteria or requirements, covering functional paths, negative and boundary cases, and non-functional requirements such as performance, accessibility, security and privacy, with traceability back to each criterion. Use when writing test cases or test ideas for a new story or feature, doing test analysis and design, or checking an AI-drafted test list for gaps.

  ---
  name: test-cases-from-acceptance-criteria
  description: Drafts a reviewable set of test cases from a user story, acceptance criteria or requirements, covering functional paths, negative and boundary cases, and non-functional requirements such as performance, accessibility, security and privacy, with traceability back to each criterion. Use when writing test cases or test ideas for a new story or feature, doing test analysis and design, or checking an AI-drafted test list for gaps.
  metadata:
    title: "Test cases from acceptance criteria"
    summary: "Drafts test cases from a user story, covering edge cases and non-functional needs"
  ---
  
  # Test Cases from Acceptance Criteria
  
  Turn a story's acceptance criteria into a traceable, risk-aware set of test cases that a tester reviews before use.
  
  ## What to gather first
  
  Ask for anything missing before you start:
  
  - The user story or requirement text and its acceptance criteria.
  - The product area, the users affected, and any known risky or fragile parts.
  - Non-functional expectations: performance, accessibility, security, privacy, compatibility, localisation.
  - Supported platforms, browsers, devices or API versions.
  - Existing test cases for the area, to avoid duplicates, and the team's test case format.
  - Which test data may be used. Never use real customer data unless the team confirms it is allowed.
  
  Ask for anything missing once. If the user can't provide it or asks you to go ahead, proceed and record every assumption.
  
  ## Steps
  
  1. Number every acceptance criterion (AC1, AC2 …). If the story has none, list the testable statements you can find and mark them as inferred.
  2. List ambiguities and contradictions in the requirements before writing any cases. Do not resolve them by guessing; record them as questions.
  3. For each criterion, write at least one positive case that proves it works as described.
  4. For each criterion, add negative cases: invalid, missing, duplicated or malformed input, and actions by users without permission.
  5. Add boundary cases for every limit, range, length, date, quantity or state transition mentioned or implied.
  6. Work through non-functional requirements explicitly, one category at a time, even if the story does not mention them. For each category below, write cases or state why it does not apply:
     - performance and load (response time under expected and peak use);
     - accessibility (keyboard use, screen reader labels, contrast, focus order);
     - security (authorisation, input handling, session behaviour);
     - privacy and data handling (what is stored, shown, logged or exported);
     - compatibility and localisation (browsers, devices, languages);
     - error recovery (offline, timeouts, retries).
  7. Add Regression cases for interactions with existing features that the story might break, and trace each one to the risk it covers.
  8. Give each case a priority based on risk: impact on users if it fails and likelihood of failure.
  9. Mark each case as a candidate for automation, manual testing or exploratory follow-up.
  10. Build a traceability table so every criterion maps to cases, and every case maps to a criterion or risk.
  11. Remove duplicate or invented cases that do not trace to a requirement or a stated risk.
  
  ## Output format
  
  ```markdown
  ## Test design: <story title or ID>
  
  ### Requirements questions
  
  - <ambiguity or contradiction, and who should answer>
  
  ### Test cases
  
  | ID    | Traces to | Type                                                         | Title | Preconditions | Steps | Expected result | Priority     | Execution                   |
  | ----- | --------- | ------------------------------------------------------------ | ----- | ------------- | ----- | --------------- | ------------ | --------------------------- |
  | TC-01 | AC1       | Positive / Negative / Boundary / Non-functional / Regression |       |               |       |                 | High/Med/Low | Automate / Manual / Explore |
  
  ### Non-functional coverage
  
  | Category                       | Cases | Not applicable because |
  | ------------------------------ | ----- | ---------------------- |
  | Performance                    |       |                        |
  | Accessibility                  |       |                        |
  | Security                       |       |                        |
  | Privacy                        |       |                        |
  | Compatibility and localisation |       |                        |
  | Error recovery                 |       |                        |
  
  ### Traceability
  
  | Criterion or risk | Cases |
  | ----------------- | ----- |
  | AC1               |       |
  
  ### Suggested exploratory charters
  
  - Explore <area> with <resources> to discover <risk>
  
  ### Assumptions and open questions
  
  - <assumption or question, and who can confirm>
  ```
  
  ## Watch out for
  
  - Treating your own draft as final. It is a starting point for a tester's review, not a finished test plan.
  - Missing non-functional requirements because the story only describes behaviour. The explicit step above is not optional.
  - Inventing requirements. An expected result must come from the story, a linked document or an agreed rule; otherwise mark it as a question.
  - Producing many similar cases instead of covering the risky ones. More cases is not better coverage.
  - Real personal data in test data or examples. Never reproduce secrets, tokens, credentials or personal data from tickets, specs or existing tests in the output; refer to them by location and type, and tell the user to rotate anything exposed.
  - Expected results that are vague ("works correctly"). Each must be observable.
  
  ## Before you finish
  
  - [ ] Every acceptance criterion has at least one positive and one negative or boundary case.
  - [ ] Every non-functional category has cases or a reason it does not apply.
  - [ ] Every case traces to a criterion or a named risk.
  - [ ] Ambiguities are listed as questions, not silently resolved.
  - [ ] No real customer or personal data appears.
  - [ ] Clearly flag your assumptions, inferred requirements, and which priorities and expected results a tester or product owner must confirm.
  

• Agent test change auditChecks agent-changed tests for edits that could hide a real bug or create a false passagent-test-change-auditView · Copy · Download

  Copy instructions Download .zip Copy link to this

  How to use it

  • In any AI chat, such as ChatGPT, Microsoft 365 Copilot or Claude: choose Copy instructions, paste them at the start of a new chat, then add your material. Our suggestion
  • Install it in Claude (web or desktop): download the .zip, then go to Customize › Skills › + › Create skill › Upload a skill. Code execution must be turned on. Claude help (opens in new tab)
  • Install it in a coding agent such as Claude Code, GitHub Copilot or Codex: unzip the folder into the tool's skills folder. Folder locations

  Show the full file

  File name SKILL.md, in a folder named agent-test-change-audit. When to use it: Audits test changes made by a coding or testing agent, such as healed locators, adjusted waits, updated assertions, skipped, disabled or deleted tests, and flags changes that could hide a real bug or create a false pass. Use when reviewing a pull request where an agent wrote or repaired tests, when a test suite suddenly turns green after agent work, when tests were auto-healed or marked skipped, or when checking generated test code for gaps.

  ---
  name: agent-test-change-audit
  description: Audits test changes made by a coding or testing agent, such as healed locators, adjusted waits, updated assertions, skipped, disabled or deleted tests, and flags changes that could hide a real bug or create a false pass. Use when reviewing a pull request where an agent wrote or repaired tests, when a test suite suddenly turns green after agent work, when tests were auto-healed or marked skipped, or when checking generated test code for gaps.
  metadata:
    title: "Agent test change audit"
    summary: "Checks agent-changed tests for edits that could hide a real bug or create a false pass"
  ---
  
  # Agent Test Change Audit
  
  Check whether tests changed by an agent still test the right thing, and flag every change that could hide a real defect.
  
  ## What to gather first
  
  Ask for anything missing before you start:
  
  - The diff, branch or pull request containing the test changes, and the base branch.
  - Why the agent changed the tests (task description, failure logs, agent notes).
  - Whether application code changed in the same diff.
  - Recent test run results before and after the change, if available.
  - The requirements or acceptance criteria the affected tests are meant to cover.
  
  Ask for anything missing once. If the user can't provide it or asks you to go ahead, proceed and record every assumption.
  
  ## Steps
  
  1. List every changed test file and classify each change: new test, deleted test, skipped or disabled test, changed assertion, changed locator, changed wait or timeout, changed test data, changed setup or mocks, retries added.
  2. Treat deletions, skips, disabled tests and added retries as high attention. For each, find the reason given. If no reason is given, flag it.
  3. For every changed assertion, compare old and new expected values. Decide whether the new value reflects an intended requirement change or simply matches whatever the application now does. Look for weakened checks: exact values replaced by "exists", "contains" or "not null", or removed assertions.
  4. For every changed locator, check that it still targets the same user-facing element and purpose, not a different element that happens to match.
  5. For changed waits and timeouts, check whether a longer wait hides a real performance problem or race condition.
  6. For changed mocks, stubs and test data, check whether the test still exercises real behaviour or now tests the mock.
  7. If a test was skipped because the agent believed the feature is broken, treat it as a possible defect. Write down how a human can reproduce it.
  8. If application code changed in the same diff, check whether tests were edited to match the new behaviour rather than the requirement.
  9. Check new tests for false passes: assertions that can never fail, missing assertions, or tests that pass even when the feature is removed. To confirm a test fails when the behaviour it checks is broken:
     - ask the user for explicit permission first, and skip this check if they do not give it;
     - work only in a separate throwaway worktree or copy, never in the user's working tree;
     - check that `git status` is clean before you start and again after you finish;
     - list every temporary edit in the output, and never commit or push them.
  10. Map each affected test back to its requirement and note any requirement that lost coverage.
  11. Recommend an action per change: accept, question, revert, or raise a defect.
  
  ## Output format
  
  ```markdown
  ## Test change audit: <PR or branch>
  
  ### Summary
  
  - Tests added: <n> | changed: <n> | skipped or disabled: <n> | deleted: <n>
  - Application code changed in same diff: Yes / No
  - Overall: Safe to merge / Needs answers / Do not merge yet
  
  ### High-attention changes
  
  | Test | Change type | Before | After | Reason given | Risk | Recommended action |
  | ---- | ----------- | ------ | ----- | ------------ | ---- | ------------------ |
  
  ### Possible defects hidden by test changes
  
  | Test | Suspected defect | How to reproduce | Suggested severity |
  | ---- | ---------------- | ---------------- | ------------------ |
  
  ### Coverage lost
  
  | Requirement | Previously covered by | Now covered by |
  | ----------- | --------------------- | -------------- |
  
  ### Other changes reviewed
  
  - <test>: <change> — <accept / question>
  
  ### Temporary edits made to check for false passes
  
  - <file and edit, worktree or copy used, git status before and after — or "none">
  
  ### Questions for the author
  
  - <question>
  
  ### Assumptions and open questions
  
  - <assumption or question, and who can confirm>
  ```
  
  ## Watch out for
  
  - A green suite is not proof the product works. Ask what changed to make it green.
  - A healed test that passes against the wrong element or a weaker check.
  - Skipped tests quietly left in the suite. A skip may be the only signal that a feature is broken, and someone must confirm whether the bug is real.
  - Tests and code changed together in a way that makes the tests agree with a bug.
  - Added retries or longer timeouts that hide flakiness, race conditions or slowness.
  - Snapshot or golden files regenerated wholesale, which accepts every difference at once.
  - Never reproduce secrets, tokens, credentials or personal data from test data, fixtures or logs in the output; refer to them by location and type, and tell the user to rotate anything exposed.
  
  ## Before you finish
  
  - [ ] Every deleted, skipped, disabled or retried test is listed with its reason or "none given".
  - [ ] Every changed assertion was compared before and after.
  - [ ] Possible hidden defects have reproduction notes.
  - [ ] Any deliberate breakage had explicit permission, ran in a throwaway worktree or copy with a clean git status before and after, is listed in the output, and was never committed.
  - [ ] Clearly flag your assumptions, which skipped tests may be real bugs, and every accept or revert decision a human must make.
  

• LLM feature eval planDrafts an evaluation plan and starter test set for a feature built on a language modelllm-feature-eval-planView · Copy · Download

  Copy instructions Download .zip Copy link to this

  How to use it

  • In any AI chat, such as ChatGPT, Microsoft 365 Copilot or Claude: choose Copy instructions, paste them at the start of a new chat, then add your material. Our suggestion
  • Install it in Claude (web or desktop): download the .zip, then go to Customize › Skills › + › Create skill › Upload a skill. Code execution must be turned on. Claude help (opens in new tab)
  • Install it in a coding agent such as Claude Code, GitHub Copilot or Codex: unzip the folder into the tool's skills folder. Folder locations

  Show the full file

  File name SKILL.md, in a folder named llm-feature-eval-plan. When to use it: Drafts an evaluation plan and a starter eval dataset for a product feature built on a large language model, where the same input can give different outputs, covering labelled examples, edge and adversarial cases, a Pass/Fail grading method and regression runs on every change. Use when quality criteria for a chatbot, summariser, AI assistant or other generative AI feature exist, or alongside defining them, to build the eval dataset, grading method and regression runs. Defining what good output looks like is a product decision; if there are no criteria yet, draft them with the product manager first.

  ---
  name: llm-feature-eval-plan
  description: Drafts an evaluation plan and a starter eval dataset for a product feature built on a large language model, where the same input can give different outputs, covering labelled examples, edge and adversarial cases, a Pass/Fail grading method and regression runs on every change. Use when quality criteria for a chatbot, summariser, AI assistant or other generative AI feature exist, or alongside defining them, to build the eval dataset, grading method and regression runs. Defining what good output looks like is a product decision; if there are no criteria yet, draft them with the product manager first.
  metadata:
    title: "LLM feature eval plan"
    summary: "Drafts an evaluation plan and starter test set for a feature built on a language model"
  ---
  
  # LLM Feature Eval Plan
  
  Draft an eval plan and starter dataset so a team can judge an AI feature's output quality consistently and repeatably.
  
  ## What to gather first
  
  Ask for anything missing before you start:
  
  - What the feature does and who uses it.
  - The agreed quality criteria (Pass if / Fail if), usually drafted by the product manager from real interactions. If none exist, say so.
  - The prompt or system instructions, and any tools or data sources the feature can use.
  - Real or realistic example inputs, and whether they contain personal or confidential data.
  - Known failures, complaints or bug reports.
  - Constraints: response time, cost limits, languages, content the feature must never produce.
  - How the team currently tests the feature, if at all.
  
  Ask for anything missing once. If the user can't provide it or asks you to go ahead, proceed and record every assumption.
  
  ## Steps
  
  1. Write the feature's purpose in one sentence and list the user tasks it must support.
  2. Start from the agreed quality criteria. Deciding what good output looks like is a product decision, so do not redefine it. If no criteria exist, draft provisional ones as separate binary statements ("Pass if…, Fail if…"), for example on correctness against source, following instructions, format, refusal when appropriate and safety, and mark each as needing product agreement.
  3. Separate criteria that can be checked by code (format, length, required fields, forbidden terms, response time) from criteria that need human or model-based judgement.
  4. Draft a starter dataset of inputs grouped by category: typical requests, edge cases (empty, very long, mixed language, ambiguous), out-of-scope requests, and adversarial inputs such as instructions hidden in user content that try to override the feature's rules.
  5. For each input, write the expected characteristics of a good answer rather than one exact expected string.
  6. Replace or remove any personal or confidential data in examples. Use synthetic stand-ins and note where real data was anonymised.
  7. Plan grading and labelling: each output gets Pass or Fail per criterion, with an optional "needs human review" flag when the grader is unsure. Say who labels outputs, who reviews flagged ones, how disagreements are resolved, and how many examples need human labels before trusting any automated grader.
  8. If a model is used as a grader, plan to validate it against the human labels first and record where it disagrees.
  9. Because output varies, plan to run each input more than once and judge consistency as well as quality.
  10. Define pass thresholds per criterion as proposals for the team to agree, not as fixed numbers.
  11. Plan when evals run: on every prompt, model, tool or data change, and before each release. Say where results are stored and who reviews regressions.
  12. Add non-functional checks: response time, cost per request, behaviour on timeouts and errors, and what is logged.
  
  ## Output format
  
  ```markdown
  # Eval plan: <feature name>
  
  ## Purpose and user tasks
  
  - <task>
  
  ## Quality criteria
  
  | Criterion | Pass if | Fail if | Checked by                  | Status               |
  | --------- | ------- | ------- | --------------------------- | -------------------- |
  |           |         |         | Code / Human / Model grader | Agreed / Provisional |
  
  ## Starter dataset
  
  | ID     | Category                                    | Input (synthetic) | Good answer should | Must not |
  | ------ | ------------------------------------------- | ----------------- | ------------------ | -------- |
  | EV-001 | Typical / Edge / Out of scope / Adversarial |                   |                    |          |
  
  ## Grading and labelling
  
  - Grades: Pass / Fail per criterion, plus optional "needs human review" flag
  - Labellers, review of flagged outputs, and disagreement process: <...>
  - Model grader validation: <plan, or "not used">
  - Runs per input: <n, proposed>
  
  ## Proposed thresholds
  
  | Criterion | Proposed threshold | Needs agreement from |
  | --------- | ------------------ | -------------------- |
  
  ## When evals run
  
  - <trigger> → <who reviews>
  
  ## Non-functional checks
  
  - <check>
  
  ## Assumptions and open questions
  
  - <assumption, risk or question, and who can confirm>
  ```
  
  ## Watch out for
  
  - Writing one exact expected output and asserting equality. It will fail on harmless variation and pass on nothing useful.
  - Trusting a model grader that has never been compared with human labels.
  - Only typical inputs. Adversarial and out-of-scope inputs are where harmful failures appear.
  - Real user data copied into eval datasets or shared outside approved systems. Never reproduce secrets, tokens, credentials or personal data from prompts, logs or bug reports in the output; refer to them by location and type, and tell the user to rotate anything exposed.
  - Middle grades such as "borderline" that let unclear outputs slip through. Grade Pass or Fail and flag uncertain cases for human review.
  - Rewriting the product's quality criteria. Raise disagreements as questions for the product manager.
  - Answers that sound fluent but are wrong. Check against the source, not just readability.
  - Treating a single good run as a pass when output varies between runs.
  
  ## Before you finish
  
  - [ ] Every criterion has Pass if / Fail if wording, a checking method, and is marked agreed or provisional.
  - [ ] The dataset includes typical, edge, out-of-scope and adversarial inputs.
  - [ ] No real personal or confidential data appears in examples.
  - [ ] Thresholds are marked as proposals for the team to agree.
  - [ ] Clearly flag your assumptions, which criteria and thresholds need product or team agreement, and which examples a human must label.