Quality assurance
Analysing complaints, capturing inspection records, spotting deviations: where AI helps in QA and what evidence duties come with it.
What this is about
| Task | Value |
|---|---|
| Grouping complaint texts and finding clusters | very high |
| Capturing inspection records from paper or PDF | high |
| Classifying deviation reports | high |
| Searching for links between batch and complaint | medium |
| Naming causes | low, stays with the department |
| Granting releases | not without human confirmation |
Introduction
- 01
Sharpen the categories
Two experts classify 100 complaints independently. Below 70 percent agreement, the definition is the problem.
- 02
Work up the history
Group two years of complaint text and name the groups. That often yields a finding immediately, without any model being trained.
- 03
Then classify continuously
Automatic classification goes live only once the categories hold.
- 04
Secure the feedback
Every correction by QA is a new label and feeds the next analysis.
- Report accuracy per category, not as an overall figure. The rare categories are the interesting ones.
- No threshold without a cost calculation: what does a missed cluster cost, what does a false alarm?
- Free-text complaint fields contain customer names; pseudonymise before analysis.
Evidence for audits
An auditor asks not about the model but about control of the process. Five items carry:
| Item | Meaning |
|---|---|
| Purpose and boundary of the system | What it does and expressly does not |
| Validation with date and result | Fixed evaluation set, quality per category |
| Model version and change history | Every change is a change to the process |
| Thresholds with reasoning and sign-off | Who decided this and when |
| Monitoring in operation | How degradation is noticed |
Those five simultaneously cover the requirements of ISO 9001 clause 8.5.1 and, where classified as high risk, substantial parts of Annex IV of the AI Act. See Preparing for audit.
Validation, concretely
- An evaluation set of 200 to 500 real cases with expert-assigned labels.
- The set is versioned and rerun on every model change.
- Inter-expert agreement is reported alongside; it is the ceiling.
- Per-category results with confidence intervals, not only point values.
- The cases where the system was wrong, with examples, in the documentation.
Regulated environments
In GxP-regulated settings further requirements apply: computer system validation, change control, access restriction, complete audit trails, and the requirement that electronic records meet standards of completeness, legibility and durability. A self-learning system that changes in operation is routinely incompatible with that: the usual solution is a frozen model state with documented release per version.
Choosing a task in quality assurance
FREE ACCOUNT
Selection sheet and templates: quality assurance
A sheet that picks the first task out of the ones quality assurance actually has, with two tested templates and the review steps for them.
Worksheet3 items