How to Know Whether Your AI Workflow Actually Works

You will produce an AI Workflow Evaluation Card, a five-case synthetic test set, and an adopt, revise, or stop decision.

Anyone evaluating a prompt, skill, model-assisted process, agent, or automation before recurring use.

• A workflow you can run locally or safely
• A precise claimed outcome
• A realistic baseline
• Five synthetic or authorized cases
• A human reviewer who understands the artifact

1. 1. Turn 'works' into a claim

   Name the artifact, user, cases covered, quality threshold, and maximum acceptable failure. Separate execution success from usefulness.

2. 2. Choose a realistic baseline

   Compare against the current manual method, template, or simpler automation. Record human time and correction, not just model output.

3. 3. Build five cases

   Include a normal case, incomplete input, ambiguity, a dominant failure case, and an out-of-scope request. Use synthetic data unless real data is authorized.

4. 4. Write the rubric first

   Define weighted criteria and hard failures before seeing outputs. Include support, completeness, instruction following, usability, safety, and correction effort.

5. 5. Run and preserve evidence

   Record versions, inputs, outputs, latency, cost where relevant, human interventions, score, and notes. Do not discard bad runs.

6. 6. Analyze failures

   For each failure record cause, detectability, consequence, recovery, and whether a gate or redesign is needed. Plausible hidden errors deserve more weight.

7. 7. Make a bounded decision

   Adopt only for demonstrated cases, revise and retest, or stop. Define monitoring, version-change triggers, and the evidence that suspends use.

Synthetic example

Scenario

A product-research team is evaluating a prompt that turns batches of interview notes into customer-needs themes.

Input

Five synthetic note sets including strong themes, thin notes, contradictory evidence, confidential details, and a request for unsupported market sizing.

Result

The workflow groups strong themes well but overstates thin evidence and repeats confidential details. Decision: revise. Add evidence counts, privacy redaction, and refusal to estimate market size; then rerun all five cases.

• Testing only the happy path
• Writing the rubric after seeing results
• Comparing against an unrealistically weak baseline
• Averaging away a dangerous hard failure
• Ignoring human correction time
• Claiming general reliability from one model and prompt version

• □The success claim names covered cases and thresholds
• □Five distinct case types are present
• □The baseline is realistic
• □The rubric predates scoring
• □Hard failures cannot be hidden by an average
• □Human effort and failure detectability are recorded
• □The decision includes monitoring and a suspension condition

ai-workflow-evaluation-card.md

# AI Workflow Evaluation Card

## Claim
For [users and cases], this workflow produces [artifact] meeting [threshold], with no more than [acceptable failure].

## Version
- Model/class:
- Prompt or skill:
- Tools:
- Date:

## Baseline
- Current method:
- Human time:
- Known limitations:

## Evaluation cases
1. Normal:
2. Incomplete:
3. Ambiguous:
4. Dominant failure:
5. Out of scope:

## Rubric
| Criterion | Weight | Hard fail? |
| --- | ---: | --- |
| Evidence/support | | |
| Completeness | | |
| Instruction following | | |
| Safety/privacy | | |
| Usability | | |
| Human correction | | |

## Results and failures
- Artifact evidence:
- Failure cause:
- Detectability:
- Consequence:
- Recovery:

## Decision
[Adopt for bounded cases / Revise and retest / Stop]

## Monitoring
- Review frequency:
- Version-change trigger:
- Suspension condition:

1. 0:00 Contrast a green run status with a flawed artifact
2. 0:35 Write the observable success claim
3. 1:15 Choose the baseline and five cases
4. 2:05 Write the rubric before running
5. 2:55 Inspect one hidden and one visible failure
6. 3:50 Compare human correction effort
7. 4:25 Make the bounded adoption decision