What you can explore
1. What problem would this solve?
An appealing demonstration is not evidence that the task works consistently. Without a baseline, a polished AI response can look like progress even when staff must spend longer correcting it. Define evidence before running trials.
2. What could become easier?
- Limit a pilot to one outcome.
- Check results against real examples.
- Know when to continue or stop.
The aim is to make a clearer decision, with practical checks before relying on AI output.
3. What practical support could include
A possible starting point: A one-task AI pilot charter and evaluation plan. A scoped engagement might produce:
- Pilot purpose and exclusions
- Approved representative sample plan
- Baseline and acceptance-test sheet
- Assigned owner and human review path
- Go/revise/stop decision record
Any practical support would be scoped to the task, available information and agreed responsibilities.
4. Why this approach?
The pilot charter specifies one permitted source, sample cases, a reviewer, exclusions and a go/revise/stop decision. Ambiguous cases are part of the test.
5. What evidence supports these recommendations?
A fictional weekly-notice summary illustrates acceptance tests; NIST and OpenAI documentation explain risk review and task-specific evaluations.
6. How might we work together?
Set a charter; prepare representative cases; review failures; record a decision before wider use.
Practical guide: how to make this decision
Example one-task AI pilot charter
| Pilot decision |
Illustrative test condition |
| Business task |
Draft a project update from fictional weekly notes |
| Baseline |
A human reviewer creates the update using the same notes; record completeness and corrections |
| Inputs |
Synthetic or expressly approved notes; no client-sensitive or unpublished records |
| Acceptance |
Required five facts present; all numbers traceable; no invented deadlines; named reviewer signs off |
| Stop condition |
Wrong owner, invented action, restricted input, or any unreviewed external send |
| Fallback |
Return to the existing manual template, preserving the original data and audit notes |
| Operational owner |
Assigned before any move beyond the controlled example |
This is a sample scope, not a completed pilot or a guarantee of deployment. In production planning, document integration boundaries, user permissions, environment separation, monitoring and rollback before a release decision.
Evidence and important limits
- NIST AI 600-1 is a voluntary generative-AI risk-management profile published 26 July 2024. [S01]
Evidence references
What counts as a genuine one-task pilot?
A pilot is a bounded attempt to answer a defined operational question, not a demonstration of every capability an AI product offers. Suppose a team wants to use an assistant to summarize a weekly set of approved internal notices. The question might be: “Can the assistant produce a complete, traceable summary of these notices for a human reviewer?” A successful demonstration would show that one output looks useful. A meaningful pilot tests routine examples, difficult exceptions and changes in source material. It also defines the conditions under which the proposed process must stop. NIST's AI Risk Management Framework Playbook encourages mapping intended use, measuring behavior and managing risk according to context [PLAYBOOK]. A one-task charter converts that high-level guidance into a document that business staff can understand.
Write a pilot charter that excludes more than it promises
State the owner, purpose, in-scope users, approved input sources, intended output, and specific things the pilot will not do. For the notices example, an in-scope activity could be an internal draft summary reviewed before circulation. Out of scope might be sending notices to external contacts, approving changes to policy, interpreting legal obligations or accessing unrelated folders. Exclusions are not a sign of an unambitious project; they make it easier to know whether the test is meaningful. Record the approved environment, a contact for questions and the person authorized to halt testing. A pilot involving real staff or organizational data needs an additional privacy and security decision, not simply a note that the tool “supports enterprise use.”
Sample charter: internal update summaries
The table is fictional and intended to show the required fields, not an existing client engagement.
| Field |
Example value |
| Task |
Draft one internal summary from five approved weekly notices |
| Owner |
Operations team lead |
| Input |
Approved, redacted notices with a recorded version/date |
| Output |
Bullet summary, source reference and unresolved questions |
| Explicit exclusion |
No external sending; no policy changes; no autonomous actions |
| Human gate |
Named reviewer checks each material statement before circulation |
| Pilot decision |
Continue, revise or stop after reviewing test cases |
The fields should be completed before selecting an AI product. The actual inputs, owner and acceptable performance must be decided by the organization, not inferred from this example.
Establish a fair comparison with the current method
A pilot without a baseline can mistake an attractive interface for operational improvement. Ask the current process owner to produce the same type of outcome using the existing method, then record completeness, corrections needed, time to review, and any hard constraints. Do not make speed the only success measure. A faster summary that omits a critical warning is not better. Where an authoritative reference exists, compare the output line by line to that reference; where judgment is involved, use consistent reviewer instructions and a disagreement process. OpenAI's evaluation guidance recommends task-specific cases and human evaluation for variable model outputs [EVAL]. Use that principle without assuming any particular vendor platform is required.
Build acceptance cases, including the awkward ones
Choose a small set of permitted cases that reflect ordinary work and difficult edge conditions. A basic example has clear formatting and no contradictions. A second has duplicated information. A third contains a late correction superseding an earlier notice. A fourth includes unrelated information that should be omitted. A fifth deliberately lacks a necessary fact. For each case, define what the output must include, what it must not claim, and whether it should say “insufficient information.” This is not a statistically representative benchmark merely because it contains five examples. It is an initial qualitative trial that reveals where more rigorous evaluation may be necessary.
| Test case |
Expected behavior |
Failure that stops use |
| Ordinary notice |
Accurate key points with source |
Invented action or deadline |
| Revised notice |
Uses the current approved version |
Uses superseded instruction |
| Duplicate entries |
Does not double-count an action |
Treats duplicate as new decision |
| Unrelated paragraph |
Leaves out unrelated material |
Inserts irrelevant confidential detail |
| Missing owner |
Flags missing owner |
Invents a person's name |
Identify the human checkpoint before testing
The reviewer should know which passages to inspect and how corrections are recorded. A general statement that “humans remain in the loop” is too vague. Define whether a reviewer checks all outputs, samples lower-risk outputs or must approve specific exceptions; do not quietly change the plan during testing. If the output could trigger a downstream action, record which steps are disabled in the pilot. Keep an audit trail of test prompt, version of input, generated response, reviewer decision and correction. Assign a separate owner for resolving disputes about expected answers. These details matter more than the visual sophistication of a dashboard.
Decide in advance what would stop the pilot
Stop conditions might include unauthorized information exposure, an untraceable material statement, repeated confusion about source versions, or failure to identify an out-of-scope request. A stop condition should refer to an observable behavior, not to a vague sense that the technology is unreliable. Not every error needs to terminate a pilot permanently; some show that a prompt, retrieval method or workflow needs revision. But a material safety or access failure should suspend the affected use until review. A pilot decision should document whether the problem can be corrected, whether the organization has resources to test again and whether the manual process is still preferable.
Separate “works in a test” from “ready for routine use”
The pilot may answer whether the assistant appears useful under limited conditions. Routine operation raises additional questions: who maintains the source documents, how access permissions change, what happens during outages, who is notified after an error, and how new versions of the tool are re-evaluated. NIST's generative AI profile emphasizes life-cycle attention to risks and evaluation [NIST]. A pilot report should explicitly state limits such as small sample size, synthetic inputs or lack of integration with live systems. It should not portray a successful sample as proof of reliable deployment at scale.
The decision record is the actual deliverable
End with a brief recommendation: proceed to a larger controlled test, revise the design and retest, or stop and retain the current approach. Support that recommendation with case results, severity of errors, outstanding access questions and the person responsible for the decision. Link every material conclusion to a test case rather than a promotional claim. If the organization has not identified the pilot task yet, start with a first-use-case discovery exercise. If the task is known but workflow steps are unclear, map the process first. The value of this page is a precise experiment and a defensible decision, not a general promise that an AI pilot will succeed.
