AI implementation that starts with a real workflow.

From pilot selection to responsible rollout

Working principle: choose one real task, measure the existing process, constrain the AI system, evaluate with representative examples, and scale only when evidence supports it.

What does AI implementation involve?

AI implementation is the process of introducing an artificial intelligence capability into a defined workflow, making its output usable for the people responsible for that work, and evaluating whether the resulting process performs acceptably. A prototype can demonstrate that a model produces plausible answers. Implementation asks a harder question: can a team depend on the process with appropriate data handling, quality checks, escalation and accountability? That difference matters to small organizations as much as large enterprises. A useful project states who owns the workflow, which activities the AI may assist, how errors will be detected, and what triggers a return to manual operation. It also distinguishes a model capability from the surrounding software, permissions, records and operational procedures that make a system reliable. A pilot is not a promise of productivity gains; it is an opportunity to collect evidence under controlled conditions.

How should a team choose its first use case?

Start with a bounded, frequent task that has observable inputs and outputs. Examples include sorting incoming enquiries, drafting a first version of a summary from approved documents, identifying missing fields in an intake packet, or preparing a research brief for human review. Prefer a process with a knowledgeable owner and accessible examples of correct results. A workflow that is rare, ambiguous, privacy-sensitive or legally consequential may be a poor first experiment even if it appears valuable. For each candidate, assess task frequency, baseline handling time, quality of current data, cost of mistakes, expected review burden, integration complexity and whether success can be measured. Record assumptions as assumptions. If expected savings depend on an invented productivity percentage, there is not yet a credible business case. Select one task with sufficient evidence to test, then design the pilot around the real constraint limiting the existing process.

What must be measured before any AI pilot?

A baseline should describe the existing method before the AI-assisted alternative is introduced. Collect a representative sample of tasks and track the time required, rework rate, error categories, exceptions, user satisfaction and direct operating inputs if available. Keep the measurement method consistent between baseline and pilot. For information work, quality often matters more than raw speed: a draft produced quickly but requiring extensive correction may provide limited net benefit. Separate elapsed time from active human effort, and report both when possible. Record who labels an output correct, what rubric they use and whether different reviewers agree. A reproducible test set reduces the temptation to judge a system on a few memorable demonstrations. A baseline need not involve expensive instrumentation; even a carefully managed spreadsheet can establish whether a later comparison is meaningful.

Which controls should be in place before connecting organizational data?

Map the data entering a system before choosing a vendor or an AI agent design. Identify personal data, confidential commercial information, intellectual property, privileged records, sensitive employee information and any information governed by contractual restrictions. Determine who can access each source, where the information is processed, how long logs are retained, and whether the service uses submitted information for model improvement. Review relevant vendor documentation and contractual settings rather than assuming that every deployment mode behaves identically. Implement least-privilege access and avoid giving a pilot broader permissions than its task requires. Define a human escalation path for uncertain or restricted cases. NIST’s AI Risk Management Framework provides a structure for considering governance, mapping, measurement and management of AI risk; it is a voluntary resource, not an automatic certification of a particular deployment.

When is a retrieval-based solution appropriate?

Retrieval-augmented generation, often abbreviated RAG, combines a model with a retrieval step that selects material from a defined source collection. It can be useful when an answer must reflect internal policies, frequently revised documentation or traceable reference materials. Its value depends on document quality, access control, search configuration, citation behavior and evaluation. Retrieval does not ensure factual correctness: a system can select irrelevant passages, omit a qualification, or combine statements from incompatible versions. A team should test whether users receive the right document, whether access boundaries hold, and whether the answer accurately represents the source. Versioned documents and a clear ownership process can matter more than adding a larger model. When a task only requires a fixed rule, a deterministic workflow may be simpler and easier to audit than model-generated interpretation.

How are AI agents different from ordinary automation?

Traditional automation usually follows a defined sequence of actions and rules. An AI agent may use a model to select tools or decide intermediate steps toward a task, which introduces more flexible behavior and more opportunities for unintended actions. The relevant design question is not whether a workflow deserves the label “agent”; it is what decisions the software can make and what it is authorized to change. Separate read-only lookup from actions that send messages, alter records, spend money or modify permissions. Start with explicit tool allowlists, constrained scopes and approvals for consequential operations. Record tool calls and failures so that incidents can be investigated. An agent that can suggest a revised customer response under supervision is a different risk proposition from one that can issue refunds without human confirmation. The amount of autonomy should follow the tested reliability of the task, not marketing terminology.

What should an evaluation framework include?

A useful evaluation plan tests normal cases, edge cases, missing information, contradictory inputs and situations where the correct action is to refuse or escalate. Assign a named reviewer to each task family and agree upon pass criteria before assessing results. For a document-processing system, checks may include correct field extraction, preservation of source references, handling of unreadable material and protection against disclosing unrelated records. For a drafting assistant, review factual fidelity, instructions followed, tone and whether it introduces unsupported claims. Track how often a person must intervene and how long that intervention takes. Re-run the same evaluation after substantial model, prompt, data or workflow changes. A single benchmark score does not capture the performance of a complete organizational process. NIST's Generative AI Profile can help teams identify relevant risk considerations when designing this evaluation process.

How should AI implementation training be structured?

Training should be tied to what a learner actually does within the workflow. The operator needs to know how to provide usable inputs, interpret outputs and catch common failure modes. The supervisor needs to understand approvals, logging, incident response and performance reporting. The system owner needs to manage access, vendor configuration and changes to the process. A short orientation followed by supervised practice on realistic examples is usually more informative than a generic presentation about model features. Document a competence checklist and provide reference material that can be consulted after training. Teams evaluating a formal learning pathway can use the existing AI courses and workshops hub and the enterprise AI curriculum resources to compare formats. UNESCO's AI competency work offers a useful perspective on critical judgment and human-centered use, although institutional curricula must be adapted to workplace roles.

How should an organization decide whether to scale?

Scale decisions should be made using pre-agreed evidence rather than anecdotal enthusiasm. Compare pilot results against the baseline, accounting for human review, exceptions, error correction, support effort, ongoing subscriptions and maintenance. Identify whether gains hold across different users and task variations. A successful pilot may justify expanding only to a small adjacent use case; it does not prove that the technology is suitable for an entire department. Confirm who will monitor quality, who can pause the workflow and how updates will be approved. Establish a change log for prompts, model configurations, retrieval sources, permission sets and test cases. Where outcomes are mixed, the responsible decision may be to simplify the workflow, improve source data, choose a deterministic method or stop. A negative pilot result can still be valuable if it prevents a costly rollout.

What common implementation failures should be anticipated?

Frequent project weaknesses include vague success criteria, inaccessible or outdated source data, missing domain reviewers, premature integration into high-stakes processes and an unclear operating owner. Teams may overestimate what a model remembers between sessions or assume that a fluent response implies accurate reasoning. Another recurring challenge is adoption: employees may use a tool inconsistently when it is disconnected from their established work or when quality checking creates more effort than the original task. These risks are reduced by involving users at design time, measuring baseline performance and establishing simple instructions for exceptions. Privacy and security controls must be assessed before exposing production records. Resistance to a new process should be understood as information about usability or organizational incentives, not automatically dismissed as unwillingness to innovate.

A practical 30/60/90-day planning framework

A 30/60/90-day framework can organize work without promising that a particular implementation will be finished or profitable within that period. In the first phase, define a bounded use case, document the baseline, classify data and assign owners. In the second phase, test a prototype against representative tasks, collect failure examples and revise the process. In the third phase, run limited operational use with training, monitoring and clear rollback procedures. These are planning milestones rather than universal deadlines. Complex integrations, regulated information, procurement reviews or inadequate data quality may require substantially more time. Deliverables should be concrete: an approved process map, access matrix, evaluation log, operating instructions and a go/no-go decision. The decision must identify remaining risks and the evidence supporting any expansion.

Where can teams find useful evidence and supporting guidance?

For governance and risk planning, consult the NIST AI Risk Management Framework and the NIST Generative AI Profile. For broader policy-oriented principles, the OECD AI Principles discuss trustworthy AI and related policy considerations. For learning design and critical AI competency, UNESCO materials can help structure the human side of adoption. These sources provide frameworks and considerations; they do not prove performance for a particular model, vendor or workflow. Vendor-specific security, retention, connector and permission claims should be checked in the vendor's current technical and contractual documentation. Internally, record the precise document version and date relied upon whenever a consequential decision is made. A clear evidence register makes a future evaluation repeatable and helps prevent temporary assumptions from becoming unexamined operating rules.

Decision matrix: choose an implementation approach

Primary research and further reading

• NIST Artificial Intelligence Risk Management Framework
• NIST Generative AI Profile
• OECD AI Principles
• UNESCO Recommendation on the Ethics of AI

Sources are guidance for research and planning. The procedures on this page are an educational synthesis and must be adapted to actual organizational, technical, contractual and regulatory requirements.