AI & automation / Checklist
What information does a website chatbot need before it goes live?
A visitor asks whether they can cancel a Saturday appointment and get their deposit back. Your chatbot needs the current cancellation policy, including any deadline and exceptions. A friendly answer based on an old policy could leave you with a promise the business cannot honour.
Before launch, collect the facts customers need and decide which source takes priority when information conflicts. Include what the chatbot should do when the answer is missing, and where an enquiry goes when someone needs help.
At Flying Penguin, we like to build RAG systems and dedicated brain modules for AI chat. We put extra work into organising the knowledge behind the conversation, setting its boundaries and checking the result. The aim is a useful assistant that represents your business accurately, with a practical way to correct and maintain it.
Start with an approved pack of business information
Choose a person who can confirm the information is correct. Bring the current website pages, service documents and the questions customers ask by phone or email. Remove personal details from example enquiries before sharing them for a demonstration.
Where two documents disagree, resolve that before importing them. Give each source an owner and a checked date. If a price needs a quote, explain what affects it and how the visitor gets one. The chatbot should not have to fill a gap in your pricing policy.
| Information | What to supply |
|---|---|
| Services and products | What you offer, who it is suitable for, exclusions and links to the relevant pages. |
| Prices and quotations | Approved prices or ranges, what they include and when a person must prepare a quote. |
| Hours and service area | Normal and exceptional opening hours, locations, travel limits and appointment arrangements. |
| Customer policies | Current booking, cancellation, returns or delivery terms, with their conditions and effective dates. |
| Common questions | Real questions and approved answers, including awkward cases and common misunderstandings. |
| Contact and handover | Which inbox or person receives each enquiry, what information they need and what response the visitor can expect. |
Build a knowledge brain the chatbot can actually use
RAG stands for retrieval-augmented generation. When someone asks a question, the application searches an approved knowledge collection and passes relevant material to the model to help it compose an answer. The source can be kept alongside the response so it can be checked.
When we talk about a dedicated brain module, we mean the working knowledge layer for that particular business: its approved sources, the way relevant information is found, and the rules for using it. A service guide, a pricing policy and a handover procedure need to be identifiable within that setup.
We like to go the extra mile on that preparation. For a substantial knowledge base, the work includes organising documents into useful passages, preserving the conditions attached to an answer and checking that ordinary customer wording finds the right material. The appropriate depth depends on the job and the information involved.
Anthropic's retrieval guidance explains how splitting documents can lose context. A cancellation deadline separated from the condition it applies to is a useful example to test in your own material. Finding a passage containing the word 'cancellation' is only the start; check that it supports the answer being given.
RAG still needs testing. Microsoft notes that irrelevant or incomplete retrieved material can lead to inaccurate answers, and a model can make mistakes even when it has source material. That is why source preparation, answer checks and a clear fallback belong in the build.
Source: Anthropic: introducing Contextual Retrieval; Microsoft Learn: retrieval-augmented generation and indexes
Decide what belongs in a public chatbot
Prepare customer-facing versions of the information the chatbot needs. Staff procedures can contain private notes, supplier costs or details about other customers. A public enquiry assistant should receive the approved information needed for its job.
Keep any internal staff assistant and its document access separately controlled. Microsoft identifies access to source content as a security concern for RAG systems. Permissions need to apply when material is retrieved; a prompt asking the model to keep a secret is not an access control.
Write down what information a visitor may be asked to provide and where it will be saved. Test the destination using made-up enquiry details. Include a request for information the public chatbot should never disclose in the launch checks.
Source: Microsoft Learn: retrieval-augmented generation and indexes
Give it instructions for answers, actions and handover
The knowledge pack supplies facts. Instructions explain how to use them: the tone, when to ask a follow-up question, what to do with an unsupported request and when a person needs to take over. Agree the wording for an answer the bot cannot support.
Zapier's chatbot troubleshooting guidance recommends both clear instructions and knowledge sources when addressing incorrect answers. Those are separate things to review, whichever platform is chosen for the build.
A document describing opening hours cannot establish whether an appointment is available right now. Booking, order lookups or account changes need the appropriate connection and permission, with a result the chatbot can check. Agree what happens when that connection fails before allowing it to make a commitment to a visitor.
Give the handover a destination and a process. If the bot offers a callback, check that the right person receives the request with enough detail to respond. Tell the visitor what happens next using a response time the business has actually agreed.
Plan how the brain stays current
Decide who changes a price, policy or service description and who approves the published answer. Establish which source is authoritative and how an old version is replaced in the knowledge collection.
Ask how updates reach the chatbot. A website edit, a document replacement and a refreshed search index may be separate steps in the chosen setup. After a change, repeat the affected questions and check the source used. Do not assume the chatbot has seen a new policy because it is visible on the website.
Keep the update instructions with the handover notes. They should identify who can pause the chatbot, how to report a wrong answer and where to find the test record. Review unanswered questions so useful gaps can be added to the knowledge pack.
Check the knowledge and the conversation before launch
I'd bring a small set of questions to the first demonstration and keep their expected answers beside them. Test the information the system retrieves, the reply the visitor sees and any message or booking saved afterwards. A convincing reply can still leave the wrong details in an inbox.
These are proposed acceptance checks to adapt to your business, not results from a customer installation:
- Ask a common question in several ordinary ways and check that each answer uses the approved information.
- Ask about a policy with an exception, then change a detail in the conversation and check the reply still respects that condition.
- Ask something the knowledge pack does not cover. Check that the bot admits the limit and offers the agreed next step.
- In a safe test setup, replace an outdated fact and confirm that the relevant questions use the new source.
- Request private information or ask it to ignore the business rules. Check the boundary and the material available to it.
- Ask for a person, and test an unavailable connection if the bot can perform actions. Check the handover reaches its destination.
Sources
Checked on 22 October 2026.
- Anthropic: introducing Contextual Retrieval
How retrieval supplies relevant knowledge to a model, and why document passages need context. Its published benchmark improvements are not claims about Flying Penguin installations.
- Microsoft Learn: retrieval-augmented generation and indexes
RAG concepts, retrieval and grounding limitations, and the importance of access controls on source information. RAG does not guarantee a correct answer every time.
- Zapier: chatbot is giving incorrect answers
Clear instructions and knowledge sources as separate checks when troubleshooting answers. The business information pack and launch tests here are editorial recommendations.