AI & automation / Checklist
Will your chatbot know when you change the website?
You change your opening hours on the website, publish the page and carry on with the day. Will the chatbot give somebody the new hours? That depends on how it gets its information and whether the update has reached it.
This came up while we were adding content to Flying Penguin. We needed to check the website chat and the phone agent separately. Publishing the pages was one part of the work; checking the knowledge sources and the answers was another.
We like building RAG systems and the wider knowledge modules behind an assistant because we can organise that process around the business. RAG lets the assistant look up relevant material when answering. The source still needs maintaining, and the answer still needs checking.
Find out which copy the assistant reads
Ask your supplier where the assistant gets a price or policy from. It might search an imported copy of your website, use an uploaded document or query a business system. Those sources have different update arrangements.
A document exported from another app becomes a snapshot unless the integration keeps it in sync. Retell explicitly notes that exported files need uploading again after the original changes. Microsoft’s RAG guidance also separates the content being indexed from the passages retrieved to answer a question.
Give each important fact an agreed source. If the public price page says one thing and an old PDF says another, decide which should be current and remove or correct the conflicting material.
Source: Retell: knowledge base setup and retrieval tuning; Microsoft: retrieval-augmented generation in Azure AI Search
Check new pages as well as changed pages
Refreshing a known URL means fetching that page again. Discovering a new URL means finding a page the assistant did not already have. A system can do the first successfully while never seeing your new service page.
Retell documents separate automatic refresh and URL-path crawling features. Its refresh can keep the previous copy if fetching fails or a time limit is reached. That keeps the knowledge base available, but means a scheduled refresh alone does not prove every page is current.
After adding a page, check that its address is in the assistant’s intended public sources. After changing a page, check that the imported text contains the changed detail. Keep unpublished articles and private records out of a public assistant’s library.
Put an answer check beside the website change
Use a small checklist for the facts you have changed. Suppose you alter Saturday opening hours. Check the live contact page, confirm the new hours reached the knowledge source and ask the assistant when you are open on Saturdays.
Try a second wording too, such as whether someone can visit on Saturday afternoon. It helps you see whether the assistant can apply the hours to the question. Where the platform exposes retrieved passages, read the passage used for that answer.
If the answer is wrong, keep the question and reply. Work back through the source and retrieval result before changing the wording of the instructions. The fault could be an old copy, a missing passage or the assistant misreading information it did receive.
Flying PenguinFrom the workshopHas the new opening time reached both assistants?
A website edit is one step in keeping an assistant current.
| Place to check | What you want to establish |
|---|---|
| Public page | The approved hours are visible at the live address |
| Knowledge source | The imported text contains those hours and the refresh completed |
| Website chat | Both a direct question and a practical visit question use the current hours |
| Phone assistant | The same questions produce a current answer on this separate channel |
Check the phone agent separately
Your website chat and phone receptionist may use different knowledge bases even when they sound similar. An update in one dashboard may leave the other unchanged.
Write down which channels need the changed information and who confirms each one. Ask for evidence of source coverage and a completed import before testing the reply. If the assistants share retrieval, check the actual published agents still use it.
For live prices, stock or available appointments, discuss a connection to the current business system. A periodically imported page may be unsuitable for a fact that changes during the day. The assistant also needs a useful response when that live lookup is unavailable.
Remove the information you have stopped offering
A discontinued service needs the same attention as a new one. Search for old copies in uploaded documents and custom knowledge notes, as well as the web page. If a source has been replaced, check the old version is no longer available to answer from.
Keep a record of the change and when it was checked. Avoid leaving customer data or credentials in that record. The owner needs to see what changed, which channels passed and what is still waiting for a fix.
Agree who looks after failed updates
Ask where failed refreshes appear and who is expected to act on them. Set the check frequency around how often the business changes and the effect of an outdated answer. A changed safety instruction needs a different response from a new blog paragraph.
The maintenance job needs clear sources, checks after changes and a person responsible for failed updates. Keep that process with the chatbot’s handover so the business can see what is current.
Sources
Checked on 23 October 2026.
- Retell: knowledge base setup and retrieval tuning
Imported sources, exported-file limitations, distinct refresh and crawl functions, retained content after failed refreshes and inspection of retrieved passages.
- Microsoft: retrieval-augmented generation in Azure AI Search
Indexing, incremental updates and retrieval of relevant passages as parts of a maintained RAG system.
