Your website is usually the best place to start teaching your chatbot. Liyobot scans your site, shows you the pages it found, and reads only the ones you choose. Each page becomes searchable as soon as it’s been read.
Scan your site#
Enter your website address
Open + Add source, choose the Website tab and type your Website URL, usually your home page.
Pick a depth
Whole site looks everywhere. 1 level or 2 levels only follows links that many steps from the address you entered, which is handy for one section of a large site.
Click Scan site
“We list the pages found first, so you can choose what gets crawled.” Nothing is read or counted against your allowance yet.
A scan finds up to 1,000 pages. It follows links on your site and also checks your sitemaps, including sitemap.xml, WordPress sitemaps and any listed in robots.txt. Important pages, such as your home page and short top-level pages, are listed first. Archives, tags, pagination and feeds sink to the bottom.
Choose your pages#
The results show “Found N pages · M selected” and a checklist of pages with their PAGE address and DEPTH (Root, 1, 2 and so on). Use the Filter paths box to narrow the list, for example to everything under /services, then tick or untick pages.
Missing a page? Click Add link and paste its address. Only public pages on the same website can be added. If the address isn’t valid, or is already in the list, you’ll be told.
Crawl and follow progress#
Click Crawl N pages to start, or Cancel to go back. Pages are read one by one and progress shows in the sources table below, for example “Scraping 12/40 pages…”. You can leave the page while it works. The bot can already use pages that have finished, even before the whole crawl is done.
Page allowance#
Each chatbot can index a set number of pages. This is a hard limit:
| Plan | Indexed pages per chatbot |
|---|---|
| Starter | 100 |
| Growth | 1,000 |
| Business | 2,000 |
| Enterprise | Custom |
If a crawl would go over, Liyobot tells you how many pages remain, for example “This crawl would add 80 pages but only 30 remain…”. Untick some pages, remove pages you don’t need from an existing source, or upgrade. Pages found as listings also count toward the allowance.
Automatic retries#
Websites sometimes fail to load for a moment. Each page is tried several times, and pages that still fail are retried automatically in the background, up to five times, waiting longer between each attempt. Pages that don’t exist (404 errors) or are private aren’t retried, because they won’t succeed. You can also retry failed pages yourself from View pages.
Add pages from a list (Bulk links)#
Some websites hide pages from a scan, for example pages that aren’t linked from anywhere or are loaded by scripts. If you already have the links, from your own list or another tool, add them directly with the Bulk links tab.
Open the Bulk links tab
Click + Add source and choose Bulk links.
Paste your links
Separate them with commas, spaces or new lines. You can leave out
https://; Liyobot adds it. A preview shows how many valid links it found, removes duplicates, and lists anything it doesn’t recognise as a web link so you can fix it.Check the sites
Links are grouped by website, for example “12 links on example.com · 3 on partner.com”. Click a site to see its links. Up to 1,000 links per site can be added at a time.
Click Add links
Liyobot creates one website source per site and reads exactly those pages, one by one. Each site shows ✓ or the reason it failed; links from a failed site stay in the box so you can try again.
Tips for choosing pages#
- Include pages that answer customer questions: services, pricing, FAQs, contact details, opening hours and policies.
- Skip pages with no useful answers: login, cart, checkout, account, tag and archive pages. They use up your allowance without helping.
- Avoid duplicates. Paginated lists and category archives often repeat content found elsewhere.
- Fill gaps with text. If important information isn’t on your website, add it as a text snippet.