Glossary
Crawler (URL, Website)
An automated program that systematically browses and extracts content from websites to build searchable indexes and knowledge bases. In Ethora, crawlers feed your AI chat bot with up-to-date content from your own site.
General definition
A website URL crawler is an automated program that systematically browses and extracts content from websites. It works through three stages — discovery, extraction and indexing:
- Start with initial URLs and download their HTML while respecting server constraints.
- Parse and process the information found on each page.
- Create indexed repositories that can be searched and retrieved later.
Crawlers in the Ethora ecosystem
In Ethora, crawlers build an intelligence layer for business applications rather than indexing the open web. When an AI chat bot needs to answer a customer question, the crawler ensures it has access to the most current and accurate information directly from your website.
Crawled content is automatically organized into searchable, contextual chunks that are ideal for RAG retrieval — eliminating manual data entry for products, services and policies. Crawling is available across Ethora’s embeddable AI widgets, no-code builder, WordPress plugin and AI SDK, keeping the knowledge base in continuous sync with your site.
