Scrape Website Data with Python: From Requests + BeautifulSoup to AI-Ready Pipelines

A few years ago, web scraping meant saving rows to a CSV. The interesting endpoint now is a vector database – an AI knowledge base that answers questions, a structured JSON feed for an LLM agent, and an embeddings store that powers a chatbot grounded in live data.

The classic Requests + BeautifulSoup pair still handles static HTML fine. But most modern sites render content client-side. React, Next.js, Vue — a naive GET request sees an empty HTML shell where the actual content will never arrive. Based on community analysis in 2026, well over half of top-traffic sites render significant content client-side, which breaks any scraper that doesn’t run JavaScript. You need different tools for different sites, and the right pipeline to turn the output into something an LLM can use.

This guide covers the full stack: sync scraping with Requests + BS4, async fan-out with httpx, JavaScript rendering with Playwright, scaled crawls with Scrapy, LLM-friendly extraction with Crawl4AI, structured output via GPT function calling, and the full scrape → chunk → embed → retrieve RAG pipeline.

Ethics and legal notice: Always check robots.txt before scraping. Respect Crawl-delay directives. Don’t scrape authentication-walled content. Rate-limit your requests — 1 req/second per domain is a reasonable default. Terms of Service violations can have legal consequences.

Setup: Install Everything Once

Start with a virtual environment, then install all the libraries this guide uses:

# Create and activate venv
python -m venv venv && source venv/bin/activate  # Linux/Mac
# python -m venv venv && venv\Scripts\activate  # Windows

# Install all libraries
pip install requests httpx beautifulsoup4 lxml playwright scrapy crawl4ai openai

# Playwright needs its browser binaries separately
playwright install chromium

requests uses simple HTTP synchronization. httpx works the same way, but with asynchronous support and HTTP/2. BeautifulSoup with lxml performs HTML parsing. playwright uses an actual browser to handle JavaScript rendering. scrapy comes as a complete framework where you need to do retries and de-duplication with item pipelines. crawl4ai can crawl the data and provide it back as markdown directly for your RAG pipeline. openai is used for LLM extraction calls.

Static Pages: Requests + BeautifulSoup

If the site returns full HTML on the first GET – news sites, documentation, most blogs, many ecommerce product pages – Requests + BeautifulSoup is still the right call. No browser process, no overhead, fast.

GET request

import requests
headers = {'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7)'}
r = requests.get('https://example.com', headers=headers, timeout=10)
r.raise_for_status()  # Raises HTTPError on 4xx/5xx — always include this

The default Python user-agent string gets blocked by many servers – set a real browser one. And always set timeout=10. Without it, a slow or dead server hangs your script indefinitely.

Parse with BeautifulSoup

from bs4 import BeautifulSoup

soup = BeautifulSoup(r.text, 'lxml')

# Title
title = soup.title.string

# All paragraph text inside article elements
paragraphs = [p.get_text(strip=True) for p in soup.select('article p')]

# All links
links = [a['href'] for a in soup.find_all('a', href=True)]

# Images with alt text
images = [{'src': img['src'], 'alt': img.get('alt', '')} for img in soup.select('img')]

Save to CSV

import csv

rows = [[title, p] for p in paragraphs]
with open('output.csv', 'w', newline='', encoding='utf-8') as f:
    csv.writer(f).writerows(rows)

Parser choice

Using ‘lxml’ as the parser argument is the fastest way. ‘html.parser’ is slower, but it’s built-in with Python’s standard library – so no extra installs. ‘html5lib’ is the most tolerant of broken markup, which matters for the kinds of HTML that government sites and old CMSes produce. Default to lxml and only switch if you hit malformed pages that BS4 is misreading.

Going faster: httpx and async scraping

Sync requests are fine for one page. But if there are 500 URLs, you’ll wait for one request to finish before making another. This wastes your time. The httpx module allows asynchronous requests with an interface almost the same as requests. Thus, the switch takes around 10 minutes. HTTP/2 and connection pooling are supported out of the box.

import httpx, asyncio
async def fetch(client, url):
    r = await client.get(url, timeout=10)
    r.raise_for_status()
    return r.text
async def scrape_many(urls):
    sem = asyncio.Semaphore(10)  # Max 10 concurrent requests
    async def bounded(client, url):
        async with sem:
            return await fetch(client, url)
    async with httpx.AsyncClient(
        headers={'User-Agent': 'Mozilla/5.0'}
    ) as client:
        return await asyncio.gather(*(bounded(client, u) for u in urls))
htmls = asyncio.run(scrape_many(['https://...', 'https://...']))

The asyncio.Semaphore(10) cap is important. Without it, 500 URLs fire 500 simultaneous requests, which will get your IP banned faster than anything else. Throughput gain over sequential requests: 10–50x depending on how much of the time is network wait vs CPU work. 

Which to use in 2026

For new projects, just start with httpx – HTTP/2, async, nearly identical API. The only reason to keep using requests is an existing codebase that’s already built around it. 

JavaScript-Rendered Pages: Playwright

React, Next.js, Vue, Angular – regardless of which SPA you use, it is going to render the HTML via JavaScript after the page is loaded. Make a simple GET request to any of them, and you will receive an HTML skeleton containing a root div element. Nothing for BeautifulSoup to analyze here, a browser is necessary.

Playwright is what most teams use in 2026. Selenium still works, but it’s slower, noisier, and the API hasn’t kept up. Puppeteer requires JavaScript. Playwright has solid Python bindings, real async support, and handles Chromium, Firefox, and WebKit.

Basic synchronous example

from playwright.sync_api import sync_playwright

with sync_playwright() as p:
    browser = p.chromium.launch(headless=True)
    page = browser.new_page()
    page.goto('https://example.com', wait_until='networkidle')
    html = page.content()  # Full DOM after JS execution
    browser.close()

# Now parse with BeautifulSoup as normal
soup = BeautifulSoup(html, 'lxml')

Key Playwright tactics

wait_until=’networkidle’ waits until there are no in-flight network requests for 500ms – works for most SPAs. If the page lazy-loads on scroll or waits for a specific element, page.wait_for_selector(‘.product-card’) is more reliable than a timer. 

# Infinite scroll: scroll to bottom in a loop
for _ in range(5):
    page.evaluate('window.scrollTo(0, document.body.scrollHeight)')
    page.wait_for_timeout(1000)

# Intercept XHR and grab JSON directly (often easier than DOM parsing)
def handle_response(response):
    if 'api/products' in response.url:
        print(response.json())
page.on('response', handle_response)

# Block images and fonts to speed up page loads
page.route('**/*.{png,jpg,gif,woff,woff2}', lambda r: r.abort())

On React apps, the XHR intercept trick is often cleaner than DOM scraping. The API response is already structured JSON – you don’t need to reverse-engineer the HTML layout at all.

Playwright is slow – use it only when you have to
5–20x slower per page than a plain HTTP request. Always try Requests first and check whether the actual content is in the initial HTML response. Pull in Playwright only when it isn’t.

Static Pages: Requests + BeautifulSoup

One-off scripts break down fast when you’re crawling thousands of URLs – you need retries when servers flake out, deduplication so you don’t hit the same URL twice, rate limiting so you don’t get banned, and a way to clean and export the output. Scrapy handles all of that. It’s been the production crawler framework in Python for a decade, and it’s still the right tool for serious crawl jobs.

Once you use Scrapy, you don’t need to write manually: request management with CONCURRENT_REQUESTS and DOWNLOAD_DELAY, automatic retries via RetryMiddleware, robots.txt compliance by setting ROBOTSTXT_OBEY = True, URL deduplication, item pipelines for cleaning and validating scraped data, and export to JSON/CSV/XML with a single flag.

Minimal spider

# Save as myproject/spiders/blog_spider.py
import scrapy

class BlogSpider(scrapy.Spider):
    name = 'blog'
    start_urls = ['https://example.com/blog']

    def parse(self, response):
        for post in response.css('article'):
            yield {
                'title': post.css('h2::text').get(),
                'url': post.css('a::attr(href)').get(),
                'summary': post.css('p::text').get('').strip(),
            }

        # Follow "next page" link automatically
        next_page = response.css('a.next::attr(href)').get()
        if next_page:
            yield response.follow(next_page, self.parse)

# Run and output to JSON
scrapy crawl blog -O posts.json

If you need to distribute across multiple machines, Scrapy-Redis swaps the in-memory request queue for a Redis-backed one. The great thing is your spider code stays the same – only the queue backend changes.

LLM-Assisted Scraping: Crawl4AI and Structured Extraction

CSS selectors have a fragility problem: the site redesigns, .product-title becomes .product-name, and your scraper silently returns nothing for a week before someone notices. LLM extraction sidesteps this – the model reads the HTML or markdown and pulls out the fields regardless of what the class names are. 

Crawl4AI

Crawl4AI is an open-source async crawler that crawls a page – including JS-rendered ones – and returns clean markdown. The v0.8.x release added adaptive crawling (stops when enough info is gathered), BFS/DFS/BestFirst deep crawl strategies, and built-in proxy rotation. The output from fit_markdown has navigation and boilerplate stripped out – you can feed it directly into an embedding pipeline. 

import asyncio
from crawl4ai import AsyncWebCrawler
async def main():
    async with AsyncWebCrawler() as crawler:
        result = await crawler.arun(url='https://example.com/docs')
        # result.markdown.raw_markdown — full markdown
        # result.markdown.fit_markdown — boilerplate stripped
        print(result.markdown.fit_markdown)
asyncio.run(main())

For multiple pages at once, arun_many() handles concurrency for you. The fit_markdown output is usually ready to chunk and embed without any further cleaning steps.

Structured output via OpenAI function calling

When you need specific fields – product name, price, SKU, availability – define a JSON schema and let the LLM fill it in. Outperforms hand-written selectors on any site that changes layout regularly. 

from openai import OpenAI
import json

client = OpenAI()

# html = the page HTML or markdown (truncate to ~50K chars for cost control)
response = client.chat.completions.create(
    model='gpt-4o-mini',
    response_format={
        'type': 'json_schema',
        'json_schema': {
            'name': 'product',
            'schema': {
                'type': 'object',
                'properties': {
                    'name':         {'type': 'string'},
                    'price_usd':    {'type': 'number'},
                    'in_stock':     {'type': 'boolean'},
                    'description':  {'type': 'string'},
                },
                'required': ['name', 'price_usd', 'in_stock'],
            }
        }
    },
    messages=[{
        'role': 'user',
        'content': f'Extract the product fields from this page: {html[:50000]}'
    }]
)

product = json.loads(response.choices[0].message.content)

Cost: gpt-4o-mini is $0.15 per million input tokens. A 50K-character page is roughly 12K tokens – about $0.0018 per extraction call. 100K pages = ~$180. That’s cheaper than the engineering hours spent maintaining selectors that break every few months when the site gets redesigned. 

Anti-Bot, Ethics, and robots.txt

This is the section most tutorials skip, and then people are surprised when their IP gets banned or they get a legal letter.

robots.txt

Check https://target.com/robots.txt before you write a single line of scraping code. Ignoring it won’t necessarily get you sued, but it’s legally murky in multiple jurisdictions, and you’ll almost certainly get blocked. Check it programmatically so your scraper respects it automatically:

from urllib import robotparser

rp = robotparser.RobotFileParser()
rp.set_url('https://example.com/robots.txt')
rp.read()

if rp.can_fetch('*', 'https://example.com/products'):
    # OK to scrape
    pass
else:
    print('Blocked by robots.txt')

Rate limiting

1 request per second per domain is a reasonable baseline. Hammering a site with 50 concurrent requests is the fastest way to get banned and is genuinely inconsiderate. Use time.sleep(1) in sync code. In async code, a Semaphore plus a sleep inside the bounded function works well.

Most large sites prohibit commercial scraping in their Terms of Service. The legal picture is genuinely complicated – public data, no auth bypass, and a polite rate is generally defensible in US law (see hiQ v. LinkedIn 2022). Scraping behind a login is a different situation entirely. If you’re unsure, ask a lawyer before shipping to production, not after.

Anti-bot services

Cloudflare, DataDome, and PerimeterX fingerprint headless browsers by looking for automation signals that Playwright leaves by default. The playwright-stealth package patches the most obvious ones. For heavy workloads on protected sites, managed services like ScrapingBee or ZenRows deal with the anti-bot layer for you – at a cost.

Proxies

For large jobs, residential proxy services (Bright Data, Oxylabs, Smartproxy) rotate your outbound IPs across real residential addresses. Cost is roughly $5-15 per GB. Don’t bother with small one-off scripts – just slow down and set a sensible User-Agent.

From Scrape to RAG: Feeding Scraped Data into an AI Knowledge Base

Most scraping projects in 2026 don’t end with a CSV. They end with a chatbot that can answer questions about the scraped content, or a search interface grounded in real data. That means a vector store, not a spreadsheet. Here’s the full pipeline:

Step 1: Scrape to clean markdown

Markdown preserves the document hierarchy – headers, lists, links – better than stripping to plain text. Crawl4AI’s fit_markdown removes the nav, footer, and cookie banners while keeping the actual content. For JS-rendered sites, it handles the browser internally.

from crawl4ai import AsyncWebCrawler

async with AsyncWebCrawler() as crawler:
    pages = await crawler.arun_many(urls=sitemap_urls, max_concurrent=5)
    docs = [p.markdown.fit_markdown for p in pages if p.success]

Step 2: Chunk by headers

Split on H1/H2 boundaries. Target 500-1,000 tokens per chunk with about 100 tokens of overlap so context doesn’t get severed mid-sentence. LangChain’s MarkdownHeaderTextSplitter does this cleanly, or just split on # and ## if you want to avoid the LangChain dependency.

from langchain_text_splitters import MarkdownHeaderTextSplitter

splitter = MarkdownHeaderTextSplitter(
    headers_to_split_on=[("#", "H1"), ("##", "H2"), ("###", "H3")]
)
chunks = splitter.split_text(markdown_doc)

Step 3: Embed 

OpenAI’s text-embedding-3-small is $0.02 per million tokens. A corpus of 10,000 pages typically costs a couple of dollars to embed. If you want to keep data off OpenAI’s infrastructure, nomic-embed-text or bge-large-en-v1.5 via Ollama are solid alternatives.

from openai import OpenAI

client = OpenAI()
texts = [chunk.page_content for chunk in chunks]

# Batch up to 2048 inputs per call
response = client.embeddings.create(
    model='text-embedding-3-small',
    input=texts
)
vectors = [item.embedding for item in response.data]

Step 4: Store in a vector database

If you already run Postgres, pgvector is the zero-friction option – a single extension, standard SQL queries, nothing new to operate. Qdrant is the best dedicated open-source vector DB if you want something purpose-built with a decent free tier. Pinecone is fully managed if you’d rather not run the DB yourself. Weaviate handles hybrid search (vector plus keyword) if you need both.

# Example: upsert to Qdrant
from qdrant_client import QdrantClient
from qdrant_client.models import PointStruct

client_q = QdrantClient(':memory:')  # or url='http://localhost:6333'
points = [
    PointStruct(id=i, vector=vectors[i], payload={'text': texts[i]})
    for i in range(len(vectors))
]
client_q.upsert(collection_name='docs', points=points)

Step 5: Retrieve and augment

Embed the user’s question, find the top-k closest chunks by cosine similarity, inject them into the prompt as context. If precision matters more than speed, add a reranker between retrieval and generation – Cohere Rerank and BGE Reranker both work well here.

# Query time
q_vec = client.embeddings.create(model='text-embedding-3-small', input=[query]).data[0].embedding
hits = client_q.search(collection_name='docs', query_vector=q_vec, limit=5)
context = '\n\n'.join([h.payload['text'] for h in hits])

answer = client.chat.completions.create(
    model='gpt-4o-mini',
    messages=[
        {'role': 'system', 'content': 'Answer using only the provided context.'},
        {'role': 'user', 'content': f'Context:\n{context}\n\nQuestion: {query}'}
    ]
)

Step 6: Incremental refresh

Don’t re-embed the whole corpus on every crawl. Hash each page’s content, store the hash, and only re-embed pages where the hash changed. Weekly full crawl with daily diff checks is usually enough. This cuts the embedding cost by 80-95% once the initial corpus is built.

Build Your Own: Skip the Pipeline with Ethora RAG Crawler

The pipeline above works. But it’s also six moving parts: crawler, chunker, embedder, vector store, retrieval layer, and LLM connection. If you just need a chatbot grounded in a website, that’s a lot of infrastructure to own. 

Ethora’s AI Bots SDK with RAG Crawler does the whole thing in one module. Point it at a URL or sitemap. It crawls the site – static and JS-rendered – strips boilerplate, chunks by headers, embeds, and writes to either a managed vector store or your own pgvector/Qdrant. Then it connects directly to the AI Bots SDK so you can query the result from within your chat interface.

Incremental crawls use content hashing so only changed pages get re-embedded – the same optimization you’d build anyway, already there. The BYO-LLM and BYO-embedding setup means you can point it at nomic-embed-text running locally on your own GPU, and nothing leaves your network. That’s the full stack for a HIPAA-compliant chat SDK where prompts genuinely can’t touch OpenAI’s API.

The Ethora MCP server exposes the knowledge base to LLMs running in Cursor, VS Code, or Claude Desktop.

The RAG Crawler is included in the free tier. Point it at your URL, and your chatbot has context without building the pipeline yourself.