Retrieval-Augmented Generation (RAG) pipelines live and die by their embeddings. If you feed raw, unoptimized web data into a text chunker, your vector database will be poisoned by navigation menus, footer links, cookie banners, and inline CSS.
Naive implementations often request an HTML page, run a regex to strip tags, and pass the resulting text wall into a character splitter. This destroys structural context. A chunk might end mid-sentence, or worse, blend a critical paragraph with a site's privacy policy. When the LLM retrieves this context, the output hallucinates or misses the point entirely.
To build accurate RAG pipelines, data optimization must happen before chunking. You need a systematic approach to extract clean, semantically intact content from public web sources.
Phase 1: Reliable Data Ingestion
Modern web applications are client-side rendered. A simple HTTP GET request often returns an empty root div and a bundle of JavaScript. If your pipeline relies on static HTML fetching, it will miss the actual content entirely.
To get the data, you need to execute JavaScript, wait for network idle states, and capture the final DOM. Doing this at scale requires orchestrating headless browsers, managing rotating IP pools, and dealing with anti-bot handling.
Instead of maintaining that infrastructure, you can delegate the rendering phase to an API. Here is how you fetch the fully rendered DOM using our API.
Fetching the Rendered DOM
We require the raw HTML after all JavaScript has executed. Below are examples of fetching a target URL using both standard cURL and the dedicated SDK.
```bash title="Terminal" {2-3}
curl -X POST https://api.alterlab.io/v1/scrape \
-H "X-API-Key: YOUR_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"url": "https://example.com/technical-article",
"render_js": true
}'
For Python-based data pipelines, the [Python SDK](https://alterlab.io/web-scraping-api-python) handles the request and response parsing seamlessly.
```python title="ingest.py" {4-6}
client = alterlab.Client(api_key=os.getenv("ALTERLAB_API_KEY"))
response = client.scrape("https://example.com/technical-article", render_js=True)
raw_html = response.html
Once you have the raw_html, the actual extraction work begins.
Phase 2: Algorithmic Noise Reduction
A rendered web page contains the content you want, wrapped in hundreds of DOM nodes you don't. Injecting headers, footers, sidebars, and hidden modal text into your vector database degrades retrieval accuracy.
We need to prune the DOM tree before extracting text. This process is known as boilerplate removal.
Targeted DOM Pruning
Using a library like BeautifulSoup, we can violently prune elements that historically never contain primary content. This includes <script>, <style>, <nav>, <footer>, and specific ARIA roles.
```python title="cleaner.py" {11-12, 17-18}
from bs4 import BeautifulSoup
def prune_dom_noise(html_content: str) -> str:
soup = BeautifulSoup(html_content, 'html.parser')
# Define tags that are universally noise in a RAG context
noise_tags = [
'script', 'style', 'noscript', 'nav', 'footer', 'header',
'aside', 'iframe', 'canvas', 'svg', 'form'
]
for tag in soup(noise_tags):
tag.decompose()
# Remove elements based on common CSS class naming conventions
# that indicate non-core content
noise_classes = ['ad', 'banner', 'sidebar', 'menu', 'popup', 'cookie']
for element in soup.find_all(class_=lambda x: x and any(c in x.lower() for c in noise_classes)):
element.decompose()
# Remove elements explicitly marked as presentation or navigation
for element in soup.find_all(attrs={"role": ["navigation", "banner", "contentinfo"]}):
element.decompose()
return str(soup)
cleaned_html = prune_dom_noise(raw_html)
By decomposing these nodes entirely, we reduce the token payload by up to 80% and eliminate the most common sources of embedding pollution. The remaining HTML is a semantic shell of the actual article, product description, or documentation page.
### Advanced: Readability Scoring
For heavily unstructured pages, simple pruning isn't enough. You may need to implement a readability algorithm (similar to Mozilla's Readability.js). These algorithms score DOM nodes based on paragraph density, comma count, and text-to-tag ratios. Nodes with high scores are retained; low-scoring nodes are discarded. Libraries like `readability-lxml` in Python can automate this secondary filtering pass if your target domain layouts are highly unpredictable.
## Phase 3: Structural Mapping to Markdown
With a clean HTML string, the next mistake engineers make is calling `soup.get_text()`.
Stripping all tags converts structured data into a flat wall of text. You lose the distinction between an `<h1>` page title and a `<p>` paragraph. You lose the rows and columns of `<table>` data.
Vector databases don't understand HTML well, but LLMs and modern text splitters understand Markdown natively. Converting clean HTML to Markdown preserves semantic hierarchy. A Markdown header (`##`) signals a context shift to a text chunker, ensuring that chunks are broken precisely at section boundaries rather than arbitrarily at a character limit.
```python title="mapper.py" {6-8}
from cleaner import prune_dom_noise
def html_to_markdown(html_content: str) -> str:
cleaned_html = prune_dom_noise(html_content)
# Convert to markdown, explicitly preserving structures that LLMs understand
md = markdownify.markdownify(
cleaned_html,
heading_style="ATX",
strip=['img', 'a'], # Optional: strip links/images if they distract from core text
bullets="-",
strong_em_symbol="**"
)
# Clean up excessive empty lines generated by tag stripping
clean_md = "\n".join([line for line in md.splitlines() if line.strip()])
return clean_md
markdown_data = html_to_markdown(raw_html)
The Final Step: Intelligent Chunking
Because you preserved the structure in Markdown, you can now use a specialized text splitter. Instead of blindly chopping text every 1,000 characters, you can split by Markdown headers.
If you are using LangChain, the MarkdownHeaderTextSplitter consumes the output of your pipeline perfectly:
```python title="chunker.py" {3-5}
from langchain_text_splitters import MarkdownHeaderTextSplitter
headers_to_split_on = [
("#", "Header 1"),
("##", "Header 2"),
("###", "Header 3"),
]
markdown_splitter = MarkdownHeaderTextSplitter(
headers_to_split_on=headers_to_split_on
)
This returns semantic chunks bounded by actual page sections
md_header_splits = markdown_splitter.split_text(markdown_data)
If a section under an `<h2>` tag is 800 characters long, it becomes a single, highly cohesive vector embedding. The metadata attached to the chunk will include the header names, giving the LLM precise context about where this text lived in the original document hierarchy.
## Takeaways
Optimizing extraction before chunking dramatically reduces hallucination rates in RAG pipelines.
1. **Never scrape raw HTML directly into a text splitter.** Get the final rendered DOM to ensure you aren't missing data.
2. **Prune aggressively.** Strip `<nav>`, `<footer>`, and `<script>` tags to prevent UI text from polluting your embeddings.
3. **Map HTML to Markdown.** Preserve structural indicators like headers and tables.
4. **Chunk by semantics, not by characters.** Use Markdown-aware splitters to keep logically grouped text in the same vector.
By treating data extraction and transformation as a first-class citizen in your RAG architecture, you ensure your LLM is retrieving high-signal, zero-noise context. For further configuration details on optimizing your extraction pipelines, refer to our [API docs](https://alterlab.io/docs).
Top comments (0)