A retrieval system can return the correct source and still generate a poor answer when its corpus is full of navigation, repeated sidebars, consent text, and stale fragments.
The fix begins upstream. A useful web-to-RAG pipeline treats source discovery, retrieval, cleaning, chunking, embedding, refresh, and evaluation as separate stages with evidence preserved between them.
Why RAG collection is different
Selector-based scraping usually targets known fields. RAG collection needs the meaningful body of a page while preserving semantic structure: headings, paragraphs, lists, tables, code, links, and source metadata.
The goal is not the smallest possible output. It is the smallest output that still contains the evidence needed to answer questions correctly.
Stage one: discover sources
Choose discovery based on the corpus:
- Sitemap or URL map: useful when the publisher exposes a stable inventory.
- Scoped crawl: useful when pages are linked but not listed in one manifest.
- Manual curation: useful for a small, high-value corpus where inclusion needs editorial judgment.
- Application feed or API: preferable when the source already publishes a structured and supported interface.
Purify exposes map and crawl operations, but discovery should still have an explicit domain boundary, path rules, deduplication policy, and stopping condition.
Store why each URL was included. That decision becomes valuable when a corpus grows or a source should later be removed.
Stage two: retrieve and clean
Use the current POST contract for a hosted scrape:
import requests
def scrape_clean(url: str, api_key: str) -> dict:
response = requests.post(
"https://purify.verifly.pro/api/v1/scrape",
json={"url": url},
headers={"Authorization": f"Bearer {api_key}"},
)
response.raise_for_status()
return response.json()
result = scrape_clean("https://example.com/docs", "YOUR_API_KEY")
clean_content = result["content"]The active response uses content, not the legacy markdown field. Request-level diagnostics are nested:
tokens.original_estimate;tokens.cleaned_estimate;tokens.savings_percent;timing.total_ms;timing.navigation_ms;timing.cleaning_ms.
Do not treat tokens.savings_percent as a quality score. Review whether the cleaner preserved the title, argument, lists, tables, examples, and citations your retrieval system needs.
Stage three: preserve provenance
Before chunking, store a source record alongside the clean content:
source_record = {
"source_url": source_url,
"final_url": result.get("final_url"),
"retrieved_at": retrieved_at,
"content": result["content"],
"tokens": result.get("tokens", {}),
"timing": result.get("timing", {}),
"content_hash": content_hash,
}Also keep the extractor version and request options in your own run manifest. Without them, a changed result is difficult to explain.
Stage four: chunk by meaning
Heading-aware chunking is a practical starting point for documentation and editorial content:
def chunk_by_heading(content: str, source_url: str) -> list[dict]:
chunks = []
heading = "Introduction"
lines = []
def flush():
if lines:
chunks.append({
"text": "\n".join(lines).strip(),
"heading": heading,
"source_url": source_url,
})
for line in content.splitlines():
if line.startswith("## "):
flush()
heading = line.removeprefix("## ").strip()
lines = [line]
else:
lines.append(line)
flush()
return [chunk for chunk in chunks if chunk["text"]]Real documents complicate this pattern. A table may belong with the paragraph that explains it. A code block may depend on the heading above it. A short section may need its parent heading for context.
Avoid publishing one “ideal” chunk size as a universal rule. Select candidate policies, evaluate retrieval on representative questions, and inspect which evidence each policy returns.
Stage five: embed and store
The vector record should contain more than an embedding:
- chunk text;
- source URL and final URL;
- section heading and hierarchy;
- retrieval timestamp;
- content hash and chunk hash;
- extractor and embedding model versions;
- access or tenancy metadata needed by your application.
The hashes help avoid unnecessary re-embedding and make changed content visible. Version fields let you distinguish a source update from a pipeline update.
Use the embedding provider and model that fit your language coverage, privacy requirements, and evaluation results. Check the provider's current documentation and pricing rather than copying a model or cost assumption from an old tutorial.
Stage six: retrieve and answer
At query time:
- embed the question with the same compatible model family used for the corpus;
- apply access filters before or during retrieval;
- retrieve candidate chunks;
- optionally rerank them;
- pass the selected evidence and source identifiers to the generation model;
- require citations and an explicit “not enough evidence” behavior.
Log the retrieved chunk identifiers for every evaluated answer. Without retrieval traces, it is easy to blame the generation model for a collection or ranking problem.
Refresh without losing control
Web pages change. A refresh pipeline should:
- revisit sources according to their change rate and importance;
- compare content hashes;
- re-chunk and re-embed only when policy or content changed;
- preserve previous artifacts long enough to diagnose regressions;
- remove records when a source is no longer authorized or included;
- surface repeated access or extraction failures for review.
A fixed schedule is not automatically correct. Fast-moving sources and stable reference documents need different refresh policies.
Evaluate every stage
Create a labeled evaluation set with real user questions and expected supporting passages. Track:
- source discovery coverage;
- extraction retention and noise;
- retrieval recall for supporting evidence;
- citation correctness;
- unsupported-answer rate;
- freshness after source updates;
- errors and latency by pipeline stage.
Earlier versions of this article included exact token-cost and retrieval-cost tables without a published reproducibility manifest. Those tables have been removed. Model pricing changes, and a recorded token reduction on one corpus does not predict another corpus.
Use the token and timing diagnostics from your own Purify responses, publish the fixture and runner manifest internally, and calculate cost using the current provider price for your chosen models.
Common failure modes
Embedding raw page chrome. Repeated navigation can dominate retrieval because it appears across many pages.
Removing too much. A very compact extraction can omit the table, warning, or code example that answers the question.
Dropping provenance. A chunk without its source and retrieval time cannot support a trustworthy citation.
Testing with copied wording. Queries copied from the source make retrieval look better than it will for real users.
Ignoring access filters. Retrieval must not expose a chunk merely because it is semantically similar.
Skipping refresh evidence. If old and new artifacts are not traceable, regressions become difficult to distinguish from source changes.
A practical starting sequence
- Curate a representative source set and question set.
- Retrieve with the current POST API and store full response diagnostics.
- Review content retention before embedding anything.
- Test several chunking policies against the labeled questions.
- Store provenance, hashes, and version metadata with every chunk.
- Add refresh and deletion handling before the corpus becomes large.
- Keep retrieval traces and citations in the evaluation output.
The quality of RAG answers is constrained by the evidence pipeline. Clean the page, preserve the record, and make every transformation inspectable.