How to reduce AI token costs when scraping the web

When a model receives raw HTML, it may process navigation, scripts, styles, consent text, repeated links, and other page furniture alongside the content you actually wanted.

Cleaning the page first can reduce that unnecessary input. The important word is can: the result depends on the source, rendering mode, extraction policy, and what your application considers relevant.

Recorded reference, not a verified benchmark

An earlier version of this article presented the following token counts as a benchmark. They are retained as a recorded reference only. A reproducibility manifest covering fixtures, timestamps, runner configuration, raw outputs, and the measurement script has not yet been published.

| Recorded fixture | Raw HTML | Clean content | Recorded reduction | |---|---:|---:|---:| | GitHub README | 14,847 | 1,026 | 93.1% | | BBC News article | 32,591 | 1,843 | 94.3% | | Wikipedia page | 28,103 | 3,412 | 87.9% | | Hacker News front page | 5,230 | 631 | 87.9% | | Xiaohongshu post | 51,208 | 892 | 98.3% |

These figures are not a guarantee, a verified cross-vendor comparison, or evidence for every page. Use your own fixtures before making a cost or capacity decision.

Measure the workload you actually have

Build a small corpus that represents your production sources. Include different page structures, content lengths, rendering requirements, and failure cases.

For each request, keep:

• the source URL and retrieval timestamp;
• the exact Purify and runner versions;
• request options and rendering mode;
• the cleaned content;
• tokens.original_estimate;
• tokens.cleaned_estimate;
• tokens.savings_percent;
• timing.total_ms, timing.navigation_ms, and timing.cleaning_ms;
• a human review of missing and unwanted material.

Then calculate a distribution across the corpus. An average alone can hide pages where extraction removed too much or retained too much.

Why tag stripping is not enough

Calling .get_text() on an HTML document removes tags but also discards useful structure. Headings become indistinguishable from paragraphs, tables lose their relationships, and code can collapse into ordinary text.

A content-cleaning pipeline usually needs to:

1. identify the main content region;
2. remove navigation, scripts, styling, repeated furniture, and unrelated sidebars;
3. preserve headings, lists, links, tables, and code where relevant;
4. return metadata and diagnostics that make the result auditable.

Aggressive cleaning has a real failure mode: it can delete context your application needed. Always inspect retention alongside token reduction.

Use the current API contract

The hosted scrape endpoint accepts a POST request with a JSON body:

import requests
 
def clean_scrape(url: str, api_key: str) -> dict:
    response = requests.post(
        "https://purify.verifly.pro/api/v1/scrape",
        json={"url": url},
        headers={"Authorization": f"Bearer {api_key}"},
    )
    response.raise_for_status()
    return response.json()
 
result = clean_scrape("https://example.com/article", "YOUR_API_KEY")
clean_content = result["content"]
token_report = result["tokens"]
timing_report = result["timing"]

The active response contract uses:

• content for cleaned page content;
• tokens.original_estimate, tokens.cleaned_estimate, and tokens.savings_percent for token diagnostics;
• timing.total_ms, timing.navigation_ms, and timing.cleaning_ms for timing diagnostics.

Use these active fields when updating older integrations.

Convert tokens into cost carefully

Model prices change. Instead of copying a static cost table, take the cleaned-token distribution from your own run and multiply it by the current input price published by the model provider you actually use.

Include the complete system cost: retrieval, browser capacity, retries, model input, storage, monitoring, and engineering time. Token reduction is one part of the decision, not the entire business case.

Benefits beyond cost

Cleaner input can make more room in the model context and reduce the chance that navigation or consent text appears in an answer. It can also improve retrieval quality when embeddings represent the document rather than the page chrome.

Those outcomes still require evaluation. Test answer quality, citation accuracy, and missing-content cases against a labeled question set.

Hosted or self-hosted

Use the hosted API when you want Purify to operate the service. Use the Apache-licensed core when your team wants to own deployment and the data boundary. Self-hosting removes a managed provider from the path, but it does not remove infrastructure or operational cost.

The reliable rule is simple: clean first, measure on your own corpus, and keep enough evidence to understand what was removed.