You ship a docs fix on Tuesday. On Wednesday you find out you accidentally deleted the API-key section, broke a pricing table, and shipped a typo into your homepage hero. Nobody noticed for a week — because nobody was diffing the pages you just published.
Manual page comparison doesn't scale. You can't eyeball 400 marketing pages, docs pages, and legal pages after every deploy. What you need is automated page comparison: a pipeline that captures what your pages look like today, compares them to what they looked like yesterday (or to production), and flags meaningful content drift before your users do.
This guide shows you how to build that pipeline with DiffCheck, the web-page diff API in the FetchAPI suite, plus Markdownify for the clean-text half of the workflow. DiffCheck is currently coming soon to the FetchAPI suite, so we'll also show the live, working fallback you can run today.
The naive approach — fetch the page twice and run a text diff — falls apart immediately:
Automatic comparison only works when the page is normalized first (boilerplate stripped, content reduced to clean text) and the diff is returned semantically — added, removed, and changed passages you can act on.
DiffCheck accepts two shapes, depending on whether you're comparing two URLs or diffing against a stored snapshot.
Compare two URLs:
curl -s -X POST "https://fetchapi.tech/v1/diffcheck" \
-H "Content-Type: application/json" \
-d '{
"url_a": "https://staging.example.com/pricing",
"url_b": "https://example.com/pricing"
}'
Compare two text blocks — perfect for diffing a page against a snapshot you stored yourself:
curl -s -X POST "https://fetchapi.tech/v1/diffcheck" \
-H "Content-Type: application/json" \
-d '{
"text_a": "Pro plan — $49/month",
"text_b": "Pro plan — $39/month"
}'
Both return the same structured shape:
{
"added": ["AI-powered insights"],
"removed": ["Priority support"],
"unchanged": ["Pro", "plan", "$49", "$39", "month"],
"summary": "3 words changed"
}
Or, in single-URL mode, DiffCheck tracks the page's history for you and returns just what moved since the last check:
curl -s "https://fetchapi.tech/v1/diffcheck?url=https://example.com/pricing"
{
"changed": true,
"diff": [
{ "type": "change", "old": "$49/month", "new": "$39/month" },
{ "type": "add", "text": "Includes AI-powered insights" }
]
}
Clean. Semantic. Immediately actionable — and trivial to assert on in a CI job.
Because DiffCheck is still rolling out, here's the same automatic comparison pattern using the live Markdownify endpoint. Markdownify converts any URL to clean markdown — boilerplate stripped — so the diff is meaningful instead of noisy:
curl -s "https://fetchapi.tech/v1/markdownify?url=https://example.com"
{
"content": "# Example Domain\n\nThis domain is for use in documentation examples without needing permission.\n\n[Learn more](https://iana.org/domains/example)",
"title": "",
"description": "",
"url": "https://example.com",
"word_count": 20,
"char_count": 167
}
You convert both pages to markdown, then diff the content fields. Snapshot storage plus Markdownify gives you a working regression pipeline right now; swap in DiffCheck later for one-call semantics.
The highest-value automation is catching content regressions before they reach users. Treat page content like code: assert on it in CI.
import os, requests, sys
MARKDOWNIFY = "https://fetchapi.tech/v1/markdownify"
def content(url):
r = requests.get(MARKDOWNIFY, params={"url": url}, timeout=30)
r.raise_for_status()
return r.json()["content"]
def compare(a, b, tolerance=0.05):
ta, tb = content(a), content(b)
if ta == tb:
return True
# crude similarity guard: large divergence = regression
import difflib
ratio = difflib.SequenceMatcher(None, ta, tb).ratio()
print(f"similarity {ratio:.3f} between {a} and {b}")
return ratio >= 1 - tolerance
if __name__ == "__main__":
ok = compare(
"https://staging.example.com/pricing",
"https://example.com/pricing",
)
sys.exit(0 if ok else 1)
Run this in a GitHub Actions step after your staging deploy. A non-zero exit fails the build and blocks the release:
# .github/workflows/content-regression.yml
name: Content regression
on:
deployment_status:
jobs:
diff:
if: github.event.deployment_status.state == 'success'
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- run: pip install requests
- run: python scripts/compare_pages.py
Now a deleted section or a broken pricing table fails the pipeline instead of shipping silently.
Some pages change on their own — competitor sites, partner docs, legal pages, dependency changelogs. You don't want to diff against a fixed baseline; you want to diff against the last known state so you catch every change, even ones that happened while your monitor was down.
#!/bin/bash
set -euo pipefail
URL="${1:?usage: driftwatch.sh <url>}"
STATE="/var/lib/driftwatch/$(echo "$URL" | md5sum | cut -d' ' -f1).md"
mkdir -p "$(dirname "$STATE")"
# Clean current content via the live Markdownify endpoint
curl -s "https://fetchapi.tech/v1/markdownify?url=$URL" \
| jq -r '.content' > /tmp/driftwatch_now.md
if [ ! -f "$STATE" ]; then
cp /tmp/driftwatch_now.md "$STATE"
echo "Baseline captured for $URL"
exit 0
fi
# Diff against last known state (swap for POST /v1/diffcheck when live)
if ! diff -q "$STATE" /tmp/driftwatch_now.md >/dev/null; then
echo "Change detected: $URL"
diff -u "$STATE" /tmp/driftwatch_now.md | head -60
# alert here (Slack/webhook/agent), then update state
cp /tmp/driftwatch_now.md "$STATE"
else
echo "No change: $URL"
fi
Schedule it and you have zero-touch monitoring:
0 */6 * * * /usr/local/bin/driftwatch.sh https://docs.dependency.com/changelog
Feeding full page content into an LLM every run is expensive and wasteful. Automatic comparison lets your agent skip the token spend entirely when nothing changed — and re-read only what did:
def page_changed(url: str, previous_content: str) -> bool:
"""True if the page's clean content differs from what we last saw."""
r = requests.post("https://fetchapi.tech/v1/diffcheck", json={
"text_a": previous_content,
"url_b": url, # DiffCheck normalizes the URL via Markdownify first
}, timeout=30)
data = r.json()
return len(data.get("added", [])) + len(data.get("removed", [])) > 3
doc = fetch_markdown(url) # Markdownify
if page_changed(url, doc):
summary = llm.summarize(fetch_markdown(url))
notify(summary)
The pair is the point: DiffCheck tells you whether to spend tokens; Markdownify gives you the clean text when the answer is yes. For more on the extraction side, see Markdownify best practices for LLM content extraction.
Automated page comparison is one of the highest-leverage things you can add to a docs site, a marketing site, or a competitive-intelligence pipeline — and the whole thing is a snapshot, two API calls, and a diff.
DiffCheck (GET /v1/diffcheck and POST /v1/diffcheck) is finishing its rollout, so watch the FetchAPI blog for the launch announcement. In the meantime, Markdownify is live and the fallback above runs perfectly today — build the pipeline now, then switch to one-call semantics the moment DiffCheck ships.
Read the full API reference in the docs, and start comparing pages automatically instead of finding out a week too late.