← Back to blog

Markdownify API Best Practices for LLM Content Extraction: A Practical Guide

Every AI agent, RAG pipeline, and LLM-powered research tool has the same bottleneck: the web is written in HTML, and LLMs don't read HTML. Feed a raw webpage to a language model and you waste context window on navigation menus, cookie banners, inline scripts, and tracking pixels — while the actual content gets diluted into noise.

The fix is a clean intermediate representation, and markdown is the lingua franca of LLM content extraction. It's lightweight, structured, and close enough to plain text that models consume it efficiently. The Markdownify API from FetchAPI turns any URL into clean markdown with a single HTTP GET — no headless browser, no parsing library, no cleanup pipeline. This guide covers the best practices that separate a toy integration from a production-grade content pipeline.

What Markdownify Returns

The endpoint is dead simple. Pass a URL, get markdown back as JSON:

curl "https://fetchapi.tech/v1/markdownify?url=https://example.com"

{
  "content": "# Example Domain\n\nThis domain is for use in documentation examples without needing permission. Avoid use in operations.\n\n[Learn more](https://iana.org/domains/example)",
  "title": "",
  "description": "",
  "url": "https://example.com",
  "word_count": 20,
  "char_count": 167
}

Every field earns its keep:

content — the page body as clean markdown, with links preserved as [text](href) and headings as real # levels. This is what you feed to your LLM.
title / description — extracted page metadata, useful for cataloguing.
word_count / char_count — instant token-budget math before you spend a single embedding or completion call.
url — the final URL after any redirects, so you always know what you actually fetched.

Best Practice 1: Convert Before You Chunk, Chunk Before You Embed

The most common mistake in RAG pipelines is chunking raw HTML and hoping the embedding model sorts it out. It won't. A typical article page is 60–120 KB of HTML; the same article as markdown is often 5–15 KB. That's a 10x token reduction before any processing, plus a massive drop in embedding noise.

The correct order is:

URL → Markdownify → clean markdown → semantic chunking → embeddings → vector store

Markdown structure gives you natural chunk boundaries for free. Headings become section delimiters, code blocks stay intact, and lists keep their itemization. A simple heading-aware chunker over content outperforms a naive fixed-size token splitter on HTML every time.

Best Practice 2: Budget Tokens with word_count Before Calling Your LLM

Because the response includes word_count and char_count, you can decide before paying for tokens whether a page is worth processing. A rough rule of thumb: 1 word ≈ 1.3 tokens for English text. If a page is 5,000 words, that's roughly 6,500 tokens of context — under most modern context windows. If it's 50,000 words, you know to summarize instead of stuff.

import requests

r = requests.get("https://fetchapi.tech/v1/markdownify",
                 params={"url": "https://news.ycombinator.com"})
data = r.json()

estimated_tokens = int(data["word_count"] * 1.3)
print(f"{data['word_count']} words, ~{estimated_tokens} tokens")

if estimated_tokens < 8000:
    # feed the full markdown to your model
    pass
else:
    # summarize section by section instead
    pass

Best Practice 3: Pair Markdownify with OGSnap for Link Previews

Markdownify gives you the body; OGSnap gives you the metadata. When you're building an agent that reads links shared in chat, email, or Slack, fetch both in parallel:

# Extract the article body as markdown
curl "https://fetchapi.tech/v1/markdownify?url=https://example.com/blog/post"

# Extract Open Graph metadata for a rich preview
curl "https://fetchapi.tech/v1/ogsnap?url=https://example.com/blog/post"

The OG snapshot gives you og:title, og:description, and og:image for the preview card, while the markdown gives the model the actual content to answer questions about. Two parallel requests, one coherent tool.

Best Practice 4: Combine with the Transcript API for Video Pages

YouTube watch pages are terrible LLM sources — the useful content is in the captions, not the HTML. When your crawler hits a YouTube URL, don't markdownify it; hit the YouTube Transcript API instead:

curl "https://fetchapi.tech/v1/transcript?url=https://www.youtube.com/watch?v=dQw4w9WgXcQ&format=text"

A smart router checks the URL host first: youtube.com / youtu.be → transcript endpoint; everything else → Markdownify. You get the actual content of the video instead of a page of JavaScript shells and "related videos" links.

Best Practice 5: Cache Aggressively, Keyed by URL

Web content changes, but not that often. For most pages, a 24-hour cache with a URL key will serve 95% of requests from storage and save you from hammering the API — and from re-embedding identical content. If you want to be clever, cache the content hash and only re-fetch when the hash changes:

curl -s "https://fetchapi.tech/v1/markdownify?url=https://example.com" | sha256sum

For pages you monitor frequently, pair this with DiffCheck when it ships — fetch once, diff forever, and only re-extract when something actually changed.

Best Practice 6: Keep Code Blocks and Tables Intact

Markdownify preserves fenced code blocks and markdown tables, which is exactly what you want for documentation pages. When you chunk, respect those structures: never split a code block across two chunks, and keep tables whole — a table split mid-row becomes gibberish to an embedding model. If your chunker sees a `` fence or a|` table header, treat it as a hard boundary.

Best Practice 7: Handle Errors Like a Production System

The API is an HTTP service, so treat it like one:

Check status codes. A 4xx means the URL is bad, not the API. Log it and move on.
Retry with backoff on 5xx and network timeouts — exponential backoff with jitter, capped at 5 retries.
Respect rate limits. If you're bulk-crawling, add a tiny delay between requests and use an API key from the dashboard so usage is tracked.
Validate the output. If content is empty or word_count is 0, the page was probably a JavaScript-rendered SPA. Mark it as needs_browser in your pipeline instead of embedding a blank string.

Putting It Together: A "Read This Page" Agent Tool

Here's a complete, copy-pasteable tool definition for an agent that reads any web page — YouTube video or article — and answers questions about it:

import requests

def read_page(url: str) -> str:
    """Fetch any URL as clean, LLM-ready text."""
    if "youtube.com" in url or "youtu.be" in url:
        r = requests.get("https://fetchapi.tech/v1/transcript",
                         params={"url": url, "format": "text"})
        return r.json().get("text", "")

    r = requests.get("https://fetchapi.tech/v1/markdownify",
                     params={"url": url})
    data = r.json()
    return f"# {data['title']}\n\n{data['content']}"

That's the entire web-reading layer of an agent — 15 lines. The transcript endpoint handles video, Markdownify handles everything else, and the markdown structure gives your model headings, links, and lists it can actually reason over.

Why This Matters

Token cost and answer quality are the two numbers that decide whether an LLM application survives contact with production. Markdownify attacks both: you send fewer tokens per page, and the tokens you send are the ones that matter. Convert early, chunk on structure, cache by URL, budget with word_count, and route video pages to the transcript API — do those five things and your extraction pipeline will be cheaper, faster, and more accurate than 90% of the hand-rolled scrapers out there.

The best practice summary in one line: never feed raw HTML to a model when a markdown API call gets you the same content at a tenth of the tokens. Try it with a real page:

curl "https://fetchapi.tech/v1/markdownify?url=https://en.wikipedia.org/wiki/Retrieval-augmented_generation"