Scraping Basics and Limits

Outcome: a clear rule for which pages you will scrape, a written boundary for what you will not, and a fallback design for the rows that fail.

Surface: Sync GTM app and MCP server
Level: Beginner
Uses: Web scraper column
Credits: ~2 for a 5-row test
Prerequisite: A table with a domain column

What a scrape actually does

It fetches a URL and returns what came back. That is the whole mechanism, and both its strengths and its failures follow from it.

Strengths. Deterministic — the same page returns the same text. Cheap relative to an AI call. Exact — you get the literal words, not a paraphrase.

Weaknesses. Blind to anything not at that URL. Brittle when the page changes. Silent when it fails, because an error page is still text.

Reference: Web scrapers.

Static versus rendered

Static pages

The HTML that arrives contains the content. Most marketing sites, blogs, docs and pricing pages. A plain fetch works and costs the least.

Client-rendered pages

The HTML that arrives is a shell; JavaScript fills it in afterwards. Common on app dashboards, some modern marketing sites, and anything built as a single-page app.

Symptom: you can read the page in a browser and the scrape returns almost nothing. Fix: use the rendering option, which costs more and is slower.

Which one you have

Test it. Scrape one page and look at the character count of the raw output. Under a few hundred characters on a page you know is dense means a shell.

Why scrapes fail

Failure	Symptom	What to do
Client rendering	Near-empty output, page fine in browser	Switch to the rendering scraper
Bot blocking	Consistent failure on one domain, others fine	Slow the run; accept the domain as unscrapable
Rate limiting	Failures start partway through a run	Reduce concurrency
Dead or parked domain	Content unrelated to the company	A data problem — fix the domain
Redirect to a different company	Content about an acquirer	Also a data problem; often a real signal
Path does not exist	404 page returned as text	Check the path exists before assuming the fact is absent

The last row is the one that quietly corrupts a dataset. A 404 page is text, so the scrape “succeeds”, and any AI column reading it will confidently classify a page that does not exist. Always check output length and look for 404 markers before treating a result as content.

Plan for 10–20% failure on any real list. That is normal, not a misconfiguration.

The boundaries

Short list, not negotiable.

Nothing behind a login. Authenticating to scrape breaches the terms you agreed to, and in several jurisdictions more than that.
No personal data at scale from social platforms. Use the dedicated enrichments, which operate under their own agreements — LinkedIn enrichments exist for this reason.
Respect explicit prohibitions. robots.txt and terms of use that name scraping.
Do not republish. Reading a page to qualify an account is a different act from copying its content into your own material.
Personal data carries obligations regardless of source. Public does not mean unregulated — GDPR and equivalents apply to scraped data the same as bought data.

If a source needs a workaround, treat that as the answer rather than the obstacle.

Design for failure

Every scrape column needs three companions:

A status column — did it succeed, return empty, or error?
A length check — a formula flagging output below a sensible threshold.
A fallback — an AI research column that runs only where the scrape failed, or an explicit “unknown” that a filter can see.

Without these, a 15% failure rate becomes 15% of your rows silently carrying wrong conclusions.

Do this now

Pick five domains you know

Mix in a large company, a small one and something built recently.

Scrape the homepage of each

Read the raw output, not a summary.

Record the character count

Anything conspicuously short is a rendering problem or a block.

Retry the short ones with rendering

Compare. Note which sites needed it.

Try one path that does not exist

Scrape /pricing on a site you know has no pricing page. Look at what comes back — this is what a false positive looks like.

Write your boundary list

Which sources you will scrape, which you will not, and why.

Check your work

You can tell a static page from a client-rendered one by the output
You know what a 404 looks like coming back from a scrape
A status column and length check are on your scrape column
Your boundary list is written down

Where this breaks

Scaling a scrape before testing failure handling is the expensive version of this mistake. At five rows you read every result; at 5,000 you read none, and the 700 failures blend into the dataset as though they were answers. Build the status column and the length check on the five-row test, not after the full run.

Further automation

Track your failure rate per run. A domain set that suddenly fails at 40% instead of 15% usually means a scraper option changed or the target sites moved to client rendering — both fixable, and both invisible unless you are watching the number.

Next lesson

03 — Scraping a single page, pulling named facts off a URL you construct per row.

Reference for this lesson: Web scrapers, Scrape domain details, Data storage.