Scraping a Company’s Own Site
Outcome: a scrape column pulling one specific fact from a named page, tested on rows where you can check the answer against the live site.
- Surface
- Sync GTM app
- Level
- Beginner
- Uses
- Web scraper column · scrape_emails_from_website
- Credits
- ~0.5–1 per page read
- Prerequisite
- Lesson 09
Scrape or research?
They overlap, and picking the wrong one wastes a column.
| Use a scrape when | Use AI research when |
|---|---|
| You want the literal text or list on a page | You want a judgement about what the page means |
The page is predictable — /pricing, /integrations | The fact could be on any of several pages |
| You need the exact values: tiers, prices, logos | You need a classification: B2B or B2C, enterprise or SMB |
| You will parse the result yourself | You want a constrained answer straight away |
The strongest pattern is both: scrape the page, then run an AI column over the scraped text. You pay for one fetch and get a reliable, constrained answer grounded in real content rather than the model’s memory.
Reference: Web scrapers.
Pages worth reading
| Page | What it tells you |
|---|---|
/pricing | Whether they sell self-serve or enterprise, price band, whether they publish at all |
/integrations or /partners | Their stack, and whether you appear on it |
/customers or /case-studies | Who they sell to — the fastest read on their real ICP |
/about or /team | Size, seniority mix, location |
/careers or /jobs | What they are building right now |
/blog latest post | Whether the company is active, and what they currently care about |
/contact | Addresses and phone numbers, especially for local businesses |
A no-pricing-page result is itself data: it usually means enterprise sales with a quote process, which changes both your qualification and your message.
Sites that fight back
Three common failures and what to do about each.
Client-side rendering
The page loads content with JavaScript after the initial response, so a naive fetch returns an empty shell. Symptom: near-empty results on a site you can clearly read in a browser. Use the rendering scraper option rather than a plain fetch.
Blocking
Rate limits, bot detection, or a hard block. Symptom: consistent failures on one domain while others succeed. Slow the run, and accept that some sites are not scrapable — route those rows to an AI research column instead.
Redirects and dead domains
The domain moved, parked, or belongs to an acquirer now. Symptom: content that does not match the company at all. This is usually a data problem from lesson 06, not a scraping problem — fix the domain.
Expect 10–20% of rows to fail on any real list. Build the fallback into the design: a scrape_status column, and an AI research column that runs only where the scrape came back empty.
What not to scrape
A short list, and it is not negotiable.
- Anything behind a login. Terms of service, and in several jurisdictions more than that.
- Personal data at scale from social platforms. Use the dedicated enrichments, which operate under their own agreements.
- Sites with an explicit prohibition in
robots.txtor terms you have read. - Content you plan to republish. Reading a page to qualify an account is a different act from copying it.
If a source needs a workaround to reach, treat that as the answer rather than the obstacle.
Do this now
Pick one page and one fact
/pricing and “is there a published price?” is a good first choice — high signal, easy to verify.
Build the URL
Concatenate your domain column with the path in a formula column. Free, and it makes failures readable.
Run on five rows
Read the raw output. Confirm you got page content, not an error page or a shell.
Add the AI layer
An AI column over the scraped text: “Based on this page content, is a price published? Answer: Yes, No, Unknown.”
Verify by hand
Open all five sites yourself and compare. This is the only real test.
Add the fallback
A column that runs AI research directly where the scrape returned nothing.
Widen to qualified rows
Only qualified rows, never the whole table.
Check your work
- Scraped output contains real page content on the rows that succeeded
- A status column distinguishes empty results from failures
- You verified five rows against the live site by hand
- A fallback path exists for the rows that failed
- Nothing you scraped was behind a login
Where this breaks
Silent partial failure is the one to watch. A scrape that returns a cookie banner, a 404 page or a JavaScript shell is not empty — it is text, and an AI column will confidently classify it. The result is a column full of answers derived from error pages. Always check length and content of the raw scrape before the AI layer reads it, and treat anything under a couple of hundred characters as a failure.
Further automation
Scraped facts have different shelf lives. Pricing and integrations change quarterly; a latest blog post changes weekly. Set the refresh cadence per column rather than re-running the whole table, and use the scrape only where a database genuinely cannot answer — it is the most expensive way to learn a company’s headcount.
Next lesson
11 — Enriching people, the stage where the credits actually go.
Reference for this lesson: Web scrapers, Scrape pricing plans, AI agents, Research & Web Scraping.