Every SEO team automates reporting eventually. The second or third time you paste a number from one tab into another, you start writing a script — and that script is where the trouble begins. Not because automation is wrong, but because the moment a machine is pulling the numbers, nobody is reading the terms of the place it pulls them from.
The takeaway up front: most of a monthly SEO report can be assembled entirely from official APIs, and the small remainder is where teams quietly cross a line. Knowing which is which — and encoding the rules in the collector rather than in a policy document nobody opens — is the difference between a pipeline you can hand to a client and one you'd rather they never ask about.
Sort your report into three tiers before you write any code
Take last month's report and label every number with where it came from. Almost everything lands in one of three tiers, and each has completely different rules.
Tier 1 — your own data, via an official API
The bulk of a good report, and all of it permitted, structured, and free or cheap:
- Google Search Console API — queries, clicks, impressions, average position, and page-level performance for your own property; ground truth for how your site performs in Google.
- GA4 Data API — sessions, conversions, and revenue attributed to organic search, so rankings connect to money.
- Bing Webmaster Tools API — the same shape of data for Bing.
- Google Business Profile API — views, calls, and direction requests for local reporting.
- Google Ads API — paid data, when the report covers both channels.
A Search Console pull is an ordinary authenticated POST:
POST https://searchconsole.googleapis.com/webmasters/v3/sites/https%3A%2F%2Fexample.com%2F/searchAnalytics/query
Authorization: Bearer YOUR_OAUTH_TOKEN
Content-Type: application/json
{
"startDate": "2026-07-01",
"endDate": "2026-07-31",
"dimensions": ["query", "page"],
"rowLimit": 25000,
"startRow": 0
}
The reason to reach for this first isn't ethics, it's engineering: an API contract is versioned and stable, so your pipeline doesn't shatter the next time someone changes a page layout. Reliability and permission point the same way here — plan for the day they don't.
Tier 2 — licensed third-party data
Rank trackers, backlink indexes, keyword-volume providers, and site-audit crawlers all expose APIs. You're allowed to use them; the constraint isn't access but redistribution. Most vendor terms permit internal use and reporting to the account holder, and restrict bulk export, resale, or embedding data into a product you sell. This is the tier teams break most often, and always by accident — the classic being an agency that pulls backlink data on one seat and pipes it into a white-labelled dashboard for forty clients. That's a licence breach, not a scraping problem, and no amount of polite rate limiting fixes it. Read the API terms once and keep a note of what you may show, and to whom, beside the credentials.
Tier 3 — the genuine gaps, where no official API exists
A short list, but it matters: the live SERP layout for a specific city or device, a competitor's newly published URLs and on-page changes, review counts and ratings on third-party directories, and marketplace listing positions. There's no front door for these — if you need them, you're collecting public web pages.
Notice how small tier 3 is. Most reports that "require scraping" require it for two or three numbers, and the honest first question is whether those numbers change a decision at all — the same discipline that should already govern your SEO analytics and reporting.
Where teams quietly cross a line
None of these start as a decision to break rules. They start as a deadline.
Scraping Google's results directly and calling it rank tracking. Automated querying of Google Search is against Google's terms. Commercial rank trackers exist precisely so you don't have to; buying one is cheaper than the engineering, and it moves the compliance question to a vendor whose business is answering it.
Ignoring robots.txt because "it isn't legally binding". The legal status varies by jurisdiction, and it's the wrong frame anyway: robots.txt is the operator telling you in machine-readable form which paths their automation budget will tolerate, and it's the first thing anyone checks when a complaint arrives.
Unbounded concurrency. Someone sets a worker pool to 200 because the job finished faster, and a scheduled report becomes a small denial-of-service against a competitor's site. Politeness here is a technical setting, not a sentiment.
Collecting behind a login or paywall. Authenticated content is not public data, and automating past an authentication boundary is a categorically different act from fetching a public page. Don't.
Sweeping up personal data. Reviewer names and contact details on directory sites are personal data under GDPR and similar regimes, however easy they were to fetch. Collect the aggregate — rating, count — not the people.
Build the constraints into the collector, not the policy doc
Rules that live in a document get followed by whoever read it. Rules that live in the HTTP client get followed by everyone, including the cron job you forgot about. Give your pipeline one fetch function and make it structurally impossible to be rude. Three mechanisms carry most of the weight:
- Fetch and cache
robots.txtper host, check every URL against it before building the request, and honourCrawl-delay. - Token-bucket per host, not globally — one slow target shouldn't starve the others, and a fast one shouldn't get hammered.
- Back off on the signals servers send you. A
429or503withRetry-Afteris an explicit instruction; obey it, and use exponential backoff otherwise.
import time, urllib.robotparser as rp
from urllib.parse import urlparse
import requests
UA = "WeSEOReportBot/1.0 (+https://example.com/bot; [email protected])"
_robots, _next_ok = {}, {}
def allowed(url):
host = urlparse(url).netloc
if host not in _robots:
p = rp.RobotFileParser()
p.set_url(f"https://{host}/robots.txt")
p.read() # a missing robots.txt reads as allow-all
_robots[host] = p
return _robots[host].can_fetch(UA, url)
def polite_get(url, min_gap=2.0):
if not allowed(url):
raise PermissionError(f"robots.txt disallows {url}")
host = urlparse(url).netloc
delay = _robots[host].crawl_delay(UA) or min_gap
wait = _next_ok.get(host, 0) - time.monotonic()
if wait > 0:
time.sleep(wait)
r = requests.get(url, headers={"User-Agent": UA}, timeout=30)
_next_ok[host] = time.monotonic() + delay
if r.status_code in (429, 503):
time.sleep(float(r.headers.get("Retry-After", 60)))
return polite_get(url, min_gap)
return r
Two details are worth more than they look. Identify yourself — a real User-Agent with a contact URL means an annoyed sysadmin emails you instead of blocking your IP range. And cache aggressively: a competitor's pricing page doesn't change hourly, so a 24-hour cache cuts request volume by an order of magnitude with no loss of fidelity. The politest request is the one you didn't need to send.
Anti-bot challenges: a completeness problem, not a permission one
Here's the situation that trips up otherwise careful pipelines. A page is public, robots.txt allows it, the terms permit collection, your rate limit is polite — and the response is a Cloudflare Turnstile or a reCAPTCHA rather than the page. The row silently drops. Do that on a tenth of your checks and the "competitor publishing cadence" chart in your report is fiction, and nobody knows.
That's a completeness failure, and it deserves a completeness fix: a solving step that lets the collector finish a request it was already entitled to make. CaptchaAI is one option worth evaluating because its economics suit scheduled reporting rather than bursty scraping — pricing is thread-based, so you buy concurrent threads with unlimited solves per thread, no per-CAPTCHA fee and no surcharge by type. Published tiers run from BASIC at $15/month for 5 threads through ADVANCE at $90/month for 50 and ENTERPRISE at $300/month for 200, so a nightly job needing four workers has a predictable bill. The interface is the legacy 2Captcha-shaped protocol, so it drops into most existing stacks: submit to /in.php, poll /res.php roughly every 5 seconds until it stops returning CAPCHA_NOT_READY.
curl -s "https://ocr.captchaai.com/in.php" \
-d "key=YOUR_API_KEY" -d "method=turnstile" \
-d "sitekey=0x4AAA..." -d "pageurl=https://example.com/pricing" \
-d "proxy=user:[email protected]:8080" -d "proxytype=HTTP" -d "json=1"
# {"status":1,"request":"2122988149"}
# poll every ~5s
curl -s "https://ocr.captchaai.com/res.php?key=YOUR_API_KEY&action=get&id=2122988149&json=1"
# {"status":0,"request":"CAPCHA_NOT_READY"} → keep polling
# {"status":1,"request":"0.KBw..."} → inject and continue
Handle ERROR_ZERO_BALANCE and ERROR_UNSOLVABLE explicitly rather than retrying blindly — the first is a billing alert, the second means that URL needs a human. Published success rates and latencies (>99% under 10 seconds for Turnstile, >99.5% for reCAPTCHA v2) are the vendor's own figures; benchmark them against your own targets rather than assuming.
The boundary matters more than the tooling: this is for pages you are permitted to collect. It is not a way around a Disallow, a login, a paywall, or terms that say no. A solver removes a technical obstacle; it never creates permission you didn't have.
Make the report itself auditable
Stamp every row with its source and collection timestamp, and put a coverage figure on the dashboard — records expected versus records stored, with an alert below 95%. A dip usually means an API quota, an exhausted proxy pool, or a spike in unsolved challenges, and you'll see it before a client acts on a hollow number. Then keep a one-page register of each reported number, its source, and the terms that govern it. That register is the difference between "we automate our reporting" and "we can show you exactly where every figure came from."
FAQ
Is it legal to scrape Google search results for rank tracking?
Automated querying of Google Search is against Google's terms of service, whatever local law says about scraping generally. Use a commercial rank tracker with its own API instead — the vendor carries the compliance question, and the cost is almost always lower than maintaining the collector yourself. For your own site's positions, Search Console's API gives you average position directly.
Does robots.txt apply to a reporting script that only fetches a few pages a day?
Yes — it addresses automated agents, not volume, and makes no exception for small jobs. Compliance is cheap at that scale: check the file once, cache it for a day, honour any Crawl-delay. If a path you need is disallowed, look for an official API, a public data export, or simply ask the site owner — permission in writing beats a clever workaround.
Can I put third-party API data into a white-labelled client dashboard?
Check the licence — this is the most commonly broken rule in agency reporting. Most SEO data vendors permit internal use and reporting to the account holder, and restrict redistribution, resale, or embedding data into a product; some sell a specific reseller tier for exactly this. It's a five-minute read that prevents an account termination.
Where does a CAPTCHA solver legitimately fit in a reporting pipeline?
Only on collection you were already entitled to do: a public page, allowed by robots.txt and the site's terms, fetched at a polite rate, that happens to serve a challenge. There it stops silent row loss and keeps your dataset honest. It has no place in anything touching authentication, account creation, or paywalled content — and if a site's terms forbid automated access, the challenge isn't the obstacle, the answer is.
How much of a monthly report can realistically be automated?
Nearly all of the assembly — traffic, conversion, ranking, and profile numbers all come from APIs. What doesn't automate is the part clients actually pay for: which change caused which movement, what it means, and what happens next month. Automate the assembly so a human has time to write the analysis.
Automate the assembly, keep the judgement
A reporting pipeline you can defend looks boring by design: official APIs for everything they cover, a licence note beside every third-party key, a single fetch function that reads robots.txt and rate-limits itself, caching that keeps volume low, and a coverage metric that tells you when the data is thin. If anti-bot challenges are punching holes in collection you're genuinely permitted to run, benchmark a solving step — CaptchaAI is a reasonable place to measure that, since thread-based pricing makes a scheduled job's cost predictable — and keep it strictly on the permitted side of the line.
Then spend the hours you saved on the part no script can do — the interpretation. And if you'd rather not build any of it, see what a fixed-price SEO package covers and how WeSEO's transparent work log shows every deliverable behind the numbers: no ranking guarantees, just the work and the evidence for it.
