抓取241个英国地方议会的规划门户网站—— 迄今为止已获取260万份决策信息
20 分•作者: mebkorea•3 个月前
我一直在抓取 241 个英国地方议会的规划门户网站——目前已获得 260 万份决策数据。
英国的规划数据在技术上是公开的。但实际上，这些数据被锁在 400 多个不同的议会门户网站后面，其中一些门户网站仍在运行定制的 ASP.NET 系统，看起来像是 2004 年的产物，还有一些则位于 AWS WAF 后面，所有这些门户网站的模式都略有不同。我花了四个月的时间来抓取这些数据。目前，我已经覆盖了英格兰、苏格兰和威尔士的 241 个议会，并获得了 260 万份决策数据。
抓取数据的问题
大多数英国议会运行着少数几种门户网站系统，其中 Idox 最为常见。理论上这让事情变得简单。但实际上，每个议会都对其进行了不同的配置，有些议会通过 TLS 指纹识别来阻止非浏览器请求，有些议会设置了速率限制，会在 10 分钟内封禁你，还有少数议会运行着前面提到的定制 ASP.NET 系统。
我最终编写了几个抓取器：一个基于标准请求的抓取器，一个基于 Playwright 的抓取器（用于阻止任何看起来不像真实浏览器的议会），以及一个 curl_cffi 抓取器（用于 TLS 指纹识别）。有些议会我仍然无法获取数据。利物浦的门户网站位于 AWS WAF 后面，并带有 JavaScript 挑战。我有一个基于 Playwright 的抓取器可以解决这个挑战并重复使用 cookie，但 WAF 在大约 10 个请求后会限制 IP 的速率，然后封禁我一天。因此，我从旧的抓取中获得了 6 万份利物浦的决策数据，但没有简单的方法可以添加更多数据。
我的发现
大多数人关注的是批准率。全国范围内的批准率约为 88%，但它在议会内部的各个选区之间差异很大，不仅仅是在议会之间。
更有趣的发现来自决策时间数据。在 119 个英格兰和威尔士议会中，2025 年有 36.5% 的房屋扩建申请错过了 8 周的法定目标，高于 2019 年的 27.9%。吉尔福德的情况最糟糕：66% 的决策超过目标，平均需要 13.3 周。
目前的情况
提供邮编查询（免费）和付费 PDF 报告（19 英镑/79 英镑）。目前还没有付费客户，这没关系。我一直专注于数据质量和覆盖范围。
网站是 planninglens.co.uk，如果你想查看。欢迎就抓取方面的问题进行提问——这才是真正有趣的问题所在。
查看原文
I've been scraping 241 UK council planning portals – 2.6M decisions so far<p>UK planning data is technically public. In practice it's locked behind 400+ different council portals, some still running bespoke ASP.NET that looks like it dates from 2004, some behind AWS WAF, all with subtly different schemas. I've spent four months scraping them. I'm now at 241 councils and 2.6 million decisions across England, Scotland and Wales.<p>The scraping problem<p>Most UK councils run one of a handful of portal systems, Idox being the most common. In theory this makes things easy. In practice every council has configured theirs differently, some block non-browser requests via TLS fingerprinting, some have rate limits that will get you banned inside 10 minutes, and a handful are running the aforementioned bespoke ASP.NET.<p>I ended up writing several scrapers: a standard requests-based one, a Playwright-based one for councils that block anything that doesn't look like a real browser, and a curl_cffi one for TLS fingerprinting. Some councils I still can't get. Liverpool's portal sits behind AWS WAF with a JavaScript challenge. I have a working Playwright-based scraper that solves the challenge once and reuses cookies, but the WAF rate-limits the IP after about 10 requests and then blocks me for a day. So I have 60k Liverpool decisions from an old scrape and no easy way to add more.<p>What I found<p>The approval rate stuff is what most people come for. Nationally it's around 88%, but it varies wildly by ward within a council, not just between councils.<p>The more interesting finding came from the time-to-decision data. Across 119 English and Welsh councils, 36.5% of home extension applications missed the statutory 8-week target in 2025, up from 27.9% in 2019. Guildford is the worst at scale: 66% of decisions over target, averaging 13.3 weeks.<p>What it is now<p>A postcode checker (free) and paid PDF reports (£19/£79). Zero paying customers so far, which is fine. I've been heads down on data quality and coverage.<p>Site is planninglens.co.uk if you want to poke around. AMA on the scraping side – that's where the interesting problems are.