Lesson plan
Common Crawl
The Common Crawl Foundation (Common Crawl) is a nonprofit 501(c)(3) organization that crawls the web and freely provides its archives and datasets to the public. Access to the data is free on Amazon Web Services, but users may incur storage and compute costs. Common Crawl was founded by Gil Elbaz. The data had mostly been primarily used by researchers and some startups until the 2020s, when AI companies started training large language models using the data. In November 2025, an investigation by The Atlantic revealed that Common Crawl misled publishers when it claimed it respected paywalls in its scraping and it was not honoring requests from publishers to have their content removed from its databases.
Article summary
The Common Crawl Foundation (Common Crawl) is a nonprofit 501(c)(3) organization that crawls the web and freely provides its archives and datasets to the public. Access to the data is free on Amazon Web Services, but users may incur storage and compute costs. Common Crawl was founded by Gil Elbaz. The data had mostly been primarily used by researchers and some startups until the 2020s, when AI companies started training large language models using the data. In November 2025, an investigation by The Atlantic revealed that Common Crawl misled publishers when it claimed it respected paywalls in its scraping and it was not honoring requests from publishers to have their content removed from its databases.
Key claims
- date of foundation: 2007 — source: Wikipedia, revision 1376544662, retrieved 2026-10-20 · confidence 0.95
Discussion questions
- Summarise the article in your own words. Which single claim would you verify first, and why?
- The record states that the date of foundation is "2007". What evidence would count against this claim?
- Which claim in this article is most likely to be revised as the sources update? How would you check its current status?
- If two sources disagreed about one of these facts, what would you need in order to trust one over the other?
Bibliography
- Wikipedia, “Common Crawl”, revision 1376544662, retrieved 2026-10-20 https://en.wikipedia.org/w/index.php?oldid=1376544662
Key claims and bibliography are generated from the same tracked claims and sources as the article page.