Building a scraper that scales

This is a series where I explore what a scalable web scraper implemented in Python might look like, starting from a basic script to a Python package and more

Part I: The Motivation to Build a Scraper in Python

Explore the journey of building a robust web scraper for analyzing Belgium’s property market. Learn how we transitioned from Selenium to Beautiful Soup for efficiency, used Poetry and Typer for better dependency management, amongst other tools. This blog is part one of a series aimed at creating a scalable data collection and analysis tool

Series: Building a Scraper that scales

Published: Oct 24, 2023 Last edited: Oct 28, 2024 7 min read technology