Large-Scale Data Extraction & Cataloguing

Problem

Product and pricing information needed for day-to-day work was spread across more than fifteen supplier websites and portals, each with its own layout, categorisation, and update cadence. Finding and comparing information meant visiting sites individually and copying data out by hand — slow, inconsistent, and quickly out of date.

Constraints

Every supplier site was structured differently, some rendered content dynamically (requiring a real browser to see it at all), and none offered a usable data export or API. The extraction approach had to be resilient to sites changing their markup without warning, and had to avoid hammering supplier sites aggressively enough to cause problems.

Approach

I built a Python pipeline that uses `requests` and BeautifulSoup for straightforward static pages, and Selenium for sites that render content dynamically, with per-source parsing logic isolated so a layout change on one supplier's site doesn't break extraction from the rest. Every source's extracted data is normalised into one common schema — same fields, same units, same category taxonomy — and loaded into a queryable catalogue that Power BI reads from directly.

Outcome

Product and pricing information across all supplier sources became queryable from one place instead of fifteen-plus separate manual lookups. Because normalisation happens once, centrally, comparisons across suppliers became straightforward instead of a manual, error-prone exercise every time someone needed one.

What I'd do differently

I'd invest earlier in automated detection of source layout changes — a lightweight check that flags when a supplier site's structure no longer matches what the parser expects, rather than discovering it when a scheduled run quietly returns incomplete data. Per-source isolation meant a broken parser didn't take down the whole pipeline, but silent partial failure on one source was still possible early on.

Architecture

Isolating extraction logic per source — rather than one generic scraper trying to handle every layout — was the decision that made this maintainable: a single supplier changing their site breaks one small module, not the whole pipeline.