Booking.com Moves 80 to 90 PB of Data Off Hadoop to AWS and Snowflake in a Year

Booking.com has finished moving a large part of its data lake off an aging Hadoop setup and onto a cloud platform built on AWS S3 and Snowflake. Its engineers say the job took a year and ran without stopping the business.

The company's own estimate puts the old Hadoop 2 estate at about 250 petabytes, or roughly 500 PB once replication is counted. About 80 to 90 PB, excluding replicas, went to the new platform, which Booking.com calls the Booking Data Exchange, or BDX. The authors say more moves to BDX are planned. For scale, they point to the US Library of Congress digital collections, reported at 21 PB in 2022.

Not everything made the trip. The team chose to leave a large share of old datasets behind rather than pay to copy them and carry over clutter. The post gives two figures for what stayed, about 200 PB including replicas in one section and about 110 PB of raw data in another, and does not reconcile them.

The reasons for the move are practical. Hadoop 2 was near the end of its life, and leadership decided against upgrading to Hadoop 3 or buying new on-premises hardware. Access inside the old system was seen as too broad, rising regulatory demands were eating time, and the business wanted to scale for summer peaks and feed generative AI projects with data joined across flights, accommodation and car rentals.

The migration covered thousands of tables, transformation workflows and the dashboards that depend on them, across dozens of teams. Rather than one big switch, the engineers worked in waves. Teams defined Airflow workflows in YAML instead of Python, a central tool tracked dependencies and status, and intermediate tables were copied across so each team could move at its own pace. Finance teams moved early because of strict data quality needs, while customer service moved later because of upstream dependencies. Hundreds of people attended workshops on Spark, dbt and Airflow.

Booking.com says the move brought better data governance, more flexibility and freed up resources, and that it is now looking at data lineage, semantic layers and AI agents.

This is a company engineering blog, so the account is the company's own and no costs or savings are given. For hotel and rental operators, the relevant point is indirect. The data platform feeds the ranking, pricing and recommendation tools that decide how properties appear on Booking.com, and the company is clearly rebuilding it with AI in mind.