The BBC Preserves 100 Years of History Using Amazon S3

BBC Archives Technology and Services is the custodian of the BBC’s 100-year-old archives, with 16 million assets collected throughout the broadcaster’s existence—from historical film to modern digital media. The team focuses on three goals: unifying all of the archives’ delivery content, making the archives more accessible through digitization, and preserving the media with modern solutions. “We want to have a forward-looking strategy, with tools like flexible storage and compute that facilitate the use of machine learning,” says Brendan Mallon, head of product and services in BBC Archives Technology and Services. “Our goal is to safeguard the content in the archives so that it’s accessible for another 100 years.”

The broadcaster’s data had been split into separate genre repositories for news, sports, radio, and programs. It wanted to standardize its supply chain and workflows to develop a sustainable and centralized plan for archiving. It had been successfully using on-premises infrastructure, but in 2017 it saw a need to reduce the complexity of its systems. Content aggregation was unnecessarily onerous with the existing, disparate datasets. So, the team embarked on a 5-year mission to consolidate its various storage application layers and become more sustainable.

Because the BBC had been running its media asset processing system on AWS for years, it was a practical next step to migrate to an AWS solution that could contribute to the longer-term preservation strategy. “We wanted a consistent approach to extract value from inconsistent datasets, create an authoritative single catalogue that matches the media, and drive value for our audiences,” says Mallon. “Using AWS, we can standardize storage for all of our content.”

Due to the scale of the archive data, a network migration was the most practical option to automate. To complete the content migration, the BBC used its existing infrastructure powered by AWS Direct Connect, which is a cloud service that helps users create a dedicated network connection to AWS for smooth and reliable data transfers at a massive scale for near real-time analysis, rapid data backup, or broadcast media processing. In November 2022, after about 12 months of planning and consulting with Cloudfirst.io (Cloudfirst), an AWS Partner, the migration kicked off.

At peak, the team migrated 120 TB of data per day, using AWS Direct Connect to transfer sizable amounts of content to AWS. Within 10 months, the team had transferred 25 PB of data out to the cloud. By doing so, it could retire one of its legacy tape-based media repositories and develop a next-generation abstraction between media asset management systems and public cloud storage. “We were able to retire half the archive’s physical infrastructure,” says Mark Glanville, senior technical architect in BBC Archives Technology and Services. “This frees up a huge amount of technical space and power in some precious real estate in Central London.”

The BBC migrated most of its data to Amazon S3 Glacier Instant Retrieval. The solution provided the ideal type of flexible storage for the sorts of data that were stored in the BBC’s archives. Because the team had used AWS for its media asset processing system, which housed about 3 PB of media, it had experience with Amazon S3 solutions. “We worked alongside the AWS team to select the right storage class for the bulk of our content,” says Glanville. “By using Amazon S3 Glacier Instant Retrieval, we can benefit from expedited retrieval from the archive while having cost flexibility.”

The team uses a combination of Amazon S3 Glacier Instant Retrieval and Amazon S3 Intelligent-Tiering, a cloud storage class that delivers automatic storage cost savings without performance impact or operational overhead. The BBC can choose between these two storage classes depending on its expected level of access without compromising on performance.

“By using Amazon S3 Glacier Instant Retrieval and Amazon S3 Intelligent-Tiering, we get archive-like pricing models for content that we previously had in relatively hot storage,” says Tom Cartwright, executive product manager at the BBC. “This is really valuable because we can make decisions early on in a project about where our data lives.”