Overview
Mountpoint for Amazon S3 is an open source file client that you can use to mount an S3 bucket on your compute instance and access it as a local file system. It automatically translates local file system API calls to REST API calls on S3 objects. Mountpoint for Amazon S3 is optimized for high-throughput performance. It builds on the AWS Common Runtime (CRT) library, which is purpose-built for high performance and low-resource usage to make efficient use of your compute resources.
Benefits
When to use Mountpoint for Amazon S3
Mountpoint for Amazon S3 is ideal for workloads that read large datasets (terabytes to petabytes in size) and require the elasticity and high throughput of Amazon Simple Storage Service (Amazon S3). Common use cases include large-scale machine learning (ML) training, autonomous vehicle simulation, genomics analysis, and image rendering. While these workloads read large datasets over several compute instances, they write sequentially to a file from a single node. This means they do not need shared file system features such as locking.
Mountpoint for Amazon S3 is designed around the tenet that it exposes the native performance of Amazon S3 and doesn’t support file system operations that can’t be efficiently implemented against S3 object APIs. This means Mountpoint for Amazon S3 doesn't try to emulate shared file system features that have no close analog in S3 object APIs. For applications that require shared file system features such as file locking and POSIX permissions, you can use Amazon FSx for Lustre with a data repository association to your S3 bucket.
How it works
Customers
-
Continental
Continental develops pioneering technologies and services for sustainable, connected mobility of people and their goods. Founded in 1871, this technology company offers safe, efficient, intelligent, and affordable solutions for vehicles, machines, traffic, and transportation.
Continental’s Advanced Driver Assistance Systems (ADAS) technologies enable a high level of driving comfort and crash avoidance with features like Adaptive Cruise Control and Emergency Brake Assist. The validation of ADAS development requires large sets of input data stored in Amazon S3 to be re-simulated in compute workloads running on Amazon EC2 instances. Mountpoint for Amazon S3 allows us to reduce the idle time spent waiting to copy data to EC2 instances for processing. With its high-throughput performance, Mountpoint for Amazon S3 widens our choice of EC2 instance types, significantly reducing our compute costs.
The An Binh Nguyen, Product Owner for Cloud-based Simulation Platform, Continental -
Untold Studios
Untold Studios is a BAFTA-, EMMY-, and GRAMMY-nominated studio, shaping culture through music, TV, and advertising. Untold Studios develops original programming, produces music and advertising content, and crafts world-class VFX, all made possible by next-generation technology.
Being able to quickly iterate on infrastructure and take advantage of new technologies is why Untold Studios runs on AWS. Mountpoint for Amazon S3 reduces the cost of storage and simplifies architecture for our render workflows that require high-throughput access to read and write TBs of transient files. It gives our applications direct access to Amazon S3's elastic throughput and storage over a file interface, freeing up performance and capacity on our primary file system for artists and projects.
Sam Reid, Head of Technology, Untold Studios -
Amagi
Amagi is a global media technology SaaS leader, providing end-to-end cloud-managed live video, on-demand video, and monetization solutions for broadcast and streaming TV.
At Amagi, we rely heavily on Amazon S3 to store large amounts of video, audio, graphics, and metadata. In the past, we tried various FUSE-based file systems with S3-based storage infrastructure, but they failed to meet our SLA requirements for broadcast-grade playout systems. We are thrilled with the sustained high read throughput we have achieved in long-term experiments with Mountpoint for Amazon S3. We prioritize cost-effective solutions for our customers while maintaining high SLA standards, and Mountpoint for Amazon S3 will enable us to do so across multiple products.
Arpit Malani, Engineering Manager, Platform & Streaming TV, Amagi -
Rivian
Rivian Automotive, Inc., is an American electric vehicle manufacturer and automotive technology and outdoor recreation company founded in 2009.
We store petabytes of data in Amazon S3 to run simulations and data pipelines that improve the accuracy of our autonomous driving systems, and provide quick feedback to autonomy model development. Our internal tools read simulation datasets and write results over a file system interface. This is where Mountpoint for Amazon S3 has been a game changer for us— delivering high throughput access to objects over a file system interface without any code changes. We were able to improve our file download times 4x by using Mountpoint, and can easily scale to terabits per second of aggregate throughput, allowing our teams to finish jobs faster.
Narendra Nath Challa, Staff Software Engineer, Data Ops, Autonomy, Rivian