AWS Direct Connect and AWS Local Zones interoperability patterns

In December 2019, we announced our first Local Zone in Los Angeles. As a refresher, AWS Local Zones are a type of infrastructure deployment that place compute, storage, database, and other AWS services close to large population, industry, and IT locations. Local Zones extend the capabilities of an AWS Region – what we called “parent” AWS Region. As of November 2022, AWS has expanded Local Zones to 21 metropolitan areas, and announced plans to launch in 30 metros across 25 countries globally. This means that you can deliver single-digit millisecond latency applications to your end-users in more locations.

Accessing applications on a Local Zone is similar to how you would do it in an AWS Region. You can provide access through the public Internet, so employees and end users can use public IPs to access the applications. However, if you want private access, then you can use either a VPN or AWS Direct Connect. This blog will focus on the second option.

In June 2022, AWS announced AWS Direct Connect support for all AWS Local Zones. Prior to this announcement, connectivity from a Direct Connect location to a workload running in a Local Zone (except for the Los Angeles one) would flow through the parent AWS Region of that specific Local Zone. This translated into an increased latency when you wanted a private and dedicated connection between your on-premises locations and Local Zones. With this announcement, network traffic takes the shortest path between Direct Connect locations and Local Zones, both for current and future ones. This new routing behavior reduces the distance that network traffic must travel, thereby decreasing latency and helping make your applications more responsive.

This post will provide a deep dive into the relation between Local Zones and Direct Connect. First, it will cover common use cases that we’ve seen in our customers, and we’ll discuss how certain industries can benefit from the low latency, private, and dedicated connectivity with Local Zones using Direct Connect. After that, we will discuss the different reference architectures that you can have when connecting your on-premises locations and Local Zones using Direct Connect. Finally, we’ll review some architecture considerations when building the architectures described.

Use cases

You can leverage Local Zones when you want to bring AWS resources close to your end-users or on-premises locations, and Direct Connect to achieve low latency, private, and dedicated connectivity between resources in AWS and on-premises workloads. The most common customer use cases are as follows:

• Media and Entertainment (M&E) content creation: For M&E customers, moving the on-premises workstations to AWS helped them accelerate content creation by removing capacity constraints while improving the security and operational efficiency. But now, media workstations require five milliseconds of latency (or less) from the offices to the virtual instances located in AWS. Having a consistent experience is key when working on remote workloads. Customers can place those virtual instances in Local Zones and connect them to their offices using Direct Connect, thereby keeping the single-digit millisecond connectivity they required. This means that they can run their latency-sensitive workloads such as live production or video editing. Netflix is one example of an M&E customer utilizing Local Zones and Direct Connect for their media workstations.
• Payment processors: For financial services customers, payment processing applications are crucial in their operations. Payment processes require a high exchange of data every day, as well as a fast and consistent performance for the end-users, so high-bandwidth and low latency are primary requirements. We also shouldn’t forget about data residency, as there may be regulatory requirements imposing data to be handled and/or stored in a specific geographic location. Most financial services customers are already using Direct Connect to communicate their on-premises locations to AWS using dedicated and private connections. Local Zones can also be included in their architectures, as they can bring their cloud-native solutions closer to their on-premises locations and end-users, thereby reducing the latency while keeping the high-bandwidth and consistency of the connectivity by using Direct Connect directly to the Local Zones.
• Mobile 5G networks: Telecom operators are leveraging AWS global infrastructure to either extend the capabilities of their 5G mobile networks, or directly build cloud-native 5G networks like Dish Wireless. AWS backbone and services help them orchestrate a secure and scalable software-driven network, simplify their operations, and re-imagine customer experience by using artificial intelligence (AI) and advanced data analytics. Local Zones and Direct Connect are important parts of this network topology, getting those solutions closer to the end-users, as well as achieving the high-bandwidth and low latency requirements needed by the 5G networks.
• Enterprise migration: Some enterprises are interested in achieving hybrid deployment or migrations in specific locations. We see this pattern regularly in the Los Angeles Local Zones location, where customers are using it to migrate complex and interdependent applications from on-premises to AWS. These customers have told us that it can be daunting to migrate a portfolio of interdependent applications to the cloud all at the same time, and in many cases that they don’t currently have plans to fully vacate their facilities. Using Direct Connect, these customers want to establish a hybrid environment that provides single-digit millisecond latency communication between applications running in the Local Zone and on-premises installations. In turn, this enables customers to migrate applications incrementally, simplifying migrations drastically and also enabling ongoing hybrid deployments with Local Zones.
• Real-time gaming: It’s paramount to get low latencies all of the time in gaming applications, as the slightest increase translates to a bad experience. Approximately 20 milliseconds (ms) is considered a very good latency for real-time gaming, and between 20 to 100 ms is a nice range to have for acceptable performance. However, up to 150 ms will increase lag, thereby reducing the end-users experience. That’s why our customers have been placing their latency-sensitive game servers in Local Zones, to run real-time and interactive multiplayer game sessions. In addition, if those game servers need hybrid connectivity with on-premises servers closer in the area, then utilizing Direct Connect connections helps maintain the high throughput and single-digit millisecond latency required for the applications.

You can find more customers’ use cases on the Local Zones website. Let’s now review different architectures showing how you can use Direct Connect to create a dedicated, private, and low-latency connection between your workloads in AWS and on-premises locations.

Local Zones and Direct Connect reference architectures

When placing your workloads in Local Zones, you proceed as if you are doing it in AWS Regions, as you do it inside a subnet. When you create subnets associated to a specific Local Zone, you’re extending the Amazon Virtual Private Cloud (Amazon VPC) to that Local Zone. Therefore, you can either have VPCs with subnets in both an AWS Region and Local Zones (Figure 1), or VPCs with subnets only in Local Zones (Figure 2).

Figure 1. VPC with subnets on both AWS Parent Region and AWS Local Zone

Figure 2. VPC with subnets only in either an AWS Region or AWS Local Zone

When connecting your on-premises environment to your VPCs via Direct Connect, you utilize Virtual Interfaces (VIFs). There are three types of VIFs – private, transit, or public – which are used to connect to different types of endpoints inside of AWS. Regardless of where your subnets are located, the VIFs are configured in the same way. However, depending on the setup you choose, you may utilize the direct communication to the Local Zone from the Direct Connect location, or you must pass through the parent AWS Region to reach the subnets in the Local Zone.

In this section, we’ll detail different reference architectures when connecting Local Zones and on-premises locations via Direct Connect, and how the traffic is routed through the AWS backbone. That way, you’ll know exactly how to architect your applications to achieve the lowest latency possible for your hybrid setup.

Private VIF attached to a Virtual Private Gateway (VGW)

In the first option, you are connecting your on-premises locations directly to Amazon VPC via Direct Connect using a private VIF attached to a Virtual Private Gateway (VGW). This option applies either when you have VPCs with subnets in both an AWS Region and Local Zone, or VPCs with subnets only in a Local Zone.

Figure 3. Private VIF to connect on-premises locations and Amazon VPC via AWS Direct Connect

In this scenario, the remote/on-premises CIDR ranges are advertised via the private VIF to AWS, as Direct Connect only supports dynamic routing using Border Gateway Protocol (BGP). To have those entries propagated in the subnet route table, you must enable route propagation in the VGW. This option is recommended over creating those routes manually in your VPC route tables. An example of what the VPC route table would look like is as follows:

When using a Private VIF attached directly to a VGW, traffic from the Direct Connect location destined to resources in the Local Zone will take the shortest path, and they won’t transverse via the parent AWS Region.

Note that in this use case, as you aren’t using a Direct Connect Gateway (DXGW), the AWS Region to connect from the Direct Connect location(s) must be the associated AWS Region. In the list of Direct Connect locations, you can find the associated AWS Region of each specific location.

Private VIF attached to a DXGW

A slight modification to the first approach above involves the use of a DXGW. In this case, the private VIF is attached to the Direct Connect Gateway, which also has associated the VPC’s VGW. This option applies either when you have VPCs with subnets in both an AWS Region and Local Zone, or VPCs with subnets only in a Local Zone.

Figure 4. Private VIF to connect on-premises locations and Amazon VPC via AWS Direct Connect gateway and AWS Direct Connect

In a similar pattern to the first approach, the remote/on-premises CIDR ranges are advertised via the private VIF to AWS – in this case to the DXGW. Then the DXGW propagates the routes learned from the private VIF to any VPC to which it is associated. To have those entries propagated in the subnet route table, you must enable route propagation in the VGW (recommended option). An example of what the VPC route table would look like is as follows:

When using a Private VIF attached to a DXGW, traffic from the Direct Connect location destined to resources in the Local Zone will take the shortest path, and they won’t transverse via the parent AWS Region.

Transit VIF attached to a DXGW

The third option when connecting to a Local Zone through Direct Connect involves the use of a transit VIF and AWS Transit Gateway. In this case, the transit VIF is attached to the DXGW, the Transit Gateway is associated to the DXGW, and the VPCs are attached to the Transit Gateway.

This option applies only when you have VPCs with subnets in both an AWS Region and a Local Zone. This is because currently, when creating a VPC attachment in the Transit Gateway, you can’t place Transit Gateway ENIs in subnets in a Local Zone.

Figure 5. Transit VIF to connect on-premises locations and AWS Transit Gateway via AWS Direct Connect

In this case, the remote/on-premises CIDR ranges are advertised via the transit VIF to the DXGW. You can propagate these routes into any Transit Gateway route table by configuring the propagation of the DXGW attachment to those route tables – this option is recommended over the creation of static routes.

In addition, in the VPC route tables you must manually create a route to the on-premises CIDR ranges via the Transit Gateway attachment (TGW ENIs). That way, resources in that VPC can forward traffic destined to on-premises workloads via the Transit Gateway. An example of what the VPC route table would look like is as follows:

When using a Transit VIF attached to a DXGW, traffic from the Direct Connect location destined to resources in the Local Zone will be routed to the Transit Gateway in the parent AWS Region. The traffic will be forwarded to the VPC via the Transit Gateway attachment (TGW ENIs) placed in a subnet in an Availability Zone in the parent AWS Region. From there, the traffic will be routed to the subnet in the Local Zone using the underlying fabric of the VPC. Therefore, you’ll experience an increase in latency between on-premises resources and Local Zones when using this option.

Private VIF and Transit VIF attached to the VPC at the same time

The last option we’ll be covering is the connection to a VPC with subnets both in a parent AWS Region and a Local Zone using two paths: a private VIF to connect directly to subnets in the Local Zone, and a transit VIF to connect via a Transit Gateway to subnets located in the parent AWS Region.

Figure 6. Transit VIF connecting on-premises locations to AWS Transit Gateway, and Private VIF connecting to subnets in Local Zone

In this case, the remote/on-premises CIDR ranges are advertised both via the private VIF and the transit VIF to AWS – you can decide to advertise the same ranges or different ones if you want to achieve traffic segmentation. Regarding the VPC routing in the Local Zone’s subnets, the on-premises ranges will be dynamically advertised by the VGW – if route propagation is enabled. To access other resources within AWS using the Transit Gateway, you can create a static route pointing to the Transit Gateway attachment. An example of what the VPC route table would look like is as follows:

For the VPC routing in the parent AWS Region’s subnets, both the on-premises ranges and AWS’ ranges are accessed via the Transit Gateway. Therefore, you must add static routes pointing to the Transit Gateway attachment. An example of what the VPC route table would look like is as follows:

When using a transit VIF and a private VIF to access the VPC:

• Traffic from the Direct Connect location destined to resources in the Local Zone should use the private VIF path – to take the shortest path to the Local Zone and not traverse through the parent AWS Region.
• Traffic from the Direct Connect location destined to resources in the parent AWS Region should use the transit VIF path – to reach the Transit Gateway.
• In addition, traffic from resources in the Local Zone destined to resources in other VPCs in the parent AWS Region can be routed via the Transit Gateway.

Architecture considerations

• When designing your Direct Connect setup, the same considerations apply regardless of the place where you’re connecting inside AWS – either an AWS Region or Local Zone:
  • For high-availability and resiliency, create at least two Direct Connect connections. It’s recommended that those connections be configured in different Direct Connect locations. Check the Direct Connect Resiliency Toolkit for more information.
  • Direct Connect connections are dedicated lines, but they aren’t encrypted by default. You can utilize MACsec in your Direct Connect connections (for 10 or 100 Gbps options) to encrypt your traffic between your on-premises device and the Direct Connect router – check which Direct Connect locations have MACsec available. Furthermore, note that when using AWS Site-to-Site VPN on top of Direct Connect to encrypt the traffic, you will pass through the parent AWS Region to reach the Local Zone. Therefore, if MACsec isn’t available, then consider using layer-7 encryption protocols (TLS for example) to encrypt your traffic when connecting directly the Local Zone and the Direct Connect location(s).
  • When deciding the connection type of your Direct Connect connection, consider the number of VIFs that you’ll need. For example, in the last reference architecture that we covered, you will need a private VIF to reach directly subnets in the Local Zone and a transit VIF to connect to a Transit Gateway in the parent AWS Region. To have two VIFs in your Direct Connect connection, you must have a Dedicated connection – as with Hosted connections you can only have one VIF per connection.
  • The Direct Connect pricing doesn’t change when connecting to subnets placed in Local Zones.
• When spinning up your resources in Local Zones, note the following:
  • Check which AWS services are available, as the supported list varies depending on the Local Zone needed.
  • The same is true with instance types, as currently Local Zones support T3, C5d, R5d, and G4dn. Some Local Zones, such as the Los Angeles Local Zones, offer more instance types, including C5, M5, R5, and I3en. You can use the Instance Types section of the Amazon Elastic Compute Cloud (Amazon EC2) Console or DescribeInstanceTypeOfferings API to discover and compare available instance types in the different Local Zones.
  • Amazon EC2 Instances and other AWS resources in Local Zones will have different prices than in the parent region. Check each specific AWS service’s pricing information to check the differences.
  • Data Transfer charges in Local Zones is the same as in the Availability Zones in the US today.

Conclusion

You can now connect your on-premises locations to Local Zones using the shortest path possible in the AWS backbone via Direct Connect connections. With this routing configuration, you can achieve single-digit millisecond latencies when using Direct Connect connections close to the Local Zone, while you utilize dedicated and private lines for your hybrid architecture. This applies for current and future Local Zones.

In this post, we’ve discussed common use cases that we’ve seen from our customers when using Local Zones and Direct Connect. Additionally, we’ve covered different ways to connect your workloads in Local Zones and on-premises locations using Direct Connect, and how the traffic flow works when trying to use the shortest path possible in the AWS backbone. To know more about Local Zones and Direct Connect, check the corresponding sections in the AWS documentation.

About the authors