AWS TGW VPC Attachment with 3rd party VPC using AWS RAM

Introduction

This is a continuation of the discussion about integrating 3rd party AWS VPCs with your Aviatrix environment. There have been already two articles posted by me:

an article describing a few possible options (high-level): https://cloud-cod.com/index.php/2023/12/06/connecting-3rd-party-aws-vpcs-with-aviatrix-environment/
an article presenting how to configure one of the options (detailed configuration steps) with leveraging AWS Transit Gateway orchestrated feature: https://cloud-cod.com/index.php/2023/12/27/aviatrix-orchestrated-aws-tgw-vpc-attachment-with-3rd-party-vpc/

The following blog post guides you on how to create an integration between 3rd party AWS VPC(s) and your Aviatrix environment using AWS TGW but with the caveat that you cannot use the Aviatrix AWS TGW-Orchestrator feature. One of the reasons might be that you are not allowed to onboard 3rd party Account to your Aviatrix Controller.

The solution presented here leverages AWS Resource Access Manager to share AWS Transit Gateway between AWS Accounts. This is a perfect use case for using RAM.

Initial Setup

The initial setup is the same as the one used by AWS TGW-the O demo (https://cloud-cod.com/index.php/2023/12/27/aviatrix-orchestrated-aws-tgw-vpc-attachment-with-3rd-party-vpc/).

I have kept our test environment very simple. Your real PROD environment would be of course much more sophisticated. My test environment consists of the following:

one Aviatrix Transit VPC (10.150.0.0/16)
two Spoke VPCs (10.251.0.0/24 and 10.252.0.0/24) connected to Transit VPC through Aviatrix Spoke Gateways. The Spoke 10-251 (called by me as Integration-Spoke) will be the one attached to AWS TGW.

I used the following modules to set up the Aviatrix environment:

There is also one VPC (10.80.0.0/16) that does not belong to our organization (an Account out of our control). What is more, I have deployed test VMs in 3rd party VPC and 10-252 Spoke VPC that will help us with a test scenario later.

Test Scenario Diagram

The goal is to achieve connectivity between VMthe instance in 3rd party VPC and VM in Spoke 10-252 VPC (the Spoke VPC connected to Aviatrix Transit).

Configuration of AWS TGW and its Attachments

AWS TGW Creation

Let’s create AWS Transit Gateway (AWS TGW). We will use aws_ec2_transit_gateway Terraform resource. Please also notice that AWS TGW will be created in the Account owned and controlled by us.

				
					resource "aws_ec2_transit_gateway" "aws_integration_tgw" {
  description = "aws_integration_tgw"
  tags        = {"Name" = "aws_integration_tgw"}
}

AWS TGW has been created:

Attaching Integration-Spoke to AWS TGW

The Aviatrix Integration Spoke VPC will be connected to AWS TGW using a VPC Attachment.

Please notice that you must provide Subnet IDs and there can be only one Subnet ID per AZ. My Integration Spoke has been created using the MC-Spoke module. The module creates 2 Public Subnets in AZ-A and 2 Private Subnets in AZ-B. I just provided both Public Subnet IDs in the code.

				
					resource "aws_ec2_transit_gateway_vpc_attachment" "aws_tgw_attach_avtx_spoke_integration" {
  subnet_ids         = ["subnet-01ff108c94428d7dc", "subnet-065d7a7fbcd7402fc"] # only one subnet per AZ is allowed
  transit_gateway_id = aws_ec2_transit_gateway.aws_integration_tgw.id
  vpc_id             = module.mc-spoke-aws.spoke_gateway.vpc_id
}

The Integration-Spoke VPC has been attached:

Creating AWS RAM Resource Share

Now, let’s attach 3rd party VPC to AWS TGW. But please remember that AWS TGW has been created in our Account and the 3rd party VPC is created in a different Account. To attach 3rd party VPC we must share our AWS TGW. Sharing is possible thanks to the AWS Resource Access Manager (RAM) service.

We must create a Resource Share first and allow external Accounts to use it.

				
					resource "aws_ram_resource_share" "aws_integration_tgw_ram_share" {
  name                      = "aws_integration_tgw_ram_share"
  allow_external_principals = true
}

The resource share is there:

Next step is to associate AWS TGW and 3rd party Account ID with Resource Share:

				
					resource "aws_ram_principal_association" "example" {
  principal          = "<3rd-party-Account-ID>"
  resource_share_arn = aws_ram_resource_share.aws_integration_tgw_ram_share.arn
}

resource "aws_ram_resource_association" "aws_ram_share_association" {
  resource_arn       = "arn:aws:ec2:eu-central-1:<Account-ID>:transit-gateway/tgw-0b2c705f976924bb5"
  resource_share_arn = aws_ram_resource_share.aws_integration_tgw_ram_share.arn
}

Once done, there is a Pending Invitation in 3rd party Account.

Please select the Resource Share and Accept it:

After accepting the Resource Share, you can see the Transit Gateway (in 3rd party Account Management Console):

The Resource Share is also visible in our main Account of course:

Attaching 3rd party VPC to AWS TGW

Now, having AWS TGW shared between Accounts, we can proceed with attaching 3rd party VPC to AWS TGW. The attachment must be requested by 3rd party Account.

The attachment is now in “Pending Acceptance” state (snapshot from 3rd party Account Management Console):

Go to main Account: VPC > Transit gateways > Transit gateways attachments. You can see the Attachment requested by 3rd party.

You can now accept the Attachment request:

The Attachment State is now Available:

Will all that be enough to ensure the connectivity between the 3rd party VPC EC2 Instance and 10-252-Spoke EC2 instance work? The answer is: No.

Please keep in mind that with “manual” (either using Console or Terraform) provisioning of AWS TGW the routes are not automatically created in attached VPCs. This is opposite to the solution I presented previously with the Aviatrix feature called AWS TGW Orchestrator. The AWS TGW Orchestrator automates the creation of routes on our behalf. Without AWS TGW Orchestrator we have to take care of the routing by ourselves.

Checking Routing

How does the routing look like now? AWS TGW Route Table shows that it knows both attached VPCs CIDRs. However, it is not aware of the remaining VPCs existing in Aviatrix environment (in our case: Spoke-10-152). So this is routing issue #1.

There are more routing issues. The 3rd party Subnets do not have any Routes pointing to AWS TGW. Those Route Tables know only its local VPC CIDR. This is routing issue #2.

If we look at the route table used by a Subnet where Aviatrix Spoke Gateways (10-251 Spoke) are deployed, there is no route for 10.80.0.0/16. We can see the route 10.0.0.0/8 pointing to Aviatrix Spoke Gateway’s ENI. It means that if we try to send traffic from Spoke 10-252 and traffic gets to Integration Spoke Gateways it will be lost there. This is routing issue #3.

There is yet another problem (routing issue #4): Integration-Spoke Gateways do not advertise 3rd party VPC CIDR to Aviatrix Transit (and also farther down to Spokes). If we look at the Aviatrix Transit Route Table it does not know AWS 3rd party VPC CIDR:

I have presented all those 4 routing issues in the diagram below. All of them must be fixed by us.

Fixing Routing Issues

Let’s start with routing issue #2: lack of routes in 3rd party VPC pointing to AWS TGW. I will create the route in the AWS Management Console. For sake of simplicity I will use 10.0.0.0/8:

Next step: correct routing issue #1 – lack of route in AWS TGW Route Table. This time I will add a route for 10.0.0.0/8 using Terraform:

				
					resource "aws_ec2_transit_gateway_route" "aws_tgw_route" {
  destination_cidr_block         = "10.0.0.0/8"
  transit_gateway_attachment_id  = aws_ec2_transit_gateway_vpc_attachment.aws_tgw_attach_avtx_spoke_integration.id
  transit_gateway_route_table_id = "tgw-rtb-00024edaef7a7e3a0"
}

The route has been created:

Now, let’s fix the Route Table used by Integration Spoke Gateways (issue #3). I will add a route for 3rd party VPC (10.80.0.0/16) towards AWS TGW using Terraform. What is more, there are two Route Tables that we have to modify because our 2 Integration Spoke Gateways (Primary and HA) reside in different Subnets and those Subnets are associated with different Route Tables.

				
					resource "aws_route" "integration-spoke-rt-route" {
  route_table_id            = "rtb-0223c9605a17e037d"
  destination_cidr_block    = "10.80.0.0/16"
  transit_gateway_id        = aws_ec2_transit_gateway.aws_integration_tgw.id
}

resource "aws_route" "integration-spoke-hagw-rt-route" {
  route_table_id            = "rtb-03391e19a4d31bbc6"
  destination_cidr_block    = "10.80.0.0/16"
  transit_gateway_id        = aws_ec2_transit_gateway.aws_integration_tgw.id
}

The route has been added:

Last but not least, let’s fix routing issue #4 – lack of advertisement of 3rd party VPC CIDR to Aviatrix Transit. I am going to advertise both the Spoke CIDR itself (10.251.0.0/24) and 10.80.0.0/16 (3rd party VPC CIDR) from Integration-Spoke Gateways to Aviatrix Transit to ensure that the remaining Spokes will get those routes. I will use the “included_advertised_spoke_routes” argument (that maps to “Customized Spoke Advertised VPC CIDRs”) of the mc-spoke module:

				
					module "mc-spoke-aws" {
  source  = "terraform-aviatrix-modules/mc-spoke/aviatrix"
  version = "1.6.6"
  # insert the 22 required variables here
  cloud      = "AWS"
  account    = "AWS-Jakub-2"
  cidr       = "10.251.0.0/24"
  name       = "AWS-Spoke-10-251"
  region     = "eu-central-1"
  transit_gw = module.mc_transit_aws.transit_gateway.gw_name
  attached   = true
  gw_name    = "aws-10-251-sgw"
  included_advertised_spoke_routes = "10.80.0.0/16"
}

Let’s verify. The CIDR 10.80.0.0/16 is know by Transit Gateways and also by Spoke 10-252 Gateways:

The routing has been fixed. We can try to verify with ping:

The communication is working as desired. Furthermore, we can check mtr to verify the path:

Summary

As you can see, the integration scenario is working. However, there was a lot of effort to have it implemented. I had to solve 4 routing issues (lack of routes in Route Tables and lack of proper advertisement) to make it function as required.

If you compare this demo to the previous demo where I showed the AWS TGW Orchestrator feature (https://cloud-cod.com/index.php/2023/12/27/aviatrix-orchestrated-aws-tgw-vpc-attachment-with-3rd-party-vpc/) you could easily see the value of AWS TGW Orchestrator because I did not have to modify any Route Table when I used AWS TGW-O.