
SITEMAP
Introduction
The previous article ( https://cloud-cod.com/index.php/2025/01/03/azure-devops-pipeline-for-deploying-aviatrix-controller-and-copilot/ ) showed how to create the Azure DevOps Pipeline for your Terraform code, which stores the Terraform tfsate file(s) in the Azure Storage Blob Container. The Pipeline we deployed used Microsoft-hosted Agents. The solution works quite nicely, but we saw that a different Agent with a different Public IP executed each Pipeline Stage.
Of course, we wanted to harden the access to the Azure Storage as much as possible. Therefore, we used the Azure Storage Firewall Rules. Using the Microsoft-hosted Agent forced us to update the Firewalls Rules with the Agent’s Public IP each time the Stage was triggered and remove the Agent’s Public IP at the end of each Stage. This led to additional steps in our Tasks. Because of Azure API lag/delay, we also had to add extra “sleep” time. The result was that the execution of the Pipeline took more time than it should have. Additionally, sometimes the Azure API is sluggish, which could cause the Pipeline to fail. Of course, you could increase the “sleep” time to solve it, but it will add even more delay. It is not the perfect solution, isn’t it?..
This blog article shows one way of solving the issues described above. This time, we will leverage the Self-hosted Agent (a VM executing the Pipeline) created inside our VNET. What is important is that we will also establish a Private Endpoint to connect to the Azure Storage privately (traffic will not flow through the Internet). Thanks to that, the Azure Storage Firewall could be set to block ALL public access. Great advantage from the security standpoint!
Azure Storage for Terraform tfstate file(s)
We will use the same Storage Account created in the previous blog article. If you do not have it, please create it:
# Create a Storage Account Container for Azure DevOpS Pipeline TF backend
resource "azurerm_resource_group" "avx-mgmt-tf-storage-rg" {
name = "avx-mgmt-storage-rg"
location = "westeurope"
}
resource "azurerm_storage_account" "avx-mgmt-tf-storage-account" {
name = "avxmgmttfstorageaccount" # name can only consist of lowercase letters and numbers, and must be between 3 and 24 characters long
resource_group_name = azurerm_resource_group.avx-mgmt-tf-storage-rg.name
location = "westeurope"
account_tier = "Standard"
account_replication_type = "LRS" # LRS - Locally Redundant Storage is ok for Test/Non-Prod Storage Account
network_rules {
default_action = "Deny"
ip_rules = [""]
}
# allow_nested_items_to_be_public = false
# public_network_access_enabled = false
}
# Storage Account Container used to keep the Terraform tfstate for the code from my previous article
resource "azurerm_storage_container" "avx-mgmt-tf-storage-container" {
name = "avx-mgmt-tf-storage-container"
storage_account_id = azurerm_storage_account.avx-mgmt-tf-storage-account.id
}
# Storage Account Container to be used by the code presented in this article
resource "azurerm_storage_container" "avxiatrix-env-tf-storage-container" {
name = "avxiatrix-env-tf-storage-container"
storage_account_id = azurerm_storage_account.avx-mgmt-tf-storage-account.id
}
As you can see above, I am still using the Firewall Rules to allow my Public IP and to ensure that the pipeline I showed in my previous blog article still works (because it is using the Microsoft-hosted Agents). If you would like to block Public Access completely, please remove the “network_rules” section from your Terraform code and use the following two arguments instead:
- allow_nested_items_to_be_public = false
- public_network_access_enabled = false
You will see the Public Access is Disabled:
The container (“avxiatrix-env-tf-storage-container”) is there:
Azure DevOps Project and Repo
I will create a separate Project within my Organization in Azure DevOps.
# Create a Project in your Organization
# You must have a Personal Access Token
resource "azuredevops_project" "aviatrix-infra-tf" {
name = "Aviatrix-Infra-TF"
description = "Project that contains Aviatrix Gateways"
visibility = "private"
version_control = "Git"
work_item_template = "Basic"
features = {
"testplans" = "disabled"
"artifacts" = "disabled"
"boards" = "disabled"
"repositories" = "enabled"
"pipelines" = "enabled"
}
}
# Repository that contains code for Aviatrix Controller and Copilot
resource "azuredevops_git_repository" "aviatrix-infra-tf-gitrepo" {
project_id = azuredevops_project.aviatrix-infra-tf.id
name = "Aviatrix-Infra-Git-Repo"
initialization {
init_type = "Clean"
}
lifecycle {
ignore_changes = [
initialization,
]
}
}
resource "azuredevops_build_definition" "aviatrix-infra-tf-pipeline-deploy" {
project_id = azuredevops_project.aviatrix-infra-tf.id
name = "avx-env-pipeline-deploy"
ci_trigger {
use_yaml = true
}
repository {
repo_type = "TfsGit"
repo_id = azuredevops_git_repository.aviatrix-infra-tf-gitrepo.id
branch_name = azuredevops_git_repository.aviatrix-infra-tf-gitrepo.default_branch
yml_path = "azure-pipelines.yml"
}
}
resource "azuredevops_build_definition" "aviatrix-infra-tf-pipeline-destroy" {
project_id = azuredevops_project.aviatrix-infra-tf.id
name = "avx-env-pipeline-destroy"
ci_trigger {
use_yaml = true
}
repository {
repo_type = "TfsGit"
repo_id = azuredevops_git_repository.aviatrix-infra-tf-gitrepo.id
branch_name = azuredevops_git_repository.aviatrix-infra-tf-gitrepo.default_branch
yml_path = "azure-pipelines-destroy.yml"
}
}
# Authorizaton required. Otherwise not able to access Azure Storage
resource "azuredevops_serviceendpoint_azurerm" "avx-ado-svcendpoint-azurerm-02" {
project_id = azuredevops_project.aviatrix-infra-tf.id
service_endpoint_name = "AzureRM Service Endpoint for Aviatrix Gateways deployment"
service_endpoint_authentication_scheme = "ServicePrincipal"
azurerm_spn_tenantid = ""
azurerm_subscription_id = ""
azurerm_subscription_name = ""
}
resource "azuredevops_resource_authorization" "avx-ado-resource-auth-02" {
project_id = azuredevops_project.aviatrix-infra-tf.id
resource_id = azuredevops_serviceendpoint_azurerm.avx-ado-svcendpoint-azurerm-02.id
authorized = true
}
The Service Principal we used for the Service Endpoint has a Contributor role at the Azure Subscription level:
Network Resources for Azure Agent VM
Let’s create the following network resources (e.g., VNET, Subnet, Route Table, NSG, NIC, SG). The Agent will be deployed inside the VNET.
## Create Azure Resource Group
resource "azurerm_resource_group" "az-devops-rg" {
name = var.az-devops-rg
location = var.az-devops-region
}
## Create VNET and Subnet for Azure Agent
resource "azurerm_virtual_network" "az-devops-vnet" {
name = var.az-devops-vnet-name
location = azurerm_resource_group.az-devops-rg.location
resource_group_name = azurerm_resource_group.az-devops-rg.name
address_space = var.az-devops-vnet-address-space
}
resource "azurerm_subnet" "az-devops-vnet-subnet" {
name = var.az-devops-subnet-name
resource_group_name = azurerm_resource_group.az-devops-rg.name
virtual_network_name = azurerm_virtual_network.az-devops-vnet.name
address_prefixes = var.az-devops-subnet-address-space
}
resource "azurerm_route_table" "az-devops-subnet-rt-public" {
name = "avx-devops-subnet-rt-public"
location = azurerm_resource_group.az-devops-rg.location
resource_group_name = azurerm_resource_group.az-devops-rg.name
route {
name = "default"
address_prefix = "0.0.0.0/0"
next_hop_type = "Internet"
}
}
resource "azurerm_subnet_route_table_association" "az-devops-subnet-rt-association" {
subnet_id = azurerm_subnet.az-devops-vnet-subnet.id
route_table_id = azurerm_route_table.az-devops-subnet-rt-public.id
}
# NSG for Azure Agent VM instance
resource "azurerm_network_security_group" "az-devops-agent-nsg" {
name = var.az-devops-agent-nsg
location = azurerm_resource_group.az-devops-rg.location
resource_group_name = azurerm_resource_group.az-devops-rg.name
security_rule {
name = "https"
priority = 100
direction = "Inbound"
access = "Allow"
protocol = "Tcp"
source_port_range = "*"
destination_port_range = "22"
source_address_prefixes = var.az-admin-public-ips
destination_address_prefix = "*"
description = "https-access-to-controller"
}
# lifecycle {
# ignore_changes = [security_rule]
# }
}
resource "azurerm_public_ip" "az-devops-agent-public-ip" {
name = var.az-devops-agent-pip-name
location = azurerm_resource_group.az-devops-rg.location
resource_group_name = azurerm_resource_group.az-devops-rg.name
allocation_method = "Static"
idle_timeout_in_minutes = 30
domain_name_label = "azpipeline"
sku = "Basic"
}
# Azure Agent VM NIC
resource "azurerm_network_interface" "az-devops-agent-iface-01" {
name = var.az-devops-agent-nic-name
location = azurerm_resource_group.az-devops-rg.location
resource_group_name = azurerm_resource_group.az-devops-rg.name
ip_configuration {
name = var.az-devops-agent-nic-ipconf-name
subnet_id = azurerm_subnet.az-devops-vnet-subnet.id
private_ip_address_allocation = "Static"
private_ip_address = var.az-devops-agent-prv-ip
public_ip_address_id = azurerm_public_ip.az-devops-agent-public-ip.id
}
}
resource "azurerm_network_interface_security_group_association" "az-devops-agent-nsg-association" {
network_interface_id = azurerm_network_interface.az-devops-agent-iface-01.id
network_security_group_id = azurerm_network_security_group.az-devops-agent-nsg.id
}
Agent Pool and Pipeline Authorizations
Let’s create the Agent Pool and authorize our Project to use it.
# Agent Pool
resource "azuredevops_agent_pool" "ado-agent-pool-01" {
name = "ado-agent-pool-01"
auto_provision = false
auto_update = false
}
# Authorization for the Pipeline
resource "azuredevops_agent_queue" "ado-agent-queue-02" {
project_id = azuredevops_project.aviatrix-infra-tf.id
agent_pool_id = azuredevops_agent_pool.ado-agent-pool-01.id
}
resource "azuredevops_pipeline_authorization" "ado-pipeline-authorization-02" {
project_id = azuredevops_project.aviatrix-infra-tf.id
resource_id = azuredevops_agent_queue.ado-agent-queue-02.id
type = "queue"
}
The authorization is there (as you can see, I have authorized two Projects):
Azure Private Endpoint for Storage
The following code creates the Azure Private Endpoint (to access the Azure Storage) and a few DNS-related resources. Why DNS? These DNS resources are required to resolve the Azure service’s private endpoint name to its private IP address. Without it, DNS resolution will return a public IP when you access the storage account using its public DNS name (<storage-account-name>.blob.core.windows.net). This can result in the connection attempting to go through the public internet, defeating a private endpoint’s purpose.
# Azure Private Endpoint
resource "azurerm_private_endpoint" "az-private-endpoint-storage" {
name = "az-private-endpoint-storage"
location = azurerm_resource_group.az-devops-rg.location
resource_group_name = azurerm_resource_group.az-devops-rg.name
subnet_id = azurerm_subnet.az-devops-vnet-subnet.id
private_service_connection {
name = "storage-blob-connection"
private_connection_resource_id = azurerm_storage_account.avx-mgmt-tf-storage-account.id
subresource_names = ["blob"]
is_manual_connection = false
}
}
# DNS Configuration
# Private DNS Zone for Blob Storage
resource "azurerm_private_dns_zone" "dns-zone" {
name = "privatelink.blob.core.windows.net"
resource_group_name = azurerm_resource_group.az-devops-rg.name
}
# Link DNS Zone to VNET
resource "azurerm_private_dns_zone_virtual_network_link" "vnet-dns-link" {
name = "vnet-dns-link"
resource_group_name = azurerm_resource_group.az-devops-rg.name
private_dns_zone_name = azurerm_private_dns_zone.dns-zone.name
virtual_network_id = azurerm_virtual_network.az-devops-vnet.id
}
# A Record for the Private Endpoint
resource "azurerm_private_dns_a_record" "stroage-a-record" {
name = azurerm_storage_account.avx-mgmt-tf-storage-account.name
zone_name = azurerm_private_dns_zone.dns-zone.name
resource_group_name = azurerm_resource_group.az-devops-rg.name
ttl = 300
records = [azurerm_private_endpoint.az-private-endpoint-storage.private_service_connection[0].private_ip_address]
}
Azure Self-Hosted Agent
The code for the VM creation. Please note that using Key Vault to store the admin password is recommended. In this case, your Service Principal will also require the IAM Role for Key Vault.
resource "azurerm_linux_virtual_machine" "az-devops-agent-vm" {
admin_username = var.az-devops-agent-admin-name
admin_password = "" # you could use Key Vault to store it
name = var.az-devops-agent-vm-name
disable_password_authentication = false
location = azurerm_resource_group.az-devops-rg.location
network_interface_ids = [azurerm_network_interface.az-devops-agent-iface-01.id]
resource_group_name = azurerm_resource_group.az-devops-rg.name
size = var.az-devops-agent-vm-size
user_data = base64encode(templatefile("userdata.tftpl", {}))
os_disk {
name = var.az-devops-agent-os-disk-name
caching = "ReadWrite"
storage_account_type = "Standard_LRS"
disk_size_gb = "128"
}
source_image_reference {
publisher = "Canonical"
offer = "0001-com-ubuntu-server-jammy"
sku = "22_04-lts"
version = "latest"
}
tags = {
Environment = "DevOps-Pipeline-Agent"
}
lifecycle {
ignore_changes = [
admin_password
]
}
}
You could use userdata file to install the required components (Terraform, Agent, AZ CLI) or just SSH to the VM and execute the commands by yourself.
The Azure self-hosted Agents require Git (2.9.0 or higher) to be installed (all the requirements: https://learn.microsoft.com/en-us/azure/devops/pipelines/agents/linux-agent?view=azure-devops). If you deploy the Azure VM, you will get Git already installed:
Terraform installation ( https://developer.hashicorp.com/terraform/tutorials/aws-get-started/install-cli ):
#! /usr/bin/env bash
sudo apt-get update && sudo apt-get install -y gnupg software-properties-common
wget -O- https://apt.releases.hashicorp.com/gpg | \
gpg --dearmor | \
sudo tee /usr/share/keyrings/hashicorp-archive-keyring.gpg > /dev/null
gpg --no-default-keyring \
--keyring /usr/share/keyrings/hashicorp-archive-keyring.gpg \
--fingerprint
echo "deb [signed-by=/usr/share/keyrings/hashicorp-archive-keyring.gpg] \
https://apt.releases.hashicorp.com $(lsb_release -cs) main" | \
sudo tee /etc/apt/sources.list.d/hashicorp.list
sudo apt update -y
sudo apt-get install terraform
Azure Pipeline Self-hosted Agent installation ( https://learn.microsoft.com/en-us/cli/azure/install-azure-cli-linux?pivots=apt ):
wget https://vstsagentpackage.azureedge.net/agent/3.248.0/vsts-agent-linux-x64-3.248.0.tar.gz
mkdir myagent && cd myagent
tar zxvf ~/vsts-agent-linux-x64-3.248.0.tar.gz
./config.sh --url https://dev.azure.com/ --auth pat --token --pool --agent
sudo ./svc.sh install
sudo ./svc.sh start
sudo ./svc.sh status
AZ CLI installation ( https://learn.microsoft.com/en-us/cli/azure/install-azure-cli-linux?pivots=apt ):
curl -sL https://aka.ms/InstallAzureCLIDeb | sudo bash
az login
Verification:
- DNS, Azure Storage Account is reachable via private IP 10.100.0.4
- Terraform
- AZ CLI
Remaining Terraform Code
The remaining Terraform code is presented below.
The providers.tf file:
provider "azurerm" {
features {}
}
provider "azuredevops" {
org_service_url = var.ado_org_service_url
}
The variables.tf file:
variable "ado_org_service_url" {
type = string
description = "Organization Service URL"
default = "https://dev.azure.com//"
}
# DevOps Part - for Self-hosted DevOps Pipeline Agent
variable "az-devops-rg" {
type = string
description = "The name of the Resource Group used by DevOps Pipeline Agent"
}
variable "az-devops-region" {
type = string
description = "The Region to be used"
}
variable "az-devops-vnet-name" {
type = string
description = "The name of the DevOps Pipeline Agent components VNET"
}
variable "az-devops-vnet-address-space" {
type = list(string)
description = "The Address Space assigned to the DevOps Pipeline VNET"
}
variable "az-devops-subnet-name" {
type = string
description = "The name of the DevOps Pipeline Subnet"
}
variable "az-devops-subnet-address-space" {
type = list(string)
description = "The Address Space assigned to the DevOps Pipeline Subnet"
}
# Azure DevOps Agent
variable "az-devops-agent-nsg" {
type = string
description = "The name of the DevOps Pipeline Agent VM NSG"
}
variable "az-admin-public-ips" {
type = list(string)
description = "The list of Public IPs that need access to the Agent VM"
}
variable "az-devops-agent-pip-name" {
type = string
description = "The name of the Public IP used by the DevOps Pipeline Agent"
}
variable "az-devops-agent-nic-name" {
type = string
description = "The name of the Network interface used by the DevOps Pipeline Agent"
}
variable "az-devops-agent-nic-ipconf-name" {
type = string
description = "The name of the IP Config assigned to the DevOps Pipeline Agent NIC"
}
variable "az-devops-agent-prv-ip" {
type = string
description = "The Private IP address to be assigned to the DevOps Pipeline Agent VM NIC"
}
variable "az-devops-agent-vm-name" {
type = string
description = "The name of the DevOps Pipeline Agent VM instance"
}
variable "az-devops-agent-vm-size" {
type = string
description = "The family size of the DevOps Pipeline Agent VM instance"
}
variable "az-devops-agent-os-disk-name" {
type = string
description = "The name of the DevOps Pipeline Agent OS disk"
}
variable "az-devops-agent-admin-name" {
type = string
description = "The name of the Admin Account for the DevOps Pipeline Agent"
}
The versions.tf file with Terraform backend:
terraform {
required_providers {
azuredevops = {
source = "microsoft/azuredevops"
version = ">=0.1.0"
}
azurerm = {
source = "hashicorp/azurerm"
version = ">= 2.39"
}
}
# Backend for TF STATE FILE
cloud {
organization = ""
workspaces {
name = ""
}
}
}
Example of the terraform.auto.tfvars file:
# DevOps Pipeline
az-devops-rg = "az-devops-rg"
az-devops-region = "West Europe"
az-devops-vnet-name = "az-devops-vnet-01"
az-devops-vnet-address-space = ["10.100.0.0/24"]
az-devops-subnet-name = "az-devops-vnet-subnet-01"
az-devops-subnet-address-space = ["10.100.0.0/25"]
# DevOps Agent
az-devops-agent-nsg = "az-devops-agent-nsg"
az-devops-agent-pip-name = "az-devops-agent-public-ip"
az-devops-agent-nic-name = "az-devops-agent-nic-01"
az-devops-agent-nic-ipconf-name = "az-devops-agent-nic-ipconf-01"
az-devops-agent-prv-ip = "10.100.0.10"
az-devops-agent-admin-name = "avxadmin"
az-devops-agent-vm-name = "az-devops-agent-vm"
az-devops-agent-vm-size = "Standard_D4s_v5"
az-devops-agent-os-disk-name = "az-devops-agent-os-disk"
Pipeline Test
Let’s check whether the Pipeline runs successfully. I will execute the terraform init and the terraform plan commands to confirm that they finish without errors.
Just to let you know, terraform init does not require additional configuraiton. Still, for terraform plan you have to successfully authenticate to the provider you want to use (AzureRM in my case). There are a few ways to pass the Client ID, Client Secret, Tenant ID, and Subscription ID to Terraform. For the sake of simplicity I will show how to use the Pipeline Variables.
I will create four Pipeline Variables and marked them as Secret:
trigger: none
pool: ado-agent-pool-01
steps:
- script: |
echo "Using self-hosted Agent"
export ARM_CLIENT_ID=$(ARM_CLIENT_ID)
export ARM_CLIENT_SECRET=$(ARM_CLIENT_SECRET)
export ARM_TENANT_ID=$(ARM_TENANT_ID)
export ARM_SUBSCRIPTION_ID=$(ARM_SUBSCRIPTION_ID)
terraform init
terraform plan
displayName: 'Terraform Init and Plan'
env:
ARM_CLIENT_ID: $(ARM_CLIENT_ID)
ARM_CLIENT_SECRET: $(ARM_CLIENT_SECRET)
ARM_TENANT_ID: $(ARM_TENANT_ID)
ARM_SUBSCRIPTION_ID: $(ARM_SUBSCRIPTION_ID)
The result:
Summary
I hope this article has been informative and helpful, providing valuable insights on the Azure Pipeline Self-Hosted Agents topic and the integration with the Azure Storage Blob Containers, and Terraform. Thank you for taking the time to read, and I look forward to sharing more with you in the future!