Azure Pipeline Self-hosted Agents and Azure Storage for Terraform State

Azure Pipeline Self Hosted Agent
SITEMAP

Introduction

The previous article ( https://cloud-cod.com/index.php/2025/01/03/azure-devops-pipeline-for-deploying-aviatrix-controller-and-copilot/ ) showed how to create the Azure DevOps Pipeline for your Terraform code, which stores the Terraform tfsate file(s) in the Azure Storage Blob Container. The Pipeline we deployed used Microsoft-hosted Agents. The solution works quite nicely, but we saw that a different Agent with a different Public IP executed each Pipeline Stage.

Of course, we wanted to harden the access to the Azure Storage as much as possible. Therefore, we used the Azure Storage Firewall Rules. Using the Microsoft-hosted Agent forced us to update the Firewalls Rules with the Agent’s Public IP each time the Stage was triggered and remove the Agent’s Public IP at the end of each Stage. This led to additional steps in our Tasks. Because of Azure API lag/delay, we also had to add extra “sleep” time. The result was that the execution of the Pipeline took more time than it should have. Additionally, sometimes the Azure API is sluggish, which could cause the Pipeline to fail. Of course, you could increase the “sleep” time to solve it, but it will add even more delay. It is not the perfect solution, isn’t it?..

This blog article shows one way of solving the issues described above. This time, we will leverage the Self-hosted Agent (a VM executing the Pipeline) created inside our VNET. What is important is that we will also establish a Private Endpoint to connect to the Azure Storage privately (traffic will not flow through the Internet). Thanks to that, the Azure Storage Firewall could be set to block ALL public access. Great advantage from the security standpoint!

Azure Storage for Terraform tfstate file(s)

We will use the same Storage Account created in the previous blog article. If you do not have it, please create it:

				
					# Create a Storage Account Container for Azure DevOpS Pipeline TF backend
resource "azurerm_resource_group" "avx-mgmt-tf-storage-rg" {
  name     = "avx-mgmt-storage-rg"
  location = "westeurope"
}

resource "azurerm_storage_account" "avx-mgmt-tf-storage-account" {
  name                     = "avxmgmttfstorageaccount"  # name can only consist of lowercase letters and numbers, and must be between 3 and 24 characters long
  resource_group_name      = azurerm_resource_group.avx-mgmt-tf-storage-rg.name
  location                 = "westeurope"
  account_tier             = "Standard"
  account_replication_type = "LRS"  # LRS - Locally Redundant Storage is ok for Test/Non-Prod Storage Account
  
  network_rules {
     default_action             = "Deny"
     ip_rules = ["<your-public-ip>"]
   }

  # allow_nested_items_to_be_public = false
  # public_network_access_enabled   = false
}

# Storage Account Container used to keep the Terraform tfstate for the code from my previous article
resource "azurerm_storage_container" "avx-mgmt-tf-storage-container" {
  name                 = "avx-mgmt-tf-storage-container"
  storage_account_id   = azurerm_storage_account.avx-mgmt-tf-storage-account.id
}

# Storage Account Container to be used by the code presented in this article
resource "azurerm_storage_container" "avxiatrix-env-tf-storage-container" {
  name                 = "avxiatrix-env-tf-storage-container"
  storage_account_id   = azurerm_storage_account.avx-mgmt-tf-storage-account.id
}
				
			

As you can see above, I am still using the Firewall Rules to allow my Public IP and to ensure that the pipeline I showed in my previous blog article still works (because it is using the Microsoft-hosted Agents). If you would like to block Public Access completely, please remove the “network_rules” section from your Terraform code and use the following two arguments instead:

  • allow_nested_items_to_be_public = false
  • public_network_access_enabled = false

You will see the Public Access is Disabled:

The container (“avxiatrix-env-tf-storage-container”) is there:

Azure DevOps Project and Repo

I will create a separate Project within my Organization in Azure DevOps.

				
					# Create a Project in your Organization
# You must have a Personal Access Token
resource "azuredevops_project" "aviatrix-infra-tf" {
  name               = "Aviatrix-Infra-TF"
  description        = "Project that contains Aviatrix Gateways"
  visibility         = "private"
  version_control    = "Git"
  work_item_template = "Basic"

  features = {
    "testplans"    = "disabled"
    "artifacts"    = "disabled"
    "boards"       = "disabled"
    "repositories" = "enabled"
    "pipelines"    = "enabled"
  }
}

# Repository that contains code for Aviatrix Controller and Copilot
resource "azuredevops_git_repository" "aviatrix-infra-tf-gitrepo" {
  project_id = azuredevops_project.aviatrix-infra-tf.id
  name       = "Aviatrix-Infra-Git-Repo"
  initialization {
    init_type = "Clean"
  }
  lifecycle {
    ignore_changes = [
      initialization,
    ]
  }
}

resource "azuredevops_build_definition" "aviatrix-infra-tf-pipeline-deploy" {
  project_id = azuredevops_project.aviatrix-infra-tf.id
  name       = "avx-env-pipeline-deploy"

  ci_trigger {
    use_yaml = true
  }

  repository {
    repo_type   = "TfsGit"
    repo_id     = azuredevops_git_repository.aviatrix-infra-tf-gitrepo.id
    branch_name = azuredevops_git_repository.aviatrix-infra-tf-gitrepo.default_branch
    yml_path    = "azure-pipelines.yml"
  }

}

resource "azuredevops_build_definition" "aviatrix-infra-tf-pipeline-destroy" {
  project_id = azuredevops_project.aviatrix-infra-tf.id
  name       = "avx-env-pipeline-destroy"

  ci_trigger {
    use_yaml = true
  }

  repository {
    repo_type   = "TfsGit"
    repo_id     = azuredevops_git_repository.aviatrix-infra-tf-gitrepo.id
    branch_name = azuredevops_git_repository.aviatrix-infra-tf-gitrepo.default_branch
    yml_path    = "azure-pipelines-destroy.yml"
  }
}

# Authorizaton required. Otherwise not able to access Azure Storage
resource "azuredevops_serviceendpoint_azurerm" "avx-ado-svcendpoint-azurerm-02" {
  project_id                             = azuredevops_project.aviatrix-infra-tf.id
  service_endpoint_name                  = "AzureRM Service Endpoint for Aviatrix Gateways deployment"
  service_endpoint_authentication_scheme = "ServicePrincipal"
  azurerm_spn_tenantid                   = "<Your-Tenant-ID>"
  azurerm_subscription_id                = "<Your-Subscription-ID>"
  azurerm_subscription_name              = "<Your-Subscription-Name>"
}

resource "azuredevops_resource_authorization" "avx-ado-resource-auth-02" {
  project_id  = azuredevops_project.aviatrix-infra-tf.id
  resource_id = azuredevops_serviceendpoint_azurerm.avx-ado-svcendpoint-azurerm-02.id
  authorized  = true
}

				
			

The Service Principal we used for the Service Endpoint has a Contributor role at the Azure Subscription level:

Network Resources for Azure Agent VM

Let’s create the following network resources (e.g., VNET, Subnet, Route Table, NSG, NIC, SG). The Agent will be deployed inside the VNET.

				
					## Create Azure Resource Group
resource "azurerm_resource_group" "az-devops-rg" {
  name     = var.az-devops-rg
  location = var.az-devops-region
}

## Create VNET and Subnet for Azure Agent
resource "azurerm_virtual_network" "az-devops-vnet" {
  name                = var.az-devops-vnet-name
  location            = azurerm_resource_group.az-devops-rg.location
  resource_group_name = azurerm_resource_group.az-devops-rg.name
  address_space       = var.az-devops-vnet-address-space
}

resource "azurerm_subnet" "az-devops-vnet-subnet" {
  name                 = var.az-devops-subnet-name
  resource_group_name  = azurerm_resource_group.az-devops-rg.name
  virtual_network_name = azurerm_virtual_network.az-devops-vnet.name
  address_prefixes     = var.az-devops-subnet-address-space
}

resource "azurerm_route_table" "az-devops-subnet-rt-public" {
  name                = "avx-devops-subnet-rt-public"
  location            = azurerm_resource_group.az-devops-rg.location
  resource_group_name = azurerm_resource_group.az-devops-rg.name

  route {
    name           = "default"
    address_prefix = "0.0.0.0/0"
    next_hop_type  = "Internet"
  }
}

resource "azurerm_subnet_route_table_association" "az-devops-subnet-rt-association" {
  subnet_id      = azurerm_subnet.az-devops-vnet-subnet.id
  route_table_id = azurerm_route_table.az-devops-subnet-rt-public.id
}

# NSG for Azure Agent VM instance
resource "azurerm_network_security_group" "az-devops-agent-nsg" {
  name                = var.az-devops-agent-nsg
  location            = azurerm_resource_group.az-devops-rg.location
  resource_group_name = azurerm_resource_group.az-devops-rg.name

  security_rule {
    name                   = "https"
    priority               = 100
    direction              = "Inbound"
    access                 = "Allow"
    protocol               = "Tcp"
    source_port_range      = "*"
    destination_port_range = "22"
    source_address_prefixes    = var.az-admin-public-ips
    destination_address_prefix = "*"
    description                = "https-access-to-controller"
  }

  # lifecycle {
  #   ignore_changes = [security_rule]
  # }
}

resource "azurerm_public_ip" "az-devops-agent-public-ip" {
  name                = var.az-devops-agent-pip-name
  location            = azurerm_resource_group.az-devops-rg.location
  resource_group_name = azurerm_resource_group.az-devops-rg.name
  allocation_method       = "Static"
  idle_timeout_in_minutes = 30
  domain_name_label       = "azpipeline"
  sku                     = "Basic"
}

# Azure Agent VM NIC
resource "azurerm_network_interface" "az-devops-agent-iface-01" {
  name                = var.az-devops-agent-nic-name
  location            = azurerm_resource_group.az-devops-rg.location
  resource_group_name = azurerm_resource_group.az-devops-rg.name

  ip_configuration {
    name                          = var.az-devops-agent-nic-ipconf-name
    subnet_id                     = azurerm_subnet.az-devops-vnet-subnet.id
    private_ip_address_allocation = "Static"
    private_ip_address            = var.az-devops-agent-prv-ip
    public_ip_address_id          = azurerm_public_ip.az-devops-agent-public-ip.id
  }
}

resource "azurerm_network_interface_security_group_association" "az-devops-agent-nsg-association" {
  network_interface_id      = azurerm_network_interface.az-devops-agent-iface-01.id
  network_security_group_id = azurerm_network_security_group.az-devops-agent-nsg.id
}
				
			

Agent Pool and Pipeline Authorizations

Let’s create the Agent Pool and authorize our Project to use it.

				
					# Agent Pool
resource "azuredevops_agent_pool" "ado-agent-pool-01" {
  name           = "ado-agent-pool-01"
  auto_provision = false
  auto_update    = false
}

# Authorization for the Pipeline
resource "azuredevops_agent_queue" "ado-agent-queue-02" {
  project_id    = azuredevops_project.aviatrix-infra-tf.id
  agent_pool_id = azuredevops_agent_pool.ado-agent-pool-01.id
}

resource "azuredevops_pipeline_authorization" "ado-pipeline-authorization-02" {
  project_id  = azuredevops_project.aviatrix-infra-tf.id
  resource_id = azuredevops_agent_queue.ado-agent-queue-02.id
  type        = "queue"
}
				
			

The authorization is there (as you can see, I have authorized two Projects):

Azure Private Endpoint for Storage

The following code creates the Azure Private Endpoint (to access the Azure Storage) and a few DNS-related resources. Why DNS? These DNS resources are required to resolve the Azure service’s private endpoint name to its private IP address. Without it, DNS resolution will return a public IP when you access the storage account using its public DNS name (<storage-account-name>.blob.core.windows.net). This can result in the connection attempting to go through the public internet, defeating a private endpoint’s purpose.

				
					# Azure Private Endpoint
resource "azurerm_private_endpoint" "az-private-endpoint-storage" {
  name                = "az-private-endpoint-storage"
  location            = azurerm_resource_group.az-devops-rg.location
  resource_group_name = azurerm_resource_group.az-devops-rg.name
  subnet_id           = azurerm_subnet.az-devops-vnet-subnet.id

  private_service_connection {
    name                           = "storage-blob-connection"
    private_connection_resource_id = azurerm_storage_account.avx-mgmt-tf-storage-account.id
    subresource_names              = ["blob"]
    is_manual_connection           = false
  }
}

# DNS Configuration
# Private DNS Zone for Blob Storage
resource "azurerm_private_dns_zone" "dns-zone" {
  name                = "privatelink.blob.core.windows.net"
  resource_group_name = azurerm_resource_group.az-devops-rg.name
}

# Link DNS Zone to VNET
resource "azurerm_private_dns_zone_virtual_network_link" "vnet-dns-link" {
  name                  = "vnet-dns-link"
  resource_group_name   = azurerm_resource_group.az-devops-rg.name
  private_dns_zone_name = azurerm_private_dns_zone.dns-zone.name
  virtual_network_id    = azurerm_virtual_network.az-devops-vnet.id
}

# A Record for the Private Endpoint
resource "azurerm_private_dns_a_record" "stroage-a-record" {
  name                = azurerm_storage_account.avx-mgmt-tf-storage-account.name
  zone_name           = azurerm_private_dns_zone.dns-zone.name
  resource_group_name = azurerm_resource_group.az-devops-rg.name
  ttl                 = 300
  records             = [azurerm_private_endpoint.az-private-endpoint-storage.private_service_connection[0].private_ip_address]
}
				
			

Azure Self-Hosted Agent

The code for the VM creation. Please note that using Key Vault to store the admin password is recommended. In this case, your Service Principal will also require the IAM Role for Key Vault.

				
					resource "azurerm_linux_virtual_machine" "az-devops-agent-vm" {
  admin_username                  = var.az-devops-agent-admin-name
  admin_password                  = "<your-admin-password>" # you could use Key Vault to store it
  name                            = var.az-devops-agent-vm-name
  disable_password_authentication = false
  location                        = azurerm_resource_group.az-devops-rg.location
  network_interface_ids           = [azurerm_network_interface.az-devops-agent-iface-01.id]
  resource_group_name             = azurerm_resource_group.az-devops-rg.name
  size                            = var.az-devops-agent-vm-size

  user_data = base64encode(templatefile("userdata.tftpl", {}))

  os_disk {
    name                 = var.az-devops-agent-os-disk-name
    caching              = "ReadWrite"
    storage_account_type = "Standard_LRS"
    disk_size_gb         = "128"
  }

  source_image_reference {
    publisher = "Canonical"
    offer     = "0001-com-ubuntu-server-jammy"
    sku       = "22_04-lts"
    version   = "latest"
  }

  tags = {
    Environment        = "DevOps-Pipeline-Agent"
  }

  lifecycle {
    ignore_changes = [
      admin_password
    ]
  }
}
				
			

You could use userdata file to install the required components (Terraform, Agent, AZ CLI) or just SSH to the VM and execute the commands by yourself.

The Azure self-hosted Agents require Git (2.9.0 or higher) to be installed (all the requirements: https://learn.microsoft.com/en-us/azure/devops/pipelines/agents/linux-agent?view=azure-devops). If you deploy the Azure VM, you will get Git already installed:

				
					#! /usr/bin/env bash
sudo apt-get update && sudo apt-get install -y gnupg software-properties-common
wget -O- https://apt.releases.hashicorp.com/gpg | \
gpg --dearmor | \
sudo tee /usr/share/keyrings/hashicorp-archive-keyring.gpg > /dev/null
gpg --no-default-keyring \
--keyring /usr/share/keyrings/hashicorp-archive-keyring.gpg \
--fingerprint
echo "deb [signed-by=/usr/share/keyrings/hashicorp-archive-keyring.gpg] \
https://apt.releases.hashicorp.com $(lsb_release -cs) main" | \
sudo tee /etc/apt/sources.list.d/hashicorp.list
sudo apt update -y
sudo apt-get install terraform
				
			

Azure Pipeline Self-hosted Agent installation ( https://learn.microsoft.com/en-us/cli/azure/install-azure-cli-linux?pivots=apt ):

				
					wget https://vstsagentpackage.azureedge.net/agent/3.248.0/vsts-agent-linux-x64-3.248.0.tar.gz
mkdir myagent && cd myagent
tar zxvf ~/vsts-agent-linux-x64-3.248.0.tar.gz
./config.sh --url https://dev.azure.com/<your-org> --auth pat --token <personal-access-token> --pool <your-agent-pool> --agent <your-Agent-name>
sudo ./svc.sh install
sudo ./svc.sh start
sudo ./svc.sh status
				
			
				
					curl -sL https://aka.ms/InstallAzureCLIDeb | sudo bash
az login
				
			

Verification:

  • DNS, Azure Storage Account is reachable via private IP 10.100.0.4
  • Terraform
  • AZ CLI

Remaining Terraform Code

The remaining Terraform code is presented below.

The providers.tf file:

				
					provider "azurerm" {
  features {}
}

provider "azuredevops" {
  org_service_url = var.ado_org_service_url
}


				
			

The variables.tf file:

				
					variable "ado_org_service_url" {
  type        = string
  description = "Organization Service URL"  
  default     = "https://dev.azure.com/<your-org>/"
}

# DevOps Part - for Self-hosted DevOps Pipeline Agent
variable "az-devops-rg" {
  type        = string
  description = "The name of the Resource Group used by DevOps Pipeline Agent"
}

variable "az-devops-region" {
  type        = string
  description = "The Region to be used"
}

variable "az-devops-vnet-name" {
  type        = string
  description = "The name of the DevOps Pipeline Agent components VNET"
}

variable "az-devops-vnet-address-space" {
  type        = list(string)
  description = "The Address Space assigned to the DevOps Pipeline VNET"
}

variable "az-devops-subnet-name" {
  type        = string
  description = "The name of the DevOps Pipeline Subnet"
}

variable "az-devops-subnet-address-space" {
  type        = list(string)
  description = "The Address Space assigned to the DevOps Pipeline Subnet" 
}

# Azure DevOps Agent
variable "az-devops-agent-nsg" {
  type        = string
  description = "The name of the DevOps Pipeline Agent VM NSG"  
}

variable "az-admin-public-ips" {
  type        = list(string)
  description = "The list of Public IPs that need access to the Agent VM" 
}

variable "az-devops-agent-pip-name" {
  type        = string
  description = "The name of the Public IP used by the DevOps Pipeline Agent"
}

variable "az-devops-agent-nic-name" {
  type        = string
  description = "The name of the Network interface used by the DevOps Pipeline Agent"     
}

variable "az-devops-agent-nic-ipconf-name" {
  type        = string
  description = "The name of the IP Config assigned to the DevOps Pipeline Agent NIC"   
}

variable "az-devops-agent-prv-ip" {
  type        = string
  description = "The Private IP address to be assigned to the DevOps Pipeline Agent VM NIC"   
}

variable "az-devops-agent-vm-name" {
  type        = string
  description = "The name of the DevOps Pipeline Agent VM instance"     
}

variable "az-devops-agent-vm-size" {
  type        = string
  description = "The family size of the DevOps Pipeline Agent VM instance"    
}

variable "az-devops-agent-os-disk-name" {
  type        = string
  description = "The name of the DevOps Pipeline Agent OS disk"  
}

variable "az-devops-agent-admin-name" {
  type        = string
  description = "The name of the Admin Account for the DevOps Pipeline Agent"     
}


				
			

The versions.tf file with Terraform backend:

				
					terraform { 
  required_providers {
    azuredevops = {
      source  = "microsoft/azuredevops"
      version = ">=0.1.0"
    }

    azurerm = {
      source  = "hashicorp/azurerm"
      version = ">= 2.39"
    }
  }

# Backend for TF STATE FILE
  cloud { 
    organization = "<your-org>" 

    workspaces { 
      name = "<workspace-name>" 
    } 
  } 
}
				
			

Example of the terraform.auto.tfvars file:

				
					# DevOps Pipeline
az-devops-rg                   = "az-devops-rg"
az-devops-region               = "West Europe"
az-devops-vnet-name            = "az-devops-vnet-01"
az-devops-vnet-address-space   = ["10.100.0.0/24"]
az-devops-subnet-name          = "az-devops-vnet-subnet-01"
az-devops-subnet-address-space = ["10.100.0.0/25"]

# DevOps Agent
az-devops-agent-nsg             = "az-devops-agent-nsg"
az-devops-agent-pip-name        = "az-devops-agent-public-ip" 
az-devops-agent-nic-name        = "az-devops-agent-nic-01"
az-devops-agent-nic-ipconf-name = "az-devops-agent-nic-ipconf-01"
az-devops-agent-prv-ip          = "10.100.0.10"
az-devops-agent-admin-name      = "avxadmin"
az-devops-agent-vm-name         = "az-devops-agent-vm"
az-devops-agent-vm-size         = "Standard_D4s_v5"
az-devops-agent-os-disk-name    = "az-devops-agent-os-disk"
				
			

Pipeline Test

Let’s check whether the Pipeline runs successfully. I will execute the terraform init and the terraform plan commands to confirm that they finish without errors.

Just to let you know, terraform init does not require additional configuraiton. Still, for terraform plan you have to successfully authenticate to the provider you want to use (AzureRM in my case). There are a few ways to pass the Client ID, Client Secret, Tenant ID, and Subscription ID to Terraform. For the sake of simplicity I will show how to use the Pipeline Variables.

I will create four Pipeline Variables and marked them as Secret:

				
					trigger: none

pool: ado-agent-pool-01

steps:
- script: |
    echo "Using self-hosted Agent"
    export ARM_CLIENT_ID=$(ARM_CLIENT_ID)
    export ARM_CLIENT_SECRET=$(ARM_CLIENT_SECRET)
    export ARM_TENANT_ID=$(ARM_TENANT_ID)
    export ARM_SUBSCRIPTION_ID=$(ARM_SUBSCRIPTION_ID)
    
    terraform init
    terraform plan
  displayName: 'Terraform Init and Plan'
  env:
    ARM_CLIENT_ID: $(ARM_CLIENT_ID)
    ARM_CLIENT_SECRET: $(ARM_CLIENT_SECRET)
    ARM_TENANT_ID: $(ARM_TENANT_ID)
    ARM_SUBSCRIPTION_ID: $(ARM_SUBSCRIPTION_ID)

				
			

The result:

Summary

I hope this article has been informative and helpful, providing valuable insights on the Azure Pipeline Self-Hosted Agents topic and the integration with the Azure Storage Blob Containers, and Terraform. Thank you for taking the time to read, and I look forward to sharing more with you in the future!

Leave a Reply

Your email address will not be published. Required fields are marked *