Tekton Pipelines in the Azure Kubernetes Service Cluster

In this article, we will explore the installation of Azure Kubernetes Service (AKS) with the integration of Azure Container Instances as Virtual Nodes. On top of that, we will provision Tekton Pipelines and deploy a sample application to the on-demand Virtual Node. Azure Kubernetes Service (AKS) is an Azure flavored managed Kubernetes Service. Kubernetes masters are managed by Azure and you only manage the worker nodes. Azure Container Instances (ACI) is a Microsoft Azure managed service that allows you to schedule containers directly on the public cloud, without virtual machines (VMs). We will integrate it in the AKS cluster as a Virtual Node. Virtual nodes are often used for scaling solutions in AKS. Virtual Nodes is a network communication between the AKS cluster and pods that run in Azure Container Instances (ACI). The virtual nodes add-on for AKS is based on the open-source project Virtual Kubelet. Tekton Pipelines is a powerful Kubernetes-native open-source CI/CD project that allows you to test, build and deploy across cloud providers and on-premise data centers. The Ancient Greek noun tektōn (τέκτων) is a common term for an artisan/craftsman. For this tutorial, we will use Azure Cloud Shell as a working environment. It is a nice free interactive shell that has common Azure tools preinstalled and configured to use with your account. Prerequisites Azure subscription with contributor access To use Azure Container Instances we need Microsoft.ContainerInstance service to be registered. Check that Microsoft.ContainerInstance service provider is registered with your subscription, as shown in the following example: az provider list --query "[?contains(namespace,'Microsoft.ContainerInstance')]" -o table Microsoft.ContainerInstance should be in the Registered status. If not then go ahead and register it with the AZ CLI command: az provider register --namespace Microsoft.ContainerInstance Prepare dependency resources Resource group Now that prerequisites are sorted we will start by creating a resource group. Every resource in Azure should be in a particular resource group. In this example, I’m going to create a resource group tekton-lab in the westeurope region az group create --name tekton-lab --location westeurope Virtual Network and subnets In this instruction, we will deploy Azure Kubernetes Cluster with advanced networking (Azure CNI) since virtual nodes will not work with AKS default basic networking (kubenet). Let’s create a virtual network tekton-lab-Vnet and a subnet myAKSSubnet using the az network command: az network vnet create \ --resource-group tekton-lab \ --name tekton-lab-Vnet \ --address-prefixes 10.0.0.0/8 \ --subnet-name myAKSSubnet \ --subnet-prefix 10.240.0.0/16 Now we need to provision additional subnet for virtual nodes: az network vnet subnet create \ --resource-group tekton-lab \ --vnet-name tekton-lab-Vnet \ --name tekton-lab-VirtualNodeSubnet \ --address-prefixes 10.241.0.0/16 Service principal We will manually create a service principal and use it as our AKS cluster identity. This will allow it to interact with other Azure resources. az ad sp create-for-rbac Note: if you will face an error after running az ad sp command in Cloud Shell you need to login with az cli az login and retry the command expected output: { "appId": "435be37-8310-f3d6-47af-e67c3184879", "displayName": "azure-cli-2022-03-05-17-32-55", "password": "Adkp1X38BLC31vmsEx7TV~k~bN.eLKT3T", "tenant": "1e2447db-4827-18fc--82f24e-84c3-0d00f" } Copy the appId and password. We will use those values in the next steps. Grant permissions to AKS to use virtual network We need to configure correct permissions for the AKS service principle to use network resources. To do that we need to obtain the resource ID of our network: az network vnet show --resource-group tekton-lab --name tekton-lab-Vnet --query id -o tsv Expected output: /subscriptions/54a18e-db49-4300-afcc-01b049e24/resourceGroups/tekton-lab/providers/Microsoft.Network/virtualNetworks/tekton-lab-Vnet Then we will create a role assignment with az role assignment create command to allow AKS cluster to use the virtual network: az role assignment create --assignee 435be37-8310-f3d6-47af-e67c3184879 --scope /subscriptions/54a18e-db49-4300-afcc-01b049e24/resourceGroups/tekton-lab/providers/Microsoft.Network/virtualNetworks/tekton-lab-Vnet --role Contributor Create AKS cluster Get the ID of the subnet we created for the Azure Kubernetes Service cluster az network vnet subnet show --resource-group tekton-lab --vnet-name tekton-lab-Vnet --name myAKSSubnet --query id -o tsv Expected output: /subscriptions/54a1468e-db49-4300-afcc-01f8b0493e24/resourceGroups/tekton-lab/providers/Microsoft.Network/virtualNetworks/tekton-lab-Vnet/subnets/myAKSSubnet Now we will install AKS with az aks create command. It will take a few minutes to spin up the AKS cluster: az aks create \ --resource-group tekton-lab \ --name myTektonAKSCluster \ --node-count 1 \ --node-vm-size standard_b2s \ --generate-ssh-keys \ --network-plugin azure \ --service-cidr 10.0.0.0/16 \ --dns-service-ip 10.0.0.10 \ --docker-bridge-address 172.17.0.1/16 \ --vnet-subnet-id /subscriptions/54a1468e-db49-4300-afcc-01f8b0493e24/resourceGroups/tekton-lab/providers/Microsoft.Network/virtualNetworks/tekton-lab-Vnet/subnets/myAKSSubnet \ --service-principal 435be37-8310-f3d6-47af-e67c3184879 \ --client-secret Adkp1X38BLC31vmsEx7TV~k~bN.eLKT3T For client secret use of service principle, we created in previous steps. For service-principal use of service principle. For vnet-subnet-id use the ID of the subnet in the previous step. Enable addon for virtual nodes We will use the az aks enable-addons command in order to enable virtual nodes and use the tekton-lab-VirtualNodeSubnet subnet that we created beforehand: az aks enable-addons \ --resource-group tekton-lab \ --name myTektonAKSCluster \ --addons virtual-node \ --subnet-name tekton-lab-VirtualNodeSubnet Get kubectl access To get access to a newly created AKS cluster with kubectl run the following command: az aks get-credentials --resource-group tekton-lab --name myTektonAKSCluster Expected output: Merged "myTektonAKSCluster" as current context in /home/viachaslau/.kube/config Check virtual node To verify the connection to your cluster and the availability of the ACI virtual node, use the kubectl get command to return a list of the cluster nodes. kubectl get nodes You should be able to see your Azure Container Instances virtual nodes ready! Tekton Let’s install the latest version of Tekton with the following command: kubectl apply --filename https://storage.googleapis.com/tekton-releases/pipeline/latest/release.yaml You can check the installation progress with the get pods command: kubectl get pods --namespace tekton-pipelines Now that Tekton is up and running let’s add Tekton Dashboard. It is a nice web-based UI for Tekton Pipelines. kubectl apply --filename https://github.com/tektoncd/dashboard/releases/latest/download/tekton-dashboard-release.yaml Tekton Dashboard By default, Tekton Dashboard is accessible through its cluster IP type service. To quickly get web access from Cloud Shell we can set up port forwarding with Tekton Dashboard: kubectl --namespace tekton-pipelines port-forward svc/tekton-dashboard 9097:9097 Get access to the Tekton Dashboard by clicking “Web preview” in Cloud Shell and then specify port 9097 Tekton task Let’s create a sample Tekton task. TheTask is a series of Steps that you define in a particular order of execution as part of your pipeline. Task has inputs and could produce outputs. A Task executes as a Pod on your Kubernetes cluster. Let’s create a tekton-task.yml file with the following content: apiVersion: tekton.dev/v1beta1 kind: Task metadata: name: hello spec: steps: - name: hello image: ubuntu command: - echo args: - "Hello World!" And then apply this file with kubectl apply command: kubectl apply -f tekton-task.yml If you navigate to the Tasks section in Tekton Dashboard you will see the “hello” task appeared: Here is the YAML configuration of the “hello” task: To run tasks specifically on the virtual node we need to get rid of its default taint. To get your node taints use the kubectl describe command: kubectl describe node virtual-node-aci-linux To remove taint you need to apply the same taint but with “-” character: kubectl taint nodes virtual-node-aci-linux virtual-kubelet.io/provider=azure:NoSchedule- Expected output: node/virtual-node-aci-linux untainted To double-check that there is no taint left you can describe the node again: Now that taints are sorted we can create a TaskRun for that task with node selector configured in UI. This way we will make sure that the Tekton task will run on Azure Container Instances. If you face any weird issues with your pipelines you might want to double-check Tekton Pipelines and Tekton Dashboard versions in the compatibility matrix and pick not the latest version Navigate to the TaskRuns section on the left menu and you will see that TaskRun “hello” succeded: Summary: AKS Clusters with Azure Container Integration In this tutorial, we created an AKS cluster with the integration of Azure Container Instances as Virtual Nodes with corresponding dependencies. We successfully installed Tekton Pipelines and Tekton Dashboard in the Azure Kubernetes Service cluster. Finally, we were able to schedule a sample Tekton task in an on-demand Virtual node. Clean-up Here is what to do to clean up resources created in this tutorial” Delete Tekton installation kubectl delete -f kubectl apply --filename https://github.com/tektoncd/dashboard/releases/latest/download/tekton-dashbo0 kubectl delete --filename https://storage.googleapis.com/tekton-releases/pipeline/latest/release.yaml To get rid of your virtual nodes, you can turn off them with az aks disable-addons command az aks disable-addons --resource-group tekton-lab --name myTektonAKSCluster --addons virtual-node Delete resource group with AKS: az group delete --name tekton-lab --yes --no-wait In this article, we will explore the installation of Azure Kubernetes Service (AKS) with the integration of Azure Container Instances as Virtual Nodes. On top of that, we will provision Tekton Pipelines and deploy a sample application to the on-demand Virtual Node. Azure Kubernetes Service (AKS) is an Azure flavored managed Kubernetes Service. Kubernetes masters are managed by Azure and you only manage the worker nodes. Azure Kubernetes Service (AKS) Azure Container Instances (ACI) is a Microsoft Azure managed service that allows you to schedule containers directly on the public cloud, without virtual machines (VMs). We will integrate it in the AKS cluster as a Virtual Node. Virtual nodes are often used for scaling solutions in AKS. Azure Container Instances (ACI) Virtual Nodes is a network communication between the AKS cluster and pods that run in Azure Container Instances (ACI). The virtual nodes add-on for AKS is based on the open-source project Virtual Kubelet. Virtual Nodes Tekton Pipelines is a powerful Kubernetes-native open-source CI/CD project that allows you to test, build and deploy across cloud providers and on-premise data centers. The Ancient Greek noun tektōn (τέκτων) is a common term for an artisan/craftsman. Tekton Pipelines For this tutorial, we will use Azure Cloud Shell as a working environment. It is a nice free interactive shell that has common Azure tools preinstalled and configured to use with your account. Prerequisites Azure subscription with contributor access Azure subscription with contributor access To use Azure Container Instances we need Microsoft.ContainerInstance service to be registered. Check that Microsoft.ContainerInstance service provider is registered with your subscription, as shown in the following example: az provider list --query "[?contains(namespace,'Microsoft.ContainerInstance')]" -o table az provider list --query "[?contains(namespace,'Microsoft.ContainerInstance')]" -o table Microsoft.ContainerInstance should be in the Registered status. If not then go ahead and register it with the AZ CLI command: Registered az provider register --namespace Microsoft.ContainerInstance az provider register --namespace Microsoft.ContainerInstance Prepare dependency resources Resource group Now that prerequisites are sorted we will start by creating a resource group. Every resource in Azure should be in a particular resource group. In this example, I’m going to create a resource group tekton-lab in the westeurope region tekton-lab westeurope az group create --name tekton-lab --location westeurope az group create --name tekton-lab --location westeurope Virtual Network and subnets In this instruction, we will deploy Azure Kubernetes Cluster with advanced networking (Azure CNI) since virtual nodes will not work with AKS default basic networking (kubenet). basic Let’s create a virtual network tekton-lab-Vnet and a subnet myAKSSubnet using the az network command: tekton-lab-Vnet myAKSSubnet az network az network vnet create \ --resource-group tekton-lab \ --name tekton-lab-Vnet \ --address-prefixes 10.0.0.0/8 \ --subnet-name myAKSSubnet \ --subnet-prefix 10.240.0.0/16 az network vnet create \ --resource-group tekton-lab \ --name tekton-lab-Vnet \ --address-prefixes 10.0.0.0/8 \ --subnet-name myAKSSubnet \ --subnet-prefix 10.240.0.0/16 Now we need to provision additional subnet for virtual nodes: az network vnet subnet create \ --resource-group tekton-lab \ --vnet-name tekton-lab-Vnet \ --name tekton-lab-VirtualNodeSubnet \ --address-prefixes 10.241.0.0/16 az network vnet subnet create \ --resource-group tekton-lab \ --vnet-name tekton-lab-Vnet \ --name tekton-lab-VirtualNodeSubnet \ --address-prefixes 10.241.0.0/16 Service principal We will manually create a service principal and use it as our AKS cluster identity. This will allow it to interact with other Azure resources. az ad sp create-for-rbac az ad sp create-for-rbac Note: if you will face an error after running az ad sp command in Cloud Shell you need to login with az cli az login and retry the command Note: if you will face an error after running az ad sp command in Cloud Shell you need to login with az cli az login and retry the command az ad sp az login and expected output: { "appId": "435be37-8310-f3d6-47af-e67c3184879", "displayName": "azure-cli-2022-03-05-17-32-55", "password": "Adkp1X38BLC31vmsEx7TV~k~bN.eLKT3T", "tenant": "1e2447db-4827-18fc--82f24e-84c3-0d00f" } { "appId": "435be37-8310-f3d6-47af-e67c3184879", "displayName": "azure-cli-2022-03-05-17-32-55", "password": "Adkp1X38BLC31vmsEx7TV~k~bN.eLKT3T", "tenant": "1e2447db-4827-18fc--82f24e-84c3-0d00f" } Copy the appId and password. We will use those values in the next steps. appId password. Grant permissions to AKS to use virtual network We need to configure correct permissions for the AKS service principle to use network resources. To do that we need to obtain the resource ID of our network: az network vnet show --resource-group tekton-lab --name tekton-lab-Vnet --query id -o tsv az network vnet show --resource-group tekton-lab --name tekton-lab-Vnet --query id -o tsv Expected output: /subscriptions/54a18e-db49-4300-afcc-01b049e24/resourceGroups/tekton-lab/providers/Microsoft.Network/virtualNetworks/tekton-lab-Vnet /subscriptions/54a18e-db49-4300-afcc-01b049e24/resourceGroups/tekton-lab/providers/Microsoft.Network/virtualNetworks/tekton-lab-Vnet Then we will create a role assignment with az role assignment create command to allow AKS cluster to use the virtual network: az role assignment create az role assignment create --assignee 435be37-8310-f3d6-47af-e67c3184879 --scope /subscriptions/54a18e-db49-4300-afcc-01b049e24/resourceGroups/tekton-lab/providers/Microsoft.Network/virtualNetworks/tekton-lab-Vnet --role Contributor az role assignment create --assignee 435be37-8310-f3d6-47af-e67c3184879 --scope /subscriptions/54a18e-db49-4300-afcc-01b049e24/resourceGroups/tekton-lab/providers/Microsoft.Network/virtualNetworks/tekton-lab-Vnet --role Contributor Create AKS cluster Get the ID of the subnet we created for the Azure Kubernetes Service cluster az network vnet subnet show --resource-group tekton-lab --vnet-name tekton-lab-Vnet --name myAKSSubnet --query id -o tsv az network vnet subnet show --resource-group tekton-lab --vnet-name tekton-lab-Vnet --name myAKSSubnet --query id -o tsv Expected output: /subscriptions/54a1468e-db49-4300-afcc-01f8b0493e24/resourceGroups/tekton-lab/providers/Microsoft.Network/virtualNetworks/tekton-lab-Vnet/subnets/myAKSSubnet /subscriptions/54a1468e-db49-4300-afcc-01f8b0493e24/resourceGroups/tekton-lab/providers/Microsoft.Network/virtualNetworks/tekton-lab-Vnet/subnets/myAKSSubnet Now we will install AKS with az aks create command. It will take a few minutes to spin up the AKS cluster: az aks create az aks create \ --resource-group tekton-lab \ --name myTektonAKSCluster \ --node-count 1 \ --node-vm-size standard_b2s \ --generate-ssh-keys \ --network-plugin azure \ --service-cidr 10.0.0.0/16 \ --dns-service-ip 10.0.0.10 \ --docker-bridge-address 172.17.0.1/16 \ --vnet-subnet-id /subscriptions/54a1468e-db49-4300-afcc-01f8b0493e24/resourceGroups/tekton-lab/providers/Microsoft.Network/virtualNetworks/tekton-lab-Vnet/subnets/myAKSSubnet \ --service-principal 435be37-8310-f3d6-47af-e67c3184879 \ --client-secret Adkp1X38BLC31vmsEx7TV~k~bN.eLKT3T az aks create \ --resource-group tekton-lab \ --name myTektonAKSCluster \ --node-count 1 \ --node-vm-size standard_b2s \ --generate-ssh-keys \ --network-plugin azure \ --service-cidr 10.0.0.0/16 \ --dns-service-ip 10.0.0.10 \ --docker-bridge-address 172.17.0.1/16 \ --vnet-subnet-id /subscriptions/54a1468e-db49-4300-afcc-01f8b0493e24/resourceGroups/tekton-lab/providers/Microsoft.Network/virtualNetworks/tekton-lab-Vnet/subnets/myAKSSubnet \ --service-principal 435be37-8310-f3d6-47af-e67c3184879 \ --client-secret Adkp1X38BLC31vmsEx7TV~k~bN.eLKT3T For client secret use of service principle, we created in previous steps. For service-principal use of service principle. For vnet-subnet-id use the ID of the subnet in the previous step. Enable addon for virtual nodes We will use the az aks enable-addons command in order to enable virtual nodes and use the tekton-lab-VirtualNodeSubnet subnet that we created beforehand: az aks enable-addons tekton-lab-VirtualNodeSubnet az aks enable-addons \ --resource-group tekton-lab \ --name myTektonAKSCluster \ --addons virtual-node \ --subnet-name tekton-lab-VirtualNodeSubnet az aks enable-addons \ --resource-group tekton-lab \ --name myTektonAKSCluster \ --addons virtual-node \ --subnet-name tekton-lab-VirtualNodeSubnet Get kubectl access To get access to a newly created AKS cluster with kubectl run the following command: kubectl az aks get-credentials --resource-group tekton-lab --name myTektonAKSCluster az aks get-credentials --resource-group tekton-lab --name myTektonAKSCluster Expected output: Merged "myTektonAKSCluster" as current context in /home/viachaslau/.kube/config Merged "myTektonAKSCluster" as current context in /home/viachaslau/.kube/config Check virtual node To verify the connection to your cluster and the availability of the ACI virtual node, use the kubectl get command to return a list of the cluster nodes. kubectl get kubectl get nodes kubectl get nodes You should be able to see your Azure Container Instances virtual nodes ready! Tekton Let’s install the latest version of Tekton with the following command: kubectl apply --filename https://storage.googleapis.com/tekton-releases/pipeline/latest/release.yaml kubectl apply --filename https://storage.googleapis.com/tekton-releases/pipeline/latest/release.yaml You can check the installation progress with the get pods command: get pods kubectl get pods --namespace tekton-pipelines kubectl get pods --namespace tekton-pipelines Now that Tekton is up and running let’s add Tekton Dashboard. It is a nice web-based UI for Tekton Pipelines. kubectl apply --filename https://github.com/tektoncd/dashboard/releases/latest/download/tekton-dashboard-release.yaml kubectl apply --filename https://github.com/tektoncd/dashboard/releases/latest/download/tekton-dashboard-release.yaml Tekton Dashboard By default, Tekton Dashboard is accessible through its cluster IP type service. To quickly get web access from Cloud Shell we can set up port forwarding with Tekton Dashboard: cluster IP kubectl --namespace tekton-pipelines port-forward svc/tekton-dashboard 9097:9097 kubectl --namespace tekton-pipelines port-forward svc/tekton-dashboard 9097:9097 Get access to the Tekton Dashboard by clicking “Web preview” in Cloud Shell and then specify port 9097 9097 Tekton task Let’s create a sample Tekton task. The Task is a series of Steps that you define in a particular order of execution as part of your pipeline. Task has inputs and could produce outputs. A Task executes as a Pod on your Kubernetes cluster. Task Steps Task Let’s create a tekton-task.yml file with the following content: apiVersion: tekton.dev/v1beta1 kind: Task metadata: name: hello spec: steps: - name: hello image: ubuntu command: - echo args: - "Hello World!" apiVersion: tekton.dev/v1beta1 kind: Task metadata: name: hello spec: steps: - name: hello image: ubuntu command: - echo args: - "Hello World!" And then apply this file with kubectl apply command: kubectl apply -f tekton-task.yml kubectl apply -f tekton-task.yml If you navigate to the Tasks section in Tekton Dashboard you will see the “hello” task appeared: Here is the YAML configuration of the “hello” task: To run tasks specifically on the virtual node we need to get rid of its default taint. To get your node taints use the kubectl describe command: kubectl describe kubectl describe node virtual-node-aci-linux kubectl describe node virtual-node-aci-linux To remove taint you need to apply the same taint but with “-” character: kubectl taint nodes virtual-node-aci-linux virtual-kubelet.io/provider=azure:NoSchedule- kubectl taint nodes virtual-node-aci-linux virtual-kubelet.io/provider=azure:NoSchedule- Expected output: node/virtual-node-aci-linux untainted node/virtual-node-aci-linux untainted To double-check that there is no taint left you can describe the node again: Now that taints are sorted we can create a TaskRun for that task with node selector configured in UI. This way we will make sure that the Tekton task will run on Azure Container Instances. If you face any weird issues with your pipelines you might want to double-check Tekton Pipelines and Tekton Dashboard versions in the compatibility matrix and pick not the latest version If you face any weird issues with your pipelines you might want to double-check Tekton Pipelines and Tekton Dashboard versions in the compatibility matrix and pick not the latest version compatibility matrix Navigate to the TaskRuns section on the left menu and you will see that TaskRun “hello” succeded: Summary: AKS Clusters with Azure Container Integration In this tutorial, we created an AKS cluster with the integration of Azure Container Instances as Virtual Nodes with corresponding dependencies. We successfully installed Tekton Pipelines and Tekton Dashboard in the Azure Kubernetes Service cluster. Finally, we were able to schedule a sample Tekton task in an on-demand Virtual node. Clean-up Here is what to do to clean up resources created in this tutorial” Delete Tekton installation kubectl delete -f kubectl apply --filename https://github.com/tektoncd/dashboard/releases/latest/download/tekton-dashbo0 kubectl delete --filename https://storage.googleapis.com/tekton-releases/pipeline/latest/release.yaml kubectl delete -f kubectl apply --filename https://github.com/tektoncd/dashboard/releases/latest/download/tekton-dashbo0 kubectl delete --filename https://storage.googleapis.com/tekton-releases/pipeline/latest/release.yaml To get rid of your virtual nodes, you can turn off them with az aks disable-addons command az aks disable-addons az aks disable-addons --resource-group tekton-lab --name myTektonAKSCluster --addons virtual-node az aks disable-addons --resource-group tekton-lab --name myTektonAKSCluster --addons virtual-node Delete resource group with AKS: az group delete --name tekton-lab --yes --no-wait az group delete --name tekton-lab --yes --no-wait