Thomas Ngo

Software Engineer

6 minute read

In the past, we explored how to create a simple CI/CD pipeline. Then, we went through the steps to create a CI/CD pipeline for multiple release sites. In this journal, we will take a leap forward by leveraging Azure DevOps more effectively through the use of templates.

Motivation

Previously, we created an azure-pipeline.yaml that contained all the stages, jobs, and steps. This approach works for simple projects and when deploying a single site. However, as the pipeline grows with more instructions or when new stages, jobs, or steps are introduced, the YAML file becomes increasingly complex and harder to manage. To simplify this process, we can utilize templates provided by Azure DevOps Services. This helps us to reuse code and clean up our pipeline.

Recap

Here’s our azure-pipeline.yaml from a previous journal:

# Starter pipeline
# Start with a minimal pipeline that you can customize to build and deploy your code.
# Add steps that build, run tests, deploy, and more:
# https://aka.ms/yaml

trigger:
  - main
  - azure-pipelines

variables:
  solution: "**/*.sln"
  buildPlatform: "x64"
  buildConfiguration: "Release"

stages:
  - stage: BuildFinalRuleDAC
    jobs:
      - job: Build
        pool:
          name: default
          demands:
            - Agent.Name -equals agent1
            - Agent.Version -gtVersion 2.153.1
        steps:
          - task: NuGetCommand@2
            inputs:
              command: "restore"
              restoreSolution: "**/*.sln"
              feedsToUse: "select"
          - task: UseDotNet@2
            inputs:
              packageType: "sdk"
              version: "6.x"
          - task: DotNetCoreCLI@2
            inputs:
              command: "restore"
              projects: "FinalRuleDAC/FinalRuleDAC.csproj"
          - task: DotNetCoreCLI@2
            inputs:
              command: "build"
              projects: "FinalRuleDAC/FinalRuleDAC.csproj"
              configuration: "$(buildConfiguration)"
  - stage: BuildProviderDMS
    dependsOn: BuildFinalRuleDAC
    jobs:
      - job: Build
        pool:
          name: default
          demands:
            - Agent.Name -equals agent1
            - Agent.Version -gtVersion 2.153.1
        steps:
          - task: DotNetCoreCLI@2
            inputs:
              command: "restore"
              projects: "ProviderDMS/ProviderDMS.csproj"
          - task: DotNetCoreCLI@2
            inputs:
              command: "build"
              projects: "ProviderDMS/ProviderDMS.csproj"
              configuration: "$(buildConfiguration)"
  - stage: CreateProviderDMSArtifact
    dependsOn: BuildProviderDMS
    jobs:
      - job: CreateArtifact
        pool:
          name: default
          demands:
            - Agent.Name -equals agent1
            - Agent.Version -gtVersion 2.153.1
        steps:
          - task: UseDotNet@2
            inputs:
              packageType: "sdk"
              version: "6.x"
          - task: DotNetCoreCLI@2
            inputs:
              command: "publish"
              projects: "ProviderDMS/ProviderDMS.csproj"
              zipAfterPublish: false
              arguments: "--configuration $(buildConfiguration) --runtime win-x64 --self-contained --output $(Build.ArtifactStagingDirectory) /p:PublishReadyToRun=true"
          - task: DownloadSecureFile@1
            inputs:
              secureFile: "appsettings.stage.json"
          - task: PowerShell@2
            inputs:
              targetType: inline
              script: Copy-Item -Path "$(Agent.TempDirectory)/appsettings.stage.json" -Destination "$(Build.ArtifactStagingDirectory)/ProviderDMS/appsettings.json"
            displayName: "Copy stage appsettings"
          - task: DownloadSecureFile@1
            inputs:
              secureFile: "site.stage.css"
          - task: PowerShell@2
            inputs:
              targetType: inline
              script: Copy-Item -Path "$(Agent.TempDirectory)/site.stage.css" -Destination "$(Build.ArtifactStagingDirectory)/ProviderDMS/wwwroot/css/site.css"
            displayName: "Copy stage site.css"
          - task: PublishBuildArtifacts@1
            displayName: "Publish Artifact"
            inputs:
              PathtoPublish: "$(build.artifactstagingdirectory)"
              ArtifactName: publishStage
            condition: succeededOrFailed()

AAs you can see, it’s quite long and difficult to manage. Let’s break it down into smaller pieces and apply the stage template

Stage Template

Let’s create the build.yaml template first. Here’s how it looks:

stages:
  - stage: BuildFinalRuleDAC
    displayName: DAC
    jobs:
      - job: Build
        pool:
          name: default
          demands:
            - Agent.Name -equals agent1
            - Agent.Version -gtVersion 2.153.1
        steps:
          - script: |
              echo "System.DefaultWorkingDirectory: $(System.DefaultWorkingDirectory)"
            displayName: "Show System.DefaultWorkingDirectory"
          - task: DotNetCoreCLI@2
            inputs:
              command: "restore"
              projects: "FinalRuleDAC/FinalRuleDAC.csproj"
          - task: DotNetCoreCLI@2
            inputs:
              command: "build"
              projects: "FinalRuleDAC/FinalRuleDAC.csproj"
              configuration: "$(buildConfiguration)"
  - stage: BuildProviderApi
    displayName: ProviderApi(KDXTi)
    dependsOn: BuildFinalRuleDAC
    jobs:
      - job: Build
        pool:
          name: default
          demands:
            - Agent.Name -equals agent1
            - Agent.Version -gtVersion 2.153.1
        steps:
          - task: DotNetCoreCLI@2
            inputs:
              command: "restore"
              projects: "ProviderApi/ProviderApi.csproj"
          - task: DotNetCoreCLI@2
            inputs:
              command: "build"
              projects: "ProviderApi/ProviderApi.csproj"
              configuration: "$(buildConfiguration)"
  - stage: BuildProviderDMS
    displayName: ProviderDMS(Web UI)
    dependsOn: BuildFinalRuleDAC
    jobs:
      - job: Build
        pool:
          name: default
          demands:
            - Agent.Name -equals agent1
            - Agent.Version -gtVersion 2.153.1
        steps:
          - task: DotNetCoreCLI@2
            inputs:
              command: "restore"
              projects: "ProviderDMS/ProviderDMS.csproj"
          - task: DotNetCoreCLI@2
            inputs:
              command: "build"
              projects: "ProviderDMS/ProviderDMS.csproj"
              configuration: "$(buildConfiguration)"

Next, we create the publish templates for the Web API and Blazor projects. In this example, DAC is the data access layer we wrote to query data from an SQL Server database. ProviderApi is a web API service for validating records based on specific business logic. ProviderDMS is our Blazor-based web server project.

Here are examples of publishProviderApi.yaml and publishProviderDMS.yaml:

# publishProviderApi.yaml
parameters:
  dependsOn: []
  version: ""

stages:
  - stage: $_CreateProviderApiArtifact
    displayName: ProviderApi(KDXTi) Artifact ($)
    dependsOn: $
    jobs:
      - job: $_CreateArtifact
        pool:
          name: default
          demands:
            - Agent.Name -equals agent1
            - Agent.Version -gtVersion 2.153.1
        steps:
          - task: UseDotNet@2
            inputs:
              packageType: "sdk"
              version: "8.x"
          - task: DotNetCoreCLI@2
            inputs:
              command: "publish"
              projects: "ProviderApi/ProviderApi.csproj"
              publishWebProjects: false
              zipAfterPublish: false
              arguments: "--configuration $(buildConfiguration) --runtime win-x64 --self-contained --output $(Build.ArtifactStagingDirectory)/$ /p:PublishReadyToRun=true"
          # - task: DownloadSecureFile@1
          #   inputs:
          #     secureFile: 'appsettings.$.providerapi.json'
          # - task: PowerShell@2
          #   inputs:
          #     targetType: inline
          #     script: Copy-Item -Path "$(Agent.TempDirectory)/appsettings.$.providerapi.json" -Destination "$(Build.ArtifactStagingDirectory)/$/ProviderApi/appsettings.json"
          #   displayName: 'Copy $ ProviderApi appsettings'
          - task: PublishBuildArtifacts@1
            displayName: "Publish Artifact"
            inputs:
              PathtoPublish: "$(build.artifactstagingdirectory)/$"
              ArtifactName: publish$
            condition: succeededOrFailed()

# pubishProviderDMS.yaml
parameters:
  dependsOn: []
  version: ""

stages:
  - stage: $_CreateProviderDMSArtifact
    displayName: ProviderDMS Artifact ($)
    dependsOn: $
    jobs:
      - job: $_CreateArtifact
        pool:
          name: default
          demands:
            - Agent.Name -equals agent1
            - Agent.Version -gtVersion 2.153.1
        steps:
          - script: |
              echo "Build.ArtifactStagingDirectory: $(Build.ArtifactStagingDirectory)"
            displayName: "Show Build.ArtifactStagingDirectory"
          - task: UseDotNet@2
            inputs:
              packageType: "sdk"
              version: "6.x"
          - task: DotNetCoreCLI@2
            inputs:
              command: "publish"
              projects: "ProviderDMS/ProviderDMS.csproj"
              zipAfterPublish: false
              arguments: "--configuration $(buildConfiguration) --runtime win-x64 --self-contained --output $(Build.ArtifactStagingDirectory)/$ /p:PublishReadyToRun=true"
          # - task: DownloadSecureFile@1
          #   inputs:
          #     secureFile: 'appsettings.$.providerdms.json'
          # - task: PowerShell@2
          #   inputs:
          #     targetType: inline
          #     script: Copy-Item -Path "$(Agent.TempDirectory)/appsettings.$.providerdms.json" -Destination "$(Build.ArtifactStagingDirectory)/$/ProviderDMS/appsettings.json"
          #   displayName: 'Copy $ appsettings'
          - task: DownloadSecureFile@1
            inputs:
              secureFile: "site.$.css"
          - task: PowerShell@2
            inputs:
              targetType: inline
              script: Copy-Item -Path "$(Agent.TempDirectory)/site.$.css" -Destination "$(Build.ArtifactStagingDirectory)/$/ProviderDMS/wwwroot/css/site.css"
            displayName: "Copy $ site.css"
          - task: PublishBuildArtifacts@1
            displayName: "Publish Artifact"
            inputs:
              PathtoPublish: "$(build.artifactstagingdirectory)/$"
              ArtifactName: publish$
            condition: succeededOrFailed()

Note: As observed from the information above, parameters are used in these templates. The reason for this is that we have three different sites that need to be deployed (Stage, MHP, and SUD). Each site uses the same source code but requires different configurations in appsettings.json. Therefore, instead of dumping all the instructions (e.g., jobs, tasks, and steps) into one giant pipeline, we can leverage template parameters and variables to help us achieve this goal.

Main Build Pipeline

With the above setup, we can achieve a cleaner and shorter version of azure-pipeline.yaml, which looks like the following:

# Starter pipeline
# Start with a minimal pipeline that you can customize to build and deploy your code.
# Add steps that build, run tests, deploy, and more:
# https://aka.ms/yaml

trigger:
  - develop

variables:
  solution: "**/*.sln"
  buildPlatform: "Any CPU"
  buildConfiguration: "Release"

stages:
  - template: cicd-templates/build.yaml
  - template: cicd-templates/publishProviderDMS.yaml
    parameters:
      dependsOn: "BuildProviderDMS"
      version: "stage"
  - template: cicd-templates/publishProviderApi.yaml
    parameters:
      dependsOn: "stage_CreateProviderDMSArtifact"
      version: "stage"
  - template: cicd-templates/publishProviderDMS.yaml
    parameters:
      dependsOn: "stage_CreateProviderApiArtifact"
      version: "mhp"
  - template: cicd-templates/publishProviderApi.yaml
    parameters:
      dependsOn: "mhp_CreateProviderDMSArtifact"
      version: "mhp"
  - template: cicd-templates/publishProviderDMS.yaml
    parameters:
      dependsOn: "mhp_CreateProviderApiArtifact"
      version: "sud"
  - template: cicd-templates/publishProviderApi.yaml
    parameters:
      dependsOn: "sud_CreateProviderDMSArtifact"
      version: "sud"

File Transform

We are not done yet because when we deploy the code, it will use the same appsettings.json that we have on our local environment. To replace the configurations inside **/*.json files, we can use one of two methods:

  1. Secure Files
  2. Variable Groups and JSON Variable Substitution in the Release Pipeline

Secure Files

To use this method, we first need to prepare all the appsettings.<version>*.json files that we need so we can download and replace them in our artifacts.

secure-file

Then, in our template YAML, we can use this PowerShell command to perform the replacements:

- task: DownloadSecureFile@1
  inputs:
    secureFile: "appsettings.$.providerdms.json"
- task: PowerShell@2
  inputs:
    targetType: inline
    script: Copy-Item -Path "$(Agent.TempDirectory)/appsettings.$.providerdms.json" -Destination "$(Build.ArtifactStagingDirectory)/$/ProviderDMS/appsettings.json"
  displayName: "Copy $ appsettings"

With this approach, after the artifact is created, the appsettings.json file will be replaced with the content of the appsettings.json that we placed inside the Secure Files.

JSON variable subsitution

A more elegant approach is to define a variable group for each release stage. This is called File Transform & Variable Subsitution Options in the release pipeline. Azure DevOps Services is smart enough to find the key-value pairs that we want to replace inside appsettings.json if we provide it with a variable group containing the exact keys.

First, let’s create variable groups like the following:

variable-group

mhp

stage

Make sure these defined variables match the ones inside our appsettings.json located in the local development environment.

For example, this is my appsettings.json for the above variables:

example-appsettings

Then, in the release pipeline, we instruct the Azure DevOps Service to look for appsettings.json using wildcard or absolute/relative path to the appsetting.

release-cicd

Lastly, we need to ensure that Azure applies the correct appsettings for the correct stage in the release pipeline. This is called “scope” in the release pipeline.

For instance, here’s how I set up my release pipeline variable group:

scope

Conclusion

That’s it! Now we have learned how to separate our main pipeline using templates. We have also gained the knowledge of how to replace configurations or *.json files using Secure Files and Variable Groups.

Tags:

Updated:

You may also enjoy