“Efficient Pipeline Orchestration: Triggering and Tracking Multiple Pipelines in Azure DevOps”

In the fast-paced world of software development, automating and orchestrating your workflows is crucial to ensure smooth and efficient delivery of software. Azure DevOps provides a powerful platform for managing and executing pipelines, and one of its key features is the ability to trigger and track multiple pipelines from a single pipeline. In this blog post, we’ll explore how to set up this orchestration, track the completion status of these pipelines, and display the pass percentage once they’ve finished.

Why Trigger Multiple Pipelines?

Before we dive into the technical details, let’s understand why triggering multiple pipelines from a single pipeline is valuable:

1. Modularization: Breaking down your complex application into smaller, manageable components is a fundamental principle of DevOps. Triggering multiple pipelines allows you to build and test each component separately, promoting modularity and making it easier to identify and fix issues.
2. Parallel Execution: By triggering multiple pipelines simultaneously, you can significantly reduce your overall build and deployment time. This parallelism accelerates the software delivery process, helping you meet tight deadlines.
3. Isolation and Testing: Separate pipelines enable isolation and thorough testing of different parts of your application. This isolation ensures that a failure in one component doesn’t disrupt the entire workflow.

Real world use cases:

1. Microservices Architecture: In a microservices-based application, each microservice may have its own pipeline for building, testing, and deploying. Triggering these individual pipelines and coordinating their releases is essential to maintain the overall application.
2. Multi-Environment Deployments: When deploying an application across different environments (e.g., development, testing, staging, production), separate pipelines are needed for each environment. These pipelines ensure that code changes are properly validated and promoted through the release stages.
3. Release Variants: For applications with multiple release variants (e.g., free, pro, enterprise), separate pipelines can be triggered for each variant. This allows for customized testing and deployment processes for each variant.
4. Mobile App Releases: In mobile app development, you may have pipelines for Android and iOS platforms. When a new feature or bug fix is ready, you trigger both Android and iOS pipelines to ensure simultaneous releases on both platforms.
5. Infrastructure as Code (IaC): When managing infrastructure with IaC tools like Terraform or Ansible, you can have separate pipelines for provisioning infrastructure, testing configurations, and deploying changes. Coordinating these pipelines ensures that the infrastructure is in sync with application updates.
6. Plugin or Extension Ecosystem: If you’re maintaining a platform with a plugin or extension ecosystem, each plugin or extension may have its own pipeline for development and deployment. A top-level pipeline can trigger individual plugin pipelines when updates are ready.
7. Cross-Platform Applications: Applications that run on multiple platforms (e.g., Windows, Linux, macOS) may require separate pipelines for each platform-specific build and testing. These pipelines are triggered to ensure compatibility across platforms.
8. Security Scanning and Compliance: Security is paramount, and multiple pipelines can be used to scan code for vulnerabilities, perform penetration testing, and ensure compliance with security standards. Triggering these pipelines is crucial for secure releases.
9. Feature Flag Rollouts: When using feature flags to enable or disable features in real-time, triggering pipelines to manage flag rollouts and monitor their impact on the application is essential.
10. A/B Testing: For A/B testing, you may need separate pipelines to manage the deployment of different feature variations to specific user segments and gather data for analysis.
11. Third-Party Integrations: Applications that rely on third-party services may need pipelines to validate integrations with those services. These pipelines ensure that the application functions correctly with external dependencies.
12. Database Schema Updates: When making changes to the database schema, separate pipelines for schema migrations, testing, and data migration may be necessary. Coordinating these pipelines helps maintain data integrity.

Now, let’s walk through the steps to achieve this in Azure DevOps:

Step 1: Set Up the Parent Pipeline

1. Create a new Azure DevOps pipeline or use an existing one to serve as the parent pipeline.
2. Define your stages and jobs within this pipeline. Each stage represents a different component or task you want to build, test, or deploy.

Step 2: Trigger Multiple Pipelines

To trigger child pipelines from your parent pipeline, you can use the Azure DevOps REST API, Azure CLI, or YAML pipeline syntax. For example, using PowerShell:

# Define the variables
$azureDevOpsOrgUrl = "https://dev.azure.com/<organization>"
$azureDevOpsProject = "<project>"
$parentPipelineName = "<parent_pipeline_name>"
$childPipelineNames = @("<child_pipeline1>", "<child_pipeline2>", "<child_pipeline3>")
$timeoutInMinutes = 30

# Authenticate with Azure DevOps
az login

# Trigger child pipelines from the parent pipeline
$runIds = @()
foreach ($pipelineName in $childPipelineNames) {
    $runId = az pipelines run queue `
        --name $pipelineName `
        --org $azureDevOpsOrgUrl `
        --project $azureDevOpsProject `
        --pipeline-parameters "{}" `
        --branch master `
        --output json |
        ConvertFrom-Json
    $runIds += $runId.id
    Write-Host "Triggered $pipelineName with Run ID: $($runId.id)"
}

Step 3: Track Completion Status

Tracking the completion status of the child pipelines can be achieved in several ways:

1. Azure DevOps REST API: You can use the API to query the status of individual pipeline runs. Periodically poll the API to check for completion and gather results.
2. Azure DevOps Service Connection: Create a service connection to your Azure DevOps organization, allowing you to interact with your pipelines programmatically.

Step 4: Calculate Pass Percentage

Once all child pipelines have completed, calculate the pass percentage by aggregating test results, if applicable. You may need to extract test results from various sources and calculate the ratio of passed tests to total tests.

Step 5: Display Pass Percentage

Display the calculated pass percentage in a suitable format, such as a dashboard, email notification, or a custom report. This provides a clear picture of the quality of your software components.

Complete script for this is as follows:

# Define your Azure DevOps organization and project details
$organizationName = "YourOrganizationName"
$projectName = "YourProjectName"
$pipelineId = "YourPipelineID"

# Azure DevOps Personal Access Token (PAT) for authentication
$patToken = "YourPersonalAccessToken"

# Define the Azure DevOps REST API URL for getting pipeline runs
$baseUrl = "https://dev.azure.com/$organizationName/$projectName/_apis/pipelines/$pipelineId/runs?api-version=6.0"

# Function to get pipeline runs and check their status
function Get-PipelineStatus {
    $headers = @{
        Authorization = "Basic " + [Convert]::ToBase64String([Text.Encoding]::ASCII.GetBytes(":$($patToken)"))
    }

    $response = Invoke-RestMethod -Uri $baseUrl -Headers $headers -Method Get
    $latestRun = $response.value | Sort-Object -Property finishTime -Descending | Select-Object -First 1

    # Check if the pipeline has completed
    if ($latestRun.status -eq "completed") {
        $completionStatus = $latestRun.result
        $passPercentage = $latestRun.passedTests / $latestRun.totalTests * 100

        Write-Host "Pipeline Completion Status: $completionStatus"
        Write-Host "Pass Percentage: $passPercentage%"
    } else {
        Write-Host "Pipeline is still running..."
    }
}

# Define the polling interval (in seconds)
$pollingInterval = 60

# Infinite loop to continuously poll the pipeline status
while ($true) {
    Get-PipelineStatus
    Start-Sleep -Seconds $pollingInterval
}

Conclusion

Triggering and tracking multiple pipelines from a single Azure pipeline is a powerful technique to enhance your DevOps processes. It promotes modularity, parallelism, and efficient testing, all of which contribute to faster and more reliable software delivery. By calculating and displaying pass percentages, you gain valuable insights into the quality of your software, enabling continuous improvement in your development practices.

Embrace this automation and orchestration strategy in Azure DevOps, and you’ll be on your way to achieving greater efficiency and quality in your software delivery pipeline.

Thank you for reading and happy coding!

If you liked this article, please feel free to connect with me on LinkedIn.