Featured image

Introduction

In today’s fast-paced software development environment, maintaining high-quality code with efficient testing workflows is paramount. Azure Test Plans offers a robust platform for managing both manual and automated testing efforts within Azure DevOps. This comprehensive guide delves deep into integrating automated tests within Azure Test Plans, enabling teams to run tests on demand, monitor results effectively, and optimize their pipelines with best practices tailored for intermediate to advanced users.

Why Integrate Automated Testing with Azure Test Plans?

Integrating automated tests directly into Azure Test Plans brings several advantages:

  • User-Friendly Execution: Testers without deep knowledge of build or release pipelines can effortlessly run automated tests.
  • Selective and On-Demand Runs: Instead of running all tests on schedule, testers can select specific tests to execute as needed.
  • Efficient Failure Recovery: Easily rerun tests that failed due to environmental issues or after fixes.

This workflow bridges the gap between continuous integration pipelines and test management, providing a seamless experience for testers and developers alike.

Prerequisites

Before setting up automated test execution from Azure Test Plans, ensure the following:

Access and Permissions

  • Azure DevOps Access: At minimum, Basic access level with work item viewing permissions.
  • Test Plans Access: To create and manage test plans, suites, and configurations, Basic + Test Plans access or equivalent Visual Studio subscriptions (Enterprise, Test Professional, MSDN Platforms).
  • Permissions:
    • Edit work items and test artifacts.
    • Manage test plans and test suites.
    • Create and manage releases and deployment.

Tools and Environment

  • Test Plans populated with automated test cases associated with test methods in Visual Studio (2015 or newer).
  • Build Pipeline producing build artifacts that contain the test binaries.
  • Release Pipeline configured to run tests, either existing or created from the “Run automated tests from Test Manager” template.

Setting Up Automated Testing Integration

Step 1: Configure Test Plan Settings

Navigate to the Test Plans page in Azure DevOps, select your test plan, and open the shortcut menu to choose Test plan settings.

  • Select the build pipeline that produces your test binaries.
  • Choose a specific build number or opt to always use the latest build.
  • Choose an existing release pipeline created from the appropriate template or create a new one.
  • Assign meaningful names to the pipeline and stages for clarity.

Step 3: Configure the Release Pipeline Tasks

  • Visual Studio Test Platform Installer: Add this task if Visual Studio Test Platform is not pre-installed on agents.
  • Visual Studio Test Task (version 3 recommended):
    • Set Select tests using to Test run to ensure tests are selected based on the test run context.
    • Choose toolsInstaller for the test platform version.
    • For UI tests on physical browsers or thick clients, configure the agent to run in interactive mode with autologon.
    • Specify any run settings or configuration files that provide parameters such as URLs or connection strings.

Step 4: Agent Job and Parallelism

  • Verify that the agent queue contains the necessary test machines.
  • For faster execution, configure parallelism by enabling multiple executions and specifying the number of agents.

Ensure the build pipeline producing the test binaries is linked as an artifact source in your release pipeline. This linkage is critical for the release pipeline to access test binaries.

Running Automated Tests On Demand

  1. In the Test Plans web portal, open the test plan and locate the test suite containing your automated tests.
  2. Select one or more test cases to run.
  3. Click Run for web application to trigger the automated test execution.

The system automatically validates the selected tests, permissions, and pipeline configurations before creating a test run and triggering the release pipeline.

Monitoring Test Progress

  • Access the View test run link to monitor execution and analyze results.
  • Detailed error messages, stack traces, console logs, and attachments for failed tests assist in debugging.
  • The Runs page provides an overview of all test runs, with drill-down capabilities.

Advanced Usage and Best Practices

Passing Parameters to Tests

Pass runtime parameters such as app URLs or database strings through a .runsettings file configured in the Visual Studio Test task. This flexibility allows tests to adapt to different environments without code changes.

<?xml version="1.0" encoding="utf-8"?>
<RunSettings>
  <TestRunParameters>
    <Parameter name="AppUrl" value="https://staging.myapp.com"/>
  </TestRunParameters>
</RunSettings>

Integrate this file in your pipeline and override parameters as needed.

Handling UI Tests

  • For UI tests using Selenium or Coded UI running on physical browsers, ensure agents run interactively with autologon enabled to allow browser automation.
  • For headless browsers, interactive mode is not required.

Running Tests in Parallel

  • To optimize test execution times, leverage agent pools with multiple agents.
  • Configure stage triggers to allow multiple releases to deploy simultaneously if your application supports parallel test execution.

Managing Permissions

Assign necessary release permissions to users running tests:

  • Create and manage releases
  • Edit release stages
  • Manage deployments

Review Release permissions documentation for detailed guidance.

Troubleshooting Common Issues

Issue Resolution
Release pipeline/stage not listed Ensure the build pipeline is linked as the primary artifact in the release pipeline.
Insufficient permission to trigger release Assign Create releases and Manage deployments permissions to the user.
No automated tests found Verify automation status on test cases and that test binaries exist in artifacts.
Tests stuck or error during execution Confirm correct Visual Studio Test task version and pipeline configuration.

FAQ Highlights

Can multiple testers run tests in parallel using the same release pipeline?

Yes, provided there are enough agents and parallel job licenses. Avoid running the exact same tests simultaneously to prevent data overwrites.

Can I override build or release stage at runtime?

Yes, use the Run with options feature to select specific builds and release stages per test run.

Does Microsoft Test Manager support this workflow?

No. Automated test execution from Azure Test Plans is supported only in the web-based Azure DevOps interface.

Real-World Scenario: Integrating Automated UI Tests for a Web App

Consider a team developing a web application with nightly builds. The team maintains a suite of Selenium UI tests automated through Visual Studio.

  • The build pipeline compiles the app and publishes test binaries.
  • A release pipeline is configured with Visual Studio Test tasks to run these Selenium tests.
  • Testers use Azure Test Plans to select specific UI tests to run on demand against the latest build.
  • The release pipeline runs tests on interactive agents configured with Google Chrome.
  • Test results are immediately visible in Azure Test Plans, enabling quick feedback and bug triage.

This setup streamlines regression testing and enhances collaboration between developers and testers.

Conclusion

Integrating automated tests into Azure Test Plans empowers teams to manage, execute, and analyze tests efficiently within a unified platform. By following best practices such as proper pipeline configuration, parameterization, and agent management, teams can reduce manual overhead and accelerate quality assurance cycles.

This in-depth guide provides both foundational setup instructions and advanced tips to maximize your Azure Test Plans automated testing workflow. Embrace these practices to elevate your DevOps testing strategy and deliver reliable software faster.

References

Author: Joseph Perez