Running in CI/CD

Learn how to set up and run tests with TestDriver in CI/CD pipelines using GitHub Actions as an example for seamless end-to-end testing.

This guide explains how to set up TestDriver in CI/CD pipelines, using GitHub Actions as an example. The same principles can be applied to other CI/CD platforms like GitLab CI, Jenkins, or CircleCI.

Prerequisites

1. TestDriver API Key: Obtain your API key from TestDriver and store it as a CI/CD secret (for example, TD_API_KEY in GitHub Actions).
2. Test Files: Ensure your test files are saved in the testdriver/ directory of your repository.

Create a CI/CD workflow

This example uses GitHub Actions, but the same approach can be adapted for other CI/CD platforms:

1. Create a new file in your repository: .github/workflows/testdriver.yaml.
2. Add the following workflow configuration:

Example GitHub Actions workflow

.github/workflows/testdriver.yaml

name: TestDriver / Run / Regressions

on:
  push:
    branches: ["main"]
  pull_request:
  workflow_dispatch:
  schedule:
    - cron: '0 13 * * *'  # Every day at 1 PM UTC (adjust if needed for timezone)

permissions:
  actions: read
  contents: read
  statuses: write
  pull-requests: write

jobs:
  gather-test-files:
    name: Setup Test Matrix (./testdriver/*.yaml)
    runs-on: ubuntu-latest
    outputs:
      test_files: ${{ steps.test_list.outputs.files }}
    steps:
      - name: Check out repository
        uses: actions/checkout@v4
        with:
          ref: ${{ github.event.ref }}
      - name: Find all test files and extract filenames
        id: test_list
        run: |
          FILES=$(ls ./testdriver/*.yaml)
          FILENAMES=$(basename -a $FILES)
          FILES_JSON=$(echo "$FILENAMES" | jq -R -s -c 'split("\n")[:-1]')
          echo "files=$FILES_JSON" >> $GITHUB_OUTPUT

  test:
    needs: gather-test-files
    strategy:
      matrix:
        test: ${{ fromJson(needs.gather-test-files.outputs.test_files) }}
      fail-fast: false
    name: ${{ matrix.test }}
    steps:
      - name: Check out repository
        uses: actions/checkout@v4
        with:
          ref: ${{ github.event.ref }}
          fetch-depth: 0

      - name: Set up Node.js
        uses: actions/setup-node@v4
        with:
          node-version: "20"
          cache: "npm"

      - name: Install dependencies
        run: NODE_ENV=production npm ci

      - name: Display filename being tested
        run: echo "Running job for file: ${{ matrix.test }}"

      - name: Run TestDriver
        run: npx testdriverai@latest run testdriver/${{ matrix.test }} --headless
        env:
          TD_API_KEY: ${{ secrets.TD_API_KEY }}
          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
          FORCE_COLOR: "3"
          TD_WEBSITE: ${{ vars.TD_WEBSITE || 'https://your-app.com' }}
          TD_THIS_FILE: ${{ matrix.test }}

Trigger the workflow

1. Push changes to the main branch or open a pull request.
2. Alternatively, manually trigger the workflow from the Actions tab in your GitHub repository.
3. The scheduled trigger will run tests daily at 1 PM UTC for regression testing.

View results

1. Navigate to the Actions tab in your GitHub repository (or equivalent CI/CD dashboard).
2. Select the workflow run to view the test results.

For other CI/CD platforms, adapt the workflow syntax to match your platform’s requirements while keeping the same TestDriver CLI commands and environment variables.

Debugging tests

TestDriver provides a powerful debugging interface through its platform. This interface allows you to analyze test runs, identify failures, and optimize your test suite with detailed visual and textual feedback.

1. Step-by-step execution logs

• View each step of the test execution, including:
  • The action performed (for example, clicking a button, typing text).
  • The expected outcome (for example, verifying a specific element is visible).
  • The result (pass, fail, or skipped).
• Logs provide detailed context for each step, making it easier to pinpoint where and why a test failed.

2. Visual feedback

• Screenshots: See what the application looked like at each step of the test.
• GIF Previews: Watch a replay of the entire test execution to understand the flow and identify UI issues.
• Highlighting: Elements interacted with during the test are highlighted in screenshots, helping you verify that the correct elements were targeted.

3. Error details

• For failed steps, TestDriver provides:
  • Error messages: Detailed descriptions of what went wrong (for example, “Element not found”).
  • Stack traces: For advanced debugging of backend or script-related issues.
  • Suggestions: Recommendations for fixing common issues, such as adjusting prompts or improving element descriptions.

4. Test history

• Access the history of test runs to:
  • Compare results across different builds or environments.
  • Identify flaky tests by analyzing patterns in failures.
  • Track improvements or regressions over time.

5. Environment context

• View the environment details for each test run, including:
  • Operating system and browser version.
  • Screen resolution and viewport size.
  • Network conditions (if applicable).

6. Collaboration tools

• Share test results with your team by generating a shareable link.
• Add comments or annotations to specific steps to facilitate discussions and debugging.