Skip to main content
Version: v2 (current)

Azure DevOps Integration with PanDev Metrics

import IdealImage from "@theme/IdealImage"; import Head from "@docusaurus/Head";

TL;DR. This guide connects Azure DevOps to PanDev Metrics in about 10 minutes. You will create a Personal Access Token (PAT) with Code, Build, and Work Items scopes, paste it into PanDev Metrics, pick projects, and verify that pull-request, pipeline, and Boards webhooks arrive. Audience: admin.

Before you begin​

  • PanDev Metrics version ≥ 1.0.250 (Cloud users always meet this)
  • Admin access to your PanDev Metrics tenant
  • An Azure DevOps user (service account preferred) with Project Administrator access on the projects to analyse
  • For on-prem PanDev Metrics: outbound 443 to dev.azure.com or to your Azure DevOps Server host
  • For Azure DevOps to reach an on-prem PanDev Metrics deployment: the PanDev Metrics host must be reachable from Azure DevOps at the public webhook URL

:::tip Use a service account Create a Microsoft Entra ID user dedicated to PanDev Metrics (for example pandev-metrics-bot@acme.onmicrosoft.com) and add it to the relevant Azure DevOps organization. This keeps automated PR comments distinct from human reviewers. :::

Step 1 — Create a Personal Access Token in Azure DevOps​

  1. Sign in to Azure DevOps as the service account.
  2. Select the user icon in the top-right and choose Personal access tokens.
  3. Select New Token and configure:
FieldValue
Namepandev-metrics
OrganizationYour organization (or All accessible organizations)
ExpirationCustom — 1 year recommended
ScopesCustom defined (see below)
  1. Under Custom defined, enable these scopes:
ScopeAccessWhy
CodeReadRead commits, branches, files for analysis
Pull Request ThreadsRead & writeRead PR metadata; post quality comments on PRs
BuildReadRead pipeline runs for deployment-frequency metrics
Work ItemsReadLink Boards work items to commits and PRs
  1. Select Create and copy the token — Azure DevOps shows it only once.

:::warning Save the token immediately The token is displayed only once. If you close the dialog without copying it, you must create a new token. :::

Step 2 — Open the Azure DevOps integration screen in PanDev Metrics​

Sign in to PanDev Metrics as a workspace admin and open Settings → Integrations → Azure DevOps.

Step 3 — Enter the organization URL and PAT​

Fill in the form:

FieldValue
Organization URLhttps://dev.azure.com/<your-org> for Services; https://azuredevops.acme.example/<your-collection> for Server
Personal access tokenValue from Step 1

Select Check connection. PanDev Metrics calls the Azure DevOps API and reports the authenticated user plus the count of accessible projects.

When the check passes, select Activate.

Step 4 — Pick the projects to monitor​

PanDev Metrics shows the list of Azure DevOps projects visible to the PAT. Select the ones you want to analyse and save.

PanDev Metrics starts a one-time backfill of the last 30 days of commits, pull requests, and pipeline runs.

Step 5 — Verify webhook registration​

PanDev Metrics registers Azure DevOps service hooks automatically per selected project. To check, open Project settings → Service hooks on Azure DevOps. You should see hooks pointing at the PanDev Metrics ingest URL.

For reference, PanDev Metrics handles these endpoints (URLs are filled in automatically during activation):

EventPanDev Metrics endpoint (informational)
Repository / project lifecycle/v1/azure/webhook/project
Pull-request events/v1/azure/webhook/merge-request
Azure Boards work-item events/v1/azure/webhook/boards

Recent deliveries should return HTTP 200 within seconds.

Verify​

Open Settings → Integrations → Azure DevOps in PanDev Metrics. You should see:

  • A green Connected badge
  • The count of selected projects
  • The timestamp of the Last sync (no older than 30 minutes once backfill completes)

Then open a dashboard with DORA metrics — within ~15 minutes after activation, deployment-frequency and PR-cycle-time tiles should reflect Azure DevOps data. Filter Lead Time by your Azure project to confirm Boards work items are linked to commits.

Troubleshooting​

Check connection returns "TF400813" or "401 Unauthorized"

The PAT is invalid, expired, or scoped to the wrong organization. Re-generate the PAT (Step 1) and make sure Organization matches the org you're connecting from PanDev Metrics. If you set the PAT to All accessible organizations, the service account must be a member of each org you want to monitor.

Service hooks register but Boards events don't appear

The PAT lacks Work Items: Read. Re-create the PAT with that scope and update it in PanDev Metrics under Settings → Integrations → Azure DevOps → Advanced → Update token. Boards events resume within minutes.

Pipeline runs missing for some projects

The PAT lacks Build: Read, or the service account doesn't have View build pipeline permission on the affected project. Add the user to a security group with build-read access (for example Build Administrators at the project level).

PR comments don't appear

The PAT lacks Pull Request Threads: Read & write, or Post PR comments is disabled in PanDev Metrics. Open Settings → Integrations → Azure DevOps → Advanced to confirm. The setting honours per-project overrides — confirm the target project isn't excluded.

Azure DevOps Server returns 404 on the API call

For Server, the Organization URL must include the collection: https://azuredevops.acme.example/DefaultCollection. Without the collection segment, the REST API returns 404. Update the URL in Settings → Integrations → Azure DevOps.

What gets collected​

PanDev Metrics ingests, per selected project:

  • Commits (SHA, author, timestamp, message, file stats) from Azure Repos
  • Pull requests (active / completed / abandoned) and their reviews, comments, and approvals
  • Pipeline runs (build and release) — start time, end time, result, environment
  • Work items linked to commits and PRs (from Azure Boards)

Source code itself is not stored. For details, see How the plugin works.

FAQ​

Are Azure Boards work items included automatically?​

Yes — when the PAT includes Work Items: Read, PanDev Metrics links Boards items to commits and PRs by the standard #1234 mention in commit messages and PR descriptions. For a dedicated Boards setup guide, see Azure Boards as a task tracker.

What's the minimum PAT scope set?​

For repos-only analytics: Code: Read and Pull Request Threads: Read & write. For DORA deployment-frequency metrics from pipelines: add Build: Read. For Boards linkage: add Work Items: Read. Including all four is the standard PanDev Metrics setup.

Does PanDev Metrics work with Azure DevOps Server?​

Yes. The same PAT-based flow works on Azure DevOps Server. The Organization URL must include the collection (for example https://azuredevops.acme.example/DefaultCollection). On-prem PanDev Metrics needs network reachability to the Azure DevOps Server host on port 443.

How often is data refreshed?​

Service hooks deliver pull-request, pipeline, and work-item events in real time — usually within seconds. PanDev Metrics also runs a reconciliation poll every 15 minutes to catch any missed deliveries.

How long does the initial backfill take?​

PanDev Metrics backfills the last 30 days when you activate the integration. For 10 projects this typically takes 5-15 minutes. Larger organizations with many pipeline runs can take an hour or more.

Can I rotate the PAT without re-installing?​

Yes. Generate a new PAT in Azure DevOps with the same scopes, then open Settings → Integrations → Azure DevOps → Advanced → Update token in PanDev Metrics. Service hooks stay registered and no backfill is re-triggered.

Can I limit the integration to specific projects?​

Yes. After activation, deselect projects on the integration screen at any time. PanDev Metrics removes service-hook registrations on the deselected projects within minutes.

Next steps​