Skip to main content
Version: v2 (current)

Azure DevOps Integration with PanDev Metrics

import IdealImage from "@theme/IdealImage"; import Head from "@docusaurus/Head";

TL;DR. This guide connects Azure DevOps to PanDev Metrics in about 10 minutes. You will create a Personal Access Token (PAT) with Code, Pull Request Threads, Build, and Work Items scopes (and a service account that is Project Administrator on the target projects), paste it into PanDev Metrics, pick projects, and verify that pull-request, pipeline, and Boards webhooks arrive. Audience: admin.

Before you begin​

  • PanDev Metrics version ≥ 1.0.250 (Cloud users always meet this)
  • Admin access to your PanDev Metrics tenant
  • An Azure DevOps user (service account preferred) with Project Administrator access on the projects to analyse — this grants the Edit subscriptions project permission that PanDev Metrics needs to register service hooks (there is no separate "Service Hooks" PAT scope in Azure DevOps)
  • For on-prem PanDev Metrics: outbound 443 to dev.azure.com or to your Azure DevOps Server host
  • For Azure DevOps to reach an on-prem PanDev Metrics deployment: the PanDev Metrics host must be reachable from Azure DevOps at the public webhook URL

:::tip Use a service account Create a Microsoft Entra ID user dedicated to PanDev Metrics (for example pandev-metrics-bot@acme.onmicrosoft.com) and add it to the relevant Azure DevOps organization. This keeps automated PR comments distinct from human reviewers. :::

Step 1 — Create a Personal Access Token in Azure DevOps​

  1. Sign in to Azure DevOps as the service account.
  2. Select the user icon in the top-right and choose Personal access tokens.
  3. Select New Token and configure:
FieldValue
Namepandev-metrics
OrganizationYour organization (or All accessible organizations)
ExpirationCustom — 1 year recommended
ScopesCustom defined (see below)
  1. Under Custom defined, enable these scopes:
ScopeAccessWhy
CodeRead & writeRead commits, branches, files for analysis; write is required so PanDev Metrics can register the git.* (pull-request, repository) service hooks
Pull Request ThreadsRead & writeRead PR metadata; post quality comments on PRs
BuildReadRead pipeline runs for deployment-frequency metrics
Work ItemsRead & writeLink Boards work items to commits and PRs; write is required so PanDev Metrics can register the workitem.created / workitem.updated (Boards) service hooks
  1. Select Create and copy the token — Azure DevOps shows it only once.

:::warning Service-account permissions register the hooks, not a PAT scope Azure DevOps has no "Service Hooks" PAT scope. Subscription creation (/_apis/hooks/subscriptions) is gated by the Edit subscriptions project permission, which is held by Project Administrators — that is why the service account must be a Project Administrator on every target project. The PAT only needs the scope for each hook's resource: Code for git.* and Work Items for workitem.*. If the account has those rights and the PAT has these four scopes but sync still fails, see Troubleshooting. :::

:::warning Save the token immediately The token is displayed only once. If you close the dialog without copying it, you must create a new token. :::

Step 2 — Open the Azure DevOps integration screen in PanDev Metrics​

Sign in to PanDev Metrics as a workspace admin and open Settings → Integrations → Azure DevOps.

Step 3 — Enter the organization URL and PAT​

Fill in the form:

FieldValue
Organization URLhttps://dev.azure.com/<your-org> for Services; https://azuredevops.acme.example/<your-collection> for Server
Personal access tokenValue from Step 1

Select Test connection. PanDev Metrics calls the Azure DevOps API and reports the authenticated user plus the count of accessible projects.

When the check passes, select Save.

Step 4 — Pick the projects to monitor​

PanDev Metrics shows the list of Azure DevOps projects visible to the PAT. Select the ones you want to analyse and save.

PanDev Metrics starts a one-time backfill of the last 30 days of commits, pull requests, and pipeline runs.

Step 5 — Verify webhook registration​

PanDev Metrics registers Azure DevOps service hooks automatically per selected project. To check, open Project settings → Service hooks on Azure DevOps. You should see hooks pointing at the PanDev Metrics ingest URL.

For reference, PanDev Metrics handles these endpoints (URLs are filled in automatically during activation):

EventPanDev Metrics endpoint (informational)
Repository / project lifecycle/v1/azure/webhook/project
Pull-request events/v1/azure/webhook/merge-request
Azure Boards work-item events/v1/azure/webhook/boards

Recent deliveries should return HTTP 200 within seconds.

Verify​

Open Settings → Integrations → Azure DevOps in PanDev Metrics. You should see:

  • A green Connected badge
  • The count of selected projects
  • The timestamp of the Last sync (no older than 30 minutes once backfill completes)

Then open a dashboard with DORA metrics — within ~15 minutes after activation, deployment-frequency and PR-cycle-time tiles should reflect Azure DevOps data. Filter Lead Time by your Azure project to confirm Boards work items are linked to commits.

Troubleshooting​

Test connection returns "TF400813" or "401 Unauthorized"

The PAT is invalid, expired, or scoped to the wrong organization. Re-generate the PAT (Step 1) and make sure Organization matches the org you're connecting from PanDev Metrics. If you set the PAT to All accessible organizations, the service account must be a member of each org you want to monitor.

Sync fails with 403 "No scope found" on service hooks

PanDev Metrics registers service-hook subscriptions per project (/_apis/hooks/subscriptions). Azure DevOps returns 403 "No scope found" when the PAT lacks the scope for the hook's resource — there is no separate "Service Hooks" scope:

  • git.* hooks (pull request, repository) → PAT needs Code: Read & write.
  • workitem.created / workitem.updated hooks (Boards) → PAT needs Work Items: Read & write.

Re-create the PAT with all four scopes listed in Step 1 and update it under Settings → Integrations → Azure DevOps → Advanced → Update token, then re-run the integration sync.

If the PAT already has all four scopes and you still see 403, the service account is missing the Edit subscriptions project permission — make it a Project Administrator on every target project (Project settings → Permissions → Users / Security groups).

Service hooks register but Boards events don't appear

The PAT lacks Work Items: Read (for reading work-item data) or Work Items: Read & write (required to register the workitem.* service hooks). Re-create the PAT with Work Items: Read & write and update it in PanDev Metrics under Settings → Integrations → Azure DevOps → Advanced → Update token. Boards events resume within minutes.

Pipeline runs missing for some projects

The PAT lacks Build: Read, or the service account doesn't have View build pipeline permission on the affected project. Add the user to a security group with build-read access (for example Build Administrators at the project level).

PR comments don't appear

The PAT lacks Pull Request Threads: Read & write, or Post PR comments is disabled in PanDev Metrics. Open Settings → Integrations → Azure DevOps → Advanced to confirm. The setting honours per-project overrides — confirm the target project isn't excluded.

Azure DevOps Server returns 404 on the API call

For Server, the Organization URL must include the collection: https://azuredevops.acme.example/DefaultCollection. Without the collection segment, the REST API returns 404. Update the URL in Settings → Integrations → Azure DevOps.

What gets collected​

PanDev Metrics ingests, per selected project:

  • Commits (SHA, author, timestamp, message, file stats) from Azure Repos
  • Pull requests (active / completed / abandoned) and their reviews, comments, and approvals
  • Pipeline runs (build and release) — start time, end time, result, environment
  • Work items linked to commits and PRs (from Azure Boards)

Source code itself is not stored. For details, see How the plugin works.

FAQ​

Are Azure Boards work items included automatically?​

Yes — when the PAT includes Work Items: Read & write, PanDev Metrics links Boards items to commits and PRs by the standard #1234 mention in commit messages and PR descriptions. For a dedicated Boards setup guide, see Azure Boards as a task tracker.

What's the minimum PAT scope set?​

There is no "Service Hooks" scope in Azure DevOps — registering subscriptions is gated by the Edit subscriptions project permission (Project Administrator), not by the PAT. The PAT only needs the scope for each hook's resource:

  • For repos-only analytics: Code: Read & write and Pull Request Threads: Read & write.
  • For DORA deployment-frequency metrics from pipelines: add Build: Read.
  • For Boards linkage: add Work Items: Read & write (write is needed to register the workitem.* service hooks).

All four scopes together with a Project Administrator service account is the standard PanDev Metrics setup.

Does PanDev Metrics work with Azure DevOps Server?​

Yes. The same PAT-based flow works on Azure DevOps Server. The Organization URL must include the collection (for example https://azuredevops.acme.example/DefaultCollection). On-prem PanDev Metrics needs network reachability to the Azure DevOps Server host on port 443.

How often is data refreshed?​

Service hooks deliver pull-request, pipeline, and work-item events in real time — usually within seconds. PanDev Metrics also runs a reconciliation poll every 15 minutes to catch any missed deliveries.

How long does the initial backfill take?​

PanDev Metrics backfills the last 30 days when you activate the integration. For 10 projects this typically takes 5-15 minutes. Larger organizations with many pipeline runs can take an hour or more.

Can I rotate the PAT without re-installing?​

Yes. Generate a new PAT in Azure DevOps with the same scopes, then open Settings → Integrations → Azure DevOps → Advanced → Update token in PanDev Metrics. Service hooks stay registered and no backfill is re-triggered.

Can I limit the integration to specific projects?​

Yes. After activation, deselect projects on the integration screen at any time. PanDev Metrics removes service-hook registrations on the deselected projects within minutes.

Next steps​