CEO's Guide to Engineering Team Health (Non-Technical)
Most non-technical CEOs I've met treat engineering as either a black box or a theater. Black-box CEOs ask "how's engineering?" at the executive meeting, accept "we're on track" as an answer, and act surprised four quarters later when the senior architect resigns and the product roadmap stalls. Theater CEOs become amateur engineering managers — they learn to recite DORA metrics, mispronounce "Kubernetes," and inadvertently turn every roadmap discussion into a technical argument they can't follow.
Neither failure mode is about intelligence. It's about the absence of a short, non-technical vocabulary for engineering health. First Round's 2023 State of Startups survey found 68% of first-time CEOs rate themselves "somewhat" or "very" dependent on their CTO for all engineering judgment calls — which is fine until the CTO leaves or disagrees with the board on direction.
This guide is the minimum CEO vocabulary: 6 questions that let you test whether engineering is healthy without pretending to be technical.
{/* truncate */}
What a CEO really needs to know
You don't need to know what a pull request is. You need to know whether your team is shipping, learning, recovering from mistakes, and still here in 12 months. Those four things have non-technical proxies. If your CTO can answer them in plain language, the engineering function is probably fine. If the answers are evasive or defensive, you have a signal.
Engineering dashboards are built for engineering leaders. They show DORA metrics, deployment frequencies, test coverage. A CEO staring at one of these dashboards is in the wrong mode: you're not trying to run engineering, you're trying to trust engineering. Those are different jobs.
McKinsey's 2022 Developer Velocity study tracked 440 companies and found that top-quartile engineering organizations outperformed bottom-quartile ones on revenue growth by 4-5x. The study's authors noted, carefully, that the measurable drivers were mostly cultural: tool satisfaction, onboarding speed, management clarity. Not infrastructure spend. Not headcount. Culture.
A CEO can read culture. A CEO usually can't read Kubernetes. Start from the strength.
The 6 questions every CEO should be asking
The CEO's engineering loop. Not every question needs a dashboard; most need a conversation. Data comes in only when the conversation is confused.
1. "Are we losing our best engineers, or keeping them?"
The single most predictive engineering-health signal is senior-engineer retention. Not junior. Not average. The top 20% of your engineering team. They're the ones who can leave easily, know the codebase deeply, and whose departure forces a 6-12 month rebuild of institutional knowledge.
Numbers to ask your CTO for, quarterly:
- How many engineers at the "senior" level or above have left in the last 12 months?
- How many joined at the same level?
- What's the median tenure of current senior engineers?
A healthy B2B SaaS team loses 8-15% of senior engineers per year (Stack Overflow's 2024 survey aggregates). Above 25% is a crisis signal even if the CTO's dashboard looks green. Below 5% is also a signal — either the comp is above market and you're over-spending, or top performers aren't growing and eventually will leave in a cluster.
2. "Is the team shipping, or just busy?"
Visible activity — meetings, Slack messages, Jira tickets closed — isn't the same as shipping. Deloitte's research on knowledge workers tracks the distinction: activity metrics correlate weakly with business outcomes; delivery metrics correlate strongly.
Ask: "What did we ship to customers this quarter, and what did the customers do with it?" Good answers cite specific features with specific usage numbers. Bad answers cite tickets closed or PRs merged. "We shipped 127 tickets" is a vanity metric; "we shipped the multi-currency billing feature and 23% of paying customers activated it in the first month" is a health metric.
The CEO-specific test: could a board member understand your CTO's answer without a translator? If yes, the team is probably shipping outcomes. If no, the team may be optimizing for activity.
3. "What's on fire that you're not telling me about?"
The one question every good CEO asks and every bad CEO avoids. Engineering has latent crises all the time — aging databases, third-party dependencies going out of support, security debt, a key contractor whose contract ends in 60 days. None of them are visible in a sprint board.
Ask quarterly: "If I gave you an unrestricted 2-week engineering pause, what would you use it to fix?" The answer reveals whether your CTO has been absorbing risk silently. A "nothing urgent" answer is usually wrong — there's always something. A thoughtful answer with one or two specific items is healthy.
The anti-pattern: a long list. If your CTO enumerates 8 things, they're overwhelmed and haven't been protecting you from the list.
4. "Where is the team spending time versus where we need them to spend time?"
Budget holders track "where is the money going?" CEOs should track "where is the engineering time going?" because for most B2B SaaS companies, engineering time is a bigger constraint than engineering money.
Three buckets are worth knowing:
- New-feature work (roadmap-driven, revenue-facing)
- Maintenance + support (bug fixes, customer escalations, keeping the lights on)
- Internal platform / tech debt (refactoring, infrastructure, tooling)
A healthy allocation for a Series A-B B2B SaaS company is roughly 55% feature, 25% maintenance, 20% platform. At Series C+ the feature share drops and platform rises. If your engineering team is spending >40% on maintenance, you have a quality or technical-debt problem. If it's <10% on platform, you're accumulating problems for the next CEO cycle.
PanDev Metrics tracks this split via IDE and Git telemetry cross-referenced with ticket types — the split is computed automatically rather than self-reported. Where the automation matters: self-report numbers are always rosier than reality. Engineers pattern-match to "we're roadmap-heavy" because they were told that's what leadership wants to hear. The IDE heartbeat data tells a different, more actionable story.
5. "How long does it take a new engineer to become productive?"
Onboarding ramp-up time is a hidden but load-bearing health metric. It tells you two things at once: how much institutional knowledge is undocumented, and how much slack the current team has to invest in the next generation.
Benchmark: a new mid-level engineer should produce their first meaningful merged PR within 2 weeks and reach 70-80% of team average productivity within 8-12 weeks. Teams with 16+ week ramp times are typically either understaffed or running an undocumented codebase — both signal health debt.
Ask your CTO: "When we hired [most recent senior engineer], how long before they were doing real work independently?" If the answer is "3-4 months" you have a typical team. If the answer is "I'm not sure, we're still onboarding them six months in," you have a problem that compounds with every hire. More on this in our onboarding ramp-up post.
6. "When something breaks, how fast do we recover?"
You don't need to understand MTTR numerically. You need to understand the culture around incidents. Two companies can have identical MTTRs but very different health trajectories.
Ask after the next production incident: "What would happen if we had this same bug six months from now? Would we catch it faster, slower, or the same?" A healthy engineering team gets faster over time; post-incident reviews produce changes, and changes compound. An unhealthy team has the same bug three times in a year and explains it each time.
This is a cultural read, not a numerical one. The CEO role is to ask it. A good post-mortem culture is one of the strongest engineering-health indicators a non-technical leader can evaluate.
Red flags: signals things are worse than they look
Watch for these — they indicate the engineering function is in trouble even if dashboards look green:
- "We're pivoting to AI this quarter" announced by the CTO without a clear customer or revenue link. Often a sign that the team's real velocity is unshippable, and "AI pivot" is cover.
- Senior engineers stop speaking up in meetings. When your top people go quiet, they've already left mentally. Physical resignation follows within 6-9 months.
- The "hero" pattern. A single engineer is credited with rescuing every release. Heroes are a failure mode, not a success mode — they mean your processes don't work without them, and they will burn out.
- Roadmap gets replanned more than quarterly. Occasional replanning is healthy; constant replanning means you don't know what your team is capable of delivering.
- "It's complicated" appears in every technical answer. Some things are complicated. Most aren't. A CTO who says "it's complicated" to more than 50% of your questions is either hiding, bad at communicating, or both.
How to act on this
Immediate (this week)
- Schedule a 30-minute conversation with your CTO. Use questions 1, 3, 5 as the structure.
- Ask HR for senior-engineer retention numbers separated from total engineering retention. Most dashboards blur the two.
Short-term (this quarter)
- Establish the "what's on fire" question as a standing item in your 1:1 with the CTO.
- Agree on a feature/maintenance/platform split target and track actual vs target.
- Read one engineering post-mortem per quarter, start to finish. You don't need to understand every technical detail; you need to understand how the team reasons about failure.
Long-term (this year)
- Build one technical co-founder or advisor into your inner circle who is not your CTO. You need a second opinion on engineering that's independent of reporting lines.
- Calibrate your engineering-health read by comparing notes with 2-3 peer CEOs in similar-stage companies. Benchmarks without peer context are misleading.
The metrics framework (simple version)
| Metric | What it tells you | Healthy range |
|---|---|---|
| Senior engineer retention (12mo) | Whether top talent believes in the company | 85-95% |
| Feature / Maintenance / Platform split | Where the team's hours really go | ~55 / 25 / 20% for SaaS A-B |
| Onboarding time to meaningful PR | Institutional knowledge health | Under 2 weeks |
| MTTR improvement trend | Whether the team learns from incidents | Decreasing YoY |
| Senior-engineer 1:1 cadence with CTO | Early warning signal | Weekly at minimum |
| Replanning frequency | Delivery predictability | Quarterly or less |
Six numbers. None require you to read a dashboard. All are answers your CTO should have.
The honest limit
This framework works best for B2B SaaS companies between Series A and Series C. Earlier-stage startups have too few data points (a 3-engineer team has no "senior retention rate" — it's 0% or 100%). Later-stage organizations have so much structure that "ask your CTO" doesn't scale past 200 engineers — you need VP-level direct reports and the framework needs adapting.
Our data: we see the numbers that back questions 4-6 at scale across 100+ B2B SaaS companies. Questions 1-3 are informed by customer conversations and published research, not primarily our dataset. If you're in a 20-30 person startup, treat this as a starting point, not a scorecard.
The contrarian point
Every engineering-blog post tells CEOs to "learn DORA metrics" or "understand deployment frequency." I don't. A CEO who memorizes DORA without understanding why ends up making worse decisions than one who stays in the business-outcome frame. Your job as CEO isn't to understand engineering. It's to build a trust relationship with someone who does, and to know when that relationship's signal is degrading.
The six questions above are engineered to produce that signal without requiring technical depth. If your CTO gives you non-answers to three or more, the problem isn't your technical knowledge — it's the relationship.
Related reading
- How to Justify Hiring 5 More Developers to Your CFO — the CFO-facing counterpart to this guide
- The CTO Dashboard — what your CTO should be looking at weekly (so you don't have to)
- 5 Data Patterns That Scream 'Your Developer Is Burning Out' — early warning signals you'd otherwise miss
- External: McKinsey's Developer Velocity Index — the definitive CEO-level case for why engineering quality is a business issue