Profiling IDE Extension Startup: Why Cold-Start Metrics Mislead You About Real-World User Experience

Ever opened VS Code after a weekend away, waited forever for your extensions to load, and thought: 'Why is this so slow?' You check the docs—cold-start metrics look fine. But here's the rub: cold-start numbers are a snapshot of one specific scenario, not your daily grind.

This article tears down the myth that cold-start metrics equal user experience. We'll dig into warm starts, lazy loading, and the silent killers—like network calls and config parsing—that benchmarks ignore.

Why Cold-Start Metrics Are the Wrong Yardstick

The gap between synthetic benchmarks and real usage

Most teams celebrate shaving 400 milliseconds off a cold start. They ship the update, pat themselves on the back, and move on. Meanwhile, actual users are staring at a spinner every single time they open a file—because the real bottleneck isn't the extension's first load. It's the fifth load, the tenth, the one that happens after a long coffee break. Cold-start metrics measure a pristine, factory-fresh scenario that almost nobody experiences in daily work. The synthetic benchmark runs on an empty cache, no other extensions competing for resources, no editor state to restore. That's a lie wrapped in a histogram.

How developers actually use extensions — hint: not from a clean slate

Watch a colleague for an hour. They open the IDE once in the morning, maybe twice. The rest of the session is a blur of tab switching, file jumping, branch checkout. The extension re-activates constantly: when the user changes a settings file, when they install a new language server, when the workspace root shifts because they opened a sub-folder. Cold-start profiling ignores all of that. What hurts is the warm-start cost—the time to re-hydrate a cached state, re-parse a config that changed by two lines, or re-evaluate a decorator that the user never even sees. I have seen an extension that booted in 150 milliseconds cold, then took 1.2 seconds to restore its tree view on every branch switch. Nobody measured that. They were too busy polishing the wrong number.

What cold-start misses: warm starts, incremental loading, user settings

The catch is that cold-start data captures disk reads and JIT compilation that only happen once per machine reboot. Real pain lives in three blind spots:

• Warm starts. The extension reactivates because a file changed or a config reloaded—but the benchmark never simulates that trigger.
• Incremental loading. Some extensions lazy-load features after the first paint. Cold-start stops the timer too early, missing the background crawl that steals frames five seconds later.
• User settings. Your test profile has three toggles. Your power user has fifteen plugins, custom keybindings, and a project with two hundred files. The cold-start path that works in isolation blows up under that weight.

Most teams skip this: they ship a 10% cold-start improvement that actually degrades warm-start behavior because they cached aggressively, bloating memory for the rest of the session. That hurts. A real-world user won't thank you for a snappy first launch if the extension stutters every time they save a file.

The Core Idea: Measure What Matters

Defining ‘startup’ in context: activation, not installation

Most teams measure extension startup by firing a stopwatch the moment the IDE window opens. That’s the wrong trigger. The user doesn’t care how long the installer ran—they care when the extension actually does something useful. I’ve seen a linter that took 400ms to cold-start but then blocked keystrokes for another 900ms because it ran its full dependency scan on first activation. That second window—time-to-interactive—is what frustrates a developer mid-flow. Cold-start hides that. Activate, don’t install. Define the clock tick at the moment the user’s first command hits your entry point.

Warm-start vs. cold-start: why it matters more

The catch is that cold-start scenarios dominate benchmarks but rarely dominate real life. A developer opens their IDE once in the morning—cold. Then they close and reopen it, or restart the language server, or reload a window, maybe a dozen times that day. Those are warm starts. V8 caches code, the OS caches file handles, your extension’s dependency graph sits ready in memory. Warm-start performance can be five to ten times faster than cold. That’s the number your users feel every time they hit Cmd+R.

We fixed this once by pre-loading a small AST parser in a deferred background thread. Cold-start barely budged—warm-start dropped from 1.2s to 0.3s. The team wanted to celebrate the cold-start improvement, but that was a mirage: the real win was shaving nearly a second off every reload. Most teams skip this: they optimize the one-time boot and ignore the repeated pain.

“The gap between cold and warm is where your users live. Profile the second open—stop guessing at the first.”

— Anonymous VS Code extension maintainer, after a painful refactor cycle

Key metrics that cut through the noise

Stop timing “extension activated.” Time time-to-interactive—when the extension actually responds to a user gesture, not when its activate() function returns. Measure perceived latency: the time between pressing a key and seeing a diagnostic underline, or between clicking a command and a menu appearing. That hurt we all feel? It’s usually 300–500ms of invisible work before the UI yields. Honest question—does your extension’s cold-start number tell you anything about that stutter? No.

Track impact on editor startup, a metric most tooling ignores. A heavy extension doesn’t just slow itself down—it delays everything in the extension host. One malformed async loop can push your language server’s init ten seconds back, and cold-start won’t whisper a word. I’ve watched a team spend two weeks shaving 50ms off a cold-start only to find their warm-start diagnostics still lagged by 1.4 seconds. Wrong target. Warm-start and perceived performance are the yardsticks that tell you if the user is tapping their fingers.

Flag this for development: shortcuts cost a day.

Flag this for development: shortcuts cost a day.

Under the Hood: What Cold-Start Actually Captures

What Exactly Happens When You Hit 'Reload'?

Most developers think cold-start means 'extension is slow.' In VS Code, the story is far more mechanical—and the gap between benchmark and reality is where teams waste weeks. Every extension registers a bundle of activationEvents inside package.json. When you open a file, the editor scans these declarations: onLanguage:typescript, onCommand:myTool.run, onStartupFinished. The extension host then queues the appropriate extension, resolves its dependencies, and finally runs the activate export. That sequence is what a cold-start profiler captures. So far, so good.

The tricky bit is what happens next—or rather, what doesn't. Cold-start metrics stop the clock the moment activate() returns. But the extension often hasn't finished setting up. I have seen extensions that populate a tree view, register decorators, or fetch remote configuration inside a setTimeout or a deferred Promise. The profiler logs '12 ms activation time.' Users experience a visible two-second lag before the sidebar populates. Wrong order. That hurts.

Activation Events: The On/Off Switch That Isn't

onLanguage:javascript sounds innocent. Open a JS file, fire the extension. But consider a linter that also provides hover info, code actions, and diagnostics. Each hook triggers separate registrations inside the activation function. The cold-start timer sees only the initial import chain; subsequent registrations are invisible. Most teams skip this: they profile in a pristine window with no other extensions. Real users have Copilot, GitLens, Prettier, and your extension fighting for the event loop. Caching? VS Code stores the parsed package.json and some module artifacts, but extension code itself is rarely cached across cold starts. Lazy loading changes the picture dramatically—one team I worked with chopped perceived load from 800 ms to 40 ms by moving heavy AST analysers into lazy import() calls. The cold-start number barely budged. The user-visible delay disappeared.

'We celebrated shaving 200 ms off the activation time. Users still complained about slowness because we ignored the deferred rendering after startup.'

— senior tooling engineer, post-mortem on a popular formatter extension

Why Caching Breaks Your Benchmarks

The extension host does cache some things: resolved module paths, compiled native modules, and the activation event graph itself. That means the second cold start in a session often runs 30 % faster than the first. Your CI runs one test—the first. Users run dozens of sessions per day. The first cold start sets the impression, but subsequent cold starts (or warm restarts) dominate real usage. onStartupFinished is another pitfall: it defers activation until the entire editor finishes loading. Your extension appears 'fast' because it hasn't started yet. The user opens a file, waits five seconds, triggers a command—and wonders why nothing happens. Cold-start metrics happily report zero activation cost. That's not measurement. That's self-deception.

I once fixed an extension where cold-start was 18 ms but the first hover action took 4.5 seconds. The fix? Move a JSON schema fetch from sync startup into a lazy onHover handler. Activation stayed 18 ms. The perceived performance flipped from 'broken' to 'instant.' So here's the rhetorical question the benchmarks never answer: does your extension actually finish starting up when activate() returns, or does it just pretend to? If you only optimise the cold-start number, you're polishing the decoy while the real slowness hides in deferred execution, network waterfalls, and UI thread contention. Measure what the user waits for—not what the profiler reports first.

A Walkthrough: Profiling a Real Extension

Setting up the profiler (VS Code Extension Host)

Open any large project—a monorepo with fifty packages will do—and launch VS Code’s built-in profiler via the command palette. Developer: Startup Performance. That single command fires up the Extension Host recorder, instrumenting every activation hook from onLanguage to onCustomEditor. Most teams skip this: they stare at the cold-start number in the status bar and call it done. Wrong move.

I pointed the profiler at the same Markdown preview extension in two runs. First run: fresh window, no cached modules, zero warm-up. Second run: same file, same workspace, but the OS hadn’t been rebooted and VS Code’s internal bindings were still hot. The discrepancy appeared before the first flame graph finished rendering.

Comparing cold-start vs. warm-start activation times

The cold-start trace reported 247 milliseconds to activate—below the generally accepted 200ms threshold? Barely above it. The warm-start trace logged 38 milliseconds. An 86% difference. That hurts.

The real gut-check came when I added a Promise.all() that prefetches language server config on first open. Cold-start jumped to 410ms. Warm-start stayed at 42ms because the promise resolved from a disk cache that hadn’t been evicted. A developer testing after lunch sees 42ms and ships the feature. End-users opening VS Code for the first time in the morning? They wait 410ms. That’s not a bug report—that’s a churn trigger.

“Warm-start is the lie we tell ourselves to feel fast. Cold-start is the truth users actually experience.”

— Lead maintainer, after reviewing instrumentation from 12,000 daily active installs

The catch is that most profiling guides—even official VS Code docs—recommend measuring in a “clean environment.” Clean means no other extensions, no lingering daemons, no cached bytecode. That’s not a real machine. That’s a laboratory.

Honestly — most development posts skip this.

Honestly — most development posts skip this.

Interpreting flame graphs and event timelines

Flame graphs from the cold-start run show a wide, squat stack: activate blocking on a require(‘vscode-languageclient’) that triggers a cascade of module resolution. The warm-start flame graph looks like a tall thin needle—barely 2% of the total time spent in vscode-languageclient because the engine already parsed and cached those modules during a prior session.

What usually breaks first is the onDidChangeConfiguration listener attached inside the activation. During cold-start, that listener fires synchronously before the editor paints, adding another 30–80ms of JSON parsing overhead the profiler conveniently tucks into the “activate” bubble. We fixed this by deferring configuration reads with a setImmediate wrapper—no change to the warm-start trace, but cold-start dropped by 17%.

One rhetorical question worth asking: would you rather optimize for the benchmark or for the Monday-morning experience? The flame graph doesn’t answer that. You do, by reading the raw event timeline and noticing that startup activation and user-first-interaction are two different spans. The profiler swallows both into one label. That’s the trap.

Most teams don’t profile at all—they rely on console.time or a performance.mark placed after activate returns. But that return doesn’t mean the extension is ready. It means the first microtask queue drained. Flame graphs reveal the truth: cold-start is a waterfall of hidden promises, warm-start is a trickle of reused handles. Ship the wrong metric and your users pay for it—every cold Monday.

Edge Cases: When Cold-Start Metrics Break Down

Extensions with network dependencies

Cold-start metrics assume your extension lives entirely on disk. That assumption shatters the moment code calls fetch() at activation. I once profiled a syntax-highlighter extension that seemed fast in every local benchmark—under 120ms cold. Then the team deployed it to users behind corporate VPNs. Startup time ballooned to 4.7 seconds. The extension was pulling grammar bundles from a CDN. The cold-start harness had cached those bundles. Real-world users didn't. The seam blows out when you treat network I/O as free.

And it gets worse. Some extensions lazy-load dependencies only after checking a remote feature flag. Cold-start profiling fires the flag request successfully—because your test machine sits on a fat pipe. Users in a coffee shop with flaky Wi‑Fi hit a timeout. Extension hangs. You measure 200ms warm. They experience ten seconds of blank editor. Trade-off: faster local numbers vs. furious bug reports from people who can't open their terminal.

User-specific configs that trigger additional loading

Here's where the numbers lie the hardest. Many extensions inspect settings.json, keybindings.json, or project-level .editorconfig files before they render anything. A cold-start environment usually ships with default configs—no custom themes, no wild snippets. That's not your user. Most teams skip this: one heavy editor.fontLigatures string can force a font atlas rebuild that adds 700ms. I have seen an extension that, on detecting a tasks.json with 40 entries, eagerly parsed every one during activation. The cold-start harness had zero tasks. Real projects had forty. The result? A 3x difference that never appears in CI dashboards.

The tricky bit is that config-driven loading is almost invisible. You only catch it if you profile with a production dotfile dumped into the test workspace. A rhetorical question for extension authors: have you ever run your startup profiler against a user's actual preference file? Most haven't. That hurts.

Cold-start metrics measure a machine. Real-world metrics measure a person's patience. They're not the same.

— paraphrased from a debugging session I attended; it stuck with me.

The impact of other extensions on startup order

Extension startup is not serial—but it isn't fully parallel either. The IDE activates extensions in dependency order. One extension that blocks on another's initialisation can cascade delays. Wrong order. Not yet. Your extension might look fast in isolation, then land third in a chain where the first extension does a synchronous file scan across a monorepo. You wait. Then your activate fires late. Cold-start metrics never model this because they most often run your extension alone. That's not how you ship.

I have seen a popular linter become the culprit solely because it registered a document-open listener before other extensions could finish their own hooks. The listener didn't do much—but its mere presence re-ordered the activation queue. Fixing it meant deferring the listener registration to a window.setTimeout after all startup phases completed. Small change. Huge difference for users with 12 extensions installed. The catch: cold-start benchmarks still showed identical numbers because they tested the extension solo. Real-world profiling must happen in a crowded IDE. Create a scratch profile with five popular extensions loaded, then measure again. The delta will shock you.

The Limits of Profiling Extensions

Tooling limitations: what the profiler can't see

Profiling an extension's startup is like debugging a black box with a single pinhole camera. You see the flame graph, the call stacks, the millisecond spikes—but you miss the dark matter. The extension host process itself? Invisible. Garbage collection pauses that randomly align with your load cycle? The profiler shrugs. I once spent three days chasing a 400ms blip that turned out to be the OS swapping out the editor's process memory—nothing the extension did, everything the scheduler chose. Worse, most profilers measure wall-clock time, not CPU time. That means a background system update, a disk queue, even a browser widget repainting can inflate your numbers. You optimise for a phantom. The seam blows out when you ship and users on different hardware get wildly different results.

Odd bit about tools: the dull step fails first.

Odd bit about tools: the dull step fails first.

The Hawthorne effect of measuring

Start attaching a profiler and your extension behaves. Seriously—it stops being itself. The act of profiling injects overhead: trace hooks, logging calls, serialised stack snapshots. That 80ms cold-start you just recorded? The extension was wearing a monitor suit, breathing into a tube. Remove the gear and the real startup behaves differently—sometimes faster, sometimes slower, but never exactly what the tool reported. The catch is we forget this. We treat profiler output as gospel, then wonder why production metrics disagree. Honest—I have shipped "optimised" extensions that performed worse in the field because the profiling session itself hid the I/O bottleneck.

What usually breaks first is the human side too. Developers, under observation, unconsciously tweak loading order, delay a module, skip a cache clear. Not maliciously—just the Hawthorne effect with keystrokes. You can't profile an extension the way users experience it, because users don't run profilers. They run fifteen other tabs and a video call and a sluggish VPN. Measuring changes what is measured.

Why no single metric tells the whole story

Cold-start metrics, warm-start metrics, lazy-load latency, total activation depth—each one lies by omission. Cold-start tells you nothing about memory fragmentation. Warm-start hides the cost of reinstating cached state. Total activation depth ignores the user input that triggers the real bottleneck five seconds later. That hurts. Most teams chase one number, flattening a multidimensional problem into a single bar chart. Wrong order. You need multiple data points—and not just from profilers.

'I stopped trusting single-metric dashboards after an extension scored green on cold-start but crashed the editor on every second file open.'

— Senior tooling engineer, internal post-mortem

A bare minimum set: cold-start p50 and p99, warm-start median, memory delta after first user interaction, and one trace from a machine under load. Vary the hardware between runs. Vary the time of day. Run without the profiler attached occasionally—use simple console timestamps, cheap but honest. Stop treating optimisation like a sprint. It's triage across data that disagrees with itself. If your extension passes one benchmark but fails three others, pick the failure that correlates with user complaints—not the one that flatters your profiler.

Frequently Asked Questions

Should I ignore cold-start entirely?

Not completely — but you should demote it. Cold-start matters for the first time someone opens your extension after a full IDE restart. That happens maybe once a day, not every few minutes. I have seen teams spend weeks shaving 200 milliseconds off cold-start while their extension still hangs for two seconds every time the user switches files. That trade-off hurts. The real question is not whether cold-start exists but whether it dominates the user's repeat experience. Most real-world workflows involve warm-start cycles: the IDE is already running, caches are warmed, the extension has been loaded before. Cold-start casts a shadow over those. Keep an eye on it, sure — just stop letting it dictate your optimization budget.

How do I reproduce warm-start conditions?

You simulate the user's second, third, and thirtieth extension activation — not the first. The trick is to disable then re-enable your extension without restarting the IDE host. Most profiling tools default to cold-start because they launch a fresh extension host process. Override that. On VS Code, run the 'Developer: Reload Window' command, then trigger your extension manually through a command palette action or a file-open event. That gives you warm-start. The catch is that some internal caches (like file watchers or language service snapshots) accumulate across sessions — you want those populated, not reset. A better approach: write a small test script that opens three files in sequence, measures activation time on the third one, and discards the first two measurements. That hurts the first run but mirrors how people actually work.

'We measured warm-start by profiling the fourth activation in a row — the first three were garbage data. It revealed a 1.8-second hidden bottleneck we'd been blind to for months.'

— Senior engineer, internal team retro on a linter extension

What's the best single metric to optimize?

Time-to-first-useful-action, not time-to-activate. The activate hook fires and your extension technically boots — but does the user see a gutter decoration, a hover popup, or a status bar update yet? Probably not. Most extensions do lazy hydration: they activate, register decorators, then wait for an actual file event to paint UI. That gap is where perceived lag lives. I once profiled a language server extension that claimed 40ms cold-start; the reality was a 1.2-second delay before any glyph appeared because the server needed a full workspace scan first. Optimize for the moment the cursor changes color or the lint error highlights — not for the console log that says 'extension loaded'. That single shift in focus killed more than half of our reported latency issues in a code-completion plugin. The catch is that this metric is harder to instrument — you need a visual checkpoint, not a timestamp. But that's exactly why it matters.

Practical Takeaways

Three things to measure besides cold-start

Most teams skip this: measure warm open and reload-after-tab-restore. Cold-start captures disk-IO and JIT compilation — useful for OS-level tuning, useless for predicting what a developer sees after their morning coffee when the IDE is already sitting in RAM. I have watched a bloated 417ms cold-start shrink to 98ms after caching, yet the extension still felt sluggish because the lazy-loaded decorator ran on every keystroke. The second blind spot is perceived latency under load. Open a 2,000-file project, then activate your extension. What happens? If your extension injects a tree view, does the render thread hitch? Profile there. Third: deactivation cost. Nobody benchmarks cleanup — and that's where memory leaks hide until the editor freezes at hour six.

Quick wins to improve perceived startup

Defer everything that's not pinned to the activate event. One concrete trick: swap eager registerCompletionProvider for lazy registration via an export map — the host only loads your provider when the user actually triggers autocomplete. The catch is that lazy activation can backfire. If your extension provides a hover provider and a code action, and both activate on the same trigger (e.g., cursor move), you end up paying the module load cost twice. Tune that with a single activation bucket. Pre-load shared imports inside a Promise.all — not inside a waterfall of import() calls. That sounds obvious. I still see production extensions calling import('./decorator') inside an onDidChangeTextDocument handler. Wrong order. The seam blows out: a cold-start of 220ms becomes a 600ms pause after the first keystroke.

‘Lazy is not always faster. Lazy is less wrong than eager — until your user triggers both features in the same second.’

— notes from a refactoring session that turned a 1.3-second startup into 340ms

When to use lazy activation vs. eager

Eager is safe when your extension is tiny — think a single Language Server client with one dependency. Anything beyond that: use activation events that map directly to user intent. onCommand for palette commands. onView for side panels. onLanguage for language features. But here is the trade-off: splitting across five activation events can inflate the total startup cost because the host rebuilds the module graph each time. We fixed this by consolidating three related features behind a single onLanguage:python event and dynamically importing the heavy path only when the user opens a Python file exceeding 500 lines. That's not a standard activation event — it's a runtime guard. The extension activates eagerly for Python, but the 40ms classifyProject call runs only after the 200th line boundary. Harder to profile. Far closer to what real-world users experience. Measure that.