Skip to content

perf(ci): Cache integration test dependency installations - #26720

Draft
nicohrubec wants to merge 13 commits into
developfrom
feat/benchmark-integration-install-timings
Draft

nicohrubec wants to merge 13 commits into
developfrom
feat/benchmark-integration-install-timings

Conversation

@nicohrubec

@nicohrubec nicohrubec commented Nov 21, 2026 •

Copy link
Copy Markdown
Member

What

Reuse integration-test dependency installations through a daily GitHub cache. Each scenario receives its own copy of the installed dependencies; missing combinations install normally, and custom setup cases keep fresh installs.

Why

Avoid repeatedly resolving and installing identical dependencies. Cache keys separate runtime versions, architectures and shards, and include a source hash and UTC date to refresh floating dependencies daily. A daily cleanup deletes integration caches dated before the current UTC day, preventing old daily snapshots from accumulating.

Average per shard across six shards per runtime:

Runtime Summed installation work: original Cold Warm Whole job: original Cold Warm
Node 20 50s 27s 4s 2:18 2:14 2:14
Node 22 74s 37s 3s 2:27 2:22 2:11
Node 24 71s 35s 3s 2:14 2:08 2:02
Node 26 65s 31s 3s 2:15 2:11 2:03
Node 27 alpha 66s 30s 3s 2:26 2:22 2:15
Bun 1.3.14 57s 21s 3s 2:07 2:05 2:00
Bun 1.4 61s 26s 3s 2:04 2:06 1:57
Bun canary 115s 32s 5s 2:57 2:47 2:42

Installation work is summed across workers and overlaps; it is included in job time and is not a wall-time saving. Whole-job times include setup, cache restoration and cleanup, excluding queues and the shared build. Single-run measurements.

Original · cold cache · warm cache. Measurements are from integration-focused benchmark runs.

nicohrubec and others added 2 commits November 21, 2026 08:49
Co-Authored-By: OpenAI Codex <codex@openai.com>
Co-Authored-By: OpenAI Codex <codex@openai.com>

@cursor cursor Bot left a comment •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Stale Bugbot comment from a previous run.

@nicohrubec nicohrubec changed the title test(ci): Benchmark integration dependency installation test(ci): Benchmark keyed integration dependency reuse Nov 21, 2026
nicohrubec and others added 2 commits November 21, 2026 09:39
Co-Authored-By: OpenAI Codex <codex@openai.com>
Co-Authored-By: OpenAI Codex <codex@openai.com>

@cursor cursor Bot left a comment •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Stale Bugbot comment from a previous run.

Co-Authored-By: OpenAI Codex <codex@openai.com>

@cursor cursor Bot left a comment •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Stale Bugbot comment from a previous run.

Comment thread .github/workflows/build.yml Resolved
Comment thread .github/workflows/build.yml Resolved
nicohrubec and others added 3 commits November 21, 2026 10:39
Co-Authored-By: OpenAI Codex <codex@openai.com>
Co-Authored-By: OpenAI Codex <codex@openai.com>
Co-Authored-By: OpenAI Codex <codex@openai.com>

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes and found 1 potential issue.

There are 2 total unresolved issues (including 1 from previous review).

Fix All in Cursor

❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.

Reviewed by Cursor Bugbot for commit 5e9421a. Configure here.

Co-Authored-By: OpenAI Codex <codex@openai.com>
@nicohrubec nicohrubec changed the title test(ci): Benchmark keyed integration dependency reuse perf(ci): Cache integration test dependency installations Nov 21, 2026
nicohrubec and others added 3 commits November 21, 2026 21:44
Co-Authored-By: OpenAI Codex <codex@openai.com>
Co-Authored-By: OpenAI Codex <codex@openai.com>
Co-Authored-By: OpenAI Codex <codex@openai.com>
@github-actions

github-actions Bot commented Nov 21, 2026 •

Copy link
Copy Markdown
Contributor

Optional E2E tests

Some optional checks were not successful. They do not block the merge. 7 not successful, 56 successful. See the run.

Check Status
❌ CI: Optional E2E Tests / E2E cloudflare-workers-send-to-sentry Test (optional) Failing after 2m Details
❌ CI: Optional E2E Tests / E2E cloudflare-pi-durable (latest) Test (optional) Failing after 43s Details
❌ CI: Optional E2E Tests / E2E nextjs-16-tunnel (cloudflare) Test (optional) Failing after 1m Details
❌ CI: Optional E2E Tests / E2E node-express-send-to-sentry (static trace lifecycle) Test (optional) Failing after 3m Details
❌ CI: Optional E2E Tests / E2E node-pi-durable (latest) Test (optional) Failing after 22s Details
❌ CI: Optional E2E Tests / E2E node-openai-decisions Test (optional) Failing after 26s Details
❌ CI: Optional E2E Tests / E2E node-openai-decisions (latest) Test (optional) Failing after 27s Details
Successful checks
Check Status
✅ CI: Optional E2E Tests / E2E cloudflare-mastra Test (optional) Successful in 47s Details
✅ CI: Optional E2E Tests / E2E cloudflare-pi-durable Test (optional) Successful in 46s Details
✅ CI: Optional E2E Tests / E2E deno (latest) Test (optional) Successful in 1m Details
✅ CI: Optional E2E Tests / E2E cloudflare-think (ai v6) Test (optional) Successful in 1m Details
✅ CI: Optional E2E Tests / E2E nextjs-14-supabase Test (optional) Successful in 2m Details
✅ CI: Optional E2E Tests / E2E cloudflare-flue-send-to-sentry Test (optional) Successful in 2m Details
✅ CI: Optional E2E Tests / E2E gen-ai-libraries Test (optional) Successful in 44s Details
✅ CI: Optional E2E Tests / E2E cloudflare-think Test (optional) Successful in 55s Details
✅ CI: Optional E2E Tests / E2E node-express-mcp-v2 Test (optional) Successful in 32s Details
✅ CI: Optional E2E Tests / E2E cloudflare-agent Test (optional) Successful in 46s Details
✅ CI: Optional E2E Tests / E2E express-5 (deno) Test (optional) Successful in 1m Details
✅ CI: Optional E2E Tests / E2E cloudflare-workers-without-tracing-send-to-sentry Test (optional) Successful in 59s Details
✅ CI: Optional E2E Tests / E2E astro-7 Test (optional) Successful in 1m Details
✅ CI: Optional E2E Tests / E2E gen-ai-libraries (cloudflare) Test (optional) Successful in 41s Details
✅ CI: Optional E2E Tests / E2E node-eve Test (optional) Successful in 1m Details
✅ CI: Optional E2E Tests / E2E nestjs-12 (deno) Test (optional) Successful in 55s Details
✅ CI: Optional E2E Tests / E2E nextjs-16 (cloudflare, latest) Test (optional) Successful in 2m Details
✅ CI: Optional E2E Tests / E2E astro-7 (static trace lifecycle) Test (optional) Successful in 1m Details
✅ CI: Optional E2E Tests / E2E express-5 (cloudflare) Test (optional) Successful in 53s Details
✅ CI: Optional E2E Tests / E2E nextjs-15 (latest) Test (optional) Successful in 3m Details
✅ CI: Optional E2E Tests / E2E nextjs-16 (vinext, bun) Test (optional) Successful in 2m Details
✅ CI: Optional E2E Tests / E2E gen-ai-libraries (latest) Test (optional) Successful in 47s Details
✅ CI: Optional E2E Tests / E2E express-5 (bun) Test (optional) Successful in 1m Details
✅ CI: Optional E2E Tests / E2E node-eve (0.x) Test (optional) Successful in 1m Details
✅ CI: Optional E2E Tests / E2E nestjs-12 (cloudflare) Test (optional) Successful in 40s Details
✅ CI: Optional E2E Tests / E2E node-flue (latest) Test (optional) Successful in 2m Details
✅ CI: Optional E2E Tests / E2E node-eve (content off) Test (optional) Successful in 1m Details
✅ CI: Optional E2E Tests / E2E debug-id-sourcemaps Test (optional) Successful in 34s Details
✅ CI: Optional E2E Tests / E2E nextjs-16 (vinext) Test (optional) Successful in 2m Details
✅ CI: Optional E2E Tests / E2E nestjs-12 (bun) Test (optional) Successful in 55s Details
✅ CI: Optional E2E Tests / E2E node-express-send-to-sentry Test (optional) Successful in 1m Details
✅ CI: Optional E2E Tests / E2E node-express-mcp-v1 Test (optional) Successful in 35s Details
✅ CI: Optional E2E Tests / E2E nuxt-5 Test (optional) Successful in 1m Details
✅ CI: Optional E2E Tests / E2E nuxt-5-sourcemaps (static / nuxt generate) Test (optional) Successful in 32s Details
✅ CI: Optional E2E Tests / E2E node-flue Test (optional) Successful in 1m Details
✅ CI: Optional E2E Tests / E2E nuxt-5 (development) Test (optional) Successful in 1m Details
✅ CI: Optional E2E Tests / E2E react-send-to-sentry Test (optional) Successful in 3m Details
✅ CI: Optional E2E Tests / E2E node-mastra Test (optional) Successful in 1m Details
✅ CI: Optional E2E Tests / E2E nuxt-5-cloudflare Test (optional) Successful in 27s Details
✅ CI: Optional E2E Tests / E2E node-eve (orchestrion) Test (optional) Successful in 1m Details
✅ CI: Optional E2E Tests / E2E react-router-7-framework-instrumentation Test (optional) Successful in 2m Details
✅ CI: Optional E2E Tests / E2E nuxt-5-sourcemaps (client source maps kept) Test (optional) Successful in 38s Details
✅ CI: Optional E2E Tests / E2E vue-3 (no Options API) Test (optional) Successful in 48s Details
✅ CI: Optional E2E Tests / E2E node-fastify-6 Test (optional) Successful in 36s Details
✅ CI: Optional E2E Tests / E2E nextjs-15 (turbo) Test (optional) Successful in 2m Details
✅ CI: Optional E2E Tests / E2E nuxt-5-tunnel (cloudflare) Test (optional) Successful in 28s Details
✅ CI: Optional E2E Tests / E2E solidstart-2-tunnel Test (optional) Successful in 37s Details
✅ CI: Optional E2E Tests / E2E node-mastra (latest) Test (optional) Successful in 1m Details
✅ CI: Optional E2E Tests / E2E tanstackstart-react (latest) Test (optional) Successful in 1m Details
✅ CI: Optional E2E Tests / E2E tanstackstart-react-sourcemaps Test (optional) Successful in 31s Details
✅ CI: Optional E2E Tests / E2E nuxt-5-tunnel Test (optional) Successful in 49s Details
✅ CI: Optional E2E Tests / E2E node-pi-durable Test (optional) Successful in 31s Details
✅ CI: Optional E2E Tests / E2E tanstackstart-react-cloudflare Test (optional) Successful in 36s Details
✅ CI: Optional E2E Tests / E2E nuxt-5-sourcemaps Test (optional) Successful in 30s Details
✅ CI: Optional E2E Tests / E2E solidstart-2 Test (optional) Successful in 51s Details
✅ CI: Optional E2E Tests / E2E nuxt-5 (kv-cache) Test (optional) Successful in 1m Details

Co-Authored-By: OpenAI Codex <codex@openai.com>
@github-actions

Copy link
Copy Markdown
Contributor

size-limit report 📦

Compared: cce5e67 (PR test merge using develop at 5a39728)

Against: 5a39728 (develop) · base run #67367

✅ Baseline matches: this diff contains only this PR's changes.

Path Size % Change Change
@sentry/browser 30.23 kB - -
@sentry/browser - with treeshaking flags 28.43 kB - -
@sentry/browser - with treeshaking flags tracing without tracing 28.33 kB - -
@sentry/browser (incl. Tracing) 52.81 kB - -
@sentry/browser (incl. Tracing Core) 40.33 kB - -
@sentry/browser (incl. Tracing + Span Streaming) 52.81 kB - -
@sentry/browser (incl. Tracing, Profiling) 55.69 kB - -
@sentry/browser (incl. Tracing, Replay) 92.88 kB - -
@sentry/browser (incl. Tracing, Replay) - with treeshaking flags 81.7 kB - -
@sentry/browser (incl. Tracing, Replay with Canvas) 97.22 kB - -
@sentry/browser (incl. Tracing, Replay, Feedback) 110.8 kB - -
@sentry/browser (incl. Feedback) 48.17 kB - -
@sentry/browser (incl. sendFeedback) 30.81 kB - -
@sentry/browser (incl. FeedbackAsync) 35.84 kB - -
@sentry/browser (incl. Metrics) 31.55 kB - -
@sentry/browser (incl. Logs) 31.83 kB - -
@sentry/browser (incl. Metrics & Logs) 32.49 kB - -
@sentry/react 32.24 kB - -
@sentry/react (incl. Tracing) 55.45 kB - -
@sentry/vue 38.45 kB - -
@sentry/vue (incl. Tracing) 55.75 kB - -
@sentry/svelte 30.24 kB - -
@sentry/remix (Remix 3 client bundle) 58.46 kB - -
CDN Bundle 32.16 kB - -
CDN Bundle (incl. Tracing) 53.19 kB - -
CDN Bundle (incl. Logs, Metrics) 34.95 kB - -
CDN Bundle (incl. Tracing, Logs, Metrics) 55.76 kB - -
CDN Bundle (incl. Replay, Logs, Metrics) 76.02 kB - -
CDN Bundle (incl. Tracing, Replay) 91.1 kB - -
CDN Bundle (incl. Tracing, Replay, Logs, Metrics) 93.71 kB - -
CDN Bundle (incl. Tracing, Replay, Feedback) 96.51 kB - -
CDN Bundle (incl. Tracing, Replay, Feedback, Logs, Metrics) 99.07 kB - -
CDN Bundle - uncompressed 93.91 kB - -
CDN Bundle (incl. Tracing) - uncompressed 157.26 kB - -
CDN Bundle (incl. Logs, Metrics) - uncompressed 102.24 kB - -
CDN Bundle (incl. Tracing, Logs, Metrics) - uncompressed 164.88 kB - -
CDN Bundle (incl. Replay, Logs, Metrics) - uncompressed 232.48 kB - -
CDN Bundle (incl. Tracing, Replay) - uncompressed 277.53 kB - -
CDN Bundle (incl. Tracing, Replay, Logs, Metrics) - uncompressed 285.16 kB - -
CDN Bundle (incl. Tracing, Replay, Feedback) - uncompressed 289.46 kB - -
CDN Bundle (incl. Tracing, Replay, Feedback, Logs, Metrics) - uncompressed 297.08 kB - -
@sentry/nextjs (client) 59.15 kB - -
@sentry/sveltekit (client) 54.06 kB - -
@sentry/core/server 45.28 kB - -
@sentry/core/browser 14.41 kB - -
@sentry/node 155.23 kB +0.01% +1 B 🔺
@sentry/node/import (ESM hook with diagnostics-channel injection) 83.46 kB - -
@sentry/node - without tracing 88.92 kB +0.01% +1 B 🔺
@sentry/node - without channel injection 144.15 kB -0.01% -5 B 🔽
@sentry/aws-serverless 99.55 kB -0.01% -5 B 🔽
@sentry/cloudflare (withSentry) - minified 208.88 kB - -
@sentry/cloudflare (withSentry) 484.65 kB - -
@sentry/cloudflare (withSentry, instrumented CommonJS package) - minified 1.02 MB - -
@sentry/nextjs/cloudflare (withSentry) - minified 234.11 kB - -

View base workflow run

// before a retry gets a chance to run.
const NPM_INSTALL_TIMEOUT_MS = 60_000;

async function npmInstallWithRetry(cwd: string, deps: string[]): Promise<void> {

@nicohrubec nicohrubec Nov 21, 2026 •

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this method was just moved as is to this new file

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant