Skip to content

Run cloud integration tests with -parallel 8 - #7871

Merged
denik merged 1 commit into
mainfrom
denik/integration-parallel-8
Nov 22, 2026
Merged

denik merged 1 commit into
mainfrom
denik/integration-parallel-8

Conversation

@denik

@denik denik commented Nov 21, 2026

Copy link
Copy Markdown
Contributor

Changes

Raise -parallel from 4 to 8 in the integration and integration-short tasks, the ones the cloud CI runs on main.

Why

The cloud acceptance package hits the 5h20m go test -timeout (set in #7749) on aws, azure and gcp, linux and windows, when many main runs overlap. No test hangs. The tests still running at the timeout differ in every failure, and they keep moving through the package (bundle/invariant/, resources/postgres_, volumes, apps, ...). The package is saturated:

  • The tests mostly wait on the workspace, so the package's wall time is about its total test time divided by -parallel. 2-core linux and 4-core windows runners take about the same time.
  • In a full scheduled run, the acceptance leaf tests add up to ~18.9h of test time. The package took 4h52m at -parallel 4.
  • In a quiet run, the same tests are 1.4x faster (median): ~10.3h of test time for the short suite. When ~25 main runs overlapped on 11-19, tests ran ~2x slower, and both the short and the full suite went past 5h20m.

At -parallel 8 the same work finishes in roughly half the time, which leaves margin under the timeout even when the workspace is slow.

Judgment call: 8 rather than a bigger jump. It doubles each job's load on the shared test workspaces, which already serve 4 tests per job. The cloud-select task (PR runs, at most 50 tests, 2h timeout) is unchanged.

Tests

Ran TestAccept/bundle/invariant/no_drift (180 tests) on aws with -parallel 8. It finished in 8m47s. At -parallel 4, these tests add up to ~23 min in a quiet CI run. 152 passed. The 28 cluster/job-cluster variants failed in my local environment only, on a missing instance pool (Can't find an instance pool with id: [ID]. (404 RESOURCE_DOES_NOT_EXIST)), which is unrelated to parallelism.

Dashboard

  • error/3809: 74 jobs in 39 main runs, 11-18 to 11-19.
Package github.com/databricks/cli/acceptance ended after 5h20m with no test reporting a failure.
The test binary printed: panic: test timed out after 5h20m0s
Still running (4): TestAccept/bundle/resources/volumes/recreate/DMS=, TestAccept/bundle/resources/volumes/rename_case_only/DMS=true, TestAccept/bundle/run/app-with-job/DMS=, TestAccept/bundle/run/app-with-job/DMS=true

AI agent (opus5.5, isaac_claude)

This pull request and its description were written by Isaac.

The cloud acceptance package hit its 5h20m go test timeout on all clouds
when many main runs overlapped: the tests wait on the workspace, so the
package's wall time is its total test time divided by -parallel, and at
-parallel 4 the full suite (~19h of test time) needs ~4h50m.

Co-authored-by: Isaac <no-reply@databricks.com>
@eng-dev-ecosystem-bot

Copy link
Copy Markdown
Collaborator

✅ Integration test report

Commit: b45c609

Run: 44884462367

Env ✅​pass 🙈​skip Time
✅​ aws linux-2core-8gb 273 19 3:49
✅​ aws-windows-latest-4core-16gb 275 17 4:23
✅​ azure linux-2core-8gb 272 19 3:36
✅​ azure-windows-latest-4core-16gb 274 17 3:25
✅​ gcp linux-2core-8gb 273 19 3:37
✅​ gcp-windows-latest-4core-16gb 275 17 3:24
Top 3 slowest tests (at least 2 minutes):
duration env testname
4:21 aws-windows-latest-4core-16gb TestAccept
3:24 azure-windows-latest-4core-16gb TestAccept
3:22 gcp-windows-latest-4core-16gb TestAccept

@denik
denik marked this pull request as ready for review November 21, 2026 08:41
@denik
denik requested a review from a team as a code owner November 21, 2026 08:41

@review-oss-bot review-oss-bot Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Automated review at b45c609. Each finding was verified against the code.

No blocking issues found.

This PR bumps -parallel from 4 to 8 in the integration and integration-short Taskfile tasks to reduce wall-clock time for cloud CI runs that were hitting the 5h20m go test timeout. The change is minimal and mechanical (two token edits plus a clarifying comment), and the code does exactly what it claims. No defects found.

@denik
denik added this pull request to the merge queue Nov 22, 2026
Merged via the queue into main with commit eba8f01 Nov 22, 2026
42 checks passed
@denik
denik deleted the denik/integration-parallel-8 branch November 22, 2026 20:18
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants