Repository navigation
Conversation
Contributor
There was a problem hiding this comment.
🟡 Changes recommended
A truncated response can raise an uncaught IncompleteRead, aborting the probe instead of retrying or trying the next endpoint; retry handling and a regression test are needed.
1 open issue
What changed in this PR
This PR makes Bazel’s Git LFS batch-request helper more resilient to transient failures while preserving authentication and the existing download/cache path.
Changes:
- Add bounded retries with exponential backoff and
Retry-Afterhandling. - Preserve the batch payload across endpoint fallback and avoid sensitive exception details in diagnostics.
- Add mocked regression tests and a Bazel test target.
| File | Description |
|---|---|
misc/bazel/internal/test_git_lfs_probe.py |
Tests retries, delays, fallback, authentication reuse, and safe diagnostics. |
misc/bazel/internal/git_lfs_probe.py |
Adds retry handling and preserves request data during fallback. |
misc/bazel/internal/BUILD.bazel |
Registers the Python regression suite as a Bazel test. |
Review effort: Balanced (adaptive)
Bound retries for transient batch-request errors, respect Retry-After, and preserve authenticated endpoint fallback without logging sensitive response details. Handle truncated response bodies as well as connection errors: json.load can raise IncompleteRead, which is not a ConnectionError. Cover recovery and exhausted retries using in-memory HTTP responses alongside the other deterministic regressions. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
josefs
force-pushed
the
josefs/lfs-transport-hardening
branch
from
November 20, 2026 17:03
247b106 to
ea6608d
Compare
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.

Summary
This is an attempt at deflaking CI. With some copilot investigation, it's clear that LFS connection failure is a common recurring problem. This PR adds retries in an attempt at hardening the code. Specifically it retries timeouts, connection errors (including
RemoteDisconnected), and HTTP 408/429/500/502/503/504.Evidence
Pre-change CI baseline
Seven complete UTC days: November 13–19, 2026 (
2026-11-13T00:00:00Zinclusive to2026-11-20T00:00:00Zexclusive).Automated transient-failure reports recorded 11 distinct workflow runs failing at the LFS batch/probe step. Each was confirmed against its original first-attempt job log.
RemoteDisconnectedCounting rule: select reports by their creation timestamp in the UTC window, then count each workflow run once. Exclude failures downloading the actual LFS archives, failures in other services, and duplicate downstream errors. These are counts of recorded failures, not an exhaustive CI census or a failure rate. Ten of the eleven were reported on November 17.
For the after comparison, repeat the same collection over seven complete UTC days once CI is using the landed helper. Baseline collected November 20, 2026; per-run evidence retained separately.
Regression tests
origin/main:Ran 8 tests,FAILED (failures=4, errors=12)including parameterized cases. Disconnects/timeouts escape, rate limits are not retried, and malformed-response fallback sends the wrong payload.After: The same suite passes all 8 tests, including
Retry-Afterseconds/dates, malformed and expired headers, the 30s/31s boundary, exhaustion, permanent errors, authenticated payload reuse, and safe diagnostics.python3 -m unittest discover -s misc/bazel/internal -p 'test_git_lfs_probe.py' -v— passes.bazel test //misc/bazel/internal:test_git_lfs_probe --jobs=2 --loading_phase_threads=2 --test_output=errors— 1/1 test target passes.git diff --check— clean.Network, Git subprocesses, and time are mocked. No live LFS service test or full CodeQL build was run.
Merge Danger
Door: two-way
Blast Radius: Builds
Only the LFS batch-request helper changes. Transient failures can incur bounded extra attempts and waits before endpoint fallback; cached downloads are unaffected.