diff --git a/.github/workflows/bazel-maven.yml b/.github/workflows/bazel-maven.yml index c4a320fac..dfb5e1475 100644 --- a/.github/workflows/bazel-maven.yml +++ b/.github/workflows/bazel-maven.yml @@ -6,8 +6,9 @@ name: Bazel patch compatibility (Maven) # patch is one block in MODULE.bazel that puts a repository with the # patched jar in the place of the hub's own (`override_repo`): the flows # hold the block's text, the patched build and its sha256 check, the undo, -# a version bump of the patched artifact, and a hub the block cannot -# patch. bazel-compatibility.yml holds the legs and what every ecosystem +# a hosted scan that finds the artifact in the hub's lock, a version bump +# of the patched artifact, and a hub the block cannot patch. +# bazel-compatibility.yml holds the legs and what every ecosystem # shares; this file says when the Maven legs run and on which releases. # # Each major release has an era of its own, because the ruleset decides diff --git a/crates/socket-patch-cli/CLI_CONTRACT.md b/crates/socket-patch-cli/CLI_CONTRACT.md index a19a36417..8505d1920 100644 --- a/crates/socket-patch-cli/CLI_CONTRACT.md +++ b/crates/socket-patch-cli/CLI_CONTRACT.md @@ -379,7 +379,7 @@ Pinned by `tests/scan/scan_proxy_fallback_e2e.rs`, `tests/cli/api_client_errors_ **Throttling: bounded retry, then a reported failure.** Every patch-API JSON call (the batch query, the per-package patch lists, patch views and VEX record fetches, hosted package references) retries an HTTP `429` or `503` answer up to 3 times (`SOCKET_API_MAX_RETRIES=`, `0`-`10`; `0` = no retry). The wait honors `Retry-After` (delta-seconds or HTTP-date); a `Retry-After` over 30 s is not waited out — the answer is final at once — and one under the jittered first backoff step (`0`, a past date) waits that step instead. Without one it backs off 0.5 s / 1 s / 2 s (each step up to 8 s, with jitter in its upper half). All retries in one run share a 60 s wall-clock window that opens with the run's first retry: a retry whose wait would end after it closes is refused and the answer is final. Parallel requests wait in parallel, so each still gets its retries while the run adds at most about 60 s. Nothing else is retried (401/403 still drive the proxy fallback on the first answer; the public proxy's permanent `503 "Patch API is not configured"` is never retried on any path — the batch query still degrades to the per-package path at once, and a per-package lookup or patch view answering it is the same non-throttle failure it always was, which on the legacy per-package path leaves that package unanswered, see above), and a retried answer folds exactly where the first attempt's would have, so output is identical to an unthrottled run's. A request still throttled after that is a failure in the channel its siblings use: a failed batch is the human `Warning: API batch of failed: ` line and, under `--json`, a run-level `warnings[]` entry `{code: "api_batch_failed", detail: "API batch of failed: "}` (additive; `status` stays `success`, exit 0 — the other batches' packages are reported); a failed per-package patch-list query in the agent / hosted / vendored flows is the human `Warning: could not fetch details for : ` line and, under `--json`, `{code: "patch_details_failed", detail: "could not fetch details for : "}`. When every batch (or every patch-list query) fails, the existing all-failed error envelope and exit 1 apply. The error names the exhausted retry: `Rate limit exceeded (HTTP 429, gave up after 3 retries). Please try again later.` / `API request failed with status 503: (gave up after 3 retries)` (or `(Retry-After s exceeds the 30 s retry cap)` / `(the run's 60 s retry window has closed)`); with retries off it is the pre-retry text. On the token-less legacy per-package proxy path (a proxy without `POST /patch/batch`), a package still throttled (429 / over-capacity 503) after its retries is an unanswered package (see "No patches" needs an answer above): the batch's answered packages are used and it is reported as `patch_lookup_failed` (`partialFailure`, exit 1); when it leaves no package of its batch answered, the batch fails through the batch-failure channel above. Only a PURL the service itself refuses (its own `400`) is skipped individually. Pinned by `tests/scan_api_retry_e2e.rs` and the core crate's `tests/api_retry_e2e.rs`. -**Lockfile supplement (v3.4)**: `scan` discovery is no longer limited to installed trees. The project's lockfiles (`package-lock.json`/`npm-shrinkwrap.json`, `pnpm-lock.yaml` v9, `yarn.lock` classic + berry, `bun.lock`, `vlt-lock.json` (registry nodes, Socket-hosted pins included; vendored `file` nodes are left to the vendor ledger), `Cargo.lock`, `go.sum`, `composer.lock`, `Gemfile.lock`, `uv.lock`/`poetry.lock`/pinned `requirements.txt`) are inventoried and dependencies with NO installed copy join discovery — counts, the API lookup, the rows of what a hosted or vendored run patches (an agent or report-only run, which patches installed copies, gets a stderr note that they are not installed), and the prune "scanned" set (a wiped node_modules no longer prunes lockfile-listed entries). JSON gains a top-level `lockfileOnlyPackages` count and an additive `notInstalled: true` on matching `packages[]` entries. Agent mode partitions lockfile-only patches out BEFORE download (calm `skipped`/`package_not_installed` events — never an error exit, never a manifest write); vendored mode passes them through to the vendor engine's server download. Vendored-ledger entries likewise stay discoverable on a fresh clone (the committed artifact is the dependency). The `Gemfile.lock` inventory lists a gem by its ruby-platform entry; a gem the lock resolves ONLY as precompiled platform builds (`nokogiri (1.18.10-arm64-darwin)`, no `nokogiri (1.18.10)`) joins the supplement of a hosted run alone (`scan` / `get --mode hosted`, and the in-memory hosted engine), which can redirect it to its per-platform prebuilds from the lock; agent and vendored runs see it only through an installed copy, as before. Global scans (`--global`) get no supplement. **Rush monorepos** (no root lockfile, `rush.json` present): the npm-lock inventory falls back to the Rush source-of-truth locks — `common/config/rush/pnpm-lock.yaml` plus every `common/config/subspaces/*/pnpm-lock.yaml` (`read_dir`-sorted, repo-relative paths preserved) — so a Rush repo's dependencies still join discovery. **Plug'n'Play layouts are an explicit refusal, not an empty inventory**: a `.pnp.*` loader means the npm packages are structurally unreachable in EVERY mode (under yarn PnP the installed-tree crawl is empty too — no `node_modules/`), so `scan` surfaces an additive top-level `warnings[]` array (`{code, detail}` objects, omitted when empty) carrying `yarn_pnp_unsupported` (same code as apply's refusal; remedy `yarn patch `) or `pnpm_pnp_unsupported` (pnpm's `node-linker=pnp` twin; pnpm remedies), plus a stderr `Warning: …` line on the human path. Exit code and `status` are deliberately unchanged (exit 0 / `success` — the same posture as hosted refusals, which exit 0 with no `applied` event); the warning is the machine-readable signal that nothing was checked. Pinned by `tests/e2e_safety_yarn_pnp.rs`. **A Bundler lock socket-patch cannot read is likewise a warning, not an empty inventory**: when the project holds gem files but bundler loads no `Gemfile.lock` / `gems.locked` socket-patch reads (`BUNDLE_GEMFILE` or bundler 4's `BUNDLE_LOCKFILE` naming another file, or a `Gemfile` + `gems.rb` twin, whose loaded pair depends on the bundler major that runs), `warnings[]` carries the additive `gem_lock_unsupported` (detail names the setting or the twin) and its lockfile-only gems are not discovered. Same exit-0 / `success` posture. +**Lockfile supplement (v3.4)**: `scan` discovery is no longer limited to installed trees. The project's lockfiles (`package-lock.json`/`npm-shrinkwrap.json`, `pnpm-lock.yaml` v9, `yarn.lock` classic + berry, `bun.lock`, `vlt-lock.json` (registry nodes, Socket-hosted pins included; vendored `file` nodes are left to the vendor ledger), `Cargo.lock`, `go.sum`, `composer.lock`, `Gemfile.lock`, `uv.lock`/`poetry.lock`/pinned `requirements.txt`) are inventoried and dependencies with NO installed copy join discovery — counts, the API lookup, the rows of what a hosted or vendored run patches (an agent or report-only run, which patches installed copies, gets a stderr note that they are not installed), and the prune "scanned" set (a wiped node_modules no longer prunes lockfile-listed entries). JSON gains a top-level `lockfileOnlyPackages` count and an additive `notInstalled: true` on matching `packages[]` entries. Agent mode partitions lockfile-only patches out BEFORE download (calm `skipped`/`package_not_installed` events — never an error exit, never a manifest write); vendored mode passes them through to the vendor engine's server download. Vendored-ledger entries likewise stay discoverable on a fresh clone (the committed artifact is the dependency). The `Gemfile.lock` inventory lists a gem by its ruby-platform entry; a gem the lock resolves ONLY as precompiled platform builds (`nokogiri (1.18.10-arm64-darwin)`, no `nokogiri (1.18.10)`) joins the supplement of a hosted run alone (`scan` / `get --mode hosted`, and the in-memory hosted engine), which can redirect it to its per-platform prebuilds from the lock; agent and vendored runs see it only through an installed copy, as before. Global scans (`--global`) get no supplement. **Rush monorepos** (no root lockfile, `rush.json` present): the npm-lock inventory falls back to the Rush source-of-truth locks — `common/config/rush/pnpm-lock.yaml` plus every `common/config/subspaces/*/pnpm-lock.yaml` (`read_dir`-sorted, repo-relative paths preserved) — so a Rush repo's dependencies still join discovery. **Plug'n'Play layouts are an explicit refusal, not an empty inventory**: a `.pnp.*` loader means the npm packages are structurally unreachable in EVERY mode (under yarn PnP the installed-tree crawl is empty too — no `node_modules/`), so `scan` surfaces an additive top-level `warnings[]` array (`{code, detail}` objects, omitted when empty) carrying `yarn_pnp_unsupported` (same code as apply's refusal; remedy `yarn patch `) or `pnpm_pnp_unsupported` (pnpm's `node-linker=pnp` twin; pnpm remedies), plus a stderr `Warning: …` line on the human path. Exit code and `status` are deliberately unchanged (exit 0 / `success` — the same posture as hosted refusals, which exit 0 with no `applied` event); the warning is the machine-readable signal that nothing was checked. Pinned by `tests/e2e_safety_yarn_pnp.rs`. **A Bundler lock socket-patch cannot read is likewise a warning, not an empty inventory**: when the project holds gem files but bundler loads no `Gemfile.lock` / `gems.locked` socket-patch reads (`BUNDLE_GEMFILE` or bundler 4's `BUNDLE_LOCKFILE` naming another file, or a `Gemfile` + `gems.rb` twin, whose loaded pair depends on the bundler major that runs), `warnings[]` carries the additive `gem_lock_unsupported` (detail names the setting or the twin) and its lockfile-only gems are not discovered. Same exit-0 / `success` posture. **The jars of a pinned rules_jvm_external hub join the supplement of a hosted run at the root of a Bazel workspace (additive)**: `scan --mode hosted` (the inferred default included) and `get --mode hosted` take, for each hub of the root `MODULE.bazel` that names a lock (`maven_install.json`), the jar without a classifier of each artifact the hub fetches, at the lock's version, as a Maven package of the project: it counts in `scannedPackages`, and in `lockfileOnlyPackages` with `notInstalled: true` unless a crawl found that version installed, and a patch found for it is pinned in the block of `MODULE.bazel` (the `maven_bazel_*` rows below). A jar whose repository has no known name is such a package too (a patch found for it gets `maven_bazel_artifact`). No jar is guessed for a hub that was not read with certainty: `warnings[]` then carries `maven_bazel_hub_not_read` once, and the jars of the hubs that were read are still looked up. A patch of a hub's jar is a patch of the run like any other, so the run writes for it what `get --mode hosted` writes in that checkout: beside a Gradle build the hosted Gradle pin too (`settings.gradle` and the files under `.socket/gradle/`, with that pin's warnings), and beside a `pom.xml` what the pom lane pins or says (`redirect_maven_version_not_selected`, `redirect_maven_selection_undetermined`, which speak of `pom.xml` alone). **A stale entry of the block is taken out by `scan --mode hosted` (additive)**: when every hub was read and none fetches the jar of an entry at its version any more (the lock moved on, so Bazel stops at the entry), the scan restores it before the rewrite as the withdrawn-pin sweep above restores a pin: the entry alone, under the apply lock, `--dry-run` previewing it, one `rolledBack` event (`verified` on a dry run) with `uuid`, `errorCode: "maven_bazel_entry_stale"` and `details.mode: "hosted"`, also on a scan that finds no package; a pin of the same patch in a `pom.xml` or Gradle file stays; a restore that fails is the `redirect_withdrawn_restore_failed` warning. `get` takes no stale entry out. No other mode, no run whose `--ecosystems` leaves Maven out, no `--global` run and no run of a project below the Bazel root takes a package from a hub, gives that warning or takes a stale entry out: its packages are those of the run without the workspace. A run of a project in no Bazel workspace is, byte for byte, the run it was. See docs/ecosystems.md "Bazel workspaces: Maven". **Server artifact acquisition (v5.0)**: vendoring downloads the patched artifact for the selected UUID, including on a fresh checkout with no installed package. The CLI no longer downloads pristine packages, stages patch blobs, applies patches to vendor copies, or constructs archives. Backend lock and package-identity checks still run before acquisition; git and custom-registry Cargo sources are refused with `vendor_source_unsupported`. Healthy committed artifacts are reused offline. A changed UUID downloads a fresh server artifact; an unhealthy artifact at the recorded UUID uses the exact redownload procedure below, except that a directory copy whose ledger entry has no file inventory (vendored before v5.0) is rebuilt by its backend from a fresh verified download, and a missing or stale Bun workspace mirror over a healthy canonical tarball is rewritten from that tarball by the backend; those cases report the backend's own failure codes. @@ -395,7 +395,7 @@ Pinned by `tests/scan/scan_proxy_fallback_e2e.rs`, `tests/cli/api_client_errors_ **Path-scoped scans (`scan [PATHS]...`, v5.0)**: what a PATH means depends on the mode. -**A bare hosted or vendored scan covers the checkout (additive/MINOR).** With no PATH, `scan` in hosted or vendored mode (the inferred default included) scans `--cwd` and every other project directory below it in the same git checkout, exactly as if each had been named as a PATH: its own lockfiles, ledgers, `.socket/` and inferred mode, one `== ==` heading each (`== . ==` for `--cwd` itself), the worst exit code, one `--max-new-patches` budget. A project directory is one the in-memory engine would detect as a root for the same tree: a directory holding a lockfile of a supported npm-family, PyPI, Cargo, RubyGems, Go or Composer format (a manifest alone never makes one; Maven, Gradle, sbt and NuGet projects are scanned only as `--cwd` or a named PATH), found by a walk that honours git's ignore rules (`.gitignore`, the repository's and the user's excludes; a `.ignore` file, which hides files from search tools and not from git, is not read), never follows a link, never enters `node_modules`, `.git`, `.socket`, `.yarn` or `vendor`, and never enters another repository: a directory below `--cwd` that holds a `.git` entry (a submodule, a nested clone) is left to a run of its own, since this checkout does not track its files and its own `socket.yml` was not read. A workspace member whose only lock is one the workspace root's run reads is not a project of its own: a Cargo workspace member's `Cargo.lock`, and a pnpm workspace member's `pnpm-lock.yaml` the root accounts for. Discovered directories are subject to the `socket.yml` path filters and the built-in `test/`, `tests/`, `fixtures/`, `__fixtures__/` and `testdata/` ignores, as glob-matched PATHs are; the ones left out are counted in one stderr note (`Note: N project directories under ignored paths ... were not scanned; name a directory to scan it.`), and naming a directory scans it. `--cwd` itself is scanned whenever it is a project, with or without a lockfile, as before. Nothing is searched for, and the run is the single-directory scan it always was, when: `--cwd` is not inside a git checkout (a scan from a home directory never goes looking for projects), a PATH is given, the scope is `--global`/`--global-prefix`, or the mode is agent (an agent-mode scan still names the directories it left; see below). `--json` and `--vex` report on one project directory, so with either a bare scan scans `--cwd` alone and adds the run-level `projects_not_scanned` warning naming the project directories it found and left (`N project directories below this one were not scanned (a, b): --json takes one project directory. Run one scan per directory, e.g. ...`); exit code and status are unchanged. A bare agent-mode scan patches the installed packages its crawl of `--cwd` reads, so it covers the project at `--cwd` alone. It looks for the same project directories below `--cwd` and adds the `projects_not_scanned` warning (stderr `Warning:`, `--json` `warnings[]`) naming the ones whose installed packages that crawl did not read, and the command that reaches the first, asked as the run was (`scan --mode agent`, `scan --sync`, or a bare `scan` when the mode was inferred): `N project directories below this one were not scanned (a, b): an agent-mode scan patches the packages installed for the one project directory it runs in. Run one scan per directory, e.g. `socket-patch scan --mode agent --cwd a``. More than 10 directories are listed as the first 10 `, and N more`. Exit code and status are unchanged. A directory is named when it has an install directory of its own that the crawl from `--cwd` did not read: a `node_modules` the npm walk skips (one under a hidden or build-output directory), a Python environment or a Composer vendor directory inside the project, or a gem home (the project's `vendor/bundle`, or the user-wide homes Bundler installs into by default when `--cwd` holds no Bundler project to read them for). A directory is not named when the run did read its packages (every other `node_modules` below `--cwd`, a gem home or an activated environment both share) or when it has no install directory of its own; the user-wide Python, Cargo and Go stores are never compared. Nothing is looked for with a PATH, a global scope, a non-default `--manifest-path` or outside a git checkout. With nothing found at `--cwd`, a run that names such directories prints `No packages found (see the warning).` in place of the hint to run the package manager's install. A tree too large to search adds a `projects_below_not_searched` warning: some directories may not be named. A tree with more than 500,000 entries is searched only that far, with a stderr warning. +**A bare hosted or vendored scan covers the checkout (additive/MINOR).** With no PATH, `scan` in hosted or vendored mode (the inferred default included) scans `--cwd` and every other project directory below it in the same git checkout, exactly as if each had been named as a PATH: its own lockfiles, ledgers, `.socket/` and inferred mode, one `== ==` heading each (`== . ==` for `--cwd` itself), the worst exit code, one `--max-new-patches` budget. A project directory is one the in-memory engine would detect as a root for the same tree: a directory holding a lockfile of a supported npm-family, PyPI, Cargo, RubyGems, Go or Composer format (a manifest alone never makes one; Maven, Gradle, sbt and NuGet projects are scanned only as `--cwd` or a named PATH), found by a walk that honours git's ignore rules (`.gitignore`, the repository's and the user's excludes; a `.ignore` file, which hides files from search tools and not from git, is not read), never follows a link, never enters `node_modules`, `.git`, `.socket`, `.yarn` or `vendor`, and never enters another repository: a directory below `--cwd` that holds a `.git` entry (a submodule, a nested clone) is left to a run of its own, since this checkout does not track its files and its own `socket.yml` was not read. A workspace member whose only lock is one the workspace root's run reads is not a project of its own: a Cargo workspace member's `Cargo.lock`, and a pnpm workspace member's `pnpm-lock.yaml` the root accounts for. Discovered directories are subject to the `socket.yml` path filters and the built-in `test/`, `tests/`, `fixtures/`, `__fixtures__/` and `testdata/` ignores, as glob-matched PATHs are; the ones left out are counted in one stderr note (`Note: N project directories under ignored paths ... were not scanned; name a directory to scan it.`), and naming a directory scans it. `--cwd` itself is scanned whenever it is a project, with or without a lockfile, as before; in hosted mode it also is one as the root of a Bazel workspace whose rules_jvm_external hubs give the scan packages, a `maven_bazel_hub_not_read` warning or a stale entry to take out (the lockfile supplement above), whatever other file it holds. Nothing is searched for, and the run is the single-directory scan it always was, when: `--cwd` is not inside a git checkout (a scan from a home directory never goes looking for projects), a PATH is given, the scope is `--global`/`--global-prefix`, or the mode is agent (an agent-mode scan still names the directories it left; see below). `--json` and `--vex` report on one project directory, so with either a bare scan scans `--cwd` alone and adds the run-level `projects_not_scanned` warning naming the project directories it found and left (`N project directories below this one were not scanned (a, b): --json takes one project directory. Run one scan per directory, e.g. ...`); exit code and status are unchanged. A bare agent-mode scan patches the installed packages its crawl of `--cwd` reads, so it covers the project at `--cwd` alone. It looks for the same project directories below `--cwd` and adds the `projects_not_scanned` warning (stderr `Warning:`, `--json` `warnings[]`) naming the ones whose installed packages that crawl did not read, and the command that reaches the first, asked as the run was (`scan --mode agent`, `scan --sync`, or a bare `scan` when the mode was inferred): `N project directories below this one were not scanned (a, b): an agent-mode scan patches the packages installed for the one project directory it runs in. Run one scan per directory, e.g. `socket-patch scan --mode agent --cwd a``. More than 10 directories are listed as the first 10 `, and N more`. Exit code and status are unchanged. A directory is named when it has an install directory of its own that the crawl from `--cwd` did not read: a `node_modules` the npm walk skips (one under a hidden or build-output directory), a Python environment or a Composer vendor directory inside the project, or a gem home (the project's `vendor/bundle`, or the user-wide homes Bundler installs into by default when `--cwd` holds no Bundler project to read them for). A directory is not named when the run did read its packages (every other `node_modules` below `--cwd`, a gem home or an activated environment both share) or when it has no install directory of its own; the user-wide Python, Cargo and Go stores are never compared. Nothing is looked for with a PATH, a global scope, a non-default `--manifest-path` or outside a git checkout. With nothing found at `--cwd`, a run that names such directories prints `No packages found (see the warning).` in place of the hint to run the package manager's install. A tree too large to search adds a `projects_below_not_searched` warning: some directories may not be named. A tree with more than 500,000 entries is searched only that far, with a stderr warning. **What a bare scan patched at and below `--cwd` is listed and rolled back from `--cwd` (additive/MINOR).** A bare scan covers `--cwd` and the project directories below it, so `list` and a bare `rollback` cover the project at `--cwd` and the patched project directories below it. A bare `rollback` reaches exactly as far as a bare scan from the same directory: from an agent-mode project (whose scan covers that project alone) it rolls back that project alone and names the directories below in a `projects_not_rolled_back` warning. `vex`, `apply`, `repair`, `remove` and `rollback ` act on the one project at `--cwd`, and name the patched directories below when `--cwd` holds nothing for them. The directories below are the ones a bare scan covers (`commands/projects_below.rs`: the same root detection and walk, the same `socket.yml` path filters and built-in `test/`, `tests/`, `fixtures/`, `__fixtures__/` and `testdata/` ignores, applied before any file of a directory is read; `SOCKET_NO_SOCKET_YML` bypasses the file here as `--no-socket-yml` does for the scan), kept when they hold patch state: a manifest with records, a vendor ledger with entries, or hosted pins in their lockfiles (a store that cannot be read counts, so the directory's own run reports it). Nothing is searched for when `--cwd` is not inside a git checkout, the scope is `--global`/`--global-prefix`, or `--manifest-path` is not the default (it names the one project the run is about). With no patched directory below, every command is, byte for byte, the single-project run it always was. @@ -2369,7 +2369,7 @@ command's integration tests check its output. | `removed` | `remove`, `repair`, `vendor`, `rollback`, `scan --prune` (manifest entries carry `details.manifest: true`) | A manifest entry or vendored state was removed, or (artifact-level, no `purl`) `.socket/` artifacts were swept — that GC carrier sets `bytes` and is not counted in `summary.removed`; the sweep's totals are the envelope's `gc`. | | `verified` | `apply`, `remove`, `repair`, `vendor`, `rollback`, `get`, `scan` (dry run); `vex`; `--update --dry-run` | The action *would* succeed cleanly (`files` lists previewed changes); `vex`: the patch verified and was attested. | | `rebuilt` | `repair`, `vendor` | A missing/corrupt vendored artifact was restored from its exact server download (v5.0: never a lost ledger entry — see `vendor_ledger_missing`). `summary.rebuilt` counts these. | -| `rolledBack` | `rollback`, `remove`, `scan --mode hosted` | A patched package was restored to its original files (`files` lists them), or (`rollback`) a vendored / hosted wiring was unwound (`details.mode`). `remove`: one per restored installed copy, `details.path` naming it — v5.0, MAJOR: was the GC carrier's `details.rolledBack`. `scan --mode hosted`: a pin whose patch was withdrawn was restored upstream (`errorCode: "patch_withdrawn"`). | +| `rolledBack` | `rollback`, `remove`, `scan --mode hosted` | A patched package was restored to its original files (`files` lists them), or (`rollback`) a vendored / hosted wiring was unwound (`details.mode`). `remove`: one per restored installed copy, `details.path` naming it — v5.0, MAJOR: was the GC carrier's `details.rolledBack`. `scan --mode hosted`: a pin whose patch was withdrawn was restored upstream (`errorCode: "patch_withdrawn"`), or a stale entry was taken out of the block in the `MODULE.bazel` of a Bazel workspace (`errorCode: "maven_bazel_entry_stale"`). | ### Stable `errorCode` tags @@ -2634,12 +2634,14 @@ command's integration tests check its output. | `vendor_pypi_reinstall_required` | `skipped` (advisory; human: `Warning: …`); rollback/remove `warnings[]`; `scan --prune` top-level `warnings[]` (human: `GC: …`) | a PyPI vendored revert (`vendor --revert`, rollback / remove of a vendored entry, the manifest reconcile, `--preserve-state` and `--dry-run` included) of a requirements.txt (pip), PDM, uv or Pipenv entry (#477, #1953). These tools reinstall a same-version package only when the LOCKED candidate is a URL or file that differs from the installed one, so once the lock is back on the registry release a plain `pdm sync` / `pdm install`, `uv sync` or `pipenv sync` reports nothing to do and keeps the vendored build; once a requirements line is back to `name==version`, `pip install -r` (and `uv pip install -r`) reports it already satisfied and keeps it too. The detail names the purl and the reinstall that restores the upstream bytes: `pip uninstall -y ` then `pip install -r ` for each requirements file that wired the entry (or the `uv pip uninstall` / `uv pip install -r` pair), which keeps those files' version ranges, index and hash options, `pdm sync --reinstall` (or recreating the virtualenv / `__pypackages__`), `uv sync --reinstall-package `, or Pipenv's `pipenv run pip uninstall -y && pipenv sync` (the categories from `Pipfile.lock`; `pipenv --rm && pipenv sync` for a clean virtualenv). Poetry, Hatch and PEP 751 entries do not get it. | | `redirect_pypi_reinstall_required` | rollback/remove `warnings[]` (+ human stderr), `vendor --revert` `warnings[]` | a hosted unwind (wet or `--dry-run`) that restored PyPI pins in `pdm.lock`, `uv.lock`, `Pipfile.lock` or a pip requirements file (`*.txt`) to their registry entry: the `vendor_pypi_reinstall_required` rule for hosted pins (the patched build's `direct_url.json` still names the patch server). One run-level warning naming each purl and its reinstall; a pin wired by both a lock and a requirements file (a `uv export` beside `uv.lock`) gets the lock tool's reinstall. | | `redirect_withdrawn_check_failed` | `warnings[]` (warning; human: `Warning: …`) | scan `--mode hosted`: the reference lookup for the uuids the lockfiles pin failed, so the run could not tell whether any was withdrawn. Every pin is kept; status and exit unchanged. See "Withdrawn pins". | -| `redirect_withdrawn_restore_failed` | `warnings[]` (warning; human: `Warning: …` / `Error: …`) | scan `--mode hosted`: a pin whose patch is `withdrawn` / `not_found` could not be restored to its upstream registry entry (the restore refused it, writing the restored files failed, or the apply lock was held). The pin is kept and no `rolledBack` event is recorded for it; the detail names the remedy (`git checkout -- `, or re-run / `socket-patch rollback `). A write failure is one warning naming every pin it kept. Status and exit unchanged, unless the failed write could not put a file back: see `lockfile_write_failed`. | +| `redirect_withdrawn_restore_failed` | `warnings[]` (warning; human: `Warning: …` / `Error: …`) | scan `--mode hosted`: a pin whose patch is `withdrawn` / `not_found` could not be restored to its upstream registry entry (the restore refused it, writing the restored files failed, or the apply lock was held). The pin is kept and no `rolledBack` event is recorded for it; the detail names the remedy (`git checkout -- `, or re-run / `socket-patch rollback `). A write failure is one warning naming every pin it kept. Status and exit unchanged, unless the failed write could not put a file back: see `lockfile_write_failed`. Also for a stale entry of a Bazel workspace's block that could not be taken out (`maven_bazel_entry_stale`). | | `patch_withdrawn` | `rolledBack` / `verified` (hosted event) | scan `--mode hosted`: the lockfile pinned a patch the server reports `withdrawn` or `not_found`; the pin was restored (dry run: would be) to its upstream registry entry. `reason` names the uuid and status. See "Withdrawn pins". | +| `maven_bazel_entry_stale` | `rolledBack` / `verified` (hosted event) | scan `--mode hosted` at the root of a Bazel workspace, with Maven in scope (additive): the block in `MODULE.bazel` held an entry for a jar that no pinned rules_jvm_external hub fetches at that version any more, at which Bazel stops ("does not generate repository"); the entry was taken out (dry run: would be), and with the last entry the block. Only when every hub was read, and not in a root, or for a package, that the `socket.yml` policy filters. A pin of the same patch in another build's files (a `pom.xml`, the hosted Gradle index) stays. `reason` names the uuid. A restore that fails keeps the entry with a `redirect_withdrawn_restore_failed` warning; one that leaves a file half written stops the run (`lockfile_write_failed`, exit 1), also when the scan has no package. See the lockfile supplement and docs/ecosystems.md "Bazel workspaces: Maven". | | `vendor_prebuilt_stub_invalid` | `failed` | RubyGems: the server stub lacks required attributes or is otherwise invalid; no local stub fallback is permitted. | | `bazelignore_not_updated` | run-level `warnings[]` | Go vendored and agent mode in a Bazel workspace (additive): the `.bazelignore` at the Bazel root, which keeps Bazel and gazelle out of the module copies under `.socket/vendor/golang` and `.socket/go-patches`, was not updated. It could not be read or written (a symbolic link or a directory in its place, a read-only directory, a read-only file). The detail names the line to add, or says that the lines of a copy that is gone can be deleted; the command's patch or undo stands and its exit code is unchanged. See docs/ecosystems.md "Bazel workspaces: `.bazelignore`". | | `maven_bazel_hub_unpatched` | run-level `warnings[]` (human: `Warning: …`) | Maven in a Bazel workspace, vendored and agent mode (additive): a pinned rules_jvm_external hub of the root `MODULE.bazel` fetches the jar of a Maven patch that this run vendored or applied, or tried to (`vendor`, `apply`, and `get` / `scan` in those modes; in agent mode only for a package it found installed; never under `--global`), and the block that hosted mode keeps in `MODULE.bazel` holds no entry for that jar. The hub builds the jar its own lock (`maven_install.json`) names, so the patch does not reach the Bazel build: these modes patch no hub. One warning per purl; the detail names the lock, the hub, the file and line of its `maven.install`, and the hosted command that patches a pinned hub. The events, the exit code and every written file are those of the same run without the workspace. A `vendor` whose `--ecosystems` leaves Maven out gives none. A vendored run that takes a hosted pin over (`vendor`, `scan` / `get --mode vendored`, `vendor --revert`) leaves the block's entry for it as it is. A hosted run never gives it: see the `maven_bazel_*` rows below and docs/ecosystems.md "Bazel workspaces: Maven". | -| `maven_bazel_unreadable` | vendored / agent: run-level `warnings[]` (human: `Warning: …`); scan/get `--mode hosted`: `warnings[]` (warning) | Maven in a Bazel workspace (additive): whether a rules_jvm_external hub builds the jar of a Maven patch could not be told, because something could not be read with certainty: a hub without `lock_file` in vendored and agent mode (hosted mode gives `maven_bazel_not_pinned`), a `maven.artifact` or `maven.from_toml` tag that cannot be read, a lock that is missing, empty, not format 2 or 3, larger than 16 MiB or no JSON, a `WORKSPACE` file that calls `maven_install` or cannot be read while `.bazeliskrc` or `.bazelversion` at the Bazel root names no release of Bazel 9 or later, a `maven.install` whose `name` or `lock_file` is not a plain string, a construct of `MODULE.bazel` the reader does not follow, and, in vendored and agent mode, a `maven.override` of the artifact (hosted mode gives `maven_bazel_no_effect`). Hosted mode also gives it for a hub on an isolated `use_extension`, an rc file that cannot be read (also one that an import takes from outside the workspace by a relative path, and one with a line whose reading Bazel's own source calls an oversight: a quote that is never closed, a `\` with nothing after it), a `MODULE.bazel` that is a symbolic link, and a block of `MODULE.bazel` that was changed by hand (also in its first line alone), and then writes no entry for the hub. One warning per purl; the detail names the file, the line and the reason. | +| `maven_bazel_unreadable` | vendored / agent: run-level `warnings[]` (human: `Warning: …`); scan/get `--mode hosted`: `warnings[]` (warning) | Maven in a Bazel workspace (additive): whether a rules_jvm_external hub builds the jar of a Maven patch could not be told, because something could not be read with certainty: a hub without `lock_file` in vendored and agent mode (hosted mode gives `maven_bazel_not_pinned`), a `maven.artifact` or `maven.from_toml` tag that cannot be read, a lock that is missing, empty, not format 2 or 3, larger than 16 MiB (or past the 64 MiB of locks that are read for one workspace) or no JSON, a `WORKSPACE` file that calls `maven_install` or cannot be read while `.bazeliskrc` or `.bazelversion` at the Bazel root names no release of Bazel 9 or later, a `maven.install` whose `name` or `lock_file` is not a plain string, a construct of `MODULE.bazel` the reader does not follow, and, in vendored and agent mode, a `maven.override` of the artifact (hosted mode gives `maven_bazel_no_effect`). Hosted mode also gives it for a hub on an isolated `use_extension`, an rc file that cannot be read (also one that an import takes from outside the workspace by a relative path, and one with a line whose reading Bazel's own source calls an oversight: a quote that is never closed, a `\` with nothing after it), a `MODULE.bazel` that is a symbolic link, and a block of `MODULE.bazel` that was changed by hand (also in its first line alone), and then writes no entry for the hub. One warning per purl; the detail names the file, the line and the reason. | +| `maven_bazel_hub_not_read` | scan run-level `warnings[]` (human: `Warning: …`); `get --mode hosted`: `warnings[]` | scan `--mode hosted` at the root of a Bazel workspace, with Maven in scope (additive): the scan takes the jars that the pinned rules_jvm_external hubs fetch for packages of the project (the lockfile supplement above), and something of the hubs could not be read with certainty, so it does not know every Maven artifact that Bazel builds through a hub. Given once per run, for the first thing that was not read, whatever the packages: a hub without `lock_file` (the detail then ends with how to pin it), or one of the reasons of `maven_bazel_unreadable` that concern reading the hubs (a tag or a lock that cannot be read, a `maven_install` of a `WORKSPACE` file that Bazel may read, a construct of `MODULE.bazel` the reader does not follow, a block that was changed by hand). `[:]: , so socket-patch does not know every Maven artifact that Bazel builds here through a rules_jvm_external hub: it looks for patches of the ones in the hub locks it read, and of no other`. No jar is guessed for what was not read; the jars of the hubs that were read are scanned, and a patch found for one of them gets no entry while the doubt stands (`maven_bazel_not_pinned` / `maven_bazel_unreadable` below). With no package at all a person reads `No packages found (see the warning).`. Exit code and `status` unchanged. A hosted `get` by advisory, which keeps the releases the project has by the same supplement, gives it too when the advisory has a patch of a Maven artifact. Only in a workspace that has to do with the ruleset: a call in `MODULE.bazel` or a file it includes names `rules_jvm_external`, the `WORKSPACE` file calls `maven_install`, or a `maven_install.json` lies at the Bazel root. In another Bazel workspace a file that cannot be read gives no such warning, and the run is the one without the workspace. | | `maven_bazel_not_at_root` | `warnings[]` (warning; human: `Warning: …`) | scan/get `--mode hosted` in a Bazel workspace (additive): a rules_jvm_external hub builds the jar of a Maven patch, and the project is below the root of its Bazel workspace, in whose `MODULE.bazel` the patch of a hub is written. Like the five codes below and `maven_bazel_unreadable` in hosted mode, it means that the run wrote no entry for the jar in the block of the root `MODULE.bazel`, because it could not show from the checkout that the entry works. One warning per purl, in the place of `redirect_maven_no_pom` for that purl; the detail says why and ends with what the block of the root holds for the jar, which is read for a project below the root too (the detail then names the file as "the MODULE.bazel at the root"). Nothing: "socket-patch wrote nothing to MODULE.bazel for it, so the Bazel build keeps the upstream jar". The event is the one of the same run without the hub: `skipped` / `redirect_unconfirmed` in a tree only Bazel builds, `applied` beside a `pom.xml` or Gradle pin that landed. The entry of the patch, from an earlier run: that entry stays, the event is `applied`, and the detail ends "socket-patch left the entry that MODULE.bazel holds for it as it is". The entry of another patch of the same jar: it stays too, the event is the one without the hub, and the detail ends "socket-patch wrote nothing to MODULE.bazel for it: the entry of patch remains there, and where the block is in effect the Bazel build takes that patch's jar". Exit code unchanged. With none of these codes the run writes the block: `redirect.rewrittenFiles` holds `MODULE.bazel`, the event is `applied`, and `rollback` / `remove` take the entry out. See docs/ecosystems.md "Bazel workspaces: Maven". | | `maven_bazel_not_pinned` | `warnings[]` (warning; human: `Warning: …`) | As `maven_bazel_not_at_root`: a hub of the workspace has no `lock_file` (also one that only `maven.artifact` or `maven.from_toml` tags make), so Bazel resolves its jars when it builds and none can be replaced. Also given beside a pinned hub that holds the artifact. The detail names the hub's file and line and how to pin it. | | `maven_bazel_release` | `warnings[]` (warning; human: `Warning: …`) | As `maven_bazel_not_at_root`: the Bazel release is not known to be 7.4.0 or later, the first with `override_repo` (`.bazeliskrc` or `.bazelversion` names an older one, or neither names a release of three numbers and `MODULE.bazel.lock` is missing or of a format before the one Bazel 7.5.0 writes), or the rules_jvm_external release Bazel selected is not known to be one of 6.4 to 7.2 (`MODULE.bazel.lock` holds the `source.json` of no release of it or of more than one, of a release outside that range, of an older one than the `bazel_dep` of `MODULE.bazel` asks for, or of another one than a `single_version_override` there pins; or an `archive_override`, `git_override`, `local_path_override` or `multiple_version_override` of the ruleset, or a `single_version_override` whose version is no plain string, leaves the release unknown). Versions are compared as Bazel compares them. The detail names the remedy (a `.bazelversion`, one build so that Bazel writes its lock, or an update of socket-patch) or the line of the override. | diff --git a/crates/socket-patch-cli/src/commands/get.rs b/crates/socket-patch-cli/src/commands/get.rs index 224c58da3..461b71cd5 100644 --- a/crates/socket-patch-cli/src/commands/get.rs +++ b/crates/socket-patch-cli/src/commands/get.rs @@ -926,6 +926,11 @@ async fn filter_to_installed_purls( let hosted = mode == super::scan::ScanMode::Hosted; let supplement = super::scan::project_lockfile_supplement(&ctx, &[], None, hosted).await; pnp_diags = supplement.unsupported; + // What was not read of a Bazel workspace's Maven hubs says nothing + // of patches that are for no Maven artifact. + let maven = accessible.iter().any(|p| p.purl.starts_with("pkg:maven/")); + let hub = socket_patch_core::bazel::maven::HUB_NOT_READ; + pnp_diags.retain(|diag| maven || diag.code != hub); if mode != super::scan::ScanMode::Agent { present.extend(supplement.entries.iter().map(|e| PurlKey::new(&e.purl))); // A hosted run's supplement also names the gems the lock diff --git a/crates/socket-patch-cli/src/commands/scan/discovery.rs b/crates/socket-patch-cli/src/commands/scan/discovery.rs index 5247005fe..89e67b195 100644 --- a/crates/socket-patch-cli/src/commands/scan/discovery.rs +++ b/crates/socket-patch-cli/src/commands/scan/discovery.rs @@ -7,11 +7,14 @@ use futures_util::StreamExt; use socket_patch_core::api::client::hold_back_debug; use socket_patch_core::api::ranking::{batch_supersedes, cmp_batch_infos}; use socket_patch_core::api::types::{BatchPackagePatches, PatchResponse, PatchSearchResult}; +use socket_patch_core::bazel::maven::HubJars; +use socket_patch_core::crawlers::Ecosystem; use socket_patch_core::manifest::schema::{PatchManifest, PatchRecord}; +use socket_patch_core::patch::redirect::upstream::HostedPin; use socket_patch_core::utils::concurrent::{api_concurrency_for, ordered_concurrent}; use socket_patch_core::utils::purl::{normalize_purl, strip_purl_qualifiers}; use socket_patch_core::utils::purl_key::PurlKey; -use socket_patch_core::vendor::lock_inventory::LockfileEntry; +use socket_patch_core::vendor::lock_inventory::{LockfileEntry, UnsupportedNpmLayout}; use socket_patch_core::vendor::VendorState; use std::collections::{BTreeSet, HashMap, HashSet}; @@ -41,7 +44,14 @@ pub(crate) struct LockfileSupplement { /// npm layouts the lockfile inventory REFUSED (Plug'n'Play loaders). /// Scan surfaces these as refusal warnings: under PnP the installed-tree /// crawl is empty too, so otherwise the project scans as a silent no-op. - pub(crate) unsupported: Vec, + /// Also what a hosted run did not read of the hubs of a Bazel workspace + /// ([`hub_jars`]). + pub(crate) unsupported: Vec, + /// The entries of the block in `MODULE.bazel` whose jar no hub of the + /// Bazel workspace fetches any more ([`hub_jars`]), as hosted pins of + /// that file alone: Bazel stops at each, so a hosted scan takes them + /// out (`withdrawn::sweep`). + pub(crate) stale_hub_pins: Vec, } /// Inventory the project's lockfile(s) and fabricate crawl entries for @@ -61,7 +71,8 @@ pub(crate) struct LockfileSupplement { /// (1.18.10-arm64-darwin)`: no one `.gem` stands for it, so it cannot be /// vendored from the lock, and an in-place patch needs the installed copy). /// Hosted mode redirects such a gem to its per-platform prebuilds from the -/// lock alone, so a hosted supplement names it too. +/// lock alone, so a hosted supplement names it too. It also names the jars +/// of [`hub_jars`], and what was not read of the hubs joins `unsupported`. pub(crate) async fn lockfile_supplement( ctx: &crate::commands::context::ProjectContext<'_>, crawled: &[socket_patch_core::crawlers::types::CrawledPackage], @@ -81,18 +92,46 @@ pub(crate) async fn lockfile_supplement( } else { Vec::new() }; - if entries.is_empty() && platform_gems.is_empty() { + let HubJars { + jars, + unread, + stale, + } = hub_jars(common, hosted).await; + out.unsupported + .extend(unread.map(|(code, detail)| UnsupportedNpmLayout { code, detail })); + let files = vec![socket_patch_core::bazel::maven::FILE.to_string()]; + let pin = |(purl, uuid)| HostedPin { + purl, + uuid, + files: files.clone(), + }; + out.stale_hub_pins = stale.into_iter().map(pin).collect(); + if entries.is_empty() && platform_gems.is_empty() && jars.is_empty() { return out; } let resolved = entries .iter() .map(|entry| entry.purl.as_str()) - .chain(platform_gems.iter().map(String::as_str)); + .chain(platform_gems.iter().map(String::as_str)) + .chain(jars.iter().map(String::as_str)); (out.packages, out.purls) = lockfile_only_packages(resolved, crawled, only, &common.cwd); out.entries = entries.clone(); out } +/// The purls of the jars that the pinned rules_jvm_external hubs of a Bazel +/// workspace fetch when `--cwd` is its root, what was not read of its hubs +/// and the entries of its block that are stale +/// ([`socket_patch_core::bazel::maven::hub_jars`]). Only for a hosted run +/// with Maven in scope: no other mode patches a hub. The callers are runs +/// of a project, never of the global installs. +pub(crate) async fn hub_jars(common: &GlobalArgs, hosted: bool) -> HubJars { + if !hosted || !common.ecosystem_selected(Ecosystem::Maven) { + return HubJars::default(); + } + socket_patch_core::bazel::maven::hub_jars(&common.cwd).await +} + /// The packages `resolved` (the purls the project's lockfiles resolve) with /// no crawled counterpart, fabricated as crawl entries, plus their /// [`PurlKey`]s. "Crawled" is by [`PurlKey`], the same relation diff --git a/crates/socket-patch-cli/src/commands/scan/mod.rs b/crates/socket-patch-cli/src/commands/scan/mod.rs index 270c142e5..444b368ba 100644 --- a/crates/socket-patch-cli/src/commands/scan/mod.rs +++ b/crates/socket-patch-cli/src/commands/scan/mod.rs @@ -1605,6 +1605,38 @@ fn emit_scan_error(common: &GlobalArgs, error: EnvelopeError) { emit_scan(&env); } +/// A restore of `sweep` that failed part-way and could not put its files +/// back leaves the lockfiles half restored: the run stops there with exit +/// 1 (`Err`), rather than write over them. The restore printed the +/// failure as it happened; `counts` are the scanned and the lockfile-only +/// packages of the envelope, and the last argument is what the failed +/// scan's telemetry takes. Otherwise `sweep` goes on. +async fn stop_half_restored( + common: &GlobalArgs, + sweep: Option, + counts: (usize, usize), + paths: &crate::path_scope::PathScope, + (fallback_to_proxy, mode, auth): (bool, &str, &TelemetryAuth), +) -> Result, i32> { + let half = |sweep: &withdrawn::WithdrawnSweep| sweep.half_restored().map(str::to_string); + let Some(why) = sweep.as_ref().and_then(half) else { + return Ok(sweep); + }; + track_patch_scan_failed("lockfile_write_failed", &why, fallback_to_proxy, mode, auth).await; + if common.json { + let mut env = scan_envelope(common); + env.set_extra("scannedPackages", serde_json::json!(counts.0)); + env.set_extra("lockfileOnlyPackages", serde_json::json!(counts.1)); + env.set_extra("paths", serde_json::json!(paths.raw())); + if let Some(sweep) = sweep { + sweep.record_into(&mut env); + } + env.mark_error(EnvelopeError::new("lockfile_write_failed", why)); + emit_scan(&env); + } + Err(1) +} + /// `(purl, n)` pairs summed per ecosystem, by CLI name: how many packages /// (or patches) of each ecosystem a scan found. An ecosystem with nothing /// has no entry; a purl of no known ecosystem is not counted. @@ -2267,11 +2299,23 @@ async fn scan_project( if args.common.json || args.vex.vex.is_some() { unscanned_projects = Some(others); } else { + // The root of a Bazel workspace can be without a file + // that makes `--cwd` a project, and a hosted scan has work + // there all the same: the jars that its pinned + // rules_jvm_external hubs fetch, a hub to name as not + // read, or a stale entry of the block to take out. + let mut dirs = found.dirs; + if !dirs.iter().any(|(_, explicit)| *explicit) { + let hosted = args.mode == Some(ScanMode::Hosted); + if discovery::hub_jars(&args.common, hosted).await != Default::default() { + dirs.insert(0, (args.common.cwd.clone(), true)); + } + } return Box::pin(run_project_dirs( args, invocation, mode_inferred, - found.dirs, + dirs, report, )) .await; @@ -2912,6 +2956,14 @@ async fn scan_project( let all_purls: Vec = filtered_crawled.iter().map(|p| p.purl.clone()).collect(); let package_count = all_purls.len(); + // The stale entries of a Bazel workspace's block that this run may + // take out: the policy holds them as it holds the run's packages. + let stale_hub_pins: Vec<_> = lockfile_only + .stale_hub_pins + .iter() + .filter(|pin| policy.admits_restore(&pin.purl)) + .cloned() + .collect(); if package_count == 0 { // An empty crawl is where the projects below matter most: they @@ -2943,6 +2995,21 @@ async fn scan_project( ); } status.finish(); + // Bazel stops at an entry of the block whose jar no hub fetches + // any more, whatever the project holds: such an entry is taken + // out here too (`withdrawn::sweep`, which asks nothing for it). + let stale_sweep = if hosted && policy.writes_allowed() { + Some(withdrawn::sweep(&args.common, &api_client, &[], &stale_hub_pins).await) + } else { + None + }; + let telemetry = (false, requested_mode, &telemetry_auth); + let stopped = + stop_half_restored(&args.common, stale_sweep, (0, 0), &path_scope, telemetry).await; + let stale_sweep = match stopped { + Ok(sweep) => sweep, + Err(code) => return code, + }; if human { print_layout_refusals(&layout_refusals, &project_patch_notice, args.common.silent); policy.print_warnings(args.common.silent); @@ -3009,6 +3076,9 @@ async fn scan_project( if prune { env.warnings.push(hosted::prune_ignored_warning()); } + if let Some(sweep) = stale_sweep { + sweep.record_into(&mut env); + } env.set_extra("redirect", hosted::redirect_block(Vec::new())); } else if !vendor { // `redirectState` rides the empty-discovery envelope too @@ -3388,43 +3458,26 @@ async fn scan_project( // longer serves, before the rewrite below re-pins any replacement. // Runs ahead of every "nothing found" exit: a package whose only patch // was withdrawn has no search results at all. - let mut withdrawn_sweep = if hosted && !args.common.is_global() && policy.writes_allowed() { + let withdrawn_sweep = if hosted && !args.common.is_global() && policy.writes_allowed() { let pins = withdrawn::pins_in_scope(&hosted_pin_list, &all_purls); - Some(withdrawn::sweep(&args.common, &api_client, &pins).await) + Some(withdrawn::sweep(&args.common, &api_client, &pins, &stale_hub_pins).await) } else { None }; - // A restore that failed part-way and could not put its files back - // leaves the lockfiles half restored: stop, rather than rewrite over - // them. (The restore printed the failure as it happened.) - if let Some(why) = withdrawn_sweep - .as_ref() - .and_then(|sweep| sweep.half_restored().map(str::to_string)) - { - track_patch_scan_failed( - "lockfile_write_failed", - &why, - fallback_to_proxy, - requested_mode, - &telemetry_auth, - ) - .await; - if args.common.json { - let mut env = scan_envelope(&args.common); - env.set_extra("scannedPackages", serde_json::json!(package_count)); - env.set_extra( - "lockfileOnlyPackages", - serde_json::json!(lockfile_only_count), - ); - env.set_extra("paths", serde_json::json!(path_scope.raw())); - if let Some(sweep) = withdrawn_sweep { - sweep.record_into(&mut env); - } - env.mark_error(EnvelopeError::new("lockfile_write_failed", why)); - emit_scan(&env); - } - return 1; - } + let counts = (package_count, lockfile_only_count); + let telemetry = (fallback_to_proxy, requested_mode, &telemetry_auth); + let stopped = stop_half_restored( + &args.common, + withdrawn_sweep, + counts, + &path_scope, + telemetry, + ) + .await; + let mut withdrawn_sweep = match stopped { + Ok(sweep) => sweep, + Err(code) => return code, + }; // The sweep's reference answers, reused by the hosted redirect. let sweep_references = withdrawn_sweep .as_mut() diff --git a/crates/socket-patch-cli/src/commands/scan/policy.rs b/crates/socket-patch-cli/src/commands/scan/policy.rs index 33912308d..616ccac1a 100644 --- a/crates/socket-patch-cli/src/commands/scan/policy.rs +++ b/crates/socket-patch-cli/src/commands/scan/policy.rs @@ -472,6 +472,14 @@ impl ScanPolicy { self.policy.enabled() } + /// Whether the run may take out a pin of `purl` that none of its + /// packages stands for (a stale entry of a Bazel workspace's block): + /// not in a root the policy filtered, and not for a package it + /// filters, as for a pin of a package the run covers. + pub(crate) fn admits_restore(&self, purl: &str) -> bool { + self.root_verdict.is_ok() && self.policy.admits_purl(purl).is_ok() + } + /// Set the merged recorded view (manifest > hosted pins > vendor ledger). pub(crate) fn set_recorded(&mut self, merged: Option<&PatchManifest>) { self.recorded = merged diff --git a/crates/socket-patch-cli/src/commands/scan/withdrawn.rs b/crates/socket-patch-cli/src/commands/scan/withdrawn.rs index 24bc86af1..ba46000b6 100644 --- a/crates/socket-patch-cli/src/commands/scan/withdrawn.rs +++ b/crates/socket-patch-cli/src/commands/scan/withdrawn.rs @@ -43,6 +43,15 @@ use crate::json_envelope::{Envelope, PatchAction, PatchEvent, RunWarning}; /// `reason` code on the `rolledBack` event of a pin the sweep restored. pub(super) const PATCH_WITHDRAWN: &str = "patch_withdrawn"; +/// `reason` code on the `rolledBack` event of an entry that the sweep took +/// out of the block in the `MODULE.bazel` of a Bazel workspace because no +/// pinned rules_jvm_external hub fetches its jar any more +/// ([`socket_patch_core::bazel::maven::HubJars::stale`]), and what a +/// message says of such a patch. +pub(super) const MAVEN_BAZEL_ENTRY_STALE: &str = "maven_bazel_entry_stale"; +const STALE: &str = "is for a jar that no pinned rules_jvm_external hub of the Bazel workspace \ + fetches any more, so Bazel stops at its entry in MODULE.bazel"; + /// Warning: a withdrawn pin could not be restored upstream (the restore /// refused it, the restored files could not be written, or the apply lock /// was held). The pin is unchanged; the detail names the remedy. The one @@ -159,17 +168,20 @@ pub(super) fn pins_in_scope( .collect() } -/// Check `pins` against the reference endpoint and restore the withdrawn -/// ones upstream (see the module docs). A wet restore runs under the apply -/// lock, released before the hosted rewrite takes it again. -pub(super) async fn sweep( +/// A pin the sweep restores, the `reason` code of its event, and what is +/// said of its patch. +type Gone = (HostedPin, &'static str, String); + +/// The pins of `pins` whose patch the reference endpoint says is gone for +/// good. A lookup that fails is a warning of `out`, and keeps every pin. +async fn withdrawn( common: &GlobalArgs, api_client: &ApiClient, pins: &[HostedPin], -) -> WithdrawnSweep { - let mut out = WithdrawnSweep::default(); + out: &mut WithdrawnSweep, +) -> Vec { if pins.is_empty() { - return out; + return Vec::new(); } let uuids: Vec = pins .iter() @@ -198,27 +210,61 @@ pub(super) async fn sweep( still published ({e}); every pin was kept, re-run to check again" ), ); - return out; + return Vec::new(); } }; // A uuid the response leaves out is not known to be gone: kept. - let gone: Vec<(HostedPin, String)> = pins + let gone = pins .iter() .filter_map(|pin| { let status = &references.get(&pin.uuid)?.status; - is_gone(status).then(|| (pin.clone(), status.clone())) + let why = format!("is no longer published ({status})"); + is_gone(status).then(|| (pin.clone(), PATCH_WITHDRAWN, why)) }) .collect(); out.references = Some(KnownReferences::settled(&uuids, references)); + gone +} + +/// `gone` and the pins of `stale` after it. A stale pin whose patch is +/// withdrawn too is in `gone` already, with every file that holds it: it +/// is restored, and reported, once. +fn with_stale(mut gone: Vec, stale: &[HostedPin]) -> Vec { + let known = |pin: &&HostedPin| { + let same = |(gone, ..): &Gone| (&gone.purl, &gone.uuid) == (&pin.purl, &pin.uuid); + !gone.iter().any(same) + }; + let stale: Vec<_> = stale.iter().filter(known).cloned().collect(); + gone.extend( + stale + .into_iter() + .map(|pin| (pin, MAVEN_BAZEL_ENTRY_STALE, STALE.to_string())), + ); + gone +} + +/// Check `pins` against the reference endpoint and restore the withdrawn +/// ones upstream (see the module docs), and with them the pins of `stale`, +/// which are asked about nowhere: the entries of a Bazel workspace's block +/// that the build stops at. A wet restore runs under the apply lock, +/// released before the hosted rewrite takes it again. +pub(super) async fn sweep( + common: &GlobalArgs, + api_client: &ApiClient, + pins: &[HostedPin], + stale: &[HostedPin], +) -> WithdrawnSweep { + let mut out = WithdrawnSweep::default(); + let gone = withdrawn(common, api_client, pins, &mut out).await; + let gone = with_stale(gone, stale); if gone.is_empty() { return out; } if !common.json && !common.silent { - for (pin, status) in &gone { + for (pin, _, why) in &gone { eprintln!( - "Note: patch {} for {} is no longer published ({status}); restoring its \ - upstream registry entry", + "Note: patch {} for {} {why}; restoring its upstream registry entry", pin.uuid, pin.purl ); } @@ -238,14 +284,13 @@ pub(super) async fn sweep( } Err(err) => { let (_, message) = crate::commands::lock_cli::lock_failure(&err, timeout); - for (pin, status) in &gone { + for (pin, _, why) in &gone { out.warn( common, REDIRECT_WITHDRAWN_RESTORE_FAILED, format!( - "{} is pinned to patch {}, which is no longer published \ - ({status}), but it was not restored: {message}; re-run, or run \ - `socket-patch rollback {}`", + "{} is pinned to patch {}, which {why}, but it was not restored: \ + {message}; re-run, or run `socket-patch rollback {}`", pin.purl, pin.uuid, pin.purl ), ); @@ -255,7 +300,7 @@ pub(super) async fn sweep( } }; - let to_restore: Vec = gone.iter().map(|(pin, _)| pin.clone()).collect(); + let to_restore: Vec = gone.iter().map(|(pin, ..)| pin.clone()).collect(); // The scan's own client, for a patch record the gem leg may need. let run_client = crate::args::RunApiClient::new_with(Some(api_client.clone())); let leg = @@ -267,7 +312,7 @@ pub(super) async fn sweep( } else { PatchAction::RolledBack }; - for (pin, status) in &gone { + for (pin, code, why) in &gone { if !leg.reverted.iter().any(|p| p == &pin.purl) { continue; } @@ -280,10 +325,9 @@ pub(super) async fn sweep( PatchEvent::new(action, pin.purl.as_str()) .with_uuid(pin.uuid.as_str()) .with_reason( - PATCH_WITHDRAWN, + *code, format!( - "patch {} is no longer published ({status}); the pin {verb} to the \ - upstream registry entry", + "patch {} {why}; the pin {verb} to the upstream registry entry", pin.uuid ), ), @@ -296,7 +340,7 @@ pub(super) async fn sweep( // so it kept every pin that was not refused on its own. let subject = if key == crate::commands::hosted_unwind::WRITE_FAILURE_KEY { gone.iter() - .map(|(pin, _)| pin.purl.as_str()) + .map(|(pin, ..)| pin.purl.as_str()) .filter(|purl| !leg.failed.iter().any(|(refused, _)| refused == purl)) .collect::>() .join(", ") @@ -362,6 +406,28 @@ mod tests { assert!(pins_in_scope(&pins, Vec::::new()).is_empty()); } + #[test] + fn a_stale_pin_that_is_withdrawn_too_is_taken_once() { + let both = HostedPin { + files: vec!["MODULE.bazel".to_string(), "pom.xml".to_string()], + ..pin("pkg:maven/g/a@1.0", "u1") + }; + let stale = [ + pin("pkg:maven/g/a@1.0", "u1"), + pin("pkg:maven/g/b@2.0", "u2"), + ]; + let gone = vec![(both.clone(), PATCH_WITHDRAWN, "is gone".to_string())]; + let taken: Vec<_> = with_stale(gone, &stale) + .into_iter() + .map(|(pin, code, _)| (pin, code)) + .collect(); + let expected = [ + (both, PATCH_WITHDRAWN), + (stale[1].clone(), MAVEN_BAZEL_ENTRY_STALE), + ]; + assert_eq!(taken, expected); + } + #[test] fn only_settled_answers_are_known() { let answer = |status: &str| -> PackageVendorResult { diff --git a/crates/socket-patch-cli/tests/e2e_bazel_build/maven.rs b/crates/socket-patch-cli/tests/e2e_bazel_build/maven.rs index c2d13aa7c..f0b4166fe 100644 --- a/crates/socket-patch-cli/tests/e2e_bazel_build/maven.rs +++ b/crates/socket-patch-cli/tests/e2e_bazel_build/maven.rs @@ -24,6 +24,12 @@ //! build fails on the sha256 and never goes to the upstream jar. On a //! Bazel before 7.5.0 the checkout first has no `.bazelversion`: the //! command then writes nothing and says so (`maven_bazel_release`). +//! - [`a_hosted_scan_of_a_tree_only_bazel_builds`]: the same checkout, and +//! `socket-patch scan --mode hosted --yes` in the place of the `get`. The +//! two jars of the hub's lock are its packages, neither of them +//! installed where a crawl looks; it finds the patch of the victim and +//! writes the same block. The build is patched, a second scan writes +//! nothing, and `rollback --yes` restores every file. //! - [`a_pom_patch_beside_a_pinned_hub`]: a checkout that Maven builds //! too. The one command pins both builds: the three files of a Maven pin //! and `MODULE.bazel`. The Bazel build is patched, and `rollback --yes` @@ -33,7 +39,10 @@ //! passes. The build then stops at the line of the entry's //! `override_repo(`, where the block says what to run. `socket-patch //! rollback --yes` takes the entry out, and the build takes the -//! new upstream jar. +//! new upstream jar. The API has a patch of that release too: a hosted +//! `scan` of the checkout with the stale entry takes the entry out, +//! reads the release from the lock, pins that patch, and the build +//! takes its jar. //! - [`the_users_own_override_of_the_artifact`]: the hub has a //! `maven.override` that puts a target of the checkout in the artifact's //! place. The command writes nothing and names the override's line @@ -92,7 +101,7 @@ use sha1::Sha1; use sha2::{Digest, Sha256}; use socket_patch_core::bazel::maven::{Lock, Pinned as Jar}; -use crate::envelope::event_for; +use crate::envelope::{event_for, event_triples}; use crate::harness::{changed, rewrite, Cell, Fixture, Ran, StandIn, Tree, CONSUMER, NOTHING}; use crate::hosted_memory_common::Patch; @@ -121,6 +130,8 @@ const GROUP_PATH: &str = "dev/socket/fixture"; const VICTIM: &str = "victim"; const VERSION: &str = "1.10.0"; const PURL: &str = "pkg:maven/dev.socket.fixture/victim@1.10.0"; +/// The release a flow moves the checkout to, which has a patch too. +const BUMPED: &str = "1.11.0"; /// The file of the victim's jar that `App.java` prints. const MARKER: &str = "dev/socket/fixture/victim/marker.txt"; /// The fixture's hub, under a name no other Bazel module adds to. @@ -221,6 +232,8 @@ fn serve_upstream( /// A patch of the victim that a stand-in grants. struct Granted { + /// The release of the victim it is for. + version: String, uuid: String, /// The sha256 the grant pins, of a jar with [`Granted::marker`]. sha256: String, @@ -232,27 +245,29 @@ struct Granted { marker: String, } -/// Puts the grant of a new patch of the victim on the API of `standin`, -/// as production has it: a token and the patch id in the path of the patch -/// registry, the version with the suffix only that registry serves. The -/// route of the jar answers with `tampered`, or with the jar the grant -/// pins. -fn grant(standin: &StandIn, tampered: Option<&[u8]>) -> Granted { +/// The grant of a new patch of the victim at `version`, as production has +/// it: a token and the patch id in the path of the patch registry, the +/// version with the suffix only that registry serves. The route of the jar +/// on `standin` answers with `tampered`, or with the jar the grant pins. +/// Gives the patch for the API of the stand-in too, which takes every +/// patch it grants in one call ([`StandIn::api`]). +fn grant(standin: &StandIn, version: &str, tampered: Option<&[u8]>) -> (Granted, Patch) { let uuid = uuid::Uuid::new_v4().to_string(); let token = uuid::Uuid::new_v4().to_string(); - let suffixed = format!("{VERSION}-socket.{}", &uuid[..8]); + let purl = format!("pkg:maven/{GROUP}/{VICTIM}@{version}"); + let suffixed = format!("{version}-socket.{}", &uuid[..8]); let jar = victim(&format!("PATCHED {uuid}")); let registry = format!("{PATCH_HOST}/patch-registry/maven/{token}/{uuid}/maven2"); let file = repo_path(VICTIM, &suffixed, "jar"); let route = format!("/{registry}/{file}"); standin.serve(5, &route, tampered.unwrap_or(&jar)); let artifact = format!( - "https://{PATCH_HOST}/patch/maven/{GROUP}/{VICTIM}/{VERSION}/{token}/{uuid}/{VICTIM}-{suffixed}.jar" + "https://{PATCH_HOST}/patch/maven/{GROUP}/{VICTIM}/{version}/{token}/{uuid}/{VICTIM}-{suffixed}.jar" ); let reference = json!({ "status": "granted", "url": artifact, - "purl": PURL, + "purl": purl, "artifacts": [{ "kind": "tarball", "url": artifact, @@ -263,7 +278,7 @@ fn grant(standin: &StandIn, tampered: Option<&[u8]>) -> Granted { "indexUrl": format!("https://{registry}"), "identifiers": { "name": format!("{GROUP}/{VICTIM}"), - "version": VERSION, + "version": version, "mavenGroupId": GROUP, "mavenArtifactId": VICTIM, "mavenSuffixedVersion": suffixed, @@ -271,18 +286,20 @@ fn grant(standin: &StandIn, tampered: Option<&[u8]>) -> Granted { }, }, }); - standin.api(&[Patch { - purl: PURL.to_string(), + let patch = Patch { + purl, uuid: uuid.clone(), reference, - }]); - Granted { + }; + let granted = Granted { + version: version.to_string(), sha256: sha256_hex(&jar), url: format!("https://{registry}/{file}"), route, marker: format!("OUT: PATCHED {uuid}"), uuid, - } + }; + (granted, patch) } /// The name of the repository through which the fixture's hub fetches the @@ -292,28 +309,34 @@ const REPOSITORY: &str = "dev_socket_fixture_victim_1_10_0"; /// The block of `MODULE.bazel` for `patch`, with its last line break. fn block(patch: &Granted) -> String { let Granted { - uuid, sha256, url, .. + version, + uuid, + sha256, + url, + .. } = patch; + let repository = REPOSITORY.replace("1_10_0", &version.replace('.', "_")); + let purl = format!("pkg:maven/{GROUP}/{VICTIM}@{version}"); format!( r#"# >>> socket-patch: Maven patches for Bazel (managed block: socket-patch writes and removes it) >>> socket_patch_http_file = use_repo_rule("@bazel_tools//tools/build_defs/repo:http.bzl", "http_file") socket_patch_maven = use_extension("@rules_jvm_external//:extensions.bzl", "maven") -# {PURL}, Socket patch {uuid} +# {purl}, Socket patch {uuid} socket_patch_http_file( - name = "socket_patch_{REPOSITORY}", - downloaded_file_path = "v1/{GROUP_PATH}/{VICTIM}/{VERSION}/{VICTIM}-{VERSION}.jar", + name = "socket_patch_{repository}", + downloaded_file_path = "v1/{GROUP_PATH}/{VICTIM}/{version}/{VICTIM}-{version}.jar", sha256 = "{sha256}", urls = ["{url}"], ) # If Bazel stops at the next line ("does not generate repository"), the hub's lock no longer pins -# {GROUP}:{VICTIM} at {VERSION}: run `socket-patch rollback {PURL}`. +# {GROUP}:{VICTIM} at {version}: run `socket-patch rollback {purl}`. # Do not change it to inject_repo(): the build would pass without the patch. override_repo( socket_patch_maven, - {REPOSITORY} = "socket_patch_{REPOSITORY}", + {repository} = "socket_patch_{repository}", ) # <<< socket-patch <<< "# @@ -329,8 +352,10 @@ struct Pinned { /// checkout leaves it; `coursier`, the cache the ruleset's Coursier /// fills when it pins; `gradle`, which nothing fills. tools: tempfile::TempDir, - /// The patch the stand-in grants, whose jar it serves. + /// The patch the stand-in grants, whose jar it serves, and the one it + /// grants for the victim at [`BUMPED`]. patch: Granted, + next: Granted, /// The id of the run, which every upstream jar made for it holds. run: String, /// What `//:consumer` prints while Bazel builds the upstream jar, and @@ -378,7 +403,9 @@ impl Pinned { &jar(&[]), Some((VICTIM, VERSION)), ); - let patch = grant(&standin, None); + let (patch, first) = grant(&standin, VERSION, None); + let (next, second) = grant(&standin, BUMPED, None); + standin.api(&[first, second]); // The checkout: the hub's repository is the stand-in, the lock is // the empty file the ruleset's documentation starts from. @@ -440,6 +467,7 @@ impl Pinned { standin, tools, patch, + next, run, upstream, upstream_route: format!("/{UPSTREAM}/{}", repo_path(VICTIM, VERSION, "jar")), @@ -502,6 +530,30 @@ impl Pinned { ); } + /// `socket-patch scan --mode hosted --yes` in a checkout without a + /// `pom.xml`: the two jars of the hub's lock are its packages, neither + /// installed where a crawl looks, and it pins the one `patch` is for + /// as the `get` of [`Pinned::pins`] does. `stale` is the patch of an + /// entry in the block that the lock no longer pins the jar of: the + /// scan takes that entry out first, and says so. `before` is the + /// checkout without any entry. Gives the files of the checkout. + fn scan_pins(&self, patch: &Granted, before: &Tree, stale: Option<&Granted>) -> Tree { + let test = &self.cell.test; + let scan = self.socket_patch(&["scan", "--mode", "hosted", "--yes"]); + let said = &scan.dump; + let counted = ["scannedPackages", "lockfileOnlyPackages"].map(|n| &scan.envelope[n]); + assert_eq!(counted, [&json!(2), &json!(2)], "{test}: {said}"); + let gone = |stale: &Granted| { + let purl = format!("pkg:maven/{GROUP}/{VICTIM}@{}", stale.version); + (purl, "rolledBack".into(), "maven_bazel_entry_stale".into()) + }; + let taken_out: Vec<_> = stale.map(gone).into_iter().collect(); + let events = event_triples(&scan.envelope); + assert_eq!(events[..taken_out.len()], taken_out, "{test}: {said}"); + assert_eq!(events.len(), taken_out.len() + 1, "{test}: {said}"); + self.pinned(&scan, patch, before, &[MODULE]) + } + /// A Bazel before 7.5.0 writes a lock that does not tell it from one /// without `override_repo`, so its checkout names the release as a /// launcher reads it. Gives the files of the checkout. @@ -513,23 +565,27 @@ impl Pinned { self.cell.tree() } - /// The one command of hosted mode: it reports the patch applied, - /// rewrites `files` and no other file of `before`, and gives no - /// warning. `MODULE.bazel` is the one of before with the block after - /// one blank line, and nothing has asked for the patched jar yet. - /// Gives the files of the checkout. + /// The one command of hosted mode, a `get` of the purl: what + /// [`Pinned::pinned`] holds of it. Gives the files of the checkout. fn pins(&self, before: &Tree, files: &[&str]) -> Tree { - let test = &self.cell.test; let get = self.socket_patch(&["get", PURL, "--mode", "hosted", "--yes"]); - let said = &get.dump; - let pinned = self - .cell - .pinned_by(&get, PURL, &self.patch.uuid, before, files); - assert_eq!(get.envelope["warnings"], Value::Null, "{test}: {said}"); + self.pinned(&get, &self.patch, before, files) + } + + /// What a finished hosted command did for `patch`: it reports the + /// patch applied, rewrote `files` and no other file of `before`, and + /// gave no warning. `MODULE.bazel` is the one of before with the block + /// after one blank line, and nothing has asked for the patched jar + /// yet. Gives the files of the checkout. + fn pinned(&self, ran: &Ran, patch: &Granted, before: &Tree, files: &[&str]) -> Tree { + let (test, said) = (&self.cell.test, &ran.dump); + let purl = format!("pkg:maven/{GROUP}/{VICTIM}@{}", patch.version); + let pinned = self.cell.pinned_by(ran, &purl, &patch.uuid, before, files); + assert_eq!(ran.envelope["warnings"], Value::Null, "{test}: {said}"); let text = |tree: &Tree| String::from_utf8(tree[MODULE].clone()).expect("a text file"); - let with_block = format!("{}\n{}", text(before), block(&self.patch)); + let with_block = format!("{}\n{}", text(before), block(patch)); assert_eq!(text(&pinned), with_block, "{test}: {said}"); - let asked = self.standin.asked(&self.patch.route); + let asked = self.standin.asked(&patch.route); assert_eq!(asked, 0, "{test}: the command fetched the patched jar"); pinned } @@ -563,7 +619,8 @@ impl Pinned { let mut twin = self.cell.twin(); let rogue = StandIn::start(); let tampered = victim("TAMPERED"); - let other = grant(&rogue, Some(&tampered)); + let (other, served) = grant(&rogue, VERSION, Some(&tampered)); + rogue.api(&[served]); let host = format!("{PATCH_HOST}/"); twin.route_downloads(&[ rewrite(&host, &rogue.origin()), @@ -741,9 +798,57 @@ fn a_tree_only_bazel_builds() { ); } +/// A hosted scan of the checkout without a `pom.xml`: the lock of the hub +/// is the one file that names the jars Bazel builds, and the command takes +/// them for the packages of the project. It finds the patch of the victim +/// and pins it as the `get` does: the build is patched, a second scan has +/// nothing to write, and the undo restores every file. +#[test] +#[ignore = "real Bazel; run with --ignored"] +fn a_hosted_scan_of_a_tree_only_bazel_builds() { + let test = "maven::a_hosted_scan_of_a_tree_only_bazel_builds"; + let Some(flow) = Pinned::open(test, false, &|_, _| {}) else { + return; + }; + flow.builds_upstream(); + let before = flow.names_its_bazel(); + + let pinned = flow.scan_pins(&flow.patch, &before, None); + flow.builds_patched(&pinned); + let again = flow.socket_patch(&["scan", "--mode", "hosted", "--yes"]); + let said = &again.dump; + let rewritten = &again.envelope["redirect"]["rewrittenFiles"]; + assert_eq!( + (again.code, &again.envelope["summary"]["applied"], rewritten), + (0, &json!(1), &json!([])), + "{test}: {said}" + ); + assert_eq!( + changed(&pinned, &flow.cell.tree()), + NOTHING, + "{test}: {said}" + ); + + let undo = flow.socket_patch(&["rollback", "--yes"]); + flow.cell + .undone(&undo, "rolledBack", &before, &flow.upstream); + println!( + "{test}: Bazel {} built {:?}, then {:?} from the patch host after `scan --mode hosted \ + --yes` found {PURL} in the hub's lock and appended the block to MODULE.bazel; a second \ + scan wrote nothing; `rollback` restored all {} files and the first build; {:?}", + flow.cell.release(), + flow.upstream, + flow.patch.marker, + before.len(), + flow.started.elapsed() + ); +} + /// The checkout moves the artifact one version up after the command: the /// relock passes, the build stops at the line of the entry, and the -/// command the block names there takes the entry out. +/// command the block names there takes the entry out. So does a hosted +/// scan of the checkout with the stale entry: it takes the entry out, +/// finds the new release in the lock, and pins the patch of it. #[test] #[ignore = "real Bazel; run with --ignored"] fn a_version_bump_of_the_patched_artifact() { @@ -756,7 +861,7 @@ fn a_version_bump_of_the_patched_artifact() { flow.pins(&before, &[MODULE]); // The relock evaluates the hub with the entry in place, and passes. - let bumped = flow.bump("1.11.0"); + let bumped = flow.bump(BUMPED); let module = std::fs::read_to_string(flow.cell.path(MODULE)).expect("MODULE.bazel"); let entry = module.lines().position(|line| line == "override_repo("); let line = entry.expect("the entry's override_repo") + 1; @@ -789,12 +894,26 @@ fn a_version_bump_of_the_patched_artifact() { let after = std::fs::read_to_string(flow.cell.path(MODULE)).expect("MODULE.bazel"); assert_eq!(after, unpatched, "{test}: {said}"); flow.cell.builds(&bumped); + + // A user who runs a hosted scan where the block says to roll back: + // the stale entry is in the file again, as it was when the build + // stopped. The scan takes it out as the rollback did, and the API has + // a patch of the release the lock now pins. + let rolled_back = flow.cell.tree(); + flow.cell.write(MODULE, module); + let pinned = flow.scan_pins(&flow.next, &rolled_back, Some(&flow.patch)); + flow.cell.builds_patched(&pinned, &flow.next.marker); + let asked = flow.standin.asked(&flow.next.route); + assert!(asked > 0, "{test}: the build did not fetch the patched jar"); println!( "{test}: Bazel {} built {:?}; after `get --mode hosted` and the checkout's bump to \ - 1.11.0 the relock passed and the build stopped at {at} ({stops}); `rollback {PURL}` \ - took the entry out and the build printed {bumped:?}; {:?}", + {BUMPED} the relock passed and the build stopped at {at} ({stops}); `rollback {PURL}` \ + took the entry out and the build printed {bumped:?}; with the stale entry back, `scan \ + --mode hosted --yes` took it out, pinned the patch of {BUMPED} and the build printed \ + {:?}; {:?}", flow.cell.release(), flow.upstream, + flow.next.marker, flow.started.elapsed() ); } diff --git a/crates/socket-patch-cli/tests/e2e_bazel_maven_beside.rs b/crates/socket-patch-cli/tests/e2e_bazel_maven_beside.rs index c494e7a0a..5da884d89 100644 --- a/crates/socket-patch-cli/tests/e2e_bazel_maven_beside.rs +++ b/crates/socket-patch-cli/tests/e2e_bazel_maven_beside.rs @@ -18,6 +18,13 @@ //! and says so, unless the block already does. A `--global` run says //! nothing. //! +//! A hosted `scan` at the Bazel root takes the jars that the pinned hubs +//! fetch for packages of the project, at the versions of their locks, so +//! it finds the patch of one in a tree that only Bazel builds, also from a +//! root above other projects, and names a hub it could not read. A scan +//! in another mode, one that leaves Maven out, and one where no hub of the +//! ruleset is declared are the scans without Bazel. +//! //! The vendored twin is in `vendor_jvm_cli.rs`, which has the fixture of a //! vendored Maven run. `e2e_bazel_build/maven.rs` holds the hosted flows //! against what a real Bazel builds, with a hub of the fixture's own name @@ -33,7 +40,7 @@ mod hosted_memory_common; use std::collections::BTreeMap; use std::path::PathBuf; -use envelope::{event_for, warning_codes}; +use envelope::{event_for, event_triples, warning_codes}; use hosted_memory_common::{hermetic, mount_api, read_tree, Patch, ORG}; use serde_json::{json, Value}; use socket_patch_core::hash::git_sha256::compute_git_sha256_from_bytes as git_sha256; @@ -73,6 +80,13 @@ pip = use_extension("@rules_python//python/extensions:pip.bzl", "pip") pip.parse(hub_name = "pypi", python_version = "3.11", requirements_lock = "//:maven_install.json") use_repo(pip, "pypi") "#; +/// That workspace with a file its `MODULE.bazel` includes and the checkout +/// lacks: something of it cannot be read, and nothing of it names the +/// ruleset of a Maven hub. +fn unread_other() -> Files { + let module = format!("{OTHER_HUB}include(\"//:more.MODULE.bazel\")\n"); + vec![("MODULE.bazel", module)] +} /// A lock of format 3 as a pin of [`HUB`] writes it when another Bazel /// module adds to the hub named `maven` too, which the real-Bazel flows do @@ -1086,6 +1100,735 @@ fn an_agent_run_says_nothing_of_a_hub_the_block_patches() { } } +/// A hosted scan, as a program reads it. +const SCAN: [&str; 5] = ["scan", "--mode", "hosted", "--yes", "--json"]; + +/// A hosted scan in a tree that only Bazel builds: the jars the pinned +/// hub's lock fetches are its packages, each one that no crawl found +/// installed, and the one with a patch gets its entry in the block. +#[test] +fn a_hosted_scan_takes_the_jars_of_a_pinned_hub_for_packages() { + let machine = Machine::start(); + let with_block = format!("{HUB}\n{BLOCK}"); + machine.make(&pinned(), false, false); + // A dry run says what the scan will do, and writes nothing. + let dry = machine.step(&[&SCAN[..], &["--dry-run"]].concat()); + let action = |ran: &Ran| event_for(&ran.envelope(), PURL)["action"].clone(); + assert_eq!(action(&dry), "verified", "{dry:?}"); + assert_eq!( + (dry.code, &dry.wrote), + (Some(0), &BTreeMap::new()), + "{dry:?}" + ); + + let scan = machine.step(&SCAN); + let envelope = scan.envelope(); + assert_eq!(scan.code, Some(0), "{scan:?}"); + // The victim, the consumer and what another Bazel module added. + let counted = ["scannedPackages", "lockfileOnlyPackages"].map(|count| &envelope[count]); + assert_eq!(counted, [&json!(3), &json!(3)], "{scan:?}"); + let found = envelope["packages"].as_array().expect("the packages"); + let patched: Vec<_> = found.iter().map(|package| &package["purl"]).collect(); + assert_eq!(patched, [PURL], "{scan:?}"); + assert_eq!(found[0]["notInstalled"], true, "{scan:?}"); + assert_eq!(action(&scan), "applied", "{scan:?}"); + assert_eq!( + envelope["redirect"]["rewrittenFiles"], + json!(["MODULE.bazel"]), + "{scan:?}" + ); + assert_eq!(envelope["warnings"], Value::Null, "{scan:?}"); + assert_eq!(scan.wrote.keys().collect::>(), [MODULE], "{scan:?}"); + assert_eq!( + String::from_utf8_lossy(&scan.wrote[MODULE]), + with_block, + "{scan:?}" + ); + // Again: the pin stands, and nothing is written. + let again = machine.step(&SCAN); + assert_eq!(action(&again), "applied", "{again:?}"); + assert_eq!( + (again.code, &again.wrote), + (Some(0), &BTreeMap::new()), + "{again:?}" + ); + + // A scan that names no mode is a hosted one, and a person is told + // which file to commit. + machine.make(&pinned(), false, false); + let person = machine.step(&["scan", "--yes"]); + assert_eq!(person.code, Some(0), "{person:?}"); + assert!(person.stdout.contains("Commit MODULE.bazel"), "{person:?}"); + assert!(!person.stdout.contains("No packages found"), "{person:?}"); + assert_eq!( + String::from_utf8_lossy(&person.wrote[MODULE]), + with_block, + "{person:?}" + ); + + // A jar whose repository has no known name is a package too: the run + // that finds a patch of it says why it gets no entry. + let mut odd = pinned(); + odd[1].1 = lock(&[("consumer", "2.0"), ("victim", "1.10.0+local")]); + machine.make(&odd, false, false); + let scan = machine.step(&SCAN); + assert_eq!(scan.envelope()["scannedPackages"], 3, "{scan:?}"); + assert_eq!(scan.wrote, BTreeMap::new(), "{scan:?}"); +} + +/// A Bazel root is often a project of another package manager too. The +/// jars of its hub join the packages its own lockfile names, and the root +/// is scanned once, above the projects below it. +#[test] +fn a_hosted_scan_takes_the_jars_of_a_hub_beside_a_lockfile_of_the_root() { + let machine = Machine::start(); + let package = json!({ "name": "app", "version": "1.0.0" }); + let npm = json!({ "name": "app", "version": "1.0.0", "lockfileVersion": 3, + "packages": { "": package, "node_modules/left-pad": { "version": "1.3.0" } } }); + let mut files = pinned(); + files.push(("package.json", package.to_string())); + files.push(("package-lock.json", npm.to_string())); + machine.make(&files, false, false); + let scan = machine.step(&SCAN); + let envelope = scan.envelope(); + // The npm package of the root's lock and the three jars of the hub's. + let counted = ["scannedPackages", "lockfileOnlyPackages"].map(|count| &envelope[count]); + assert_eq!(counted, [&json!(4), &json!(4)], "{scan:?}"); + assert_eq!(event_for(&envelope, PURL)["action"], "applied", "{scan:?}"); + assert_eq!(scan.wrote.keys().collect::>(), [MODULE], "{scan:?}"); + assert_eq!( + String::from_utf8_lossy(&scan.wrote[MODULE]), + format!("{HUB}\n{BLOCK}"), + "{scan:?}" + ); + + // Above another project: the root has its one section, the first. + files.push(("web/package.json", package.to_string())); + files.push(("web/package-lock.json", npm.to_string())); + machine.make(&files, false, false); + let person = machine.step(&["scan", "--yes"]); + let headings: Vec<&str> = person + .stdout + .lines() + .filter(|l| l.starts_with("== ")) + .collect(); + assert_eq!(headings, ["== . ==", "== web =="], "{person:?}"); + assert_eq!( + person.wrote.keys().collect::>(), + [MODULE], + "{person:?}" + ); +} + +/// The block for a patch `uuid` of the victim at `version`, as +/// socket-patch writes it. +/// A Gradle build that has nothing of the hub's jars. +const GRADLE: [(&str, &str); 3] = [ + ("settings.gradle", "rootProject.name = 'app'\n"), + ( + "build.gradle", + "plugins { id 'java' }\ndependencies { implementation 'com.example:other:1.0' }\n", + ), + ( + "gradle.lockfile", + "com.example:other:1.0=compileClasspath\nempty=\n", + ), +]; + +fn block_at(version: &str, uuid: &str) -> String { + let name = version.replace('.', "_"); + let block = BLOCK.replace("1.10.0", version).replace("1_10_0", &name); + block.replace(UUID, uuid).replace(&UUID[..8], &uuid[..8]) +} + +/// The hub's lock moves on after a pin, and Bazel stops at the entry of +/// the release it held. A hosted scan takes that entry out as `rollback +/// ` does, says so, and pins the patch of the release the lock holds +/// now. It leaves the entry while a hub is in doubt, and a scan that reads +/// no hub leaves it too. +#[test] +fn a_hosted_scan_takes_a_stale_entry_out_of_the_block() { + let machine = Machine::start(); + let old = "pkg:maven/dev.socket.fixture/victim@1.9.0"; + let old_uuid = "0a1b2c3d-1111-4222-8333-444455556666"; + let stale = "maven_bazel_entry_stale"; + let mut files = pinned(); + files[0].1 = format!("{HUB}\n{}", block_at("1.9.0", old_uuid)); + let triples = |ran: &Ran| event_triples(&ran.envelope()); + let event = |purl: &str, action: &str, code: &str| { + (purl.to_string(), action.to_string(), code.to_string()) + }; + + // The API has a patch of the release the lock holds: a dry run says + // what the scan will do, and writes nothing. + machine.make(&files, false, false); + let dry = machine.step(&[&SCAN[..], &["--dry-run"]].concat()); + assert_eq!( + triples(&dry), + [event(old, "verified", stale), event(PURL, "verified", "")], + "{dry:?}" + ); + assert_eq!( + (dry.code, &dry.wrote), + (Some(0), &BTreeMap::new()), + "{dry:?}" + ); + let scan = machine.step(&SCAN); + assert_eq!( + triples(&scan), + [event(old, "rolledBack", stale), event(PURL, "applied", "")], + "{scan:?}" + ); + let said = scan.envelope(); + let taken_out = &said["events"][0]; + assert_eq!(taken_out["uuid"], old_uuid, "{scan:?}"); + assert_eq!( + taken_out["reason"], + format!( + "patch {old_uuid} is for a jar that no pinned rules_jvm_external hub of the Bazel \ + workspace fetches any more, so Bazel stops at its entry in MODULE.bazel; the pin \ + was restored to the upstream registry entry" + ), + "{scan:?}" + ); + assert_eq!((scan.code, &said["warnings"]), (Some(0), &Value::Null)); + assert_eq!(scan.wrote.keys().collect::>(), [MODULE], "{scan:?}"); + assert_eq!( + String::from_utf8_lossy(&scan.wrote[MODULE]), + format!("{HUB}\n{BLOCK}"), + "{scan:?}" + ); + let again = machine.step(&SCAN); + assert_eq!(triples(&again), [event(PURL, "applied", "")], "{again:?}"); + assert_eq!(again.wrote, BTreeMap::new(), "{again:?}"); + + // No patch of the release the lock holds: the file is the one of + // before the first pin, to the byte, and a person reads why. + let moved = lock(&[("consumer", "2.0"), ("victim", "1.11.0")]); + let bump = || { + machine.make(&pinned(), false, false); + assert_eq!(machine.step(COMMANDS[0]).wrote.len(), 1); + std::fs::write(machine.root.join("proj/maven_install.json"), &moved).expect("the lock"); + }; + bump(); + let scan = machine.step(&SCAN); + assert_eq!( + triples(&scan), + [event(PURL, "rolledBack", stale)], + "{scan:?}" + ); + assert_eq!(scan.wrote.keys().collect::>(), [MODULE], "{scan:?}"); + assert_eq!( + String::from_utf8_lossy(&scan.wrote[MODULE]), + HUB, + "{scan:?}" + ); + bump(); + let person = machine.step(&["scan", "--yes"]); + let note = format!( + "Note: patch {UUID} for {PURL} is for a jar that no pinned rules_jvm_external hub of \ + the Bazel workspace fetches any more, so Bazel stops at its entry in MODULE.bazel; \ + restoring its upstream registry entry\n" + ); + assert_eq!(person.stderr.matches(¬e).count(), 1, "{person:?}"); + let restored = format!("Restored {PURL} to its upstream registry entry\n"); + assert!(person.stdout.contains(&restored), "{person:?}"); + assert_eq!( + String::from_utf8_lossy(&person.wrote[MODULE]), + HUB, + "{person:?}" + ); + + // A hub that fetches no jar at all: the scan has no package, and the + // entry goes all the same. + bump(); + let empty = json!({ "artifacts": {}, "dependencies": {}, "repositories": {}, "version": "3" }); + std::fs::write( + machine.root.join("proj/maven_install.json"), + empty.to_string(), + ) + .expect("it"); + let scan = machine.step(&SCAN); + assert_eq!(scan.envelope()["scannedPackages"], 0, "{scan:?}"); + assert_eq!( + triples(&scan), + [event(PURL, "rolledBack", stale)], + "{scan:?}" + ); + assert_eq!( + String::from_utf8_lossy(&scan.wrote[MODULE]), + HUB, + "{scan:?}" + ); + + // While a hub is in doubt it may be the one that fetches the jar: the + // entry stays. It stays in a scan that reads no hub, too. + bump(); + let module = machine.root.join("proj/MODULE.bazel"); + let pinned_text = std::fs::read_to_string(&module).expect("MODULE.bazel"); + let doubted = format!("{pinned_text}maven.install(name = \"tools\")\n"); + std::fs::write(&module, doubted).expect("MODULE.bazel"); + let scan = machine.step(&SCAN); + assert_eq!(triples(&scan), [], "{scan:?}"); + assert_eq!( + warning_codes(&scan.envelope()), + ["maven_bazel_hub_not_read"], + "{scan:?}" + ); + assert_eq!(scan.wrote, BTreeMap::new(), "{scan:?}"); + bump(); + let no_maven = machine.step(&[&SCAN[..], &["--ecosystems", "npm"]].concat()); + assert_eq!( + (no_maven.code, &no_maven.wrote), + (Some(0), &BTreeMap::new()), + "{no_maven:?}" + ); + + // Beside a `pom.xml` that still selects the release: the pin of the + // Maven build stays, and the entry alone goes. + machine.make(&pinned(), true, false); + assert_eq!(machine.step(COMMANDS[0]).wrote.len(), 4); + std::fs::write(machine.root.join("proj/maven_install.json"), &moved).expect("the lock"); + let scan = machine.step(&SCAN); + assert_eq!(scan.wrote.keys().collect::>(), [MODULE], "{scan:?}"); + assert_eq!( + String::from_utf8_lossy(&scan.wrote[MODULE]), + HUB, + "{scan:?}" + ); + assert_eq!( + triples(&scan), + [event(PURL, "rolledBack", stale), event(PURL, "applied", "")], + "{scan:?}" + ); + + // Beside a Gradle build the same: the entry alone goes, and the pin + // that the first scan wrote for Gradle stays for `list` to show and + // for `rollback ` to take out. + let mut files = pinned(); + files.extend(GRADLE.iter().map(|(path, text)| (*path, text.to_string()))); + machine.make(&files, false, false); + assert_eq!(machine.step(&SCAN).wrote.len(), 5); + std::fs::write(machine.root.join("proj/maven_install.json"), &moved).expect("the lock"); + let scan = machine.step(&SCAN); + assert_eq!(scan.wrote.keys().collect::>(), [MODULE], "{scan:?}"); + assert_eq!( + triples(&scan), + [event(PURL, "rolledBack", stale)], + "{scan:?}" + ); + let list = machine.step(&["list", "--json"]).envelope(); + assert_eq!( + event_for(&list, PURL)["details"]["lockfiles"], + json!([".socket/gradle/hosted-index.tsv"]), + "{list:#}" + ); +} + +/// The socket.yml policy holds a stale entry as it holds the packages of +/// the run: a scan of a root that the policy filters, and one whose +/// policy filters the package, leave the entry where it is. +#[test] +fn a_hosted_scan_leaves_a_stale_entry_that_the_policy_filters() { + let machine = Machine::start(); + let mut files = pinned(); + files[0].1 = format!( + "{HUB}\n{}", + block_at("1.9.0", "0a1b2c3d-1111-4222-8333-444455556666") + ); + let below: Vec<(String, String)> = files + .iter() + .map(|(path, text)| (format!("legacy/{path}"), text.clone())) + .collect(); + let mut below: Files = below + .iter() + .map(|(path, text)| (&*path.clone().leak(), text.clone())) + .collect(); + let filtered = |policy: &str| ("socket.yml", format!("version: 2\npatches:\n {policy}\n")); + below.push(filtered("ignorePaths: [\"/legacy/\"]")); + files.push(filtered( + "ignorePackages: [\"pkg:maven/dev.socket.fixture/victim\"]", + )); + for (files, dir) in [(&below, "proj/legacy"), (&files, "proj")] { + machine.make(files, false, false); + let scan = machine.step_in(&machine.root.join(dir), &SCAN); + let said = scan.envelope(); + assert_eq!( + (scan.code, &said["events"], &scan.wrote), + (Some(0), &json!([]), &BTreeMap::new()), + "{dir}: {scan:?}" + ); + } +} + +/// A checkout that Gradle or Maven builds too. A hub's jar with a patch is +/// a patch of the run like one it found installed, so the scan writes for +/// it what `get ` writes there: the Gradle pin beside the block, or +/// what the pom lane says of a `pom.xml` it cannot pin. +#[test] +fn a_hosted_scan_pins_a_hubs_jar_in_the_other_builds_as_get_does() { + let machine = Machine::start(); + // A pom whose dependency is not in the Maven local repository: what + // Maven selects cannot be told. + let other = PROJECT_POM + .replace("dev.socket.fixture", "com.example") + .replace("victim", "other"); + let rows: [(&[(&str, &str)], &[&str], &[&str]); 2] = [ + ( + &GRADLE, + &[ + "proj/.socket/gradle/.gitattributes", + "proj/.socket/gradle/hosted-index.tsv", + "proj/.socket/gradle/socket-patch.hosted.settings.gradle", + MODULE, + "proj/settings.gradle", + ], + &[ + "redirect_gradle_module_metadata_unavailable", + "redirect_gradle_detached_configs_unguarded", + ], + ), + ( + &[("pom.xml", &other)], + &[MODULE], + &["redirect_maven_selection_undetermined"], + ), + ]; + for (build, wrote, codes) in rows { + let mut files = pinned(); + files.extend(build.iter().map(|(path, text)| (*path, text.to_string()))); + machine.make(&files, false, false); + let get = machine.step(COMMANDS[0]); + machine.make(&files, false, false); + let scan = machine.step(&SCAN); + assert_eq!(scan.wrote.keys().collect::>(), wrote, "{scan:?}"); + assert_eq!(scan.wrote, get.wrote, "{scan:?}"); + let (scan, get) = (scan.envelope(), get.envelope()); + assert_eq!(event_for(&scan, PURL)["action"], "applied", "{scan:#}"); + assert_eq!(warning_codes(&scan), codes, "{scan:#}"); + assert_eq!(scan["warnings"], get["warnings"], "{scan:#}"); + } +} + +/// The version of a hub's jar is the lock's, whatever a `pom.xml` beside +/// the hub selects. Where Maven builds another release, the scan pins the +/// release Bazel builds, and says that it left the pom as it is. +#[test] +fn a_hosted_scan_pins_the_release_of_the_lock_beside_a_pom_of_another() { + let machine = Machine::start(); + let mut files = pinned(); + files.push(("pom.xml", PROJECT_POM.replace("1.10.0", "1.9.0"))); + machine.make(&files, false, false); + let scan = machine.step(&SCAN); + let envelope = scan.envelope(); + assert_eq!(event_for(&envelope, PURL)["action"], "applied", "{scan:?}"); + let codes = warning_codes(&envelope); + assert_eq!(codes, ["redirect_maven_version_not_selected"], "{scan:?}"); + assert_eq!(scan.wrote.keys().collect::>(), [MODULE], "{scan:?}"); + assert_eq!( + String::from_utf8_lossy(&scan.wrote[MODULE]), + format!("{HUB}\n{BLOCK}"), + "{scan:?}" + ); +} + +/// The words of the warning for a hub that was not read, after where and +/// why. +const NOT_READ: &str = "so socket-patch does not know every Maven artifact that Bazel builds here \ + through a rules_jvm_external hub: it looks for patches of the ones in the \ + hub locks it read, and of no other"; + +/// No jar is guessed for a hub that cannot be read: the scan says once +/// what it did not read, and takes the jars of the hubs it read. +#[test] +fn a_hub_that_a_scan_could_not_read_is_named_once() { + let machine = Machine::start(); + let scan = |bazel: &Files, args: &[&str]| { + machine.make(bazel, false, false); + machine.step(args) + }; + let warned = |ran: &Ran| ran.envelope()["warnings"].clone(); + let not_read = + |detail: String| json!([{ "code": "maven_bazel_hub_not_read", "detail": detail }]); + // A hub without a lock, alone: no package, and how to pin it. + let lockless = HUB.replace(" lock_file = \"//:maven_install.json\",\n", ""); + let unpinned = vec![("MODULE.bazel", lockless)]; + let detail = format!( + "MODULE.bazel:4: hub \"maven\" has no `lock_file`, {NOT_READ}. Give the hub a \ + `lock_file` and pin it (`REPIN=1 bazel run @maven//:pin`)" + ); + let ran = scan(&unpinned, &SCAN); + assert_eq!(ran.envelope()["scannedPackages"], 0, "{ran:?}"); + assert_eq!(warned(&ran), not_read(detail.clone()), "{ran:?}"); + assert_eq!( + (ran.code, &ran.wrote), + (Some(0), &BTreeMap::new()), + "{ran:?}" + ); + // A person is pointed at the warning, not at an install. + let person = scan(&unpinned, &["scan", "--yes"]); + assert_eq!(person.stdout, "No packages found (see the warning).\n"); + let line = format!("Warning: {detail}\n"); + assert_eq!(person.stderr.matches(&line).count(), 1, "{person:?}"); + + // A lock that is not there. + let [.., mut missing] = checkouts(); + missing.truncate(1); + let ran = scan(&missing, &SCAN); + let detail = format!("maven_install.json: it does not exist, {NOT_READ}"); + assert_eq!(warned(&ran), not_read(detail), "{ran:?}"); + assert_eq!(ran.envelope()["scannedPackages"], 0, "{ran:?}"); + + // A workspace that names no ruleset of a hub, with a file its reader + // cannot have: beside a `maven_install.json` it may have a hub. + let mut other = unread_other(); + other.push(("maven_install.json", lock(&[("victim", "1.10.0")]))); + let ran = scan(&other, &SCAN); + let gone = "more.MODULE.bazel: it does not exist; MODULE.bazel:6 includes it"; + assert_eq!( + warned(&ran), + not_read(format!("{gone}, {NOT_READ}")), + "{ran:?}" + ); + assert_eq!(ran.envelope()["scannedPackages"], 0, "{ran:?}"); + + // Beside a pinned hub: its jars are scanned, and the patch of one + // gets no entry while the other hub may build the upstream jar. + let mut beside = pinned(); + beside[0].1.push_str("maven.install(name = \"tools\")\n"); + let ran = scan(&beside, &SCAN); + let envelope = ran.envelope(); + assert_eq!(envelope["scannedPackages"], 3, "{ran:?}"); + assert_eq!( + warning_codes(&envelope), + ["maven_bazel_hub_not_read", "maven_bazel_not_pinned"], + "{ran:?}" + ); + let named = envelope["warnings"][0]["detail"].as_str().expect("text"); + assert!( + named.starts_with("MODULE.bazel:9: hub \"tools\" has no `lock_file`, so"), + "{named}" + ); + let event = event_for(&envelope, PURL); + assert_eq!( + (ran.code, &event["action"], &event["errorCode"], &ran.wrote), + ( + Some(0), + &json!("skipped"), + &json!("redirect_unconfirmed"), + &BTreeMap::new() + ), + "{ran:?}" + ); +} + +/// A person's scan from the root of a checkout covers the project +/// directories below it, and the root when it is a project. The root of a +/// Bazel workspace needs no file that makes it one: it is a project of a +/// hosted scan when its hubs have jars, or one of them was not read. Of +/// any other scan it is none, as without Bazel. +#[test] +fn a_bare_scan_covers_a_bazel_root_above_other_projects() { + let machine = Machine::start(); + let package = json!({ "name": "web", "version": "1.0.0" }); + let web = |mut bazel: Files| { + let lock = json!({ "name": "web", "version": "1.0.0", "lockfileVersion": 3, + "packages": { "": package, "node_modules/left-pad": { "version": "1.3.0" } } }); + bazel.push(("web/package.json", package.to_string())); + bazel.push(("web/package-lock.json", lock.to_string())); + bazel + }; + let scan = |bazel: Files, args: &[&str]| { + machine.make(&web(bazel), false, false); + machine.step(args) + }; + let headings = |ran: &Ran| -> Vec { + let lines = ran.stdout.lines().filter(|line| line.starts_with("== ")); + lines.map(String::from).collect() + }; + let person = ["scan", "--yes"]; + let both = ["== . ==", "== web =="]; + + let ran = scan(pinned(), &person); + assert_eq!(ran.code, Some(0), "{ran:?}"); + assert_eq!(headings(&ran), both, "{ran:?}"); + assert_eq!(ran.wrote.keys().collect::>(), [MODULE], "{ran:?}"); + assert_eq!( + String::from_utf8_lossy(&ran.wrote[MODULE]), + format!("{HUB}\n{BLOCK}"), + "{ran:?}" + ); + + let lockless = HUB.replace(" lock_file = \"//:maven_install.json\",\n", ""); + let ran = scan(vec![("MODULE.bazel", lockless)], &person); + assert_eq!(ran.code, Some(0), "{ran:?}"); + assert_eq!(headings(&ran), both, "{ran:?}"); + let named = "Warning: MODULE.bazel:4: hub \"maven\" has no `lock_file`, so"; + assert_eq!(ran.stderr.matches(named).count(), 1, "{ran:?}"); + assert_eq!(ran.wrote, BTreeMap::new(), "{ran:?}"); + + // A hub that fetches no jar, and a stale entry of the block: the + // root is scanned for that entry alone, and it goes. + let empty = json!({ "artifacts": {}, "dependencies": {}, "repositories": {}, "version": "3" }); + let stale = vec![ + ("MODULE.bazel", format!("{HUB}\n{BLOCK}")), + ("maven_install.json", empty.to_string()), + ]; + let ran = scan(stale, &person); + assert_eq!(headings(&ran), both, "{ran:?}"); + assert_eq!(String::from_utf8_lossy(&ran.wrote[MODULE]), HUB, "{ran:?}"); + + // The root is `--cwd`, which the user named: it is scanned in a + // directory that the built-in ignores leave out of a search, where + // the project below it is left out. + let under = |(path, text): (&str, String)| -> (&'static str, String) { + (Box::leak(format!("testdata/{path}").into_boxed_str()), text) + }; + let nested: Files = web(pinned()).into_iter().map(under).collect(); + machine.make(&nested, false, false); + let ran = machine.step_in(&machine.root.join("proj/testdata"), &person); + assert_eq!(headings(&ran), [""; 0], "{ran:?}"); + assert_eq!( + String::from_utf8_lossy(&ran.wrote["proj/testdata/MODULE.bazel"]), + format!("{HUB}\n{BLOCK}"), + "{ran:?}" + ); + + let [plain, other, ..] = checkouts(); + let vendored = ["scan", "--mode", "vendored", "--yes"]; + let no_maven = ["scan", "--yes", "--ecosystems", "npm"]; + let rows: [(Files, &[&str]); 4] = [ + (other, &person), + (unread_other(), &person), + (pinned(), &vendored), + (pinned(), &no_maven), + ]; + for (bazel, args) in rows { + let without = scan(plain.clone(), args); + assert_eq!(without.code, Some(0), "{args:?}: {without:?}"); + assert_eq!(headings(&without), [""; 0], "{args:?}: {without:?}"); + assert_eq!(scan(bazel, args), without, "{args:?}"); + } +} + +/// A scan that patches no hub does not take a hub's jars for packages: +/// one in vendored or agent mode, one that leaves Maven out. And a hosted +/// scan finds no hub where none is declared at the project: with a lock +/// of that name and no workspace file, with another ruleset's hub, and in +/// a project below the Bazel root. Each is the scan without Bazel. +#[test] +fn a_scan_that_reads_no_hub_reports_as_without_bazel() { + let machine = Machine::start(); + let [plain, other, ..] = checkouts(); + let lock_alone = vec![("maven_install.json", lock(&[("victim", "1.10.0")]))]; + let no_maven = [&SCAN[..], &["--ecosystems", "npm"]].concat(); + let unread = unread_other(); + let rows: [(&Files, bool, &[&str]); 8] = [ + (&pinned(), false, &["scan", "--mode", "agent", "--json"]), + (&pinned(), false, &["scan", "--mode", "vendored", "--json"]), + (&pinned(), false, &no_maven), + (&other, false, &SCAN), + (&other, true, &SCAN), + (&unread, false, &SCAN), + (&lock_alone, false, &SCAN), + (&lock_alone, true, &SCAN), + ]; + for (bazel, maven, args) in rows { + let scan = |bazel: &Files| { + machine.make(bazel, maven, false); + machine.step(args) + }; + let without = scan(&plain); + assert_eq!(without.code, Some(0), "{args:?}: {without:?}"); + assert_eq!(scan(bazel), without, "{args:?} beside {:?}", bazel[0].0); + } + let below = machine.root.join("proj/services/billing"); + let scan_below = |bazel: &Files| { + machine.make(bazel, false, false); + std::fs::create_dir_all(&below).expect("a project"); + machine.step_in(&below, &SCAN) + }; + assert_eq!(scan_below(&pinned()), scan_below(&plain)); +} + +/// `get` by an advisory keeps, of the releases the advisory has patches +/// for, the ones the project has. In hosted mode the jars of a pinned hub +/// count, as they do for a hosted scan, and a hub that was not read is +/// named. +#[test] +fn a_hosted_get_by_advisory_keeps_the_release_a_pinned_hub_fetches() { + use wiremock::matchers::{method, path}; + let machine = Machine::start(); + let ghsa = "GHSA-test-aaaa-bbbb"; + let found = json!({ "uuid": UUID, "purl": PURL, "publishedAt": "2024-01-01T00:00:00Z", + "description": "fixture", "license": "MIT", "tier": "free", "vulnerabilities": {} }); + let answer = json!({ "patches": [found], "canAccessPaidPatches": false }); + let search = wiremock::Mock::given(method("GET")) + .and(path(format!("/v0/orgs/{ORG}/patches/by-ghsa/{ghsa}"))) + .respond_with(wiremock::ResponseTemplate::new(200).set_body_json(answer)); + machine._runtime.block_on(search.mount(&machine.api)); + let get = |bazel: &Files| { + machine.make(bazel, false, false); + machine.step(&["get", ghsa, "--mode", "hosted", "--yes", "--json"]) + }; + let [plain, ..] = checkouts(); + let without = get(&plain); + let event = event_for(&without.envelope(), PURL).clone(); + assert_eq!( + (&event["action"], &event["errorCode"], &without.wrote), + ( + &json!("skipped"), + &json!("package_not_installed"), + &BTreeMap::new() + ), + "{without:?}" + ); + let beside = get(&pinned()); + let event = event_for(&beside.envelope(), PURL).clone(); + assert_eq!( + (beside.code, &event["action"]), + (Some(0), &json!("applied")) + ); + assert_eq!( + String::from_utf8_lossy(&beside.wrote[MODULE]), + format!("{HUB}\n{BLOCK}"), + "{beside:?}" + ); + let lockless = HUB.replace(" lock_file = \"//:maven_install.json\",\n", ""); + let unread = get(&vec![("MODULE.bazel", lockless)]); + let envelope = unread.envelope(); + let codes = warning_codes(&envelope); + assert_eq!(codes, ["maven_bazel_hub_not_read"], "{unread:?}"); + assert_eq!( + (&event_for(&envelope, PURL)["errorCode"], &unread.wrote), + (&json!("package_not_installed"), &BTreeMap::new()), + "{unread:?}" + ); + + // An advisory with no Maven artifact: the hub that was not read has + // nothing to do with it, and is not named. + let (npm, other) = ("pkg:npm/left-pad@1.3.0", "GHSA-test-cccc-dddd"); + let found = json!({ "uuid": UUID, "purl": npm, "publishedAt": "2024-01-01T00:00:00Z", + "description": "fixture", "license": "MIT", "tier": "free", "vulnerabilities": {} }); + let answer = json!({ "patches": [found], "canAccessPaidPatches": false }); + let search = wiremock::Mock::given(method("GET")) + .and(path(format!("/v0/orgs/{ORG}/patches/by-ghsa/{other}"))) + .respond_with(wiremock::ResponseTemplate::new(200).set_body_json(answer)); + machine._runtime.block_on(search.mount(&machine.api)); + let lockless = HUB.replace(" lock_file = \"//:maven_install.json\",\n", ""); + machine.make(&[("MODULE.bazel", lockless)], false, false); + let unrelated = machine.step(&["get", other, "--mode", "hosted", "--yes", "--json"]); + let envelope = unrelated.envelope(); + assert_eq!( + ( + &envelope["warnings"], + &event_for(&envelope, npm)["errorCode"] + ), + (&Value::Null, &json!("package_not_installed")), + "{unrelated:?}" + ); +} + #[test] fn a_global_run_says_nothing() { let machine = Machine::start(); diff --git a/crates/socket-patch-core/src/bazel/maven.rs b/crates/socket-patch-core/src/bazel/maven.rs index 188c8b8d4..c8b52bd51 100644 --- a/crates/socket-patch-core/src/bazel/maven.rs +++ b/crates/socket-patch-core/src/bazel/maven.rs @@ -21,7 +21,8 @@ //! as they are. [`jars`] are the patches of a run that a hub can take, //! [`site`] says for each whether its entry may be written, [`rewrite`] //! writes it, [`pins`] reads the block for `list`, `rollback`, `remove` -//! and `vex`, and [`without`] takes an entry out. +//! and `vex`, and [`without`] takes an entry out. [`hub_jars`] are the +//! jars a hosted scan looks for patches of: those the pinned hubs fetch. //! The block is recognised by its text alone: one that is not, byte for //! byte, what socket-patch writes for its entries was changed by hand, and //! is neither written to nor trusted. @@ -38,9 +39,10 @@ //! Bazel without a hub, and what a flag or an rc file outside the checkout //! tells Bazel. -use std::collections::{BTreeMap, HashMap, HashSet}; +use std::collections::{BTreeMap, BTreeSet, HashMap, HashSet}; use std::ops::Range; use std::path::Path; +use std::rc::Rc; use super::rc::{declared_release, mentions, ModuleLock}; use super::starlark::{Bound, Call, How, Value}; @@ -52,7 +54,7 @@ use crate::patch::redirect::{ use crate::utils::digest::is_hex64_lower; use crate::utils::fs::run_blocking; use crate::utils::line_endings::terminator; -use crate::utils::purl::{parse_maven_purl, purl_qualifier}; +use crate::utils::purl::{build_maven_purl, parse_maven_purl, purl_qualifier}; use crate::vendor::jvm::layout::artifact_path; /// The extension whose tags make a hub. @@ -68,6 +70,10 @@ pub const HUB_UNPATCHED: &str = "maven_bazel_hub_unpatched"; /// The warning for an artifact when the hubs of the workspace cannot be /// read: a hub may build it. pub const UNREADABLE: &str = "maven_bazel_unreadable"; +/// The warning of a hosted run that looks for the packages of a project +/// ([`hub_jars`]) when not every hub of the workspace could be read: the +/// jars of a hub that was not read are not among them. +pub const HUB_NOT_READ: &str = "maven_bazel_hub_not_read"; /// The warnings of a hosted run for a jar that a hub builds, or may /// build, and whose entry it does not write. Each says why, and that the @@ -114,8 +120,12 @@ const RULESETS: [&str; 10] = [ /// What a doubt about a hub without a lock ends with. const NO_LOCK: &str = "has no `lock_file`"; -/// The largest lock that is read. +/// The largest lock that is read, each lock once for every hub that names +/// it, and how many bytes of locks a scan takes its packages from in one +/// workspace ([`hub_jars`]): the first lock that does not fit is the last +/// one read. const MAX_LOCK: usize = 16 << 20; +const MAX_LOCKS: usize = 4 * MAX_LOCK; /// The tags that make a hub of their `name` or add to it, `install` /// first: the ruleset's `private/extensions/maven.bzl` has `artifact` at @@ -445,7 +455,7 @@ fn asked(ws: &Workspace) -> Asked { /// What the hubs of a workspace build: read once, asked per artifact. #[derive(Debug)] pub struct Hubs { - pinned: Vec<(Hub, Lock)>, + pinned: Vec<(Hub, Rc)>, /// The `override` tags of the root module: the [`target`] each /// replaces (`None`: its `coordinates` are no string, so any), and /// where it is. The tag's `name` is not asked for: the ruleset at 6.5 @@ -462,6 +472,11 @@ pub struct Hubs { edited: Option, /// What the root says of the release of the ruleset. asked: Asked, + /// Whether the workspace has to do with the ruleset: a call of a file + /// that was read names it, its `WORKSPACE` file calls `maven_install`, + /// or a `maven_install.json` lies at the root. In another workspace a + /// doubt is about no hub ([`hub_jars`]). + ruleset: bool, } impl Hubs { @@ -472,6 +487,17 @@ impl Hubs { /// `maven_install` or cannot be read while Bazel may read it (the /// workspace names no release, or one before 9) are doubts. pub fn read(view: &dyn Fn(&str) -> Entry, locks: &dyn Fn(&str) -> Entry) -> Hubs { + Hubs::read_within(view, locks, usize::MAX) + } + + /// [`Hubs::read`] with at most `room` bytes of locks in all: the lock + /// that does not fit is a doubt, and so is every lock after it, which + /// is not read. + fn read_within( + view: &dyn Fn(&str) -> Entry, + locks: &dyn Fn(&str) -> Entry, + mut room: usize, + ) -> Hubs { // The block is socket-patch's own: the workspace is read without // it, as the user wrote it, and with every line where it was. A // block that was changed by hand is read like any other text. @@ -515,17 +541,38 @@ impl Hubs { }); let overrides = overrides.collect(); let (mut pinned, mut lockless) = (Vec::new(), None); + let mut read_locks = HashMap::new(); + let past = || { + let bound = MAX_LOCKS >> 20; + Err(format!( + "the locks of the workspace are larger than {bound} MiB in all" + )) + }; for hub in hubs { // Where a doubt about its lock is: the lock, or the tag. let (file, line) = match &hub.lock { Some(lock) => (lock.clone(), 0), None => (hub.file.clone(), hub.line), }; - let lock = match hub.lock.as_deref().map(locks) { + let lock = match &hub.lock { None => Err(format!("hub \"{}\" {NO_LOCK}", hub.name)), - Some(Entry::Text(text)) => Lock::parse(&text), - Some(Entry::Absent) => Err("it does not exist".to_string()), - Some(Entry::Unreadable(why)) => Err(format!("it cannot be read: {why}")), + Some(path) => { + let parsed = || match (room > 0).then(|| locks(path)) { + None => past(), + Some(Entry::Text(text)) if text.len() > room => { + room = 0; + past() + } + Some(Entry::Text(text)) => { + room -= text.len(); + Lock::parse(&text).map(Rc::new) + } + Some(Entry::Absent) => Err("it does not exist".to_string()), + Some(Entry::Unreadable(why)) => Err(format!("it cannot be read: {why}")), + }; + let lock = read_locks.entry(path.clone()).or_insert_with(parsed); + lock.clone() + } }; match lock { Ok(lock) => pinned.push((hub, lock)), @@ -536,8 +583,16 @@ impl Hubs { } } } + let module = Value::Str(EXTENSION.module.to_string()); + let names = + |call: &Call| [call.arg("name"), call.arg("module_name")].contains(&Some(&module)); + let named = ws + .files + .iter() + .any(|(_, file)| file.calls.iter().any(names)); + let calls = matches!(ws.legacy, Some(Ok(_))); let before_9 = || declared_release(view).is_none_or(|release| release < (9, 0, 0)); - doubts.extend(match ws.legacy.filter(|_| before_9()) { + let legacy = match ws.legacy.filter(|_| before_9()) { Some(Ok((file, read))) => { // By its own name, or by one that a `load` gives it. let symbol = Value::Str("maven_install".to_string()); @@ -556,7 +611,10 @@ impl Hubs { }) } unread => unread.and_then(Result::err), - }); + }; + let ruleset = + named || (calls && legacy.is_some()) || view("maven_install.json") != Entry::Absent; + doubts.extend(legacy); Hubs { pinned, overrides, @@ -565,6 +623,7 @@ impl Hubs { block, edited, asked, + ruleset, } } @@ -852,13 +911,17 @@ type Declined = (&'static str, String); const KEEPS: &str = "socket-patch wrote nothing to MODULE.bazel for it, so the Bazel build keeps \ the upstream jar"; +/// Where a doubt is: its file, with its line when it is about one. +fn at(doubt: &Doubt) -> String { + match doubt.line { + 0 => doubt.file.clone(), + line => format!("{}:{line}", doubt.file), + } +} + /// Why no entry is written for a jar that a hub may build. fn unknown(doubt: &Doubt, kind: Kind) -> Declined { - let Doubt { file, line, why } = doubt; - let at = match line { - 0 => file.clone(), - line => format!("{file}:{line}"), - }; + let (at, why) = (at(doubt), &doubt.why); match kind { Kind::Unpinned(hub) => { let why = format!( @@ -1281,6 +1344,83 @@ pub fn pins(root: &Path, module: &str) -> Result, String> { Ok(hubs.block.iter().map(pin).collect()) } +/// What a hosted scan takes from the hubs of a Bazel workspace +/// ([`hub_jars`]). +#[derive(Debug, Default, PartialEq, Eq)] +pub struct HubJars { + /// The purl of each plain jar that a pinned hub fetches, at the + /// version its lock holds. + pub jars: Vec, + /// The warning ([`HUB_NOT_READ`]) for the first thing that was not + /// read. + pub unread: Option<(&'static str, String)>, + /// The entries of the block whose jar no pinned hub fetches any more, + /// when every hub was read: the purl and the id of the patch of each. + /// Bazel stops at such an entry. + pub stale: Vec<(String, String)>, +} + +/// What a hosted run takes for packages of `project` beside the ones it +/// finds installed, when `project` is the root of a Bazel workspace: the +/// purl of each plain jar that a pinned hub fetches, at the version its +/// lock holds. That is the version Bazel builds, whatever a `pom.xml` +/// beside the hub selects. With them, one warning ([`HUB_NOT_READ`]) for +/// the first thing that was not read: no jar is guessed for a hub without +/// a lock, or for one whose tags or lock cannot be read. In a workspace +/// that has nothing to do with the ruleset there is no hub to read, and +/// no warning. And the entries of the block that are stale, for the run +/// to take out. Nothing for a project below the root, where no entry is +/// written either, and for one in no Bazel workspace. +pub async fn hub_jars(project: &Path) -> HubJars { + let project = project.to_path_buf(); + run_blocking(move || hub_jars_blocking(&project)).await +} + +fn hub_jars_blocking(project: &Path) -> HubJars { + let root = super::bazel_root(project).filter(|root| root.project.as_os_str().is_empty()); + let Some(root) = root else { + return HubJars::default(); + }; + let locks = disk_up_to(&root.dir, MAX_LOCK); + let hubs = Hubs::read_within(&disk(&root.dir), &locks, MAX_LOCKS); + // Each lock once, however many hubs name it. + let (mut purls, mut seen) = (BTreeSet::new(), HashSet::new()); + let locks = hubs.pinned.iter().map(|(_, lock)| lock); + let locks = locks.filter(|lock| seen.insert(Rc::as_ptr(lock))); + for (key, jar) in locks.flat_map(|lock| &lock.jars) { + let (group, artifact) = key.split_once(':').unwrap_or_default(); + if jar.fetched { + purls.insert(build_maven_purl(group, artifact, &jar.version)); + } + } + let unread = hubs.doubt().filter(|_| hubs.ruleset).map(|(doubt, kind)| { + let pin = match kind { + Kind::Unpinned(hub) => format!( + ". Give the hub a `lock_file` and pin it (`REPIN=1 bazel run @{hub}//:pin`)" + ), + Kind::Unread | Kind::Overridden => String::new(), + }; + let detail = format!( + "{}: {}, so socket-patch does not know every Maven artifact that Bazel builds here \ + through a rules_jvm_external hub: it looks for patches of the ones in the hub \ + locks it read, and of no other{pin}", + at(doubt), + doubt.why + ); + (HUB_NOT_READ, detail) + }); + let stale = + |patch: &&Patch| hubs.built(&patch.group, &patch.artifact, &patch.version) == Built::No; + let stale = hubs.block.iter().filter(stale); + HubJars { + jars: purls.into_iter().collect(), + unread, + stale: stale + .map(|patch| (patch.purl(), patch.uuid.clone())) + .collect(), + } +} + /// The warnings for the patches `purls` of a command that ran in /// `project`: a code and a detail for each Maven jar that a hub of the /// project's Bazel workspace builds ([`HUB_UNPATCHED`]) or may build @@ -1326,14 +1466,12 @@ fn notices_blocking(project: &Path, mut purls: Vec) -> Vec<(&'static str ); (HUB_UNPATCHED, detail) } - Built::Unknown(Doubt { file, line, why }, _) => { - let at = match line { - 0 => file.clone(), - line => format!("{file}:{line}"), - }; + Built::Unknown(doubt, _) => { let detail = format!( "socket-patch could not tell whether Bazel takes this artifact from \ - a rules_jvm_external hub: {at}: {why}" + a rules_jvm_external hub: {}: {}", + at(doubt), + doubt.why ); (UNREADABLE, detail) } @@ -2172,6 +2310,238 @@ jvm.artifact(group = "com.example", artifact = "a", version = "1") assert_eq!(notices_blocking(broken.path(), Vec::new()), []); } + /// What the warning for a hub that was not read says after where and + /// why. + const NOT_READ: &str = "so socket-patch does not know every Maven artifact that Bazel builds \ + here through a rules_jvm_external hub: it looks for patches of the \ + ones in the hub locks it read, and of no other"; + + #[test] + fn a_hosted_scan_takes_the_plain_jars_that_pinned_hubs_fetch() { + let netty = |artifact| format!("pkg:maven/io.netty/netty-tcnative-{artifact}@2.0.61.Final"); + let netty = [netty("boringssl-static"), netty("classes")]; + let netty: Vec<&str> = netty.iter().map(String::as_str).collect(); + let sqlite = "pkg:maven/com.almworks.sqlite4java/sqlite4java@1.0.392"; + let local = LOCK_SKIPS.replace( + "\"version\": \"3\"", + "\"m2local\": true, \"version\": \"3\"", + ); + // A version and an artifact that no repository can be named for: + // the run that finds a patch of one says so (`maven_bazel_artifact`). + let odd = r#"{"artifacts": { + "a.b:c": {"shasums": {"jar": "00"}, "version": "1.0+local"}, + "a.b:d@e": {"shasums": {"jar": "11"}, "version": "1.0"}, + "a.b:f": {"shasums": {"jar": "22"}, "version": "1.0"}}, + "repositories": {"https://repo.example/": ["a.b:c", "a.b:d@e", "a.b:f"]}, + "version": "3"}"#; + let odd_jars = + ["c@1.0+local", "d@e@1.0", "f@1.0"].map(|jar| format!("pkg:maven/a.b/{jar}")); + let two = + format!("{PINNED}maven.install(name = \"tools\", lock_file = \"//:tools.json\")\n"); + let lockless = format!("{PINNED}maven.install(name = \"tools\")\n"); + let (v2, v3) = (lock_69(), lock_72()); + // A lock larger than the 1 MiB of a Starlark file. + let large = lock_72() + &" ".repeat(1 << 20); + let main = "maven_install.json"; + let pin = ". Give the hub a `lock_file` and pin it (`REPIN=1 bazel run @tools//:pin`)"; + // A file the reader cannot have, in a workspace with a hub and in + // one that names no ruleset of Maven's. + let gone = "more.MODULE.bazel: it does not exist; MODULE.bazel"; + let more = format!("{PINNED}include(\"//:more.MODULE.bazel\")\n"); + let go = "bazel_dep(name = \"rules_go\", version = \"0.50.1\")\n\ + include(\"//:more.MODULE.bazel\")\n"; + // The block of an earlier run, for netty's jar. + let entry = patch("netty-tcnative-classes", NETTY_UUID); + let patched = with_patch(PINNED, LABEL, entry).expect("a block"); + let patched_lockless = format!("{patched}\nmaven.install(name = \"tools\")\n"); + let rows: [(&str, &[(&str, &str)], Vec<&str>, Option, bool); 19] = [ + // The jar without a classifier of each artifact, in each format. + (PINNED, &[(main, &v2)], netty.clone(), None, false), + (PINNED, &[(main, &v3)], netty.clone(), None, false), + (PINNED, &[(main, &large)], netty.clone(), None, false), + // Not one the lock skips or has no sum of, not another + // packaging, and not one that no repository lists, + (PINNED, &[(main, LOCK_SKIPS)], vec![sqlite], None, false), + // unless the hub takes it from the Maven local repository. + ( + PINNED, + &[(main, &local)], + vec![sqlite, "pkg:maven/org.example/unlisted@1.0"], + None, + false, + ), + ( + PINNED, + &[(main, odd)], + odd_jars.iter().map(String::as_str).collect(), + None, + false, + ), + // Every pinned hub, and a jar that two of them fetch once. + ( + &two, + &[(main, &v3), ("tools.json", LOCK_SKIPS)], + [&[sqlite], &netty[..]].concat(), + None, + false, + ), + ( + &two, + &[(main, &v3), ("tools.json", &v2)], + netty.clone(), + None, + false, + ), + // What was not read is said, once, and guessed at nowhere: + // the hubs that were read still count. + ( + &lockless, + &[(main, &v3)], + netty.clone(), + Some(format!( + "MODULE.bazel:9: hub \"tools\" has no `lock_file`, {NOT_READ}{pin}" + )), + false, + ), + ( + &two, + &[(main, &v3)], + netty.clone(), + Some(format!("tools.json: it does not exist, {NOT_READ}")), + false, + ), + ( + PINNED, + &[], + vec![], + Some(format!("maven_install.json: it does not exist, {NOT_READ}")), + false, + ), + ( + &more, + &[], + vec![], + Some(format!("{gone}:9 includes it, {NOT_READ}")), + false, + ), + // Not in a workspace that has nothing to do with the ruleset: + // what its reader could not have is about no hub. A lock at + // the root says that it may have one, and so does a + // `maven_install` of its `WORKSPACE` file. + (go, &[], vec![], None, false), + ( + go, + &[(main, &v3)], + vec![], + Some(format!("{gone}:2 includes it, {NOT_READ}")), + false, + ), + ( + "module(name = \"app\")\n", + &[("WORKSPACE", "maven_install(artifacts = [])\n")], + vec![], + Some(format!( + "WORKSPACE:1: a `maven_install` of a WORKSPACE file is not read, {NOT_READ}" + )), + false, + ), + // An entry of the block is stale once every hub was read and + // none fetches its jar at that version any more. + (&patched, &[(main, &v3)], netty.clone(), None, false), + (&patched, &[(main, LOCK_SKIPS)], vec![sqlite], None, true), + ( + &patched_lockless, + &[(main, LOCK_SKIPS)], + vec![sqlite], + Some(format!( + "MODULE.bazel:32: hub \"tools\" has no `lock_file`, {NOT_READ}{pin}" + )), + false, + ), + ( + &patched, + &[], + vec![], + Some(format!("maven_install.json: it does not exist, {NOT_READ}")), + false, + ), + ]; + for (module, locks, jars, unread, stale) in rows { + let dir = project(&[&[("MODULE.bazel", module)], locks].concat()); + let expected = HubJars { + jars: jars.into_iter().map(String::from).collect(), + unread: unread.map(|detail| (HUB_NOT_READ, detail)), + stale: Vec::from_iter(stale.then(|| (NETTY_PURL.into(), NETTY_UUID.into()))), + }; + assert_eq!(hub_jars_blocking(dir.path()), expected, "{module}"); + } + // Only for the root of a workspace, where an entry is written: + // not below it, whatever its hubs are, and not without one. + for module in [PINNED, &lockless] { + let dir = project(&[("MODULE.bazel", module), (main, &v3)]); + assert!(!hub_jars_blocking(dir.path()).jars.is_empty(), "{module}"); + let below = dir.path().join("services/billing"); + std::fs::create_dir_all(&below).unwrap(); + assert_eq!(hub_jars_blocking(&below), HubJars::default(), "{module}"); + } + let plain = project(&[("pom.xml", ""), (main, &v3)]); + assert_eq!(hub_jars_blocking(plain.path()), HubJars::default()); + } + + /// A lock is read once for all the hubs that name it. A scan takes + /// the locks of one workspace within a bound: the lock past it is a + /// doubt, like one that is too large by itself, and no lock after it + /// is read. Every other reader takes them all, as before. + #[test] + fn the_locks_of_a_workspace_are_read_once_and_within_a_bound() { + let hub = |name: &str, lock: &str| { + format!("maven.install(name = \"{name}\", lock_file = \"//:{lock}.json\")\n") + }; + let hubs = [ + ("a", "a"), + ("b", "a"), + ("c", "c"), + ("d", "d"), + ("e", "e"), + ("f", "f"), + ]; + let module = hubs + .iter() + .fold(PINNED.to_string(), |module, (name, lock)| { + module + &hub(name, lock) + }); + let full = lock_72() + &" ".repeat(MAX_LOCK - lock_72().len()); + let asked = std::cell::RefCell::new(Vec::new()); + let view = |path: &str| { + asked.borrow_mut().push(path.to_string()); + match path { + "MODULE.bazel" => Entry::Text(module.clone()), + "maven_install.json" | "f.json" => Entry::Text(lock_72()), + "a.json" | "c.json" | "d.json" | "e.json" => Entry::Text(full.clone()), + _ => Entry::Absent, + } + }; + let names = |(hub, _): &(Hub, Rc)| hub.name.clone(); + let read = Hubs::read_within(&view, &view, MAX_LOCKS); + let pinned: Vec = read.pinned.iter().map(names).collect(); + assert_eq!(pinned, ["maven", "a", "b", "c", "d"]); + let doubt = read.doubt.expect("a doubt"); + let why = "the locks of the workspace are larger than 64 MiB in all"; + assert_eq!((doubt.file.as_str(), doubt.why.as_str()), ("e.json", why)); + let times = |lock: &str| asked.borrow().iter().filter(|path| *path == lock).count(); + assert_eq!( + ["a.json", "c.json", "e.json", "f.json"].map(times), + [1, 1, 1, 0] + ); + + asked.borrow_mut().clear(); + let read = Hubs::read(&view, &view); + let pinned: Vec = read.pinned.iter().map(names).collect(); + assert_eq!(pinned, ["maven", "a", "b", "c", "d", "e", "f"]); + assert!(read.doubt.is_none()); + assert_eq!(["a.json", "e.json", "f.json"].map(times), [1, 1, 1]); + } + // The block, and what a hosted run writes of it. const NETTY_UUID: &str = "0e77a000-0000-4000-8000-0000000000a1"; @@ -2446,7 +2816,7 @@ override_repo( let hubs = |module: &str| { let files = [("MODULE.bazel", module), ("maven_install.json", &lock)]; let read = read(&files); - let hub = |(hub, _): &(Hub, Lock)| format!("{} {}:{}", hub.name, hub.file, hub.line); + let hub = |(hub, _): &(Hub, _)| format!("{} {}:{}", hub.name, hub.file, hub.line); let hubs: Vec = read.pinned.iter().map(hub).collect(); (hubs, read.doubt, built(&files)) }; diff --git a/docs/ecosystems.md b/docs/ecosystems.md index ab1d2884b..9405515bc 100644 --- a/docs/ecosystems.md +++ b/docs/ecosystems.md @@ -1863,9 +1863,9 @@ Maven local repository, at the lock's sha256 too. So a patch in a `pom.xml`, a G an installed jar does not reach what Bazel builds. Hosted mode patches a pinned hub. The hub fetches each jar through a repository of its own, -and Bazel's `override_repo` (7.4.0 and later) puts another repository in its place. `socket-patch -get --mode hosted`, run at the Bazel root, appends one block to the root `MODULE.bazel`, -after one blank line: +and Bazel's `override_repo` (7.4.0 and later) puts another repository in its place. A hosted +`socket-patch scan` or `socket-patch get --mode hosted`, run at the Bazel root, appends +one block to the root `MODULE.bazel`, after one blank line: ``` # >>> socket-patch: Maven patches for Bazel (managed block: socket-patch writes and removes it) >>> @@ -1929,8 +1929,8 @@ to the `)` of its `override_repo`, and the entries stand in the order of their r Everything is read from the checkout: the root `MODULE.bazel` and the files it includes, for the tags of the `maven` extension of `rules_jvm_external` under whatever name the file gives the ruleset and the extension (a hub is the `install`, `artifact` and `from_toml` tags of one -`name`); the lock each hub names with `lock_file` (format 2 or 3, up to 16 MiB; each other file -up to 1 MiB); `.bazeliskrc`, `.bazelversion`, `MODULE.bazel.lock` and `.bazelrc` with the files +`name`); the lock each hub names with `lock_file` (format 2 or 3, up to 16 MiB, each lock once +for all the hubs that name it; each other file up to 1 MiB); `.bazeliskrc`, `.bazelversion`, `MODULE.bazel.lock` and `.bazelrc` with the files it imports (`import`, `try-import` and `try-import-if-bazel-version`, whose condition is not evaluated; a path under `%workspace%/`, or a relative one, which is read from the Bazel root). An rc file is read as Bazel reads one: a `\` before a line break joins two lines, a `#` @@ -1942,6 +1942,72 @@ no Bazel workspace (the rule is under as a jar without a classifier at the patch's version and every hub was read, nothing is read further, said or written: the run is the one it was. +A hosted `scan` at the Bazel root takes the jars that the pinned hubs fetch for packages of +the project, beside the ones it finds installed, so it finds their patches in a tree that has +no `pom.xml` and no Gradle build. A hub's jar is the one without a classifier of an artifact in +the hub's lock, when the lock has a sha256 for it, does not skip it, and lists it under a +repository (or takes it from the Maven local repository under `m2local`). Its version is the +lock's: the one Bazel builds, whatever a `pom.xml` beside the hub selects. A jar whose +repository has no known name is a package too: a patch found for it gets no entry, and +`maven_bazel_artifact` below says why. + +- **In the output.** Such a package counts in `scannedPackages`, and unless a crawl found that + version installed also in `lockfileOnlyPackages`, with `notInstalled` on its `packages[]` + entry. A hosted `get ` keeps the releases that a pinned hub fetches the same way, + and names a hub that was not read only when the advisory has a patch of a Maven artifact. +- **What was not read.** No jar is guessed. When something of the hubs could not be read with + certainty (a hub without a `lock_file`, a lock or a tag that cannot be read, a `maven_install` + of a `WORKSPACE` file that Bazel may read, a block that was changed by hand), the scan says so + once, with the file, the line and the reason: `maven_bazel_hub_not_read` in `warnings[]`, a + `Warning:` line for a person, who reads "No packages found (see the warning)." when there is + no package at all. The jars of the hubs that were read are still looked up. A patch found + for one of them gets no entry while the doubt stands (`maven_bazel_not_pinned` or + `maven_bazel_unreadable` below). The warning is for a workspace that has to do with the + ruleset: a call in `MODULE.bazel` or a file it includes names `rules_jvm_external`, the + `WORKSPACE` file calls `maven_install`, or a `maven_install.json` lies at the Bazel root. In + another Bazel workspace a file that cannot be read is about no hub, and the scan is the one + without the workspace. +- **How much is read.** A scan takes its packages from up to 64 MiB of hub locks in one + workspace. The first lock that does not fit is not taken, no lock after it is read, and the + warning above names it ("the locks of the workspace are larger than 64 MiB in all"). That + bound is the scan's alone: a patch that is found is pinned as `get ` pins it, which + reads each lock it needs up to 16 MiB. +- **A stale entry.** When the lock of a hub moves on (a version bump, an artifact that was + removed), the entry for the release it held is stale: Bazel stops at it ("does not generate + repository"). A hosted scan takes such an entry out as `socket-patch rollback ` does, + before it pins the patch of the release the lock holds now. It reports a `rolledBack` event + (`verified` in a dry run) with the reason `maven_bazel_entry_stale`, and a person reads a + `Note:` and "Restored to its upstream registry entry". With the last entry the block + goes, and `MODULE.bazel` is the file of before its first patch. An entry is stale only when + every hub was read and none fetches the jar at the entry's version: while something is in + doubt it stays. The `socket.yml` policy holds such an entry as it holds a package of the + run: a scan of a root that the policy filters, or one whose policy filters the package, + leaves it. Only the entry goes. A pin of the same patch that another build of the checkout + holds stays (a `pom.xml`; the hosted Gradle pin, also where the Gradle build has no such + artifact), since nothing in the checkout says that the other build is done with the + release: `socket-patch list` shows it and `socket-patch rollback ` takes it out. + While a Gradle pin of the old release stays, the Gradle lane declines a patch of another + release of the artifact (`redirect_gradle_version_conflict`); the entry of the block is + written all the same. +- **Beside another build.** A hub's jar with a patch is a patch of the run like any other, so a + hosted scan writes for it what `get --mode hosted` writes in that checkout. Beside a + Gradle build that is the Gradle pin too (`settings.gradle` and three files under + `.socket/gradle/`, with the warnings of that pin), whether or not the Gradle build has the + artifact. Beside a `pom.xml`, the pom lane pins the release Maven selects and says of any + other what it always says: `redirect_maven_version_not_selected` where Maven selects another + release, `redirect_maven_selection_undetermined` where it cannot tell. Those two speak of + `pom.xml` alone: the hub's jar has its entry. +- **A root above other projects.** A bare `scan` covers the project directories below `--cwd`, + and `--cwd` when it is a project. The root of a Bazel workspace needs no lockfile or build + file of its own for that: a hosted scan covers it, under the heading `== . ==`, when its hubs + have jars, something of them was not read, or the block has a stale entry. That section is a + scan of `--cwd` like any other: it also counts the packages a crawl from the root finds + installed in the projects below. +- **Not for another scan.** A scan in vendored or agent mode, one whose `--ecosystems` leaves + Maven out, a `--global` one, and one of a project below the Bazel root take no package from a + hub, give no `maven_bazel_hub_not_read` and take no stale entry out: their packages are those + of the scan without the workspace. + A hosted run writes no entry where it cannot show from the checkout that the block works. It then gives one warning, which says why and that the Bazel build keeps the upstream jar. In a tree that only Bazel builds the patch is `skipped` (`redirect_unconfirmed`) and the warning @@ -2007,11 +2073,14 @@ Limits: - Not looked at: a patch of a jar with a classifier, or of another packaging than `jar`. A lock names the plain jar by group and artifact alone, so such a patch gets no entry and no word about the hub: the run is the one without the workspace. -- `socket-patch scan` does not find the artifacts of a hub yet: it pins a hub's jar when the - package is one it found elsewhere (a `pom.xml`, a Gradle build), and `get` takes any purl. It - does not take a stale entry out; `rollback` and `remove` do. +- A stale entry is taken out by a hosted `scan` at the Bazel root, by `rollback` and by + `remove`. `get --mode hosted` pins the purl it is given and leaves a stale entry of + another release where it is, so the build still stops there until one of those three ran. +- The in-memory hosted engine (GitHub App) reads no hub and writes no block. - Not seen: a hub or a `maven.override` that another Bazel module declares (its hub takes the - root's entry where it pins the same version), a `maven_install` inside a function of a `.bzl` + root's entry where it pins the same version, so an entry that is stale for the root's hubs + may still patch that one: a hosted scan takes it out all the same), a hub that the root only + takes with `use_repo` and other modules fill, a `maven_install` inside a function of a `.bzl` file, a jar that reaches Bazel without a hub (`http_jar`, a checked-in jar, the target of a `maven.override`), what the command line or an rc file outside the checkout tells Bazel (`--ignore_dev_dependency`, `--override_repository`; the machine's and the home directory's diff --git a/scripts/loc-budget.json b/scripts/loc-budget.json index 6729c125c..893f07e38 100644 --- a/scripts/loc-budget.json +++ b/scripts/loc-budget.json @@ -4,7 +4,7 @@ "paths": [ "crates/socket-patch-core/src/bazel/" ], - "max": 2486 + "max": 2583 }, "lock-readers": { "paths": [