From 21f6573a2a24b24d64c6701eeb8d5025d51d0efe Mon Sep 17 00:00:00 2001 From: Mikola Lysenko Date: Fri, 20 Nov 2026 13:35:37 -0500 Subject: [PATCH 1/3] Test a hosted scan of a tree that only Bazel builds on real Bazel `socket-patch scan` takes its Maven packages from the JVM caches of a project that has a pom.xml, a Gradle build or a Scala-tool build. A checkout whose jars only a rules_jvm_external hub fetches has none of those, so a hosted scan there knows no package and never asks the API, although `get --mode hosted` in the same directory pins the same jar in MODULE.bazel. Two flows of tests/e2e_bazel_build/maven.rs now hold that on real Bazel, so that a change which makes scan read the hub's lock has expectations to turn over: - a_hosted_scan_of_a_tree_only_bazel_builds (new): after the ruleset's pin and a build, `scan --mode hosted --yes` exits 0 with scannedPackages 0, no package and no event, sends no batch request and changes no file, and the build prints the upstream marker as before. The stand-in's API holds a patch for the victim the whole time. - a_version_bump_of_the_patched_artifact: the build stops at the stale entry and `rollback ` takes it out, as before. The flow then puts the stale entry back, as the file is for a user who has not run the rollback. The same scan finds no package there either and leaves the entry at which Bazel stops, although the API also grants a patch for the release the lock now pins (1.11.0). For that the stand-in grants two patches of the victim, one per release: grant() takes the version and hands the patch back, and the caller mounts the API once with both, because a second mount does not replace the answers of the first. Measured with Bazel 9.3.2 on macOS arm64, two flows at a time: the five Maven flows pass in 21.5, 24.3, 24.9, 30.9 and 22.1 seconds, and none of the 166 requests behind a logging proxy went to a public Maven repository (from a cold repository cache: 64.0, 71.6, 34.2, 60.5 and 29.8 seconds, 208 requests). No product code changes. Co-Authored-By: Claude Fable 5.2 --- .../tests/e2e_bazel_build/maven.rs | 130 ++++++++++++++---- 1 file changed, 106 insertions(+), 24 deletions(-) diff --git a/crates/socket-patch-cli/tests/e2e_bazel_build/maven.rs b/crates/socket-patch-cli/tests/e2e_bazel_build/maven.rs index c2d13aa7c..737298e6a 100644 --- a/crates/socket-patch-cli/tests/e2e_bazel_build/maven.rs +++ b/crates/socket-patch-cli/tests/e2e_bazel_build/maven.rs @@ -24,6 +24,11 @@ //! build fails on the sha256 and never goes to the upstream jar. On a //! Bazel before 7.5.0 the checkout first has no `.bazelversion`: the //! command then writes nothing and says so (`maven_bazel_release`). +//! - [`a_hosted_scan_of_a_tree_only_bazel_builds`]: the same checkout, and +//! `socket-patch scan --mode hosted --yes` in the place of the `get`. It +//! knows no Maven package in a tree without a `pom.xml` or a Gradle +//! build: it asks the API about none, reports none and changes no file, +//! while the API has a patch for a jar that the build takes. //! - [`a_pom_patch_beside_a_pinned_hub`]: a checkout that Maven builds //! too. The one command pins both builds: the three files of a Maven pin //! and `MODULE.bazel`. The Bazel build is patched, and `rollback --yes` @@ -33,7 +38,9 @@ //! passes. The build then stops at the line of the entry's //! `override_repo(`, where the block says what to run. `socket-patch //! rollback --yes` takes the entry out, and the build takes the -//! new upstream jar. +//! new upstream jar. The API has a patch of that release too, and a +//! hosted `scan` of the checkout with the stale entry does not find it +//! and leaves the entry: the lock is not where it looks. //! - [`the_users_own_override_of_the_artifact`]: the hub has a //! `maven.override` that puts a target of the checkout in the artifact's //! place. The command writes nothing and names the override's line @@ -94,7 +101,7 @@ use socket_patch_core::bazel::maven::{Lock, Pinned as Jar}; use crate::envelope::event_for; use crate::harness::{changed, rewrite, Cell, Fixture, Ran, StandIn, Tree, CONSUMER, NOTHING}; -use crate::hosted_memory_common::Patch; +use crate::hosted_memory_common::{Patch, ORG}; /// The workspace the tests build, and the ruleset each Bazel release /// builds it with. One era per way the ruleset names an artifact's @@ -121,6 +128,8 @@ const GROUP_PATH: &str = "dev/socket/fixture"; const VICTIM: &str = "victim"; const VERSION: &str = "1.10.0"; const PURL: &str = "pkg:maven/dev.socket.fixture/victim@1.10.0"; +/// The release a flow moves the checkout to, which has a patch too. +const BUMPED: &str = "1.11.0"; /// The file of the victim's jar that `App.java` prints. const MARKER: &str = "dev/socket/fixture/victim/marker.txt"; /// The fixture's hub, under a name no other Bazel module adds to. @@ -232,27 +241,29 @@ struct Granted { marker: String, } -/// Puts the grant of a new patch of the victim on the API of `standin`, -/// as production has it: a token and the patch id in the path of the patch -/// registry, the version with the suffix only that registry serves. The -/// route of the jar answers with `tampered`, or with the jar the grant -/// pins. -fn grant(standin: &StandIn, tampered: Option<&[u8]>) -> Granted { +/// The grant of a new patch of the victim at `version`, as production has +/// it: a token and the patch id in the path of the patch registry, the +/// version with the suffix only that registry serves. The route of the jar +/// on `standin` answers with `tampered`, or with the jar the grant pins. +/// Gives the patch for the API of the stand-in too, which takes every +/// patch it grants in one call ([`StandIn::api`]). +fn grant(standin: &StandIn, version: &str, tampered: Option<&[u8]>) -> (Granted, Patch) { let uuid = uuid::Uuid::new_v4().to_string(); let token = uuid::Uuid::new_v4().to_string(); - let suffixed = format!("{VERSION}-socket.{}", &uuid[..8]); + let purl = format!("pkg:maven/{GROUP}/{VICTIM}@{version}"); + let suffixed = format!("{version}-socket.{}", &uuid[..8]); let jar = victim(&format!("PATCHED {uuid}")); let registry = format!("{PATCH_HOST}/patch-registry/maven/{token}/{uuid}/maven2"); let file = repo_path(VICTIM, &suffixed, "jar"); let route = format!("/{registry}/{file}"); standin.serve(5, &route, tampered.unwrap_or(&jar)); let artifact = format!( - "https://{PATCH_HOST}/patch/maven/{GROUP}/{VICTIM}/{VERSION}/{token}/{uuid}/{VICTIM}-{suffixed}.jar" + "https://{PATCH_HOST}/patch/maven/{GROUP}/{VICTIM}/{version}/{token}/{uuid}/{VICTIM}-{suffixed}.jar" ); let reference = json!({ "status": "granted", "url": artifact, - "purl": PURL, + "purl": purl, "artifacts": [{ "kind": "tarball", "url": artifact, @@ -263,7 +274,7 @@ fn grant(standin: &StandIn, tampered: Option<&[u8]>) -> Granted { "indexUrl": format!("https://{registry}"), "identifiers": { "name": format!("{GROUP}/{VICTIM}"), - "version": VERSION, + "version": version, "mavenGroupId": GROUP, "mavenArtifactId": VICTIM, "mavenSuffixedVersion": suffixed, @@ -271,18 +282,19 @@ fn grant(standin: &StandIn, tampered: Option<&[u8]>) -> Granted { }, }, }); - standin.api(&[Patch { - purl: PURL.to_string(), + let patch = Patch { + purl, uuid: uuid.clone(), reference, - }]); - Granted { + }; + let granted = Granted { sha256: sha256_hex(&jar), url: format!("https://{registry}/{file}"), route, marker: format!("OUT: PATCHED {uuid}"), uuid, - } + }; + (granted, patch) } /// The name of the repository through which the fixture's hub fetches the @@ -329,8 +341,10 @@ struct Pinned { /// checkout leaves it; `coursier`, the cache the ruleset's Coursier /// fills when it pins; `gradle`, which nothing fills. tools: tempfile::TempDir, - /// The patch the stand-in grants, whose jar it serves. + /// The patch the stand-in grants, whose jar it serves, and the one it + /// grants for the victim at [`BUMPED`]. patch: Granted, + next: Granted, /// The id of the run, which every upstream jar made for it holds. run: String, /// What `//:consumer` prints while Bazel builds the upstream jar, and @@ -378,7 +392,9 @@ impl Pinned { &jar(&[]), Some((VICTIM, VERSION)), ); - let patch = grant(&standin, None); + let (patch, first) = grant(&standin, VERSION, None); + let (next, second) = grant(&standin, BUMPED, None); + standin.api(&[first, second]); // The checkout: the hub's repository is the stand-in, the lock is // the empty file the ruleset's documentation starts from. @@ -440,6 +456,7 @@ impl Pinned { standin, tools, patch, + next, run, upstream, upstream_route: format!("/{UPSTREAM}/{}", repo_path(VICTIM, VERSION, "jar")), @@ -502,6 +519,30 @@ impl Pinned { ); } + /// `socket-patch scan --mode hosted --yes` in a checkout without a + /// `pom.xml`, whose API has a patch for a jar of the hub's lock: the + /// command knows no package there. It asks the API about none, reports + /// none and changes no file of `before`. + fn scan_finds_no_package(&self, before: &Tree) { + let test = &self.cell.test; + let batch = format!("/v0/orgs/{ORG}/patches/batch"); + let asked = self.standin.asked(&batch); + let scan = self.socket_patch(&["scan", "--mode", "hosted", "--yes"]); + let said = &scan.dump; + assert_eq!(scan.code, 0, "{test}: {said}"); + assert_eq!(scan.envelope["scannedPackages"], 0, "{test}: {said}"); + assert_eq!(scan.envelope["packages"], json!([]), "{test}: {said}"); + assert_eq!(scan.envelope["events"], json!([]), "{test}: {said}"); + let rewritten = &scan.envelope["redirect"]["rewrittenFiles"]; + assert_eq!(*rewritten, json!([]), "{test}: {said}"); + assert_eq!(self.standin.asked(&batch), asked, "{test}: {said}"); + assert_eq!( + changed(before, &self.cell.tree()), + NOTHING, + "{test}: {said}" + ); + } + /// A Bazel before 7.5.0 writes a lock that does not tell it from one /// without `override_repo`, so its checkout names the release as a /// launcher reads it. Gives the files of the checkout. @@ -563,7 +604,8 @@ impl Pinned { let mut twin = self.cell.twin(); let rogue = StandIn::start(); let tampered = victim("TAMPERED"); - let other = grant(&rogue, Some(&tampered)); + let (other, served) = grant(&rogue, VERSION, Some(&tampered)); + rogue.api(&[served]); let host = format!("{PATCH_HOST}/"); twin.route_downloads(&[ rewrite(&host, &rogue.origin()), @@ -741,9 +783,39 @@ fn a_tree_only_bazel_builds() { ); } +/// A hosted scan of the checkout without a `pom.xml`: the lock of the hub +/// is the one file that names the jars Bazel builds, and the command does +/// not read it for packages. So it finds none, although the API has a +/// patch for the victim, and the build is the one of before. +#[test] +#[ignore = "real Bazel; run with --ignored"] +fn a_hosted_scan_of_a_tree_only_bazel_builds() { + let test = "maven::a_hosted_scan_of_a_tree_only_bazel_builds"; + let Some(flow) = Pinned::open(test, false, &|_, _| {}) else { + return; + }; + flow.builds_upstream(); + let before = flow.names_its_bazel(); + + flow.scan_finds_no_package(&before); + flow.cell.builds(&flow.upstream); + let asked = flow.standin.asked(&flow.patch.route); + assert_eq!(asked, 0, "{test}: something fetched the patched jar"); + println!( + "{test}: Bazel {} built {:?} before and after `scan --mode hosted --yes`, which found \ + no package in a checkout whose lock pins {PURL} and changed none of its {} files; {:?}", + flow.cell.release(), + flow.upstream, + before.len(), + flow.started.elapsed() + ); +} + /// The checkout moves the artifact one version up after the command: the /// relock passes, the build stops at the line of the entry, and the -/// command the block names there takes the entry out. +/// command the block names there takes the entry out. A hosted scan of +/// the checkout with the stale entry does not: it leaves the entry, and +/// does not find the patch of the new release. #[test] #[ignore = "real Bazel; run with --ignored"] fn a_version_bump_of_the_patched_artifact() { @@ -756,7 +828,7 @@ fn a_version_bump_of_the_patched_artifact() { flow.pins(&before, &[MODULE]); // The relock evaluates the hub with the entry in place, and passes. - let bumped = flow.bump("1.11.0"); + let bumped = flow.bump(BUMPED); let module = std::fs::read_to_string(flow.cell.path(MODULE)).expect("MODULE.bazel"); let entry = module.lines().position(|line| line == "override_repo("); let line = entry.expect("the entry's override_repo") + 1; @@ -789,10 +861,20 @@ fn a_version_bump_of_the_patched_artifact() { let after = std::fs::read_to_string(flow.cell.path(MODULE)).expect("MODULE.bazel"); assert_eq!(after, unpatched, "{test}: {said}"); flow.cell.builds(&bumped); + + // A user who runs a hosted scan where the block says to roll back: + // the stale entry is in the file again, as it was when the build + // stopped. The API has a patch of the release the lock now pins, and + // the scan knows no package: the entry stays where Bazel stops. + flow.cell.write(MODULE, module); + flow.scan_finds_no_package(&flow.cell.tree()); + let asked = flow.standin.asked(&flow.next.route); + assert_eq!(asked, 0, "{test}: something fetched the patched jar"); println!( "{test}: Bazel {} built {:?}; after `get --mode hosted` and the checkout's bump to \ - 1.11.0 the relock passed and the build stopped at {at} ({stops}); `rollback {PURL}` \ - took the entry out and the build printed {bumped:?}; {:?}", + {BUMPED} the relock passed and the build stopped at {at} ({stops}); `rollback {PURL}` \ + took the entry out and the build printed {bumped:?}; with the stale entry back, `scan \ + --mode hosted --yes` found no package and left it; {:?}", flow.cell.release(), flow.upstream, flow.started.elapsed() From 1cafef6d53636c42ffcc863af2ac6b5b367ec8a5 Mon Sep 17 00:00:00 2001 From: Mikola Lysenko Date: Fri, 20 Nov 2026 13:35:37 -0500 Subject: [PATCH 2/3] Find the Maven artifacts a pinned rules_jvm_external hub builds in scan `socket-patch scan` takes its Maven packages from the JVM caches of a project that has a pom.xml, a Gradle build or a Scala-tool build. In a tree that only Bazel builds it said "No packages found. Run your package manager's install first.", although `get --mode hosted` in the same directory pins that jar in MODULE.bazel: a user had to name every purl. The lock of a pinned hub (maven_install.json) names each jar that Bazel fetches, at the version it builds. What changes: - A hosted scan at the root of a Bazel workspace takes the jars that the pinned hubs fetch for packages of the project, beside the ones it finds installed (bazel::maven::hub_jars, through scan's lockfile supplement). A hub's jar is the one without a classifier of an artifact that the hub's lock has a sha256 for, does not skip, and lists under a repository (or takes from the Maven local repository); its version is the lock's. It counts in scannedPackages, and in lockfileOnlyPackages with notInstalled unless a crawl found that version installed. A patch found for it is pinned as `get` pins it: one block in MODULE.bazel. A jar whose repository has no known name is a package too: a patch found for it gets maven_bazel_artifact. - The scan takes a stale entry out of the block. When the lock of a hub moves on (a version bump, an artifact that was removed), Bazel stops at the entry of the release it held ("does not generate repository"). A scan that only pinned the new release beside it would report `applied` and leave a build that fails. So the scan restores such an entry before the rewrite, through the sweep that restores a withdrawn pin (scan/withdrawn.rs, whose messages for a withdrawn pin are the ones they were): the entry alone, under the apply lock, previewed by --dry-run, one rolledBack event with the reason maven_bazel_entry_stale, also on a scan that finds no package. An entry is stale only when every hub was read and none fetches its jar at that version; a pin of the same patch in a pom.xml stays. - `get --mode hosted` keeps the releases that a pinned hub fetches, by the same supplement. - No jar is guessed for a hub that was not read with certainty (no lock_file, a lock or a tag that cannot be read, a maven_install of a WORKSPACE file). The run gives one warning, maven_bazel_hub_not_read, with the file, the line and the reason, and still looks up the jars of the hubs it read. With no package at all a person reads "No packages found (see the warning)." in the place of the advice to install. The warning is for a workspace that has to do with the ruleset: a call names rules_jvm_external, the WORKSPACE file calls maven_install, or a maven_install.json lies at the root. A Bazel workspace of other rulesets with a file that cannot be read scans as it did. - A bare scan covers the project directories below --cwd, and --cwd when it is a project. The root of a Bazel workspace needs no lockfile or build file of its own, so a bare hosted scan from a root above other projects scanned those and never the root, while the same scan with --json scanned the root. The root is now a project of a hosted scan when its hubs have jars, something of them was not read, or the block has a stale entry. - Every hosted scan at a Bazel root now reads the hubs, so Hubs::read reads each lock once for all the hubs that name it and at most 64 MiB of locks for one workspace (the lock past that is a doubt), and the scan walks each lock once. With a debug build, a checkout of a hundred hubs on one 12.8 MiB lock of 80,000 artifacts scans in 1.5 s with 175 MB, as one hub does (1.0 s), and twenty hubs with twenty such locks in 1.7 s with 426 MB. - No other run reads a hub for packages: a scan in vendored or agent mode, one whose --ecosystems leaves Maven out, a --global one, one of a project below the Bazel root, and every run of a project in no Bazel workspace are the runs they were. A hub's jar with a patch is a patch of the run like any other, so the scan writes for it what `get --mode hosted` writes in that checkout: beside a Gradle build the hosted Gradle pin too (settings.gradle and three files under .socket/gradle), and beside a pom.xml what the pom lane pins or says of that pom. Not in this change: `get ` leaves a stale entry where it is (a hosted scan, rollback and remove take it out), hubs that other Bazel modules declare or fill are not read, and the in-memory engine reads no hub. Tests: - bazel::maven: one table of nineteen workspaces for what counts as a jar of a hub (a lock over 1 MiB among them), what is said of a hub that was not read and where nothing is said, and which entry is stale; nothing for a project below the root or without a workspace. One test for the locks: read once, and within the bound. - tests/e2e_bazel_maven_beside.rs, nine tests without a Bazel: the scan as a program and as a person, as a dry run and a second time; a root with a lockfile of its own, alone and above another project; a stale entry, with and without a patch of the new release, with no package at all, while a hub is in doubt, and beside a pom that keeps its pin; a Gradle build and a pom beside the hub, each compared with what `get ` writes and says there; a pom that selects another release; each way a hub is not read; the scans that read no hub, each compared byte for byte with the same scan without the Bazel files; a root above another project; `get` by advisory. - tests/e2e_bazel_build/maven.rs: the two flows of the commit before this one, turned over. After the ruleset's pin, `scan --mode hosted --yes` reports the two jars of the lock and appends the block that `get` writes; the build prints the patched marker, a second scan writes nothing and `rollback` restores every file. After a version bump, with the stale entry in the file, the scan takes it out, pins the patch of the new release, and the build takes that jar. Measured on macOS arm64: the five Maven flows pass on Bazel 9.3.2, 8.8.2, 7.7.2 and 7.4.0 at this tree, a flow in 17 to 56 seconds, and none of the 498 requests behind a logging proxy went to a public Maven repository. With the four source files as they were, both flows fail on 9.3.2 at the count of scanned packages (0, not 2), and eight of the nine tests without a Bazel fail; the ninth holds what does not change. With the stale entries kept from the sweep alone, the version bump flow fails on 9.3.2: the scan reports `applied` and no rolledBack. Twenty single changes to the new code each fail a test; a twenty-first, the root of a bare scan taken as a discovered directory, changes nothing that a run can show, because a directory that the built-in ignores would drop has no project below it that they keep. In sixteen checkouts without a hub of the ruleset (ten without a Bazel workspace file, six in a workspace of other rulesets), fourteen scan and get commands print the same bytes, exit with the same code and write the same files as a binary of the commit before (224 runs). Over the root MODULE.bazel of 22 open-source Bazel workspaces, the 18 without a hub of the ruleset give no jar and no warning. The line budget of the Bazel group follows the code, 2486 to 2568. Co-Authored-By: Claude Fable 5.2 --- .github/workflows/bazel-maven.yml | 5 +- crates/socket-patch-cli/CLI_CONTRACT.md | 12 +- .../src/commands/scan/discovery.rs | 49 +- .../socket-patch-cli/src/commands/scan/mod.rs | 29 +- .../src/commands/scan/withdrawn.rs | 74 +- .../tests/e2e_bazel_build/maven.rs | 163 +++-- .../tests/e2e_bazel_maven_beside.rs | 662 +++++++++++++++++- crates/socket-patch-core/src/bazel/maven.rs | 385 +++++++++- docs/ecosystems.md | 77 +- scripts/loc-budget.json | 2 +- 10 files changed, 1323 insertions(+), 135 deletions(-) diff --git a/.github/workflows/bazel-maven.yml b/.github/workflows/bazel-maven.yml index c4a320fac..dfb5e1475 100644 --- a/.github/workflows/bazel-maven.yml +++ b/.github/workflows/bazel-maven.yml @@ -6,8 +6,9 @@ name: Bazel patch compatibility (Maven) # patch is one block in MODULE.bazel that puts a repository with the # patched jar in the place of the hub's own (`override_repo`): the flows # hold the block's text, the patched build and its sha256 check, the undo, -# a version bump of the patched artifact, and a hub the block cannot -# patch. bazel-compatibility.yml holds the legs and what every ecosystem +# a hosted scan that finds the artifact in the hub's lock, a version bump +# of the patched artifact, and a hub the block cannot patch. +# bazel-compatibility.yml holds the legs and what every ecosystem # shares; this file says when the Maven legs run and on which releases. # # Each major release has an era of its own, because the ruleset decides diff --git a/crates/socket-patch-cli/CLI_CONTRACT.md b/crates/socket-patch-cli/CLI_CONTRACT.md index 2596ee4c8..527dbadd3 100644 --- a/crates/socket-patch-cli/CLI_CONTRACT.md +++ b/crates/socket-patch-cli/CLI_CONTRACT.md @@ -378,7 +378,7 @@ Pinned by `tests/scan/scan_proxy_fallback_e2e.rs`, `tests/cli/api_client_errors_ **Throttling: bounded retry, then a reported failure.** Every patch-API JSON call (the batch query, the per-package patch lists, patch views and VEX record fetches, hosted package references) retries an HTTP `429` or `503` answer up to 3 times (`SOCKET_API_MAX_RETRIES=`, `0`-`10`; `0` = no retry). The wait honors `Retry-After` (delta-seconds or HTTP-date); a `Retry-After` over 30 s is not waited out — the answer is final at once — and one under the jittered first backoff step (`0`, a past date) waits that step instead. Without one it backs off 0.5 s / 1 s / 2 s (each step up to 8 s, with jitter in its upper half). All retries in one run share a 60 s wall-clock window that opens with the run's first retry: a retry whose wait would end after it closes is refused and the answer is final. Parallel requests wait in parallel, so each still gets its retries while the run adds at most about 60 s. Nothing else is retried (401/403 still drive the proxy fallback on the first answer; the public proxy's permanent `503 "Patch API is not configured"` is never retried on any path — the batch query still degrades to the per-package path at once, and a per-package lookup or patch view answering it is the same non-throttle failure it always was, which on the legacy per-package path leaves that package unanswered, see above), and a retried answer folds exactly where the first attempt's would have, so output is identical to an unthrottled run's. A request still throttled after that is a failure in the channel its siblings use: a failed batch is the human `Warning: API batch of failed: ` line and, under `--json`, a run-level `warnings[]` entry `{code: "api_batch_failed", detail: "API batch of failed: "}` (additive; `status` stays `success`, exit 0 — the other batches' packages are reported); a failed per-package patch-list query in the agent / hosted / vendored flows is the human `Warning: could not fetch details for : ` line and, under `--json`, `{code: "patch_details_failed", detail: "could not fetch details for : "}`. When every batch (or every patch-list query) fails, the existing all-failed error envelope and exit 1 apply. The error names the exhausted retry: `Rate limit exceeded (HTTP 429, gave up after 3 retries). Please try again later.` / `API request failed with status 503: (gave up after 3 retries)` (or `(Retry-After s exceeds the 30 s retry cap)` / `(the run's 60 s retry window has closed)`); with retries off it is the pre-retry text. On the token-less legacy per-package proxy path (a proxy without `POST /patch/batch`), a package still throttled (429 / over-capacity 503) after its retries is an unanswered package (see "No patches" needs an answer above): the batch's answered packages are used and it is reported as `patch_lookup_failed` (`partialFailure`, exit 1); when it leaves no package of its batch answered, the batch fails through the batch-failure channel above. Only a PURL the service itself refuses (its own `400`) is skipped individually. Pinned by `tests/scan_api_retry_e2e.rs` and the core crate's `tests/api_retry_e2e.rs`. -**Lockfile supplement (v3.4)**: `scan` discovery is no longer limited to installed trees. The project's lockfiles (`package-lock.json`/`npm-shrinkwrap.json`, `pnpm-lock.yaml` v9, `yarn.lock` classic + berry, `bun.lock`, `vlt-lock.json` (registry nodes, Socket-hosted pins included; vendored `file` nodes are left to the vendor ledger), `Cargo.lock`, `go.sum`, `composer.lock`, `Gemfile.lock`, `uv.lock`/`poetry.lock`/pinned `requirements.txt`) are inventoried and dependencies with NO installed copy join discovery — counts, the API lookup, the rows of what a hosted or vendored run patches (an agent or report-only run, which patches installed copies, gets a stderr note that they are not installed), and the prune "scanned" set (a wiped node_modules no longer prunes lockfile-listed entries). JSON gains a top-level `lockfileOnlyPackages` count and an additive `notInstalled: true` on matching `packages[]` entries. Agent mode partitions lockfile-only patches out BEFORE download (calm `skipped`/`package_not_installed` events — never an error exit, never a manifest write); vendored mode passes them through to the vendor engine's server download. Vendored-ledger entries likewise stay discoverable on a fresh clone (the committed artifact is the dependency). The `Gemfile.lock` inventory lists a gem by its ruby-platform entry; a gem the lock resolves ONLY as precompiled platform builds (`nokogiri (1.18.10-arm64-darwin)`, no `nokogiri (1.18.10)`) joins the supplement of a hosted run alone (`scan` / `get --mode hosted`, and the in-memory hosted engine), which can redirect it to its per-platform prebuilds from the lock; agent and vendored runs see it only through an installed copy, as before. Global scans (`--global`) get no supplement. **Rush monorepos** (no root lockfile, `rush.json` present): the npm-lock inventory falls back to the Rush source-of-truth locks — `common/config/rush/pnpm-lock.yaml` plus every `common/config/subspaces/*/pnpm-lock.yaml` (`read_dir`-sorted, repo-relative paths preserved) — so a Rush repo's dependencies still join discovery. **Plug'n'Play layouts are an explicit refusal, not an empty inventory**: a `.pnp.*` loader means the npm packages are structurally unreachable in EVERY mode (under yarn PnP the installed-tree crawl is empty too — no `node_modules/`), so `scan` surfaces an additive top-level `warnings[]` array (`{code, detail}` objects, omitted when empty) carrying `yarn_pnp_unsupported` (same code as apply's refusal; remedy `yarn patch `) or `pnpm_pnp_unsupported` (pnpm's `node-linker=pnp` twin; pnpm remedies), plus a stderr `Warning: …` line on the human path. Exit code and `status` are deliberately unchanged (exit 0 / `success` — the same posture as hosted refusals, which exit 0 with no `applied` event); the warning is the machine-readable signal that nothing was checked. Pinned by `tests/e2e_safety_yarn_pnp.rs`. **A Bundler lock socket-patch cannot read is likewise a warning, not an empty inventory**: when the project holds gem files but bundler loads no `Gemfile.lock` / `gems.locked` socket-patch reads (`BUNDLE_GEMFILE` or bundler 4's `BUNDLE_LOCKFILE` naming another file, or a `Gemfile` + `gems.rb` twin, whose loaded pair depends on the bundler major that runs), `warnings[]` carries the additive `gem_lock_unsupported` (detail names the setting or the twin) and its lockfile-only gems are not discovered. Same exit-0 / `success` posture. +**Lockfile supplement (v3.4)**: `scan` discovery is no longer limited to installed trees. The project's lockfiles (`package-lock.json`/`npm-shrinkwrap.json`, `pnpm-lock.yaml` v9, `yarn.lock` classic + berry, `bun.lock`, `vlt-lock.json` (registry nodes, Socket-hosted pins included; vendored `file` nodes are left to the vendor ledger), `Cargo.lock`, `go.sum`, `composer.lock`, `Gemfile.lock`, `uv.lock`/`poetry.lock`/pinned `requirements.txt`) are inventoried and dependencies with NO installed copy join discovery — counts, the API lookup, the rows of what a hosted or vendored run patches (an agent or report-only run, which patches installed copies, gets a stderr note that they are not installed), and the prune "scanned" set (a wiped node_modules no longer prunes lockfile-listed entries). JSON gains a top-level `lockfileOnlyPackages` count and an additive `notInstalled: true` on matching `packages[]` entries. Agent mode partitions lockfile-only patches out BEFORE download (calm `skipped`/`package_not_installed` events — never an error exit, never a manifest write); vendored mode passes them through to the vendor engine's server download. Vendored-ledger entries likewise stay discoverable on a fresh clone (the committed artifact is the dependency). The `Gemfile.lock` inventory lists a gem by its ruby-platform entry; a gem the lock resolves ONLY as precompiled platform builds (`nokogiri (1.18.10-arm64-darwin)`, no `nokogiri (1.18.10)`) joins the supplement of a hosted run alone (`scan` / `get --mode hosted`, and the in-memory hosted engine), which can redirect it to its per-platform prebuilds from the lock; agent and vendored runs see it only through an installed copy, as before. Global scans (`--global`) get no supplement. **Rush monorepos** (no root lockfile, `rush.json` present): the npm-lock inventory falls back to the Rush source-of-truth locks — `common/config/rush/pnpm-lock.yaml` plus every `common/config/subspaces/*/pnpm-lock.yaml` (`read_dir`-sorted, repo-relative paths preserved) — so a Rush repo's dependencies still join discovery. **Plug'n'Play layouts are an explicit refusal, not an empty inventory**: a `.pnp.*` loader means the npm packages are structurally unreachable in EVERY mode (under yarn PnP the installed-tree crawl is empty too — no `node_modules/`), so `scan` surfaces an additive top-level `warnings[]` array (`{code, detail}` objects, omitted when empty) carrying `yarn_pnp_unsupported` (same code as apply's refusal; remedy `yarn patch `) or `pnpm_pnp_unsupported` (pnpm's `node-linker=pnp` twin; pnpm remedies), plus a stderr `Warning: …` line on the human path. Exit code and `status` are deliberately unchanged (exit 0 / `success` — the same posture as hosted refusals, which exit 0 with no `applied` event); the warning is the machine-readable signal that nothing was checked. Pinned by `tests/e2e_safety_yarn_pnp.rs`. **A Bundler lock socket-patch cannot read is likewise a warning, not an empty inventory**: when the project holds gem files but bundler loads no `Gemfile.lock` / `gems.locked` socket-patch reads (`BUNDLE_GEMFILE` or bundler 4's `BUNDLE_LOCKFILE` naming another file, or a `Gemfile` + `gems.rb` twin, whose loaded pair depends on the bundler major that runs), `warnings[]` carries the additive `gem_lock_unsupported` (detail names the setting or the twin) and its lockfile-only gems are not discovered. Same exit-0 / `success` posture. **The jars of a pinned rules_jvm_external hub join the supplement of a hosted run at the root of a Bazel workspace (additive)**: `scan --mode hosted` (the inferred default included) and `get --mode hosted` take, for each hub of the root `MODULE.bazel` that names a lock (`maven_install.json`), the jar without a classifier of each artifact the hub fetches, at the lock's version, as a Maven package of the project: it counts in `scannedPackages`, and in `lockfileOnlyPackages` with `notInstalled: true` unless a crawl found that version installed, and a patch found for it is pinned in the block of `MODULE.bazel` (the `maven_bazel_*` rows below). A jar whose repository has no known name is such a package too (a patch found for it gets `maven_bazel_artifact`). No jar is guessed for a hub that was not read with certainty: `warnings[]` then carries `maven_bazel_hub_not_read` once, and the jars of the hubs that were read are still looked up. A patch of a hub's jar is a patch of the run like any other, so the run writes for it what `get --mode hosted` writes in that checkout: beside a Gradle build the hosted Gradle pin too (`settings.gradle` and the files under `.socket/gradle/`, with that pin's warnings), and beside a `pom.xml` what the pom lane pins or says (`redirect_maven_version_not_selected`, `redirect_maven_selection_undetermined`, which speak of `pom.xml` alone). **A stale entry of the block is taken out by `scan --mode hosted` (additive)**: when every hub was read and none fetches the jar of an entry at its version any more (the lock moved on, so Bazel stops at the entry), the scan restores it before the rewrite as the withdrawn-pin sweep above restores a pin: the entry alone, under the apply lock, `--dry-run` previewing it, one `rolledBack` event (`verified` on a dry run) with `uuid`, `errorCode: "maven_bazel_entry_stale"` and `details.mode: "hosted"`, also on a scan that finds no package; a pin of the same patch in a `pom.xml` or Gradle file stays; a restore that fails is the `redirect_withdrawn_restore_failed` warning. `get` takes no stale entry out. No other mode, no run whose `--ecosystems` leaves Maven out, no `--global` run and no run of a project below the Bazel root takes a package from a hub, gives that warning or takes a stale entry out: its packages are those of the run without the workspace. A run of a project in no Bazel workspace is, byte for byte, the run it was. See docs/ecosystems.md "Bazel workspaces: Maven". **Server artifact acquisition (v5.0)**: vendoring downloads the patched artifact for the selected UUID, including on a fresh checkout with no installed package. The CLI no longer downloads pristine packages, stages patch blobs, applies patches to vendor copies, or constructs archives. Backend lock and package-identity checks still run before acquisition; git and custom-registry Cargo sources are refused with `vendor_source_unsupported`. Healthy committed artifacts are reused offline. A changed UUID downloads a fresh server artifact; an unhealthy artifact at the recorded UUID uses the exact redownload procedure below, except that a directory copy whose ledger entry has no file inventory (vendored before v5.0) is rebuilt by its backend from a fresh verified download, and a missing or stale Bun workspace mirror over a healthy canonical tarball is rewritten from that tarball by the backend; those cases report the backend's own failure codes. @@ -394,7 +394,7 @@ Pinned by `tests/scan/scan_proxy_fallback_e2e.rs`, `tests/cli/api_client_errors_ **Path-scoped scans (`scan [PATHS]...`, v5.0)**: what a PATH means depends on the mode. -**A bare hosted or vendored scan covers the checkout (additive/MINOR).** With no PATH, `scan` in hosted or vendored mode (the inferred default included) scans `--cwd` and every other project directory below it in the same git checkout, exactly as if each had been named as a PATH: its own lockfiles, ledgers, `.socket/` and inferred mode, one `== ==` heading each (`== . ==` for `--cwd` itself), the worst exit code, one `--max-new-patches` budget. A project directory is one the in-memory engine would detect as a root for the same tree: a directory holding a lockfile of a supported npm-family, PyPI, Cargo, RubyGems, Go or Composer format (a manifest alone never makes one; Maven, Gradle, sbt and NuGet projects are scanned only as `--cwd` or a named PATH), found by a walk that honours git's ignore rules (`.gitignore`, the repository's and the user's excludes; a `.ignore` file, which hides files from search tools and not from git, is not read), never follows a link, never enters `node_modules`, `.git`, `.socket`, `.yarn` or `vendor`, and never enters another repository: a directory below `--cwd` that holds a `.git` entry (a submodule, a nested clone) is left to a run of its own, since this checkout does not track its files and its own `socket.yml` was not read. A workspace member whose only lock is one the workspace root's run reads is not a project of its own: a Cargo workspace member's `Cargo.lock`, and a pnpm workspace member's `pnpm-lock.yaml` the root accounts for. Discovered directories are subject to the `socket.yml` path filters and the built-in `test/`, `tests/`, `fixtures/`, `__fixtures__/` and `testdata/` ignores, as glob-matched PATHs are; the ones left out are counted in one stderr note (`Note: N project directories under ignored paths ... were not scanned; name a directory to scan it.`), and naming a directory scans it. `--cwd` itself is scanned whenever it is a project, with or without a lockfile, as before. Nothing is searched for, and the run is the single-directory scan it always was, when: `--cwd` is not inside a git checkout (a scan from a home directory never goes looking for projects), a PATH is given, the scope is `--global`/`--global-prefix`, or the mode is agent (an agent-mode scan still names the directories it left; see below). `--json` and `--vex` report on one project directory, so with either a bare scan scans `--cwd` alone and adds the run-level `projects_not_scanned` warning naming the project directories it found and left (`N project directories below this one were not scanned (a, b): --json takes one project directory. Run one scan per directory, e.g. ...`); exit code and status are unchanged. A bare agent-mode scan patches the installed packages its crawl of `--cwd` reads, so it covers the project at `--cwd` alone. It looks for the same project directories below `--cwd` and adds the `projects_not_scanned` warning (stderr `Warning:`, `--json` `warnings[]`) naming the ones whose installed packages that crawl did not read, and the command that reaches the first, asked as the run was (`scan --mode agent`, `scan --sync`, or a bare `scan` when the mode was inferred): `N project directories below this one were not scanned (a, b): an agent-mode scan patches the packages installed for the one project directory it runs in. Run one scan per directory, e.g. `socket-patch scan --mode agent --cwd a``. More than 10 directories are listed as the first 10 `, and N more`. Exit code and status are unchanged. A directory is named when it has an install directory of its own that the crawl from `--cwd` did not read: a `node_modules` the npm walk skips (one under a hidden or build-output directory), a Python environment or a Composer vendor directory inside the project, or a gem home (the project's `vendor/bundle`, or the user-wide homes Bundler installs into by default when `--cwd` holds no Bundler project to read them for). A directory is not named when the run did read its packages (every other `node_modules` below `--cwd`, a gem home or an activated environment both share) or when it has no install directory of its own; the user-wide Python, Cargo and Go stores are never compared. Nothing is looked for with a PATH, a global scope, a non-default `--manifest-path` or outside a git checkout. With nothing found at `--cwd`, a run that names such directories prints `No packages found (see the warning).` in place of the hint to run the package manager's install. A tree too large to search adds a `projects_below_not_searched` warning: some directories may not be named. A tree with more than 500,000 entries is searched only that far, with a stderr warning. +**A bare hosted or vendored scan covers the checkout (additive/MINOR).** With no PATH, `scan` in hosted or vendored mode (the inferred default included) scans `--cwd` and every other project directory below it in the same git checkout, exactly as if each had been named as a PATH: its own lockfiles, ledgers, `.socket/` and inferred mode, one `== ==` heading each (`== . ==` for `--cwd` itself), the worst exit code, one `--max-new-patches` budget. A project directory is one the in-memory engine would detect as a root for the same tree: a directory holding a lockfile of a supported npm-family, PyPI, Cargo, RubyGems, Go or Composer format (a manifest alone never makes one; Maven, Gradle, sbt and NuGet projects are scanned only as `--cwd` or a named PATH), found by a walk that honours git's ignore rules (`.gitignore`, the repository's and the user's excludes; a `.ignore` file, which hides files from search tools and not from git, is not read), never follows a link, never enters `node_modules`, `.git`, `.socket`, `.yarn` or `vendor`, and never enters another repository: a directory below `--cwd` that holds a `.git` entry (a submodule, a nested clone) is left to a run of its own, since this checkout does not track its files and its own `socket.yml` was not read. A workspace member whose only lock is one the workspace root's run reads is not a project of its own: a Cargo workspace member's `Cargo.lock`, and a pnpm workspace member's `pnpm-lock.yaml` the root accounts for. Discovered directories are subject to the `socket.yml` path filters and the built-in `test/`, `tests/`, `fixtures/`, `__fixtures__/` and `testdata/` ignores, as glob-matched PATHs are; the ones left out are counted in one stderr note (`Note: N project directories under ignored paths ... were not scanned; name a directory to scan it.`), and naming a directory scans it. `--cwd` itself is scanned whenever it is a project, with or without a lockfile, as before; in hosted mode it also is one as the root of a Bazel workspace whose rules_jvm_external hubs give the scan packages, a `maven_bazel_hub_not_read` warning or a stale entry to take out (the lockfile supplement above), whatever other file it holds. Nothing is searched for, and the run is the single-directory scan it always was, when: `--cwd` is not inside a git checkout (a scan from a home directory never goes looking for projects), a PATH is given, the scope is `--global`/`--global-prefix`, or the mode is agent (an agent-mode scan still names the directories it left; see below). `--json` and `--vex` report on one project directory, so with either a bare scan scans `--cwd` alone and adds the run-level `projects_not_scanned` warning naming the project directories it found and left (`N project directories below this one were not scanned (a, b): --json takes one project directory. Run one scan per directory, e.g. ...`); exit code and status are unchanged. A bare agent-mode scan patches the installed packages its crawl of `--cwd` reads, so it covers the project at `--cwd` alone. It looks for the same project directories below `--cwd` and adds the `projects_not_scanned` warning (stderr `Warning:`, `--json` `warnings[]`) naming the ones whose installed packages that crawl did not read, and the command that reaches the first, asked as the run was (`scan --mode agent`, `scan --sync`, or a bare `scan` when the mode was inferred): `N project directories below this one were not scanned (a, b): an agent-mode scan patches the packages installed for the one project directory it runs in. Run one scan per directory, e.g. `socket-patch scan --mode agent --cwd a``. More than 10 directories are listed as the first 10 `, and N more`. Exit code and status are unchanged. A directory is named when it has an install directory of its own that the crawl from `--cwd` did not read: a `node_modules` the npm walk skips (one under a hidden or build-output directory), a Python environment or a Composer vendor directory inside the project, or a gem home (the project's `vendor/bundle`, or the user-wide homes Bundler installs into by default when `--cwd` holds no Bundler project to read them for). A directory is not named when the run did read its packages (every other `node_modules` below `--cwd`, a gem home or an activated environment both share) or when it has no install directory of its own; the user-wide Python, Cargo and Go stores are never compared. Nothing is looked for with a PATH, a global scope, a non-default `--manifest-path` or outside a git checkout. With nothing found at `--cwd`, a run that names such directories prints `No packages found (see the warning).` in place of the hint to run the package manager's install. A tree too large to search adds a `projects_below_not_searched` warning: some directories may not be named. A tree with more than 500,000 entries is searched only that far, with a stderr warning. **What a bare scan patched at and below `--cwd` is listed and rolled back from `--cwd` (additive/MINOR).** A bare scan covers `--cwd` and the project directories below it, so `list` and a bare `rollback` cover the project at `--cwd` and the patched project directories below it. A bare `rollback` reaches exactly as far as a bare scan from the same directory: from an agent-mode project (whose scan covers that project alone) it rolls back that project alone and names the directories below in a `projects_not_rolled_back` warning. `vex`, `apply`, `repair`, `remove` and `rollback ` act on the one project at `--cwd`, and name the patched directories below when `--cwd` holds nothing for them. The directories below are the ones a bare scan covers (`commands/projects_below.rs`: the same root detection and walk, the same `socket.yml` path filters and built-in `test/`, `tests/`, `fixtures/`, `__fixtures__/` and `testdata/` ignores, applied before any file of a directory is read; `SOCKET_NO_SOCKET_YML` bypasses the file here as `--no-socket-yml` does for the scan), kept when they hold patch state: a manifest with records, a vendor ledger with entries, or hosted pins in their lockfiles (a store that cannot be read counts, so the directory's own run reports it). Nothing is searched for when `--cwd` is not inside a git checkout, the scope is `--global`/`--global-prefix`, or `--manifest-path` is not the default (it names the one project the run is about). With no patched directory below, every command is, byte for byte, the single-project run it always was. @@ -2344,7 +2344,7 @@ command's integration tests check its output. | `removed` | `remove`, `repair`, `vendor`, `rollback`, `scan --prune` (manifest entries carry `details.manifest: true`) | A manifest entry or vendored state was removed, or (artifact-level, no `purl`) `.socket/` artifacts were swept — that GC carrier sets `bytes` and is not counted in `summary.removed`; the sweep's totals are the envelope's `gc`. | | `verified` | `apply`, `remove`, `repair`, `vendor`, `rollback`, `get`, `scan` (dry run); `vex`; `--update --dry-run` | The action *would* succeed cleanly (`files` lists previewed changes); `vex`: the patch verified and was attested. | | `rebuilt` | `repair`, `vendor` | A missing/corrupt vendored artifact was restored from its exact server download (v5.0: never a lost ledger entry — see `vendor_ledger_missing`). `summary.rebuilt` counts these. | -| `rolledBack` | `rollback`, `remove`, `scan --mode hosted` | A patched package was restored to its original files (`files` lists them), or (`rollback`) a vendored / hosted wiring was unwound (`details.mode`). `remove`: one per restored installed copy, `details.path` naming it — v5.0, MAJOR: was the GC carrier's `details.rolledBack`. `scan --mode hosted`: a pin whose patch was withdrawn was restored upstream (`errorCode: "patch_withdrawn"`). | +| `rolledBack` | `rollback`, `remove`, `scan --mode hosted` | A patched package was restored to its original files (`files` lists them), or (`rollback`) a vendored / hosted wiring was unwound (`details.mode`). `remove`: one per restored installed copy, `details.path` naming it — v5.0, MAJOR: was the GC carrier's `details.rolledBack`. `scan --mode hosted`: a pin whose patch was withdrawn was restored upstream (`errorCode: "patch_withdrawn"`), or a stale entry was taken out of the block in the `MODULE.bazel` of a Bazel workspace (`errorCode: "maven_bazel_entry_stale"`). | ### Stable `errorCode` tags @@ -2604,12 +2604,14 @@ command's integration tests check its output. | `vendor_pypi_reinstall_required` | `skipped` (advisory; human: `Warning: …`); rollback/remove `warnings[]`; `scan --prune` top-level `warnings[]` (human: `GC: …`) | a PyPI vendored revert (`vendor --revert`, rollback / remove of a vendored entry, the manifest reconcile, `--preserve-state` and `--dry-run` included) of a requirements.txt (pip), PDM, uv or Pipenv entry (#477, #1953). These tools reinstall a same-version package only when the LOCKED candidate is a URL or file that differs from the installed one, so once the lock is back on the registry release a plain `pdm sync` / `pdm install`, `uv sync` or `pipenv sync` reports nothing to do and keeps the vendored build; once a requirements line is back to `name==version`, `pip install -r` (and `uv pip install -r`) reports it already satisfied and keeps it too. The detail names the purl and the reinstall that restores the upstream bytes: `pip uninstall -y ` then `pip install -r ` for each requirements file that wired the entry (or the `uv pip uninstall` / `uv pip install -r` pair), which keeps those files' version ranges, index and hash options, `pdm sync --reinstall` (or recreating the virtualenv / `__pypackages__`), `uv sync --reinstall-package `, or Pipenv's `pipenv run pip uninstall -y && pipenv sync` (the categories from `Pipfile.lock`; `pipenv --rm && pipenv sync` for a clean virtualenv). Poetry, Hatch and PEP 751 entries do not get it. | | `redirect_pypi_reinstall_required` | rollback/remove `warnings[]` (+ human stderr), `vendor --revert` `warnings[]` | a hosted unwind (wet or `--dry-run`) that restored PyPI pins in `pdm.lock`, `uv.lock`, `Pipfile.lock` or a pip requirements file (`*.txt`) to their registry entry: the `vendor_pypi_reinstall_required` rule for hosted pins (the patched build's `direct_url.json` still names the patch server). One run-level warning naming each purl and its reinstall; a pin wired by both a lock and a requirements file (a `uv export` beside `uv.lock`) gets the lock tool's reinstall. | | `redirect_withdrawn_check_failed` | `warnings[]` (warning; human: `Warning: …`) | scan `--mode hosted`: the reference lookup for the uuids the lockfiles pin failed, so the run could not tell whether any was withdrawn. Every pin is kept; status and exit unchanged. See "Withdrawn pins". | -| `redirect_withdrawn_restore_failed` | `warnings[]` (warning; human: `Warning: …` / `Error: …`) | scan `--mode hosted`: a pin whose patch is `withdrawn` / `not_found` could not be restored to its upstream registry entry (the restore refused it, writing the restored files failed, or the apply lock was held). The pin is kept and no `rolledBack` event is recorded for it; the detail names the remedy (`git checkout -- `, or re-run / `socket-patch rollback `). A write failure is one warning naming every pin it kept. Status and exit unchanged, unless the failed write could not put a file back: see `lockfile_write_failed`. | +| `redirect_withdrawn_restore_failed` | `warnings[]` (warning; human: `Warning: …` / `Error: …`) | scan `--mode hosted`: a pin whose patch is `withdrawn` / `not_found` could not be restored to its upstream registry entry (the restore refused it, writing the restored files failed, or the apply lock was held). The pin is kept and no `rolledBack` event is recorded for it; the detail names the remedy (`git checkout -- `, or re-run / `socket-patch rollback `). A write failure is one warning naming every pin it kept. Status and exit unchanged, unless the failed write could not put a file back: see `lockfile_write_failed`. Also for a stale entry of a Bazel workspace's block that could not be taken out (`maven_bazel_entry_stale`). | | `patch_withdrawn` | `rolledBack` / `verified` (hosted event) | scan `--mode hosted`: the lockfile pinned a patch the server reports `withdrawn` or `not_found`; the pin was restored (dry run: would be) to its upstream registry entry. `reason` names the uuid and status. See "Withdrawn pins". | +| `maven_bazel_entry_stale` | `rolledBack` / `verified` (hosted event) | scan `--mode hosted` at the root of a Bazel workspace, with Maven in scope (additive): the block in `MODULE.bazel` held an entry for a jar that no pinned rules_jvm_external hub fetches at that version any more, at which Bazel stops ("does not generate repository"); the entry was taken out (dry run: would be), and with the last entry the block. Only when every hub was read. `reason` names the uuid. A restore that fails keeps the entry with a `redirect_withdrawn_restore_failed` warning. See the lockfile supplement and docs/ecosystems.md "Bazel workspaces: Maven". | | `vendor_prebuilt_stub_invalid` | `failed` | RubyGems: the server stub lacks required attributes or is otherwise invalid; no local stub fallback is permitted. | | `bazelignore_not_updated` | run-level `warnings[]` | Go vendored and agent mode in a Bazel workspace (additive): the `.bazelignore` at the Bazel root, which keeps Bazel and gazelle out of the module copies under `.socket/vendor/golang` and `.socket/go-patches`, was not updated. It could not be read or written (a symbolic link or a directory in its place, a read-only directory, a read-only file). The detail names the line to add, or says that the lines of a copy that is gone can be deleted; the command's patch or undo stands and its exit code is unchanged. See docs/ecosystems.md "Bazel workspaces: `.bazelignore`". | | `maven_bazel_hub_unpatched` | run-level `warnings[]` (human: `Warning: …`) | Maven in a Bazel workspace, vendored and agent mode (additive): a pinned rules_jvm_external hub of the root `MODULE.bazel` fetches the jar of a Maven patch that this run vendored or applied, or tried to (`vendor`, `apply`, and `get` / `scan` in those modes; in agent mode only for a package it found installed; never under `--global`), and the block that hosted mode keeps in `MODULE.bazel` holds no entry for that jar. The hub builds the jar its own lock (`maven_install.json`) names, so the patch does not reach the Bazel build: these modes patch no hub. One warning per purl; the detail names the lock, the hub, the file and line of its `maven.install`, and the hosted command that patches a pinned hub. The events, the exit code and every written file are those of the same run without the workspace. A `vendor` whose `--ecosystems` leaves Maven out gives none. A vendored run that takes a hosted pin over (`vendor`, `scan` / `get --mode vendored`, `vendor --revert`) leaves the block's entry for it as it is. A hosted run never gives it: see the `maven_bazel_*` rows below and docs/ecosystems.md "Bazel workspaces: Maven". | -| `maven_bazel_unreadable` | vendored / agent: run-level `warnings[]` (human: `Warning: …`); scan/get `--mode hosted`: `warnings[]` (warning) | Maven in a Bazel workspace (additive): whether a rules_jvm_external hub builds the jar of a Maven patch could not be told, because something could not be read with certainty: a hub without `lock_file` in vendored and agent mode (hosted mode gives `maven_bazel_not_pinned`), a `maven.artifact` or `maven.from_toml` tag that cannot be read, a lock that is missing, empty, not format 2 or 3, larger than 16 MiB or no JSON, a `WORKSPACE` file that calls `maven_install` or cannot be read while `.bazeliskrc` or `.bazelversion` at the Bazel root names no release of Bazel 9 or later, a `maven.install` whose `name` or `lock_file` is not a plain string, a construct of `MODULE.bazel` the reader does not follow, and, in vendored and agent mode, a `maven.override` of the artifact (hosted mode gives `maven_bazel_no_effect`). Hosted mode also gives it for a hub on an isolated `use_extension`, an rc file that cannot be read (also one that an import takes from outside the workspace by a relative path, and one with a line whose reading Bazel's own source calls an oversight: a quote that is never closed, a `\` with nothing after it), a `MODULE.bazel` that is a symbolic link, and a block of `MODULE.bazel` that was changed by hand (also in its first line alone), and then writes no entry for the hub. One warning per purl; the detail names the file, the line and the reason. | +| `maven_bazel_unreadable` | vendored / agent: run-level `warnings[]` (human: `Warning: …`); scan/get `--mode hosted`: `warnings[]` (warning) | Maven in a Bazel workspace (additive): whether a rules_jvm_external hub builds the jar of a Maven patch could not be told, because something could not be read with certainty: a hub without `lock_file` in vendored and agent mode (hosted mode gives `maven_bazel_not_pinned`), a `maven.artifact` or `maven.from_toml` tag that cannot be read, a lock that is missing, empty, not format 2 or 3, larger than 16 MiB (or past the 64 MiB of locks that are read for one workspace) or no JSON, a `WORKSPACE` file that calls `maven_install` or cannot be read while `.bazeliskrc` or `.bazelversion` at the Bazel root names no release of Bazel 9 or later, a `maven.install` whose `name` or `lock_file` is not a plain string, a construct of `MODULE.bazel` the reader does not follow, and, in vendored and agent mode, a `maven.override` of the artifact (hosted mode gives `maven_bazel_no_effect`). Hosted mode also gives it for a hub on an isolated `use_extension`, an rc file that cannot be read (also one that an import takes from outside the workspace by a relative path, and one with a line whose reading Bazel's own source calls an oversight: a quote that is never closed, a `\` with nothing after it), a `MODULE.bazel` that is a symbolic link, and a block of `MODULE.bazel` that was changed by hand (also in its first line alone), and then writes no entry for the hub. One warning per purl; the detail names the file, the line and the reason. | +| `maven_bazel_hub_not_read` | scan run-level `warnings[]` (human: `Warning: …`); `get --mode hosted`: `warnings[]` | scan `--mode hosted` at the root of a Bazel workspace, with Maven in scope (additive): the scan takes the jars that the pinned rules_jvm_external hubs fetch for packages of the project (the lockfile supplement above), and something of the hubs could not be read with certainty, so it does not know every Maven artifact that Bazel builds through a hub. Given once per run, for the first thing that was not read, whatever the packages: a hub without `lock_file` (the detail then ends with how to pin it), or one of the reasons of `maven_bazel_unreadable` that concern reading the hubs (a tag or a lock that cannot be read, a `maven_install` of a `WORKSPACE` file that Bazel may read, a construct of `MODULE.bazel` the reader does not follow, a block that was changed by hand). `[:]: , so socket-patch does not know every Maven artifact that Bazel builds here through a rules_jvm_external hub: it looks for patches of the ones in the hub locks it read, and of no other`. No jar is guessed for what was not read; the jars of the hubs that were read are scanned, and a patch found for one of them gets no entry while the doubt stands (`maven_bazel_not_pinned` / `maven_bazel_unreadable` below). With no package at all a person reads `No packages found (see the warning).`. Exit code and `status` unchanged. A hosted `get` by advisory, which keeps the releases the project has by the same supplement, gives it too. Only in a workspace that has to do with the ruleset: a call in `MODULE.bazel` or a file it includes names `rules_jvm_external`, the `WORKSPACE` file calls `maven_install`, or a `maven_install.json` lies at the Bazel root. In another Bazel workspace a file that cannot be read gives no such warning, and the run is the one without the workspace. | | `maven_bazel_not_at_root` | `warnings[]` (warning; human: `Warning: …`) | scan/get `--mode hosted` in a Bazel workspace (additive): a rules_jvm_external hub builds the jar of a Maven patch, and the project is below the root of its Bazel workspace, in whose `MODULE.bazel` the patch of a hub is written. Like the five codes below and `maven_bazel_unreadable` in hosted mode, it means that the run wrote no entry for the jar in the block of the root `MODULE.bazel`, because it could not show from the checkout that the entry works. One warning per purl, in the place of `redirect_maven_no_pom` for that purl; the detail says why and ends with what the block of the root holds for the jar, which is read for a project below the root too (the detail then names the file as "the MODULE.bazel at the root"). Nothing: "socket-patch wrote nothing to MODULE.bazel for it, so the Bazel build keeps the upstream jar". The event is the one of the same run without the hub: `skipped` / `redirect_unconfirmed` in a tree only Bazel builds, `applied` beside a `pom.xml` or Gradle pin that landed. The entry of the patch, from an earlier run: that entry stays, the event is `applied`, and the detail ends "socket-patch left the entry that MODULE.bazel holds for it as it is". The entry of another patch of the same jar: it stays too, the event is the one without the hub, and the detail ends "socket-patch wrote nothing to MODULE.bazel for it: the entry of patch remains there, and where the block is in effect the Bazel build takes that patch's jar". Exit code unchanged. With none of these codes the run writes the block: `redirect.rewrittenFiles` holds `MODULE.bazel`, the event is `applied`, and `rollback` / `remove` take the entry out. See docs/ecosystems.md "Bazel workspaces: Maven". | | `maven_bazel_not_pinned` | `warnings[]` (warning; human: `Warning: …`) | As `maven_bazel_not_at_root`: a hub of the workspace has no `lock_file` (also one that only `maven.artifact` or `maven.from_toml` tags make), so Bazel resolves its jars when it builds and none can be replaced. Also given beside a pinned hub that holds the artifact. The detail names the hub's file and line and how to pin it. | | `maven_bazel_release` | `warnings[]` (warning; human: `Warning: …`) | As `maven_bazel_not_at_root`: the Bazel release is not known to be 7.4.0 or later, the first with `override_repo` (`.bazeliskrc` or `.bazelversion` names an older one, or neither names a release of three numbers and `MODULE.bazel.lock` is missing or of a format before the one Bazel 7.5.0 writes), or the rules_jvm_external release Bazel selected is not known to be one of 6.4 to 7.2 (`MODULE.bazel.lock` holds the `source.json` of no release of it or of more than one, of a release outside that range, of an older one than the `bazel_dep` of `MODULE.bazel` asks for, or of another one than a `single_version_override` there pins; or an `archive_override`, `git_override`, `local_path_override` or `multiple_version_override` of the ruleset, or a `single_version_override` whose version is no plain string, leaves the release unknown). Versions are compared as Bazel compares them. The detail names the remedy (a `.bazelversion`, one build so that Bazel writes its lock, or an update of socket-patch) or the line of the override. | diff --git a/crates/socket-patch-cli/src/commands/scan/discovery.rs b/crates/socket-patch-cli/src/commands/scan/discovery.rs index 5247005fe..89e67b195 100644 --- a/crates/socket-patch-cli/src/commands/scan/discovery.rs +++ b/crates/socket-patch-cli/src/commands/scan/discovery.rs @@ -7,11 +7,14 @@ use futures_util::StreamExt; use socket_patch_core::api::client::hold_back_debug; use socket_patch_core::api::ranking::{batch_supersedes, cmp_batch_infos}; use socket_patch_core::api::types::{BatchPackagePatches, PatchResponse, PatchSearchResult}; +use socket_patch_core::bazel::maven::HubJars; +use socket_patch_core::crawlers::Ecosystem; use socket_patch_core::manifest::schema::{PatchManifest, PatchRecord}; +use socket_patch_core::patch::redirect::upstream::HostedPin; use socket_patch_core::utils::concurrent::{api_concurrency_for, ordered_concurrent}; use socket_patch_core::utils::purl::{normalize_purl, strip_purl_qualifiers}; use socket_patch_core::utils::purl_key::PurlKey; -use socket_patch_core::vendor::lock_inventory::LockfileEntry; +use socket_patch_core::vendor::lock_inventory::{LockfileEntry, UnsupportedNpmLayout}; use socket_patch_core::vendor::VendorState; use std::collections::{BTreeSet, HashMap, HashSet}; @@ -41,7 +44,14 @@ pub(crate) struct LockfileSupplement { /// npm layouts the lockfile inventory REFUSED (Plug'n'Play loaders). /// Scan surfaces these as refusal warnings: under PnP the installed-tree /// crawl is empty too, so otherwise the project scans as a silent no-op. - pub(crate) unsupported: Vec, + /// Also what a hosted run did not read of the hubs of a Bazel workspace + /// ([`hub_jars`]). + pub(crate) unsupported: Vec, + /// The entries of the block in `MODULE.bazel` whose jar no hub of the + /// Bazel workspace fetches any more ([`hub_jars`]), as hosted pins of + /// that file alone: Bazel stops at each, so a hosted scan takes them + /// out (`withdrawn::sweep`). + pub(crate) stale_hub_pins: Vec, } /// Inventory the project's lockfile(s) and fabricate crawl entries for @@ -61,7 +71,8 @@ pub(crate) struct LockfileSupplement { /// (1.18.10-arm64-darwin)`: no one `.gem` stands for it, so it cannot be /// vendored from the lock, and an in-place patch needs the installed copy). /// Hosted mode redirects such a gem to its per-platform prebuilds from the -/// lock alone, so a hosted supplement names it too. +/// lock alone, so a hosted supplement names it too. It also names the jars +/// of [`hub_jars`], and what was not read of the hubs joins `unsupported`. pub(crate) async fn lockfile_supplement( ctx: &crate::commands::context::ProjectContext<'_>, crawled: &[socket_patch_core::crawlers::types::CrawledPackage], @@ -81,18 +92,46 @@ pub(crate) async fn lockfile_supplement( } else { Vec::new() }; - if entries.is_empty() && platform_gems.is_empty() { + let HubJars { + jars, + unread, + stale, + } = hub_jars(common, hosted).await; + out.unsupported + .extend(unread.map(|(code, detail)| UnsupportedNpmLayout { code, detail })); + let files = vec![socket_patch_core::bazel::maven::FILE.to_string()]; + let pin = |(purl, uuid)| HostedPin { + purl, + uuid, + files: files.clone(), + }; + out.stale_hub_pins = stale.into_iter().map(pin).collect(); + if entries.is_empty() && platform_gems.is_empty() && jars.is_empty() { return out; } let resolved = entries .iter() .map(|entry| entry.purl.as_str()) - .chain(platform_gems.iter().map(String::as_str)); + .chain(platform_gems.iter().map(String::as_str)) + .chain(jars.iter().map(String::as_str)); (out.packages, out.purls) = lockfile_only_packages(resolved, crawled, only, &common.cwd); out.entries = entries.clone(); out } +/// The purls of the jars that the pinned rules_jvm_external hubs of a Bazel +/// workspace fetch when `--cwd` is its root, what was not read of its hubs +/// and the entries of its block that are stale +/// ([`socket_patch_core::bazel::maven::hub_jars`]). Only for a hosted run +/// with Maven in scope: no other mode patches a hub. The callers are runs +/// of a project, never of the global installs. +pub(crate) async fn hub_jars(common: &GlobalArgs, hosted: bool) -> HubJars { + if !hosted || !common.ecosystem_selected(Ecosystem::Maven) { + return HubJars::default(); + } + socket_patch_core::bazel::maven::hub_jars(&common.cwd).await +} + /// The packages `resolved` (the purls the project's lockfiles resolve) with /// no crawled counterpart, fabricated as crawl entries, plus their /// [`PurlKey`]s. "Crawled" is by [`PurlKey`], the same relation diff --git a/crates/socket-patch-cli/src/commands/scan/mod.rs b/crates/socket-patch-cli/src/commands/scan/mod.rs index 50e7adabd..f4a950c61 100644 --- a/crates/socket-patch-cli/src/commands/scan/mod.rs +++ b/crates/socket-patch-cli/src/commands/scan/mod.rs @@ -2259,11 +2259,23 @@ async fn scan_project( if args.common.json || args.vex.vex.is_some() { unscanned_projects = Some(others); } else { + // The root of a Bazel workspace can be without a file + // that makes `--cwd` a project, and a hosted scan has work + // there all the same: the jars that its pinned + // rules_jvm_external hubs fetch, a hub to name as not + // read, or a stale entry of the block to take out. + let mut dirs = found.dirs; + if !dirs.iter().any(|(_, explicit)| *explicit) { + let hosted = args.mode == Some(ScanMode::Hosted); + if discovery::hub_jars(&args.common, hosted).await != Default::default() { + dirs.insert(0, (args.common.cwd.clone(), true)); + } + } return Box::pin(run_project_dirs( args, invocation, mode_inferred, - found.dirs, + dirs, report, )) .await; @@ -2927,6 +2939,15 @@ async fn scan_project( ); } status.finish(); + // Bazel stops at an entry of the block whose jar no hub fetches + // any more, whatever the project holds: such an entry is taken + // out here too (`withdrawn::sweep`, which asks nothing for it). + let stale_sweep = if hosted && policy.writes_allowed() { + let stale = &lockfile_only.stale_hub_pins; + Some(withdrawn::sweep(&args.common, &api_client, &[], stale).await) + } else { + None + }; if human { print_layout_refusals(&layout_refusals, &project_patch_notice, args.common.silent); policy.print_warnings(args.common.silent); @@ -2990,6 +3011,9 @@ async fn scan_project( if prune { env.warnings.push(hosted::prune_ignored_warning()); } + if let Some(sweep) = stale_sweep { + sweep.record_into(&mut env); + } env.set_extra("redirect", hosted::redirect_block(Vec::new())); } else if !vendor { // `redirectState` rides the empty-discovery envelope too @@ -3364,7 +3388,8 @@ async fn scan_project( // was withdrawn has no search results at all. let mut withdrawn_sweep = if hosted && !args.common.is_global() && policy.writes_allowed() { let pins = withdrawn::pins_in_scope(&hosted_pin_list, &all_purls); - Some(withdrawn::sweep(&args.common, &api_client, &pins).await) + let stale = &lockfile_only.stale_hub_pins; + Some(withdrawn::sweep(&args.common, &api_client, &pins, stale).await) } else { None }; diff --git a/crates/socket-patch-cli/src/commands/scan/withdrawn.rs b/crates/socket-patch-cli/src/commands/scan/withdrawn.rs index 24bc86af1..a8096cdcd 100644 --- a/crates/socket-patch-cli/src/commands/scan/withdrawn.rs +++ b/crates/socket-patch-cli/src/commands/scan/withdrawn.rs @@ -43,6 +43,15 @@ use crate::json_envelope::{Envelope, PatchAction, PatchEvent, RunWarning}; /// `reason` code on the `rolledBack` event of a pin the sweep restored. pub(super) const PATCH_WITHDRAWN: &str = "patch_withdrawn"; +/// `reason` code on the `rolledBack` event of an entry that the sweep took +/// out of the block in the `MODULE.bazel` of a Bazel workspace because no +/// pinned rules_jvm_external hub fetches its jar any more +/// ([`socket_patch_core::bazel::maven::HubJars::stale`]), and what a +/// message says of such a patch. +pub(super) const MAVEN_BAZEL_ENTRY_STALE: &str = "maven_bazel_entry_stale"; +const STALE: &str = "is for a jar that no pinned rules_jvm_external hub of the Bazel workspace \ + fetches any more, so Bazel stops at its entry in MODULE.bazel"; + /// Warning: a withdrawn pin could not be restored upstream (the restore /// refused it, the restored files could not be written, or the apply lock /// was held). The pin is unchanged; the detail names the remedy. The one @@ -159,17 +168,20 @@ pub(super) fn pins_in_scope( .collect() } -/// Check `pins` against the reference endpoint and restore the withdrawn -/// ones upstream (see the module docs). A wet restore runs under the apply -/// lock, released before the hosted rewrite takes it again. -pub(super) async fn sweep( +/// A pin the sweep restores, the `reason` code of its event, and what is +/// said of its patch. +type Gone = (HostedPin, &'static str, String); + +/// The pins of `pins` whose patch the reference endpoint says is gone for +/// good. A lookup that fails is a warning of `out`, and keeps every pin. +async fn withdrawn( common: &GlobalArgs, api_client: &ApiClient, pins: &[HostedPin], -) -> WithdrawnSweep { - let mut out = WithdrawnSweep::default(); + out: &mut WithdrawnSweep, +) -> Vec { if pins.is_empty() { - return out; + return Vec::new(); } let uuids: Vec = pins .iter() @@ -198,27 +210,45 @@ pub(super) async fn sweep( still published ({e}); every pin was kept, re-run to check again" ), ); - return out; + return Vec::new(); } }; // A uuid the response leaves out is not known to be gone: kept. - let gone: Vec<(HostedPin, String)> = pins + let gone = pins .iter() .filter_map(|pin| { let status = &references.get(&pin.uuid)?.status; - is_gone(status).then(|| (pin.clone(), status.clone())) + let why = format!("is no longer published ({status})"); + is_gone(status).then(|| (pin.clone(), PATCH_WITHDRAWN, why)) }) .collect(); out.references = Some(KnownReferences::settled(&uuids, references)); + gone +} + +/// Check `pins` against the reference endpoint and restore the withdrawn +/// ones upstream (see the module docs), and with them the pins of `stale`, +/// which are asked about nowhere: the entries of a Bazel workspace's block +/// that the build stops at. A wet restore runs under the apply lock, +/// released before the hosted rewrite takes it again. +pub(super) async fn sweep( + common: &GlobalArgs, + api_client: &ApiClient, + pins: &[HostedPin], + stale: &[HostedPin], +) -> WithdrawnSweep { + let mut out = WithdrawnSweep::default(); + let mut gone = withdrawn(common, api_client, pins, &mut out).await; + let stale = stale.iter().cloned(); + gone.extend(stale.map(|pin| (pin, MAVEN_BAZEL_ENTRY_STALE, STALE.to_string()))); if gone.is_empty() { return out; } if !common.json && !common.silent { - for (pin, status) in &gone { + for (pin, _, why) in &gone { eprintln!( - "Note: patch {} for {} is no longer published ({status}); restoring its \ - upstream registry entry", + "Note: patch {} for {} {why}; restoring its upstream registry entry", pin.uuid, pin.purl ); } @@ -238,14 +268,13 @@ pub(super) async fn sweep( } Err(err) => { let (_, message) = crate::commands::lock_cli::lock_failure(&err, timeout); - for (pin, status) in &gone { + for (pin, _, why) in &gone { out.warn( common, REDIRECT_WITHDRAWN_RESTORE_FAILED, format!( - "{} is pinned to patch {}, which is no longer published \ - ({status}), but it was not restored: {message}; re-run, or run \ - `socket-patch rollback {}`", + "{} is pinned to patch {}, which {why}, but it was not restored: \ + {message}; re-run, or run `socket-patch rollback {}`", pin.purl, pin.uuid, pin.purl ), ); @@ -255,7 +284,7 @@ pub(super) async fn sweep( } }; - let to_restore: Vec = gone.iter().map(|(pin, _)| pin.clone()).collect(); + let to_restore: Vec = gone.iter().map(|(pin, ..)| pin.clone()).collect(); // The scan's own client, for a patch record the gem leg may need. let run_client = crate::args::RunApiClient::new_with(Some(api_client.clone())); let leg = @@ -267,7 +296,7 @@ pub(super) async fn sweep( } else { PatchAction::RolledBack }; - for (pin, status) in &gone { + for (pin, code, why) in &gone { if !leg.reverted.iter().any(|p| p == &pin.purl) { continue; } @@ -280,10 +309,9 @@ pub(super) async fn sweep( PatchEvent::new(action, pin.purl.as_str()) .with_uuid(pin.uuid.as_str()) .with_reason( - PATCH_WITHDRAWN, + *code, format!( - "patch {} is no longer published ({status}); the pin {verb} to the \ - upstream registry entry", + "patch {} {why}; the pin {verb} to the upstream registry entry", pin.uuid ), ), @@ -296,7 +324,7 @@ pub(super) async fn sweep( // so it kept every pin that was not refused on its own. let subject = if key == crate::commands::hosted_unwind::WRITE_FAILURE_KEY { gone.iter() - .map(|(pin, _)| pin.purl.as_str()) + .map(|(pin, ..)| pin.purl.as_str()) .filter(|purl| !leg.failed.iter().any(|(refused, _)| refused == purl)) .collect::>() .join(", ") diff --git a/crates/socket-patch-cli/tests/e2e_bazel_build/maven.rs b/crates/socket-patch-cli/tests/e2e_bazel_build/maven.rs index 737298e6a..f0b4166fe 100644 --- a/crates/socket-patch-cli/tests/e2e_bazel_build/maven.rs +++ b/crates/socket-patch-cli/tests/e2e_bazel_build/maven.rs @@ -25,10 +25,11 @@ //! Bazel before 7.5.0 the checkout first has no `.bazelversion`: the //! command then writes nothing and says so (`maven_bazel_release`). //! - [`a_hosted_scan_of_a_tree_only_bazel_builds`]: the same checkout, and -//! `socket-patch scan --mode hosted --yes` in the place of the `get`. It -//! knows no Maven package in a tree without a `pom.xml` or a Gradle -//! build: it asks the API about none, reports none and changes no file, -//! while the API has a patch for a jar that the build takes. +//! `socket-patch scan --mode hosted --yes` in the place of the `get`. The +//! two jars of the hub's lock are its packages, neither of them +//! installed where a crawl looks; it finds the patch of the victim and +//! writes the same block. The build is patched, a second scan writes +//! nothing, and `rollback --yes` restores every file. //! - [`a_pom_patch_beside_a_pinned_hub`]: a checkout that Maven builds //! too. The one command pins both builds: the three files of a Maven pin //! and `MODULE.bazel`. The Bazel build is patched, and `rollback --yes` @@ -38,9 +39,10 @@ //! passes. The build then stops at the line of the entry's //! `override_repo(`, where the block says what to run. `socket-patch //! rollback --yes` takes the entry out, and the build takes the -//! new upstream jar. The API has a patch of that release too, and a -//! hosted `scan` of the checkout with the stale entry does not find it -//! and leaves the entry: the lock is not where it looks. +//! new upstream jar. The API has a patch of that release too: a hosted +//! `scan` of the checkout with the stale entry takes the entry out, +//! reads the release from the lock, pins that patch, and the build +//! takes its jar. //! - [`the_users_own_override_of_the_artifact`]: the hub has a //! `maven.override` that puts a target of the checkout in the artifact's //! place. The command writes nothing and names the override's line @@ -99,9 +101,9 @@ use sha1::Sha1; use sha2::{Digest, Sha256}; use socket_patch_core::bazel::maven::{Lock, Pinned as Jar}; -use crate::envelope::event_for; +use crate::envelope::{event_for, event_triples}; use crate::harness::{changed, rewrite, Cell, Fixture, Ran, StandIn, Tree, CONSUMER, NOTHING}; -use crate::hosted_memory_common::{Patch, ORG}; +use crate::hosted_memory_common::Patch; /// The workspace the tests build, and the ruleset each Bazel release /// builds it with. One era per way the ruleset names an artifact's @@ -230,6 +232,8 @@ fn serve_upstream( /// A patch of the victim that a stand-in grants. struct Granted { + /// The release of the victim it is for. + version: String, uuid: String, /// The sha256 the grant pins, of a jar with [`Granted::marker`]. sha256: String, @@ -288,6 +292,7 @@ fn grant(standin: &StandIn, version: &str, tampered: Option<&[u8]>) -> (Granted, reference, }; let granted = Granted { + version: version.to_string(), sha256: sha256_hex(&jar), url: format!("https://{registry}/{file}"), route, @@ -304,28 +309,34 @@ const REPOSITORY: &str = "dev_socket_fixture_victim_1_10_0"; /// The block of `MODULE.bazel` for `patch`, with its last line break. fn block(patch: &Granted) -> String { let Granted { - uuid, sha256, url, .. + version, + uuid, + sha256, + url, + .. } = patch; + let repository = REPOSITORY.replace("1_10_0", &version.replace('.', "_")); + let purl = format!("pkg:maven/{GROUP}/{VICTIM}@{version}"); format!( r#"# >>> socket-patch: Maven patches for Bazel (managed block: socket-patch writes and removes it) >>> socket_patch_http_file = use_repo_rule("@bazel_tools//tools/build_defs/repo:http.bzl", "http_file") socket_patch_maven = use_extension("@rules_jvm_external//:extensions.bzl", "maven") -# {PURL}, Socket patch {uuid} +# {purl}, Socket patch {uuid} socket_patch_http_file( - name = "socket_patch_{REPOSITORY}", - downloaded_file_path = "v1/{GROUP_PATH}/{VICTIM}/{VERSION}/{VICTIM}-{VERSION}.jar", + name = "socket_patch_{repository}", + downloaded_file_path = "v1/{GROUP_PATH}/{VICTIM}/{version}/{VICTIM}-{version}.jar", sha256 = "{sha256}", urls = ["{url}"], ) # If Bazel stops at the next line ("does not generate repository"), the hub's lock no longer pins -# {GROUP}:{VICTIM} at {VERSION}: run `socket-patch rollback {PURL}`. +# {GROUP}:{VICTIM} at {version}: run `socket-patch rollback {purl}`. # Do not change it to inject_repo(): the build would pass without the patch. override_repo( socket_patch_maven, - {REPOSITORY} = "socket_patch_{REPOSITORY}", + {repository} = "socket_patch_{repository}", ) # <<< socket-patch <<< "# @@ -520,27 +531,27 @@ impl Pinned { } /// `socket-patch scan --mode hosted --yes` in a checkout without a - /// `pom.xml`, whose API has a patch for a jar of the hub's lock: the - /// command knows no package there. It asks the API about none, reports - /// none and changes no file of `before`. - fn scan_finds_no_package(&self, before: &Tree) { + /// `pom.xml`: the two jars of the hub's lock are its packages, neither + /// installed where a crawl looks, and it pins the one `patch` is for + /// as the `get` of [`Pinned::pins`] does. `stale` is the patch of an + /// entry in the block that the lock no longer pins the jar of: the + /// scan takes that entry out first, and says so. `before` is the + /// checkout without any entry. Gives the files of the checkout. + fn scan_pins(&self, patch: &Granted, before: &Tree, stale: Option<&Granted>) -> Tree { let test = &self.cell.test; - let batch = format!("/v0/orgs/{ORG}/patches/batch"); - let asked = self.standin.asked(&batch); let scan = self.socket_patch(&["scan", "--mode", "hosted", "--yes"]); let said = &scan.dump; - assert_eq!(scan.code, 0, "{test}: {said}"); - assert_eq!(scan.envelope["scannedPackages"], 0, "{test}: {said}"); - assert_eq!(scan.envelope["packages"], json!([]), "{test}: {said}"); - assert_eq!(scan.envelope["events"], json!([]), "{test}: {said}"); - let rewritten = &scan.envelope["redirect"]["rewrittenFiles"]; - assert_eq!(*rewritten, json!([]), "{test}: {said}"); - assert_eq!(self.standin.asked(&batch), asked, "{test}: {said}"); - assert_eq!( - changed(before, &self.cell.tree()), - NOTHING, - "{test}: {said}" - ); + let counted = ["scannedPackages", "lockfileOnlyPackages"].map(|n| &scan.envelope[n]); + assert_eq!(counted, [&json!(2), &json!(2)], "{test}: {said}"); + let gone = |stale: &Granted| { + let purl = format!("pkg:maven/{GROUP}/{VICTIM}@{}", stale.version); + (purl, "rolledBack".into(), "maven_bazel_entry_stale".into()) + }; + let taken_out: Vec<_> = stale.map(gone).into_iter().collect(); + let events = event_triples(&scan.envelope); + assert_eq!(events[..taken_out.len()], taken_out, "{test}: {said}"); + assert_eq!(events.len(), taken_out.len() + 1, "{test}: {said}"); + self.pinned(&scan, patch, before, &[MODULE]) } /// A Bazel before 7.5.0 writes a lock that does not tell it from one @@ -554,23 +565,27 @@ impl Pinned { self.cell.tree() } - /// The one command of hosted mode: it reports the patch applied, - /// rewrites `files` and no other file of `before`, and gives no - /// warning. `MODULE.bazel` is the one of before with the block after - /// one blank line, and nothing has asked for the patched jar yet. - /// Gives the files of the checkout. + /// The one command of hosted mode, a `get` of the purl: what + /// [`Pinned::pinned`] holds of it. Gives the files of the checkout. fn pins(&self, before: &Tree, files: &[&str]) -> Tree { - let test = &self.cell.test; let get = self.socket_patch(&["get", PURL, "--mode", "hosted", "--yes"]); - let said = &get.dump; - let pinned = self - .cell - .pinned_by(&get, PURL, &self.patch.uuid, before, files); - assert_eq!(get.envelope["warnings"], Value::Null, "{test}: {said}"); + self.pinned(&get, &self.patch, before, files) + } + + /// What a finished hosted command did for `patch`: it reports the + /// patch applied, rewrote `files` and no other file of `before`, and + /// gave no warning. `MODULE.bazel` is the one of before with the block + /// after one blank line, and nothing has asked for the patched jar + /// yet. Gives the files of the checkout. + fn pinned(&self, ran: &Ran, patch: &Granted, before: &Tree, files: &[&str]) -> Tree { + let (test, said) = (&self.cell.test, &ran.dump); + let purl = format!("pkg:maven/{GROUP}/{VICTIM}@{}", patch.version); + let pinned = self.cell.pinned_by(ran, &purl, &patch.uuid, before, files); + assert_eq!(ran.envelope["warnings"], Value::Null, "{test}: {said}"); let text = |tree: &Tree| String::from_utf8(tree[MODULE].clone()).expect("a text file"); - let with_block = format!("{}\n{}", text(before), block(&self.patch)); + let with_block = format!("{}\n{}", text(before), block(patch)); assert_eq!(text(&pinned), with_block, "{test}: {said}"); - let asked = self.standin.asked(&self.patch.route); + let asked = self.standin.asked(&patch.route); assert_eq!(asked, 0, "{test}: the command fetched the patched jar"); pinned } @@ -784,9 +799,10 @@ fn a_tree_only_bazel_builds() { } /// A hosted scan of the checkout without a `pom.xml`: the lock of the hub -/// is the one file that names the jars Bazel builds, and the command does -/// not read it for packages. So it finds none, although the API has a -/// patch for the victim, and the build is the one of before. +/// is the one file that names the jars Bazel builds, and the command takes +/// them for the packages of the project. It finds the patch of the victim +/// and pins it as the `get` does: the build is patched, a second scan has +/// nothing to write, and the undo restores every file. #[test] #[ignore = "real Bazel; run with --ignored"] fn a_hosted_scan_of_a_tree_only_bazel_builds() { @@ -797,15 +813,32 @@ fn a_hosted_scan_of_a_tree_only_bazel_builds() { flow.builds_upstream(); let before = flow.names_its_bazel(); - flow.scan_finds_no_package(&before); - flow.cell.builds(&flow.upstream); - let asked = flow.standin.asked(&flow.patch.route); - assert_eq!(asked, 0, "{test}: something fetched the patched jar"); + let pinned = flow.scan_pins(&flow.patch, &before, None); + flow.builds_patched(&pinned); + let again = flow.socket_patch(&["scan", "--mode", "hosted", "--yes"]); + let said = &again.dump; + let rewritten = &again.envelope["redirect"]["rewrittenFiles"]; + assert_eq!( + (again.code, &again.envelope["summary"]["applied"], rewritten), + (0, &json!(1), &json!([])), + "{test}: {said}" + ); + assert_eq!( + changed(&pinned, &flow.cell.tree()), + NOTHING, + "{test}: {said}" + ); + + let undo = flow.socket_patch(&["rollback", "--yes"]); + flow.cell + .undone(&undo, "rolledBack", &before, &flow.upstream); println!( - "{test}: Bazel {} built {:?} before and after `scan --mode hosted --yes`, which found \ - no package in a checkout whose lock pins {PURL} and changed none of its {} files; {:?}", + "{test}: Bazel {} built {:?}, then {:?} from the patch host after `scan --mode hosted \ + --yes` found {PURL} in the hub's lock and appended the block to MODULE.bazel; a second \ + scan wrote nothing; `rollback` restored all {} files and the first build; {:?}", flow.cell.release(), flow.upstream, + flow.patch.marker, before.len(), flow.started.elapsed() ); @@ -813,9 +846,9 @@ fn a_hosted_scan_of_a_tree_only_bazel_builds() { /// The checkout moves the artifact one version up after the command: the /// relock passes, the build stops at the line of the entry, and the -/// command the block names there takes the entry out. A hosted scan of -/// the checkout with the stale entry does not: it leaves the entry, and -/// does not find the patch of the new release. +/// command the block names there takes the entry out. So does a hosted +/// scan of the checkout with the stale entry: it takes the entry out, +/// finds the new release in the lock, and pins the patch of it. #[test] #[ignore = "real Bazel; run with --ignored"] fn a_version_bump_of_the_patched_artifact() { @@ -864,19 +897,23 @@ fn a_version_bump_of_the_patched_artifact() { // A user who runs a hosted scan where the block says to roll back: // the stale entry is in the file again, as it was when the build - // stopped. The API has a patch of the release the lock now pins, and - // the scan knows no package: the entry stays where Bazel stops. + // stopped. The scan takes it out as the rollback did, and the API has + // a patch of the release the lock now pins. + let rolled_back = flow.cell.tree(); flow.cell.write(MODULE, module); - flow.scan_finds_no_package(&flow.cell.tree()); + let pinned = flow.scan_pins(&flow.next, &rolled_back, Some(&flow.patch)); + flow.cell.builds_patched(&pinned, &flow.next.marker); let asked = flow.standin.asked(&flow.next.route); - assert_eq!(asked, 0, "{test}: something fetched the patched jar"); + assert!(asked > 0, "{test}: the build did not fetch the patched jar"); println!( "{test}: Bazel {} built {:?}; after `get --mode hosted` and the checkout's bump to \ {BUMPED} the relock passed and the build stopped at {at} ({stops}); `rollback {PURL}` \ took the entry out and the build printed {bumped:?}; with the stale entry back, `scan \ - --mode hosted --yes` found no package and left it; {:?}", + --mode hosted --yes` took it out, pinned the patch of {BUMPED} and the build printed \ + {:?}; {:?}", flow.cell.release(), flow.upstream, + flow.next.marker, flow.started.elapsed() ); } diff --git a/crates/socket-patch-cli/tests/e2e_bazel_maven_beside.rs b/crates/socket-patch-cli/tests/e2e_bazel_maven_beside.rs index c494e7a0a..93deaa1fd 100644 --- a/crates/socket-patch-cli/tests/e2e_bazel_maven_beside.rs +++ b/crates/socket-patch-cli/tests/e2e_bazel_maven_beside.rs @@ -18,6 +18,13 @@ //! and says so, unless the block already does. A `--global` run says //! nothing. //! +//! A hosted `scan` at the Bazel root takes the jars that the pinned hubs +//! fetch for packages of the project, at the versions of their locks, so +//! it finds the patch of one in a tree that only Bazel builds, also from a +//! root above other projects, and names a hub it could not read. A scan +//! in another mode, one that leaves Maven out, and one where no hub of the +//! ruleset is declared are the scans without Bazel. +//! //! The vendored twin is in `vendor_jvm_cli.rs`, which has the fixture of a //! vendored Maven run. `e2e_bazel_build/maven.rs` holds the hosted flows //! against what a real Bazel builds, with a hub of the fixture's own name @@ -33,7 +40,7 @@ mod hosted_memory_common; use std::collections::BTreeMap; use std::path::PathBuf; -use envelope::{event_for, warning_codes}; +use envelope::{event_for, event_triples, warning_codes}; use hosted_memory_common::{hermetic, mount_api, read_tree, Patch, ORG}; use serde_json::{json, Value}; use socket_patch_core::hash::git_sha256::compute_git_sha256_from_bytes as git_sha256; @@ -73,6 +80,13 @@ pip = use_extension("@rules_python//python/extensions:pip.bzl", "pip") pip.parse(hub_name = "pypi", python_version = "3.11", requirements_lock = "//:maven_install.json") use_repo(pip, "pypi") "#; +/// That workspace with a file its `MODULE.bazel` includes and the checkout +/// lacks: something of it cannot be read, and nothing of it names the +/// ruleset of a Maven hub. +fn unread_other() -> Files { + let module = format!("{OTHER_HUB}include(\"//:more.MODULE.bazel\")\n"); + vec![("MODULE.bazel", module)] +} /// A lock of format 3 as a pin of [`HUB`] writes it when another Bazel /// module adds to the hub named `maven` too, which the real-Bazel flows do @@ -1086,6 +1100,652 @@ fn an_agent_run_says_nothing_of_a_hub_the_block_patches() { } } +/// A hosted scan, as a program reads it. +const SCAN: [&str; 5] = ["scan", "--mode", "hosted", "--yes", "--json"]; + +/// A hosted scan in a tree that only Bazel builds: the jars the pinned +/// hub's lock fetches are its packages, each one that no crawl found +/// installed, and the one with a patch gets its entry in the block. +#[test] +fn a_hosted_scan_takes_the_jars_of_a_pinned_hub_for_packages() { + let machine = Machine::start(); + let with_block = format!("{HUB}\n{BLOCK}"); + machine.make(&pinned(), false, false); + // A dry run says what the scan will do, and writes nothing. + let dry = machine.step(&[&SCAN[..], &["--dry-run"]].concat()); + let action = |ran: &Ran| event_for(&ran.envelope(), PURL)["action"].clone(); + assert_eq!(action(&dry), "verified", "{dry:?}"); + assert_eq!( + (dry.code, &dry.wrote), + (Some(0), &BTreeMap::new()), + "{dry:?}" + ); + + let scan = machine.step(&SCAN); + let envelope = scan.envelope(); + assert_eq!(scan.code, Some(0), "{scan:?}"); + // The victim, the consumer and what another Bazel module added. + let counted = ["scannedPackages", "lockfileOnlyPackages"].map(|count| &envelope[count]); + assert_eq!(counted, [&json!(3), &json!(3)], "{scan:?}"); + let found = envelope["packages"].as_array().expect("the packages"); + let patched: Vec<_> = found.iter().map(|package| &package["purl"]).collect(); + assert_eq!(patched, [PURL], "{scan:?}"); + assert_eq!(found[0]["notInstalled"], true, "{scan:?}"); + assert_eq!(action(&scan), "applied", "{scan:?}"); + assert_eq!( + envelope["redirect"]["rewrittenFiles"], + json!(["MODULE.bazel"]), + "{scan:?}" + ); + assert_eq!(envelope["warnings"], Value::Null, "{scan:?}"); + assert_eq!(scan.wrote.keys().collect::>(), [MODULE], "{scan:?}"); + assert_eq!( + String::from_utf8_lossy(&scan.wrote[MODULE]), + with_block, + "{scan:?}" + ); + // Again: the pin stands, and nothing is written. + let again = machine.step(&SCAN); + assert_eq!(action(&again), "applied", "{again:?}"); + assert_eq!( + (again.code, &again.wrote), + (Some(0), &BTreeMap::new()), + "{again:?}" + ); + + // A scan that names no mode is a hosted one, and a person is told + // which file to commit. + machine.make(&pinned(), false, false); + let person = machine.step(&["scan", "--yes"]); + assert_eq!(person.code, Some(0), "{person:?}"); + assert!(person.stdout.contains("Commit MODULE.bazel"), "{person:?}"); + assert!(!person.stdout.contains("No packages found"), "{person:?}"); + assert_eq!( + String::from_utf8_lossy(&person.wrote[MODULE]), + with_block, + "{person:?}" + ); + + // A jar whose repository has no known name is a package too: the run + // that finds a patch of it says why it gets no entry. + let mut odd = pinned(); + odd[1].1 = lock(&[("consumer", "2.0"), ("victim", "1.10.0+local")]); + machine.make(&odd, false, false); + let scan = machine.step(&SCAN); + assert_eq!(scan.envelope()["scannedPackages"], 3, "{scan:?}"); + assert_eq!(scan.wrote, BTreeMap::new(), "{scan:?}"); +} + +/// A Bazel root is often a project of another package manager too. The +/// jars of its hub join the packages its own lockfile names, and the root +/// is scanned once, above the projects below it. +#[test] +fn a_hosted_scan_takes_the_jars_of_a_hub_beside_a_lockfile_of_the_root() { + let machine = Machine::start(); + let package = json!({ "name": "app", "version": "1.0.0" }); + let npm = json!({ "name": "app", "version": "1.0.0", "lockfileVersion": 3, + "packages": { "": package, "node_modules/left-pad": { "version": "1.3.0" } } }); + let mut files = pinned(); + files.push(("package.json", package.to_string())); + files.push(("package-lock.json", npm.to_string())); + machine.make(&files, false, false); + let scan = machine.step(&SCAN); + let envelope = scan.envelope(); + // The npm package of the root's lock and the three jars of the hub's. + let counted = ["scannedPackages", "lockfileOnlyPackages"].map(|count| &envelope[count]); + assert_eq!(counted, [&json!(4), &json!(4)], "{scan:?}"); + assert_eq!(event_for(&envelope, PURL)["action"], "applied", "{scan:?}"); + assert_eq!(scan.wrote.keys().collect::>(), [MODULE], "{scan:?}"); + assert_eq!( + String::from_utf8_lossy(&scan.wrote[MODULE]), + format!("{HUB}\n{BLOCK}"), + "{scan:?}" + ); + + // Above another project: the root has its one section, the first. + files.push(("web/package.json", package.to_string())); + files.push(("web/package-lock.json", npm.to_string())); + machine.make(&files, false, false); + let person = machine.step(&["scan", "--yes"]); + let headings: Vec<&str> = person + .stdout + .lines() + .filter(|l| l.starts_with("== ")) + .collect(); + assert_eq!(headings, ["== . ==", "== web =="], "{person:?}"); + assert_eq!( + person.wrote.keys().collect::>(), + [MODULE], + "{person:?}" + ); +} + +/// The block for a patch `uuid` of the victim at `version`, as +/// socket-patch writes it. +fn block_at(version: &str, uuid: &str) -> String { + let name = version.replace('.', "_"); + let block = BLOCK.replace("1.10.0", version).replace("1_10_0", &name); + block.replace(UUID, uuid).replace(&UUID[..8], &uuid[..8]) +} + +/// The hub's lock moves on after a pin, and Bazel stops at the entry of +/// the release it held. A hosted scan takes that entry out as `rollback +/// ` does, says so, and pins the patch of the release the lock holds +/// now. It leaves the entry while a hub is in doubt, and a scan that reads +/// no hub leaves it too. +#[test] +fn a_hosted_scan_takes_a_stale_entry_out_of_the_block() { + let machine = Machine::start(); + let old = "pkg:maven/dev.socket.fixture/victim@1.9.0"; + let old_uuid = "0a1b2c3d-1111-4222-8333-444455556666"; + let stale = "maven_bazel_entry_stale"; + let mut files = pinned(); + files[0].1 = format!("{HUB}\n{}", block_at("1.9.0", old_uuid)); + let triples = |ran: &Ran| event_triples(&ran.envelope()); + let event = |purl: &str, action: &str, code: &str| { + (purl.to_string(), action.to_string(), code.to_string()) + }; + + // The API has a patch of the release the lock holds: a dry run says + // what the scan will do, and writes nothing. + machine.make(&files, false, false); + let dry = machine.step(&[&SCAN[..], &["--dry-run"]].concat()); + assert_eq!( + triples(&dry), + [event(old, "verified", stale), event(PURL, "verified", "")], + "{dry:?}" + ); + assert_eq!( + (dry.code, &dry.wrote), + (Some(0), &BTreeMap::new()), + "{dry:?}" + ); + let scan = machine.step(&SCAN); + assert_eq!( + triples(&scan), + [event(old, "rolledBack", stale), event(PURL, "applied", "")], + "{scan:?}" + ); + let said = scan.envelope(); + let taken_out = &said["events"][0]; + assert_eq!(taken_out["uuid"], old_uuid, "{scan:?}"); + assert_eq!( + taken_out["reason"], + format!( + "patch {old_uuid} is for a jar that no pinned rules_jvm_external hub of the Bazel \ + workspace fetches any more, so Bazel stops at its entry in MODULE.bazel; the pin \ + was restored to the upstream registry entry" + ), + "{scan:?}" + ); + assert_eq!((scan.code, &said["warnings"]), (Some(0), &Value::Null)); + assert_eq!(scan.wrote.keys().collect::>(), [MODULE], "{scan:?}"); + assert_eq!( + String::from_utf8_lossy(&scan.wrote[MODULE]), + format!("{HUB}\n{BLOCK}"), + "{scan:?}" + ); + let again = machine.step(&SCAN); + assert_eq!(triples(&again), [event(PURL, "applied", "")], "{again:?}"); + assert_eq!(again.wrote, BTreeMap::new(), "{again:?}"); + + // No patch of the release the lock holds: the file is the one of + // before the first pin, to the byte, and a person reads why. + let moved = lock(&[("consumer", "2.0"), ("victim", "1.11.0")]); + let bump = || { + machine.make(&pinned(), false, false); + assert_eq!(machine.step(COMMANDS[0]).wrote.len(), 1); + std::fs::write(machine.root.join("proj/maven_install.json"), &moved).expect("the lock"); + }; + bump(); + let scan = machine.step(&SCAN); + assert_eq!( + triples(&scan), + [event(PURL, "rolledBack", stale)], + "{scan:?}" + ); + assert_eq!(scan.wrote.keys().collect::>(), [MODULE], "{scan:?}"); + assert_eq!( + String::from_utf8_lossy(&scan.wrote[MODULE]), + HUB, + "{scan:?}" + ); + bump(); + let person = machine.step(&["scan", "--yes"]); + let note = format!( + "Note: patch {UUID} for {PURL} is for a jar that no pinned rules_jvm_external hub of \ + the Bazel workspace fetches any more, so Bazel stops at its entry in MODULE.bazel; \ + restoring its upstream registry entry\n" + ); + assert_eq!(person.stderr.matches(¬e).count(), 1, "{person:?}"); + let restored = format!("Restored {PURL} to its upstream registry entry\n"); + assert!(person.stdout.contains(&restored), "{person:?}"); + assert_eq!( + String::from_utf8_lossy(&person.wrote[MODULE]), + HUB, + "{person:?}" + ); + + // A hub that fetches no jar at all: the scan has no package, and the + // entry goes all the same. + bump(); + let empty = json!({ "artifacts": {}, "dependencies": {}, "repositories": {}, "version": "3" }); + std::fs::write( + machine.root.join("proj/maven_install.json"), + empty.to_string(), + ) + .expect("it"); + let scan = machine.step(&SCAN); + assert_eq!(scan.envelope()["scannedPackages"], 0, "{scan:?}"); + assert_eq!( + triples(&scan), + [event(PURL, "rolledBack", stale)], + "{scan:?}" + ); + assert_eq!( + String::from_utf8_lossy(&scan.wrote[MODULE]), + HUB, + "{scan:?}" + ); + + // While a hub is in doubt it may be the one that fetches the jar: the + // entry stays. It stays in a scan that reads no hub, too. + bump(); + let module = machine.root.join("proj/MODULE.bazel"); + let pinned_text = std::fs::read_to_string(&module).expect("MODULE.bazel"); + let doubted = format!("{pinned_text}maven.install(name = \"tools\")\n"); + std::fs::write(&module, doubted).expect("MODULE.bazel"); + let scan = machine.step(&SCAN); + assert_eq!(triples(&scan), [], "{scan:?}"); + assert_eq!( + warning_codes(&scan.envelope()), + ["maven_bazel_hub_not_read"], + "{scan:?}" + ); + assert_eq!(scan.wrote, BTreeMap::new(), "{scan:?}"); + bump(); + let no_maven = machine.step(&[&SCAN[..], &["--ecosystems", "npm"]].concat()); + assert_eq!( + (no_maven.code, &no_maven.wrote), + (Some(0), &BTreeMap::new()), + "{no_maven:?}" + ); + + // Beside a `pom.xml` that still selects the release: the pin of the + // Maven build stays, and the entry alone goes. + machine.make(&pinned(), true, false); + assert_eq!(machine.step(COMMANDS[0]).wrote.len(), 4); + std::fs::write(machine.root.join("proj/maven_install.json"), &moved).expect("the lock"); + let scan = machine.step(&SCAN); + assert_eq!(scan.wrote.keys().collect::>(), [MODULE], "{scan:?}"); + assert_eq!( + String::from_utf8_lossy(&scan.wrote[MODULE]), + HUB, + "{scan:?}" + ); + assert_eq!( + triples(&scan), + [event(PURL, "rolledBack", stale), event(PURL, "applied", "")], + "{scan:?}" + ); +} + +/// A checkout that Gradle or Maven builds too. A hub's jar with a patch is +/// a patch of the run like one it found installed, so the scan writes for +/// it what `get ` writes there: the Gradle pin beside the block, or +/// what the pom lane says of a `pom.xml` it cannot pin. +#[test] +fn a_hosted_scan_pins_a_hubs_jar_in_the_other_builds_as_get_does() { + let machine = Machine::start(); + let gradle = [ + ("settings.gradle", "rootProject.name = 'app'\n"), + ( + "build.gradle", + "plugins { id 'java' }\ndependencies { implementation 'com.example:other:1.0' }\n", + ), + ( + "gradle.lockfile", + "com.example:other:1.0=compileClasspath\nempty=\n", + ), + ]; + // A pom whose dependency is not in the Maven local repository: what + // Maven selects cannot be told. + let other = PROJECT_POM + .replace("dev.socket.fixture", "com.example") + .replace("victim", "other"); + let rows: [(&[(&str, &str)], &[&str], &[&str]); 2] = [ + ( + &gradle, + &[ + "proj/.socket/gradle/.gitattributes", + "proj/.socket/gradle/hosted-index.tsv", + "proj/.socket/gradle/socket-patch.hosted.settings.gradle", + MODULE, + "proj/settings.gradle", + ], + &[ + "redirect_gradle_module_metadata_unavailable", + "redirect_gradle_detached_configs_unguarded", + ], + ), + ( + &[("pom.xml", &other)], + &[MODULE], + &["redirect_maven_selection_undetermined"], + ), + ]; + for (build, wrote, codes) in rows { + let mut files = pinned(); + files.extend(build.iter().map(|(path, text)| (*path, text.to_string()))); + machine.make(&files, false, false); + let get = machine.step(COMMANDS[0]); + machine.make(&files, false, false); + let scan = machine.step(&SCAN); + assert_eq!(scan.wrote.keys().collect::>(), wrote, "{scan:?}"); + assert_eq!(scan.wrote, get.wrote, "{scan:?}"); + let (scan, get) = (scan.envelope(), get.envelope()); + assert_eq!(event_for(&scan, PURL)["action"], "applied", "{scan:#}"); + assert_eq!(warning_codes(&scan), codes, "{scan:#}"); + assert_eq!(scan["warnings"], get["warnings"], "{scan:#}"); + } +} + +/// The version of a hub's jar is the lock's, whatever a `pom.xml` beside +/// the hub selects. Where Maven builds another release, the scan pins the +/// release Bazel builds, and says that it left the pom as it is. +#[test] +fn a_hosted_scan_pins_the_release_of_the_lock_beside_a_pom_of_another() { + let machine = Machine::start(); + let mut files = pinned(); + files.push(("pom.xml", PROJECT_POM.replace("1.10.0", "1.9.0"))); + machine.make(&files, false, false); + let scan = machine.step(&SCAN); + let envelope = scan.envelope(); + assert_eq!(event_for(&envelope, PURL)["action"], "applied", "{scan:?}"); + let codes = warning_codes(&envelope); + assert_eq!(codes, ["redirect_maven_version_not_selected"], "{scan:?}"); + assert_eq!(scan.wrote.keys().collect::>(), [MODULE], "{scan:?}"); + assert_eq!( + String::from_utf8_lossy(&scan.wrote[MODULE]), + format!("{HUB}\n{BLOCK}"), + "{scan:?}" + ); +} + +/// The words of the warning for a hub that was not read, after where and +/// why. +const NOT_READ: &str = "so socket-patch does not know every Maven artifact that Bazel builds here \ + through a rules_jvm_external hub: it looks for patches of the ones in the \ + hub locks it read, and of no other"; + +/// No jar is guessed for a hub that cannot be read: the scan says once +/// what it did not read, and takes the jars of the hubs it read. +#[test] +fn a_hub_that_a_scan_could_not_read_is_named_once() { + let machine = Machine::start(); + let scan = |bazel: &Files, args: &[&str]| { + machine.make(bazel, false, false); + machine.step(args) + }; + let warned = |ran: &Ran| ran.envelope()["warnings"].clone(); + let not_read = + |detail: String| json!([{ "code": "maven_bazel_hub_not_read", "detail": detail }]); + // A hub without a lock, alone: no package, and how to pin it. + let lockless = HUB.replace(" lock_file = \"//:maven_install.json\",\n", ""); + let unpinned = vec![("MODULE.bazel", lockless)]; + let detail = format!( + "MODULE.bazel:4: hub \"maven\" has no `lock_file`, {NOT_READ}. Give the hub a \ + `lock_file` and pin it (`REPIN=1 bazel run @maven//:pin`)" + ); + let ran = scan(&unpinned, &SCAN); + assert_eq!(ran.envelope()["scannedPackages"], 0, "{ran:?}"); + assert_eq!(warned(&ran), not_read(detail.clone()), "{ran:?}"); + assert_eq!( + (ran.code, &ran.wrote), + (Some(0), &BTreeMap::new()), + "{ran:?}" + ); + // A person is pointed at the warning, not at an install. + let person = scan(&unpinned, &["scan", "--yes"]); + assert_eq!(person.stdout, "No packages found (see the warning).\n"); + let line = format!("Warning: {detail}\n"); + assert_eq!(person.stderr.matches(&line).count(), 1, "{person:?}"); + + // A lock that is not there. + let [.., mut missing] = checkouts(); + missing.truncate(1); + let ran = scan(&missing, &SCAN); + let detail = format!("maven_install.json: it does not exist, {NOT_READ}"); + assert_eq!(warned(&ran), not_read(detail), "{ran:?}"); + assert_eq!(ran.envelope()["scannedPackages"], 0, "{ran:?}"); + + // A workspace that names no ruleset of a hub, with a file its reader + // cannot have: beside a `maven_install.json` it may have a hub. + let mut other = unread_other(); + other.push(("maven_install.json", lock(&[("victim", "1.10.0")]))); + let ran = scan(&other, &SCAN); + let gone = "more.MODULE.bazel: it does not exist; MODULE.bazel:6 includes it"; + assert_eq!( + warned(&ran), + not_read(format!("{gone}, {NOT_READ}")), + "{ran:?}" + ); + assert_eq!(ran.envelope()["scannedPackages"], 0, "{ran:?}"); + + // Beside a pinned hub: its jars are scanned, and the patch of one + // gets no entry while the other hub may build the upstream jar. + let mut beside = pinned(); + beside[0].1.push_str("maven.install(name = \"tools\")\n"); + let ran = scan(&beside, &SCAN); + let envelope = ran.envelope(); + assert_eq!(envelope["scannedPackages"], 3, "{ran:?}"); + assert_eq!( + warning_codes(&envelope), + ["maven_bazel_hub_not_read", "maven_bazel_not_pinned"], + "{ran:?}" + ); + let named = envelope["warnings"][0]["detail"].as_str().expect("text"); + assert!( + named.starts_with("MODULE.bazel:9: hub \"tools\" has no `lock_file`, so"), + "{named}" + ); + let event = event_for(&envelope, PURL); + assert_eq!( + (ran.code, &event["action"], &event["errorCode"], &ran.wrote), + ( + Some(0), + &json!("skipped"), + &json!("redirect_unconfirmed"), + &BTreeMap::new() + ), + "{ran:?}" + ); +} + +/// A person's scan from the root of a checkout covers the project +/// directories below it, and the root when it is a project. The root of a +/// Bazel workspace needs no file that makes it one: it is a project of a +/// hosted scan when its hubs have jars, or one of them was not read. Of +/// any other scan it is none, as without Bazel. +#[test] +fn a_bare_scan_covers_a_bazel_root_above_other_projects() { + let machine = Machine::start(); + let package = json!({ "name": "web", "version": "1.0.0" }); + let web = |mut bazel: Files| { + let lock = json!({ "name": "web", "version": "1.0.0", "lockfileVersion": 3, + "packages": { "": package, "node_modules/left-pad": { "version": "1.3.0" } } }); + bazel.push(("web/package.json", package.to_string())); + bazel.push(("web/package-lock.json", lock.to_string())); + bazel + }; + let scan = |bazel: Files, args: &[&str]| { + machine.make(&web(bazel), false, false); + machine.step(args) + }; + let headings = |ran: &Ran| -> Vec { + let lines = ran.stdout.lines().filter(|line| line.starts_with("== ")); + lines.map(String::from).collect() + }; + let person = ["scan", "--yes"]; + let both = ["== . ==", "== web =="]; + + let ran = scan(pinned(), &person); + assert_eq!(ran.code, Some(0), "{ran:?}"); + assert_eq!(headings(&ran), both, "{ran:?}"); + assert_eq!(ran.wrote.keys().collect::>(), [MODULE], "{ran:?}"); + assert_eq!( + String::from_utf8_lossy(&ran.wrote[MODULE]), + format!("{HUB}\n{BLOCK}"), + "{ran:?}" + ); + + let lockless = HUB.replace(" lock_file = \"//:maven_install.json\",\n", ""); + let ran = scan(vec![("MODULE.bazel", lockless)], &person); + assert_eq!(ran.code, Some(0), "{ran:?}"); + assert_eq!(headings(&ran), both, "{ran:?}"); + let named = "Warning: MODULE.bazel:4: hub \"maven\" has no `lock_file`, so"; + assert_eq!(ran.stderr.matches(named).count(), 1, "{ran:?}"); + assert_eq!(ran.wrote, BTreeMap::new(), "{ran:?}"); + + // A hub that fetches no jar, and a stale entry of the block: the + // root is scanned for that entry alone, and it goes. + let empty = json!({ "artifacts": {}, "dependencies": {}, "repositories": {}, "version": "3" }); + let stale = vec![ + ("MODULE.bazel", format!("{HUB}\n{BLOCK}")), + ("maven_install.json", empty.to_string()), + ]; + let ran = scan(stale, &person); + assert_eq!(headings(&ran), both, "{ran:?}"); + assert_eq!(String::from_utf8_lossy(&ran.wrote[MODULE]), HUB, "{ran:?}"); + + // The root is `--cwd`, which the user named: it is scanned in a + // directory that the built-in ignores leave out of a search, where + // the project below it is left out. + let under = |(path, text): (&str, String)| -> (&'static str, String) { + (Box::leak(format!("testdata/{path}").into_boxed_str()), text) + }; + let nested: Files = web(pinned()).into_iter().map(under).collect(); + machine.make(&nested, false, false); + let ran = machine.step_in(&machine.root.join("proj/testdata"), &person); + assert_eq!(headings(&ran), [""; 0], "{ran:?}"); + assert_eq!( + String::from_utf8_lossy(&ran.wrote["proj/testdata/MODULE.bazel"]), + format!("{HUB}\n{BLOCK}"), + "{ran:?}" + ); + + let [plain, other, ..] = checkouts(); + let vendored = ["scan", "--mode", "vendored", "--yes"]; + let no_maven = ["scan", "--yes", "--ecosystems", "npm"]; + let rows: [(Files, &[&str]); 4] = [ + (other, &person), + (unread_other(), &person), + (pinned(), &vendored), + (pinned(), &no_maven), + ]; + for (bazel, args) in rows { + let without = scan(plain.clone(), args); + assert_eq!(without.code, Some(0), "{args:?}: {without:?}"); + assert_eq!(headings(&without), [""; 0], "{args:?}: {without:?}"); + assert_eq!(scan(bazel, args), without, "{args:?}"); + } +} + +/// A scan that patches no hub does not take a hub's jars for packages: +/// one in vendored or agent mode, one that leaves Maven out. And a hosted +/// scan finds no hub where none is declared at the project: with a lock +/// of that name and no workspace file, with another ruleset's hub, and in +/// a project below the Bazel root. Each is the scan without Bazel. +#[test] +fn a_scan_that_reads_no_hub_reports_as_without_bazel() { + let machine = Machine::start(); + let [plain, other, ..] = checkouts(); + let lock_alone = vec![("maven_install.json", lock(&[("victim", "1.10.0")]))]; + let no_maven = [&SCAN[..], &["--ecosystems", "npm"]].concat(); + let unread = unread_other(); + let rows: [(&Files, bool, &[&str]); 8] = [ + (&pinned(), false, &["scan", "--mode", "agent", "--json"]), + (&pinned(), false, &["scan", "--mode", "vendored", "--json"]), + (&pinned(), false, &no_maven), + (&other, false, &SCAN), + (&other, true, &SCAN), + (&unread, false, &SCAN), + (&lock_alone, false, &SCAN), + (&lock_alone, true, &SCAN), + ]; + for (bazel, maven, args) in rows { + let scan = |bazel: &Files| { + machine.make(bazel, maven, false); + machine.step(args) + }; + let without = scan(&plain); + assert_eq!(without.code, Some(0), "{args:?}: {without:?}"); + assert_eq!(scan(bazel), without, "{args:?} beside {:?}", bazel[0].0); + } + let below = machine.root.join("proj/services/billing"); + let scan_below = |bazel: &Files| { + machine.make(bazel, false, false); + std::fs::create_dir_all(&below).expect("a project"); + machine.step_in(&below, &SCAN) + }; + assert_eq!(scan_below(&pinned()), scan_below(&plain)); +} + +/// `get` by an advisory keeps, of the releases the advisory has patches +/// for, the ones the project has. In hosted mode the jars of a pinned hub +/// count, as they do for a hosted scan, and a hub that was not read is +/// named. +#[test] +fn a_hosted_get_by_advisory_keeps_the_release_a_pinned_hub_fetches() { + use wiremock::matchers::{method, path}; + let machine = Machine::start(); + let ghsa = "GHSA-test-aaaa-bbbb"; + let found = json!({ "uuid": UUID, "purl": PURL, "publishedAt": "2024-01-01T00:00:00Z", + "description": "fixture", "license": "MIT", "tier": "free", "vulnerabilities": {} }); + let answer = json!({ "patches": [found], "canAccessPaidPatches": false }); + let search = wiremock::Mock::given(method("GET")) + .and(path(format!("/v0/orgs/{ORG}/patches/by-ghsa/{ghsa}"))) + .respond_with(wiremock::ResponseTemplate::new(200).set_body_json(answer)); + machine._runtime.block_on(search.mount(&machine.api)); + let get = |bazel: &Files| { + machine.make(bazel, false, false); + machine.step(&["get", ghsa, "--mode", "hosted", "--yes", "--json"]) + }; + let [plain, ..] = checkouts(); + let without = get(&plain); + let event = event_for(&without.envelope(), PURL).clone(); + assert_eq!( + (&event["action"], &event["errorCode"], &without.wrote), + ( + &json!("skipped"), + &json!("package_not_installed"), + &BTreeMap::new() + ), + "{without:?}" + ); + let beside = get(&pinned()); + let event = event_for(&beside.envelope(), PURL).clone(); + assert_eq!( + (beside.code, &event["action"]), + (Some(0), &json!("applied")) + ); + assert_eq!( + String::from_utf8_lossy(&beside.wrote[MODULE]), + format!("{HUB}\n{BLOCK}"), + "{beside:?}" + ); + let lockless = HUB.replace(" lock_file = \"//:maven_install.json\",\n", ""); + let unread = get(&vec![("MODULE.bazel", lockless)]); + let envelope = unread.envelope(); + let codes = warning_codes(&envelope); + assert_eq!(codes, ["maven_bazel_hub_not_read"], "{unread:?}"); + assert_eq!( + (&event_for(&envelope, PURL)["errorCode"], &unread.wrote), + (&json!("package_not_installed"), &BTreeMap::new()), + "{unread:?}" + ); +} + #[test] fn a_global_run_says_nothing() { let machine = Machine::start(); diff --git a/crates/socket-patch-core/src/bazel/maven.rs b/crates/socket-patch-core/src/bazel/maven.rs index 188c8b8d4..7865c63a3 100644 --- a/crates/socket-patch-core/src/bazel/maven.rs +++ b/crates/socket-patch-core/src/bazel/maven.rs @@ -21,7 +21,8 @@ //! as they are. [`jars`] are the patches of a run that a hub can take, //! [`site`] says for each whether its entry may be written, [`rewrite`] //! writes it, [`pins`] reads the block for `list`, `rollback`, `remove` -//! and `vex`, and [`without`] takes an entry out. +//! and `vex`, and [`without`] takes an entry out. [`hub_jars`] are the +//! jars a hosted scan looks for patches of: those the pinned hubs fetch. //! The block is recognised by its text alone: one that is not, byte for //! byte, what socket-patch writes for its entries was changed by hand, and //! is neither written to nor trusted. @@ -38,9 +39,10 @@ //! Bazel without a hub, and what a flag or an rc file outside the checkout //! tells Bazel. -use std::collections::{BTreeMap, HashMap, HashSet}; +use std::collections::{BTreeMap, BTreeSet, HashMap, HashSet}; use std::ops::Range; use std::path::Path; +use std::rc::Rc; use super::rc::{declared_release, mentions, ModuleLock}; use super::starlark::{Bound, Call, How, Value}; @@ -52,7 +54,7 @@ use crate::patch::redirect::{ use crate::utils::digest::is_hex64_lower; use crate::utils::fs::run_blocking; use crate::utils::line_endings::terminator; -use crate::utils::purl::{parse_maven_purl, purl_qualifier}; +use crate::utils::purl::{build_maven_purl, parse_maven_purl, purl_qualifier}; use crate::vendor::jvm::layout::artifact_path; /// The extension whose tags make a hub. @@ -68,6 +70,10 @@ pub const HUB_UNPATCHED: &str = "maven_bazel_hub_unpatched"; /// The warning for an artifact when the hubs of the workspace cannot be /// read: a hub may build it. pub const UNREADABLE: &str = "maven_bazel_unreadable"; +/// The warning of a hosted run that looks for the packages of a project +/// ([`hub_jars`]) when not every hub of the workspace could be read: the +/// jars of a hub that was not read are not among them. +pub const HUB_NOT_READ: &str = "maven_bazel_hub_not_read"; /// The warnings of a hosted run for a jar that a hub builds, or may /// build, and whose entry it does not write. Each says why, and that the @@ -114,8 +120,10 @@ const RULESETS: [&str; 10] = [ /// What a doubt about a hub without a lock ends with. const NO_LOCK: &str = "has no `lock_file`"; -/// The largest lock that is read. +/// The largest lock that is read, and how many bytes of locks are read for +/// one workspace: each lock once, for every hub that names it. const MAX_LOCK: usize = 16 << 20; +const MAX_LOCKS: usize = 4 * MAX_LOCK; /// The tags that make a hub of their `name` or add to it, `install` /// first: the ruleset's `private/extensions/maven.bzl` has `artifact` at @@ -445,7 +453,7 @@ fn asked(ws: &Workspace) -> Asked { /// What the hubs of a workspace build: read once, asked per artifact. #[derive(Debug)] pub struct Hubs { - pinned: Vec<(Hub, Lock)>, + pinned: Vec<(Hub, Rc)>, /// The `override` tags of the root module: the [`target`] each /// replaces (`None`: its `coordinates` are no string, so any), and /// where it is. The tag's `name` is not asked for: the ruleset at 6.5 @@ -462,6 +470,11 @@ pub struct Hubs { edited: Option, /// What the root says of the release of the ruleset. asked: Asked, + /// Whether the workspace has to do with the ruleset: a call of a file + /// that was read names it, its `WORKSPACE` file calls `maven_install`, + /// or a `maven_install.json` lies at the root. In another workspace a + /// doubt is about no hub ([`hub_jars`]). + ruleset: bool, } impl Hubs { @@ -515,17 +528,31 @@ impl Hubs { }); let overrides = overrides.collect(); let (mut pinned, mut lockless) = (Vec::new(), None); + let (mut read_locks, mut room) = (HashMap::new(), MAX_LOCKS); for hub in hubs { // Where a doubt about its lock is: the lock, or the tag. let (file, line) = match &hub.lock { Some(lock) => (lock.clone(), 0), None => (hub.file.clone(), hub.line), }; - let lock = match hub.lock.as_deref().map(locks) { + let lock = match &hub.lock { None => Err(format!("hub \"{}\" {NO_LOCK}", hub.name)), - Some(Entry::Text(text)) => Lock::parse(&text), - Some(Entry::Absent) => Err("it does not exist".to_string()), - Some(Entry::Unreadable(why)) => Err(format!("it cannot be read: {why}")), + Some(path) => { + let parsed = || match locks(path) { + Entry::Text(text) if text.len() > room => Err(format!( + "the locks of the workspace are larger than {} MiB in all", + MAX_LOCKS >> 20 + )), + Entry::Text(text) => { + room -= text.len(); + Lock::parse(&text).map(Rc::new) + } + Entry::Absent => Err("it does not exist".to_string()), + Entry::Unreadable(why) => Err(format!("it cannot be read: {why}")), + }; + let lock = read_locks.entry(path.clone()).or_insert_with(parsed); + lock.clone() + } }; match lock { Ok(lock) => pinned.push((hub, lock)), @@ -536,8 +563,16 @@ impl Hubs { } } } + let module = Value::Str(EXTENSION.module.to_string()); + let names = + |call: &Call| [call.arg("name"), call.arg("module_name")].contains(&Some(&module)); + let named = ws + .files + .iter() + .any(|(_, file)| file.calls.iter().any(names)); + let calls = matches!(ws.legacy, Some(Ok(_))); let before_9 = || declared_release(view).is_none_or(|release| release < (9, 0, 0)); - doubts.extend(match ws.legacy.filter(|_| before_9()) { + let legacy = match ws.legacy.filter(|_| before_9()) { Some(Ok((file, read))) => { // By its own name, or by one that a `load` gives it. let symbol = Value::Str("maven_install".to_string()); @@ -556,7 +591,10 @@ impl Hubs { }) } unread => unread.and_then(Result::err), - }); + }; + let ruleset = + named || (calls && legacy.is_some()) || view("maven_install.json") != Entry::Absent; + doubts.extend(legacy); Hubs { pinned, overrides, @@ -565,6 +603,7 @@ impl Hubs { block, edited, asked, + ruleset, } } @@ -852,13 +891,17 @@ type Declined = (&'static str, String); const KEEPS: &str = "socket-patch wrote nothing to MODULE.bazel for it, so the Bazel build keeps \ the upstream jar"; +/// Where a doubt is: its file, with its line when it is about one. +fn at(doubt: &Doubt) -> String { + match doubt.line { + 0 => doubt.file.clone(), + line => format!("{}:{line}", doubt.file), + } +} + /// Why no entry is written for a jar that a hub may build. fn unknown(doubt: &Doubt, kind: Kind) -> Declined { - let Doubt { file, line, why } = doubt; - let at = match line { - 0 => file.clone(), - line => format!("{file}:{line}"), - }; + let (at, why) = (at(doubt), &doubt.why); match kind { Kind::Unpinned(hub) => { let why = format!( @@ -1281,6 +1324,82 @@ pub fn pins(root: &Path, module: &str) -> Result, String> { Ok(hubs.block.iter().map(pin).collect()) } +/// What a hosted scan takes from the hubs of a Bazel workspace +/// ([`hub_jars`]). +#[derive(Debug, Default, PartialEq, Eq)] +pub struct HubJars { + /// The purl of each plain jar that a pinned hub fetches, at the + /// version its lock holds. + pub jars: Vec, + /// The warning ([`HUB_NOT_READ`]) for the first thing that was not + /// read. + pub unread: Option<(&'static str, String)>, + /// The entries of the block whose jar no pinned hub fetches any more, + /// when every hub was read: the purl and the id of the patch of each. + /// Bazel stops at such an entry. + pub stale: Vec<(String, String)>, +} + +/// What a hosted run takes for packages of `project` beside the ones it +/// finds installed, when `project` is the root of a Bazel workspace: the +/// purl of each plain jar that a pinned hub fetches, at the version its +/// lock holds. That is the version Bazel builds, whatever a `pom.xml` +/// beside the hub selects. With them, one warning ([`HUB_NOT_READ`]) for +/// the first thing that was not read: no jar is guessed for a hub without +/// a lock, or for one whose tags or lock cannot be read. In a workspace +/// that has nothing to do with the ruleset there is no hub to read, and +/// no warning. And the entries of the block that are stale, for the run +/// to take out. Nothing for a project below the root, where no entry is +/// written either, and for one in no Bazel workspace. +pub async fn hub_jars(project: &Path) -> HubJars { + let project = project.to_path_buf(); + run_blocking(move || hub_jars_blocking(&project)).await +} + +fn hub_jars_blocking(project: &Path) -> HubJars { + let root = super::bazel_root(project).filter(|root| root.project.as_os_str().is_empty()); + let Some(root) = root else { + return HubJars::default(); + }; + let hubs = Hubs::read(&disk(&root.dir), &disk_up_to(&root.dir, MAX_LOCK)); + // Each lock once, however many hubs name it. + let (mut purls, mut seen) = (BTreeSet::new(), HashSet::new()); + let locks = hubs.pinned.iter().map(|(_, lock)| lock); + let locks = locks.filter(|lock| seen.insert(Rc::as_ptr(lock))); + for (key, jar) in locks.flat_map(|lock| &lock.jars) { + let (group, artifact) = key.split_once(':').unwrap_or_default(); + if jar.fetched { + purls.insert(build_maven_purl(group, artifact, &jar.version)); + } + } + let unread = hubs.doubt().filter(|_| hubs.ruleset).map(|(doubt, kind)| { + let pin = match kind { + Kind::Unpinned(hub) => format!( + ". Give the hub a `lock_file` and pin it (`REPIN=1 bazel run @{hub}//:pin`)" + ), + Kind::Unread | Kind::Overridden => String::new(), + }; + let detail = format!( + "{}: {}, so socket-patch does not know every Maven artifact that Bazel builds here \ + through a rules_jvm_external hub: it looks for patches of the ones in the hub \ + locks it read, and of no other{pin}", + at(doubt), + doubt.why + ); + (HUB_NOT_READ, detail) + }); + let stale = + |patch: &&Patch| hubs.built(&patch.group, &patch.artifact, &patch.version) == Built::No; + let stale = hubs.block.iter().filter(stale); + HubJars { + jars: purls.into_iter().collect(), + unread, + stale: stale + .map(|patch| (patch.purl(), patch.uuid.clone())) + .collect(), + } +} + /// The warnings for the patches `purls` of a command that ran in /// `project`: a code and a detail for each Maven jar that a hub of the /// project's Bazel workspace builds ([`HUB_UNPATCHED`]) or may build @@ -1326,14 +1445,12 @@ fn notices_blocking(project: &Path, mut purls: Vec) -> Vec<(&'static str ); (HUB_UNPATCHED, detail) } - Built::Unknown(Doubt { file, line, why }, _) => { - let at = match line { - 0 => file.clone(), - line => format!("{file}:{line}"), - }; + Built::Unknown(doubt, _) => { let detail = format!( "socket-patch could not tell whether Bazel takes this artifact from \ - a rules_jvm_external hub: {at}: {why}" + a rules_jvm_external hub: {}: {}", + at(doubt), + doubt.why ); (UNREADABLE, detail) } @@ -2172,6 +2289,228 @@ jvm.artifact(group = "com.example", artifact = "a", version = "1") assert_eq!(notices_blocking(broken.path(), Vec::new()), []); } + /// What the warning for a hub that was not read says after where and + /// why. + const NOT_READ: &str = "so socket-patch does not know every Maven artifact that Bazel builds \ + here through a rules_jvm_external hub: it looks for patches of the \ + ones in the hub locks it read, and of no other"; + + #[test] + fn a_hosted_scan_takes_the_plain_jars_that_pinned_hubs_fetch() { + let netty = |artifact| format!("pkg:maven/io.netty/netty-tcnative-{artifact}@2.0.61.Final"); + let netty = [netty("boringssl-static"), netty("classes")]; + let netty: Vec<&str> = netty.iter().map(String::as_str).collect(); + let sqlite = "pkg:maven/com.almworks.sqlite4java/sqlite4java@1.0.392"; + let local = LOCK_SKIPS.replace( + "\"version\": \"3\"", + "\"m2local\": true, \"version\": \"3\"", + ); + // A version and an artifact that no repository can be named for: + // the run that finds a patch of one says so (`maven_bazel_artifact`). + let odd = r#"{"artifacts": { + "a.b:c": {"shasums": {"jar": "00"}, "version": "1.0+local"}, + "a.b:d@e": {"shasums": {"jar": "11"}, "version": "1.0"}, + "a.b:f": {"shasums": {"jar": "22"}, "version": "1.0"}}, + "repositories": {"https://repo.example/": ["a.b:c", "a.b:d@e", "a.b:f"]}, + "version": "3"}"#; + let odd_jars = + ["c@1.0+local", "d@e@1.0", "f@1.0"].map(|jar| format!("pkg:maven/a.b/{jar}")); + let two = + format!("{PINNED}maven.install(name = \"tools\", lock_file = \"//:tools.json\")\n"); + let lockless = format!("{PINNED}maven.install(name = \"tools\")\n"); + let (v2, v3) = (lock_69(), lock_72()); + // A lock larger than the 1 MiB of a Starlark file. + let large = lock_72() + &" ".repeat(1 << 20); + let main = "maven_install.json"; + let pin = ". Give the hub a `lock_file` and pin it (`REPIN=1 bazel run @tools//:pin`)"; + // A file the reader cannot have, in a workspace with a hub and in + // one that names no ruleset of Maven's. + let gone = "more.MODULE.bazel: it does not exist; MODULE.bazel"; + let more = format!("{PINNED}include(\"//:more.MODULE.bazel\")\n"); + let go = "bazel_dep(name = \"rules_go\", version = \"0.50.1\")\n\ + include(\"//:more.MODULE.bazel\")\n"; + // The block of an earlier run, for netty's jar. + let entry = patch("netty-tcnative-classes", NETTY_UUID); + let patched = with_patch(PINNED, LABEL, entry).expect("a block"); + let patched_lockless = format!("{patched}\nmaven.install(name = \"tools\")\n"); + let rows: [(&str, &[(&str, &str)], Vec<&str>, Option, bool); 19] = [ + // The jar without a classifier of each artifact, in each format. + (PINNED, &[(main, &v2)], netty.clone(), None, false), + (PINNED, &[(main, &v3)], netty.clone(), None, false), + (PINNED, &[(main, &large)], netty.clone(), None, false), + // Not one the lock skips or has no sum of, not another + // packaging, and not one that no repository lists, + (PINNED, &[(main, LOCK_SKIPS)], vec![sqlite], None, false), + // unless the hub takes it from the Maven local repository. + ( + PINNED, + &[(main, &local)], + vec![sqlite, "pkg:maven/org.example/unlisted@1.0"], + None, + false, + ), + ( + PINNED, + &[(main, odd)], + odd_jars.iter().map(String::as_str).collect(), + None, + false, + ), + // Every pinned hub, and a jar that two of them fetch once. + ( + &two, + &[(main, &v3), ("tools.json", LOCK_SKIPS)], + [&[sqlite], &netty[..]].concat(), + None, + false, + ), + ( + &two, + &[(main, &v3), ("tools.json", &v2)], + netty.clone(), + None, + false, + ), + // What was not read is said, once, and guessed at nowhere: + // the hubs that were read still count. + ( + &lockless, + &[(main, &v3)], + netty.clone(), + Some(format!( + "MODULE.bazel:9: hub \"tools\" has no `lock_file`, {NOT_READ}{pin}" + )), + false, + ), + ( + &two, + &[(main, &v3)], + netty.clone(), + Some(format!("tools.json: it does not exist, {NOT_READ}")), + false, + ), + ( + PINNED, + &[], + vec![], + Some(format!("maven_install.json: it does not exist, {NOT_READ}")), + false, + ), + ( + &more, + &[], + vec![], + Some(format!("{gone}:9 includes it, {NOT_READ}")), + false, + ), + // Not in a workspace that has nothing to do with the ruleset: + // what its reader could not have is about no hub. A lock at + // the root says that it may have one, and so does a + // `maven_install` of its `WORKSPACE` file. + (go, &[], vec![], None, false), + ( + go, + &[(main, &v3)], + vec![], + Some(format!("{gone}:2 includes it, {NOT_READ}")), + false, + ), + ( + "module(name = \"app\")\n", + &[("WORKSPACE", "maven_install(artifacts = [])\n")], + vec![], + Some(format!( + "WORKSPACE:1: a `maven_install` of a WORKSPACE file is not read, {NOT_READ}" + )), + false, + ), + // An entry of the block is stale once every hub was read and + // none fetches its jar at that version any more. + (&patched, &[(main, &v3)], netty.clone(), None, false), + (&patched, &[(main, LOCK_SKIPS)], vec![sqlite], None, true), + ( + &patched_lockless, + &[(main, LOCK_SKIPS)], + vec![sqlite], + Some(format!( + "MODULE.bazel:32: hub \"tools\" has no `lock_file`, {NOT_READ}{pin}" + )), + false, + ), + ( + &patched, + &[], + vec![], + Some(format!("maven_install.json: it does not exist, {NOT_READ}")), + false, + ), + ]; + for (module, locks, jars, unread, stale) in rows { + let dir = project(&[&[("MODULE.bazel", module)], locks].concat()); + let expected = HubJars { + jars: jars.into_iter().map(String::from).collect(), + unread: unread.map(|detail| (HUB_NOT_READ, detail)), + stale: Vec::from_iter(stale.then(|| (NETTY_PURL.into(), NETTY_UUID.into()))), + }; + assert_eq!(hub_jars_blocking(dir.path()), expected, "{module}"); + } + // Only for the root of a workspace, where an entry is written: + // not below it, whatever its hubs are, and not without one. + for module in [PINNED, &lockless] { + let dir = project(&[("MODULE.bazel", module), (main, &v3)]); + assert!(!hub_jars_blocking(dir.path()).jars.is_empty(), "{module}"); + let below = dir.path().join("services/billing"); + std::fs::create_dir_all(&below).unwrap(); + assert_eq!(hub_jars_blocking(&below), HubJars::default(), "{module}"); + } + let plain = project(&[("pom.xml", ""), (main, &v3)]); + assert_eq!(hub_jars_blocking(plain.path()), HubJars::default()); + } + + /// A lock is read once for all the hubs that name it, and the locks of + /// one workspace within a bound: the lock past it is a doubt, like one + /// that is too large by itself. + #[test] + fn the_locks_of_a_workspace_are_read_once_and_within_a_bound() { + let hub = |name: &str, lock: &str| { + format!("maven.install(name = \"{name}\", lock_file = \"//:{lock}.json\")\n") + }; + let hubs = [ + ("a", "a"), + ("b", "a"), + ("c", "c"), + ("d", "d"), + ("e", "e"), + ("f", "f"), + ]; + let module = hubs + .iter() + .fold(PINNED.to_string(), |module, (name, lock)| { + module + &hub(name, lock) + }); + let full = lock_72() + &" ".repeat(MAX_LOCK - lock_72().len()); + let asked = std::cell::RefCell::new(Vec::new()); + let view = |path: &str| { + asked.borrow_mut().push(path.to_string()); + match path { + "MODULE.bazel" => Entry::Text(module.clone()), + "maven_install.json" | "f.json" => Entry::Text(lock_72()), + "a.json" | "c.json" | "d.json" | "e.json" => Entry::Text(full.clone()), + _ => Entry::Absent, + } + }; + let read = Hubs::read(&view, &view); + let names = |(hub, _): &(Hub, Rc)| hub.name.clone(); + let pinned: Vec = read.pinned.iter().map(names).collect(); + assert_eq!(pinned, ["maven", "a", "b", "c", "d", "f"]); + let doubt = read.doubt.expect("a doubt"); + let why = "the locks of the workspace are larger than 64 MiB in all"; + assert_eq!((doubt.file.as_str(), doubt.why.as_str()), ("e.json", why)); + let asked = asked.borrow(); + let times = |lock: &str| asked.iter().filter(|path| *path == lock).count(); + assert_eq!(["a.json", "c.json", "e.json"].map(times), [1, 1, 1]); + } + // The block, and what a hosted run writes of it. const NETTY_UUID: &str = "0e77a000-0000-4000-8000-0000000000a1"; @@ -2446,7 +2785,7 @@ override_repo( let hubs = |module: &str| { let files = [("MODULE.bazel", module), ("maven_install.json", &lock)]; let read = read(&files); - let hub = |(hub, _): &(Hub, Lock)| format!("{} {}:{}", hub.name, hub.file, hub.line); + let hub = |(hub, _): &(Hub, _)| format!("{} {}:{}", hub.name, hub.file, hub.line); let hubs: Vec = read.pinned.iter().map(hub).collect(); (hubs, read.doubt, built(&files)) }; diff --git a/docs/ecosystems.md b/docs/ecosystems.md index ab1d2884b..dba88d8cb 100644 --- a/docs/ecosystems.md +++ b/docs/ecosystems.md @@ -1863,9 +1863,9 @@ Maven local repository, at the lock's sha256 too. So a patch in a `pom.xml`, a G an installed jar does not reach what Bazel builds. Hosted mode patches a pinned hub. The hub fetches each jar through a repository of its own, -and Bazel's `override_repo` (7.4.0 and later) puts another repository in its place. `socket-patch -get --mode hosted`, run at the Bazel root, appends one block to the root `MODULE.bazel`, -after one blank line: +and Bazel's `override_repo` (7.4.0 and later) puts another repository in its place. A hosted +`socket-patch scan` or `socket-patch get --mode hosted`, run at the Bazel root, appends +one block to the root `MODULE.bazel`, after one blank line: ``` # >>> socket-patch: Maven patches for Bazel (managed block: socket-patch writes and removes it) >>> @@ -1929,8 +1929,9 @@ to the `)` of its `override_repo`, and the entries stand in the order of their r Everything is read from the checkout: the root `MODULE.bazel` and the files it includes, for the tags of the `maven` extension of `rules_jvm_external` under whatever name the file gives the ruleset and the extension (a hub is the `install`, `artifact` and `from_toml` tags of one -`name`); the lock each hub names with `lock_file` (format 2 or 3, up to 16 MiB; each other file -up to 1 MiB); `.bazeliskrc`, `.bazelversion`, `MODULE.bazel.lock` and `.bazelrc` with the files +`name`); the lock each hub names with `lock_file` (format 2 or 3, up to 16 MiB, each lock once +for all the hubs that name it and up to 64 MiB of locks for one workspace; each other file up +to 1 MiB); `.bazeliskrc`, `.bazelversion`, `MODULE.bazel.lock` and `.bazelrc` with the files it imports (`import`, `try-import` and `try-import-if-bazel-version`, whose condition is not evaluated; a path under `%workspace%/`, or a relative one, which is read from the Bazel root). An rc file is read as Bazel reads one: a `\` before a line break joins two lines, a `#` @@ -1942,6 +1943,59 @@ no Bazel workspace (the rule is under as a jar without a classifier at the patch's version and every hub was read, nothing is read further, said or written: the run is the one it was. +A hosted `scan` at the Bazel root takes the jars that the pinned hubs fetch for packages of +the project, beside the ones it finds installed, so it finds their patches in a tree that has +no `pom.xml` and no Gradle build. A hub's jar is the one without a classifier of an artifact in +the hub's lock, when the lock has a sha256 for it, does not skip it, and lists it under a +repository (or takes it from the Maven local repository under `m2local`). Its version is the +lock's: the one Bazel builds, whatever a `pom.xml` beside the hub selects. A jar whose +repository has no known name is a package too: a patch found for it gets no entry, and +`maven_bazel_artifact` below says why. + +- **In the output.** Such a package counts in `scannedPackages`, and unless a crawl found that + version installed also in `lockfileOnlyPackages`, with `notInstalled` on its `packages[]` + entry. A hosted `get ` keeps the releases that a pinned hub fetches the same way. +- **What was not read.** No jar is guessed. When something of the hubs could not be read with + certainty (a hub without a `lock_file`, a lock or a tag that cannot be read, a `maven_install` + of a `WORKSPACE` file that Bazel may read, a block that was changed by hand), the scan says so + once, with the file, the line and the reason: `maven_bazel_hub_not_read` in `warnings[]`, a + `Warning:` line for a person, who reads "No packages found (see the warning)." when there is + no package at all. The jars of the hubs that were read are still looked up. A patch found + for one of them gets no entry while the doubt stands (`maven_bazel_not_pinned` or + `maven_bazel_unreadable` below). The warning is for a workspace that has to do with the + ruleset: a call in `MODULE.bazel` or a file it includes names `rules_jvm_external`, the + `WORKSPACE` file calls `maven_install`, or a `maven_install.json` lies at the Bazel root. In + another Bazel workspace a file that cannot be read is about no hub, and the scan is the one + without the workspace. +- **A stale entry.** When the lock of a hub moves on (a version bump, an artifact that was + removed), the entry for the release it held is stale: Bazel stops at it ("does not generate + repository"). A hosted scan takes such an entry out as `socket-patch rollback ` does, + before it pins the patch of the release the lock holds now. It reports a `rolledBack` event + (`verified` in a dry run) with the reason `maven_bazel_entry_stale`, and a person reads a + `Note:` and "Restored to its upstream registry entry". With the last entry the block + goes, and `MODULE.bazel` is the file of before its first patch. An entry is stale only when + every hub was read and none fetches the jar at the entry's version: while something is in + doubt it stays. Only the entry goes: a pin of the same patch in a `pom.xml` or a Gradle build + that still selects the release stays. +- **Beside another build.** A hub's jar with a patch is a patch of the run like any other, so a + hosted scan writes for it what `get --mode hosted` writes in that checkout. Beside a + Gradle build that is the Gradle pin too (`settings.gradle` and three files under + `.socket/gradle/`, with the warnings of that pin), whether or not the Gradle build has the + artifact. Beside a `pom.xml`, the pom lane pins the release Maven selects and says of any + other what it always says: `redirect_maven_version_not_selected` where Maven selects another + release, `redirect_maven_selection_undetermined` where it cannot tell. Those two speak of + `pom.xml` alone: the hub's jar has its entry. +- **A root above other projects.** A bare `scan` covers the project directories below `--cwd`, + and `--cwd` when it is a project. The root of a Bazel workspace needs no lockfile or build + file of its own for that: a hosted scan covers it, under the heading `== . ==`, when its hubs + have jars, something of them was not read, or the block has a stale entry. That section is a + scan of `--cwd` like any other: it also counts the packages a crawl from the root finds + installed in the projects below. +- **Not for another scan.** A scan in vendored or agent mode, one whose `--ecosystems` leaves + Maven out, a `--global` one, and one of a project below the Bazel root take no package from a + hub, give no `maven_bazel_hub_not_read` and take no stale entry out: their packages are those + of the scan without the workspace. + A hosted run writes no entry where it cannot show from the checkout that the block works. It then gives one warning, which says why and that the Bazel build keeps the upstream jar. In a tree that only Bazel builds the patch is `skipped` (`redirect_unconfirmed`) and the warning @@ -1967,7 +2021,7 @@ project below the root, whose warning names the file as "the MODULE.bazel at the |---|---| | `maven_bazel_not_at_root` | The project is below the root of its Bazel workspace. The block belongs in the root's `MODULE.bazel`: run the command there. | | `maven_bazel_not_pinned` | A hub has no `lock_file` (also one that only `maven.artifact` or `maven.from_toml` tags make): Bazel resolves its jars when it builds. Also beside a pinned hub that holds the artifact, because the hub without a lock may build the upstream jar. Give it a `lock_file` and pin it. | -| `maven_bazel_unreadable` | Something could not be read with certainty: a `maven.install` whose `name` or `lock_file` is not a plain string, `install` tags of one hub that name different locks, a construct of `MODULE.bazel` the reader does not follow, a lock that is missing, empty, not format 2 or 3, larger than 16 MiB or no JSON, a `WORKSPACE` file that calls `maven_install` while Bazel may read it, a hub on a `use_extension` with `isolate`, an rc file that cannot be read (also one that an import takes from outside the workspace by a relative path, and one with a line whose reading Bazel's own source calls an oversight: a quote that is never closed, a `\` with nothing after it; the warning names the line), a `MODULE.bazel` that is a symbolic link, or a block that was changed by hand. | +| `maven_bazel_unreadable` | Something could not be read with certainty: a `maven.install` whose `name` or `lock_file` is not a plain string, `install` tags of one hub that name different locks, a construct of `MODULE.bazel` the reader does not follow, a lock that is missing, empty, not format 2 or 3, larger than 16 MiB (or past the 64 MiB of locks read for one workspace) or no JSON, a `WORKSPACE` file that calls `maven_install` while Bazel may read it, a hub on a `use_extension` with `isolate`, an rc file that cannot be read (also one that an import takes from outside the workspace by a relative path, and one with a line whose reading Bazel's own source calls an oversight: a quote that is never closed, a `\` with nothing after it; the warning names the line), a `MODULE.bazel` that is a symbolic link, or a block that was changed by hand. | | `maven_bazel_release` | The Bazel or the ruleset release is not known to be one the block works with. Bazel: `.bazeliskrc` or `.bazelversion` names a release before 7.4.0, or neither names a release of three numbers and `MODULE.bazel.lock` is missing or older than the format Bazel 7.5.0 writes (add a `.bazelversion`). Ruleset: `MODULE.bazel.lock` does not hold the `source.json` of exactly one rules_jvm_external release (build once), that release is not one of 6.4 to 7.2 (update socket-patch), or the lock holds another release than `MODULE.bazel` now takes: an older one than its `bazel_dep` asks for, or any other than the one a `single_version_override` of the ruleset pins (build once). Versions are compared as Bazel compares them (`7.2` is below `7.2.1`). The `bazel_dep` line alone is not taken: Bazel can select a newer release, and a `single_version_override` with a version that the lock agrees with can pin an older one. An `archive_override`, `git_override`, `local_path_override` or `multiple_version_override` of the ruleset, and a `single_version_override` whose version is no plain string, leave the release unknown: the warning names the override's line. | | `maven_bazel_artifact` | The repository of this jar has no known name (a character other than letters, digits, `.`, `_` and `-` in its group, artifact or version), the lock lists no repository for the jar (it comes from the Maven local repository under `m2local`), or the patch comes without its served version or a sha256. | | `maven_bazel_no_effect` | The build would pass on the upstream jar with the block in place: the root has a `maven.override` for the artifact, or an rc file of the checkout names `--ignore_dev_dependency` without switching it off, or names `enable_bzlmod` other than switched on (`--noenable_bzlmod`, `--enable_bzlmod=false`, and Bazel's other words for it: `0`, `no`, `f`, `n`, in any case) while the checkout names no Bazel 9 or later. | @@ -2007,11 +2061,14 @@ Limits: - Not looked at: a patch of a jar with a classifier, or of another packaging than `jar`. A lock names the plain jar by group and artifact alone, so such a patch gets no entry and no word about the hub: the run is the one without the workspace. -- `socket-patch scan` does not find the artifacts of a hub yet: it pins a hub's jar when the - package is one it found elsewhere (a `pom.xml`, a Gradle build), and `get` takes any purl. It - does not take a stale entry out; `rollback` and `remove` do. +- A stale entry is taken out by a hosted `scan` at the Bazel root, by `rollback` and by + `remove`. `get --mode hosted` pins the purl it is given and leaves a stale entry of + another release where it is, so the build still stops there until one of those three ran. +- The in-memory hosted engine (GitHub App) reads no hub and writes no block. - Not seen: a hub or a `maven.override` that another Bazel module declares (its hub takes the - root's entry where it pins the same version), a `maven_install` inside a function of a `.bzl` + root's entry where it pins the same version, so an entry that is stale for the root's hubs + may still patch that one: a hosted scan takes it out all the same), a hub that the root only + takes with `use_repo` and other modules fill, a `maven_install` inside a function of a `.bzl` file, a jar that reaches Bazel without a hub (`http_jar`, a checked-in jar, the target of a `maven.override`), what the command line or an rc file outside the checkout tells Bazel (`--ignore_dev_dependency`, `--override_repository`; the machine's and the home directory's diff --git a/scripts/loc-budget.json b/scripts/loc-budget.json index 6729c125c..cf49b1496 100644 --- a/scripts/loc-budget.json +++ b/scripts/loc-budget.json @@ -4,7 +4,7 @@ "paths": [ "crates/socket-patch-core/src/bazel/" ], - "max": 2486 + "max": 2568 }, "lock-readers": { "paths": [ From 732d0b84e16529158799b9898d5e2a4150288479 Mon Sep 17 00:00:00 2001 From: Mikola Lysenko Date: Fri, 20 Nov 2026 14:27:23 -0500 Subject: [PATCH 3/3] Hold a stale Bazel entry to the policy, and keep the scan's lock bound to the scan Six fixes from a review of the hub scan, each with a test that fails without it. - The socket.yml policy now holds a stale entry of the block as it holds a package of the run. A scan of a root that `patches.ignorePaths` filters took the entry out of that root's MODULE.bazel, and so did a scan whose policy ignores the package: the sweep got the stale pins whatever the policy said. They are filtered first (`ScanPolicy::admits_restore`). - A scan with no package at all ran the sweep and never looked at a restore that left a file half written; it went on to "No packages found" and exit 0, where the same failure on the path with packages stops the run with `lockfile_write_failed` and exit 1. Both paths now share that stop (`stop_half_restored`). - A pin that is stale and whose patch is withdrawn too was handed to the restore twice and reported as two `rolledBack` events. It is taken once, as the withdrawn pin, with every file that holds it. - The bound of 64 MiB of hub locks for one workspace was added for the scan, which now reads the hubs on every hosted run at a Bazel root, but it sat in `Hubs::read` and so also changed `get`, `list`, `vex`, `rollback` and the notices: in a workspace with more locks than that, a `get ` that pinned before declined with `maven_bazel_unreadable`. The bound is the scan's alone now; every other reader takes each lock up to 16 MiB as before. - Past the bound each further lock was still read from disk and then dropped. Now the first lock that does not fit is the last one read. - `get --mode hosted` named a Maven hub that was not read for an advisory with no Maven artifact at all. The warning is given only when the advisory has a patch of a Maven artifact. Also held by a test and said in the docs, with no change in behaviour: when a stale entry goes beside a Gradle build, the hosted Gradle pin that the first scan wrote for the same patch stays, as a pin in a pom.xml does. `list` shows it, `rollback ` takes it out, and while it stays the Gradle lane declines a patch of another release of the artifact. The line budget of the Bazel group follows the code, 2568 to 2583. Co-Authored-By: Claude Fable 5.2 --- crates/socket-patch-cli/CLI_CONTRACT.md | 4 +- crates/socket-patch-cli/src/commands/get.rs | 5 + .../socket-patch-cli/src/commands/scan/mod.rs | 78 ++++++++----- .../src/commands/scan/policy.rs | 8 ++ .../src/commands/scan/withdrawn.rs | 44 ++++++- .../tests/e2e_bazel_maven_beside.rs | 107 ++++++++++++++++-- crates/socket-patch-core/src/bazel/maven.rs | 71 ++++++++---- docs/ecosystems.md | 24 +++- scripts/loc-budget.json | 2 +- 9 files changed, 271 insertions(+), 72 deletions(-) diff --git a/crates/socket-patch-cli/CLI_CONTRACT.md b/crates/socket-patch-cli/CLI_CONTRACT.md index 527dbadd3..a4172c4b7 100644 --- a/crates/socket-patch-cli/CLI_CONTRACT.md +++ b/crates/socket-patch-cli/CLI_CONTRACT.md @@ -2606,12 +2606,12 @@ command's integration tests check its output. | `redirect_withdrawn_check_failed` | `warnings[]` (warning; human: `Warning: …`) | scan `--mode hosted`: the reference lookup for the uuids the lockfiles pin failed, so the run could not tell whether any was withdrawn. Every pin is kept; status and exit unchanged. See "Withdrawn pins". | | `redirect_withdrawn_restore_failed` | `warnings[]` (warning; human: `Warning: …` / `Error: …`) | scan `--mode hosted`: a pin whose patch is `withdrawn` / `not_found` could not be restored to its upstream registry entry (the restore refused it, writing the restored files failed, or the apply lock was held). The pin is kept and no `rolledBack` event is recorded for it; the detail names the remedy (`git checkout -- `, or re-run / `socket-patch rollback `). A write failure is one warning naming every pin it kept. Status and exit unchanged, unless the failed write could not put a file back: see `lockfile_write_failed`. Also for a stale entry of a Bazel workspace's block that could not be taken out (`maven_bazel_entry_stale`). | | `patch_withdrawn` | `rolledBack` / `verified` (hosted event) | scan `--mode hosted`: the lockfile pinned a patch the server reports `withdrawn` or `not_found`; the pin was restored (dry run: would be) to its upstream registry entry. `reason` names the uuid and status. See "Withdrawn pins". | -| `maven_bazel_entry_stale` | `rolledBack` / `verified` (hosted event) | scan `--mode hosted` at the root of a Bazel workspace, with Maven in scope (additive): the block in `MODULE.bazel` held an entry for a jar that no pinned rules_jvm_external hub fetches at that version any more, at which Bazel stops ("does not generate repository"); the entry was taken out (dry run: would be), and with the last entry the block. Only when every hub was read. `reason` names the uuid. A restore that fails keeps the entry with a `redirect_withdrawn_restore_failed` warning. See the lockfile supplement and docs/ecosystems.md "Bazel workspaces: Maven". | +| `maven_bazel_entry_stale` | `rolledBack` / `verified` (hosted event) | scan `--mode hosted` at the root of a Bazel workspace, with Maven in scope (additive): the block in `MODULE.bazel` held an entry for a jar that no pinned rules_jvm_external hub fetches at that version any more, at which Bazel stops ("does not generate repository"); the entry was taken out (dry run: would be), and with the last entry the block. Only when every hub was read, and not in a root, or for a package, that the `socket.yml` policy filters. A pin of the same patch in another build's files (a `pom.xml`, the hosted Gradle index) stays. `reason` names the uuid. A restore that fails keeps the entry with a `redirect_withdrawn_restore_failed` warning; one that leaves a file half written stops the run (`lockfile_write_failed`, exit 1), also when the scan has no package. See the lockfile supplement and docs/ecosystems.md "Bazel workspaces: Maven". | | `vendor_prebuilt_stub_invalid` | `failed` | RubyGems: the server stub lacks required attributes or is otherwise invalid; no local stub fallback is permitted. | | `bazelignore_not_updated` | run-level `warnings[]` | Go vendored and agent mode in a Bazel workspace (additive): the `.bazelignore` at the Bazel root, which keeps Bazel and gazelle out of the module copies under `.socket/vendor/golang` and `.socket/go-patches`, was not updated. It could not be read or written (a symbolic link or a directory in its place, a read-only directory, a read-only file). The detail names the line to add, or says that the lines of a copy that is gone can be deleted; the command's patch or undo stands and its exit code is unchanged. See docs/ecosystems.md "Bazel workspaces: `.bazelignore`". | | `maven_bazel_hub_unpatched` | run-level `warnings[]` (human: `Warning: …`) | Maven in a Bazel workspace, vendored and agent mode (additive): a pinned rules_jvm_external hub of the root `MODULE.bazel` fetches the jar of a Maven patch that this run vendored or applied, or tried to (`vendor`, `apply`, and `get` / `scan` in those modes; in agent mode only for a package it found installed; never under `--global`), and the block that hosted mode keeps in `MODULE.bazel` holds no entry for that jar. The hub builds the jar its own lock (`maven_install.json`) names, so the patch does not reach the Bazel build: these modes patch no hub. One warning per purl; the detail names the lock, the hub, the file and line of its `maven.install`, and the hosted command that patches a pinned hub. The events, the exit code and every written file are those of the same run without the workspace. A `vendor` whose `--ecosystems` leaves Maven out gives none. A vendored run that takes a hosted pin over (`vendor`, `scan` / `get --mode vendored`, `vendor --revert`) leaves the block's entry for it as it is. A hosted run never gives it: see the `maven_bazel_*` rows below and docs/ecosystems.md "Bazel workspaces: Maven". | | `maven_bazel_unreadable` | vendored / agent: run-level `warnings[]` (human: `Warning: …`); scan/get `--mode hosted`: `warnings[]` (warning) | Maven in a Bazel workspace (additive): whether a rules_jvm_external hub builds the jar of a Maven patch could not be told, because something could not be read with certainty: a hub without `lock_file` in vendored and agent mode (hosted mode gives `maven_bazel_not_pinned`), a `maven.artifact` or `maven.from_toml` tag that cannot be read, a lock that is missing, empty, not format 2 or 3, larger than 16 MiB (or past the 64 MiB of locks that are read for one workspace) or no JSON, a `WORKSPACE` file that calls `maven_install` or cannot be read while `.bazeliskrc` or `.bazelversion` at the Bazel root names no release of Bazel 9 or later, a `maven.install` whose `name` or `lock_file` is not a plain string, a construct of `MODULE.bazel` the reader does not follow, and, in vendored and agent mode, a `maven.override` of the artifact (hosted mode gives `maven_bazel_no_effect`). Hosted mode also gives it for a hub on an isolated `use_extension`, an rc file that cannot be read (also one that an import takes from outside the workspace by a relative path, and one with a line whose reading Bazel's own source calls an oversight: a quote that is never closed, a `\` with nothing after it), a `MODULE.bazel` that is a symbolic link, and a block of `MODULE.bazel` that was changed by hand (also in its first line alone), and then writes no entry for the hub. One warning per purl; the detail names the file, the line and the reason. | -| `maven_bazel_hub_not_read` | scan run-level `warnings[]` (human: `Warning: …`); `get --mode hosted`: `warnings[]` | scan `--mode hosted` at the root of a Bazel workspace, with Maven in scope (additive): the scan takes the jars that the pinned rules_jvm_external hubs fetch for packages of the project (the lockfile supplement above), and something of the hubs could not be read with certainty, so it does not know every Maven artifact that Bazel builds through a hub. Given once per run, for the first thing that was not read, whatever the packages: a hub without `lock_file` (the detail then ends with how to pin it), or one of the reasons of `maven_bazel_unreadable` that concern reading the hubs (a tag or a lock that cannot be read, a `maven_install` of a `WORKSPACE` file that Bazel may read, a construct of `MODULE.bazel` the reader does not follow, a block that was changed by hand). `[:]: , so socket-patch does not know every Maven artifact that Bazel builds here through a rules_jvm_external hub: it looks for patches of the ones in the hub locks it read, and of no other`. No jar is guessed for what was not read; the jars of the hubs that were read are scanned, and a patch found for one of them gets no entry while the doubt stands (`maven_bazel_not_pinned` / `maven_bazel_unreadable` below). With no package at all a person reads `No packages found (see the warning).`. Exit code and `status` unchanged. A hosted `get` by advisory, which keeps the releases the project has by the same supplement, gives it too. Only in a workspace that has to do with the ruleset: a call in `MODULE.bazel` or a file it includes names `rules_jvm_external`, the `WORKSPACE` file calls `maven_install`, or a `maven_install.json` lies at the Bazel root. In another Bazel workspace a file that cannot be read gives no such warning, and the run is the one without the workspace. | +| `maven_bazel_hub_not_read` | scan run-level `warnings[]` (human: `Warning: …`); `get --mode hosted`: `warnings[]` | scan `--mode hosted` at the root of a Bazel workspace, with Maven in scope (additive): the scan takes the jars that the pinned rules_jvm_external hubs fetch for packages of the project (the lockfile supplement above), and something of the hubs could not be read with certainty, so it does not know every Maven artifact that Bazel builds through a hub. Given once per run, for the first thing that was not read, whatever the packages: a hub without `lock_file` (the detail then ends with how to pin it), or one of the reasons of `maven_bazel_unreadable` that concern reading the hubs (a tag or a lock that cannot be read, a `maven_install` of a `WORKSPACE` file that Bazel may read, a construct of `MODULE.bazel` the reader does not follow, a block that was changed by hand). `[:]: , so socket-patch does not know every Maven artifact that Bazel builds here through a rules_jvm_external hub: it looks for patches of the ones in the hub locks it read, and of no other`. No jar is guessed for what was not read; the jars of the hubs that were read are scanned, and a patch found for one of them gets no entry while the doubt stands (`maven_bazel_not_pinned` / `maven_bazel_unreadable` below). With no package at all a person reads `No packages found (see the warning).`. Exit code and `status` unchanged. A hosted `get` by advisory, which keeps the releases the project has by the same supplement, gives it too when the advisory has a patch of a Maven artifact. Only in a workspace that has to do with the ruleset: a call in `MODULE.bazel` or a file it includes names `rules_jvm_external`, the `WORKSPACE` file calls `maven_install`, or a `maven_install.json` lies at the Bazel root. In another Bazel workspace a file that cannot be read gives no such warning, and the run is the one without the workspace. | | `maven_bazel_not_at_root` | `warnings[]` (warning; human: `Warning: …`) | scan/get `--mode hosted` in a Bazel workspace (additive): a rules_jvm_external hub builds the jar of a Maven patch, and the project is below the root of its Bazel workspace, in whose `MODULE.bazel` the patch of a hub is written. Like the five codes below and `maven_bazel_unreadable` in hosted mode, it means that the run wrote no entry for the jar in the block of the root `MODULE.bazel`, because it could not show from the checkout that the entry works. One warning per purl, in the place of `redirect_maven_no_pom` for that purl; the detail says why and ends with what the block of the root holds for the jar, which is read for a project below the root too (the detail then names the file as "the MODULE.bazel at the root"). Nothing: "socket-patch wrote nothing to MODULE.bazel for it, so the Bazel build keeps the upstream jar". The event is the one of the same run without the hub: `skipped` / `redirect_unconfirmed` in a tree only Bazel builds, `applied` beside a `pom.xml` or Gradle pin that landed. The entry of the patch, from an earlier run: that entry stays, the event is `applied`, and the detail ends "socket-patch left the entry that MODULE.bazel holds for it as it is". The entry of another patch of the same jar: it stays too, the event is the one without the hub, and the detail ends "socket-patch wrote nothing to MODULE.bazel for it: the entry of patch remains there, and where the block is in effect the Bazel build takes that patch's jar". Exit code unchanged. With none of these codes the run writes the block: `redirect.rewrittenFiles` holds `MODULE.bazel`, the event is `applied`, and `rollback` / `remove` take the entry out. See docs/ecosystems.md "Bazel workspaces: Maven". | | `maven_bazel_not_pinned` | `warnings[]` (warning; human: `Warning: …`) | As `maven_bazel_not_at_root`: a hub of the workspace has no `lock_file` (also one that only `maven.artifact` or `maven.from_toml` tags make), so Bazel resolves its jars when it builds and none can be replaced. Also given beside a pinned hub that holds the artifact. The detail names the hub's file and line and how to pin it. | | `maven_bazel_release` | `warnings[]` (warning; human: `Warning: …`) | As `maven_bazel_not_at_root`: the Bazel release is not known to be 7.4.0 or later, the first with `override_repo` (`.bazeliskrc` or `.bazelversion` names an older one, or neither names a release of three numbers and `MODULE.bazel.lock` is missing or of a format before the one Bazel 7.5.0 writes), or the rules_jvm_external release Bazel selected is not known to be one of 6.4 to 7.2 (`MODULE.bazel.lock` holds the `source.json` of no release of it or of more than one, of a release outside that range, of an older one than the `bazel_dep` of `MODULE.bazel` asks for, or of another one than a `single_version_override` there pins; or an `archive_override`, `git_override`, `local_path_override` or `multiple_version_override` of the ruleset, or a `single_version_override` whose version is no plain string, leaves the release unknown). Versions are compared as Bazel compares them. The detail names the remedy (a `.bazelversion`, one build so that Bazel writes its lock, or an update of socket-patch) or the line of the override. | diff --git a/crates/socket-patch-cli/src/commands/get.rs b/crates/socket-patch-cli/src/commands/get.rs index 021a1ecd2..814506f6d 100644 --- a/crates/socket-patch-cli/src/commands/get.rs +++ b/crates/socket-patch-cli/src/commands/get.rs @@ -930,6 +930,11 @@ async fn filter_to_installed_purls( let hosted = mode == super::scan::ScanMode::Hosted; let supplement = super::scan::project_lockfile_supplement(&ctx, &[], None, hosted).await; pnp_diags = supplement.unsupported; + // What was not read of a Bazel workspace's Maven hubs says nothing + // of patches that are for no Maven artifact. + let maven = accessible.iter().any(|p| p.purl.starts_with("pkg:maven/")); + let hub = socket_patch_core::bazel::maven::HUB_NOT_READ; + pnp_diags.retain(|diag| maven || diag.code != hub); if mode != super::scan::ScanMode::Agent { present.extend(supplement.entries.iter().map(|e| PurlKey::new(&e.purl))); // A hosted run's supplement also names the gems the lock diff --git a/crates/socket-patch-cli/src/commands/scan/mod.rs b/crates/socket-patch-cli/src/commands/scan/mod.rs index f4a950c61..140ecf49c 100644 --- a/crates/socket-patch-cli/src/commands/scan/mod.rs +++ b/crates/socket-patch-cli/src/commands/scan/mod.rs @@ -1603,6 +1603,35 @@ fn emit_scan_error(common: &GlobalArgs, error: EnvelopeError) { emit_scan(&env); } +/// A restore of `sweep` that failed part-way and could not put its files +/// back leaves the lockfiles half restored: the run stops there with exit +/// 1 (`Err`), rather than write over them. The restore printed the +/// failure as it happened; `counts` are the scanned and the lockfile-only +/// packages of the envelope. Otherwise `sweep` goes on. +fn stop_half_restored( + common: &GlobalArgs, + sweep: Option, + counts: (usize, usize), + paths: &crate::path_scope::PathScope, +) -> Result, i32> { + let half = |sweep: &withdrawn::WithdrawnSweep| sweep.half_restored().map(str::to_string); + let Some(why) = sweep.as_ref().and_then(half) else { + return Ok(sweep); + }; + if common.json { + let mut env = scan_envelope(common); + env.set_extra("scannedPackages", serde_json::json!(counts.0)); + env.set_extra("lockfileOnlyPackages", serde_json::json!(counts.1)); + env.set_extra("paths", serde_json::json!(paths.raw())); + if let Some(sweep) = sweep { + sweep.record_into(&mut env); + } + env.mark_error(EnvelopeError::new("lockfile_write_failed", why)); + emit_scan(&env); + } + Err(1) +} + /// The ecosystems `purls` span, by CLI name, sorted and deduplicated. fn scanned_ecosystems(purls: &[String]) -> Vec { purls @@ -2908,6 +2937,14 @@ async fn scan_project( let all_purls: Vec = filtered_crawled.iter().map(|p| p.purl.clone()).collect(); let package_count = all_purls.len(); + // The stale entries of a Bazel workspace's block that this run may + // take out: the policy holds them as it holds the run's packages. + let stale_hub_pins: Vec<_> = lockfile_only + .stale_hub_pins + .iter() + .filter(|pin| policy.admits_restore(&pin.purl)) + .cloned() + .collect(); if package_count == 0 { // An empty crawl is where the projects below matter most: they @@ -2943,11 +2980,14 @@ async fn scan_project( // any more, whatever the project holds: such an entry is taken // out here too (`withdrawn::sweep`, which asks nothing for it). let stale_sweep = if hosted && policy.writes_allowed() { - let stale = &lockfile_only.stale_hub_pins; - Some(withdrawn::sweep(&args.common, &api_client, &[], stale).await) + Some(withdrawn::sweep(&args.common, &api_client, &[], &stale_hub_pins).await) } else { None }; + let stale_sweep = match stop_half_restored(&args.common, stale_sweep, (0, 0), &path_scope) { + Ok(sweep) => sweep, + Err(code) => return code, + }; if human { print_layout_refusals(&layout_refusals, &project_patch_notice, args.common.silent); policy.print_warnings(args.common.silent); @@ -3386,36 +3426,18 @@ async fn scan_project( // longer serves, before the rewrite below re-pins any replacement. // Runs ahead of every "nothing found" exit: a package whose only patch // was withdrawn has no search results at all. - let mut withdrawn_sweep = if hosted && !args.common.is_global() && policy.writes_allowed() { + let withdrawn_sweep = if hosted && !args.common.is_global() && policy.writes_allowed() { let pins = withdrawn::pins_in_scope(&hosted_pin_list, &all_purls); - let stale = &lockfile_only.stale_hub_pins; - Some(withdrawn::sweep(&args.common, &api_client, &pins, stale).await) + Some(withdrawn::sweep(&args.common, &api_client, &pins, &stale_hub_pins).await) } else { None }; - // A restore that failed part-way and could not put its files back - // leaves the lockfiles half restored: stop, rather than rewrite over - // them. (The restore printed the failure as it happened.) - if let Some(why) = withdrawn_sweep - .as_ref() - .and_then(|sweep| sweep.half_restored().map(str::to_string)) - { - if args.common.json { - let mut env = scan_envelope(&args.common); - env.set_extra("scannedPackages", serde_json::json!(package_count)); - env.set_extra( - "lockfileOnlyPackages", - serde_json::json!(lockfile_only_count), - ); - env.set_extra("paths", serde_json::json!(path_scope.raw())); - if let Some(sweep) = withdrawn_sweep { - sweep.record_into(&mut env); - } - env.mark_error(EnvelopeError::new("lockfile_write_failed", why)); - emit_scan(&env); - } - return 1; - } + let counts = (package_count, lockfile_only_count); + let mut withdrawn_sweep = + match stop_half_restored(&args.common, withdrawn_sweep, counts, &path_scope) { + Ok(sweep) => sweep, + Err(code) => return code, + }; // The sweep's reference answers, reused by the hosted redirect. let sweep_references = withdrawn_sweep .as_mut() diff --git a/crates/socket-patch-cli/src/commands/scan/policy.rs b/crates/socket-patch-cli/src/commands/scan/policy.rs index 33912308d..616ccac1a 100644 --- a/crates/socket-patch-cli/src/commands/scan/policy.rs +++ b/crates/socket-patch-cli/src/commands/scan/policy.rs @@ -472,6 +472,14 @@ impl ScanPolicy { self.policy.enabled() } + /// Whether the run may take out a pin of `purl` that none of its + /// packages stands for (a stale entry of a Bazel workspace's block): + /// not in a root the policy filtered, and not for a package it + /// filters, as for a pin of a package the run covers. + pub(crate) fn admits_restore(&self, purl: &str) -> bool { + self.root_verdict.is_ok() && self.policy.admits_purl(purl).is_ok() + } + /// Set the merged recorded view (manifest > hosted pins > vendor ledger). pub(crate) fn set_recorded(&mut self, merged: Option<&PatchManifest>) { self.recorded = merged diff --git a/crates/socket-patch-cli/src/commands/scan/withdrawn.rs b/crates/socket-patch-cli/src/commands/scan/withdrawn.rs index a8096cdcd..ba46000b6 100644 --- a/crates/socket-patch-cli/src/commands/scan/withdrawn.rs +++ b/crates/socket-patch-cli/src/commands/scan/withdrawn.rs @@ -226,6 +226,23 @@ async fn withdrawn( gone } +/// `gone` and the pins of `stale` after it. A stale pin whose patch is +/// withdrawn too is in `gone` already, with every file that holds it: it +/// is restored, and reported, once. +fn with_stale(mut gone: Vec, stale: &[HostedPin]) -> Vec { + let known = |pin: &&HostedPin| { + let same = |(gone, ..): &Gone| (&gone.purl, &gone.uuid) == (&pin.purl, &pin.uuid); + !gone.iter().any(same) + }; + let stale: Vec<_> = stale.iter().filter(known).cloned().collect(); + gone.extend( + stale + .into_iter() + .map(|pin| (pin, MAVEN_BAZEL_ENTRY_STALE, STALE.to_string())), + ); + gone +} + /// Check `pins` against the reference endpoint and restore the withdrawn /// ones upstream (see the module docs), and with them the pins of `stale`, /// which are asked about nowhere: the entries of a Bazel workspace's block @@ -238,9 +255,8 @@ pub(super) async fn sweep( stale: &[HostedPin], ) -> WithdrawnSweep { let mut out = WithdrawnSweep::default(); - let mut gone = withdrawn(common, api_client, pins, &mut out).await; - let stale = stale.iter().cloned(); - gone.extend(stale.map(|pin| (pin, MAVEN_BAZEL_ENTRY_STALE, STALE.to_string()))); + let gone = withdrawn(common, api_client, pins, &mut out).await; + let gone = with_stale(gone, stale); if gone.is_empty() { return out; } @@ -390,6 +406,28 @@ mod tests { assert!(pins_in_scope(&pins, Vec::::new()).is_empty()); } + #[test] + fn a_stale_pin_that_is_withdrawn_too_is_taken_once() { + let both = HostedPin { + files: vec!["MODULE.bazel".to_string(), "pom.xml".to_string()], + ..pin("pkg:maven/g/a@1.0", "u1") + }; + let stale = [ + pin("pkg:maven/g/a@1.0", "u1"), + pin("pkg:maven/g/b@2.0", "u2"), + ]; + let gone = vec![(both.clone(), PATCH_WITHDRAWN, "is gone".to_string())]; + let taken: Vec<_> = with_stale(gone, &stale) + .into_iter() + .map(|(pin, code, _)| (pin, code)) + .collect(); + let expected = [ + (both, PATCH_WITHDRAWN), + (stale[1].clone(), MAVEN_BAZEL_ENTRY_STALE), + ]; + assert_eq!(taken, expected); + } + #[test] fn only_settled_answers_are_known() { let answer = |status: &str| -> PackageVendorResult { diff --git a/crates/socket-patch-cli/tests/e2e_bazel_maven_beside.rs b/crates/socket-patch-cli/tests/e2e_bazel_maven_beside.rs index 93deaa1fd..5da884d89 100644 --- a/crates/socket-patch-cli/tests/e2e_bazel_maven_beside.rs +++ b/crates/socket-patch-cli/tests/e2e_bazel_maven_beside.rs @@ -1222,6 +1222,19 @@ fn a_hosted_scan_takes_the_jars_of_a_hub_beside_a_lockfile_of_the_root() { /// The block for a patch `uuid` of the victim at `version`, as /// socket-patch writes it. +/// A Gradle build that has nothing of the hub's jars. +const GRADLE: [(&str, &str); 3] = [ + ("settings.gradle", "rootProject.name = 'app'\n"), + ( + "build.gradle", + "plugins { id 'java' }\ndependencies { implementation 'com.example:other:1.0' }\n", + ), + ( + "gradle.lockfile", + "com.example:other:1.0=compileClasspath\nempty=\n", + ), +]; + fn block_at(version: &str, uuid: &str) -> String { let name = version.replace('.', "_"); let block = BLOCK.replace("1.10.0", version).replace("1_10_0", &name); @@ -1388,6 +1401,64 @@ fn a_hosted_scan_takes_a_stale_entry_out_of_the_block() { [event(PURL, "rolledBack", stale), event(PURL, "applied", "")], "{scan:?}" ); + + // Beside a Gradle build the same: the entry alone goes, and the pin + // that the first scan wrote for Gradle stays for `list` to show and + // for `rollback ` to take out. + let mut files = pinned(); + files.extend(GRADLE.iter().map(|(path, text)| (*path, text.to_string()))); + machine.make(&files, false, false); + assert_eq!(machine.step(&SCAN).wrote.len(), 5); + std::fs::write(machine.root.join("proj/maven_install.json"), &moved).expect("the lock"); + let scan = machine.step(&SCAN); + assert_eq!(scan.wrote.keys().collect::>(), [MODULE], "{scan:?}"); + assert_eq!( + triples(&scan), + [event(PURL, "rolledBack", stale)], + "{scan:?}" + ); + let list = machine.step(&["list", "--json"]).envelope(); + assert_eq!( + event_for(&list, PURL)["details"]["lockfiles"], + json!([".socket/gradle/hosted-index.tsv"]), + "{list:#}" + ); +} + +/// The socket.yml policy holds a stale entry as it holds the packages of +/// the run: a scan of a root that the policy filters, and one whose +/// policy filters the package, leave the entry where it is. +#[test] +fn a_hosted_scan_leaves_a_stale_entry_that_the_policy_filters() { + let machine = Machine::start(); + let mut files = pinned(); + files[0].1 = format!( + "{HUB}\n{}", + block_at("1.9.0", "0a1b2c3d-1111-4222-8333-444455556666") + ); + let below: Vec<(String, String)> = files + .iter() + .map(|(path, text)| (format!("legacy/{path}"), text.clone())) + .collect(); + let mut below: Files = below + .iter() + .map(|(path, text)| (&*path.clone().leak(), text.clone())) + .collect(); + let filtered = |policy: &str| ("socket.yml", format!("version: 2\npatches:\n {policy}\n")); + below.push(filtered("ignorePaths: [\"/legacy/\"]")); + files.push(filtered( + "ignorePackages: [\"pkg:maven/dev.socket.fixture/victim\"]", + )); + for (files, dir) in [(&below, "proj/legacy"), (&files, "proj")] { + machine.make(files, false, false); + let scan = machine.step_in(&machine.root.join(dir), &SCAN); + let said = scan.envelope(); + assert_eq!( + (scan.code, &said["events"], &scan.wrote), + (Some(0), &json!([]), &BTreeMap::new()), + "{dir}: {scan:?}" + ); + } } /// A checkout that Gradle or Maven builds too. A hub's jar with a patch is @@ -1397,17 +1468,6 @@ fn a_hosted_scan_takes_a_stale_entry_out_of_the_block() { #[test] fn a_hosted_scan_pins_a_hubs_jar_in_the_other_builds_as_get_does() { let machine = Machine::start(); - let gradle = [ - ("settings.gradle", "rootProject.name = 'app'\n"), - ( - "build.gradle", - "plugins { id 'java' }\ndependencies { implementation 'com.example:other:1.0' }\n", - ), - ( - "gradle.lockfile", - "com.example:other:1.0=compileClasspath\nempty=\n", - ), - ]; // A pom whose dependency is not in the Maven local repository: what // Maven selects cannot be told. let other = PROJECT_POM @@ -1415,7 +1475,7 @@ fn a_hosted_scan_pins_a_hubs_jar_in_the_other_builds_as_get_does() { .replace("victim", "other"); let rows: [(&[(&str, &str)], &[&str], &[&str]); 2] = [ ( - &gradle, + &GRADLE, &[ "proj/.socket/gradle/.gitattributes", "proj/.socket/gradle/hosted-index.tsv", @@ -1744,6 +1804,29 @@ fn a_hosted_get_by_advisory_keeps_the_release_a_pinned_hub_fetches() { (&json!("package_not_installed"), &BTreeMap::new()), "{unread:?}" ); + + // An advisory with no Maven artifact: the hub that was not read has + // nothing to do with it, and is not named. + let (npm, other) = ("pkg:npm/left-pad@1.3.0", "GHSA-test-cccc-dddd"); + let found = json!({ "uuid": UUID, "purl": npm, "publishedAt": "2024-01-01T00:00:00Z", + "description": "fixture", "license": "MIT", "tier": "free", "vulnerabilities": {} }); + let answer = json!({ "patches": [found], "canAccessPaidPatches": false }); + let search = wiremock::Mock::given(method("GET")) + .and(path(format!("/v0/orgs/{ORG}/patches/by-ghsa/{other}"))) + .respond_with(wiremock::ResponseTemplate::new(200).set_body_json(answer)); + machine._runtime.block_on(search.mount(&machine.api)); + let lockless = HUB.replace(" lock_file = \"//:maven_install.json\",\n", ""); + machine.make(&[("MODULE.bazel", lockless)], false, false); + let unrelated = machine.step(&["get", other, "--mode", "hosted", "--yes", "--json"]); + let envelope = unrelated.envelope(); + assert_eq!( + ( + &envelope["warnings"], + &event_for(&envelope, npm)["errorCode"] + ), + (&Value::Null, &json!("package_not_installed")), + "{unrelated:?}" + ); } #[test] diff --git a/crates/socket-patch-core/src/bazel/maven.rs b/crates/socket-patch-core/src/bazel/maven.rs index 7865c63a3..c8b52bd51 100644 --- a/crates/socket-patch-core/src/bazel/maven.rs +++ b/crates/socket-patch-core/src/bazel/maven.rs @@ -120,8 +120,10 @@ const RULESETS: [&str; 10] = [ /// What a doubt about a hub without a lock ends with. const NO_LOCK: &str = "has no `lock_file`"; -/// The largest lock that is read, and how many bytes of locks are read for -/// one workspace: each lock once, for every hub that names it. +/// The largest lock that is read, each lock once for every hub that names +/// it, and how many bytes of locks a scan takes its packages from in one +/// workspace ([`hub_jars`]): the first lock that does not fit is the last +/// one read. const MAX_LOCK: usize = 16 << 20; const MAX_LOCKS: usize = 4 * MAX_LOCK; @@ -485,6 +487,17 @@ impl Hubs { /// `maven_install` or cannot be read while Bazel may read it (the /// workspace names no release, or one before 9) are doubts. pub fn read(view: &dyn Fn(&str) -> Entry, locks: &dyn Fn(&str) -> Entry) -> Hubs { + Hubs::read_within(view, locks, usize::MAX) + } + + /// [`Hubs::read`] with at most `room` bytes of locks in all: the lock + /// that does not fit is a doubt, and so is every lock after it, which + /// is not read. + fn read_within( + view: &dyn Fn(&str) -> Entry, + locks: &dyn Fn(&str) -> Entry, + mut room: usize, + ) -> Hubs { // The block is socket-patch's own: the workspace is read without // it, as the user wrote it, and with every line where it was. A // block that was changed by hand is read like any other text. @@ -528,7 +541,13 @@ impl Hubs { }); let overrides = overrides.collect(); let (mut pinned, mut lockless) = (Vec::new(), None); - let (mut read_locks, mut room) = (HashMap::new(), MAX_LOCKS); + let mut read_locks = HashMap::new(); + let past = || { + let bound = MAX_LOCKS >> 20; + Err(format!( + "the locks of the workspace are larger than {bound} MiB in all" + )) + }; for hub in hubs { // Where a doubt about its lock is: the lock, or the tag. let (file, line) = match &hub.lock { @@ -538,17 +557,18 @@ impl Hubs { let lock = match &hub.lock { None => Err(format!("hub \"{}\" {NO_LOCK}", hub.name)), Some(path) => { - let parsed = || match locks(path) { - Entry::Text(text) if text.len() > room => Err(format!( - "the locks of the workspace are larger than {} MiB in all", - MAX_LOCKS >> 20 - )), - Entry::Text(text) => { + let parsed = || match (room > 0).then(|| locks(path)) { + None => past(), + Some(Entry::Text(text)) if text.len() > room => { + room = 0; + past() + } + Some(Entry::Text(text)) => { room -= text.len(); Lock::parse(&text).map(Rc::new) } - Entry::Absent => Err("it does not exist".to_string()), - Entry::Unreadable(why) => Err(format!("it cannot be read: {why}")), + Some(Entry::Absent) => Err("it does not exist".to_string()), + Some(Entry::Unreadable(why)) => Err(format!("it cannot be read: {why}")), }; let lock = read_locks.entry(path.clone()).or_insert_with(parsed); lock.clone() @@ -1361,7 +1381,8 @@ fn hub_jars_blocking(project: &Path) -> HubJars { let Some(root) = root else { return HubJars::default(); }; - let hubs = Hubs::read(&disk(&root.dir), &disk_up_to(&root.dir, MAX_LOCK)); + let locks = disk_up_to(&root.dir, MAX_LOCK); + let hubs = Hubs::read_within(&disk(&root.dir), &locks, MAX_LOCKS); // Each lock once, however many hubs name it. let (mut purls, mut seen) = (BTreeSet::new(), HashSet::new()); let locks = hubs.pinned.iter().map(|(_, lock)| lock); @@ -2467,9 +2488,10 @@ jvm.artifact(group = "com.example", artifact = "a", version = "1") assert_eq!(hub_jars_blocking(plain.path()), HubJars::default()); } - /// A lock is read once for all the hubs that name it, and the locks of - /// one workspace within a bound: the lock past it is a doubt, like one - /// that is too large by itself. + /// A lock is read once for all the hubs that name it. A scan takes + /// the locks of one workspace within a bound: the lock past it is a + /// doubt, like one that is too large by itself, and no lock after it + /// is read. Every other reader takes them all, as before. #[test] fn the_locks_of_a_workspace_are_read_once_and_within_a_bound() { let hub = |name: &str, lock: &str| { @@ -2499,16 +2521,25 @@ jvm.artifact(group = "com.example", artifact = "a", version = "1") _ => Entry::Absent, } }; - let read = Hubs::read(&view, &view); let names = |(hub, _): &(Hub, Rc)| hub.name.clone(); + let read = Hubs::read_within(&view, &view, MAX_LOCKS); let pinned: Vec = read.pinned.iter().map(names).collect(); - assert_eq!(pinned, ["maven", "a", "b", "c", "d", "f"]); + assert_eq!(pinned, ["maven", "a", "b", "c", "d"]); let doubt = read.doubt.expect("a doubt"); let why = "the locks of the workspace are larger than 64 MiB in all"; assert_eq!((doubt.file.as_str(), doubt.why.as_str()), ("e.json", why)); - let asked = asked.borrow(); - let times = |lock: &str| asked.iter().filter(|path| *path == lock).count(); - assert_eq!(["a.json", "c.json", "e.json"].map(times), [1, 1, 1]); + let times = |lock: &str| asked.borrow().iter().filter(|path| *path == lock).count(); + assert_eq!( + ["a.json", "c.json", "e.json", "f.json"].map(times), + [1, 1, 1, 0] + ); + + asked.borrow_mut().clear(); + let read = Hubs::read(&view, &view); + let pinned: Vec = read.pinned.iter().map(names).collect(); + assert_eq!(pinned, ["maven", "a", "b", "c", "d", "e", "f"]); + assert!(read.doubt.is_none()); + assert_eq!(["a.json", "e.json", "f.json"].map(times), [1, 1, 1]); } // The block, and what a hosted run writes of it. diff --git a/docs/ecosystems.md b/docs/ecosystems.md index dba88d8cb..9405515bc 100644 --- a/docs/ecosystems.md +++ b/docs/ecosystems.md @@ -1930,8 +1930,7 @@ Everything is read from the checkout: the root `MODULE.bazel` and the files it i the tags of the `maven` extension of `rules_jvm_external` under whatever name the file gives the ruleset and the extension (a hub is the `install`, `artifact` and `from_toml` tags of one `name`); the lock each hub names with `lock_file` (format 2 or 3, up to 16 MiB, each lock once -for all the hubs that name it and up to 64 MiB of locks for one workspace; each other file up -to 1 MiB); `.bazeliskrc`, `.bazelversion`, `MODULE.bazel.lock` and `.bazelrc` with the files +for all the hubs that name it; each other file up to 1 MiB); `.bazeliskrc`, `.bazelversion`, `MODULE.bazel.lock` and `.bazelrc` with the files it imports (`import`, `try-import` and `try-import-if-bazel-version`, whose condition is not evaluated; a path under `%workspace%/`, or a relative one, which is read from the Bazel root). An rc file is read as Bazel reads one: a `\` before a line break joins two lines, a `#` @@ -1954,7 +1953,8 @@ repository has no known name is a package too: a patch found for it gets no entr - **In the output.** Such a package counts in `scannedPackages`, and unless a crawl found that version installed also in `lockfileOnlyPackages`, with `notInstalled` on its `packages[]` - entry. A hosted `get ` keeps the releases that a pinned hub fetches the same way. + entry. A hosted `get ` keeps the releases that a pinned hub fetches the same way, + and names a hub that was not read only when the advisory has a patch of a Maven artifact. - **What was not read.** No jar is guessed. When something of the hubs could not be read with certainty (a hub without a `lock_file`, a lock or a tag that cannot be read, a `maven_install` of a `WORKSPACE` file that Bazel may read, a block that was changed by hand), the scan says so @@ -1967,6 +1967,11 @@ repository has no known name is a package too: a patch found for it gets no entr `WORKSPACE` file calls `maven_install`, or a `maven_install.json` lies at the Bazel root. In another Bazel workspace a file that cannot be read is about no hub, and the scan is the one without the workspace. +- **How much is read.** A scan takes its packages from up to 64 MiB of hub locks in one + workspace. The first lock that does not fit is not taken, no lock after it is read, and the + warning above names it ("the locks of the workspace are larger than 64 MiB in all"). That + bound is the scan's alone: a patch that is found is pinned as `get ` pins it, which + reads each lock it needs up to 16 MiB. - **A stale entry.** When the lock of a hub moves on (a version bump, an artifact that was removed), the entry for the release it held is stale: Bazel stops at it ("does not generate repository"). A hosted scan takes such an entry out as `socket-patch rollback ` does, @@ -1975,8 +1980,15 @@ repository has no known name is a package too: a patch found for it gets no entr `Note:` and "Restored to its upstream registry entry". With the last entry the block goes, and `MODULE.bazel` is the file of before its first patch. An entry is stale only when every hub was read and none fetches the jar at the entry's version: while something is in - doubt it stays. Only the entry goes: a pin of the same patch in a `pom.xml` or a Gradle build - that still selects the release stays. + doubt it stays. The `socket.yml` policy holds such an entry as it holds a package of the + run: a scan of a root that the policy filters, or one whose policy filters the package, + leaves it. Only the entry goes. A pin of the same patch that another build of the checkout + holds stays (a `pom.xml`; the hosted Gradle pin, also where the Gradle build has no such + artifact), since nothing in the checkout says that the other build is done with the + release: `socket-patch list` shows it and `socket-patch rollback ` takes it out. + While a Gradle pin of the old release stays, the Gradle lane declines a patch of another + release of the artifact (`redirect_gradle_version_conflict`); the entry of the block is + written all the same. - **Beside another build.** A hub's jar with a patch is a patch of the run like any other, so a hosted scan writes for it what `get --mode hosted` writes in that checkout. Beside a Gradle build that is the Gradle pin too (`settings.gradle` and three files under @@ -2021,7 +2033,7 @@ project below the root, whose warning names the file as "the MODULE.bazel at the |---|---| | `maven_bazel_not_at_root` | The project is below the root of its Bazel workspace. The block belongs in the root's `MODULE.bazel`: run the command there. | | `maven_bazel_not_pinned` | A hub has no `lock_file` (also one that only `maven.artifact` or `maven.from_toml` tags make): Bazel resolves its jars when it builds. Also beside a pinned hub that holds the artifact, because the hub without a lock may build the upstream jar. Give it a `lock_file` and pin it. | -| `maven_bazel_unreadable` | Something could not be read with certainty: a `maven.install` whose `name` or `lock_file` is not a plain string, `install` tags of one hub that name different locks, a construct of `MODULE.bazel` the reader does not follow, a lock that is missing, empty, not format 2 or 3, larger than 16 MiB (or past the 64 MiB of locks read for one workspace) or no JSON, a `WORKSPACE` file that calls `maven_install` while Bazel may read it, a hub on a `use_extension` with `isolate`, an rc file that cannot be read (also one that an import takes from outside the workspace by a relative path, and one with a line whose reading Bazel's own source calls an oversight: a quote that is never closed, a `\` with nothing after it; the warning names the line), a `MODULE.bazel` that is a symbolic link, or a block that was changed by hand. | +| `maven_bazel_unreadable` | Something could not be read with certainty: a `maven.install` whose `name` or `lock_file` is not a plain string, `install` tags of one hub that name different locks, a construct of `MODULE.bazel` the reader does not follow, a lock that is missing, empty, not format 2 or 3, larger than 16 MiB or no JSON, a `WORKSPACE` file that calls `maven_install` while Bazel may read it, a hub on a `use_extension` with `isolate`, an rc file that cannot be read (also one that an import takes from outside the workspace by a relative path, and one with a line whose reading Bazel's own source calls an oversight: a quote that is never closed, a `\` with nothing after it; the warning names the line), a `MODULE.bazel` that is a symbolic link, or a block that was changed by hand. | | `maven_bazel_release` | The Bazel or the ruleset release is not known to be one the block works with. Bazel: `.bazeliskrc` or `.bazelversion` names a release before 7.4.0, or neither names a release of three numbers and `MODULE.bazel.lock` is missing or older than the format Bazel 7.5.0 writes (add a `.bazelversion`). Ruleset: `MODULE.bazel.lock` does not hold the `source.json` of exactly one rules_jvm_external release (build once), that release is not one of 6.4 to 7.2 (update socket-patch), or the lock holds another release than `MODULE.bazel` now takes: an older one than its `bazel_dep` asks for, or any other than the one a `single_version_override` of the ruleset pins (build once). Versions are compared as Bazel compares them (`7.2` is below `7.2.1`). The `bazel_dep` line alone is not taken: Bazel can select a newer release, and a `single_version_override` with a version that the lock agrees with can pin an older one. An `archive_override`, `git_override`, `local_path_override` or `multiple_version_override` of the ruleset, and a `single_version_override` whose version is no plain string, leave the release unknown: the warning names the override's line. | | `maven_bazel_artifact` | The repository of this jar has no known name (a character other than letters, digits, `.`, `_` and `-` in its group, artifact or version), the lock lists no repository for the jar (it comes from the Maven local repository under `m2local`), or the patch comes without its served version or a sha256. | | `maven_bazel_no_effect` | The build would pass on the upstream jar with the block in place: the root has a `maven.override` for the artifact, or an rc file of the checkout names `--ignore_dev_dependency` without switching it off, or names `enable_bzlmod` other than switched on (`--noenable_bzlmod`, `--enable_bzlmod=false`, and Bazel's other words for it: `0`, `no`, `f`, `n`, in any case) while the checkout names no Bazel 9 or later. | diff --git a/scripts/loc-budget.json b/scripts/loc-budget.json index cf49b1496..893f07e38 100644 --- a/scripts/loc-budget.json +++ b/scripts/loc-budget.json @@ -4,7 +4,7 @@ "paths": [ "crates/socket-patch-core/src/bazel/" ], - "max": 2568 + "max": 2583 }, "lock-readers": { "paths": [