diff --git a/crates/socket-patch-cli/CLI_CONTRACT.md b/crates/socket-patch-cli/CLI_CONTRACT.md index 2596ee4c8..e3f579e86 100644 --- a/crates/socket-patch-cli/CLI_CONTRACT.md +++ b/crates/socket-patch-cli/CLI_CONTRACT.md @@ -2607,7 +2607,7 @@ command's integration tests check its output. | `redirect_withdrawn_restore_failed` | `warnings[]` (warning; human: `Warning: …` / `Error: …`) | scan `--mode hosted`: a pin whose patch is `withdrawn` / `not_found` could not be restored to its upstream registry entry (the restore refused it, writing the restored files failed, or the apply lock was held). The pin is kept and no `rolledBack` event is recorded for it; the detail names the remedy (`git checkout -- `, or re-run / `socket-patch rollback `). A write failure is one warning naming every pin it kept. Status and exit unchanged, unless the failed write could not put a file back: see `lockfile_write_failed`. | | `patch_withdrawn` | `rolledBack` / `verified` (hosted event) | scan `--mode hosted`: the lockfile pinned a patch the server reports `withdrawn` or `not_found`; the pin was restored (dry run: would be) to its upstream registry entry. `reason` names the uuid and status. See "Withdrawn pins". | | `vendor_prebuilt_stub_invalid` | `failed` | RubyGems: the server stub lacks required attributes or is otherwise invalid; no local stub fallback is permitted. | -| `bazelignore_not_updated` | run-level `warnings[]` | Go vendored and agent mode in a Bazel workspace (additive): the `.bazelignore` at the Bazel root, which keeps Bazel and gazelle out of the module copies under `.socket/vendor/golang` and `.socket/go-patches`, was not updated. It could not be read or written (a symbolic link or a directory in its place, a read-only directory, a read-only file). The detail names the line to add, or says that the lines of a copy that is gone can be deleted; the command's patch or undo stands and its exit code is unchanged. See docs/ecosystems.md "Bazel workspaces: `.bazelignore`". | +| `bazelignore_not_updated` | run-level `warnings[]` | Go vendored and agent mode, and Cargo vendored mode under a rules_rust crate hub, in a Bazel workspace (additive): the `.bazelignore` at the Bazel root, which keeps Bazel and gazelle out of the module copies under `.socket/vendor/golang` and `.socket/go-patches` and out of the crates under `.socket/vendor/cargo`, was not updated. It could not be read or written (a symbolic link or a directory in its place, a read-only directory, a read-only file). The detail names the line to add, or says that the lines of a copy that is gone can be deleted; the command's patch or undo stands and its exit code is unchanged. See docs/ecosystems.md "Bazel workspaces: `.bazelignore`". | | `maven_bazel_hub_unpatched` | run-level `warnings[]` (human: `Warning: …`) | Maven in a Bazel workspace, vendored and agent mode (additive): a pinned rules_jvm_external hub of the root `MODULE.bazel` fetches the jar of a Maven patch that this run vendored or applied, or tried to (`vendor`, `apply`, and `get` / `scan` in those modes; in agent mode only for a package it found installed; never under `--global`), and the block that hosted mode keeps in `MODULE.bazel` holds no entry for that jar. The hub builds the jar its own lock (`maven_install.json`) names, so the patch does not reach the Bazel build: these modes patch no hub. One warning per purl; the detail names the lock, the hub, the file and line of its `maven.install`, and the hosted command that patches a pinned hub. The events, the exit code and every written file are those of the same run without the workspace. A `vendor` whose `--ecosystems` leaves Maven out gives none. A vendored run that takes a hosted pin over (`vendor`, `scan` / `get --mode vendored`, `vendor --revert`) leaves the block's entry for it as it is. A hosted run never gives it: see the `maven_bazel_*` rows below and docs/ecosystems.md "Bazel workspaces: Maven". | | `maven_bazel_unreadable` | vendored / agent: run-level `warnings[]` (human: `Warning: …`); scan/get `--mode hosted`: `warnings[]` (warning) | Maven in a Bazel workspace (additive): whether a rules_jvm_external hub builds the jar of a Maven patch could not be told, because something could not be read with certainty: a hub without `lock_file` in vendored and agent mode (hosted mode gives `maven_bazel_not_pinned`), a `maven.artifact` or `maven.from_toml` tag that cannot be read, a lock that is missing, empty, not format 2 or 3, larger than 16 MiB or no JSON, a `WORKSPACE` file that calls `maven_install` or cannot be read while `.bazeliskrc` or `.bazelversion` at the Bazel root names no release of Bazel 9 or later, a `maven.install` whose `name` or `lock_file` is not a plain string, a construct of `MODULE.bazel` the reader does not follow, and, in vendored and agent mode, a `maven.override` of the artifact (hosted mode gives `maven_bazel_no_effect`). Hosted mode also gives it for a hub on an isolated `use_extension`, an rc file that cannot be read (also one that an import takes from outside the workspace by a relative path, and one with a line whose reading Bazel's own source calls an oversight: a quote that is never closed, a `\` with nothing after it), a `MODULE.bazel` that is a symbolic link, and a block of `MODULE.bazel` that was changed by hand (also in its first line alone), and then writes no entry for the hub. One warning per purl; the detail names the file, the line and the reason. | | `maven_bazel_not_at_root` | `warnings[]` (warning; human: `Warning: …`) | scan/get `--mode hosted` in a Bazel workspace (additive): a rules_jvm_external hub builds the jar of a Maven patch, and the project is below the root of its Bazel workspace, in whose `MODULE.bazel` the patch of a hub is written. Like the five codes below and `maven_bazel_unreadable` in hosted mode, it means that the run wrote no entry for the jar in the block of the root `MODULE.bazel`, because it could not show from the checkout that the entry works. One warning per purl, in the place of `redirect_maven_no_pom` for that purl; the detail says why and ends with what the block of the root holds for the jar, which is read for a project below the root too (the detail then names the file as "the MODULE.bazel at the root"). Nothing: "socket-patch wrote nothing to MODULE.bazel for it, so the Bazel build keeps the upstream jar". The event is the one of the same run without the hub: `skipped` / `redirect_unconfirmed` in a tree only Bazel builds, `applied` beside a `pom.xml` or Gradle pin that landed. The entry of the patch, from an earlier run: that entry stays, the event is `applied`, and the detail ends "socket-patch left the entry that MODULE.bazel holds for it as it is". The entry of another patch of the same jar: it stays too, the event is the one without the hub, and the detail ends "socket-patch wrote nothing to MODULE.bazel for it: the entry of patch remains there, and where the block is in effect the Bazel build takes that patch's jar". Exit code unchanged. With none of these codes the run writes the block: `redirect.rewrittenFiles` holds `MODULE.bazel`, the event is `applied`, and `rollback` / `remove` take the entry out. See docs/ecosystems.md "Bazel workspaces: Maven". | @@ -3172,6 +3172,29 @@ detail names the entry and the declarations. Like the other unattested references, re-vendoring cannot clear it, so it is not drift. The check never starts an API client or writes lock/recovery files. `--check` conflicts with `--revert`. +One file that the ledger's inventory of a directory artifact does not list is not drift: +the `BUILD.bazel` that rules_rust's cargo-bazel (0.59.0 and newer) writes into a Cargo +copy each time Bazel evaluates a crate hub that takes the crate from that path. It is +left out of the comparison when it is a regular file of that name in the copy's own +directory, starts with cargo-bazel's banner (a line of `#`, `# @generated`, and a line +that starts with `# DO NOT MODIFY: This file is auto-generated by a crate_universe +tool.`), the project is in a Bazel workspace with a hub of its crates (the `MODULE.bazel` +at the Bazel root, or a file it includes, calls `crate.from_cargo` of rules_rust's +crate_universe extension as a statement of its own, and the call names the project's +`Cargo.lock` or root `Cargo.toml` by a label written out as a string), and a line of the +root's `.bazelignore` covers the copy, so that Bazel reads no package there. Only the +banner is read. The same holds wherever the committed +copy is verified in its place: `vex` attests the crate, `repair` leaves the copy alone, +and a second `scan --mode vendored` reports it `already_vendored` without a download. +Every recorded file, a `BUILD.bazel` the crate ships among them, and every other +unrecorded file is compared as before (`vendor_inventory_mismatch`), as is a download +staged for a copy, and outside such a workspace nothing changes. In such a workspace the +commands that can write or remove a vendored crate also keep a line for +`.socket/vendor/cargo` in the root's `.bazelignore` (the warning `bazelignore_not_updated` +when they cannot); with that line commented out, or missing, cargo-bazel's file is drift +as it was. See +[Bazel workspaces: the build file in a vendored crate](../../docs/ecosystems.md#bazel-workspaces-the-build-file-in-a-vendored-crate). + `vendor --check --local-repo ` additionally checks existing suffixed Maven jar/POM copies for conflicting bytes. A suffixed Maven or sbt pom is also read against its upstream parent poms when `--local-repo` or a Maven or Gradle cache diff --git a/crates/socket-patch-cli/src/commands/mod.rs b/crates/socket-patch-cli/src/commands/mod.rs index 07d188caf..07eed466a 100644 --- a/crates/socket-patch-cli/src/commands/mod.rs +++ b/crates/socket-patch-cli/src/commands/mod.rs @@ -54,15 +54,16 @@ pub(crate) fn project_state_in_scope(common: &crate::args::GlobalArgs) -> bool { } /// In a Bazel workspace the root's `.bazelignore` names the directories of -/// the project's patched Go module copies, so that Bazel and gazelle read -/// no package there (`socket_patch_core::bazel::ignore`). A command that can -/// write or remove such a copy calls this once, after the last step that -/// can change a copy or take one back and before it gives up its apply -/// lock. Nothing is done in a dry run or a `--global` run, and a run whose -/// `--ecosystems` leaves `golang` out only takes away the pair of a copy -/// that is gone. So does the prune pass of a scan, which comes here after -/// its scan has (`adds` false) and can only have removed a copy: a file -/// it cannot write is left to the next command's warning. +/// the project's patched Go module copies, and of the vendored crates a +/// crate hub builds, so that Bazel and gazelle read no package there +/// (`socket_patch_core::bazel::ignore`). A command that can write or +/// remove such a copy calls this once, after the last step that can change +/// a copy or take one back and before it gives up its apply lock. Nothing +/// is done in a dry run or a `--global` run, and a run whose `--ecosystems` +/// leaves `golang` or `cargo` out only takes away that ecosystem's pair of +/// a copy that is gone. So does the prune pass of a scan, which comes here +/// after its scan has (`adds` false) and can only have removed a copy: a +/// file it cannot write is left to the next command's warning. /// /// A human run is told here that the file changed, or why it could not be; /// the result is that warning again, for the command's envelope. @@ -70,31 +71,39 @@ pub(crate) async fn sync_bazelignore( common: &crate::args::GlobalArgs, adds: bool, ) -> Vec { - use socket_patch_core::{bazel::ignore, crawlers::Ecosystem::Golang}; + use socket_patch_core::bazel::ignore; + use socket_patch_core::crawlers::Ecosystem::{Cargo, Golang}; let mut warnings = Vec::new(); if common.dry_run || common.is_global() { return warnings; } // The agent copies are those of `--cwd`, the vendored ones those of - // the manifest's project. A root both are under is named once. + // the manifest's project. let mut projects = vec![common.cwd.clone()]; if common.manifest_project_is_foreign() { projects.push(common.project_root()); } let (loud, mut told) = (!common.json && !common.silent, Vec::new()); + let [go, cargo] = [Golang, Cargo].map(|eco| adds && common.ecosystem_selected(eco)); for project in projects { - match ignore::sync(&project, adds && common.ecosystem_selected(Golang)).await { - Ok(Some(file)) if loud && !told.contains(&file) => { - let (verb, what) = match file.exists() { - true => ( - "Updated", - "the lines that keep patched Go module copies under .socket out of \ - Bazel's packages", + match ignore::sync(&project, go, cargo).await { + Ok(Some((file, copies))) if loud => { + // A root that both projects are under is named once for + // each kind of copy whose lines changed there. + let (shown, copies) = (file.display(), copies.join(" and ")); + let line = match file.exists() { + true => format!( + "Updated {shown}: the lines that keep patched {copies} copies under \ + .socket out of Bazel's packages" ), - false => ("Removed", "no patched Go module copy is left under .socket"), + false => { + format!("Removed {shown}: no patched {copies} copy is left under .socket") + } }; - println!("{verb} {}: {what}", file.display()); - told.push(file); + if !told.contains(&line) { + println!("{line}"); + told.push(line); + } } Err(detail) if adds => { if loud { diff --git a/crates/socket-patch-cli/tests/e2e_bazel_build/cargo.rs b/crates/socket-patch-cli/tests/e2e_bazel_build/cargo.rs index 7c8c570e9..f0ae4ea51 100644 --- a/crates/socket-patch-cli/tests/e2e_bazel_build/cargo.rs +++ b/crates/socket-patch-cli/tests/e2e_bazel_build/cargo.rs @@ -29,18 +29,21 @@ //! patched crate file. `scan --mode vendored --yes` unpacks the patched //! crate file to `.socket/vendor/cargo//-/`, records //! it in the ledger `.socket/vendor/state.json`, points a -//! `[patch.crates-io]` entry of `Cargo.toml` at the copy, and gives the +//! `[patch.crates-io]` entry of `Cargo.toml` at the copy, gives the //! lock's entry the version of the copy (`+socket.`) and -//! no source. +//! no source, and writes the line of the vendored crates into +//! `.bazelignore` ([`IGNORED`]): the workspace has a crate hub. //! - (c) The same two Bazel commands, with no step in between, print the //! patched marker. Hosted: the stand-in patch host was asked for its index //! file of the crate and, at least twice, for the crate file, and the //! upstream one was not asked for again. Vendored: the scan had fetched -//! it, and the build asked the stand-in for nothing. Then `vex` is asked +//! it, and the build asked no host for the crate. Then `vex` is asked //! for its document. Hosted: it attests the crate, and the command of the //! mode, run a second time as on every run of a pipeline, reports the -//! patch applied again and changes no file. Vendored: what `vex` does -//! depends on the ruleset (see "Known gaps"). +//! patch applied again and changes no file. Vendored: `vendor --check` +//! passes and `vex` attests the crate, `//...` is the user's own target, +//! and the commands that analyse every target pass (see "The build file +//! in the copy"). //! - (d) The undo (`rollback --yes`, or `vendor --revert`) puts every file //! the command wrote back, byte for byte, and the build prints the //! unpatched marker again. @@ -51,12 +54,14 @@ //! pins, and the build fails on the checksum, whether the index of that //! host names the pinned checksum or the one of the file it serves; the //! upstream crate is not asked for in its place. Vendored: the first -//! build there is the patched one, and the command of the mode runs a -//! second time; the build after it writes nothing, `vendor --check` -//! passes and `vex` attests the crate. Then the library of the copy is -//! changed. Nothing in a build holds the committed copy to a checksum, as -//! nothing does for any crate a Cargo workspace takes from a path: the -//! changed library is what Bazel builds next, and it is +//! build there is the patched one, and the command of the mode, run a +//! second time, reports the copy `already_vendored`, downloads nothing +//! and changes no file; `vendor --check` passes and `vex` attests the +//! crate. Then a build file that cargo-bazel did not write is put into +//! the copy (see "The build file in the copy"), and then the library of +//! the copy is changed. Nothing in a build holds the committed copy to a +//! checksum, as nothing does for any crate a Cargo workspace takes from a +//! path: the changed library is what Bazel builds next, and it is //! `socket-patch vendor --check` that reports the copy //! (`vendor_hash_mismatch`). //! @@ -72,11 +77,11 @@ //! build writes the one of before. //! //! With rules_rust 0.56.0 Bazel's lock has no entry for the hub and a build -//! writes no file: the undo restores the checkout at once, `vendor --check` -//! passes, `vex` attests the crate, and a second scan reports the copy -//! `already_vendored` and leaves it alone. With rules_rust 0.75.0 a build -//! also writes a `BUILD.bazel` into the vendored copy, and those three -//! commands do otherwise right after it (see "Known gaps"). +//! writes no file: the undo restores the checkout at once. With rules_rust +//! 0.75.0 a build also writes a `BUILD.bazel` into the vendored copy (see +//! "The build file in the copy"). With either, `vendor --check` passes +//! right after the build, `vex` attests the crate, and a second scan +//! reports the copy `already_vendored` and leaves it alone. //! //! The two releases also differ in the home of the hub's cargo //! ([`cargo_env`]). rules_rust 0.75.0 gives it a directory inside the @@ -114,33 +119,49 @@ //! (static.rust-lang.org), and the crates it builds its own tools from //! (static.crates.io). //! -//! # Known gaps +//! # The build file in the copy //! -//! Two assertions of the vendored flow hold what the release does with -//! rules_rust 0.75.0, and that is not what a user needs. They are marked -//! `KNOWN GAP` where they stand. Both have one cause: cargo-bazel 0.75.0 -//! writes a `BUILD.bazel` into the committed copy, the scan recorded no -//! such file, and `socket-patch` then takes the copy for a damaged one -//! (`vendor_inventory_mismatch`). Right after a build that is the patched -//! one: +//! cargo-bazel renders a build file for every crate of the hub. Since +//! rules_rust 0.59.0 the one of a crate that the workspace takes from a +//! path goes to `/BUILD.bazel`, into the checkout +//! (`crate_universe/src/rendering.rs`, `render_build_files`), each time +//! Bazel evaluates the hub, which a build does when a file the hub reads +//! has changed since the last one. The hub does not build the crate from +//! there: it copies the directory into a repository of its own and renders +//! the `BUILD.bazel` of that repository again, over the copied one +//! (`crate_universe/private/local_crate_mirror.bzl`). What the flow holds +//! of that: //! -//! - `vendor --check` exits 1. -//! - `vex` leaves the patch out and, with nothing else to attest, exits 1 -//! and writes no document. +//! - Where the ruleset writes the file, it starts with the banner of every +//! file cargo-bazel renders ([`BANNER`]), and `socket-patch` takes a +//! `BUILD.bazel` that starts so, in the copy's own directory of a +//! project whose Bazel workspace has a crate hub, for no change of the +//! copy: the scan recorded no such file, and the commands above pass all +//! the same. +//! - Bazel would read that file as a package of the user's workspace. The +//! crate has a dependency ([`DEP`]), so the file names a repository of +//! the hub, which the main repository does not see, and the commands +//! that analyse every target fail on it: `bazel cquery //...` and +//! `bazel query 'deps(//...)'` ([`analyses`]). The line that the scan +//! wrote into `.bazelignore` keeps the copies out of Bazel's packages: +//! `//...` is the user's own target ([`every_target`]), both commands +//! pass, and the build is the patched one, because the hub reads the +//! copy by its path, not as a package. A user who comments the line out +//! has the package back, and the two commands fail where the ruleset +//! writes the file. +//! - With either ruleset, a build file that someone else put there under +//! the same banner does not reach the crate: the build still prints the +//! patched marker. `vendor --check` passes on it: the banner is all that +//! `socket-patch` reads of the file. Under the line it is no package. +//! With the line commented out it is one, and `bazel build //...` builds +//! its target. //! -//! The flow also holds how a user gets out of it today. A second -//! `scan --mode vendored` downloads the crate again, reports the copy -//! `rebuilt` and removes the file. Nothing the hub reads has changed by -//! then, so the build after that scan does not write the file again, and -//! the check and `vex` pass. `vendor --revert` removes the file with the -//! copy. -//! -//! A fix of the product makes the two commands pass right after the build, -//! as they do with rules_rust 0.56.0. `intact` of the vendored flow is then -//! true with every ruleset, and the branches marked `KNOWN GAP` go, with -//! the second download of the crate. +//! What else is a changed copy, and where the line is written, is held +//! without a Bazel +//! ([`only_the_build_file_of_cargo_bazel_is_no_change_of_a_vendored_copy`], +//! [`the_vendored_crates_get_a_bazelignore_line_where_a_crate_hub_builds_them`]). -use std::path::{Path, PathBuf}; +use std::path::Path; use std::time::Instant; use base64::Engine as _; @@ -150,8 +171,8 @@ use socket_patch_core::hash::git_sha256::compute_git_sha256_from_bytes; use crate::envelope::{event_for, has_event}; use crate::harness::{ - changed, fixture_path, Cell, Fixture, Ran, Release, StandIn, Tree, CONSUMER, NOTHING, - REPO_CACHE, + cell_without_bazel, changed, fixture_path, Cell, Fixture, Ran, Release, StandIn, Tree, + CONSUMER, NOTHING, REPO_CACHE, }; use crate::hosted_memory_common::{Patch, ORG}; @@ -183,24 +204,39 @@ const NAME: &str = "socket-patch-bazel-fixture-with-a-name-that-is-too-long-for- const VERSION: &str = "1.0.0"; const PURL: &str = "pkg:cargo/socket-patch-bazel-fixture-with-a-name-that-is-too-long-for-crates-io@1.0.0"; -/// Its `Cargo.toml`. The other file of a crate file is `src/lib.rs`. +/// The crate that one depends on, at the same version, which the same +/// registry holds under a name crates.io cannot hold either. Its crate +/// file is the same in every run. With a dependency, the build file that +/// cargo-bazel renders for [`NAME`] names a repository of the hub. +const DEP: &str = + "socket-patch-bazel-fixture-dependency-with-a-name-that-is-too-long-for-crates-io"; +/// The `Cargo.toml` of each. The other file of a crate file is `src/lib.rs`. const CRATE_MANIFEST: &str = "[package]\n\ name = \"socket-patch-bazel-fixture-with-a-name-that-is-too-long-for-crates-io\"\n\ version = \"1.0.0\"\n\ + edition = \"2021\"\n\ + \n\ + [dependencies]\n\ + socket-patch-bazel-fixture-dependency-with-a-name-that-is-too-long-for-crates-io = \"1.0.0\"\n"; +const DEP_MANIFEST: &str = "[package]\n\ + name = \"socket-patch-bazel-fixture-dependency-with-a-name-that-is-too-long-for-crates-io\"\n\ + version = \"1.0.0\"\n\ edition = \"2021\"\n"; /// The files of the checkout that cargo and the hub read, and Bazel's lock. const CONFIG: &str = ".cargo/config.toml"; const LOCK: &str = "Cargo.lock"; const MANIFEST: &str = "Cargo.toml"; const BAZEL_LOCK: &str = "MODULE.bazel.lock"; -/// How a `Cargo.lock` names crates.io, with or without a mirror. +/// How a `Cargo.lock` names crates.io, with or without a mirror, and how +/// the index of another registry names it. const CRATES_IO: &str = "registry+https://github.com/rust-lang/crates.io-index"; +const CRATES_IO_INDEX: &str = "https://github.com/rust-lang/crates.io-index"; /// What the fixture's lock holds as the checksum of the crate: the sum of /// no file. const PLACEHOLDER: &str = "0000000000000000000000000000000000000000000000000000000000000000"; /// The route prefixes under which the stand-in plays the two hosts of /// crates.io, and the routes of the crate there: its file of the sparse -/// index, and its crate file. +/// index, and its crate file. The dependency has the same two. const INDEX: &str = "index.crates.io"; const FILES: &str = "static.crates.io"; const UPSTREAM_ENTRY: &str = @@ -209,6 +245,20 @@ const UPSTREAM_CRATE: &str = "/static.crates.io/crates/socket-patch-bazel-fixtur /// The access-token level of a hosted URL, which `socket-patch` writes as /// the API hands it out. const TOKEN: &str = "33333333-3333-4333-8333-333333333333"; +/// How every file cargo-bazel renders begins: its banner, up to the name +/// of the tool (`crate_universe/src/rendering/templates/partials/header.j2`, +/// the same since 2022). The command that renders the file again follows. +const BANNER: &str = + "###############################################################################\n\ + # @generated\n\ + # DO NOT MODIFY: This file is auto-generated by a crate_universe tool."; +/// The two lines `socket-patch` keeps in the `.bazelignore` of a workspace +/// with a crate hub while the project has vendored crates, and the same +/// with the path line commented out: a user's "no" to them. +const IGNORED: &str = "# socket-patch: patched crate copies, not Bazel packages (to opt out, \ + comment out the next line)\n.socket/vendor/cargo\n"; +const DECLINED: &str = "# socket-patch: patched crate copies, not Bazel packages (to opt out, \ + comment out the next line)\n# .socket/vendor/cargo\n"; /// The cargo home of every `socket-patch` child: no directory, so a scan /// finds the crate in the lock alone, as in a checkout only Bazel built. const NO_CARGO_HOME: &str = "/nonexistent/cargo-home"; @@ -219,17 +269,18 @@ fn sha256_hex(bytes: &[u8]) -> String { hex::encode(Sha256::digest(bytes)) } -/// The crate with `lib_rs` as its library, as a registry serves it. Every -/// sum a test holds a build to is computed from the bytes it serves, so -/// nothing depends on the exact bytes the archive crates emit. -fn crate_file(lib_rs: &str) -> Vec { +/// The crate `name` with `manifest` and with `lib_rs` as its library, as a +/// registry serves it. Every sum a test holds a build to is computed from +/// the bytes it serves, so nothing depends on the exact bytes the archive +/// crates emit. +fn crate_file(name: &str, manifest: &str, lib_rs: &str) -> Vec { let gzip = flate2::write::GzEncoder::new(Vec::new(), flate2::Compression::default()); let mut archive = tar::Builder::new(gzip); - for (name, text) in [("Cargo.toml", CRATE_MANIFEST), ("src/lib.rs", lib_rs)] { + for (file, text) in [("Cargo.toml", manifest), ("src/lib.rs", lib_rs)] { let mut header = tar::Header::new_ustar(); header.set_size(text.len() as u64); header.set_mode(0o644); - let path = format!("{NAME}-{VERSION}/{name}"); + let path = format!("{name}-{VERSION}/{file}"); archive .append_data(&mut header, path, text.as_bytes()) .expect("tar"); @@ -251,7 +302,7 @@ impl Package { fn new(word: &str, id: &str) -> Package { let lib_rs = format!("pub const STATUS: &str = \"{word} {id}\";\n"); Package { - crate_file: crate_file(&lib_rs), + crate_file: crate_file(NAME, CRATE_MANIFEST, &lib_rs), marker: format!("OUT: {word} {id}"), lib_rs, } @@ -268,16 +319,33 @@ impl Package { } /// The crate's file of a sparse index that lists this crate file: one - /// line per version. - fn index_line(&self) -> String { - let version = json!({ - "name": NAME, "vers": VERSION, "deps": [], "features": {}, "yanked": false, - "cksum": self.checksum(), + /// line per version. It names the dependency as being in `registry`, + /// an index: crates.io's own on the patch host, as cargo wants it of a + /// registry that is not the dependency's, and none on crates.io itself. + fn index_line(&self, registry: Option<&str>) -> String { + let dependency = json!({ + "name": DEP, "req": format!("^{VERSION}"), "features": [], "optional": false, + "default_features": true, "target": null, "kind": "normal", "registry": registry, }); - format!("{version}\n") + index_line(NAME, &self.checksum(), json!([dependency])) } } +/// One line of a crate's file of a sparse index. +fn index_line(name: &str, checksum: &str, deps: serde_json::Value) -> String { + let version = json!({ + "name": name, "vers": VERSION, "deps": deps, "features": {}, "yanked": false, + "cksum": checksum, + }); + format!("{version}\n") +} + +/// The crate file of [`DEP`]. +fn dependency_file() -> Vec { + let lib_rs = "pub const DEPENDENCY: &str = \"of the fixture\";\n"; + crate_file(DEP, DEP_MANIFEST, lib_rs) +} + /// One patch of the crate, under an id that is new for every run. struct Patched { uuid: String, @@ -295,7 +363,7 @@ impl Patched { let index = format!("/patch-registry/cargo/{TOKEN}/{uuid}/index"); Patched { package: Package::new("PATCHED", &uuid), - entry: format!("{index}/{}", index_path()), + entry: format!("{index}/{}", index_path(NAME)), route: format!("/patch/cargo/{NAME}/{VERSION}/{TOKEN}/{uuid}/{NAME}-{VERSION}.crate"), index, uuid, @@ -313,10 +381,15 @@ impl Patched { } } -/// The path of the crate's file in a sparse index: cargo's layout for a -/// name of four characters or more. -fn index_path() -> String { - format!("{}/{}/{NAME}", &NAME[..2], &NAME[2..4]) +/// The path of the file of the crate `name` in a sparse index: cargo's +/// layout for a name of four characters or more. +fn index_path(name: &str) -> String { + format!("{}/{}/{name}", &name[..2], &name[2..4]) +} + +/// What the lock's entry of the crate says of its dependency. +fn depends() -> String { + format!("dependencies = [\n \"{DEP}\",\n]\n") } /// The entry of a lock that resolves the crate from `source` and pins its @@ -324,20 +397,40 @@ fn index_path() -> String { fn lock_entry(source: &str, checksum: &str) -> String { format!( "[[package]]\nname = \"{NAME}\"\nversion = \"{VERSION}\"\n\ - source = \"{source}\"\nchecksum = \"{checksum}\"\n" + source = \"{source}\"\nchecksum = \"{checksum}\"\n{}", + depends() + ) +} + +/// The entry of a lock that pins the crate file of the dependency to +/// `checksum`. +fn dependency_entry(checksum: &str) -> String { + format!( + "[[package]]\nname = \"{DEP}\"\nversion = \"{VERSION}\"\n\ + source = \"{CRATES_IO}\"\nchecksum = \"{checksum}\"\n" ) } -/// The fixture's lock with the checksum of `upstream` in place of the -/// placeholder: the lock a test writes into its workspace copy before the -/// first build. +/// The fixture's lock with the checksum of `upstream`, and the one of the +/// dependency, in place of the placeholders: the lock a test writes into +/// its workspace copy before the first build. fn lock_pinning(upstream: &Package) -> String { let path = fixture_path(FIXTURE.dir).join("common").join(LOCK); - let lock = std::fs::read_to_string(&path).unwrap_or_else(|e| panic!("{}: {e}", path.display())); - let unpinned = lock_entry(CRATES_IO, PLACEHOLDER); - let entries = lock.matches(&unpinned).count(); - assert_eq!(entries, 1, "{}: not one\n{unpinned}", path.display()); - lock.replace(&unpinned, &lock_entry(CRATES_IO, &upstream.checksum())) + let mut lock = + std::fs::read_to_string(&path).unwrap_or_else(|e| panic!("{}: {e}", path.display())); + let dependency = dependency_entry(&sha256_hex(&dependency_file())); + for (unpinned, pinned) in [ + ( + lock_entry(CRATES_IO, PLACEHOLDER), + lock_entry(CRATES_IO, &upstream.checksum()), + ), + (dependency_entry(PLACEHOLDER), dependency), + ] { + let entries = lock.matches(&unpinned).count(); + assert_eq!(entries, 1, "{}: not one\n{unpinned}", path.display()); + lock = lock.replace(&unpinned, &pinned); + } + lock } /// The `.cargo/config.toml` of a checkout whose crates come from a mirror @@ -359,15 +452,22 @@ fn mirror_config(origin: &str) -> String { /// A stand-in that holds `upstream` as crates.io does: the `config.json` of /// the sparse index, which names the download host the way the real one /// does (a base URL, to which cargo adds `///download`), the -/// crate's file of the index, and the crate file. +/// crate's file of the index, and the crate file. And the same two of the +/// dependency. fn registry(upstream: &Package) -> StandIn { let standin = StandIn::start(); let origin = standin.origin(); let config = json!({ "dl": format!("{origin}/{FILES}/crates"), "api": origin }); let config = serde_json::to_vec(&config).expect("a JSON document"); standin.serve(5, &format!("/{INDEX}/config.json"), &config); - standin.serve(5, UPSTREAM_ENTRY, upstream.index_line().as_bytes()); + standin.serve(5, UPSTREAM_ENTRY, upstream.index_line(None).as_bytes()); standin.serve(5, UPSTREAM_CRATE, &upstream.crate_file); + let dependency = dependency_file(); + let listed = index_line(DEP, &sha256_hex(&dependency), json!([])); + let entry = format!("/{INDEX}/{}", index_path(DEP)); + standin.serve(5, &entry, listed.as_bytes()); + let file = format!("/{FILES}/crates/{DEP}/{VERSION}/download"); + standin.serve(5, &file, &dependency); standin } @@ -431,7 +531,8 @@ fn grant(standin: &StandIn, upstream: &Package, patch: &Patched, served: &[u8]) ); let config = serde_json::to_vec(&json!({ "dl": dl, "api": origin })).expect("a JSON document"); standin.serve(5, &format!("{}/config.json", patch.index), &config); - standin.serve(5, &patch.entry, patch.package.index_line().as_bytes()); + let listed = patch.package.index_line(Some(CRATES_IO_INDEX)); + standin.serve(5, &patch.entry, listed.as_bytes()); standin.serve(5, &patch.route, served); } @@ -474,14 +575,31 @@ fn open(test: &str) -> Option<(tempfile::TempDir, Cell, StandIn, Package)> { /// the patch server (a hosted pin is recognised by that origin) and as the /// index `rollback` reads the upstream checksum from. fn socket_patch(cell: &Cell, standin: &StandIn, args: &[&str]) -> Ran { + let (origin, index) = hosts(standin); + cell.socket_patch(&origin, args, &env(&origin, &index)) +} + +/// [`socket_patch`] as a person runs it: what it printed on stdout and on +/// stderr. +fn person(cell: &Cell, standin: &StandIn, args: &[&str]) -> [String; 2] { + let (origin, index) = hosts(standin); + cell.socket_patch_person(&origin, args, &env(&origin, &index)) +} + +/// The origin of `standin`, and the index of crates.io there. +fn hosts(standin: &StandIn) -> (String, String) { let origin = standin.origin(); let index = format!("{origin}/{INDEX}"); - let env = [ - ("SOCKET_PATCH_SERVER_URL", origin.as_str()), - ("SOCKET_CRATES_INDEX", index.as_str()), + (origin, index) +} + +/// What a `socket-patch` child of a flow is told of its hosts. +fn env<'a>(origin: &'a str, index: &'a str) -> [(&'a str, &'a str); 3] { + [ + ("SOCKET_PATCH_SERVER_URL", origin), + ("SOCKET_CRATES_INDEX", index), ("CARGO_HOME", NO_CARGO_HOME), - ]; - cell.socket_patch(&origin, args, &env) + ] } /// A text file of a tree. @@ -642,20 +760,44 @@ fn refuses(twin: &Cell, refusal: &str) { ); } -/// `vex` in the checkout of `cell`, which Bazel has built in and which -/// holds the files of `built`. Its document goes beside the checkout, and +/// What Bazel takes the checkout of `cell` for: `bazel build //...` passes, +/// and these are the targets `bazel query //...` lists, sorted. +fn every_target(cell: &Cell) -> Vec { + let build = cell.bazel("build", &["//..."]); + assert!(build.ok(), "{}: {}", cell.test, build.dump()); + let query = cell.bazel("query", &["//..."]); + assert!(query.ok(), "{}: {}", cell.test, query.dump()); + let mut targets: Vec = query.stdout().lines().map(str::to_string).collect(); + targets.sort(); + targets +} + +/// Whether the commands that analyse every target of the checkout of +/// `cell` pass: `bazel cquery //...` and `bazel query 'deps(//...)'`, which +/// agree. One that fails has met a repository of the hub, which the main +/// repository does not see. +fn analyses(cell: &Cell) -> bool { + let runs = [("cquery", "//..."), ("query", "deps(//...)")]; + let runs = runs.map(|(command, targets)| cell.bazel(command, &[targets])); + for run in &runs { + let unseen = run.stderr().contains("No repository visible as '@crates__"); + assert_ne!(run.ok(), unseen, "{}: {}", cell.test, run.dump()); + } + assert_eq!(runs[0].ok(), runs[1].ok(), "{}", cell.test); + runs[0].ok() +} + +/// `vex` in the checkout of `cell`, which holds the files of `built`. Its +/// document goes beside the checkout ([`Cell::document`]), and /// no file of the checkout changes. With `attested` it exits 0 and the -/// document names the crate as patched by `patch`. Without, which is a -/// known gap (module docs), it exits 1 and writes no document: the patch is -/// left out over the inventory of the vendored copy +/// document names the crate as patched by `patch`. Without, for a vendored +/// copy that is a changed one, it exits 1 and writes no document: the +/// patch is left out over the inventory of the copy /// (`vendor_inventory_mismatch`), and nothing else is there to attest. fn vex(cell: &Cell, standin: &StandIn, patch: &Patched, built: &Tree, attested: bool) { let test = &cell.test; - let info = cell.bazel("info", &["workspace"]); - assert!(info.ok(), "{test}: {}", info.dump()); - let checkout = PathBuf::from(info.stdout().trim()); - let document = checkout.parent().expect("a scratch root").join("vex.json"); - assert!(!document.exists(), "{test}: {}", document.display()); + let document = cell.document("vex"); + let _ = std::fs::remove_file(&document); let output = document.to_string_lossy(); let vex = socket_patch(cell, standin, &["vex", "--output", &output]); let said = &vex.dump; @@ -668,10 +810,6 @@ fn vex(cell: &Cell, standin: &StandIn, patch: &Patched, built: &Tree, attested: let names = written.contains(&format!("\"{PURL}\"")) && written.contains(&by); assert!(names, "{test}: {said}\n{written}"); } else { - // KNOWN GAP: what the release does, not what a user needs. Once - // `vex` attests a vendored crate whose copy holds the `BUILD.bazel` - // a build wrote, no flow passes `attested = false`, and this branch - // goes with the parameter. assert_eq!(vex.code, 1, "{test}: {said}"); let left_out = has_event(&vex.envelope, "skipped", "vendor_inventory_mismatch", PURL); assert!(left_out, "{test}: {said}"); @@ -804,7 +942,8 @@ fn hosted_from_cargo_with_config() { // first build of a machine does. rules_rust 0.56.0 keeps the home it is // given from build to build, and cargo does not ask again for the index // file of a crate the lock pins while its home holds one. - other.serve(1, &second.entry, tampered.index_line().as_bytes()); + let listed = tampered.index_line(Some(CRATES_IO_INDEX)); + other.serve(1, &second.entry, listed.as_bytes()); let third_home = cargo_home(); twin.repo_env = cargo_env(third_home.path()); let asked = other.asked(&second.entry); @@ -863,10 +1002,12 @@ fn vendored_from_cargo_with_config() { assert_eq!(scan.envelope["summary"]["downloaded"], 1, "{test}: {said}"); assert_eq!(scan.envelope["summary"]["applied"], 1, "{test}: {said}"); // The unpacked crate and its ledger under `.socket/vendor`, the two - // files that wire it in, and no other. + // files that wire it in, the line that keeps Bazel out of the copies, + // and no other. let copy = patch.vendored(); let (copied_manifest, library) = (format!("{copy}/Cargo.toml"), format!("{copy}/src/lib.rs")); let written = [ + ".bazelignore".to_string(), copied_manifest.clone(), library.clone(), format!( @@ -879,6 +1020,7 @@ fn vendored_from_cargo_with_config() { ]; let vendored = cell.tree(); assert_eq!(changed(&before, &vendored), written, "{test}: {said}"); + assert_eq!(text(&vendored, ".bazelignore"), IGNORED, "{test}: {said}"); // The committed library is the one of the crate file the patch host // served. The copy says it is the patched build of the version, a // `[patch.crates-io]` entry sends the crate there, and the lock's entry @@ -901,7 +1043,10 @@ fn vendored_from_cargo_with_config() { "{test}: {MANIFEST}:\n{manifest}" ); let lock = text(&vendored, LOCK); - let entry = format!("[[package]]\nname = \"{NAME}\"\nversion = \"{tagged}\"\n"); + let entry = format!( + "[[package]]\nname = \"{NAME}\"\nversion = \"{tagged}\"\n{}", + depends() + ); assert!( lock.ends_with(&entry) && !lock.contains(&upstream.checksum()), "{test}: {LOCK}:\n{lock}" @@ -911,15 +1056,22 @@ fn vendored_from_cargo_with_config() { } // (c) The same build, no step in between and no repin: patched, from - // the committed copy. Bazel asks the stand-in for nothing. - let requests = standin.requests(); + // the committed copy. Bazel asks no host for the crate: what the hub's + // cargo fetches again is the dependency. + let routes = [UPSTREAM_ENTRY, UPSTREAM_CRATE, patch.route.as_str()]; + let asked = || routes.map(|route| standin.asked(route)); + let requests = asked(); cell.builds(&patch.package.marker); - assert_eq!(standin.requests(), requests, "{test}: Bazel asked a host"); + assert_eq!( + asked(), + requests, + "{test}: Bazel asked a host for the crate" + ); // Where the ruleset watches, the build rewrote Bazel's lock, and - // cargo-bazel wrote a `BUILD.bazel` into the committed copy. That file - // is no part of what the scan recorded: the copy is no longer `intact`, - // and `vendor --check` fails right after a build that is the patched - // one. Elsewhere the build wrote no file and the check passes. + // cargo-bazel wrote a `BUILD.bazel` into the committed copy, under its + // banner. That file is no part of what the scan recorded, and no change + // of the copy: `vendor --check` passes right after the build, as it + // does where the build wrote no file. let built = cell.tree(); let build_file = format!("{copy}/BUILD.bazel"); let by_bazel: &[&str] = if watches(&cell) { @@ -928,34 +1080,44 @@ fn vendored_from_cargo_with_config() { &[] }; assert_eq!(changed(&vendored, &built), by_bazel, "{test}"); - let intact = !watches(&cell); - let check = socket_patch(&cell, &standin, &["vendor", "--check"]); - let said = &check.dump; - if intact { - assert_eq!(check.code, 0, "{test}: {said}"); - } else { - // KNOWN GAP: what the release does, not what a user needs. Once - // `vendor --check` passes on a copy that holds the `BUILD.bazel` a - // build wrote, `intact` is true with every ruleset, and this branch - // goes. - assert_eq!(check.code, 1, "{test}: {said}"); - let event = event_for(&check.envelope, PURL); - assert_eq!(event["errorCode"], "vendor_check_failed", "{test}: {said}"); - let reason = event["reason"].as_str().unwrap_or_default(); - assert!( - reason.contains("vendor_inventory_mismatch"), - "{test}: {said}" - ); + if watches(&cell) { + let rendered = text(&built, &build_file); + assert!(rendered.starts_with(BANNER), "{test}: {rendered}"); } + let check = socket_patch(&cell, &standin, &["vendor", "--check"]); + assert_eq!(check.code, 0, "{test}: {}", check.dump); // From here on the checkout stands: the next build changes no file, and // Bazel's lock holds when Bazel may not update it. cell.builds_patched(&built, &patch.package.marker); - // `vex` attests the crate of an intact copy, and leaves the patch of - // the other out (the KNOWN GAP of `vex`). - vex(&cell, &standin, &patch, &built, intact); + vex(&cell, &standin, &patch, &built, true); + // What Bazel takes the copy for. Under the line the scan wrote it is no + // package: `//...` is the consumer alone, and the commands that analyse + // every target pass. A user who comments the line out has the package + // back, where the ruleset writes the build file: `//...` names targets + // there, those commands fail on the repository of the dependency, and + // the file Bazel now reads is a change of the copy again. + assert_eq!(every_target(&cell), [CONSUMER], "{test}"); + assert!(analyses(&cell), "{test}: under the line of the scan"); + cell.write(".bazelignore", DECLINED); + let targets = every_target(&cell); + let package = format!("//{copy}:"); + let (in_copy, own): (Vec<&str>, Vec<&str>) = + (targets.iter().map(String::as_str)).partition(|target| target.starts_with(&package)); + let in_copy = in_copy.len(); + assert_eq!(own, [CONSUMER], "{test}: {targets:?}"); + assert_eq!(in_copy > 0, watches(&cell), "{test}: {targets:?}"); + assert_eq!(analyses(&cell), !watches(&cell), "{test}: without the line"); + let check = socket_patch(&cell, &standin, &["vendor", "--check"]); + let read = i32::from(watches(&cell)); + assert_eq!(check.code, read, "{test}: {}", check.dump); + let declined = ["failed there", "passed"][usize::from(!watches(&cell))]; + cell.write(".bazelignore", IGNORED); + cell.builds(&patch.package.marker); + assert_eq!(changed(&built, &cell.tree()), NOTHING, "{test}"); // (d) The undo. The copy goes, with the `BUILD.bazel` a build wrote - // into it, and the two files are the ones of before at once. + // into it and with the line in `.bazelignore`, and the two files are + // the ones of before at once. let undo = socket_patch(&cell, &standin, &["vendor", "--revert"]); undone(&cell, &undo, "removed", &before, &upstream.marker); @@ -968,11 +1130,9 @@ fn vendored_from_cargo_with_config() { twin.builds(&second.package.marker); let second_built = twin.tree(); // The same command a second time, as every run of a pipeline has it, - // ends with the copy `already_vendored`. An intact one it leaves alone. - // The other it first downloads again (`rebuilt`), which removes the - // `BUILD.bazel`. No file the hub reads has changed, so the next build, - // still the patched one, writes nothing: from here on `vendor --check` - // passes and `vex` attests the crate with either ruleset. + // finds the copy `already_vendored` and leaves it alone, with the + // `BUILD.bazel` the build wrote into it: nothing is downloaded and no + // file changes. `vendor --check` passes and `vex` attests the crate. let downloads = other.asked(&second.route); let again = socket_patch(&twin, &other, &["scan", "--mode", "vendored", "--yes"]); let said = &again.dump; @@ -981,16 +1141,52 @@ fn vendored_from_cargo_with_config() { assert!(in_sync, "{test}: {said}"); let rebuilt = again.envelope["summary"]["rebuilt"] == 1; let downloaded = other.asked(&second.route) > downloads; - assert_eq!((rebuilt, downloaded), (!intact, !intact), "{test}: {said}"); + assert_eq!((rebuilt, downloaded), (false, false), "{test}: {said}"); let second_copy = second.vendored(); let second_build_file = format!("{second_copy}/BUILD.bazel"); - let removed: &[&str] = if intact { &[] } else { &[&second_build_file] }; - let repaired = twin.tree(); - assert_eq!(changed(&second_built, &repaired), removed, "{test}: {said}"); - twin.builds_patched(&repaired, &second.package.marker); + assert_eq!( + changed(&second_built, &twin.tree()), + NOTHING, + "{test}: {said}" + ); + twin.builds_patched(&second_built, &second.package.marker); + let check = socket_patch(&twin, &other, &["vendor", "--check"]); + assert_eq!(check.code, 0, "{test}: {}", check.dump); + vex(&twin, &other, &second, &second_built, true); + // A build file that cargo-bazel did not write, under its banner. It + // does not reach the crate, which Bazel builds as before, and under + // the line of the scan it is no package. The banner is all that + // `socket-patch` reads of such a file, so `vendor --check` passes on + // this one too. With the line commented out it is a package of the + // workspace, whose target `//...` builds, and `vendor --check` takes + // the copy for a changed one. + let planted = format!( + "{BANNER}\n\ngenrule(\n name = \"planted\",\n outs = [\"planted.txt\"],\n \ + cmd = \"touch $@\",\n)\n" + ); + twin.write(&second_build_file, &planted); + twin.builds(&second.package.marker); + assert_eq!(every_target(&twin), [CONSUMER], "{test}"); let check = socket_patch(&twin, &other, &["vendor", "--check"]); assert_eq!(check.code, 0, "{test}: {}", check.dump); - vex(&twin, &other, &second, &repaired, true); + twin.write(".bazelignore", DECLINED); + let planted_target = format!("//{second_copy}:planted"); + assert_eq!( + every_target(&twin), + [planted_target.as_str(), CONSUMER], + "{test}" + ); + let check = socket_patch(&twin, &other, &["vendor", "--check"]); + let said = &check.dump; + assert_eq!(check.code, 1, "{test}: {said}"); + let reason = event_for(&check.envelope, PURL)["reason"] + .as_str() + .unwrap_or_default(); + assert!( + reason.contains("vendor_inventory_mismatch"), + "{test}: {said}" + ); + twin.write(".bazelignore", IGNORED); // Then another library in the committed copy. Nothing in a build holds // the copy to a checksum: Bazel builds what is there. `vendor --check` // is what finds it. @@ -1007,42 +1203,227 @@ fn vendored_from_cargo_with_config() { println!( "{test}: Bazel {release} built {:?} from {copy} after `scan --mode vendored`, with no \ - repin and no request to the stand-in, and wrote {by_bazel:?} itself; right after \ - that build {}; `vendor --revert` restored the checkout, and the next build printed \ - {:?} and left all {} files as before; on a second machine, after its first patched \ - build, a second scan {}, the next build wrote nothing, `vendor --check` passed and \ - `vex` attested the crate; a library then changed in the committed copy was built \ - ({:?}), and `vendor --check` reported vendor_hash_mismatch; {:?}", + repin and no request for the crate, and wrote {by_bazel:?} itself; right after \ + that build `vendor --check` passed and `vex` attested the crate; under the \ + `.bazelignore` line of the scan `//...` held the consumer alone and `cquery //...` \ + and `query 'deps(//...)'` passed, and with the line commented out `//...` held \ + {in_copy} target(s) in the copy, the two commands {declined} and `vendor --check` \ + exited {read}; `vendor --revert` \ + restored the checkout, and the next build printed {:?} and left all {} files as \ + before; on a second machine, after its first patched build, a second scan reported \ + the copy already_vendored, downloaded nothing and changed no file, `vendor --check` \ + passed and `vex` attested the crate; a BUILD.bazel of someone else's under \ + cargo-bazel's banner left the build patched and `vendor --check` passed, under the \ + line it was no package, and with the line commented out `//...` built its target and \ + `vendor --check` reported vendor_inventory_mismatch; a \ + library then changed in the committed copy was built ({:?}), and `vendor --check` \ + reported vendor_hash_mismatch; {:?}", patch.package.marker, - if intact { - "`vendor --check` passed and `vex` attested the crate" - } else { - "(KNOWN GAPS) `vendor --check` failed on the BUILD.bazel \ - (vendor_inventory_mismatch), and `vex` left the patch out under the same code and \ - wrote no document" - }, upstream.marker, before.len(), - if intact { - "reported the copy already_vendored and changed nothing" - } else { - "downloaded the crate again (rebuilt) and removed the BUILD.bazel" - }, tampered.marker, started.elapsed() ); } +/// The module file of the fixture with rules_rust 0.75.0: one that makes a +/// crate hub. +fn hub() -> String { + let module = fixture_path(FIXTURE.dir).join("e9/MODULE.bazel"); + std::fs::read_to_string(module).expect("a fixture file") +} + +/// Without a Bazel: the fixture's Cargo project in a checkout of its own, +/// with the file `marks` at its root, which holds `module` and makes the +/// checkout a Bazel workspace when it is one of Bazel's names; the +/// stand-in that holds the crate and one patch of it; and that patch, +/// which `scan --mode vendored --yes` has applied there. +fn vendored_checkout(marks: &str, module: &str) -> (Cell, StandIn, Patched) { + let cell = cell_without_bazel("9.3.2"); + let upstream = Package::upstream(); + let (standin, patch) = (registry(&upstream), Patched::new()); + grant(&standin, &upstream, &patch, &patch.package.crate_file); + let manifest = fixture_path(FIXTURE.dir).join("common").join(MANIFEST); + cell.write(MANIFEST, std::fs::read(manifest).expect("a fixture file")); + cell.write(LOCK, lock_pinning(&upstream)); + cell.write(CONFIG, mirror_config(&standin.origin())); + cell.write(".git/HEAD", "ref: refs/heads/main\n"); + cell.write(marks, module); + let scan = socket_patch(&cell, &standin, &["scan", "--mode", "vendored", "--yes"]); + assert_eq!(scan.code, 0, "{}", scan.dump); + assert_eq!(scan.envelope["summary"]["applied"], 1, "{}", scan.dump); + (cell, standin, patch) +} + +/// What `socket-patch` does in `cell` with a vendored copy that holds +/// `extra`, a file the scan did not record. A copy it takes for `changed`: +/// `vendor --check` exits 1 over the inventory, `vex` leaves the patch out, +/// and a second scan downloads the crate again and removes the file. The +/// other: the check passes, `vex` attests the crate of `patch`, and a +/// second scan downloads nothing and leaves every file as it is. `repair` +/// after that scan does the same with either. +fn takes_for(cell: &Cell, standin: &StandIn, patch: &Patched, extra: &str, is_changed: bool) { + let left = cell.tree(); + let check = socket_patch(cell, standin, &["vendor", "--check"]); + assert_eq!(check.code, i32::from(is_changed), "{extra}: {}", check.dump); + let reason = event_for(&check.envelope, PURL)["reason"] + .as_str() + .unwrap_or_default(); + let mismatch = reason.contains("vendor_inventory_mismatch"); + assert_eq!(mismatch, is_changed, "{extra}: {}", check.dump); + vex(cell, standin, patch, &left, !is_changed); + let downloads = standin.asked(&patch.route); + let again = socket_patch(cell, standin, &["scan", "--mode", "vendored", "--yes"]); + let said = &again.dump; + assert_eq!(again.code, 0, "{extra}: {said}"); + let in_sync = has_event(&again.envelope, "skipped", "already_vendored", PURL); + assert!(in_sync, "{extra}: {said}"); + let rebuilt = again.envelope["summary"]["rebuilt"] == 1; + let downloaded = standin.asked(&patch.route) - downloads; + let count = usize::from(is_changed); + assert_eq!( + (rebuilt, downloaded), + (is_changed, count), + "{extra}: {said}" + ); + let removed: &[&str] = if is_changed { &[extra] } else { &[] }; + let scanned = cell.tree(); + assert_eq!(changed(&left, &scanned), removed, "{extra}: {said}"); + // `repair` has nothing to restore by now, the file of the other copy + // included. + let repair = socket_patch(cell, standin, &["repair"]); + assert_eq!(repair.code, 0, "{extra}: {}", repair.dump); + let restored = standin.asked(&patch.route) - downloads - downloaded; + assert_eq!(restored, 0, "{extra}: {}", repair.dump); + assert_eq!(changed(&scanned, &cell.tree()), NOTHING, "{extra}"); +} + +/// The module file of [`hub`] with the Cargo files of another directory of +/// the checkout: a hub of another Cargo workspace. +fn hub_elsewhere() -> String { + let (hub, ours) = (hub(), "\"//:Cargo"); + assert_eq!(hub.matches(ours).count(), 2, "{hub}"); + hub.replace(ours, "\"//other:Cargo") +} + +/// Without a Bazel: a build file in a vendored copy that the scan did not +/// record. The first is where cargo-bazel writes one and starts as its +/// files do, in a Bazel workspace with a hub of the project's crates, +/// under the `.bazelignore` line of the scan: that one is no change of the +/// copy. The others differ from it in one thing each, and are the changed +/// copy they were: a checkout that is no Bazel workspace (a directory +/// named like a workspace file marks none), a workspace whose module file +/// makes no crate hub, or the hub of another Cargo workspace, a path line +/// that the user commented out, no banner, Bazel's other name for a build +/// file, and a directory of the crate ([`takes_for`]). +#[test] +fn only_the_build_file_of_cargo_bazel_is_no_change_of_a_vendored_copy() { + let rendered = format!("{BANNER} To \n# regenerate this file, run the following:\n"); + let (hub, elsewhere, bazel) = (hub(), hub_elsewhere(), "MODULE.bazel"); + let no_hub = "module(name = \"socket_fixture\", version = \"0.0.0\")\n"; + let (build, mine) = ("BUILD.bazel", "genrule(name = \"x\")\n"); + for (marks, module, lines, file, text, is_changed) in [ + (bazel, hub.as_str(), None, build, rendered.as_str(), false), + ("WORKSPACE/README", &hub, None, build, &rendered, true), + (bazel, no_hub, None, build, &rendered, true), + (bazel, &elsewhere, None, build, &rendered, true), + (bazel, &hub, Some(DECLINED), build, &rendered, true), + (bazel, &hub, None, build, mine, true), + (bazel, &hub, None, "BUILD", &rendered, true), + (bazel, &hub, None, "src/BUILD.bazel", &rendered, true), + ] { + let (cell, standin, patch) = vendored_checkout(marks, module); + if let Some(lines) = lines { + cell.write(".bazelignore", lines); + } + let extra = format!("{}/{file}", patch.vendored()); + cell.write(&extra, text); + takes_for(&cell, &standin, &patch, &extra, is_changed); + } +} + +/// Without a Bazel: the `.bazelignore` line of the vendored crates. Where +/// the module files of the workspace make a crate hub, `scan --mode +/// vendored` writes the pair and `vendor --revert` takes it away with the +/// copies, and a person is told of each. A run that leaves Cargo out adds +/// no pair, and a path line that the user commented out stays as it is. +/// In a workspace without such a hub, in one whose hub reads another Cargo +/// workspace, and in no workspace, the scan writes no such file. +#[test] +fn the_vendored_crates_get_a_bazelignore_line_where_a_crate_hub_builds_them() { + let no_hub = "module(name = \"socket_fixture\", version = \"0.0.0\")\n"; + let elsewhere = hub_elsewhere(); + let none = [ + ("MODULE.bazel", no_hub), + ("MODULE.bazel", elsewhere.as_str()), + ("WORKSPACE/README", ""), + ]; + for (marks, module) in none { + let (cell, _standin, _patch) = vendored_checkout(marks, module); + assert!(!cell.tree().contains_key(".bazelignore"), "{marks}"); + } + + let (cell, standin, _patch) = vendored_checkout("MODULE.bazel", &hub()); + let file = || cell.tree().get(".bazelignore").cloned(); + assert_eq!(file().as_deref(), Some(IGNORED.as_bytes())); + // A run that leaves Cargo out adds none, whatever else it has in scope; + // the same run without the option does, and says so to a person. + std::fs::remove_file(cell.path(".bazelignore")).expect("remove the file"); + let repair = socket_patch(&cell, &standin, &["repair", "--ecosystems", "golang"]); + assert_eq!((repair.code, file()), (0, None), "{}", repair.dump); + let told = |[stdout, stderr]: [String; 2], verb: &str, what: &str| { + let line = stdout.lines().find(|line| line.contains(".bazelignore")); + let said = line.is_some_and(|line| line.starts_with(verb) && line.ends_with(what)); + assert!(said, "{verb} ... {what}:\n{stdout}\n{stderr}"); + }; + let updated = ".bazelignore: the lines that keep patched crate copies under .socket out \ + of Bazel's packages"; + told(person(&cell, &standin, &["repair"]), "Updated ", updated); + assert_eq!(file().as_deref(), Some(IGNORED.as_bytes())); + // The user's "no", over a scan and a repair. + cell.write(".bazelignore", DECLINED); + for command in [&["scan", "--mode", "vendored", "--yes"][..], &["repair"]] { + let ran = socket_patch(&cell, &standin, command); + assert_eq!(ran.code, 0, "{}", ran.dump); + let warnings = ran.envelope["warnings"].as_array(); + let warned = |w: &serde_json::Value| w["code"] == "bazelignore_not_updated"; + assert!( + !warnings.is_some_and(|w| w.iter().any(warned)), + "{}", + ran.dump + ); + assert_eq!(file().as_deref(), Some(DECLINED.as_bytes()), "{command:?}"); + } + // The pair goes with the copies, out of a file that keeps the user's + // line, and the file the scan made goes with its last pair. + cell.write(".bazelignore", format!("node_modules\n{IGNORED}")); + let undo = socket_patch(&cell, &standin, &["vendor", "--revert"]); + assert_eq!(undo.code, 0, "{}", undo.dump); + assert_eq!(file().as_deref(), Some(&b"node_modules\n"[..])); + std::fs::remove_file(cell.path(".bazelignore")).expect("remove the file"); + let scan = socket_patch(&cell, &standin, &["scan", "--mode", "vendored", "--yes"]); + assert_eq!(scan.code, 0, "{}", scan.dump); + assert_eq!(file().as_deref(), Some(IGNORED.as_bytes())); + let gone = ".bazelignore: no patched crate copy is left under .socket"; + told( + person(&cell, &standin, &["vendor", "--revert"]), + "Removed ", + gone, + ); + assert_eq!(file(), None); +} + /// Without a Bazel, what stays fixed now that no checksum is committed: the /// fixture's directories and the legs of its workflow are those of its -/// table; the fixture's lock holds the one entry a test pins, with the -/// placeholder, and pinning changes nothing else; two runs serve two -/// upstream crate files; the fixture depends on the crate under the names -/// written out above, and commits no cargo config; every era's hub names -/// the config a test writes and has no cargo-bazel lock; and crates.io -/// cannot hold the name. +/// table; the fixture's lock holds the two entries a test pins, the +/// crate's and its dependency's, with the placeholder, and pinning changes +/// nothing else; two runs serve two upstream crate files and one of the +/// dependency; the fixture depends on the crate under the names written +/// out above, and commits no cargo config; every era's hub names the +/// config a test writes and has no cargo-bazel lock; and crates.io can +/// hold neither name. #[test] -fn a_test_pins_the_one_crate_of_the_fixture_lock() { +fn a_test_pins_the_crates_of_the_fixture_lock() { FIXTURE.check(); let root = fixture_path(FIXTURE.dir); let read = |rel: &str| std::fs::read_to_string(root.join(rel)).expect(rel); @@ -1051,9 +1432,14 @@ fn a_test_pins_the_one_crate_of_the_fixture_lock() { let pinned = lock_pinning(&upstream); let entry = lock_entry(CRATES_IO, &upstream.checksum()); assert!(pinned.contains(&entry), "no\n{entry}in\n{pinned}"); - let unpinned = pinned.replace(&entry, &lock_entry(CRATES_IO, PLACEHOLDER)); + let dependency = dependency_entry(&sha256_hex(&dependency_file())); + assert!(pinned.contains(&dependency), "no\n{dependency}in\n{pinned}"); + let unpinned = pinned + .replace(&entry, &lock_entry(CRATES_IO, PLACEHOLDER)) + .replace(&dependency, &dependency_entry(PLACEHOLDER)); assert_eq!(unpinned, read("common/Cargo.lock")); assert_ne!(Package::upstream().crate_file, upstream.crate_file); + assert_eq!(dependency_file(), dependency_file()); let dependency = format!("\n[dependencies]\n{NAME} = \"{VERSION}\"\n"); assert!(read("common/Cargo.toml").ends_with(&dependency)); @@ -1070,10 +1456,14 @@ fn a_test_pins_the_one_crate_of_the_fixture_lock() { assert!(module.contains(&hub), "{era}: no\n{hub}in\n{module}"); } - assert!(NAME.len() > 64, "crates.io can hold {NAME}"); - assert!(CRATE_MANIFEST.contains(&format!("name = \"{NAME}\"\nversion = \"{VERSION}\"\n"))); + for (name, manifest) in [(NAME, CRATE_MANIFEST), (DEP, DEP_MANIFEST)] { + assert!(name.len() > 64, "crates.io can hold {name}"); + assert!(manifest.contains(&format!("name = \"{name}\"\nversion = \"{VERSION}\"\n"))); + } + let needs = format!("\n[dependencies]\n{DEP} = \"{VERSION}\"\n"); + assert!(CRATE_MANIFEST.ends_with(&needs)); assert_eq!(PURL, format!("pkg:cargo/{NAME}@{VERSION}")); - assert_eq!(UPSTREAM_ENTRY, format!("/{INDEX}/{}", index_path())); + assert_eq!(UPSTREAM_ENTRY, format!("/{INDEX}/{}", index_path(NAME))); assert_eq!( UPSTREAM_CRATE, format!("/{FILES}/crates/{NAME}/{VERSION}/download") diff --git a/crates/socket-patch-cli/tests/fixtures/bazel/rules_rust/bzlmod/common/Cargo.lock b/crates/socket-patch-cli/tests/fixtures/bazel/rules_rust/bzlmod/common/Cargo.lock index d64d2c6a5..29cc0d08c 100644 --- a/crates/socket-patch-cli/tests/fixtures/bazel/rules_rust/bzlmod/common/Cargo.lock +++ b/crates/socket-patch-cli/tests/fixtures/bazel/rules_rust/bzlmod/common/Cargo.lock @@ -9,8 +9,17 @@ dependencies = [ "socket-patch-bazel-fixture-with-a-name-that-is-too-long-for-crates-io", ] +[[package]] +name = "socket-patch-bazel-fixture-dependency-with-a-name-that-is-too-long-for-crates-io" +version = "1.0.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "0000000000000000000000000000000000000000000000000000000000000000" + [[package]] name = "socket-patch-bazel-fixture-with-a-name-that-is-too-long-for-crates-io" version = "1.0.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "0000000000000000000000000000000000000000000000000000000000000000" +dependencies = [ + "socket-patch-bazel-fixture-dependency-with-a-name-that-is-too-long-for-crates-io", +] diff --git a/crates/socket-patch-core/src/bazel/cargo_bazel.rs b/crates/socket-patch-core/src/bazel/cargo_bazel.rs new file mode 100644 index 000000000..630d13db1 --- /dev/null +++ b/crates/socket-patch-core/src/bazel/cargo_bazel.rs @@ -0,0 +1,295 @@ +//! The build file that cargo-bazel, the tool of rules_rust's crate hubs, +//! writes into a crate a Cargo workspace takes from a path. +//! +//! Vendored mode keeps a patched crate under `.socket/vendor/cargo` and +//! points a `[patch.crates-io]` path entry at it. From rules_rust 0.59.0 on, +//! cargo-bazel renders the build file of such a crate to +//! `/BUILD.bazel` in the checkout, each time Bazel evaluates the hub +//! (`crate_universe/src/rendering.rs`, `render_build_files`). The scan +//! recorded no such file, so the copy would read as a changed one after a +//! build that is the patched one. +//! +//! A hub (`crate.from_cargo`) does not build the crate from that file: it +//! copies the directory into a repository of its own and renders the build +//! file of that repository again +//! (`crate_universe/private/local_crate_mirror.bzl`; measured on Bazel +//! 9.3.2, 8.8.2 and 7.7.2, where another file under the same banner left +//! the build as it was). Bazel still reads the file in the checkout as a +//! package of the workspace, and that package names repositories only the +//! hub sees: for a crate with a dependency `bazel cquery //...` and +//! `bazel query 'deps(//...)'` fail on it. So where the module files of +//! the workspace make such a hub of the project's crates ([`hub`]), the +//! copies get a `.bazelignore` line ([`super::ignore`]), and under such a +//! line, where Bazel reads no package in the copy, the file is no change +//! of the copy ([`wrote_build_file`]). +//! +//! Nowhere else. Read in the same source and not run: where `crates_vendor` +//! renders build files into the checkout (`mode = "local"`), they name such +//! a crate by that package, so there the file is the one the crate is built +//! from and a `.bazelignore` line would take the crate away from the build. + +use std::io::Read as _; +use std::path::Path; + +use super::starlark::Value; +use super::workspace::{disk, Extension, Repo, Workspace}; +use super::BazelRoot; +use crate::utils::fs::{open_regular_file_sync, run_blocking}; + +/// The name of that file, in the directory of the crate. +pub(crate) const BUILD_FILE: &str = "BUILD.bazel"; + +/// How every file cargo-bazel renders begins: the first three lines of its +/// banner, the third up to the name of the tool (`partials/header.j2`, +/// unchanged since 2022). The command that renders the file again follows. +const BANNER: [&str; 3] = [ + "###############################################################################", + "# @generated", + "# DO NOT MODIFY: This file is auto-generated by a crate_universe tool.", +]; + +/// The files of rules_rust that define the extension of its crate hubs: +/// the one its documentation names, and the older name it still takes. +const EXTENSION_FILES: [&str; 2] = [ + "//crate_universe:extensions.bzl", + "//crate_universe:extension.bzl", +]; + +/// Whether the module files of the Bazel workspace `root` make a hub of +/// the project's crates: a `crate.from_cargo` call that is a statement of +/// its own ([`super::starlark::Call::whole`]: it runs, once) and names the +/// project's `Cargo.lock` as its `cargo_lockfile`, or the project's +/// `Cargo.toml` among its `manifests`, by a label of the main repository +/// that is written out as a string. A call in a comprehension or behind a +/// condition may never run, and one that names its files in another way, +/// or the files of another Cargo workspace of the checkout, does not show +/// that it reads this project: each is none, and socket-patch then leaves +/// the copies and the `.bazelignore` as they are. A hub of the older +/// `WORKSPACE` system is not looked for. +pub(crate) fn hub(root: &BazelRoot) -> bool { + let workspace = Workspace::read(&disk(&root.dir)); + // Bazel has `/` between the parts of a path on every system. + let project = super::ignore::from_root(root, Path::new("")); + let ours = ["Cargo.lock", "Cargo.toml"].map(|file| format!("{project}{file}")); + let names = |label: &str| { + let main = |label: &super::workspace::Label| label.repo == Repo::Main; + let label = workspace.label(label).filter(main); + label.is_some_and(|label| ours.contains(&label.path())) + }; + EXTENSION_FILES.iter().any(|file| { + let (module, name) = ("rules_rust", "crate"); + let extension = Extension { module, file, name }; + let tags = workspace.tags(&extension, "from_cargo"); + tags.calls.iter().any(|tag| { + let lock = match tag.call.arg("cargo_lockfile") { + Some(Value::Str(label)) => names(label), + _ => false, + }; + let manifests = match tag.call.arg("manifests") { + Some(Value::List(labels)) => labels.iter().any(|label| names(label)), + _ => false, + }; + tag.call.whole && (lock || manifests) + }) + }) +} + +/// Whether the [`BUILD_FILE`] of `crate_dir`, a crate of the project at +/// `project`, is one cargo-bazel wrote and Bazel does not read: a file that +/// begins with [`BANNER`], in a project whose Bazel workspace +/// ([`super::bazel_root`]) has a [`hub`] of its crates and a `.bazelignore` +/// line that covers the crate ([`super::ignore::ignores`]). The banner says +/// which tool the file names as its writer. It does not vouch for the rest +/// of the file, which differs from one machine to the next, so the file is +/// let through only where it is no package of the workspace: without the +/// line it is one, whoever wrote it, and stays a change of the copy. A +/// link is followed: the one caller asks about a file that its own walk of +/// the copy found to be a regular one. +pub(crate) async fn wrote_build_file(project: &Path, crate_dir: &Path) -> bool { + let (project, dir) = (project.to_path_buf(), crate_dir.to_path_buf()); + run_blocking(move || { + // What cannot be read has no banner. + let mut start = Vec::new(); + let _ = open_regular_file_sync(&dir.join(BUILD_FILE)) + .and_then(|(file, _)| file.take(512).read_to_end(&mut start)); + let start = String::from_utf8_lossy(&start); + let mut lines = start.lines(); + let banner = BANNER[..2].iter().all(|line| lines.next() == Some(line)) + && lines.next().is_some_and(|line| line.starts_with(BANNER[2])); + // The cheaper question first: the module files are read last. + let unread = |root: &BazelRoot| { + let copy = dir.strip_prefix(&project); + copy.is_ok_and(|copy| super::ignore::ignores(root, copy)) + }; + banner && super::bazel_root(&project).is_some_and(|root| unread(&root) && hub(&root)) + }) + .await +} + +#[cfg(test)] +pub(crate) mod tests { + use super::*; + use std::fs; + + /// A `MODULE.bazel` with the hub of the rules_rust README. + pub(crate) const HUB: &str = "bazel_dep(name = \"rules_rust\", version = \"0.75.0\")\n\ + crate = use_extension(\"@rules_rust//crate_universe:extensions.bzl\", \"crate\")\n\ + crate.from_cargo(\n name = \"crates\",\n cargo_lockfile = \"//:Cargo.lock\",\n \ + manifests = [\"//:Cargo.toml\"],\n)\n\ + use_repo(crate, \"crates\")\n"; + + /// A checkout with the file `marks` at its root, which holds `module`, + /// a crate in `krate`, whose build file holds `text`, and a + /// `.bazelignore` line for that directory. + fn checkout(marks: &str, module: &str, text: &str) -> tempfile::TempDir { + let tmp = tempfile::tempdir().unwrap(); + fs::create_dir_all(tmp.path().join(".git")).unwrap(); + fs::create_dir_all(tmp.path().join("krate")).unwrap(); + fs::write(tmp.path().join(marks), module).unwrap(); + fs::write(tmp.path().join("krate").join(BUILD_FILE), text).unwrap(); + fs::write(tmp.path().join(".bazelignore"), "krate\n").unwrap(); + tmp + } + + async fn wrote(marks: &str, module: &str, text: &str) -> bool { + let tmp = checkout(marks, module, text); + wrote_build_file(tmp.path(), &tmp.path().join("krate")).await + } + + /// The first lines of the file a build on rules_rust 0.75.0 wrote into + /// a vendored crate (Bazel 9.3.2). + const WRITTEN: &str = + "###############################################################################\n\ + # @generated\n\ + # DO NOT MODIFY: This file is auto-generated by a crate_universe tool. To \n\ + # regenerate this file, run the following:\n\ + #\n\ + # bazel mod show_repo 'socket_fixture'\n\ + ###############################################################################\n\ + \n\ + load(\"@rules_rust//rust:defs.bzl\", \"rust_library\")\n"; + + #[tokio::test] + async fn only_a_file_under_the_banner_in_a_workspace_with_a_crate_hub_is_cargo_bazels() { + const MODULE: &str = "MODULE.bazel"; + assert!(wrote(MODULE, HUB, WRITTEN).await); + assert!(wrote(MODULE, HUB, &WRITTEN.replace('\n', "\r\n")).await); + // Each line of the banner is needed, in its place, from the start + // of the file. + let lines: Vec<&str> = WRITTEN.split_inclusive('\n').collect(); + for missing in 0..3 { + let mut left = lines.clone(); + left.remove(missing); + assert!(!wrote(MODULE, HUB, &left.concat()).await, "{missing}"); + } + // And each line is the banner's own. + for other in [ + String::new(), + format!("\n{WRITTEN}"), + format!("# mine\n{WRITTEN}"), + WRITTEN.replacen('#', "", 1), + WRITTEN.replace("# @generated", "# @generated by hand"), + WRITTEN.replace("crate_universe tool", "tool"), + "genrule(name = \"x\")\n".to_string(), + ] { + assert!(!wrote(MODULE, HUB, &other).await, "{other:?}"); + } + // No Bazel workspace, or one whose module files make no such hub: + // the file is then nobody's. + assert!(!wrote("Cargo.toml", HUB, WRITTEN).await); + for module in [ + "", + "bazel_dep(name = \"rules_rust\", version = \"0.75.0\")\n", + &HUB.replace("from_cargo", "from_specs"), + &HUB.replace("@rules_rust//", "@rules_mine//"), + &HUB.replace("extensions.bzl", "other.bzl"), + &HUB.replace("\"crate\")", "\"other\")"), + ] { + assert!(!wrote(MODULE, module, WRITTEN).await, "{module}"); + } + assert!(!wrote("WORKSPACE", HUB, WRITTEN).await); + // The hub under the older name of its file, under another name for + // the ruleset, and in a file the root includes. + let renamed = HUB + .replace("0.75.0\")", "0.75.0\", repo_name = \"rr\")") + .replace("@rules_rust//", "@rr//"); + for module in [HUB.replace("extensions.bzl", "extension.bzl"), renamed] { + assert!(wrote(MODULE, &module, WRITTEN).await, "{module}"); + } + let tmp = checkout(MODULE, "include(\"//:rust.MODULE.bazel\")\n", WRITTEN); + fs::write(tmp.path().join("rust.MODULE.bazel"), HUB).unwrap(); + assert!(wrote_build_file(tmp.path(), &tmp.path().join("krate")).await); + } + + /// The file is let through only where Bazel does not read it: under a + /// `.bazelignore` line that covers the crate. Without one, or with one + /// the user commented out, it is a package of the workspace, whoever + /// wrote it. + #[tokio::test] + async fn a_build_file_that_bazel_reads_is_not_let_through() { + let tmp = checkout("MODULE.bazel", HUB, WRITTEN); + let (file, krate) = (tmp.path().join(".bazelignore"), tmp.path().join("krate")); + for (lines, covered) in [ + ("krate\n", true), + ("node_modules\nkrate/\n", true), + ("", false), + ("# krate\n", false), + ("krate/src\n", false), + ("other\n", false), + ] { + fs::write(&file, lines).unwrap(); + let wrote = wrote_build_file(tmp.path(), &krate).await; + assert_eq!(wrote, covered, "{lines:?}"); + } + fs::remove_file(&file).unwrap(); + assert!(!wrote_build_file(tmp.path(), &krate).await, "no file"); + } + + /// A hub of the project's crates is a `from_cargo` call that runs and + /// names the project's lock or manifest in so many words. + #[tokio::test] + async fn only_a_call_that_names_the_projects_cargo_files_is_its_hub() { + const MODULE: &str = "MODULE.bazel"; + let lock = " cargo_lockfile = \"//:Cargo.lock\",\n"; + let manifests = " manifests = [\"//:Cargo.toml\"],\n"; + assert!(HUB.contains(lock) && HUB.contains(manifests)); + let elsewhere = |hub: &str| hub.replace("\"//:", "\"//other:"); + // Either file names the project, under any name for the main + // repository. + for module in [ + HUB.replace(lock, ""), + HUB.replace(manifests, ""), + elsewhere(HUB).replace("\"//other:Cargo.lock\"", "\"@//:Cargo.lock\""), + elsewhere(HUB).replace("//other:Cargo.toml", "@@//:Cargo.toml"), + ] { + assert!(wrote(MODULE, &module, WRITTEN).await, "{module}"); + } + // The files of another Cargo workspace, of another repository, or + // not written out; and a call that may never run. + for module in [ + elsewhere(HUB), + HUB.replace("\"//:", "\"@other//:"), + HUB.replace("\"//:Cargo.lock\"", "LOCK") + .replace("[\"//:Cargo.toml\"]", "MANIFESTS"), + HUB.replace("crate.from_cargo(", "False and crate.from_cargo("), + HUB.replace("crate.from_cargo(", "[crate.from_cargo(") + .replace(")\nuse_repo", ") for _ in []]\nuse_repo"), + ] { + assert!(!wrote(MODULE, &module, WRITTEN).await, "{module}"); + } + // A project below the root is named from the root. + let tmp = tempfile::tempdir().unwrap(); + let (svc, krate) = (tmp.path().join("svc"), tmp.path().join("svc/krate")); + fs::create_dir_all(tmp.path().join(".git")).unwrap(); + fs::create_dir_all(&krate).unwrap(); + fs::write(krate.join(BUILD_FILE), WRITTEN).unwrap(); + fs::write(tmp.path().join(".bazelignore"), "svc/krate\n").unwrap(); + for (module, ours) in [ + (elsewhere(HUB).replace("other", "svc"), true), + (HUB.into(), false), + ] { + fs::write(tmp.path().join(MODULE), &module).unwrap(); + assert_eq!(wrote_build_file(&svc, &krate).await, ours, "{module}"); + } + } +} diff --git a/crates/socket-patch-core/src/bazel/ignore.rs b/crates/socket-patch-core/src/bazel/ignore.rs index aad0e4476..5d8807133 100644 --- a/crates/socket-patch-core/src/bazel/ignore.rs +++ b/crates/socket-patch-core/src/bazel/ignore.rs @@ -1,5 +1,6 @@ -//! The `.bazelignore` lines that keep patched Go module copies out of the -//! packages of a Bazel workspace. +//! The `.bazelignore` lines that keep patched Go module copies, and the +//! patched crates a crate hub builds, out of the packages of a Bazel +//! workspace. //! //! Vendored and agent mode patch a Go module by keeping a copy of it under //! the project's `.socket` directory, which a `go.mod` `replace` points @@ -10,33 +11,63 @@ //! fails to load rules_go from it. A `.bazelignore` line keeps both out. //! Bazel reads the file at the root of the workspace only and takes each //! line as a literal directory path from that root. +//! +//! Vendored mode keeps a patched crate under `.socket/vendor/cargo`, and a +//! rules_rust crate hub writes a `BUILD.bazel` into it +//! ([`super::cargo_bazel`]). The hub reads the crate by its path, and the +//! package that file makes names repositories only the hub sees, so a +//! command that analyses every target fails on it (`bazel cquery //...`, +//! `bazel query 'deps(//...)'`). With a line the build is the same and the +//! package is none. Only where the module files make such a hub of the +//! project's crates: other tools of the ruleset build a crate from that +//! package. use std::path::{Path, PathBuf}; +use super::{cargo_bazel, BazelRoot}; use crate::utils::fs::{atomic_write_sync, read_regular_to_bytes_sync, run_blocking}; +use crate::vendor::cargo_config::CARGO_VENDOR_DIR; use crate::vendor::go_mod_edit::{GO_PATCHES_DIR, GO_VENDOR_DIR}; /// The warning code of a `.bazelignore` that [`sync`] could not update. pub const NOT_UPDATED: &str = "bazelignore_not_updated"; -/// The line above every line [`sync`] writes. -const COMMENT: &str = - "# socket-patch: patched Go module copies, not Bazel packages (to opt out, comment out the next line)"; +/// The directories [`sync`] keeps a line for, from the project, what the +/// copies in each are, and whether they are the crates, whose pair is +/// added only under a hub that builds them. +const COPIES: [(&str, &str, bool); 3] = [ + (GO_VENDOR_DIR, "Go module", false), + (GO_PATCHES_DIR, "Go module", false), + (CARGO_VENDOR_DIR, "crate", true), +]; + +/// The line above the line [`sync`] writes for a directory of `what` +/// copies. +fn comment(what: &str) -> String { + format!( + "{MARK} patched {what} copies, not Bazel packages (to opt out, comment out the next line)" + ) +} -/// How [`COMMENT`] starts, in this wording and in any other. A line is +/// How [`comment`] starts, in this wording and in any other. A line is /// [`sync`]'s to remove only when it stands under a line that starts so, /// and a user who puts a `#` before that line has declined it: [`sync`] /// leaves both lines alone. const MARK: &str = "# socket-patch:"; -/// Make the `.bazelignore` at the Bazel root of `project` agree with the Go -/// module copies of the project: for each of `.socket/vendor/golang` and -/// `.socket/go-patches`, one line under [`COMMENT`] while the directory -/// exists and the file neither covers nor declines it ([`settled`]), and no -/// such pair once it is gone; without `add`, pairs are only taken away. A -/// directory that cannot be probed keeps what the file says of it. A file -/// left empty is deleted. Outside a Bazel workspace ([`super::bazel_root`]) -/// nothing is read or written, and that lookup is all the call costs. +/// Make the `.bazelignore` at the Bazel root of `project` agree with the +/// copies of the project: for each of `.socket/vendor/golang`, +/// `.socket/go-patches` and `.socket/vendor/cargo` ([`COPIES`]), one line +/// under its comment line while the directory exists and the file neither +/// covers nor declines it ([`settled`]), and no such pair once it is gone. +/// A pair is added only for an ecosystem the run has in scope (`go`, +/// `cargo`), and for the crates only where the workspace has a hub of the +/// project's crates ([`cargo_bazel::hub`], whose files are read when a +/// pair for that directory is to be added and at no other time): elsewhere +/// a directory that exists keeps what the file says of it, as does one +/// that cannot be probed. A file left empty is deleted. +/// Outside a Bazel workspace ([`super::bazel_root`]) nothing is read or +/// written, and that lookup is all the call costs. /// /// The pair follows the directory, not the `replace`: a rollback that /// keeps the copy on disk keeps its BUILD files in the workspace too. @@ -45,23 +76,43 @@ const MARK: &str = "# socket-patch:"; /// its project's, so the edit, from its read to its write, is made under a /// lock of the root ([`lock`]). /// -/// `Ok(Some)` is the file, when it was changed. `Err` is the detail of a +/// `Ok(Some)` is the file, when it was changed, and whose lines came or +/// went: `Go module`, `crate`, or both, in that order. `Err` is the detail +/// of a /// [`NOT_UPDATED`] warning: the file could not be read or written, or is a /// link, which socket-patch does not write through. The command that asked /// has done its work either way. -pub async fn sync(project: &Path, add: bool) -> Result, String> { +pub async fn sync(project: &Path, go: bool, cargo: bool) -> Result, String> { let project = project.to_path_buf(); - run_blocking(move || sync_blocking(&project, add)).await + run_blocking(move || sync_blocking(&project, go, cargo)).await } -fn sync_blocking(project: &Path, add: bool) -> Result, String> { +/// The file [`sync`] changed, and the copies its changed lines are about. +pub type Changed = (PathBuf, Vec<&'static str>); + +/// The directory `dir` of the project, from the Bazel root `root`, as +/// Bazel writes a path: its parts with `/` after each, on every system. +/// Empty for the root itself. +pub(crate) fn from_root(root: &BazelRoot, dir: &Path) -> String { + (root.project.join(dir).components()) + .map(|part| format!("{}/", part.as_os_str().to_string_lossy())) + .collect() +} + +/// Whether a line of the `.bazelignore` at `root` makes Bazel skip `dir`, +/// a directory from the project ([`covers`]). A file that cannot be read +/// covers nothing. +pub(crate) fn ignores(root: &BazelRoot, dir: &Path) -> bool { + let dir = from_root(root, dir); + let file = read_regular_to_bytes_sync(&root.dir.join(".bazelignore")); + file.is_ok_and(|file| covers(&file, dir.trim_end_matches('/'))) +} + +fn sync_blocking(project: &Path, go: bool, cargo: bool) -> Result, String> { let Some(root) = super::bazel_root(project) else { return Ok(None); }; - // Bazel wants `/` on every system. - let from_root: String = (root.project.components()) - .map(|part| format!("{}/", part.as_os_str().to_string_lossy())) - .collect(); + let from_root = from_root(&root, Path::new("")); let project = root.dir.join(&root.project); // `None`: it cannot be told (a directory above it that may not be // searched). Only a directory that is not found is gone. @@ -69,8 +120,24 @@ fn sync_blocking(project: &Path, add: bool) -> Result, String> { Ok(meta) => Some(meta.is_dir()), Err(e) => (e.kind() == std::io::ErrorKind::NotFound).then_some(false), }; - let wanted = - [GO_VENDOR_DIR, GO_PATCHES_DIR].map(|dir| (format!("{from_root}{dir}"), exists(dir))); + let mut wanted = COPIES.map(|(dir, what, crates)| { + // A copy that is there gets a pair only from a run that has its + // ecosystem in scope. + let adds = if crates { cargo } else { go }; + let state = exists(dir).filter(|there| adds || !there); + (format!("{from_root}{dir}"), state, what) + }); + // And the crates only under a hub that builds them. Its files are read + // once, and only when `text` would get a pair for the crates. + let mut hub = None; + let mut under_a_hub = |wanted: &mut [(String, Option, &'static str); 3], text: &[u8]| { + for ((.., crates), (line, state, _)) in COPIES.iter().zip(wanted.iter_mut()) { + let adds = *crates && *state == Some(true) && !settled(text, line); + if adds && !*hub.get_or_insert_with(|| cargo_bazel::hub(&root)) { + *state = None; + } + } + }; let file = root.dir.join(".bazelignore"); let read = || match read_regular_to_bytes_sync(&file) { @@ -80,7 +147,8 @@ fn sync_blocking(project: &Path, add: bool) -> Result, String> { // Most runs find nothing to change, and take no lock to find that. A // file that cannot be read is an edit only with a copy to name in it. let unlocked = read().unwrap_or_default(); - if edited(&unlocked, &wanted, add) == unlocked { + under_a_hub(&mut wanted, &unlocked); + if edited(&unlocked, &wanted) == unlocked { return Ok(None); } let _lock = lock(&root.dir); @@ -88,7 +156,8 @@ fn sync_blocking(project: &Path, add: bool) -> Result, String> { Ok(old) => (old, None), Err(e) => (Vec::new(), Some(e)), }; - let new = edited(&old, &wanted, add); + under_a_hub(&mut wanted, &old); + let new = edited(&old, &wanted); // The root is canonical: on Windows that spelling has a prefix no user // types and no tool prints. let shown = crate::utils::pnpm_workspace::without_verbatim_prefix(file.clone()); @@ -101,7 +170,18 @@ fn sync_blocking(project: &Path, add: bool) -> Result, String> { "it is a symbolic link, which socket-patch does not write through", ), None => match write(&file, &old, &new) { - Ok(()) => return Ok(Some(shown)), + Ok(()) => { + // Whose lines the file has more or fewer of than before. + let lines = |text: &[u8], line: &str| { + let is = |l: &&[u8]| content(l) == line.as_bytes(); + text.split(|byte| *byte == b'\n').filter(is).count() + }; + let differs = |line: &str| lines(&old, line) != lines(&new, line); + let whose = wanted.iter().filter(|(line, ..)| differs(line)); + let mut whose: Vec<_> = whose.map(|(.., what)| *what).collect(); + whose.dedup(); + return Ok(Some((shown, whose))); + } Err(e) => e, }, }; @@ -109,14 +189,14 @@ fn sync_blocking(project: &Path, add: bool) -> Result, String> { // behind for a directory that is gone ignores nothing. let missing: Vec = wanted .iter() - .filter(|(line, exists)| add && *exists == Some(true) && !settled(&old, line)) - .map(|(line, _)| format!("`{line}`")) + .filter(|(line, state, _)| *state == Some(true) && !settled(&old, line)) + .map(|(line, ..)| format!("`{line}`")) .collect(); let advice = if !missing.is_empty() { format!( "add {} to it on a line of its own, with no space before or after, so that Bazel \ - and gazelle do not read the patched Go module copies there as packages (`bazel run \ - //:gazelle` or `bazel build //...` can otherwise break the build)", + and gazelle do not read the patched copies there as packages (`bazel run \ + //:gazelle`, `bazel build //...` or `bazel cquery //...` can otherwise fail)", missing.join(" and ") ) } else { @@ -216,19 +296,20 @@ fn settled(old: &[u8], path: &str) -> bool { covers(old, path) || lines.windows(2).any(declined) } -/// `old` as [`sync`] leaves it for `wanted`, the line of each directory and -/// whether the directory exists (`None`: unknown, and nothing changes for -/// it): without the pairs of the directories that are gone, and, with -/// `add`, with a pair at its end, in the file's own line ending, for each -/// directory that exists and is not [`settled`]. A file whose last line has -/// no line break gets one before the pair and none after it, and loses that -/// one (the comment line's own) with a pair that ends the file that way, so -/// that the file without the pair is the file of before, byte for byte. -fn edited(old: &[u8], wanted: &[(String, Option)], add: bool) -> Vec { +/// `old` as [`sync`] leaves it for `wanted`, the line of each directory, +/// whether the directory exists (`None`: unknown, or not this run's to +/// add, and nothing changes for it) and what its copies are: without the +/// pairs of the directories that are gone, and with a pair at its end, in +/// the file's own line ending, for each directory that exists and is not +/// [`settled`]. A file whose last line has no line break gets one before +/// the pair and none after it, and loses that one (the comment line's own) +/// with a pair that ends the file that way, so that the file without the +/// pair is the file of before, byte for byte. +fn edited(old: &[u8], wanted: &[(String, Option, &str)]) -> Vec { let gone = |line: &[u8]| { wanted .iter() - .any(|(l, exists)| *exists == Some(false) && l.as_bytes() == content(line)) + .any(|(l, exists, _)| *exists == Some(false) && l.as_bytes() == content(line)) }; let lines: Vec<&[u8]> = old.split_inclusive(|byte| *byte == b'\n').collect(); let mut new = Vec::with_capacity(old.len()); @@ -249,13 +330,14 @@ fn edited(old: &[u8], wanted: &[(String, Option)], add: bool) -> Vec { } } let nl = crate::utils::line_endings::terminator(&String::from_utf8_lossy(old)); - for (line, exists) in wanted { - if !add || *exists != Some(true) || settled(&new, line) { + for (line, exists, what) in wanted { + if *exists != Some(true) || settled(&new, line) { continue; } + let comment = comment(what); let pair = match new.is_empty() || new.ends_with(b"\n") { - true => format!("{COMMENT}{nl}{line}{nl}"), - false => format!("{nl}{COMMENT}{nl}{line}"), + true => format!("{comment}{nl}{line}{nl}"), + false => format!("{nl}{comment}{nl}{line}"), }; new.extend_from_slice(pair.as_bytes()); } @@ -269,13 +351,17 @@ mod tests { const VENDOR: &str = ".socket/vendor/golang"; const PATCHES: &str = ".socket/go-patches"; + const CRATES: &str = ".socket/vendor/cargo"; + /// The line above the line of a directory of Go module copies. + const COMMENT: &str = + "# socket-patch: patched Go module copies, not Bazel packages (to opt out, comment out the next line)"; /// [`edited`] for the two directories of a project at the root, each /// existing or not. fn edit(old: &str, vendor: bool, patches: bool) -> String { let wanted = [(VENDOR, vendor), (PATCHES, patches)]; - let wanted = wanted.map(|(line, exists)| (line.to_string(), Some(exists))); - String::from_utf8(edited(old.as_bytes(), &wanted, true)).unwrap() + let wanted = wanted.map(|(line, exists)| (line.to_string(), Some(exists), "Go module")); + String::from_utf8(edited(old.as_bytes(), &wanted)).unwrap() } fn pair(line: &str) -> String { @@ -511,17 +597,21 @@ mod tests { (tmp, root, project) } - /// [`super::sync`] for a run that has Go in scope. - async fn sync(project: &Path) -> Result, String> { - super::sync(project, true).await + /// [`super::sync`] for a run that has every ecosystem in scope. + async fn sync(project: &Path) -> Result, String> { + super::sync(project, true, true).await + } + + /// The file [`sync`] says it changed, as the test spells it, for lines + /// of Go module copies. + fn changed(root: &Path) -> Result, String> { + changed_for(root, "Go module") } - /// The file [`sync`] says it changed, as the test spells it. - fn changed(root: &Path) -> Result, String> { + fn changed_for(root: &Path, whose: &'static str) -> Result, String> { let file = root.join(".bazelignore"); - Ok(Some(crate::utils::pnpm_workspace::without_verbatim_prefix( - file, - ))) + let file = crate::utils::pnpm_workspace::without_verbatim_prefix(file); + Ok(Some((file, whose.split(" and ").collect()))) } #[tokio::test] @@ -602,23 +692,24 @@ mod tests { async fn a_run_that_leaves_go_out_only_takes_pairs_away() { let (_tmp, root, project) = project("", VENDOR); let file = root.join(".bazelignore"); - assert_eq!(super::sync(&project, false).await, Ok(None)); + let sync = |project: PathBuf| async move { super::sync(&project, false, true).await }; + assert_eq!(sync(project.clone()).await, Ok(None)); assert!(!file.exists()); fs::create_dir(&file).unwrap(); - assert_eq!(super::sync(&project, false).await, Ok(None), "unreadable"); + assert_eq!(sync(project.clone()).await, Ok(None), "unreadable"); fs::remove_dir(&file).unwrap(); // The pair of the copy that is there stays, and the one of the copy // that is gone goes. let kept = format!("node_modules\n{}", pair(VENDOR)); fs::write(&file, format!("{kept}{}", pair(PATCHES))).unwrap(); - assert_eq!(super::sync(&project, false).await, changed(&root)); + assert_eq!(sync(project.clone()).await, changed(&root)); assert_eq!(fs::read_to_string(&file).unwrap(), kept); fs::remove_dir_all(project.join(VENDOR)).unwrap(); let mut bits = fs::metadata(&file).unwrap().permissions(); bits.set_readonly(true); fs::set_permissions(&file, bits).unwrap(); if fs::OpenOptions::new().append(true).open(&file).is_err() { - let detail = super::sync(&project, false).await.expect_err("read-only"); + let detail = sync(project.clone()).await.expect_err("read-only"); assert!(detail.contains("can be deleted"), "{detail}"); assert_eq!(fs::read_to_string(&file).unwrap(), kept); } @@ -660,6 +751,62 @@ mod tests { } } + /// The vendored crates get their pair, under a comment line of their + /// own, where the module files of the workspace make a crate hub and + /// the run has Cargo in scope. Elsewhere a directory that is there + /// keeps what the file says of it, and one that is gone loses its pair + /// as any other does. The caller is told whose lines changed. + #[tokio::test] + async fn the_crates_get_a_pair_only_where_a_hub_builds_them() { + use crate::bazel::cargo_bazel::tests::HUB; + let crates = format!( + "# socket-patch: patched crate copies, not Bazel packages (to opt out, comment out \ + the next line)\n{}\n", + "svc/.socket/vendor/cargo" + ); + let (_tmp, root, project) = project("svc", CRATES); + let (file, module) = (root.join(".bazelignore"), root.join("MODULE.bazel")); + // No hub: no pair, and the pair that is there stays. + for old in [None, Some(crates.as_str())] { + if let Some(old) = old { + fs::write(&file, old).unwrap(); + } + assert_eq!(sync(&project).await, Ok(None), "{old:?}"); + assert_eq!(fs::read_to_string(&file).ok().as_deref(), old); + } + fs::remove_file(&file).unwrap(); + // Nor under the hub of another Cargo workspace of the checkout. + fs::write(&module, HUB).unwrap(); + assert_eq!(sync(&project).await, Ok(None), "the root's hub"); + assert!(!file.exists()); + let hub = HUB.replace("\"//:", "\"//svc:"); + fs::write(&module, &hub).unwrap(); + // A run that leaves Cargo out adds none either. + assert_eq!(super::sync(&project, true, false).await, Ok(None)); + assert!(!file.exists()); + assert_eq!(sync(&project).await, changed_for(&root, "crate")); + assert_eq!(fs::read_to_string(&file).unwrap(), crates); + assert_eq!(sync(&project).await, Ok(None), "a second run"); + // With a Go copy beside them, and then without the crates. + fs::create_dir_all(project.join(VENDOR)).unwrap(); + assert_eq!(sync(&project).await, changed(&root)); + let both = format!("{crates}{}", pair("svc/.socket/vendor/golang")); + assert_eq!(fs::read_to_string(&file).unwrap(), both); + fs::remove_dir_all(project.join(CRATES)).unwrap(); + fs::remove_dir_all(project.join(VENDOR)).unwrap(); + fs::write(&module, "").unwrap(); + let gone = super::sync(&project, false, false).await; + assert_eq!(gone, changed_for(&root, "Go module and crate")); + assert!(!file.exists()); + // A user's "no" holds for the crates as for any other directory. + fs::write(&module, &hub).unwrap(); + fs::create_dir_all(project.join(CRATES)).unwrap(); + let declined = crates.replace("\nsvc/", "\n# svc/"); + fs::write(&file, &declined).unwrap(); + assert_eq!(sync(&project).await, Ok(None)); + assert_eq!(fs::read_to_string(&file).unwrap(), declined); + } + #[tokio::test] async fn a_project_below_the_root_is_named_from_the_root() { let (_tmp, root, project) = project("svc/go", PATCHES); diff --git a/crates/socket-patch-core/src/bazel/mod.rs b/crates/socket-patch-core/src/bazel/mod.rs index f2d19a28d..e9eef5e16 100644 --- a/crates/socket-patch-core/src/bazel/mod.rs +++ b/crates/socket-patch-core/src/bazel/mod.rs @@ -9,6 +9,7 @@ pub const WORKSPACE_FILES: [&str; 4] = ["MODULE.bazel", "REPO.bazel", "WORKSPACE", "WORKSPACE.bazel"]; pub mod built; +pub(crate) mod cargo_bazel; pub mod ignore; pub mod maven; pub mod rc; diff --git a/crates/socket-patch-core/src/vendor/cargo_config.rs b/crates/socket-patch-core/src/vendor/cargo_config.rs index a68e51620..38cc68978 100644 --- a/crates/socket-patch-core/src/vendor/cargo_config.rs +++ b/crates/socket-patch-core/src/vendor/cargo_config.rs @@ -54,7 +54,7 @@ fn config_doc(content: &str) -> Result, toml_edit::TomlError> { /// Project-relative root of the vendor backend's committed crate copies. An /// entry whose `path` is under this prefix is socket-owned. -const CARGO_VENDOR_DIR: &str = ".socket/vendor/cargo"; +pub(crate) const CARGO_VENDOR_DIR: &str = ".socket/vendor/cargo"; /// Cargo's legacy extensionless project config: when it exists cargo reads /// it INSTEAD of [`CONFIG_TOML`] (and warns that the `.toml` is ignored). diff --git a/crates/socket-patch-core/src/vendor/verify.rs b/crates/socket-patch-core/src/vendor/verify.rs index 72cdcb9c5..7cfaf05cf 100644 --- a/crates/socket-patch-core/src/vendor/verify.rs +++ b/crates/socket-patch-core/src/vendor/verify.rs @@ -20,7 +20,11 @@ //! `vendor_manifest_unverifiable` //! (dir-shaped artifacts with a recorded [`file_inventory`] additionally //! verify their FULL file tree — missing, extra and modified unpatched -//! files all fail; entries without one keep member-only verification). +//! files all fail; entries without one keep member-only verification). One +//! extra file is none: the build file cargo-bazel writes into a vendored +//! crate of a Bazel workspace with a hub of the project's crates, where a +//! `.bazelignore` line keeps Bazel from reading it +//! ([`crate::bazel::cargo_bazel`]). //! //! [`file_inventory`]: super::state::VendorArtifact::file_inventory @@ -28,6 +32,7 @@ use std::collections::{BTreeMap, HashMap}; use std::io::Read; use std::path::{Path, PathBuf}; +use crate::bazel::cargo_bazel::{self, BUILD_FILE}; use crate::hash::git_sha256::compute_git_sha256_from_bytes; use crate::manifest::schema::{PatchFileInfo, PatchRecord}; use crate::patch::apply::{ @@ -163,8 +168,19 @@ pub async fn verify_vendored_patch_record( // (missing / extra / modified unpatched files, the stub gemspec). // Pre-inventory entries carry `None` and keep member-only behavior. if let Some(inventory) = &entry.artifact.file_inventory { - let cargo_uuid = (entry.ecosystem == "cargo").then_some(entry.uuid.as_str()); - verify_dir_inventory(&artifact, inventory, cargo_uuid).await?; + let mut actual = dir_inventory(&artifact).await?; + // One file the inventory does not list is no extra file of a + // cargo copy in its place: the `BUILD.bazel` in the copy's own + // directory that cargo-bazel wrote there and Bazel does not + // read. A build file the crate ships is in the inventory and + // is held to its bytes, like every other recorded file. + let unlisted = cargo_uuid.is_some() + && actual.contains_key(BUILD_FILE) + && !inventory.contains_key(BUILD_FILE); + if unlisted && cargo_bazel::wrote_build_file(project_root, &artifact).await { + actual.remove(BUILD_FILE); + } + inventory_matches(&artifact, &actual, inventory, cargo_uuid).await?; } return Ok(()); } @@ -909,9 +925,24 @@ pub(super) async fn verify_dir_inventory( inventory: &BTreeMap, cargo_uuid: Option<&str>, ) -> Result<(), String> { - let actual = compute_dir_inventory(dir) - .await - .map_err(|_| "vendor_artifact_unreadable".to_string())?; + let actual = dir_inventory(dir).await?; + inventory_matches(dir, &actual, inventory, cargo_uuid).await +} + +/// The files under `dir` ([`compute_dir_inventory`]), or the routing tag +/// of a tree that cannot be walked. +async fn dir_inventory(dir: &Path) -> Result, String> { + let actual = compute_dir_inventory(dir).await; + actual.map_err(|_| "vendor_artifact_unreadable".to_string()) +} + +/// [`verify_dir_inventory`] for `actual`, the files found under `dir`. +async fn inventory_matches( + dir: &Path, + actual: &BTreeMap, + inventory: &BTreeMap, + cargo_uuid: Option<&str>, +) -> Result<(), String> { if actual.len() != inventory.len() { return Err("vendor_inventory_mismatch".to_string()); } @@ -1621,6 +1652,172 @@ mod tests { ); } + /// In a Bazel workspace with a hub of the project's crates and a + /// `.bazelignore` line over the copies, the `BUILD.bazel` that + /// cargo-bazel wrote into the directory of a vendored crate is no extra + /// file of the copy (first row). Every other row differs from it in one + /// thing, and is the changed copy it was before. So is the first with + /// one more change or without the line, and a build file the crate + /// ships is a recorded file. + #[tokio::test] + async fn cargo_inventory_leaves_out_only_the_build_file_cargo_bazel_wrote() { + use crate::bazel::cargo_bazel::tests::HUB; + // How the file a build on rules_rust 0.75.0 wrote begins. + const WRITTEN: &str = + "###############################################################################\n\ + # @generated\n\ + # DO NOT MODIFY: This file is auto-generated by a crate_universe tool. To \n\ + # regenerate this file, run the following:\n"; + const OWN: &str = "exports_files([\"Cargo.toml\"])\n"; + const BUILD: &str = "BUILD.bazel"; + const BAZEL: &str = "MODULE.bazel"; + const REL: &str = ".socket/vendor/cargo/9f6b2c4e-1d3a-4f6b-8c2d-7e5a9b1c3d5f/cfg-if-1.0.4"; + const IGNORE: &str = ".bazelignore"; + // A checkout with the file `marks` at its root, which holds + // `module`, the crate in it, and the line that keeps Bazel out of + // the copies. + let checkout = |marks: &str, module: &str| { + let tmp = tempfile::tempdir().unwrap(); + let dir = tmp.path().join(REL); + std::fs::create_dir_all(dir.join("src")).unwrap(); + std::fs::create_dir(tmp.path().join(".git")).unwrap(); + std::fs::write(tmp.path().join(marks), module).unwrap(); + std::fs::write(tmp.path().join(IGNORE), ".socket/vendor/cargo\n").unwrap(); + let manifest = "[package]\nname = \"cfg-if\"\nversion = \"1.0.4\"\n"; + std::fs::write(dir.join("Cargo.toml"), manifest).unwrap(); + std::fs::write(dir.join("README.md"), "read me\n").unwrap(); + std::fs::write(dir.join("src/lib.rs"), PATCHED).unwrap(); + (tmp, dir) + }; + let changed = Err("vendor_inventory_mismatch".to_string()); + + for (why, marks, module, ecosystem, file, text) in [ + ("cargo-bazel's file", BAZEL, HUB, "cargo", BUILD, WRITTEN), + ( + "no Bazel workspace", + "Cargo.lock", + HUB, + "cargo", + BUILD, + WRITTEN, + ), + ("no crate hub", BAZEL, "", "cargo", BUILD, WRITTEN), + ("another ecosystem", BAZEL, HUB, "gem", BUILD, WRITTEN), + ("no banner", BAZEL, HUB, "cargo", BUILD, OWN), + ("Bazel's other name", BAZEL, HUB, "cargo", "BUILD", WRITTEN), + ( + "a directory", + BAZEL, + HUB, + "cargo", + "src/BUILD.bazel", + WRITTEN, + ), + ] { + let (tmp, dir) = checkout(marks, module); + let key = ["src/lib.rs", "package/src/lib.rs"][usize::from(ecosystem == "cargo")]; + let rec = record(UUID, key); + let mut ent = entry(ecosystem, UUID, REL); + let inventory = compute_dir_inventory(&dir).await.unwrap(); + ent.artifact.file_inventory = Some(inventory.clone()); + let got = verify_vendored_patch_record(tmp.path(), &ent, &rec).await; + assert_eq!(got, Ok(()), "{why}: as the scan left it"); + std::fs::write(dir.join(file), text).unwrap(); + let verdict = match why { + "cargo-bazel's file" => Ok(()), + _ => changed.clone(), + }; + let got = verify_vendored_patch_record(tmp.path(), &ent, &rec).await; + assert_eq!(got, verdict, "{why}"); + // A download staged for a copy is held to the recorded files + // alone: its check knows no such file. + let staged = verify_dir_inventory(&dir, &inventory, Some(UUID)).await; + assert_eq!(staged, changed, "{why}"); + } + + let (tmp, dir) = checkout(BAZEL, HUB); + let rec = record(UUID, "package/src/lib.rs"); + let mut ent = entry("cargo", UUID, REL); + ent.artifact.file_inventory = Some(compute_dir_inventory(&dir).await.unwrap()); + std::fs::write(dir.join(BUILD), WRITTEN).unwrap(); + // Bazel must not read the file. Without a line that covers the + // copy, or under one the user commented out, it is a package of + // the workspace, whoever wrote it, and a change of the copy. + for (lines, verdict) in [ + ("", changed.clone()), + ("# .socket/vendor/cargo\n", changed.clone()), + (".socket/vendor/golang\n", changed.clone()), + (".socket\n", Ok(())), + ] { + std::fs::write(tmp.path().join(IGNORE), lines).unwrap(); + let got = verify_vendored_patch_record(tmp.path(), &ent, &rec).await; + assert_eq!(got, verdict, "{lines:?}"); + } + std::fs::remove_file(tmp.path().join(IGNORE)).unwrap(); + let got = verify_vendored_patch_record(tmp.path(), &ent, &rec).await; + assert_eq!(got, changed, "no .bazelignore"); + std::fs::write(tmp.path().join(IGNORE), ".socket/vendor/cargo\n").unwrap(); + for (file, text, back) in [ + ("src/x.rs", "", None), + ("src/BUILD.bazel", WRITTEN, None), + ("README.md", "", Some("read me\n")), + ] { + std::fs::write(dir.join(file), text).unwrap(); + let got = verify_vendored_patch_record(tmp.path(), &ent, &rec).await; + assert_eq!(got, changed, "{file} beside cargo-bazel's file"); + match back { + Some(text) => std::fs::write(dir.join(file), text).unwrap(), + None => std::fs::remove_file(dir.join(file)).unwrap(), + } + let got = verify_vendored_patch_record(tmp.path(), &ent, &rec).await; + assert_eq!(got, Ok(()), "{file} as recorded again"); + } + // A link of that name is no file of a copy, wherever it leads. + #[cfg(unix)] + { + std::fs::rename(dir.join(BUILD), tmp.path().join(BUILD)).unwrap(); + std::os::unix::fs::symlink(tmp.path().join(BUILD), dir.join(BUILD)).unwrap(); + let got = verify_vendored_patch_record(tmp.path(), &ent, &rec).await; + assert_eq!(got, Err("vendor_artifact_unreadable".to_string())); + std::fs::remove_file(dir.join(BUILD)).unwrap(); + } + // A build file the crate ships is a recorded file: cargo-bazel's + // over it is a changed one. + std::fs::write(dir.join(BUILD), OWN).unwrap(); + ent.artifact.file_inventory = Some(compute_dir_inventory(&dir).await.unwrap()); + let got = verify_vendored_patch_record(tmp.path(), &ent, &rec).await; + assert_eq!(got, Ok(()), "the crate's own build file"); + std::fs::write(dir.join(BUILD), WRITTEN).unwrap(); + let got = verify_vendored_patch_record(tmp.path(), &ent, &rec).await; + assert_eq!(got, changed, "cargo-bazel's file over the crate's own"); + // So is one the crate ships under the banner: as recorded it is + // the copy's, and with another byte a changed one. + ent.artifact.file_inventory = Some(compute_dir_inventory(&dir).await.unwrap()); + let got = verify_vendored_patch_record(tmp.path(), &ent, &rec).await; + assert_eq!(got, Ok(()), "the crate's own build file, under the banner"); + std::fs::write(dir.join(BUILD), format!("{WRITTEN}\n")).unwrap(); + let got = verify_vendored_patch_record(tmp.path(), &ent, &rec).await; + assert_eq!(got, changed, "another byte in a recorded build file"); + + // The workspace is the project's. A crate that ships module files + // of its own, in a project without Bazel, is in none. + let (tmp, dir) = checkout("Cargo.lock", ""); + std::fs::write(dir.join(BAZEL), HUB).unwrap(); + ent.artifact.file_inventory = Some(compute_dir_inventory(&dir).await.unwrap()); + let got = verify_vendored_patch_record(tmp.path(), &ent, &rec).await; + assert_eq!( + got, + Ok(()), + "a crate with module files, as the scan left it" + ); + std::fs::write(dir.join(BUILD), WRITTEN).unwrap(); + let got = verify_vendored_patch_record(tmp.path(), &ent, &rec).await; + assert_eq!( + got, changed, + "the module files of the crate make no workspace" + ); + } + /// The MEMBER path (a record keyed on the copy's own `Cargo.toml`, the /// shape a patch that edits the crate's manifest takes) applies the same /// uuid pin as the inventory path: the tag is dropped only when it is diff --git a/docs/ecosystems.md b/docs/ecosystems.md index ab1d2884b..7eaae2adb 100644 --- a/docs/ecosystems.md +++ b/docs/ecosystems.md @@ -2232,6 +2232,112 @@ version = "1.0.4+socket." hosted pin left there. Hosted scan and `vendor` refuse the same directories (`cargo_manifest_not_workspace_root`). +## Bazel workspaces: the build file in a vendored crate + +A rules_rust crate hub (`crate.from_cargo`) builds a vendored crate from its copy under +`.socket/vendor/cargo`: the build after `scan --mode vendored` is the patched one, with no +repin. From rules_rust 0.59.0 on, the hub's tool, cargo-bazel, also writes a `BUILD.bazel` +into the copy's own directory each time Bazel evaluates the hub, as it does for every crate +a Cargo workspace takes from a path. The scan recorded no such file, and Bazel reads it as +a package of your workspace. + +socket-patch does two things about that file, both only where the Bazel workspace has a hub +of the project's crates: `MODULE.bazel` at the Bazel root (see +[`.bazelignore`](#bazel-workspaces-bazelignore) for where that is, and for the checkouts in +which no root is looked for), or a file it includes, calls `from_cargo` on the extension +`crate` of `@rules_rust//crate_universe:extensions.bzl` (or of `extension.bzl`, its older +name), and that call + +- is a statement of its own: one in a comprehension, behind an `and` or in a condition may + never run, and is passed over; +- names the project's `Cargo.lock` as its `cargo_lockfile`, or the project's root + `Cargo.toml` among its `manifests`, by a label of your own repository that is written + out as a string (`"//:Cargo.lock"`, `"//services/api:Cargo.toml"`). + +A hub that names the files of another Cargo workspace of the checkout, or names its files +through a constant, is not taken for a hub of this project: socket-patch then does neither +of the two things, and what follows a build is what is described under "Limits". + +**Bazel is kept out of the copies.** While the project has vendored crates, the +`.bazelignore` at the Bazel root holds the line `.socket/vendor/cargo` (after the project's +path from that root, for a project below it), under the comment line +`# socket-patch: patched crate copies, not Bazel packages (to opt out, comment out the next line)`. +`scan --mode vendored` writes the two lines and `vendor --revert` takes them away; every +rule of the Go lines in [Bazel workspaces: `.bazelignore`](#bazel-workspaces-bazelignore) +holds for them: which commands bring the file in line, what covers a directory, the opt-out, +the warning `bazelignore_not_updated`. A command whose `--ecosystems` leaves `cargo` out +adds no line. What the line is for, measured with rules_rust 0.75.0 on Bazel 9.3.2 and 8.8.2 +(with rules_rust 0.56.0, on 7.7.2, a build writes no file): + +- The hub does not build the crate from the file in the checkout. It reads the copy by its + path, copies the directory into a repository of its own and renders the build file there + again. So the build with the line is the same patched one, and another file under the + same banner does not change what the crate is built from. +- Without the line the file makes the copy a package of your workspace: `bazel query //...` + lists its targets, and `bazel build //...` builds those not tagged `manual` (cargo-bazel + tags the crate's library so). That package names repositories only the hub sees, so for a + vendored crate with a dependency the commands that analyse every target fail on it: + `bazel cquery //...` and `bazel query 'deps(//...)'` end with `No repository visible as + '@crates__' from main repository`. With the line both pass, and `//...` holds + your own targets alone. + +**Under that line, the file is no change of the copy.** `vendor --check` passes, `vex` +attests the crate, `repair` has nothing to restore, and a second `scan --mode vendored` +reports the copy `already_vendored` and downloads nothing. socket-patch never writes the +file, and removes it only with the copy: on `vendor --revert`, or when it replaces a copy +that is changed in another way. Only this file is left out of the comparison with the +ledger's inventory: + +- a regular file named `BUILD.bazel` in the copy's own directory that the inventory does + not list (a build file the crate ships is listed, and is held to the crate's bytes); +- that starts with the banner of every file cargo-bazel renders: a line of `#`, the line + `# @generated`, and a line that starts with `# DO NOT MODIFY: This file is auto-generated + by a crate_universe tool.`; +- in a project whose Bazel workspace has a hub of its crates, as above; +- in a copy that a line of the root's `.bazelignore` covers: the line socket-patch writes, + or one of your own for the directory or one above it (`.socket`). + +In any other case a file like it is a changed copy, as before (`vendor_inventory_mismatch`). +Every recorded file is still held to its bytes, any other file the scan did not record is +still a changed copy, and a download that `repair` or a scan stages for a copy is held to +the inventory alone. Of the build file itself socket-patch reads the banner and nothing +after it: the rest differs from one machine and one ruleset release to the next, so no +recorded copy of it could be compared. That is why the line is part of the rule: a +`BUILD.bazel` that someone else puts into a copy under the same banner is let through only +where Bazel reads no package there, and where Bazel does read it, it is a changed copy. + +Limits: + +- With the path line commented out (the opt-out), or where the line could not be written + (`bazelignore_not_updated`), or in a checkout vendored by an earlier release until a + command that writes the line has run (`scan --mode vendored`, `repair`, `apply`), the copy + is a package again and cargo-bazel's file is a changed copy again: after the next build + those two commands fail, `vendor --check` exits 1, `vex` leaves the patch out, and the + next vendored scan downloads the crate again. +- Both parts need the Bazel root, which is looked for as the `.bazelignore` section says: + above the project's own directory only up to the root of its git checkout, and only when + that checkout's `.git` belongs to you, to root, or to the user `sudo` ran for. A project + that is a checkout of its own below the root (a submodule), or a checkout that a CI + container mounts from another user, has no Bazel root for socket-patch: both parts are + off there, and the commands above fail after a build although they pass on a developer's + machine. +- A workspace whose module files make no hub of the project's crates gets neither: no line + is written, and a `BUILD.bazel` in a copy is a changed copy. That holds for a hub of the + older `WORKSPACE` system (`crates_repository`) and for rules_rust's `crates_vendor`. By + rules_rust's source, which no build here has run, `crates_vendor` with `mode = "local"` + renders build files into the checkout that name a path crate by its package: there the + file in the copy is what Bazel builds the crate from, and a `.bazelignore` line would take + the crate out of the build. A workspace that has both such a vendor target and a hub over + the same Cargo workspace must comment the line out. +- Bazel leaves an ignored directory out of every `glob()` of a package above it. A target + that hands the project's files to a `cargo` of its own through such a glob no longer + receives the copies. From rules_rust 0.59.0 on, the build file that cargo-bazel writes + into a copy takes that copy out of the glob in the same way. +- Not covered: a crate that ships a `BUILD.bazel` of its own. By its source, which no build + here has run for such a crate, cargo-bazel writes its file over that one. The inventory + lists the crate's, so the copy reads as changed after a build, as it did before, until a + second `scan --mode vendored` or `repair` restores the crate's file. + ## Go: directory replaces and go.sum Both Go modes work through a `go.mod` `replace` directive pointing at a committed @@ -2397,8 +2503,10 @@ Side effects and limits: copy takes the copy out of the glob in the same way. With the line it builds when it names the files of the copy one by one beside the glob, or runs outside the sandbox. Such a workspace can opt out as above. -- Only Go module copies are covered. The unpacked packages other ecosystems keep under - `.socket/vendor/` get no line. +- Only Go module copies are covered, and the vendored crates of a workspace with a + rules_rust crate hub, which have a comment line of their own (see + [Bazel workspaces: the build file in a vendored crate](#bazel-workspaces-the-build-file-in-a-vendored-crate)). + The unpacked packages other ecosystems keep under `.socket/vendor/` get no line. - A Bazel root that socket-patch does not look for gets no line, and no warning is given: add the line by hand, with the directory's path from that root. That is a root above the root of the project's git checkout (the project is a git submodule, or several checkouts sit below diff --git a/scripts/loc-budget.json b/scripts/loc-budget.json index 6729c125c..82d689e84 100644 --- a/scripts/loc-budget.json +++ b/scripts/loc-budget.json @@ -4,7 +4,7 @@ "paths": [ "crates/socket-patch-core/src/bazel/" ], - "max": 2486 + "max": 2591 }, "lock-readers": { "paths": [