Repository navigation
Conversation
…ecorded HTTP Stacked on #7848. ## Changes Adds `internal/testutil/hostenv`, a test helper that makes a test binary stop picking up the host's Databricks and coding-agent settings. It also catches tests that send HTTP requests outside the machine or leave files in HOME. Every aitools test package now calls it from its `TestMain`: ```go func TestMain(m *testing.M) { finish := hostenv.Isolate() os.Exit(finish(m.Run())) } ``` `hostenv.Isolate()`: - unsets all `DATABRICKS_*` variables, plus a fixed list of variables used to detect coding agents and locate agent config directories (`CLAUDECODE`, `CLAUDE_CONFIG_DIR`, `CODEX_HOME`, `XDG_CONFIG_HOME`, `APPDATA`, ...). - points `HOME` and `USERPROFILE` at a new empty temp directory. - sets `HTTP_PROXY`/`HTTPS_PROXY` to a local recorder that answers 502, and clears `NO_PROXY`/`no_proxy`. Go does not proxy loopback addresses, so `httptest` servers still work. The function it returns removes the temp HOME. If a request went through the recorder or a test left files in HOME, it prints what it found and changes a zero exit code to 1. What it does not cover: only HTTP clients that take their proxy from the environment (as `http.DefaultTransport` does) are routed to the recorder. Clients with their own proxy settings and non-HTTP connections are not. `PATH` is unchanged, so host binaries can still be found, except in `libs/aitools/mcp/agentconfig`, whose `TestMain` keeps its `agents.LookPath` override. Files outside HOME are not checked. Packages that now call it: `cmd/aitools`, `experimental/aitools/cmd`, `experimental/aitools/lib/{middlewares,session}`, `libs/aitools/{agents,autoinstall,installer,mcp,mcp/agentconfig,mcp/proxy,mcpconfig}`. `TestAitoolsPackagesIsolate` fails if a directory with tests under `cmd/aitools`, `libs/aitools` or `experimental/aitools` has no `hostenv.Isolate()` call in its test files. Tests that sent requests through the recorder now stay local: - `cmd/aitools` sets `DATABRICKS_FORCE_EMBEDDED_COMPAT=true` in `TestMain`, so commands use the embedded compatibility manifest and don't fetch it from GitHub. `TestUpdateWithoutStateKeepsTheInstallGuidance` clears that variable, so it pins `DATABRICKS_SKILLS_REF` instead. - `TestInstallKeepsNameWhenRepoDirChanges` pins `DATABRICKS_SKILLS_REF`, so the install doesn't look up the latest release on GitHub. - Two `mcp serve` challenge tests build their `.test` workspace client with a host-metadata resolver that returns no metadata. The `libs/aitools/mcp` binding tests use a resolver that returns no metadata for `.test` hosts and still asks a test's own server. `TestServeBindHostOnlyProfileNeedsLogin` points its profile at a local server that returns 404 for everything. - `libs/aitools/mcp/agentconfig` moves its `TestMain` into `main_test.go`. `internal/testutil/hostenv/**` is added to the `test-exp-aitools` sources and packages, so a change to the helper runs the aitools job. A `tools/testmask` case covers that. ## Why The aitools tests read agent config dirs, `HOME` and `~/.databrickscfg`, and some of them build SDK clients. On a clean CI runner, a test that forgets to set up its own environment still passes. On a developer's machine, or inside a coding agent, the same test can pick up the real agent setup and profiles, or send requests to real services. Failed requests and stray writes were also easy to miss, because the test that caused them often passed anyway. Clearing those inputs in every aitools test binary, and failing the binary when a test sends a request out or writes to HOME, catches these problems on every run. ## Tests - New unit tests in `internal/testutil/hostenv` check that the variables are cleared and HOME starts empty. They also check that requests (plain HTTP and `CONNECT`) and leftover HOME files are reported and that the exit code is set. A re-executed test binary checks that an inherited `NO_PROXY=*` is ignored. Another test checks that every aitools test package calls `hostenv.Isolate()`. - `task test-exp-aitools` passes with every aitools package calling `hostenv.Isolate()`. Co-authored-by: Isaac <no-reply@databricks.com>
There was a problem hiding this comment.
Automated review at 7a7a37a. Each finding was verified against the code.
Not ready yet: 1 blocking issue(s), tagged [blocking].
This PR adds internal/testutil/hostenv, a TestMain-level isolation helper that unsets agent and Databricks environment variables, redirects HOME, and routes stray HTTP through a local recorder. The approach is sound, but the agent-detection variable list is already out of sync with its stated reference before merging.
| "GEMINI_CLI_HOME", | ||
| "GOOSE_PATH_ROOT", | ||
| "GOOSE_TERMINAL", | ||
| "KIRO", |
There was a problem hiding this comment.
[blocking] OMNIGENT is in acceptance/acceptance_test.go:279 but missing here, so the two lists are already out of sync on arrival — the comment on line 21 says "keep in sync with the list in acceptance/acceptance_test.go". Running aitools tests inside an Omnigent session would leak the variable through, letting the SDK's user-agent pick up the agent marker that Isolate is meant to strip. Can you add it between KIRO and OPENCLAW_SHELL?
✅ Integration test reportCommit: 7a7a37a
Top 3 slowest tests (at least 2 minutes):
|
Stacked on #7848.
Changes
Adds
internal/testutil/hostenv, a test helper that makes a test binary stop picking up the host's Databricks and coding-agent settings. It also catches tests that send HTTP requests outside the machine or leave files in HOME. Every aitools test package now calls it from itsTestMain:hostenv.Isolate():DATABRICKS_*variables, plus a fixed list of variables used to detect coding agents and locate agent config directories (CLAUDECODE,CLAUDE_CONFIG_DIR,CODEX_HOME,XDG_CONFIG_HOME,APPDATA, ...).HOMEandUSERPROFILEat a new empty temp directory.HTTP_PROXY/HTTPS_PROXYto a local recorder that answers 502, and clearsNO_PROXY/no_proxy. Go does not proxy loopback addresses, sohttptestservers still work.The function it returns removes the temp HOME. If a request went through the recorder or a test left files in HOME, it prints what it found and changes a zero exit code to 1.
What it does not cover: only HTTP clients that take their proxy from the environment (as
http.DefaultTransportdoes) are routed to the recorder. Clients with their own proxy settings and non-HTTP connections are not.PATHis unchanged, so host binaries can still be found, except inlibs/aitools/mcp/agentconfig, whoseTestMainkeeps itsagents.LookPathoverride. Files outside HOME are not checked.Packages that now call it:
cmd/aitools,experimental/aitools/cmd,experimental/aitools/lib/{middlewares,session},libs/aitools/{agents,autoinstall,installer,mcp,mcp/agentconfig,mcp/proxy,mcpconfig}.TestAitoolsPackagesIsolatefails if a directory with tests undercmd/aitools,libs/aitoolsorexperimental/aitoolshas nohostenv.Isolate()call in its test files.Tests that sent requests through the recorder now stay local:
cmd/aitoolssetsDATABRICKS_FORCE_EMBEDDED_COMPAT=trueinTestMain, so commands use the embedded compatibility manifest and don't fetch it from GitHub.TestUpdateWithoutStateKeepsTheInstallGuidanceclears that variable, so it pinsDATABRICKS_SKILLS_REFinstead.TestInstallKeepsNameWhenRepoDirChangespinsDATABRICKS_SKILLS_REF, so the install doesn't look up the latest release on GitHub.mcp servechallenge tests build their.testworkspace client with a host-metadata resolver that returns no metadata. Thelibs/aitools/mcpbinding tests use a resolver that returns no metadata for.testhosts and still asks a test's own server.TestServeBindHostOnlyProfileNeedsLoginpoints its profile at a local server that returns 404 for everything.libs/aitools/mcp/agentconfigmoves itsTestMainintomain_test.go.internal/testutil/hostenv/**is added to thetest-exp-aitoolssources and packages, so a change to the helper runs the aitools job. Atools/testmaskcase covers that.Why
The aitools tests read agent config dirs,
HOMEand~/.databrickscfg, and some of them build SDK clients. On a clean CI runner, a test that forgets to set up its own environment still passes. On a developer's machine, or inside a coding agent, the same test can pick up the real agent setup and profiles, or send requests to real services. Failed requests and stray writes were also easy to miss, because the test that caused them often passed anyway. Clearing those inputs in every aitools test binary, and failing the binary when a test sends a request out or writes to HOME, catches these problems on every run.Tests
internal/testutil/hostenvcheck that the variables are cleared and HOME starts empty. They also check that requests (plain HTTP andCONNECT) and leftover HOME files are reported and that the exit code is set. A re-executed test binary checks that an inheritedNO_PROXY=*is ignored. Another test checks that every aitools test package callshostenv.Isolate().task test-exp-aitoolspasses with every aitools package callinghostenv.Isolate().