Skip to content

aitools: run aitools unit tests with a clean env, an empty HOME and recorded HTTP - #7872

Open
anton-107 wants to merge 1 commit into
aitools-ci-dependent-testsfrom
aitools-tests-clean-env
Open

anton-107 wants to merge 1 commit into
aitools-ci-dependent-testsfrom
aitools-tests-clean-env

Conversation

@anton-107

Copy link
Copy Markdown
Contributor

Stacked on #7848.

Changes

Adds internal/testutil/hostenv, a test helper that makes a test binary stop picking up the host's Databricks and coding-agent settings. It also catches tests that send HTTP requests outside the machine or leave files in HOME. Every aitools test package now calls it from its TestMain:

func TestMain(m *testing.M) {
	finish := hostenv.Isolate()
	os.Exit(finish(m.Run()))
}

hostenv.Isolate():

  • unsets all DATABRICKS_* variables, plus a fixed list of variables used to detect coding agents and locate agent config directories (CLAUDECODE, CLAUDE_CONFIG_DIR, CODEX_HOME, XDG_CONFIG_HOME, APPDATA, ...).
  • points HOME and USERPROFILE at a new empty temp directory.
  • sets HTTP_PROXY/HTTPS_PROXY to a local recorder that answers 502, and clears NO_PROXY/no_proxy. Go does not proxy loopback addresses, so httptest servers still work.

The function it returns removes the temp HOME. If a request went through the recorder or a test left files in HOME, it prints what it found and changes a zero exit code to 1.

What it does not cover: only HTTP clients that take their proxy from the environment (as http.DefaultTransport does) are routed to the recorder. Clients with their own proxy settings and non-HTTP connections are not. PATH is unchanged, so host binaries can still be found, except in libs/aitools/mcp/agentconfig, whose TestMain keeps its agents.LookPath override. Files outside HOME are not checked.

Packages that now call it: cmd/aitools, experimental/aitools/cmd, experimental/aitools/lib/{middlewares,session}, libs/aitools/{agents,autoinstall,installer,mcp,mcp/agentconfig,mcp/proxy,mcpconfig}. TestAitoolsPackagesIsolate fails if a directory with tests under cmd/aitools, libs/aitools or experimental/aitools has no hostenv.Isolate() call in its test files.

Tests that sent requests through the recorder now stay local:

  • cmd/aitools sets DATABRICKS_FORCE_EMBEDDED_COMPAT=true in TestMain, so commands use the embedded compatibility manifest and don't fetch it from GitHub. TestUpdateWithoutStateKeepsTheInstallGuidance clears that variable, so it pins DATABRICKS_SKILLS_REF instead.
  • TestInstallKeepsNameWhenRepoDirChanges pins DATABRICKS_SKILLS_REF, so the install doesn't look up the latest release on GitHub.
  • Two mcp serve challenge tests build their .test workspace client with a host-metadata resolver that returns no metadata. The libs/aitools/mcp binding tests use a resolver that returns no metadata for .test hosts and still asks a test's own server. TestServeBindHostOnlyProfileNeedsLogin points its profile at a local server that returns 404 for everything.
  • libs/aitools/mcp/agentconfig moves its TestMain into main_test.go.

internal/testutil/hostenv/** is added to the test-exp-aitools sources and packages, so a change to the helper runs the aitools job. A tools/testmask case covers that.

Why

The aitools tests read agent config dirs, HOME and ~/.databrickscfg, and some of them build SDK clients. On a clean CI runner, a test that forgets to set up its own environment still passes. On a developer's machine, or inside a coding agent, the same test can pick up the real agent setup and profiles, or send requests to real services. Failed requests and stray writes were also easy to miss, because the test that caused them often passed anyway. Clearing those inputs in every aitools test binary, and failing the binary when a test sends a request out or writes to HOME, catches these problems on every run.

Tests

  • New unit tests in internal/testutil/hostenv check that the variables are cleared and HOME starts empty. They also check that requests (plain HTTP and CONNECT) and leftover HOME files are reported and that the exit code is set. A re-executed test binary checks that an inherited NO_PROXY=* is ignored. Another test checks that every aitools test package calls hostenv.Isolate().
  • task test-exp-aitools passes with every aitools package calling hostenv.Isolate().

…ecorded HTTP

Stacked on #7848.

## Changes

Adds `internal/testutil/hostenv`, a test helper that makes a test binary stop picking up the host's Databricks and coding-agent settings. It also catches tests that send HTTP requests outside the machine or leave files in HOME. Every aitools test package now calls it from its `TestMain`:

```go
func TestMain(m *testing.M) {
	finish := hostenv.Isolate()
	os.Exit(finish(m.Run()))
}
```

`hostenv.Isolate()`:

- unsets all `DATABRICKS_*` variables, plus a fixed list of variables used to detect coding agents and locate agent config directories (`CLAUDECODE`, `CLAUDE_CONFIG_DIR`, `CODEX_HOME`, `XDG_CONFIG_HOME`, `APPDATA`, ...).
- points `HOME` and `USERPROFILE` at a new empty temp directory.
- sets `HTTP_PROXY`/`HTTPS_PROXY` to a local recorder that answers 502, and clears `NO_PROXY`/`no_proxy`. Go does not proxy loopback addresses, so `httptest` servers still work.

The function it returns removes the temp HOME. If a request went through the recorder or a test left files in HOME, it prints what it found and changes a zero exit code to 1.

What it does not cover: only HTTP clients that take their proxy from the environment (as `http.DefaultTransport` does) are routed to the recorder. Clients with their own proxy settings and non-HTTP connections are not. `PATH` is unchanged, so host binaries can still be found, except in `libs/aitools/mcp/agentconfig`, whose `TestMain` keeps its `agents.LookPath` override. Files outside HOME are not checked.

Packages that now call it: `cmd/aitools`, `experimental/aitools/cmd`, `experimental/aitools/lib/{middlewares,session}`, `libs/aitools/{agents,autoinstall,installer,mcp,mcp/agentconfig,mcp/proxy,mcpconfig}`. `TestAitoolsPackagesIsolate` fails if a directory with tests under `cmd/aitools`, `libs/aitools` or `experimental/aitools` has no `hostenv.Isolate()` call in its test files.

Tests that sent requests through the recorder now stay local:

- `cmd/aitools` sets `DATABRICKS_FORCE_EMBEDDED_COMPAT=true` in `TestMain`, so commands use the embedded compatibility manifest and don't fetch it from GitHub. `TestUpdateWithoutStateKeepsTheInstallGuidance` clears that variable, so it pins `DATABRICKS_SKILLS_REF` instead.
- `TestInstallKeepsNameWhenRepoDirChanges` pins `DATABRICKS_SKILLS_REF`, so the install doesn't look up the latest release on GitHub.
- Two `mcp serve` challenge tests build their `.test` workspace client with a host-metadata resolver that returns no metadata. The `libs/aitools/mcp` binding tests use a resolver that returns no metadata for `.test` hosts and still asks a test's own server. `TestServeBindHostOnlyProfileNeedsLogin` points its profile at a local server that returns 404 for everything.
- `libs/aitools/mcp/agentconfig` moves its `TestMain` into `main_test.go`.

`internal/testutil/hostenv/**` is added to the `test-exp-aitools` sources and packages, so a change to the helper runs the aitools job. A `tools/testmask` case covers that.

## Why

The aitools tests read agent config dirs, `HOME` and `~/.databrickscfg`, and some of them build SDK clients. On a clean CI runner, a test that forgets to set up its own environment still passes. On a developer's machine, or inside a coding agent, the same test can pick up the real agent setup and profiles, or send requests to real services. Failed requests and stray writes were also easy to miss, because the test that caused them often passed anyway. Clearing those inputs in every aitools test binary, and failing the binary when a test sends a request out or writes to HOME, catches these problems on every run.

## Tests

- New unit tests in `internal/testutil/hostenv` check that the variables are cleared and HOME starts empty. They also check that requests (plain HTTP and `CONNECT`) and leftover HOME files are reported and that the exit code is set. A re-executed test binary checks that an inherited `NO_PROXY=*` is ignored. Another test checks that every aitools test package calls `hostenv.Isolate()`.
- `task test-exp-aitools` passes with every aitools package calling `hostenv.Isolate()`.

Co-authored-by: Isaac <no-reply@databricks.com>
@anton-107
anton-107 requested review from a team and lennartkats-db as code owners November 21, 2026 10:30

@review-oss-bot review-oss-bot Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Automated review at 7a7a37a. Each finding was verified against the code.

Not ready yet: 1 blocking issue(s), tagged [blocking].

This PR adds internal/testutil/hostenv, a TestMain-level isolation helper that unsets agent and Databricks environment variables, redirects HOME, and routes stray HTTP through a local recorder. The approach is sound, but the agent-detection variable list is already out of sync with its stated reference before merging.

"GEMINI_CLI_HOME",
"GOOSE_PATH_ROOT",
"GOOSE_TERMINAL",
"KIRO",

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[blocking] OMNIGENT is in acceptance/acceptance_test.go:279 but missing here, so the two lists are already out of sync on arrival — the comment on line 21 says "keep in sync with the list in acceptance/acceptance_test.go". Running aitools tests inside an Omnigent session would leak the variable through, letting the SDK's user-agent pick up the agent marker that Isolate is meant to strip. Can you add it between KIRO and OPENCLAW_SHELL?

@eng-dev-ecosystem-bot

Copy link
Copy Markdown
Collaborator

✅ Integration test report

Commit: 7a7a37a

Run: 44894755381

Env 💚​RECOVERED ✅​pass 🙈​skip Time
💚​ aws linux-2core-8gb 1 272 19 3:46
💚​ aws-windows-latest-4core-16gb 1 274 17 4:17
💚​ azure linux-2core-8gb 1 271 19 3:46
💚​ azure-windows-latest-4core-16gb 1 273 17 4:04
💚​ gcp linux-2core-8gb 1 272 19 3:49
💚​ gcp-windows-latest-4core-16gb 1 274 17 3:59
Test Name aws linux-2core-8gb aws-windows-latest-4core-16gb azure linux-2core-8gb azure-windows-latest-4core-16gb gcp linux-2core-8gb gcp-windows-latest-4core-16gb
💚​ TestAccept 💚​R 💚​R 💚​R 💚​R 💚​R 💚​R
Top 3 slowest tests (at least 2 minutes):
duration env testname
4:16 aws-windows-latest-4core-16gb TestAccept
4:03 azure-windows-latest-4core-16gb TestAccept
3:58 gcp-windows-latest-4core-16gb TestAccept

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants