Build a Safari WebDriver Regression Gate on a Cloud Mac

After a remote team merges web changes, the most easily overlooked issues are not related to the Chromium path, but to Safari-specific form, focus, and layout behavior. Instead of opening a remote desktop before every release and clicking through pages manually, keep a fixed Safari WebDriver regression suite on a GPUMini cloud Mac. Run the same critical flows against every release candidate and preserve enough page context to investigate each failure.

Establish a Repeatable Execution Boundary First

Safari automation depends on a macOS GUI session, so it cannot simply reuse the execution model of a headless browser. Create a dedicated test user, keep its GUI session logged in, and ensure that automated jobs never overlap with interactive remote access. Run only one Safari session at a time within the same user profile to prevent browser history, downloaded files, and window state from contaminating one another.

When preparing a node for the first time, enable and diagnose the driver from an interactive terminal:

sudo safaridriver --enable
safaridriver --diagnose
mkdir -p "$HOME/browser-tests/artifacts"

--enable is a node provisioning step and should not be included in every automated job. Routine jobs should run only the diagnostic command. If diagnostics fail, stop the test instead of producing a batch of uninformative timeout errors. The Python environment should also pin dependency versions. Record a specific version of selenium in the project's dependency file rather than installing the latest release on every run.

The first baseline for browser regression testing is not merely that the browser starts, but that the same commit, test data, and wait conditions produce the same conclusion.

Isolate External Variability with Local Fixtures

End-to-end flows such as sign-in, search, and checkout can use a test environment, but component interactions and browser compatibility checks should rely on local fixtures whenever possible. Keep pages for forms, dialogs, uploads, and keyboard navigation in the repository and serve them over the loopback interface. This removes variability caused by DNS, certificates, and external APIs.

set -euo pipefail

ROOT="$(cd "$(dirname "$0")" && pwd)"
ARTIFACTS="$ROOT/artifacts"
mkdir -p "$ARTIFACTS"

python3 -m http.server 8080 \
  --bind 127.0.0.1 \
  --directory "$ROOT/fixtures" \
  >"$ARTIFACTS/http-server.log" 2>&1 &
SERVER_PID=$!

cleanup() {
  kill "$SERVER_PID" 2>/dev/null || true
}
trap cleanup EXIT INT TERM

"$ROOT/.venv/bin/python" "$ROOT/tests/safari_smoke.py"

Before reserving a fixed port, check whether it is already in use with lsof -nP -iTCP:8080 -sTCP:LISTEN. If several jobs share a node, do not kill processes indiscriminately. Either enforce mutual exclusion at the scheduler level or assign an explicit port to each job and record it in the test configuration.

Express Wait Conditions as Business States

One of the most common brittle patterns is sleeping for a fixed number of seconds after a click. Even a small change in machine load can make a short sleep produce false failures, while an excessively long sleep slows the entire suite. Wait for an observable business state instead, such as a button becoming clickable, a result list appearing, or status text changing to the expected value.

from pathlib import Path
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC

artifacts = Path("artifacts")
driver = webdriver.Safari()
driver.set_page_load_timeout(20)

try:
    driver.get("http://127.0.0.1:8080/login.html")
    wait = WebDriverWait(driver, 12)

    wait.until(
        EC.visibility_of_element_located((By.ID, "email"))
    ).send_keys("tester@local.invalid")

    driver.find_element(By.ID, "continue").click()

    status = wait.until(
        EC.visibility_of_element_located((By.ID, "result"))
    )
    assert status.text == "Ready"
finally:
    driver.quit()

Prefer stable id values or dedicated test attributes for locators. Avoid hierarchical selectors that change whenever the layout is adjusted. Assertions should verify the outcome rather than merely confirm that an element exists. The presence of a submit button does not mean submission succeeded, and the presence of a list container does not mean its data has rendered.

Make Every Failure Reproducible from Its Evidence

The most expensive browser test result is “CI timed out, but there is no context.” Whenever a test case fails, immediately save a screenshot, the page source, the current URL, the test case name, and timing information. Waiting until the entire suite finishes before taking a screenshot often captures only a page that has already navigated elsewhere or closed.

Screenshot filenames should include a test case identifier and a unique run number, but they must not contain tokens, email addresses, or project secrets. Page source may also include test data, so redact it before archiving and define an explicit retention period.

Persist Evidence Immediately at the Point of Failure

The test framework's failure hook should call a shared evidence-capture function. Because the save operation itself can fail, screenshot and page-source write errors must be handled separately. An archival error must never overwrite the original assertion failure.

Recover Leftover Sessions Instead of Retrying Blindly

When a job is interrupted, Safari or its driver process may continue holding the session. Retrying immediately on the next run often results in repeated session creation failures. First check for running jobs owned by the test user and confirm that no other valid test is active before terminating a leftover session. Never perform broad process-name-based cleanup on a shared node.

Retries are appropriate only for known transient session creation failures, and then only once. Assertion failures, page structure changes, and mismatched business states must not be retried automatically, because doing so can disguise a real regression as an intermittent issue. If a single retry succeeds, record it separately as an instability event rather than marking the job as entirely healthy.

Finally, split the gate into two layers: run local-fixture smoke tests on every commit, then run a small set of critical end-to-end flows against each release candidate. Before release, verify that driver diagnostics pass, the test user has exclusive access, the fixture service is reachable, explicit waits contain no fixed sleeps, failure evidence is readable, and the cleanup function always runs. The result is not a set of browser scripts that “occasionally pass,” but a Safari regression baseline that can track changes reliably over time.

Frequently asked questions