stealth

func NewClient ¶

func NewClient(opts ...ClientOption) (*BrowserClient, error)

NewClient creates a BrowserClient with the given options.

func PlatformMatchedProfile ¶

func PlatformMatchedProfile() BrowserProfile

PlatformMatchedProfile returns a BrowserProfile whose OS matches the actual runtime platform (runtime.GOOS).

func RandomProfile ¶

func RandomProfile(opts ...ProfileOption) BrowserProfile

RandomProfile returns a random BrowserProfile matching the given filters. With no options, returns any profile.

When no profile satisfies every filter it never substitutes an unrelated browser: if a browser filter was given, the OS and mobile filters are relaxed (the browser is kept, e.g. WithBrowser("safari")+WithOS("macos") yields an iOS Safari profile); if that still matches nothing, or no browser filter was given, it returns the zero BrowserProfile, whose UserAgent is "". Callers that filter must check for that.

func NewByparrSolver ¶

func NewByparrSolver(cfg ByparrConfig) *ByparrSolver

NewByparrSolver creates a CookieProvider that calls Byparr/FlareSolverr.

func WithBackend ¶

func WithBackend(factory BackendFactory) ClientOption

WithBackend sets a custom backend factory for creating the HTTPDoer. If not set, the default bogdanfinn/tls-client backend is used.

func WithCookieJar ¶ added in v1.28.0

func WithCookieJar(jar http.CookieJar) ClientOption

WithCookieJar makes the client store and send cookies through jar instead of its own private jar. Pass the same jar to several clients to share one cookie store, as the tabs and workers of one browser do: a cookie set by a response to one client is sent by the next request of another.

The jar decides everything about a cookie: which domain and path it is scoped to, expiry and deletion (a cookie set with MaxAge < 0 is removed), and the order cookies are sent in. net/http/cookiejar orders them as RFC 6265 does and as Chrome does: longer path first, then earliest creation, so cookies from one response keep their Set-Cookie order. The caller can seed, read or delete cookies by calling the jar directly, which lets it drop an explicit Cookie header (a Cookie header set on a request is sent in addition to the jar's cookies, so a cookie in both is sent twice).

The jar must be safe for concurrent use (net/http/cookiejar is). A nil jar keeps the default per-client jar. The tls-client default jar is not changed: it keys cookies by the last two host labels, ignores Domain and Path, and returns one response's cookies in reverse Set-Cookie order. Works with both backends and does not touch the TLS or HTTP/2 wire.

func WithCookieSolver ¶

func WithCookieSolver(provider CookieProvider) ClientOption

WithCookieSolver enables Cloudflare challenge solving via a CookieProvider. Automatically adds CloudflareDetectMiddleware and CloudflareCookieMiddleware.

func WithDebug ¶

func WithDebug() ClientOption

WithDebug enables request/response logging via slog.Debug. Automatically adds LoggingMiddleware to the middleware chain.

func WithDialControl ¶

func WithDialControl(fn func(network, address string) error) ClientOption

WithDialControl overrides the connect-time (tier-1) SSRF guard: the rebind-proof check run on the already-resolved address immediately before connect(2), on both backends' dialers. Pass go-kit/httputil's SSRFGuards() dial closure (or DenyBlockedAddress) here for the framework's full policy. A nil fn disables the connect-time guard.

On a PROXIED client this hook sees only the proxy's address, never the real target — pair it with WithRequestURLGuard (tier 3), which does. The client passes the configured proxy's own endpoint (its resolved addresses and port, following SetProxy) without consulting fn, so a proxy on a private address stays reachable; every other address still goes through fn. The consequence: a stricter caller guard is bypassed for those proxy endpoints (every proxy set on this client recently, at most the last 64 distinct URLs), because they are operator configuration.

func WithFetchMode ¶ added in v1.27.0

func WithFetchMode() ClientOption

WithFetchMode makes the client behave like a Chrome page issuing fetch() and XMLHttpRequest calls rather than top-level navigations (the default). Two things change, and both match what Chrome sends for such requests:

• HTTP/2: the HEADERS frame carries exclusive priority weight 220 instead of the navigation weight 256.
• Headers: a request without its own "priority" header gets "priority: u=1, i". A priority header the caller sets is kept as is.

The mode belongs to the whole client because HTTP/2 priority is a property of the connection's transport, not of a single request; a caller that mixes navigations and fetches builds one client of each (they then use separate connections, which Chrome does not). The cookie jar is per client.

It needs a Chrome-family TLS profile (chrome_*, brave_*) and a Chromium identity (chrome, edge, brave); on Firefox or Safari, or an identity that names one of them, NewClient returns ErrFetchModeUnsupported rather than present a Chrome request shape over another browser's fingerprint. It is opt-in for every profile, chrome_131 included: without it the wire is unchanged. With WithStdHTTP only the header part applies (net/http cannot set a priority).

func WithFollowRedirects ¶

func WithFollowRedirects() ClientOption

WithFollowRedirects enables redirect following (disabled by default).

func WithHTTP3 ¶

func WithHTTP3() ClientOption

WithHTTP3 enables HTTP/3 QUIC support (tls-client backend only).

func WithHeaderOrder ¶

func WithHeaderOrder(order []string) ClientOption

WithHeaderOrder sets the default HTTP header ordering for requests.

func WithIdentity ¶ added in v1.21.0

func WithIdentity(id BrowserIdentity) ClientOption

WithIdentity sets the browser identity — TLS profile and User-Agent (plus Client Hints) — together, so they can never be configured apart and drift. The identity's TLSProfile is installed as the backend's fingerprint, and Identity() returns the supplied identity verbatim. If ClientHints is nil for a Chromium UA, it is derived from the User-Agent via ClientHintsHeaders so the identity the client reports is always complete.

WithIdentity and WithProfile may both be passed; WithIdentity wins for the profile (it sets c.profile too) and supplies the UA, which WithProfile alone cannot do. Use WithIdentity when the caller already holds a BrowserProfile from RandomProfile/PlatformMatchedProfile and wants the client's reported identity to match exactly.

func WithInsecureSkipVerify ¶

func WithInsecureSkipVerify() ClientOption

WithInsecureSkipVerify disables TLS certificate verification. WARNING: this makes connections vulnerable to man-in-the-middle attacks. Use only for local testing (httptest.NewTLSServer) or explicit MITM proxy inspection. Never use in production against public endpoints.

func WithOxBrowser ¶

func WithOxBrowser(url string) ClientOption

WithOxBrowser enables ox-browser integration for CF solving and smart fetch. If no CookieProvider is already set, adds OxBrowserSolver + CloudflareDetectMiddleware. Always adds SmartFetchMiddleware as a fallback for CF-challenged responses. url is the ox-browser base URL (e.g. "http://127.0.0.1:8901"). Calls to ox-browser carry X-Internal-Secret from INTERNAL_SERVICE_SECRET when set — unless a proxy pool routes them through an external proxy, where the header would transit the proxy and is not sent.

func WithProfile ¶

func WithProfile(p TLSProfile) ClientOption

WithProfile sets the TLS client profile for fingerprint impersonation. The zero value "" means "no preference" and keeps the default profile; a non-empty name with no backend mapping makes NewClient fail with ErrUnknownTLSProfile.

func WithProxy ¶

func WithProxy(url string) ClientOption

WithProxy sets the proxy URL (e.g. "socks5://user:pass@host:port").

func WithProxyPool ¶

func WithProxyPool(pool ProxyPoolProvider) ClientOption

WithProxyPool enables per-request proxy rotation. Each call to Do() will cycle to the next proxy in the pool.

func WithRedirectGuard ¶

func WithRedirectGuard(fn func(req *http.Request, via []*http.Request) error) ClientOption

WithRedirectGuard overrides the per-hop (tier-2) SSRF guard: a CheckRedirect-shaped closure run on every redirect hop the backend follows (only relevant with WithFollowRedirects). It MUST re-own a hop cap — a custom CheckRedirect replaces net/http's built-in 10-hop limit. Pass go-kit/httputil's SSRFGuards() redirect closure here. A nil fn disables the per-hop guard (the backend then falls back to its built-in redirect cap).

On the std backend, req and via are the exact *http.Request values net/http's own redirect loop uses (Body/Cancel/ctx and all). On the tls backend, req and via are adapted from bogdanfinn/fhttp's redirect chain: Method, URL, and Header are populated faithfully hop-by-hop (fhttp builds its own via chain from real prior requests, same shape as net/http's), but Body/GetBody/Cancel/Context and any other *http.Request field are left at their zero value — a guard that needs only len(via)/via[i].URL/via[i].Host (the go-kit SSRFGuards() shape) works unchanged on both backends; a guard reading Body or Context should not assume either backend populates them.

func WithRequestURLGuard ¶

func WithRequestURLGuard(fn func(ctx context.Context, u *url.URL) error) ClientOption

WithRequestURLGuard overrides the pre-request (tier-3) SSRF guard: a check on the INITIAL target URL, evaluated before the (possibly proxied) fetch. This is the ONLY tier that guards a proxied fetch's initial target, since a proxied dial control (tier 1) sees only the proxy. Signature matches go-kit/httputil.CheckURL, so a consumer can wire it directly. A nil fn disables the pre-request guard.

func WithRetryOnBlock ¶

func WithRetryOnBlock(n int) ClientOption

WithRetryOnBlock enables automatic retry with proxy rotation when the server returns a block status (403, 429). Each retry uses the next proxy from the pool. Requires WithProxyPool. n is the number of extra retries (total attempts = 1 + n).

func WithStdHTTP ¶

func WithStdHTTP() ClientOption

WithStdHTTP uses the standard net/http backend instead of tls-client. No TLS fingerprinting — useful for testing and CGO-free environments.

func WithTimeout ¶

func WithTimeout(seconds int) ClientOption

WithTimeout sets the request timeout in seconds.

func WithWebshareCountry ¶

func WithWebshareCountry(apiKey string, countries ...string) ClientOption

WithWebshareCountry creates a Webshare backbone proxy pool targeting the given ISO-2 country codes (e.g. "US", "GB"). Defaults to ["US"] if no countries are given. The API call is made eagerly; if it fails, NewClient returns the error.

The client owns the pool this option builds: call BrowserClient.Close to stop the pool's periodic credential refresher when the client is done.

func WithWebshareRotating ¶

func WithWebshareRotating(username, password string, countries ...string) ClientOption

WithWebshareRotating creates a Webshare rotating-endpoint proxy pool without calling the Webshare API. Uses username-CC-rotate syntax. Defaults to ["US"] when no countries are given.

The client owns the pool this option builds (BrowserClient.Close releases it); a rotating pool runs no refresher, so closing it is a no-op today.

func WithoutSSRFGuard ¶

func WithoutSSRFGuard() ClientOption

WithoutSSRFGuard clears all three SSRF guard tiers (dial, redirect, pre-request), restoring the pre-guard byte-for-byte behavior.

FOR TESTS ONLY. A BrowserClient is fail-closed by construction — its built-in stdlib default-deny refuses loopback/private/link-local targets, which is exactly what an httptest (127.0.0.1) suite needs to opt out of. Never call this from production code; the fleet fitness function forbids it outside _test.go.

func DetectCloudflare ¶

func DetectCloudflare(resp *Response) *CloudflareError

DetectCloudflare inspects a Response for Cloudflare challenge markers. Returns nil if the response is not a Cloudflare challenge.

Signal precedence (under the server~cloudflare gate):

1. Block markers (any status) — a block page that also mentions challenge-platform or carries cf-mitigated must stay a block, not be promoted to a solvable challenge.
2. cf-mitigated: challenge header (any status) — the documented, status-independent challenge-page signal. Top precedence among challenge signals.
3. Body markers (any status): turnstile, then challenge-platform/_cf_chl_opt.

Status is a kind hint, not a gate: 200 → ChallengeManagedAt200, otherwise ChallengeJS. The fix is additive (recognise more challenges), never promotive (a 403 WAF block carrying neither cf-mitigated nor block/body markers returns nil — promoting unknown 403s would burn solver time on hard blocks).

Cloudflare moved the JS/managed-challenge status from 503 to 403 on 2023-03-01; cf-mitigated: challenge is set on all challenge-page types regardless of status and is the only valid value for that header.

• https://developers.cloudflare.com/cloudflare-challenges/challenge-types/challenge-pages/detect-response/
• https://community.cloudflare.com/t/community-update-status-code-for-javascript-challenge-changing-from-503-to-403/445724
• https://developers.cloudflare.com/rules/custom-errors/reference/error-page-types/

func ClientHintsMiddleware ¶

func ClientHintsMiddleware(next Handler) Handler

ClientHintsMiddleware auto-injects sec-ch-ua-* headers for Chromium-based profiles. This ensures Client Hints match the User-Agent, preventing fingerprint mismatch detection.

func CloudflareDetectMiddleware ¶

func CloudflareDetectMiddleware(next Handler) Handler

CloudflareDetectMiddleware inspects responses for Cloudflare challenges. If a challenge is detected, it returns a *CloudflareError (use errors.As to extract). Non-challenge responses pass through unchanged.

func LoggingMiddleware ¶

func LoggingMiddleware(next Handler) Handler

LoggingMiddleware logs each request's method, URL, status, and latency via slog.

func Chain ¶

func Chain(middlewares ...Middleware) Middleware

Chain composes multiple middlewares into a single Middleware. Middlewares are applied in order: first middleware wraps the outermost layer.

Chain(logging, retry, rateLimit)(handler)
// execution order: logging → retry → rateLimit → handler

func CloudflareCookieMiddleware ¶

func CloudflareCookieMiddleware(provider CookieProvider) Middleware

CloudflareCookieMiddleware returns a middleware that:

1. Injects cached cf_clearance cookies from the provider before each request.
2. On Cloudflare challenge response, calls provider.Solve() to get a cookie and retries once.

func RateLimitMiddleware ¶

func RateLimitMiddleware(limiter *ratelimit.DomainLimiter) Middleware

RateLimitMiddleware returns a middleware that enforces per-domain rate limits. Blocks until the request is allowed or context deadline is exceeded.

func RateLimitMiddlewareWithContext ¶

func RateLimitMiddlewareWithContext(ctx context.Context, limiter *ratelimit.DomainLimiter) Middleware

RateLimitMiddlewareWithContext returns a rate limit middleware that respects context cancellation.

func RetryMiddleware ¶

func RetryMiddleware(cfg RetryConfig) Middleware

RetryMiddleware returns a middleware that retries failed requests using the given config.

func RetryMiddlewareWithContext ¶

func RetryMiddlewareWithContext(ctx context.Context, cfg RetryConfig) Middleware

RetryMiddlewareWithContext returns a retry middleware that respects context cancellation.

func SmartFetchMiddleware ¶

func SmartFetchMiddleware(oxClient *OxBrowserClient) Middleware

SmartFetchMiddleware returns a middleware that detects Cloudflare challenges and delegates to ox-browser /fetch-smart for automatic solving. On CF detection, the original response is replaced with ox-browser's solved response. Non-CF responses pass through unchanged.

func NewOxBrowserClient ¶

func NewOxBrowserClient(baseURL string) *OxBrowserClient

NewOxBrowserClient creates a client for ox-browser at the given base URL. Does NOT route through proxy. Use NewOxBrowserClientWithProxy for stealth scenarios. Requests to ox-browser carry the X-Internal-Secret header from INTERNAL_SERVICE_SECRET when that env var is set (go-kit svcauth); the header is scoped to baseURL's origin and stripped on requests and redirect hops to anywhere else.

func NewOxBrowserClientWithProxy ¶

func NewOxBrowserClientWithProxy(baseURL string, proxyFn func(*http.Request) (*url.URL, error)) *OxBrowserClient

NewOxBrowserClientWithProxy creates a client for ox-browser that routes all requests (to the ox-browser API itself) through the given proxy function. proxyFn is compatible with http.Transport.Proxy — pass proxyPool.TransportProxy() to ensure each call to ox-browser uses a fresh rotated residential IP.

No X-Internal-Secret is attached on this path, by design: a plain-http request to ox-browser transits the external proxy in cleartext, so the secret would be visible to the proxy operator.

func NewOxBrowserSolver ¶

func NewOxBrowserSolver(cfg OxBrowserSolverConfig) *OxBrowserSolver

NewOxBrowserSolver creates a CookieProvider backed by ox-browser. Its calls to ox-browser carry X-Internal-Secret from INTERNAL_SERVICE_SECRET when set — unless cfg.ProxyFn routes them through an external proxy (see NewOxBrowserClientWithProxy).

func WithBrowser ¶

func WithBrowser(b string) ProfileOption

WithBrowser filters profiles by browser. Valid values: "chrome", "firefox", "safari", "edge".

func WithMobile ¶

func WithMobile(mobile bool) ProfileOption

WithMobile filters for mobile or desktop profiles.

func WithOS ¶

func WithOS(os string) ProfileOption

WithOS filters profiles by operating system. Valid values: "windows", "macos", "linux", "android", "ios".

func NewRetryTracker ¶

func NewRetryTracker(maxAttempts int, ttl time.Duration) *RetryTracker

NewRetryTracker creates a tracker that allows up to maxAttempts per URL before blocking retries. Entries expire after ttl (lazy eviction).

func NewSiteIntel ¶

func NewSiteIntel(client *OxBrowserClient) *SiteIntel

NewSiteIntel creates a SiteIntel cache backed by ox-browser /analyze.