incognigo

Incognigo

Incognigo is a dependency-free Go library for detecting and transforming personally identifiable information (PII) before it reaches logs, external services, or other data destinations.

The current development line is designed to fail closed, bound input-driven work, avoid source data in library diagnostics, and support concurrent reuse. The public API remains a v0 API: compatibility-breaking improvements are still possible before v1 and are recorded in CHANGELOG.md.

Incognigo provides:

• configurable policy and priority per PII type;
• byte, string, and bounded-retention stream processing;
• a protecting log/slog handler;
• built-in ASCII email, IPv4/IPv6, payment-card, NANP phone, narrow global E.164/telephone-URI, IBAN, and MAC detectors, plus opt-in US and GB identifier packs;
• source-content-free inspection results with ordinal confidence and fixed evidence metadata;
• removal, fixed replacement, byte masking, HMAC tokenization, and reversible AES-256-GCM tokenization; and
• extension interfaces for application-owned detectors and transformers.

The built-in detectors are deliberately syntactic and limited. Incognigo does not establish that a match belongs to a person, and it does not include names, postal addresses, national phone formats outside NANP, phone-number assignment metadata, government identifiers outside the explicit locale packs, or contextual free-text entity recognition. It does not detect all PII and cannot guarantee anonymization.

Requirements and installation

Incognigo supports Go 1.25 and Go 1.26. Use the latest patch release available for either line.

go get github.com/Chadi00/incognigo@latest

The module has no third-party runtime or test dependencies.

Quick start

This complete example protects all six core built-in PII types with seven detectors (NANP and E.164 share the phone policy). Locale-pack detectors remain explicitly opt-in. Always treat construction or processing errors as protection failures; never fall back to the original input.

package main

import (
	"context"
	"fmt"
	"log"

	"github.com/Chadi00/incognigo"
	"github.com/Chadi00/incognigo/detector"
	"github.com/Chadi00/incognigo/transform"
)

func main() {
	if err := run(); err != nil {
		log.Fatal(err)
	}
}

func run() error {
	engine, err := incognigo.NewEngine(incognigo.Config{
		Detectors: []incognigo.DetectorRegistration{
			detector.Email(),
			detector.IPAddress(),
			detector.PaymentCard(),
			detector.NANPPhone(),
			detector.E164Phone(),
			detector.IBAN(),
			detector.MACAddress(),
		},
		Policies: []incognigo.Policy{
			{
				Type:        incognigo.PIITypeEmailAddress,
				Priority:    100,
				Transformer: transform.Replace([]byte("[email]")),
				Determinism: incognigo.Deterministic,
			},
			{
				Type:        incognigo.PIITypeIPAddress,
				Priority:    90,
				Transformer: transform.Replace([]byte("[ip]")),
				Determinism: incognigo.Deterministic,
			},
			{
				Type:        incognigo.PIITypePaymentCardNumber,
				Priority:    80,
				Transformer: transform.Replace([]byte("[payment-card]")),
				Determinism: incognigo.Deterministic,
			},
			{
				Type:        incognigo.PIITypePhoneNumber,
				Priority:    110,
				Transformer: transform.Replace([]byte("[phone]")),
				Determinism: incognigo.Deterministic,
			},
			{
				Type:        incognigo.PIITypeIBAN,
				Priority:    60,
				Transformer: transform.Replace([]byte("[iban]")),
				Determinism: incognigo.Deterministic,
			},
			{
				Type:        incognigo.PIITypeMACAddress,
				Priority:    50,
				Transformer: transform.Replace([]byte("[mac]")),
				Determinism: incognigo.Deterministic,
			},
		},
	})
	if err != nil {
		return fmt.Errorf("configure PII protection: %w", err)
	}

	protected, err := engine.ProtectString(
		context.Background(),
		"email owner@example.test from 192.0.2.1 "+
			"with card 4111 1111 1111 1111, call 212-555-0100 or "+
			"tel:+442079460958, "+
			"IBAN GB29NWBK60161331926819, MAC 00:1A:2B:3C:4D:5E",
	)
	if err != nil {
		return fmt.Errorf("protect data: %w", err)
	}
	fmt.Println(protected)
	return nil
}

Output:

email [email] from [ip] with card [payment-card], call [phone] or tel:[phone], IBAN [iban], MAC [mac]

The repository's Example... functions compile and execute under go test. They cover every public workflow, including custom extensions, all built-in detectors and transformations, streaming, structured logging, detokenization, and inspectable failures.

Processing APIs

Engine.ProtectBytes is the canonical batch API. It never mutates or retains its input, and every successful result is newly owned by the caller. Engine.ProtectString is a byte-preserving adapter over the same path. On failure they return no partial or unprotected output.

protectedBytes, err := engine.ProtectBytes(ctx, inputBytes)
protectedString, err := engine.ProtectString(ctx, inputString)

Engine.Inspect runs the same validation, candidate budget, confidence thresholds, and overlap resolution without transforming source bytes:

findings, err := engine.Inspect(ctx, inputBytes)

Each finding contains only type, exact original byte span, detector/version, locale, ordinal confidence, fixed evidence flags, configured threshold, and outcome. It contains no source excerpt, copied match, source hash, or provider error. The metadata is content-derived and is not automatically safe to log.

Engine.ContainsPII is the metadata-free, early-exit trust-boundary check. It stops on the first valid detector candidate, including candidates below policy confidence thresholds, and returns only a boolean:

containsPII, err := engine.ContainsPII(ctx, inputBytes)

An error means the result is indeterminate and must fail closed. The sloghandler strict-key policy uses this path.

Streaming

Engine.ProtectStream processes an arbitrary-length io.Reader while bounding undecided source retention:

var destination bytes.Buffer
written, err := engine.ProtectStream(
	ctx,
	&destination,
	strings.NewReader("contact owner@example.test from 192.0.2.1"),
)

written counts only protected bytes accepted by the destination. A reader, detector, transformer, or writer failure after an earlier write can leave a protected prefix. Incognigo never appends undecided or unprotected source bytes after that prefix. Use a transactional destination or buffer before commit when delivery must be all-or-nothing.

For file output, write to a restricted temporary file on the destination filesystem, sync and close it after success, then atomically rename it. For databases or object stores, write inside a transaction or to an uncommitted staging object and publish only after ProtectStream succeeds. Remove or abort the staging destination on every error.

Structured logging

For production logging, start with an opt-in profile. The telephony profile is strict: it scans free text, directly redacts known PSTN/SIP fields case-insensitively, rejects PII-looking attribute keys and group names, protects numeric values as strings, and rejects remaining unsupported native representations:

downstream := slog.NewJSONHandler(os.Stdout, nil)
handler, err := sloghandler.NewWithOptions(
	engine,
	downstream,
	sloghandler.TelephonyProfile(),
)
if err != nil {
	return err
}

logger := slog.New(handler)
logger.Info(
	"routing tel:+1-416-555-0100",
	"ANI", "local extension 42",
	"To", "sip:+12125550100@voice.example.test",
)

TelephonyProfile directly replaces phone_number, phone, ani, dnis, from, to, caller_id, contact, and common asserted-identity fields even when their values are local, malformed, or outside detector syntax. It also inherits common credential-field redactions. ProductionProfile keeps native bool, duration, and time values for generic services; StrictProductionProfile rejects them. All profiles are starting points returned by value, so applications can review and amend their FieldPolicies before construction.

Custom exact field policies use the same complete-value behavior:

options := sloghandler.ProductionProfile()
options.FieldPolicies = options.FieldPolicies.With(sloghandler.FieldPolicy{
	Key:         "customer_reference",
	Replacement: "[REDACTED]",
})
handler, err := sloghandler.NewWithOptions(engine, downstream, options)

Matching is exact rather than substring-based. Enable CaseInsensitiveFieldMatching for unstable header/key casing. A matched field is replaced before resolving slog.LogValuer, formatting opaque values, or traversing a group, so known-sensitive values do not depend on free-text detection.

With RejectPIILookingKeys, every non-empty attribute key and group name is inspected by the configured engine. Any detector candidate—including one below its policy confidence threshold—returns sloghandler.ErrUnsafeKey and prevents downstream handling. The error never includes the key. Keys are not transformed because renaming can create collisions and alter group semantics.

String and exact []byte values keep their logical kinds when they are not field-redacted. slog.LogValuer values are resolved, groups are traversed without recursive stack growth, and other non-nil slog.Any values are eagerly converted with fmt.Sprint, protected, and passed downstream as strings. This conversion prevents a downstream-only serializer from bypassing protection, but intentionally loses the original dynamic type.

Attribute/group names that pass strict inspection, nil Any values, native values allowed by the selected options, and record metadata are preserved. Downstream handler errors are returned unchanged. A canceled record context does not cancel protection, as required by the slog.Handler contract.

Native integers, unsigned integers, and floats can be converted to protected strings explicitly. Strict mode rejects unprotected native representations:

handler, err := sloghandler.NewWithOptions(
	engine,
	downstream,
	sloghandler.Options{
		ProtectNumericValues:    true,
		RejectUnsupportedValues: true,
	},
)

Numeric conversion and field replacement change downstream schemas. The zero-value Options remains compatibility-oriented and opt-in profiles do not change sloghandler.New.

See examples/http for a small net/http service using the strict telephony setup without a framework.

Transformations

Detection and transformation are independent. A policy can choose any transformer for its PII type.

Masking counts bytes, not Unicode code points. Callers choose the retained prefix and suffix, one replacement byte, and whether unexpectedly short values are fully masked or rejected without output.

masker, err := transform.NewMasker(transform.MaskOptions{
	PrefixBytes: 2,
	SuffixBytes: 4,
	MaskByte:    '*',
	ShortInput:  transform.ShortInputMaskAll,
})

All built-in transformations are immutable and concurrency-safe after construction. Redaction and masking are directly idempotent; end-to-end engine idempotence additionally depends on whether detectors recognize transformed output. Tokenization is not idempotent.

Safe production configuration

For logs and telemetry, fixed replacement or removal is the recommended baseline: it does not preserve value fragments, equality, or reversibility. Combine it with a production sloghandler profile, register every detector needed by the service domain, keep MinimumConfidence at ConfidenceWeak/zero when recall is the security priority, and bound limits for the largest intended record.

Masking is appropriate only when the explicitly retained prefix/suffix is permitted to leave the trust boundary. HMAC tokenization is appropriate only when cross-event correlation is required and equality/frequency disclosure is accepted. Reversible tokenization is appropriate only when an authorized recovery path is a requirement; load its random 32-byte key from a secret boundary, rotate it, and never log it.

For PSTN/SIP services, a safe starting point is:

detectors := []incognigo.DetectorRegistration{
	detector.Email(),
	detector.IPAddress(),
	detector.NANPPhone(),
	detector.E164Phone(),
}
// Configure replacement policies for email, IP, and the shared phone type,
// then wrap the downstream handler with sloghandler.TelephonyProfile().

Free-text detection and exact field policy are complementary: scanning catches PII embedded in messages and generic values; direct field redaction covers known-sensitive ani, dnis, from, to, and similar values even when their syntax is not recognized. Strict key inspection closes the third path where caller-controlled PII becomes a key or group name.

Give the phone policy a higher overlap priority than email when SIP subscriber forms should be classified as phone numbers. A value such as +12125550100@voice.example.test is also a syntactically valid email dot-atom/domain; deterministic policy priority decides which transformation wins.

Custom extensions

Applications can implement incognigo.Detector and incognigo.Transformer. A detector emits through the borrowed incognigo.MatchSink; Add validates authorization, spans, match length, and the configured budget before engine-owned storage grows. Detectors must stop when Add returns false and must never retain or call the sink concurrently. Registrations declare stable detector IDs, algorithm versions, confidence, evidence, authorized PII types, and finite match/lookaround bounds.

Transformers can additionally implement incognigo.SizedTransformer. When every selected transformer does so, batch output size is checked before any replacement is allocated and the output buffer is allocated exactly once. All built-in transformers implement this contract.

Custom extensions remain trusted code: they must treat borrowed values as read-only, must not retain or disclose them, must honor cancellation, must not panic, and must be safe for concurrent calls. Incognigo cannot enforce that an extension does not retain data, start hidden work, race, or allocate internally.

NewEngine copies configuration slices and metadata, but retains detector and transformer implementations as executable interfaces.

Built-in detector formats

Built-ins are deterministic, byte-oriented syntactic detectors. Non-ASCII and malformed UTF-8 bytes are barriers, original byte offsets are preserved, and input is never normalized. Detection resumes after a barrier, so a valid ASCII candidate that follows a non-ASCII byte can still match independently.

Additional format details:

• Email local parts contain 1–64 bytes made from ASCII letters, digits, dots, and !#$%&'*+-/=?^_`{|}~. Dots separate non-empty atoms. Domain labels contain 1–63 letters, digits, or interior hyphens; the domain is at most 253 bytes and the complete address at most 254 bytes. A terminal period is surrounding punctuation.
• IPv4 spans the full 0.0.0.0 through 255.255.255.255 range and uses a strict allocation-free parser. IPv6 is validated with net/netip. Colon-label recovery is bounded to eight prefix components and 64 bytes per component; longer delimiter chains are intentionally outside the detector's contract.
• Payment-card rendered length is at most 37 bytes. Passing Luhn does not prove that an account was issued or is active.
• NANP examples include 2125550100, 212-555-0100, (212) 555-0100, 1-212-555-0100, +12125550100, and +1 (212) 555-0100.
• E.164/global examples include +12125550100, tel:+358-555-1234567, sip:+12125550100@voice.example.test, and sips:+44.20.7946.0958@voice.example.test. The detector enforces the international 15-digit maximum but deliberately does not embed country calling-code or national-plan metadata. Its scope follows the maximum in ITU-T E.164, the telephone-subscriber syntax in RFC 3966, and the SIP user placement in RFC 3261.

detector.StructuredGlobal() returns independent registrations for email, IP, payment card, IBAN, and MAC. Phone numbers remain excluded to preserve profile compatibility and because national forms need explicit region metadata; add E164Phone and/or NANPPhone explicitly. detector/us.Identifiers() and detector/gb.Identifiers() are opt-in locale packs for checksum- or range-validated identifiers; their package documentation states the exact scope. Adding a detector to any profile is a behavioral compatibility change.

The locale packs do not query assignment registries. Checksum/range validity does not prove issuance or identity, and compact numeric values can collide with unrelated identifiers.

External detection evaluation

The checked-in detector cases are conformance fixtures, not statistical accuracy evidence. They prove intended formats and exact byte spans remain stable; no precision or recall claim is derived from them. The generated built-in detector coverage report inventories matched formats, exercised non-matches/limitations, detector versions, and fixture counts for every built-in package. The built-ins are deterministic rules rather than trained models, but fixtures authored with those rules still cannot measure generalization.

Incognigo v0.2.0 was instead evaluated on a frozen revision of the independent, Apache-2.0 Gravitee PII Detection Dataset: 175,881 English documents with 781,052 labeled spans. Eight directly mapped labels supplied 153,912 unique scored spans. Exact type-and-span results were:

The dataset is mostly synthetic, English-only, and its global phone label was broader than the NANP-only phone configuration evaluated in v0.2.0. The new opt-in E.164 detector has not been scored into that frozen result. The results are useful independent coverage evidence, not an estimate for arbitrary production traffic. See EVALUATION.md for the pinned revision and checksum, full TP/FP/FN counts, methodology, limitations, and one-command reproduction.

Behavioral contracts

Configuration and overlap resolution

Engines must be created with incognigo.NewEngine. Detector IDs and policies are unique. Every type declared by a detector has exactly one policy, and every policy corresponds to a declared type. Invalid or typed-nil implementations are rejected. A nil or zero-valued engine fails closed.

Overlapping candidates are preferred deterministically by:

1. policy priority, descending;
2. span length, descending;
3. start offset, ascending;
4. end offset, ascending;
5. PII type, lexicographically ascending;
6. detector ID, lexicographically ascending; and
7. detector emission sequence, ascending.

Candidates are greedily accepted only when they do not overlap an accepted span. Adjacent spans do not overlap. Identical accepted matches collapse.

Byte and ownership semantics

All offsets and limits count bytes. Incognigo performs no Unicode normalization. Malformed UTF-8 outside a match is preserved exactly.

Detector input, match sinks, and transformer values are borrowed, call-scoped views. Extensions transfer ownership of returned slices. Batch processing never mutates or retains caller input and always returns newly owned output, including empty and no-match results.

One engine may be reused concurrently when its custom extensions are also concurrency-safe. Built-in detectors and transformations are immutable and safe for concurrent use.

Limits

Zero-valued fields select these defaults:

Maximum values are inclusive. Negative limits are invalid. Total stream input and output are not bounded by the batch byte limits.

The match sink stops built-in work at limit+1; low-confidence candidates are budgeted before filtering. Built-in and conforming-transformer output is sized before replacement allocation. Arbitrary custom transformers remain trusted and can allocate internally.

Streaming validation requires one full window to commit at least max(1, MaxStreamBufferBytes/8) new source bytes. The exact progress formula is buffer - maxLookbehind - max(MaxMatchBytes+LookaheadBytes) + 1. Configurations below this threshold are rejected to prevent near-full-window rescans for every source byte.

Failures and diagnostics

Incognigo-generated runtime failures are *incognigo.Error values. Use errors.Is with the fixed sentinels and errors.As for allowlisted metadata:

var protectionErr *incognigo.Error
if errors.As(err, &protectionErr) {
	actual, maximum, hasCounts := protectionErr.Counts()
	_ = actual
	_ = maximum
	_ = hasCounts
}

Library errors may contain only fixed operation/field/limit names, validated detector IDs and PII types, and numeric counts. Arbitrary detector, transformer, reader, and writer errors are discarded rather than wrapped because their messages may contain source data. Batch failures return no partial output. Transformer output returned with an error is discarded.

Canceled operations still match their Incognigo category and the fixed context.Canceled or context.DeadlineExceeded sentinel. An extension cannot forge cancellation by merely returning one of those errors while the supplied context remains active.

Never return, write, enqueue, or log the original input after protection fails. This fallback is unsafe:

protected, err := engine.ProtectBytes(ctx, input)
if err != nil {
	protected = input // UNSAFE: turns a protection failure into disclosure.
}
_, _ = destination.Write(protected)

Fail closed or send an input-independent replacement instead:

protected, err := engine.ProtectBytes(ctx, input)
if err != nil {
	return fmt.Errorf("protect outbound data: %w", err)
}
if _, err := destination.Write(protected); err != nil {
	return fmt.Errorf("write protected data: %w", err)
}

The same rule applies to slog: do not retry a failed protecting handler with the unwrapped downstream handler. Standard slog.Logger calls do not return handler errors; a protecting handler still drops the record before downstream delivery. Applications that must observe logging failures should invoke or wrap the handler at an application-owned boundary and emit only input-independent health signals.

Tokenization design

Both tokenizers require a caller-supplied, unpredictable 32-byte key and a public key ID. They accept arbitrary bytes, bind the PII type, use unpadded base64url output, default to an 8 MiB direct-call limit, and perform no network, file, secret-manager, database, or token-vault access.

Deterministic irreversible tokens use:

icg.v1.hmac-sha256.<key-id>.<32-byte-hmac>

The HMAC input is:

"incognigo\0token\0v1\0hmac-sha256\0"
|| complete-token-header
|| uint64be(len(pii-type)) || pii-type
|| uint64be(len(value))    || value

Randomized reversible tokens use:

icg.v1.aes-256-gcm.<key-id>.<12-byte-nonce || ciphertext || 16-byte-tag>

AES-GCM authenticates:

"incognigo\0token\0v1\0aes-256-gcm\0"
|| complete-token-header
|| uint64be(len(pii-type)) || pii-type

The complete header ends after the key-ID separator. Parsers reject unknown versions, algorithms, malformed encodings, wrong PII types, tampering, and non-canonical payloads.

Threat model and key rotation

HMAC tokens expose equality and frequency. Possession of their key permits offline guessing, especially for low-entropy values. Reversible tokens expose encoded plaintext length and become readable and forgeable if their key is compromised. Neither mode prevents disclosure from other plaintext copies, process memory, crash dumps, malicious extensions, or application code authorized to detokenize.

Go does not guarantee zeroization of compiler-managed key or plaintext memory. Callers own secret storage, process controls, aggregate usage accounting, and rotation. One reversible key must not encrypt more than 2^32 values across all processes and tokenizer instances.

To rotate a reversible key:

1. create a tokenizer with the new key as ActiveKey;
2. retain required prior keys in DecryptionKeys;
3. route new transformations through the new instance;
4. optionally detokenize and re-tokenize persisted old values; and
5. remove an old key only when recovery and retention policies permit.

Missing keys are never fetched automatically. A recognized reversible token with an unavailable key returns ErrTokenKeyUnavailable; malformed or unauthenticated tokens return ErrInvalidToken; oversized values return ErrTokenValueTooLarge. Every failure returns nil plaintext.

Test vectors

The following vectors use the synthetic value person@example.test, PII type email_address, key ID 2026-01, and a test-only key containing bytes 00 01 ... 1f.

icg.v1.hmac-sha256.2026-01.3rDSpGYbv2Mt9LkfD0SiBefyvnDQN47gLdInd8STLyk

The reversible decryption vector uses the fixed nonce 00 01 02 03 04 05 06 07 08 09 0a 0b only for verification:

icg.v1.aes-256-gcm.2026-01.AAECAwQFBgcICQoLN2ekaKqLgn71IPr73YxWGeal84tlruM4lmxUjjKxV2SbAHM

Production tokenization always obtains fresh randomness from Go's cryptographic implementation.

Security and limitations

Incognigo processes sensitive data at application trust boundaries. Report a suspected privacy bypass, source-data disclosure, unbounded resource path, concurrency leak, or cryptographic defect through GitHub's private security advisory flow:

https://github.com/Chadi00/incognigo/security/advisories/new

Do not include customer data, production logs, credentials, real PII, or public breach data. Use a minimal synthetic reproduction. If private advisories are unavailable, contact the repository owner privately without sending exploit details in the initial message.

Operational limitations:

• syntactic detection cannot guarantee that every production PII value is found;
• unsupported formats pass through unchanged unless a custom detector handles them or an exact field policy replaces the complete value;
• attribute/group keys are not transformed; opt-in strict inspection rejects PII-looking names, field policies and numeric protection can change downstream schemas, and strict profiles can reject other native values;
• custom extensions are trusted code;
• streaming can leave a protected prefix after failure;
• deterministic tokens disclose equality and reversible tokens depend on caller-owned key protection; and
• during v0, exported APIs may change to address material safety, correctness, or usability issues.

Never treat protection failure as permission to send the original value.

Development and release checks

Test fixtures, fuzz seeds, examples, benchmarks, issues, and pull requests must use synthetic data. Never commit customer samples, real-person data, public breach data, raw vulnerability-report values, or secrets. The canonical detector corpora live under detector/testdata.

Common checks:

go vet ./...
go test ./...
go test -race ./...
go test -run '^$' -bench . -benchmem -benchtime=1x ./...
go test -v ./detector -run '^TestConformanceCorpora$'
./scripts/check-detector-fuzz-coverage.sh
./scripts/generate-detector-coverage.sh --check
./scripts/evaluate-gravitee.sh evaluation.md
./scripts/fuzz.sh 30s
./scripts/api-diff.sh v0.1.0

The complete release gate checks formatting, module tidiness, the zero-dependency policy, vetting, ordinary and race tests, benchmark compilation, hot-path allocation and 25% same-runner median slowdown guards, conformance fixtures, every fuzz target, and pinned vulnerability analysis:

./scripts/release-check.sh 30s

compatibility-manifest.json records persisted token formats, PII and detector IDs, profile composition, algorithm/metadata versions, confidence order, overlap rules, and streaming progress. MIGRATING.md records adoption and rollout impact for detector behavior changes. Release-candidate CI retains a public API diff and raw benchmark output. Every release has a committed snapshot under benchmarks, and tag builds add a fresh checksummed platform snapshot beside the SHA-bound source archive and signed GitHub/Sigstore provenance evidence.

Pull requests affecting detection, transformation, streaming, logging, public or persisted behavior must explain failure behavior, resource bounds, test and fuzz coverage, and compatibility impact.

License

Incognigo is licensed under the Apache License 2.0.