Killing the “Keys to the Kingdom”: Automated Just-In-Time Policy Elevator on Google Cloud

How we eliminated permanent standing privileges on Google Cloud using project-scoped IAM CEL conditions, audit-log watchdogs, and ETag-verified policy revocation.

Rwintrob

6 min read

·

Sep 15, 2026

--

Listen

Share

Press enter or click to view image in full size

Disclaimer: This package, and its associated code are provided as-is for educational and architectural demonstration purposes. This is not an official Google product, nor is it supported by Google Cloud Customer Support or Google engineering teams. Use in production environments at your own discretion and risk.

The High-Privilege Paradox in Digital-Native Companies

Imagine a common Tuesday afternoon scenario: A senior DevOps engineer needs to temporarily override a restrictive Organization Policy — such as compute.disableSerialPortAccess or compute.vmExternalIpAccess—on a single staging project to troubleshoot a networking failure.

In Google Cloud’s resource hierarchy, roles/orgpolicy.policyAdmin is an Organization-level role that cannot be granted inside a single project's standard IAM policy. Platform teams usually fall into one of two traps:

1. Standing Privileges: Granting permanent Organization Policy Administrator rights at the org root, where a single leaked token compromises the entire cloud estate.
2. Ticket Bottlenecks: Routing every temporary exception through a manual Security Operations queue, stalling engineering velocity and leaving forgotten permissions behind.

To solve this without requiring enterprise licensing tiers, we built JIT Policy Elevator — an open-source Just-In-Time (JIT) IAM elevation broker for Google Cloud.

Architectural Blueprint: Zero Standing Privileges by Design

Hosted on Google Cloud Run behind Identity-Aware Proxy (IAP), the system replaces permanent role assignments with short-lived, self-service access governed by five guardrails:

1. Project-Level Blast Radius Isolation (IAM CEL Conditions)
2. Domain Allowlisting & Strict Separation of Duties (SoD)
3. Human-in-the-Loop Peer Approval
4. Active Watchdog Monitoring with Early Auto-Rollback
5. Empirical Post-Revocation Verification (ETag Checks)

Press enter or click to view image in full size

1. The Scoping Secret: Isolating Org-Level Roles to a Single Project

To grant roles/orgpolicy.policyAdmin without exposing the entire organization, iam_manager.py injects a Google Cloud IAM v3 Common Expression Language (CEL) condition directly into the Organization IAM policy:

# iam_manager.py: Constructing the Project-Scoped CEL Binding
cond_title = f"JIT-Project-{target_project_id}-{role_slug}"
cond_expr = f'resource.name.startsWith("projects/{target_project_id}")'
target_binding = {
    "role": target_role,  # e.g., "roles/orgpolicy.policyAdmin"
    "members": [f"user:{requester_email}"],
    "condition": {
        "title": cond_title,
        "description": f"JIT {target_role} scoped strictly to project {target_project_id}",
        "expression": cond_expr
    }
}

Because GCP evaluates resource.name.startsWith("projects/{target_project_id}") at runtime, the engineer can modify Organization Policies exclusively on that target project. Any attempt to mutate sibling projects or the organization root is denied by GCP's control plane. The same engine also supports high-impact roles across infrastructure and security (roles/resourcemanager.projectIamAdmin, roles/compute.admin, roles/container.admin, roles/secretmanager.admin, and roles/storage.admin).

2. Identity Guardrails & Separation of Duties

In main.py, every elevation request passes three checks before state is persisted:

• IAP Identity Extraction: User identity is extracted from Google Cloud’s signed X-Goog-Authenticated-User-Email header, preventing spoofing.
• Domain Allowlisting: Both requester and approver are verified against ALLOWED_DOMAINS.
• Separation of Duties (SoD): Requesters cannot approve their own requests:

# main.py: Enforcing Separation of Duties
if requester.lower() == approver.lower():
    raise HTTPException(
        status_code=400,
        detail="Separation of Duties violation: Requester cannot designate themselves as their own approver."
    )

Once validated, notifier.py sends an HTML approval prompt via SendGrid, Mailgun, or SMTP.

Press enter or click to view image in full size

3. The Secret Weapon: Active Watchdog Agent with Early Auto-Rollback

Traditional JIT tools rely on passive timers: if an engineer requests a 60-minute window to run a 12-second gcloud org-policies set-policy command, the privilege remains exposed for the remaining 59 minutes and 48 seconds.

Our event-driven Watchdog worker (watchdog.py) closes this gap automatically:

Press enter or click to view image in full size

1.Telemetry Mapping: Maps elevated roles to specific Admin Activity audit methods (e.g., UpdatePolicy, SetPolicy, DeletePolicy).

2. Activity Detection: Monitors Cloud Audit Logs for matching calls by requester_email on projects/{target_project_id}, shifting the grant from IDLE to OPERATION_IN_PROGRESS.

3. Post-Activity Quiet Window: Once activity stops for a configurable quiet window (default: 15 seconds for single-shot CLI changes), the task is marked complete.

4. Early Rollback: Triggers immediate revocation (WATCHDOG_AUTO_REVOKED), shrinking exposure from hours to seconds. A background fail-safe worker also enforces hard TTL expirations if no activity ever occurs.

Press enter or click to view image in full size

4. Empirical ETag Verification

Because distributed IAM policies are subject to eventual consistency and ETag collisions, assuming a revocation succeeded just because an API call returned 200 OK is risky. Every revocation is verified by re-fetching the live policy (verify_permission_removed in iam_manager.py):

# iam_manager.py: Empirical Non-Existence Verification
policy = self._resourcemanager.organizations().getIamPolicy(
    resource=f"organizations/{org_id}",
    body={"options": {"requestedPolicyVersion": 3}}
).execute()
is_present = any(
    b.get("role") == target_role 
    and b.get("condition", {}).get("title") == cond_title
    and normalized_member in b.get("members", [])
    for b in policy.get("bindings", [])
)
if not is_present:
    return VerificationResult(
        verified_removed=True,
        timestamp=datetime.now(timezone.utc),
        policy_etag=policy.get("etag")
    )

Only when verified_removed is True does the grant close, recording the resulting Organization Policy ETag (policy_etag) in the audit log as proof of removal.

5. Immutable Dual-Write Auditing & Cloud-Native Deployment

Every state transition in audit.py dual-writes structured JSON to Google Cloud Logging (for SIEM alerting) and Google Cloud Firestore (for historical search via /api/audit).

Deploying to Cloud Run takes a single script (deploy.sh), which provisions a least-privilege service account and deploys the container behind IAP:

export GCP_PROJECT_ID="your-security-tooling-project"
export GCP_ORGANIZATION_ID="123456789012"
export ALLOWED_DOMAINS="yourcompany.com"
./deploy.sh

6. Architectural Trade-offs & Lessons Learned

Building an automated revocation engine on distributed cloud primitives revealed three important operational trade-offs:

• Audit Log Latency vs. Early Revocation: While GCP Admin Activity logs typically arrive within seconds, ingestion spikes of 30–60 seconds can happen. To prevent premature revocation before an engineer starts working, the Watchdog remains in IDLE (governed by the max-duration TTL) until the first matching audit log transitions it to OPERATION_IN_PROGRESS. In production, expanding the quiet window buffer to 60–120 seconds safely absorbs logging pipeline jitter.
• Interactive Troubleshooting (“Think Time”): A 15-second quiet window works well for pre-scripted CLI commands (gcloud org-policies set-policy or terraform apply), but cuts off engineers mid-investigation during interactive debugging. Quiet windows must be role-specific: tight (15–30s) for single-shot governance overrides, and longer (5–10 minutes or manual UI revocation) for interactive troubleshooting roles (roles/compute.admin, roles/container.admin).
• IAM Policy Quotas & ETag Contention: GCP limits total IAM bindings per resource and uses optimistic concurrency control via ETags. Writing hundreds of concurrent sessions to the Organization root would cause 409 Conflict / 412 Precondition Failed collisions and policy bloat. We mitigate this by wrapping policy mutations in exponential backoff retry loops and reserving Org-root CEL bindings strictly for true org-level roles (roles/orgpolicy.policyAdmin), while binding project roles directly to the target project's IAM policy (projects/{target_project_id}).Explore the Code & Contribute

Conclusion

Eliminating standing privileges on Google Cloud doesn’t require slowing down engineering teams or buying heavyweight enterprise suites.
By combining IAM v3 CEL conditions, an event-driven Watchdog worker, and empirical ETag verification, JIT Policy Elevator ensures high-privilege access exists only for the target project, only with peer approval, and only for the exact time required to get the job done.

You can find the relevant code in my Github repo at:
https://github.com/rwintrob/policy_elevator