🏦 The Glass Box Bank: Engineering an Auditable AI Architecture
In regulated banking, the biggest risk isn’t just fraud — it’s the inability to explain why you flagged it. I’ve architected a Governed Data Product to solve the “Black Box” dilemma, moving beyond detection into Forensic Explain ability.
Architected a Medallion-based Data Mesh to solve the AI “Black Box” problem in fraud detection. The experiment bridges the gap between high-volume processing and regulatory-grade explain ability.
Silver Layer (Apache Iceberg): Engineered an anonymized feature store using Spark for PCA-based dimensionality reduction.
Gold Layer (Data Vault 2.0): Implemented a Hub-and-Satellite schema in DuckDB/dbt to anchor business keys and store Llama 3 forensic narratives.
Governance Loop: A self-healing pipeline that identifies “unexplained” fraud flags and automatically triggers AI inference to close the audit gap (BCBS 239).
Tech Stack: dbt | DuckDB | Apache Iceberg | Llama 3 | Data Vault 2.0
The Federated Data Mesh that decouples scale from governance:
Engineering: Leveraged Apache Spark & Iceberg for high-volume PCA feature engineering.
Storage Pattern: Used Data Vault 2.0 (dbt + DuckDB) to separate immutable business keys (Hubs) from AI interpretations (Satellites).
XAI Engine: Integrated Llama 3 (Ollama) to translate abstract mathematical vectors into plain-English forensic narratives.
Governance & Data Management:
Instead of treating compliance as an afterthought, I engineered it as a Self-Healing Loop
Full Traceability: 100% lineage from the Gold Dashboard back to the raw Silver source.
Audit Readiness: Every insight is anchored to a permanent Hub with load_date and record_source metadata.
Automated Oversight: The system identifies “unexplained” fraud flags and automatically triggers the AI pipeline to close the documentation gap.
Business Outcome:
A system where auditors can drill down from an AI anomaly score directly to the raw transaction, with the behavioural “Digital Fingerprint” visualized and explained in the vault.
Deep Dive: Understanding the PCA Vectors
What is a PCA Vector?
PCA “squashes” hundreds of raw banking dimensions into uncorrelated variables called Principal Components (V1…Vn).
The Math: Eigenvalues & Variance
- Eigenvalues (λ): These represent the strength of a signal.
- The Audit: Our pipeline ensures the first 5 components capture >90 of total variance, ensuring the “signal” of fraud isn’t lost in the “noise” of normal spending.
Interpreting the “V” Signals
Component likely Interpretation (In Banking Context)V17Geospatial Velocity: High negative values often indicate “impossible travel” scenarios.V14Temporal Shift: Detects unusual spending times or sudden merchant category switches.V12Profile Match: Measures how well the transaction fits the historical behaviour of the cardholder.
The 3 Core Rationales (The “Why”)
Reason A: Mathematical Efficiency (Orthogonality)
Get Kartheek Darbha’s stories in your inbox
Join Medium for free to get updates from this writer.
Problem: In raw data, “Time of Day” and “Shop Open Hours” are redundant (correlated). This confuses models (Multicollinearity).
Solution: PCA creates features that are Orthogonal (completely independent). $V1$ has zero correlation with $V2$. This makes model training faster and more stable.
Reason B: Signal Concentration
Problem: Real data is noisy.
Solution: PCA sorts information by importance. $V1$ holds the most variance (information), $V2$ the second most, etc. By the time you get to $V28$, you’ve captured 99% of the signal and filtered out the random noise.
Reason C: Security by Design
Rationale: If a hacker steals the $V$ dataset, they can’t reverse-engineer it to find out where you live or what you bought. It is mathematically irreversible without the original Eigenvectors.
“I built this entire architecture using DuckDB, Apache Iceberg, and Llama 3. Check out the full breakdown and code on my GitHub [Link].”
