In the rapidly evolving world of decentralized finance, the ability to trace illicit activity across fragmented networks has become a civilizational necessity. Dominic Jainy, an expert in high-scale data engineering and blockchain intelligence, understands that the difference between a successful investigation and a cold trail often comes down to the milliseconds of latency in a data pipeline. At TRM Labs, the engineering team doesn’t just manage databases; they build a self-service, petabyte-scale lakehouse architecture capable of processing an exabyte of data annually. By integrating AI-driven orchestration and standardized schemas across more than 55 blockchains, they provide the structural advantage needed to disrupt money laundering, terrorism financing, and global fraud.
The following discussion explores the technical rigor required to maintain a real-time global view of blockchain activity, the architectural shifts from legacy systems to modern lakehouses, and how AI agents are beginning to handle the operational heavy lifting of data reliability.
High-throughput networks like Solana can reach 90,000 transactions per second. How do you design ingestion pipelines to handle this volume without sacrificing data freshness, and what specific architectural trade-offs are necessary to ensure stable performance during sudden spikes in network activity?
Handling 90,000 transactions per second requires moving away from traditional batch processing toward a high-throughput, streaming-first infrastructure. At this scale, the primary design challenge is maintaining a balance between write speed and the immediate availability of data for investigators who need “fresh” results to track active hacks. To achieve this, we utilize a specialized serving layer and high-throughput write paths that can absorb these massive bursts without bottlenecking the downstream analytical engines. We often have to make explicit trade-offs between cost efficiency and latency, choosing to scale compute resources horizontally during periods of high network congestion to ensure our service-level objectives remain intact. This ensures that even when a network like Solana is under extreme load, the data remains queryable within seconds of the block being minted.
Moving massive workloads to a StarRocks and Iceberg lakehouse architecture can significantly reduce operational complexity. What were the primary technical hurdles during this transition, and how do you ensure zero downtime for customer-facing APIs while backfilling petabytes of historical blockchain data?
The transition to a StarRocks and Iceberg lakehouse was a massive undertaking that involved moving over 6 petabytes of blockchain intelligence data while keeping the lights on for our customers. One of the biggest hurdles was migrating the “Next Gen Address Transfers” workload, which is one of our most business-critical datasets, without introducing any lag in the user experience. We managed this by running parallel systems and utilizing the Iceberg format to handle large-scale updates and deletes more efficiently than traditional storage methods. Because the lakehouse allows for fast, cost-efficient analytics over cloud object storage, we were able to reduce backfill times from several days to just a few hours. This speed is what ultimately allowed us to switch over to the new architecture with zero impact on the APIs that financial institutions and government agencies rely on daily.
Onboarding dozens of new blockchains within a single year requires moving from manual configurations to a self-service model. How do you standardize schemas across diverse chain architectures, and what internal workflows allow teams to deploy new data products in days rather than quarters?
To onboard 20+ new blockchains in 2025 alone, we had to stop treating every new chain as a unique snowflake and start treating them as standardized inputs. We developed a self-service model where engineers and analysts follow a “standardized workflow” that includes predefined schemas and automated testing suites. By plugging into our common pipeline infrastructure, a new chain can be ingested, normalized, and made queryable without requiring a bespoke engineering project for each one. This shift from manual coding to a platform-based approach is why we can now ship over 25 new data products a year, such as Universal Wallet Screening and Portfolio Balance. It effectively turns a quarterly engineering roadmap into a weekly deployment cycle, allowing our coverage to stay ahead of the rapidly expanding crypto ecosystem.
In blockchain intelligence, data correctness and completeness are vital for tracing illicit funds. How do you define and measure these service-level objectives beyond simple uptime, and what is the step-by-step protocol when a pipeline fails to meet a critical reliability target?
In our domain, “uptime” is a shallow metric because a system can be online while serving incomplete or “stale” data, which is dangerous for an investigator. We move beyond “vibes” by strictly measuring freshness, correctness, and completeness as our primary service-level objectives (SLOs). Every one of our 750+ Airflow DAGs is monitored, and if a pipeline misses a specific freshness target, it is automatically escalated as a formal incident. The protocol involves an immediate triage by our on-call engineers, supported by AI-driven monitoring tools that help identify if the lag is due to a network-level event or an internal infrastructure bottleneck. We treat data quality issues with the same severity as a total system outage because we know that if our data is wrong, an investigator might lose the only window they have to freeze stolen assets.
Integrating AI agents into a data platform can automate routine tasks like incident triage and cost optimization. How are these agents specifically deployed within an orchestration layer of hundreds of DAGs, and what metrics do you use to evaluate their impact on engineering productivity?
We have begun embedding AI agents directly into our orchestration layer to handle the “babysitting” tasks that typically drain an engineer’s time, such as responding to minor pipeline failures or optimizing query costs. These agents assist with incident triage by analyzing the logs of our millions of daily tasks to identify the root cause of a failure before a human even opens a laptop. We measure their impact through metrics like “time to recover from incidents” and the reduction in manual operational tickets per engineer. The goal is to create an anti-fragile platform where the system learns from previous failures to improve its own reliability. This allows our team to spend less time on maintenance and more time on high-level architecture, which is essential when managing a platform at the exabyte scale.
Graph traversals that trace funds across 55+ chains must account for complex cross-chain swaps and entity screening. What are the primary computational challenges of maintaining a unified view of these transfers, and how does this infrastructure provide a structural advantage to investigators?
The primary computational challenge is the sheer complexity of the “full graph,” where a single entity might move funds across dozens of different chains and hundreds of thousands of addresses. Running graph traversals over petabytes of data to find links between a terrorist financing cell and a seemingly unrelated wallet requires immense processing power and highly optimized data modeling. Our infrastructure provides a structural advantage by unifying these disparate data points into a single, queryable view, allowing investigators to see “Universal Wallet Screening” and “Entity Screening” in real-time. By automating the discovery of cross-chain swaps, we help good actors move faster than the criminals who are trying to hide behind the technical complexity of the blockchain. It turns what used to be weeks of manual forensic work into a series of clicks that can happen in the heat of an active investigation.
What is your forecast for the future of blockchain intelligence?
I believe we are moving toward an era of “AI-native” data platforms where the gap between raw blockchain data and actionable intelligence disappears almost entirely. In the coming years, the scale of data will grow from petabytes to exabytes as more traditional financial assets move on-chain, and manual data engineering will no longer be able to keep pace. We will see AI agents not just monitoring pipelines, but autonomously designing them and identifying emerging patterns of illicit activity before a human researcher even knows what to look for. For investigators, this means the “structural advantage” will shift from simply having access to data to having an intelligent system that provides real-time, explainable risk scores across every transaction on earth. The systems we are building today are the foundation for a future where blockchain is no longer a “black box” for law enforcement, but the most transparent financial system in human history.