Dominic Jainy stands at the intersection of emerging infrastructure and intelligent systems, bringing a wealth of expertise in how artificial intelligence and blockchain are reshaping our digital world. As an IT professional who has spent years dissecting the mechanics of machine learning, he offers a unique vantage point on the evolving relationship between the data that powers our web and the AI models that interpret it. Today, we delve into a groundbreaking collaboration between Cloudflare and OpenAI that signals a massive shift in how the internet is indexed, explored, and utilized by generative search engines.
The following discussion explores the transition from traditional, link-based search models to discovery systems fueled by real-time network signals. We examine the technical challenges of crawling the modern web, such as the drain of duplicate content and the risks of compromised pages, while also addressing the crucial balance between AI visibility and publisher control. This conversation highlights how deep network integration is enabling AI assistants to access a broader, more accurate range of human knowledge than ever before.
Traditional search engines have long prioritized pages with high inbound link volume. As AI tools shift their focus toward network signals and metadata to identify content, how does this redefine what we consider a “valuable” webpage in 2026?
The definition of value is shifting from popularity to utility. In the old model, a page was deemed important if many other sites linked to it, which often left high-quality but niche information—like a specific product manual or a local restaurant’s updated hours—buried in the depths of the web. By using network signals from roughly 1% of Cloudflare’s global traffic, AI systems can now identify pages that people are actually visiting and using, regardless of how many backlinks they have. This means a “valuable” page is now defined by its ability to answer a specific query accurately rather than its SEO standing. We are moving toward a more meritocratic web where the relevance of the information takes precedence over the social proof of the link.
The recent pilot program managed to uncover hundreds of millions of previously unknown pages using only a small fraction of network traffic. What does this massive discovery rate suggest about the limitations of our previous crawling methods?
It reveals that traditional crawlers were essentially flying blind, missing vast oceans of data because they relied on a breadcrumb trail of links that doesn’t always exist. By tapping into network-level metadata, the pilot enabled the discovery of hundreds of millions of pages that had stayed off the radar of conventional search engines for years. This suggests that the “visible web” was actually much smaller than we realized, and our previous methods were inherently biased toward older, more established domains. This new approach allows crawlers to be far more surgical, finding fresh or updated content the moment it generates traffic rather than waiting for a search engine to eventually stumble upon it.
We often hear about how automated traffic and “dead” pages can clutter search indexes. How does filtering these signals at the network level create a more efficient ecosystem for both AI companies and website owners?
Filtering at the network level is like having a high-tech gatekeeper that prevents wasted energy. In this pilot, signals helped identify when multiple URLs were pointing to identical content or when a site was redirecting thousands of addresses to a single homepage, which are common issues that normally drain crawler resources. For the AI companies, this means they aren’t wasting compute power on duplicate or non-existent pages, and for site owners, it significantly reduces the “crawler tax”—the heavy server load caused by bots constantly pinging unchanged pages. By knowing when a page is actually worth revisiting based on traffic patterns, the system ensures that search resources are directed toward content that truly adds value to the user.
There is a significant distinction made between using metadata for discovery and using page content for model training. How does this technical separation help address the ongoing friction between AI developers and content publishers?
This distinction is a critical olive branch to the publishing community. By using metadata strictly to help a crawler find a page, the system honors the domain owner’s choices about whether that content can actually be ingested for training. Cloudflare’s “AI Crawl Control” tools give publishers the power to say, “You can list me in your search results, but you cannot use my writing to train your next large language model.” This separation ensures that discovery doesn’t automatically equate to exploitation. It allows for a world where a website can still gain visibility and traffic from an AI assistant without losing control over its intellectual property.
For queries involving niche or local information, like passport requirements or specific restaurant menus, accuracy is the only metric that matters. How does expanding the index to include these “invisible” pages fundamentally change the user experience for someone using an AI assistant?
It turns the AI from a generalist into a specialist that can provide hyper-local, real-time answers. When the index includes those hundreds of millions of previously hidden pages, the AI no longer has to guess or provide outdated info; it can cite the exact PDF manual for a 10-year-old appliance or the specific opening hours of a neighborhood bistro. This reduces “hallucinations” because the AI has a direct, verified source to pull from rather than relying on stale data from a few years ago. The user experience becomes much more dependable, as the assistant moves from providing general summaries to delivering specific, actionable facts.
Security was a major pillar of this research, specifically regarding the identification of compromised pages. In what ways can network-level signals act as a first line of defense to keep search results safe for the average user?
Network signals are incredibly effective at spotting anomalies that a standard crawler might miss, such as a legitimate site that has been injected with a fake login form. During the pilot, the companies tested signals that could flag these compromised pages, allowing the AI to exclude them from its index before a user ever sees them. Because the network sees the behavior of the traffic—such as where data is being sent or if a page is behaving like a phishing site—it can provide a layer of security that is far more proactive than waiting for a user to report a bad link. This essentially creates a “cleaner” version of the internet for AI users, where the risk of encountering malware or scams is significantly mitigated at the source.
What is your forecast for the future of web indexing and AI discovery?
I expect we will see a total decoupling of “crawling” from “indexing” as we know it today. In the next few years, the internet will likely move toward a “push” model where network providers and site owners feed real-time change signals directly to AI models, making the concept of a monthly or weekly “crawl” obsolete. We will see a much more dynamic, living index where the gap between a page being published and a page being searchable shrinks to mere seconds. This will result in an AI-driven web that is not only more expansive—containing billions of niche pages we can’t even find today—but also far more efficient, as the majority of bot traffic will be replaced by these intelligent, targeted network signals.