An Empirical Inquiry into AI Scraping and Public Data in India

by Agastya Shukla and Manraj Singh Chandpuri

• Introduction

In 2025, a developer running TechPays, reported that AI crawlers generated around 90% of the site’s traffic in a month and broke the 500GB outbound data barrier by 300GB, triggering $90 in overage penalties on their hosting plan. Traditional signals of restricted use like “robots.txt” were of no use, as many did not comply. This illustrates that the web’s default tools were built for human access, not industrial scale automated extraction. When signals like robots.txt fail, “publicly viewable” starts behaving like “permissionless for automation,” even when neither the publishers nor individuals intended that outcome. Further, In Rajya Sabha’s Question No. 588, it was clarified that web scraping of publicly available data for AI training is regulated under the Information Technology Act, 2000 (“IT Act”), and may attract liability under §43, which is built around unauthorised access and extraction. 

This gap sits at the centre of this piece’s question: Is data scraping for AI training lawful in India? And it also reframes what that question really asks. The legal problem is no longer about whether personal data can be seen or encountered online. It is whether mere visibility can do the legal work of ‘permission,’ allowing actors to extract, copy, store, and reuse that data at scale for model training, even though the downstream effects are neither foreseen by, nor meaningfully controllable by, the data principal.

• The DPDP’s ‘Publicly Available’ Test

In India, §3(c)(ii) of the Digital Personal Data Protection Act, 2023 (“DPDP”), excludes the personal data that has been “made or caused to be made publicly available” by the Data Principal out of the Act’s scope. The phrasing sounds clear, however, the problem is that the statute gives no internal test for what “publicly available” means in the contexts that matter most for AI scraping. Notably, this gap is time sensitive and The Ministry of Electronics and Information Technology (“MeitY”) Notification specifically staggers DPDP’s enforcement as §3 is slated to commence only after 18 months after the Gazette Notification. The window before it turns into a routine defence is precisely when the Data Protection Board (“DPB”) or courts should crystallise an “intentional publicness” reading with respect to scraping data for AI training. This silence becomes most consequential in AI training because AI pipelines do not “read” content the way humans do. They ingest data into opaque, hard-to-reverse computational workflows and then deduplicate it, embed it, aggregate it, and reproduce its informational value downstream, long after the original page has changed or disappeared. 

In this setting, a broad reading of §3(c)(ii), treating “publicly available” as “technically reachable on an open URL” acts as a structural escape hatch. The moment personal data becomes visible online, the DPDP could stop applying, even though AI training is precisely the kind of processing that magnifies privacy risk through aggregation and persistence. Recent commentary also flag this definitional gap in §3(c)(ii) for AI scraping, though it largely stops at identifying the ambiguity without offering a workable test for intentional publicness or a scalable permission architecture for automated access. 

Why Agency Matters When DPDP Protection is Removed

To see why this matters, we must place §3(c)(ii) next to DPDP’s consent architecture. DPDP defines consent within §6 in high-agency terms that it must “…be free, specific, informed, unconditional, and unambiguous with a clear affirmative action.” If §3(c)(ii) is read as a passive, technical condition so that visibility online alone strips data principals of DPDP protection, then the exception would defeat the Act’s own agency baseline. It converts DPDP into validating that “publicly viewable” becomes “lawfully scrapable for AI training,” without any way for data principals to signal limited intent.

This agency baseline matters more after the DPDP Rules, 2025 (notified 14 Nov. 2025) operationalised the statute’s compliance structure around purpose-linked, administrable processing. So, if §3(c)(ii) removes DPDP entirely, the exclusion must rest on some substitute signal of intentional, informed choice. If “publicly available” is treated as that substitute, it must plausibly reflect deliberate sharing with knowledge of downstream uses, including automated extraction and model training. The argument that follows uses consent not as a proposed solution for AI scraping, but as a diagnostic test that exposes the flaw in a broad interpretation of §3(c)(ii) that can “publicly available” really stand in for agency when the Act’s own baseline demands clear affirmative action?

This is the DPDP’s legal gap in its sharpest form. A consent based law on one side, and a vague public data carve-out on the other, applied to AI systems. The rest of this analysis asks: First, why §3(c)(ii) cannot mean anything visible online, and must instead require intentional public sharing. Second, if consent cannot realistically be taken at web scale, what practical system can make AI access permission-based so “public” remains public-for-humans, not permissionless for scrapers training models. 

• Intentional Publicness in Comparative Data Protection Law

Before turning to the empirical question of whether consent can fill the gap left, if §3(c)(ii) is read too broadly. Comparative analysis validates that data protection systems refuse to treat mere online visibility as a substitute for intentional permission. European Union (“EU”) regulators consistently draw a distinction between accessible and information that is public by choice. The UK Information Commissioner’s Office cautions that “publicly accessible” data is not necessarily data that the individual chose to make public, while European Data Protection Board (“EDPB”) guidance similarly stresses that default platform settings don’t suffice. The General Data Protection Regulation (“GDPR”) within Article 9(2)(e) contains an exception allowing processing of special category data when the data has been “manifestly made public” by the data subject. Only clear affirmative action and full awareness that data will be public satisfy the requirement. 

In Meta v Bundeskartellamt, the Court of Justice of the European Union (“CJEU”) held that even sensitive data posted on a website is not manifestly public unless the user intended to make it so by clear affirmative action and in advance choice of data subjects made with full knowledge of the facts. As Taner Kuru notes, EU data-protection bodies view this exception as rare and reserve it for situations where data subjects have unambiguously relinquished control. This suggests that §3(c)(ii) of the DPDP should be read as an intentional publicness clause rather than a blanket exemption for anything reachable on the web. European doctrine also accounts for context and downstream use. GDPR recital 47 requires attention to what data subjects could reasonably expect, and the CJEU in Meta, rejected the idea that public posts justify unlimited aggregation where users could not have anticipated such use. Extending this reasoning to AI training, Kuru notes that individuals sharing content online cannot reasonably expect their contributions to be absorbed into large-scale, persistent training corpora that mix sensitive and non-sensitive data and enable opaque downstream inference.

• Why Consent Collapses in Open-Web Scraping

After exploring that visibility cannot substitute for intentional publicness as a matter of principle. The analysis now shows why consent cannot repair that substitution once §3(c)(ii) is read broadly. Our empirical audit confirms that across the sampled 100 + URLs, three conditions that consent presupposes consistently fail.

Consent presupposes that data principals are identifiable and reachable, and that a governance relationship exists. Across the audited URLs, both conditions fail. Contact information is often absent, and there is no contractual framework through which consent can be requested, granted, or withdrawn. Across our 50 URL audit, public webpages overwhelmingly signal human-facing access, and not reuse for training. With limited exceptions such as Wikipedia, most pages are for contextual human reading within professional or social frames.

They contain no affirmative signal authorising general-purpose AI training. Treating such pages as training-ready merely because they are visible collapses the distinction between public attention and permission for reuse. Figure 1 illustrates this divergence between intended human access and unintended scraping pipelines.

Importantly, the web already reflects this distinction through bot-facing intent signals. The widespread use of the Robots Exclusion Protocol (robots.txt) shows that publishers routinely allow human access while restricting automated collection. Empirical research has proven that these signals are frequently ignored particularly by AI crawlers and demonstrates not the absence of intent, but the absence of enforceability.

• Why Governance Works Behind Login Walls and Fails on the Open Web

Taken together, lack of addressability, absence of privity and human-oriented design show that consent is not merely unworkable in scraping, in fact it is structurally mismatched. Our audit of 50 account based websites, show why consent and notice function in closed ecosystems. In these systems, permission precedes access and thus, making governance operational.

The Open Web Has Intent But No Enforcement

The open web, by contrast contains intent but lacks enforcement. This is visible across three common publication models:

1. Newspapers allow human readership, but may also prohibit AI training, directing recuse to licensing. E.g., the Guardian’s robots.txt, as seen below states that uses such as LLMs, machine learning or AI are not permitted, directing would-be users to licensing instead.

2. Personal websites increasingly publish ai.txt or robots.txt files that condition automated reuse through licensing or permissions. While, personal websites may look like the strongest case of data made public by the individual, yet some such as Rasoul Unlimited, regulate how automated systems may interpret, display and reuse their content. 

3. Conditional-permission sites allow training only under explicit constraints like attribution, rate limits, non-commercial use, etc. Web Mimarisi’s ai.txt for example, allows training data use, but only under stated constraints.

Why Scraping Governance Must Be Inverted

All three models show intent. Only the last makes that intent enforceable at the point of access. This reveals a structural asymmetry. On the open web, rules are expressed after access through non-binding signals such as robots.txt and ai.txt, which scrapers can and often do ignore. Once data is extracted en bloc, consent cannot be reconstructed, purpose limitation cannot be enforced, and downstream reuse becomes opaque and irreversible.

A broad reading of §3(c)(ii) compounds this failure. Treating technical accessibility as legal permission removes DPDP protection precisely where consent cannot operate and enforcement is weakest. The solution is inversion. Scraping governance must operate ex ante by making automated access conditional on explicit, machine-readable permission declared before collection. Public data should remain public-for-humans by default, while training-grade machine access should require a clear legal and technical signal. Without this inversion, §3(c)(ii) risks becoming a permanent escape hatch for mass AI extraction.

• The  “ORPA–Agent Wrap” Solution

Fundamentally, to regulate a tech-generated lapse, a tech-enabled solution becomes quintessential. As a policy measure, the authors suggest Operational Readable Permission Architecture (ORPA)–AgentWrap is a targeted governance solution for AI scraping that preserves open human access while making automated extraction auditable and enforceable. It separates visibility from permission as data remains publicly viewable to humans, but high-scale machine access requires permission under §43 of the IT Act, 2000. The core functionality of ORPA Governance thus intervenes only when access shifts from human reading to automated reuse. 

Unlike robots.txt, which merely signals intent without consequence, this model translates intent into verifiable permission, identity, and evidentiary records. By generating more audit trails where access conditions are bypassed, such AgentWrap converts disputes over scraping from abstract claims into enforceable violations. Accordingly, data scraping for AI training cannot be treated as lawful by default merely because personal data is visible online; interpreting §3(c)(ii) of the DPDP as a blanket licence for automated extraction would undermine the statute’s agency-based structure. A permission-first architecture of this kind therefore offers a principled means of preserving the openness of the web while preventing visibility from collapsing into unrestricted machine appropriation. ORPA operates only at the point where access transforms from human use to automated extraction, converting expressive intent into enforceable permission and auditable control.