agentage.org / Articles / Guide
Respecting Text and Data Mining Opt-Outs: Collection Ethics
A practical guide to navigating EU disclosure laws, machine-readable opt-outs, and the strict boundary between analysis and publication.
The TDM Exception in Brief
The European Union recognizes a specific legal carve-out for Text and Data Mining (TDM) within copyright law. This exception allows researchers and commercial entities to copy and analyze copyrighted works for the purpose of pattern recognition and data extraction. The law distinguishes between the act of mining for insights and the act of redistributing the source material itself. Agencies operating under this rule must ensure their activities remain within the scope of analysis rather than reproduction.
This legal framework applies to content that is publicly accessible on the internet. It does not grant permission to bypass paywalls or access content that requires a subscription fee. The exception assumes the data is available to the public without technical barriers like mandatory login credentials. If a site blocks access or charges for entry, the TDM exception generally does not apply to those specific restricted areas.
The core requirement is that the resulting output must not substitute the original work. An agency can extract facts, trends, and structural data from a corpus of articles. However, the final published product must be a new synthesis or a summary that adds value. Direct copying of headlines, lead paragraphs, or full articles violates the spirit and letter of the regulation, regardless of the mining process used.
Machine-Readable Opt-Outs
Publishers now have the right to explicitly forbid text and data mining through machine-readable signals. These signals are embedded directly into the code of the webpage or the server configuration. If a site owner sets a TDM opt-out flag, automated crawlers must respect this instruction immediately. Ignoring such a signal transforms a potentially legal activity into a breach of the publisher's rights.
These opt-outs often appear as specific directives in the robots.txt file or as custom meta tags within the HTML header. The technology allows site owners to target specific types of agents, such as those used for training large language models. A crawler designed for ethical compliance checks for these signals before initiating any download or parsing process. Failure to detect these flags is not considered a valid defense in legal disputes.
The presence of an opt-out means the data is off-limits for automated analysis. This applies even if the content is visible to a human user in a browser. The machine-readable nature of the restriction ensures that software agents can process the rule without human intervention. Agencies must update their scanning logic regularly to recognize new standards as they emerge from industry groups and regulatory bodies.
- Check for specific TDM directives in the robots.txt file before crawling.
- Scan HTML meta tags for opt-out signals targeting AI training or mining.
- Respect the opt-out immediately upon detection without attempting workarounds.
- Maintain logs of opt-out checks to demonstrate compliance during audits.
Robots.txt in Practice
The robots.txt file serves as the primary communication channel between website owners and web crawlers. It lists the paths that bots are allowed or forbidden to access. While traditionally used for search engine indexing, it now functions as a critical compliance tool for AI data collection. A standard directive like 'Disallow: /' blocks all automated access to the site entirely.
Modern implementations extend beyond simple path blocking. They can specify different rules for different user agents. A news aggregator might be allowed to read headlines while a training bot is blocked from accessing full articles. Crawlers must parse these rules accurately for every specific agent identity they deploy. Misidentifying the user agent can lead to accidental violations of the site owner's wishes.
Compliance requires more than just reading the file once. Websites update their robots.txt frequently to reflect changing policies or technical needs. An ethical collection pipeline re-checks this file at regular intervals or before every new crawl session. If a site changes its rules to block a previously allowed path, the crawler must stop fetching that content immediately.
Rate limiting is another essential component of respecting robots.txt. Even if a path is allowed, aggressive fetching can overwhelm a server. Best practices dictate a delay between requests, often ranging from one to fifteen seconds depending on the site size. This prevents the crawler from causing denial-of-service conditions or degrading the experience for human visitors.
Analysis Storage Versus Publication
A strict separation exists between storing data for analysis and publishing it for readers. The TDM exception permits the temporary or permanent storage of full texts solely for the purpose of computational analysis. This storage must be secure and inaccessible to the general public. The moment the stored text is displayed to a user as a news story, the legal basis shifts from mining to distribution.
Agencies must implement technical controls to prevent accidental leakage of raw text. Databases used for vector search or model training should be isolated from the public-facing content management system. Only derived insights, summaries, or factual statements extracted from the text should enter the publication workflow. The original source text remains in the secure analysis environment.
When synthesizing news from multiple sources, the agency must create original wording. Relying on the exact phrasing of the source article constitutes copyright infringement. The output must be a new work that cites the source but does not reproduce its expression. This distinction protects the intellectual property of the original publisher while allowing the agency to report on the underlying facts.
Images and multimedia assets require even stricter handling. Unlike text, which has specific exceptions for factual reporting, images rarely fall under fair use for news aggregation. Agencies should avoid downloading or displaying images from source sites unless they hold a license or the image is explicitly marked as free to use. Using a screenshot of an article for analysis is different from publishing that screenshot as part of a story.
Platform Terms and APIs
Many major platforms restrict automated access through their Terms of Service (ToS). These agreements often override general legal exceptions by contract. For example, social media networks may prohibit scraping even if the TDM exception technically applies to the content. Violating a ToS can lead to IP bans, legal action, or loss of API access. Agencies must review the ToS of every platform they intend to crawl.
Official APIs provide a compliant alternative to scraping for many data sources. Platforms like Reddit, YouTube, and academic databases offer structured interfaces for data retrieval. These APIs often include built-in rate limits and usage policies that simplify compliance. Using an API ensures that the data is delivered in a format intended for machine consumption, reducing the risk of technical violations.
Third-party scraping services exist but carry their own risks. Some providers operate in legal gray areas by using residential proxies to bypass blocks. Relying on such services can implicate the hiring agency in the provider's non-compliant behavior. Ethical agencies prefer direct interaction with the source or authorized intermediaries that guarantee adherence to robots.txt and ToS.
The landscape of platform rules changes rapidly. A policy that permits scraping today might be revoked tomorrow. Agencies must monitor announcements from platform operators and adjust their collection strategies accordingly. Staying informed about legal developments, such as recent lawsuits against scraping firms, helps organizations anticipate future restrictions and adapt their workflows proactively.
Key takeaways
- The EU TDM exception allows analysis of public content only if no machine-readable opt-out exists.
- Full texts can be stored for internal mining but must never be republished as original content.
- Robots.txt directives and platform Terms of Service are binding constraints for all automated crawlers.
- Synthesized news must use original wording and cannot reproduce the source's specific expression.
- Official APIs are the safest route for collecting data from social media and academic platforms.
Related questions
Can I scrape a news site if it has no paywall?
Not necessarily. Even without a paywall, the site may have a robots.txt file or a TDM opt-out signal that forbids automated access. You must check these machine-readable instructions before starting any crawl. If an opt-out is present, the content is off-limits for mining regardless of its public visibility.
Is it legal to store scraped articles for later analysis?
Under the EU TDM exception, storing full texts for analysis is permitted provided the publisher has not opted out. However, this storage must be secure and used strictly for computational purposes. You cannot publish the stored text or substantial parts of it without a separate license or explicit permission from the copyright holder.
How do I handle images from source websites?
Images generally do not fall under the same text-mining exceptions as written content. You should avoid downloading or displaying images from other publishers unless you have a license or the image is explicitly marked as free to use. Rely on your own photography or licensed stock images instead of scraping visual assets.
AI-assisted: drafted with a language model from the author's research notes, then edited and checked before publication. Author: Zoltán Varga, AI visibility researcher and AI ecosystem architect, Budapest.
