Understanding llms.txt, AI Guidance Pages, and Updates to robots.txt:

Why They’re Needed and How They Fit Together

As artificial intelligence continues to reshape how information is discovered, indexed, and reused, website owners are facing new challenges.

Traditional web crawling and modern AI-driven content consumption behave very differently, meaning the old rules no longer cover all the scenarios.

Search engines once carried the primary responsibility for indexing content, ranking pages, and directing users to relevant results. However, today large language models (LLMs) are capable of reading, summarising, and even repurposing site content in ways that go far beyond the expectations set when robots.txt was first introduced.

This shift has sparked a growing need for mechanisms that clearly articulate how website content may be accessed and used by AI agents. Three connected tools are emerging to help address this: the familiar robots.txt file, the newer and more AI-specific llms.txt file, and a human-readable AI guidance page. These elements together provide clarity, transparency, and control in a landscape where AI-driven crawling is rapidly becoming the norm.

Why robots.txt Alone Is No Longer Enough

Historically, robots.txt has been used to tell search engine crawlers which parts of a website they may or may not access. Instructions such as Disallow: and Allow: became the standard way for site owners to manage visibility and control crawler load.

However, modern AI crawlers often behave quite differently from traditional bots. They may:

• Access content not for indexing but for training or model improvement.
• Perform broad, site-wide scraping intended to build datasets.
• Use patterns of crawling that weren’t anticipated by the original robots.txt conventions.

Because the original robots.txt specification predates LLMs by decades, it offers no explicit rules around training usage, data retention, or content licensing. While many AI agents still respect robots.txt, it does not provide the granularity required to distinguish between search indexing and AI training—two very different uses of content.

For this reason, many organisations have begun updating their robots.txt files with AI-specific instructions, often naming particular bots or adding clearer disallow directives. These adjustments do not formally extend the standard, but they help signal site owners’ intentions and provide a first level of control.

Introducing llms.txt: A Dedicated Space for AI Permissions

The emergence of llms.txt represents a natural evolution in machine-readable web governance. Placed at the root of your domain, this new file is intended to offer a dedicated, structured set of instructions specifically for LLMs and AI crawlers.

A typical llms.txt file might:

• List AI agents (such as “GPTBot” or “ClaudeBot”) and specify whether they may access content.
• Differentiate between viewing and training permissions.
• Provide links to licensing terms or usage policies.
• Declare allowed and disallowed datasets or retrieval patterns.

While llms.txt is not yet a formal standard, it is rapidly being adopted as a best practice across the industry. It acts as the AI-era equivalent of robots.txt, providing a clean, purpose-built space for communicating expectations to AI systems.

Crucially, llms.txt also supports greater flexibility. Because it’s an emerging convention, it can evolve quickly to reflect the needs of content creators, publishers, educators, and platforms—i.e. groups whose work may be sensitive to how AI models use their data.

The Role of an AI Guidance Page

Alongside these machine-readable files sits the AI guidance page: a human-readable document that explains your site’s AI usage policies in clear, accessible terms.

An AI guidance page typically includes:

• A summary of your site’s stance on AI crawling and training.
• Stating priority content landing pages along with an explanation – content seen as important to your business.
• Explanations of any restrictions or licensing terms.
• Links to your robots.txt and llms.txt files.
• Contact information for queries or permissions.
• Notes on how you expect third parties to handle your content.

This page helps ensure there is no ambiguity. While robots.txt and llms.txt speak to machines, the AI guidance page speaks to people—developers, researchers, AI partners, and users who want to understand how your content may be used.

For organisations concerned about intellectual property, copyright, or brand integrity, this transparency provides an essential layer of protection. For organisations that welcome AI-related usage, it serves as a simple way to communicate permissions and foster collaboration.

How the Three Components Fit Together

These three tools form a coherent ecosystem:

• robots.txt remains the first point of contact for all crawlers, including AI agents that still respect traditional rules.
• llms.txt offers a more detailed, AI-specific set of permissions—addressing gaps that robots.txt cannot fill.
• The AI guidance page provides a human-friendly interpretation of your policies, ensuring clarity and accountability.

Together, they offer website owners a modern framework for managing AI interaction across both technical and legal dimensions. As LLMs continue to expand their role in search, content understanding, and knowledge retrieval, these tools help ensure the balance between innovation and responsible content use remains clear, fair, and transparent.

Exciting time are ahead. Time to embrace it!

For further information or if you need assistance for your business, please contact us either on email enquiries@figadigital.com or call us on, Freephone, 0800 802 1968.

Backlinks are one of the most important elements of search engine optimisation (SEO), and their importance should not be underestimated by anyone looking to grow a website’s visibility online.

At the high-level, a backlink is a hyperlink from one website to another. When another site links to your content, it acts as a signal to search engines, like Google, that your website is credible, trustworthy, and worth referencing. This has a direct impact on how well your website ranks in search engine results pages (SERPs).

One of the main reasons backlinks are so vital is that they help search engines determine the authority and relevance of your website. Google’s algorithms are designed to deliver the most relevant and high-quality results to users. Backlinks essentially serve as a vote of confidence. The more high-quality sites that link to your pages, the more likely it is that Google will view your content as valuable. This boosts your chances of appearing higher in search results, which in turn drives more organic traffic to your website.

However, it’s not just about the number of backlinks—quality matters far more than quantity. Links from well-established, reputable sites carry much more weight than links from high-spam or low-authority sources. For example, a backlink from a national newspaper or a respected industry blog will have a much greater positive effect on your SEO than multiple links from unknown directories or unrelated websites. Earning high-quality backlinks requires well-crafted, useful content that others naturally want to reference or share.

Backlinks are also beneficial because they can drive direct referral traffic. If a user is reading a blog post or an article on another site and clicks on a link to your website, they’ve already shown interest in the topic and are more likely to engage with your content. This type of traffic is highly targeted and often has a better conversion rate than other sources given the visitor is already engaged.

Another key advantage of building backlinks is the opportunity to establish authority within your niche. When your site is regularly cited by others in your industry, it positions you as a thought leader. This can open doors to further opportunities such as guest blogging, media coverage, collaborations, and even partnerships. It’s a cycle that feeds itself—the more your site is seen as authoritative, the more likely others are to link to it.

Backlinks also help with indexing. Search engine bots use links to crawl the web, and having a network of backlinks pointing to your site can make it easier and quicker for search engines to discover and index your new content. Without backlinks, your site might take longer to be recognised or may not be indexed at all.

FIGA Digital are backlinks, SEO and PPC experts with over 10 years of experience delivering value added sales and new clients. Click here to see our quick flipbook on how we undertake backlinks work on behalf of our clients.

For a free Backlinks and SEO review of your business, please contact us on either email enquiries@figadigital.com or call on Freephone 0800 802 1968.

Consent Mode V2 (CMv2), an enhanced version of Google’s Consent Mode, is an advanced tool designed to address the evolving requirements of data privacy and consent management.

This tool is integral for business and websites operating within the European Union (EU) and other regions that enforce stringent data privacy regulations, such as the General Data Protection Regulation (GDPR).

CMv2 allows websites to adjust their behaviour based on the consent status of users, thereby ensuring compliance while maintaining a degree of analytical and advertising functionality.

Overview

CMv2 builds upon its predecessor by providing more granular control and flexibility over how user data is handled, depending on the user’s consent choices. This mode allows websites to specify whether they have received consent for the use of analytics, advertisement personalisation, and other functionalities that involve user data.

When a user visits a website, they are typically prompted with a consent notice, requesting permission for cookies and data processing. With CMv2, the website can adapt its operations based on the user’s response. For instance, if a user consents to analytics but not to personalised ads, the website can adjust its data collection and processing, accordingly, ensuring that it respects the user’s preferences while still gathering valuable insights.

Key Features

• Granular Consent Handling – allows for more detailed consent management, distinguishing between different types of consent such as analytics, advertisement personalisation, and functional cookies. This granularity ensures that user preferences are precisely respected.
• Dynamic Behaviour Adjustment – depending on the consent status, websites can dynamically adjust their behaviour. This means that if a user does not consent to certain types of data processing, the relevant scripts and cookies are either not loaded or are modified to operate in a limited capacity.
• Improved Reporting and Insights – even when full consent is not granted, websites can gather aggregated & non-identifiable data. This allows for a level of performance monitoring and insight gathering without compromising user privacy.
• Compliance with Privacy Regulations – websites can better comply with regulations like GDPR, which mandates that user consent must be obtained prior to data processing activities. This tool helps demonstrate compliance by ensuring that consent preferences are respected and properly managed.
• Integration with Consent Management Platforms (CMPs) – CMv2 is designed to work seamlessly with various CMPs that manage the consent process on websites. This integration ensures that consent preferences are consistently enforced across different tools and platforms.

Business Benefits

Enhanced User Trust

By respecting user consent choices and providing transparency in data handling practices, businesses can enhance trust among their users. Users are increasingly aware of their data privacy rights, and demonstrating a commitment to respecting these rights can improve brand reputation and loyalty.

Optimised Marketing Efforts

CMv2 allows businesses to optimise their marketing efforts even in scenarios where full user consent is not granted. For instance, businesses can still gather valuable insights through aggregated data that can inform marketing strategies and campaign effectiveness without infringing on user privacy.

Legal Compliance

One of the most significant benefits is the ability to comply with stringent data protection regulations. Non-compliance with regulations, like GDPR, can result in hefty fines and legal repercussions. CMv2 provides a structured and reliable way to manage consent, reducing the risk of non-compliance.

Implementation Considerations

Implementing CMv2 requires a thoughtful approach. Businesses need to ensure that their websites and applications are configured correctly to fully leverage the capabilities. This involves integrating with CMPs, setting up appropriate triggers and scripts to handle consent statuses, and continuously monitoring and updating the system to accommodate changes in privacy laws and user preferences.

CMv2 represents a significant advancement in managing user consent and data privacy. By providing granular control, dynamic adjustment capabilities, and integration with CMPs, it allows businesses to balance compliance with privacy regulations and the need for valuable data insights. As data privacy concerns continue to grow, tools like CMv2 will be crucial for businesses aiming to build trust, optimise marketing efforts, and ensure legal compliance.

Subscriptions

There seems to be quite a few CMP’s that tend to require signing up to a monthly subscription if you are a business.

Recommended Plugin

However, through research, we found a free plugin for WordPress, that is totally free, customisable, and without 3^rd party branding.

This plugin is called GDPR Cookie Compliance by Moove Agency.  

Find out more by clicking this WordPress link. We highly recommend this plugin.

If you need some guidance or require us to configure CMv2, please contact us either on email enquiries@figadigital.com or call on Freephone 0800 802 1968.

We can’t believe it, and this is now a hat trick, but we have won our 3rd award, PPC Services Company of the Year 2023, from the SME Awards.

Our first Award was awarded in 2021 for the county of one of our drop-in zones, Hampshire, but the latest two awards are both UK wide.

We do work very hard on generating leads for our SME clients using our own method of PPC & SEO.

To have this recognised, feels like a ‘pat on the back’, which we all need sometimes for sure.

Thank you again to the SME Awards for this amazing recognition.