Should You Let AI Bots Crawl Your Website?
Adobe Express Generative AI (Text-to-Image)
The artificial intelligence (AI) landscape is complex and rapidly evolving with an increasing number of diverse user applications. Large language models (LLMs), like OpenAI’s ChatGPT, leverage publicly available data, much of which is on the internet, in order to both understand and generate original content.
Because a company’s website is a valuable digital asset that can increase overall enterprise value, business owners should carefully consider the opportunities and potential risks of either enabling or disabling the ability of AI bots to consume their website content.
Assessing the Opportunities & Risks
Disadvantages
AI bots may pose the risk of intellectual property or copyright infringement. This has been a major concern for media and publishing industries, artists, and firms with unique product or service descriptions.
In our experience, AI models like ChatGPT do not regularly cite references for the content it generates. Even when prompted, citation results have been inaccurate or out-of-date. (Hopefully, this improves over time.)
Consumption of your company’s website content may pose ethical considerations for data privacy (i.e., healthcare) and misuse.
AI models may make mistakes and develop their own bias based on the content they consume.
Your company’s content may be associated with misleading or misrepresentative content (but this is not a risk necessarily specific to generative AI models).
Advantages
Online visibility: The world’s largest search engine, Google, is enhanced by AI. If you block bots from crawling your website, you may be missing out on organic search traffic.
For example, the Google Search Generative Experience bot will summarize the top 3 positions in the search engine results page. Any other pages with lower rankings are virtually invisible.
Joining the AI landscape may boost your website’s organic website ranking (search engine optimization) over time.
Thought Leadership: Enabling AI bot crawling provides an opportunity for organizations to educate and improve the accuracy of these LLMs, which are consulted by a rapidly growing public audience.
Potential Referrals: A wedding photographer booked a new client, who used ChatGPT to search for “experienced wedding photographers in Portland, Oregon.” His firm ended up in the bot’s shortlist of regional photographers.
Recommendation: Enable AI Crawlers for Websites
After evaluating the advantages and disadvantages of exposing online content to AI bot consumption, business owners’s decisions are ultimately influenced by their industry structure and unique characteristics of their business. Unless your industry or business has concerns about content control and use, we generally recommend allowing AI bots to crawl your company’s website.
While we produce original content on our website and AI presents a potential risk for content ownership, we decided that allowing AI bots to crawl our content is key to improving our search engine visibility over time. We predict that Google’s AI-boosted Search Generative Experience is going to be pivotal driver of website search rankings going forward.
“Organic search is what drives more than half of all website visits and should be an important part of any organization’s digital marketing strategy, whether you’re a multinational mega-corporation or a mom-and-pop shop.”
- Search Engine Journal
By exposing content to the new and improved AI-powered Google search and other AI bots, companies can improve their online visibility, maximize their opportunities for public influence, and increase referrals and conversions.
If you are in the process of determining whether the advantages of enabling the consumption of your online content by AI bots outweigh the risks, please contact us to develop a unique website strategy designed to enhance your company’s value.
How Do AI Models See Themselves?
We experimented with text-to-image AI generation for this article. Our first prompt of “AI bot” resulted in series of images depicting a dolphin swimming. It makes one wonder how the various AI models see themselves…
Related Article
Manage Your Digital Assets to Maximize Company Value
Good digital asset management practices protect your business and allow you to leverage up on what makes your brand and business unique.
