{"id":67299,"date":"2025-05-12T07:00:00","date_gmt":"2025-05-12T14:00:00","guid":{"rendered":"https:\/\/dhblog.dream.press\/blog\/?p=67299"},"modified":"2025-05-26T11:19:20","modified_gmt":"2025-05-26T18:19:20","slug":"block-web-crawlers","status":"publish","type":"post","link":"https:\/\/www-dev.dreamhost.com\/blog\/block-web-crawlers\/","title":{"rendered":"How To Stop Web Crawlers and Bots From Slowing Down Your Website"},"content":{"rendered":"\n<p>Your website is like a coffee shop. People come in and browse the menu. Some order lattes, sit, sip, and leave.<\/p>\n\n\n\n<p>But what if half your &#8220;customers&#8221; just occupy tables, waste your baristas&#8217; time, and never buy coffee?<\/p>\n\n\n\n<p>Meanwhile, real customers leave due to no tables and slow service?<\/p>\n\n\n\n<p>Well, that\u2019s the world of web crawlers and bots.<\/p>\n\n\n\n<p>These automated programs gobble up your bandwidth, slow down your site, and drive away actual customers.<\/p>\n\n\n\n<p>Recent studies show that <a target=\"_blank\" href=\"https:\/\/www.imperva.com\/resources\/resource-library\/reports\/2025-bad-bot-report\/\">almost 51% of internet traffic<\/a> comes from bots. That&#8217;s right \u2014 <strong>more than half<\/strong> of your digital visitors may just be wasting your server resources.<\/p>\n\n\n\n<p>But don&#8217;t panic!<\/p>\n\n\n\n<p>This guide will help you spot trouble and control your site&#8217;s performance, all without coding or calling your techy cousin.<\/p>\n\n\n\n<h2 id=\"h-a-quick-refresher-on-bots\" class=\"wp-block-heading\">A Quick Refresher on Bots<\/h2>\n\n\n\n<p>Bots are automated software programs that perform tasks on the internet without human intervention. They:<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Visit websites<\/li>\n\n\n\n<li>Interact with digital content<\/li>\n\n\n\n<li>And execute specific functions based on their programming.&nbsp;<\/li>\n<\/ul>\n\n\n\n<p>Some bots analyze and index your site (potentially improving search engine rankings.) Some spend their time scraping your content for AI training datasets \u2014 or worse \u2014 posting spam, generating fake reviews, or looking for exploits and security holes in your website.<\/p>\n\n\n\n<p>Of course, not all bots are created equal. Some are critical to the health and visibility of your website. Others are arguably neutral, and a few are downright toxic. Knowing the difference \u2014 and deciding which bots to block and which to allow \u2014 is crucial for protecting your site and its reputation.<\/p>\n\n\n\n<h2 id=\"h-good-bot-bad-bot-what-s-what\" class=\"wp-block-heading\">Good Bot, Bad Bot: What\u2019s What?<\/h2>\n\n\n\n<figure class=\"wp-block-image size-full\"><img decoding=\"async\" width=\"1600\" height=\"1334\" data-src=\"https:\/\/www.dreamhost.com\/blog\/wp-content\/uploads\/2025\/05\/01_bad_bot_vs_good_bot_vs_human_traffic_2024.webp\" alt=\"Donut chart of web traffic 2023 showing Human at 49% (-1.4%), Bad Bot at 37% (+5%), and Good Bot at 14% (-3.6%).\" class=\"wp-image-67301 lazyload\" data-srcset=\"https:\/\/www-dev.dreamhost.com\/blog\/wp-content\/uploads\/2025\/05\/01_bad_bot_vs_good_bot_vs_human_traffic_2024.webp 1600w, https:\/\/www-dev.dreamhost.com\/blog\/wp-content\/uploads\/2025\/05\/01_bad_bot_vs_good_bot_vs_human_traffic_2024-300x250.webp 300w, https:\/\/www-dev.dreamhost.com\/blog\/wp-content\/uploads\/2025\/05\/01_bad_bot_vs_good_bot_vs_human_traffic_2024-1024x854.webp 1024w, https:\/\/www-dev.dreamhost.com\/blog\/wp-content\/uploads\/2025\/05\/01_bad_bot_vs_good_bot_vs_human_traffic_2024-768x640.webp 768w, https:\/\/www-dev.dreamhost.com\/blog\/wp-content\/uploads\/2025\/05\/01_bad_bot_vs_good_bot_vs_human_traffic_2024-1536x1281.webp 1536w, https:\/\/www-dev.dreamhost.com\/blog\/wp-content\/uploads\/2025\/05\/01_bad_bot_vs_good_bot_vs_human_traffic_2024-600x500.webp 600w, https:\/\/www-dev.dreamhost.com\/blog\/wp-content\/uploads\/2025\/05\/01_bad_bot_vs_good_bot_vs_human_traffic_2024-1200x1001.webp 1200w, https:\/\/www-dev.dreamhost.com\/blog\/wp-content\/uploads\/2025\/05\/01_bad_bot_vs_good_bot_vs_human_traffic_2024-730x609.webp 730w, https:\/\/www-dev.dreamhost.com\/blog\/wp-content\/uploads\/2025\/05\/01_bad_bot_vs_good_bot_vs_human_traffic_2024-1460x1217.webp 1460w, https:\/\/www-dev.dreamhost.com\/blog\/wp-content\/uploads\/2025\/05\/01_bad_bot_vs_good_bot_vs_human_traffic_2024-784x654.webp 784w, https:\/\/www-dev.dreamhost.com\/blog\/wp-content\/uploads\/2025\/05\/01_bad_bot_vs_good_bot_vs_human_traffic_2024-1568x1307.webp 1568w, https:\/\/www-dev.dreamhost.com\/blog\/wp-content\/uploads\/2025\/05\/01_bad_bot_vs_good_bot_vs_human_traffic_2024-877x731.webp 877w\" data-sizes=\"(max-width: 1600px) 100vw, 1600px\" src=\"data:image\/svg+xml;base64,PHN2ZyB3aWR0aD0iMSIgaGVpZ2h0PSIxIiB4bWxucz0iaHR0cDovL3d3dy53My5vcmcvMjAwMC9zdmciPjwvc3ZnPg==\" style=\"--smush-placeholder-width: 1600px; --smush-placeholder-aspect-ratio: 1600\/1334;\" \/><\/figure>\n\n\n\n<p>Bots make up the internet.<\/p>\n\n\n\n<p>For instance, Google\u2019s bot visits every page on the internet and adds it to their databases for ranking. This bot assists in providing valuable search traffic, which is important for the health of your website.<\/p>\n\n\n\n<p>But, not every bot is going to provide value, and some are just outright bad. Here\u2019s what to keep and what to block.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">The VIP Bots (Keep These)<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Search engine crawlers like <a target=\"_blank\" href=\"https:\/\/www.dreamhost.com\/glossary\/seo\/googlebot\/\">Googlebot<\/a> and Bingbot are examples of these crawlers. Don\u2019t block them, or you\u2019ll become invisible online.<\/li>\n\n\n\n<li>Analytics bots gather data about your site&#8217;s performance, like the <a target=\"_blank\" href=\"https:\/\/www.dreamhost.com\/blog\/pagespeed-insights-report\/\">Google Pagespeed Insights<\/a> bot or the GTmetrix bot.<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\">The Troublemakers (Need Managing)<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Content scrapers that steal your content for use elsewhere<\/li>\n\n\n\n<li>Spam bots that flood your forms and comments with junk<\/li>\n\n\n\n<li>Bad actors who attempt to hack accounts or exploit vulnerabilities<\/li>\n<\/ul>\n\n\n\n<p>The bad bots scale might surprise you. In 2024, advanced bots made up <a target=\"_blank\" href=\"https:\/\/www.imperva.com\/resources\/resource-library\/reports\/2025-bad-bot-report\/\">55%<\/a> of all advanced bad bot traffic, while good ones accounted for 44%.<\/p>\n\n\n\n<p>Those advanced bots are sneaky \u2014 they can mimic human behavior, including mouse movements and clicks, making them more difficult to detect.<\/p>\n\n\n\n<div class=\"article-newsletter article-newsletter--gradient\">\n\n\n<h2>Get Content Delivered Straight to Your Inbox<\/h2><p>Subscribe now to receive all the latest updates, delivered directly to your inbox.<\/p><form class=\"nwsl-form\" id=\"newsletter_block_\" novalidate><div class=\"messages\"><\/div><div class=\"form-group\"><label for=\"input_newsletter_block_\"><input type=\"email\"name=\"email\"id=\"input_newsletter_block_\"placeholder=\"Enter your email address\"novalidatedisabled=\"disabled\"\/><\/label><button type=\"submit\"class=\"btn btn--brand\"disabled=\"disabled\"><span>Sign Me Up!<\/span><svg width=\"21\" height=\"14\" viewBox=\"0 0 21 14\" fill=\"none\" xmlns=\"http:\/\/www.w3.org\/2000\/svg\">\n<path d=\"M13.8523 0.42524L12.9323 1.34521C12.7095 1.56801 12.7132 1.9304 12.9404 2.14865L16.7241 5.7823H0.5625C0.251859 5.7823 0 6.03416 0 6.3448V7.6573C0 7.96794 0.251859 8.2198 0.5625 8.2198H16.7241L12.9405 11.8535C12.7132 12.0717 12.7095 12.4341 12.9323 12.6569L13.8523 13.5769C14.072 13.7965 14.4281 13.7965 14.6478 13.5769L20.8259 7.39879C21.0456 7.17913 21.0456 6.82298 20.8259 6.60327L14.6477 0.42524C14.4281 0.205584 14.0719 0.205584 13.8523 0.42524Z\" fill=\"white\"\/>\n<\/svg>\n<\/button><\/div><\/form><\/div>\n\n\n<h2 id=\"h2_are-bots-bogging-down-your-website-look-for-these-warning-signs\" class=\"wp-block-heading\">Are Bots Bogging Down Your Website? Look for These Warning Signs<\/h2>\n\n\n\n<p>Before jumping into solutions, let&#8217;s make sure bots are actually your problem. Check out the signs below.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">Red Flags in Your Analytics<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>Traffic spikes without explanation:<\/strong> If your visitor count suddenly jumps but sales don&#8217;t, bots might be the culprit.<\/li>\n\n\n\n<li><strong>Everything s-l-o-w-s down:<\/strong> Pages take longer to load, frustrating real customers who might leave for good. Aberdeen shows that <a target=\"_blank\" href=\"https:\/\/www.aberdeen.com\/techpro-essentials\/gone-in-three-seconds-your-customers-have-no-patience-for-a-slow-website\/\">40% of visitors<\/a> abandon websites that take over three seconds to load, which leads to\u2026<\/li>\n\n\n\n<li><strong>High bounce rates:<\/strong> above 90% often indicate bot activity.<\/li>\n\n\n\n<li><strong>Weird session patterns:<\/strong> Humans don&#8217;t typically visit for just milliseconds or stay on one page for hours.<\/li>\n\n\n\n<li><strong>You start getting<\/strong><a target=\"_blank\" href=\"https:\/\/www.dreamhost.com\/blog\/driving-traffic-to-your-website\/\"><strong> <\/strong><strong>lots of unusual traffic<\/strong><\/a><strong>:<\/strong> Especially from countries where you don&#8217;t do business. That&#8217;s suspicious.<\/li>\n\n\n\n<li><strong>Form submissions with random text:<\/strong> Classic bot behavior.<\/li>\n\n\n\n<li><strong>Your server gets overwhelmed:<\/strong> Imagine seeing 100 customers at once, but 75 are just window shopping.<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\">Check Your Server Logs<\/h3>\n\n\n\n<p>Your website&#8217;s server logs contain records of every visitor.<\/p>\n\n\n\n<p><strong>Here&#8217;s what to look for:<\/strong><\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Too many subsequent requests from the same IP address<\/li>\n\n\n\n<li>Strange user-agent strings (the identification that bots provide)<\/li>\n\n\n\n<li>Requests for unusual URLs that don&#8217;t exist on your site<\/li>\n<\/ul>\n\n\n\n\n\n<div class=\"glossary-term\">\n\t<a\n\t\tclass=\"glossary-term__above-title\"\n\t\thref=\"\"\n\t\ttarget=\"_blank\"\n\t\trel=\"noopener noreferrer\"\n\t>\n\t\t<svg xmlns=\"http:\/\/www.w3.org\/2000\/svg\" viewBox=\"0 0 640 512\"><path d=\"M320 32c-8.1 0-16.1 1.4-23.7 4.1L15.8 137.4C6.3 140.9 0 149.9 0 160s6.3 19.1 15.8 22.6l57.9 20.9C57.3 229.3 48 259.8 48 291.9v28.1c0 28.4-10.8 57.7-22.3 80.8c-6.5 13-13.9 25.8-22.5 37.6C0 442.7-.9 448.3 .9 453.4s6 8.9 11.2 10.2l64 16c4.2 1.1 8.7 .3 12.4-2s6.3-6.1 7.1-10.4c8.6-42.8 4.3-81.2-2.1-108.7C90.3 344.3 86 329.8 80 316.5V291.9c0-30.2 10.2-58.7 27.9-81.5c12.9-15.5 29.6-28 49.2-35.7l157-61.7c8.2-3.2 17.5 .8 20.7 9s-.8 17.5-9 20.7l-157 61.7c-12.4 4.9-23.3 12.4-32.2 21.6l159.6 57.6c7.6 2.7 15.6 4.1 23.7 4.1s16.1-1.4 23.7-4.1L624.2 182.6c9.5-3.4 15.8-12.5 15.8-22.6s-6.3-19.1-15.8-22.6L343.7 36.1C336.1 33.4 328.1 32 320 32zM128 408c0 35.3 86 72 192 72s192-36.7 192-72L496.7 262.6 354.5 314c-11.1 4-22.8 6-34.5 6s-23.5-2-34.5-6L143.3 262.6 128 408z\"\/><\/svg>\n\t\t<span><\/span>\n\t\t<svg xmlns=\"http:\/\/www.w3.org\/2000\/svg\" viewBox=\"0 0 384 512\"><path d=\"M342.6 233.4c12.5 12.5 12.5 32.8 0 45.3l-192 192c-12.5 12.5-32.8 12.5-45.3 0s-12.5-32.8 0-45.3L274.7 256 105.4 86.6c-12.5-12.5-12.5-32.8 0-45.3s32.8-12.5 45.3 0l192 192z\"\/><\/svg>\n\t<\/a>\n    <h3>User Agent<\/h3>\n    <p>A user agent is a type of software that retrieves and renders web content so that users can interact with it. The most common examples are web browsers and email readers.<\/p>\n            <a\n            href=\"https:\/\/www.dreamhost.com\/glossary\/seo\/user-agent\/\"\n                        class=\"btn btn--white-outline btn--sm btn--round\"\n                                    target=\"_blank\"\n            rel=\"noopener noreferrer\"\n            >\n                            Read More                    <\/a>\n\n<\/div>\n\n\n\n<p>A legitimate Googlebot request might look like this in your logs:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>66.249.78.17 - - &#091;13\/Jul\/2015:07:18:58 -0400] \"GET \/robots.txt HTTP\/1.1\" 200 0 \"-\" \"Mozilla\/5.0 (compatible; Googlebot\/2.1; +http:\/\/www.google.com\/bot.html)\"<\/code><\/pre>\n\n\n\n<p>If you see patterns that don&#8217;t match normal human browsing behavior, it&#8217;s time to take action.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">The GPTBot Problem as AI Crawlers Surge<\/h3>\n\n\n\n<p>Recently, many website owners have reported issues with AI crawlers generating abnormal traffic patterns.<\/p>\n\n\n\n<p>According to Imperva&#8217;s research, OpenAI&#8217;s <a target=\"_blank\" href=\"https:\/\/vercel.com\/blog\/the-rise-of-the-ai-crawler\">GPTBot made 569 million requests<\/a> in a single month while Claude&#8217;s bot made 370 million across Vercel&#8217;s network.<\/p>\n\n\n\n<p>Look for:<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>Error spikes in your logs:<\/strong> If you suddenly see hundreds or thousands of 404 errors, check if they&#8217;re from AI crawlers.<\/li>\n<\/ul>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>Extremely long, nonsensical URLs:<\/strong> AI bots might request bizarre URLs like the following:<\/li>\n<\/ul>\n\n\n\n<pre class=\"wp-block-code\"><code>\/Odonto-lieyectoresli-541.aspx\/assets\/js\/plugins\/Docs\/Productos\/assets\/js\/Docs\/Productos\/assets\/js\/assets\/js\/assets\/js\/vendor\/images2021\/Docs\/...<\/code><\/pre>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>Recursive parameters<\/strong>: Look for endless repeating parameters, for example:<\/li>\n<\/ul>\n\n\n\n<pre class=\"wp-block-code\"><code>amp;amp;amp;page=6&amp;amp;amp;amp;page=6<\/code><\/pre>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>Bandwidth spikes:<\/strong> Readthedocs, a renowned technical documentation company, stated that one AI crawler downloaded <a target=\"_blank\" href=\"https:\/\/about.readthedocs.com\/blog\/2024\/07\/ai-crawlers-abuse\/\">73TB of ZIP files<\/a>, with 10TB downloaded in a single day, costing them over $5,000 in bandwidth charges.<\/li>\n<\/ul>\n\n\n\n<p>These patterns can indicate AI crawlers that are either malfunctioning or being manipulated to cause problems.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">When To Get Technical Help<\/h3>\n\n\n\n<p>If you spot these signs but don&#8217;t know what to do next, it&#8217;s time to bring in professional help. Ask your developer to check specific user agents like this one:<\/p>\n\n\n\n<p><strong>Mozilla\/5.0 AppleWebKit\/537.36 (KHTML, like Gecko; compatible; GPTBot\/1.2; +<\/strong><a target=\"_blank\" href=\"https:\/\/openai.com\/gptbot\"><strong>https:\/\/openai.com\/gptbot<\/strong><\/a><strong>)<\/strong><\/p>\n\n\n\n<p>There are many recorded <a target=\"_blank\" href=\"https:\/\/developers.netlify.com\/guides\/blocking-ai-bots-and-controlling-crawlers\/\">user agent strings for other AI crawlers<\/a> that you can look up on Google to block. <strong>Do note<\/strong> that the strings change, meaning you might end up with quite a large list over time.<\/p>\n\n\n\n<p><em>\ud83d\udc49 Don\u2019t have a developer on speed dial? <\/em><a target=\"_blank\" href=\"https:\/\/www.dreamhost.com\/pro-services\/management\/dreamcare\/\"><em>DreamHost&#8217;s DreamCare team<\/em><\/a><em> can analyze your logs and implement protection measures. They&#8217;ve seen these issues before and know exactly how to handle them.<\/em><\/p>\n\n\n\n<h2 id=\"h2_your-bot-busting-toolkit-5-simple-steps-to-take-back-control\" class=\"wp-block-heading\">Your Bot-Busting Toolkit: 5 Simple Steps To Take Back Control<\/h2>\n\n\n\n<p>Now for the good part: how to stop these bots from slowing down your site. Roll up your sleeves and let&#8217;s get to work.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">1. Create a Proper robots.txt File<\/h3>\n\n\n\n<figure class=\"wp-block-image size-full\"><img decoding=\"async\" width=\"1600\" height=\"911\" data-src=\"https:\/\/www.dreamhost.com\/blog\/wp-content\/uploads\/2025\/05\/02_create_a_proper_robots_txt_file.webp\" alt=\"robots.txt file from dreamhost.com showing crawler directives that disallow indexing of specific paths while providing sitemap location.\" class=\"wp-image-67302 lazyload\" data-srcset=\"https:\/\/www-dev.dreamhost.com\/blog\/wp-content\/uploads\/2025\/05\/02_create_a_proper_robots_txt_file.webp 1600w, https:\/\/www-dev.dreamhost.com\/blog\/wp-content\/uploads\/2025\/05\/02_create_a_proper_robots_txt_file-300x171.webp 300w, https:\/\/www-dev.dreamhost.com\/blog\/wp-content\/uploads\/2025\/05\/02_create_a_proper_robots_txt_file-1024x583.webp 1024w, https:\/\/www-dev.dreamhost.com\/blog\/wp-content\/uploads\/2025\/05\/02_create_a_proper_robots_txt_file-768x437.webp 768w, https:\/\/www-dev.dreamhost.com\/blog\/wp-content\/uploads\/2025\/05\/02_create_a_proper_robots_txt_file-1536x875.webp 1536w, https:\/\/www-dev.dreamhost.com\/blog\/wp-content\/uploads\/2025\/05\/02_create_a_proper_robots_txt_file-600x342.webp 600w, https:\/\/www-dev.dreamhost.com\/blog\/wp-content\/uploads\/2025\/05\/02_create_a_proper_robots_txt_file-1200x683.webp 1200w, https:\/\/www-dev.dreamhost.com\/blog\/wp-content\/uploads\/2025\/05\/02_create_a_proper_robots_txt_file-730x416.webp 730w, https:\/\/www-dev.dreamhost.com\/blog\/wp-content\/uploads\/2025\/05\/02_create_a_proper_robots_txt_file-1460x831.webp 1460w, https:\/\/www-dev.dreamhost.com\/blog\/wp-content\/uploads\/2025\/05\/02_create_a_proper_robots_txt_file-784x446.webp 784w, https:\/\/www-dev.dreamhost.com\/blog\/wp-content\/uploads\/2025\/05\/02_create_a_proper_robots_txt_file-1568x893.webp 1568w, https:\/\/www-dev.dreamhost.com\/blog\/wp-content\/uploads\/2025\/05\/02_create_a_proper_robots_txt_file-877x499.webp 877w\" data-sizes=\"(max-width: 1600px) 100vw, 1600px\" src=\"data:image\/svg+xml;base64,PHN2ZyB3aWR0aD0iMSIgaGVpZ2h0PSIxIiB4bWxucz0iaHR0cDovL3d3dy53My5vcmcvMjAwMC9zdmciPjwvc3ZnPg==\" style=\"--smush-placeholder-width: 1600px; --smush-placeholder-aspect-ratio: 1600\/911;\" \/><\/figure>\n\n\n\n<p>The robots.txt simple text file sits in your root directory and tells well-behaved bots which parts of your site they shouldn&#8217;t access.<\/p>\n\n\n\n<p>You can access the robots.txt for pretty much any website by adding a \/robots.txt to its domain. For instance, if you want to see the robots.txt file for DreamHost, add robots.txt at the end of the domain like this: <a target=\"_blank\" href=\"https:\/\/dreamhost.com\/robots.txt\">https:\/\/dreamhost.com\/robots.txt<\/a><\/p>\n\n\n\n<p>There\u2019s no obligation for any of the bots to accept the rules.<\/p>\n\n\n\n<p>But polite bots will respect it, and the troublemakers can choose to ignore the rules. It\u2019s best to add a robots.txt anyway so the good bots don\u2019t start indexing admin login, post-checkout pages, thank you pages, etc.<\/p>\n\n\n\n<p><strong>How to Implement<\/strong><\/p>\n\n\n\n<p>1. Create a plain text file named <strong>robots.txt<\/strong><\/p>\n\n\n\n<p>2. Add your instructions using this format:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>User-agent: *          # This line applies to all bots\nDisallow: \/admin\/      # Don't crawl the admin area\nDisallow: \/private\/    # Stay out of private folders\nCrawl-delay: 10        # Wait 10 seconds between requests\nUser-agent: Googlebot  # Special rules just for Google\nAllow: \/               # Google can access everything<\/code><\/pre>\n\n\n\n<p>3. Upload the file to your website&#8217;s root directory (so it&#8217;s at<strong> yourdomain.com\/robots.txt<\/strong>)<\/p>\n\n\n\n<p>The &#8220;Crawl-delay&#8221; directive is your secret weapon here. It forces bots to wait between requests, preventing them from hammering your server.<\/p>\n\n\n\n<p>Most major crawlers respect this, although Googlebot follows its own system (which you can control through Google Search Console).<\/p>\n\n\n\n<p><strong>Pro tip: <\/strong>Test your robots.txt with <a target=\"_blank\" href=\"https:\/\/www.google.com\/webmasters\/tools\/robots-testing-tool\">Google&#8217;s robots.txt testing tool<\/a> to ensure you haven&#8217;t accidentally blocked important content.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">2. Set Up Rate Limiting<\/h3>\n\n\n\n<p>Rate limiting restricts how many requests a single visitor can make within a specific period.<\/p>\n\n\n\n<p>It prevents bots from overwhelming your server so normal humans can browse your site without interruption.<\/p>\n\n\n\n<p><strong>How to Implement<\/strong><\/p>\n\n\n\n<p>If you&#8217;re using Apache (common for WordPress sites), add these lines to your .htaccess file:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>&lt;IfModule mod_rewrite.c&gt;\nRewriteEngine On\nRewriteCond %{REQUEST_URI} !(.css|.js|.png|.jpg|.gif|robots.txt)$ &#091;NC]\nRewriteCond %{HTTP_USER_AGENT} !^Googlebot &#091;NC]\nRewriteCond %{HTTP_USER_AGENT} !^Bingbot &#091;NC]\n# Allow max 3 requests in 10 seconds per IP\nRewriteCond %{REMOTE_ADDR} ^(&#091;0-9]+.&#091;0-9]+.&#091;0-9]+.&#091;0-9]+)$\nRewriteRule .* - &#091;F,L]\n&lt;\/IfModule&gt;<\/code><\/pre>\n\n\n\n\n\n<div class=\"glossary-term\">\n\t<a\n\t\tclass=\"glossary-term__above-title\"\n\t\thref=\"\"\n\t\ttarget=\"_blank\"\n\t\trel=\"noopener noreferrer\"\n\t>\n\t\t<svg xmlns=\"http:\/\/www.w3.org\/2000\/svg\" viewBox=\"0 0 640 512\"><path d=\"M320 32c-8.1 0-16.1 1.4-23.7 4.1L15.8 137.4C6.3 140.9 0 149.9 0 160s6.3 19.1 15.8 22.6l57.9 20.9C57.3 229.3 48 259.8 48 291.9v28.1c0 28.4-10.8 57.7-22.3 80.8c-6.5 13-13.9 25.8-22.5 37.6C0 442.7-.9 448.3 .9 453.4s6 8.9 11.2 10.2l64 16c4.2 1.1 8.7 .3 12.4-2s6.3-6.1 7.1-10.4c8.6-42.8 4.3-81.2-2.1-108.7C90.3 344.3 86 329.8 80 316.5V291.9c0-30.2 10.2-58.7 27.9-81.5c12.9-15.5 29.6-28 49.2-35.7l157-61.7c8.2-3.2 17.5 .8 20.7 9s-.8 17.5-9 20.7l-157 61.7c-12.4 4.9-23.3 12.4-32.2 21.6l159.6 57.6c7.6 2.7 15.6 4.1 23.7 4.1s16.1-1.4 23.7-4.1L624.2 182.6c9.5-3.4 15.8-12.5 15.8-22.6s-6.3-19.1-15.8-22.6L343.7 36.1C336.1 33.4 328.1 32 320 32zM128 408c0 35.3 86 72 192 72s192-36.7 192-72L496.7 262.6 354.5 314c-11.1 4-22.8 6-34.5 6s-23.5-2-34.5-6L143.3 262.6 128 408z\"\/><\/svg>\n\t\t<span><\/span>\n\t\t<svg xmlns=\"http:\/\/www.w3.org\/2000\/svg\" viewBox=\"0 0 384 512\"><path d=\"M342.6 233.4c12.5 12.5 12.5 32.8 0 45.3l-192 192c-12.5 12.5-32.8 12.5-45.3 0s-12.5-32.8 0-45.3L274.7 256 105.4 86.6c-12.5-12.5-12.5-32.8 0-45.3s32.8-12.5 45.3 0l192 192z\"\/><\/svg>\n\t<\/a>\n    <h3>.htaccess<\/h3>\n    <p>\u201c.htaccess\u201d is a configuration file used by the Apache web server software. The .htaccess file contains directives (instructions) that tell Apache how to behave for a particular website or directory.<\/p>\n            <a\n            href=\"https:\/\/www.dreamhost.com\/glossary\/hosting\/htaccess\/\"\n                        class=\"btn btn--white-outline btn--sm btn--round\"\n                                    target=\"_blank\"\n            rel=\"noopener noreferrer\"\n            >\n                            Read More                    <\/a>\n\n<\/div>\n\n\n\n<p>If you&#8217;re on Nginx, add this to your server configuration:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>limit_req_zone $binary_remote_addr zone=one:10m rate=30r\/m;\nserver {\n    ...\n    location \/ {\n        limit_req zone=one burst=5;\n        ...\n    }\n}<\/code><\/pre>\n\n\n\n<p>Many hosting control panels, like cPanel or Plesk, also offer rate-limiting tools in their security sections.<\/p>\n\n\n<div class=\"single__related-article\">\n\t<span>Related Article<\/span>\n\t<div class=\"single__related-article__wrap\">\n\t\t<div class=\"single__related-article__title\">\n\t\t\tNGINX vs. Apache: How to Choose the Best Web Server\n\t\t<\/div>\n\t\t<a href=\"https:\/\/www-dev.dreamhost.com\/blog\/nginx-vs-apache\/\" class=\"btn btn--sm btn--brand\" target=\"_blank\" rel=\"noopener\">\n\t\t\tRead More\n\t\t<\/a>\n\t<\/div>\n<\/div>\n\n\n<p><strong>Pro tip:<\/strong> Start with conservative limits (like 30 requests per minute) and monitor your site. You can always tighten restrictions if bot traffic continues.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">3. Use a Content Delivery Network (CDN)<\/h3>\n\n\n\n<p><a target=\"_blank\" href=\"https:\/\/www.dreamhost.com\/features\/cdn\/\">CDNs<\/a> do two good things for you:<\/p>\n\n\n\n<ol class=\"wp-block-list\">\n<li>Distribute content across global server networks so your website is delivered quickly worldwide<\/li>\n\n\n\n<li>Filter traffic before it reaches the website to block any irrelevant bots and attacks<\/li>\n<\/ol>\n\n\n\n<p>The \u201cirrelevant bots\u201d part is what matters to us for now, but the other benefits are useful too. Most CDNs include built-in bot management that identifies and blocks suspicious visitors automatically.<\/p>\n\n\n\n<p><strong>How to Implement<\/strong><\/p>\n\n\n\n<ol class=\"wp-block-list\">\n<li>Sign up for a CDN service like <a target=\"_blank\" href=\"https:\/\/www.dreamhost.com\/features\/cdn\/\">DreamHost CDN<\/a>, Cloudflare, Amazon CloudFront, or Fastly.<\/li>\n\n\n\n<li>Follow the setup instructions (may require changing name servers).<\/li>\n\n\n\n<li>Configure the security settings to enable bot protection.<\/li>\n<\/ol>\n\n\n\n<p>If your hosting service offers a CDN by default, you eliminate all the steps since your website will automatically be hosted on CDN.<\/p>\n\n\n\n<p><strong>Once set up, your CDN will:<\/strong><\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Cache static content to reduce server load.<\/li>\n\n\n\n<li>Filter suspicious traffic before it reaches your site.<\/li>\n\n\n\n<li>Apply machine learning to differentiate between legitimate and malicious requests.<\/li>\n\n\n\n<li>Block known malicious actors automatically.<\/li>\n<\/ul>\n\n\n\n<p><strong>Pro tip:<\/strong> Cloudflare&#8217;s free tier includes <a target=\"_blank\" href=\"https:\/\/www.cloudflare.com\/en-in\/application-services\/products\/bot-management\/\">basic bot protection<\/a> that works well for most small business sites. Their paid plans offer more advanced options if you need them.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">4. Add CAPTCHA for Sensitive Actions<\/h3>\n\n\n\n<figure class=\"wp-block-image size-full\"><img decoding=\"async\" width=\"1600\" height=\"1008\" data-src=\"https:\/\/www.dreamhost.com\/blog\/wp-content\/uploads\/2025\/05\/03_add_a_captcha.webp\" alt=\"CAPTCHA verification during checkout process with postal\/zip code field, distorted text challenge, and &quot;Add Credit Card&quot; button showing $58.39 due.\" class=\"wp-image-67303 lazyload\" data-srcset=\"https:\/\/www-dev.dreamhost.com\/blog\/wp-content\/uploads\/2025\/05\/03_add_a_captcha.webp 1600w, https:\/\/www-dev.dreamhost.com\/blog\/wp-content\/uploads\/2025\/05\/03_add_a_captcha-300x189.webp 300w, https:\/\/www-dev.dreamhost.com\/blog\/wp-content\/uploads\/2025\/05\/03_add_a_captcha-1024x645.webp 1024w, https:\/\/www-dev.dreamhost.com\/blog\/wp-content\/uploads\/2025\/05\/03_add_a_captcha-768x484.webp 768w, https:\/\/www-dev.dreamhost.com\/blog\/wp-content\/uploads\/2025\/05\/03_add_a_captcha-1536x968.webp 1536w, https:\/\/www-dev.dreamhost.com\/blog\/wp-content\/uploads\/2025\/05\/03_add_a_captcha-600x378.webp 600w, https:\/\/www-dev.dreamhost.com\/blog\/wp-content\/uploads\/2025\/05\/03_add_a_captcha-1200x756.webp 1200w, https:\/\/www-dev.dreamhost.com\/blog\/wp-content\/uploads\/2025\/05\/03_add_a_captcha-730x460.webp 730w, https:\/\/www-dev.dreamhost.com\/blog\/wp-content\/uploads\/2025\/05\/03_add_a_captcha-1460x920.webp 1460w, https:\/\/www-dev.dreamhost.com\/blog\/wp-content\/uploads\/2025\/05\/03_add_a_captcha-784x494.webp 784w, https:\/\/www-dev.dreamhost.com\/blog\/wp-content\/uploads\/2025\/05\/03_add_a_captcha-1568x988.webp 1568w, https:\/\/www-dev.dreamhost.com\/blog\/wp-content\/uploads\/2025\/05\/03_add_a_captcha-877x553.webp 877w\" data-sizes=\"(max-width: 1600px) 100vw, 1600px\" src=\"data:image\/svg+xml;base64,PHN2ZyB3aWR0aD0iMSIgaGVpZ2h0PSIxIiB4bWxucz0iaHR0cDovL3d3dy53My5vcmcvMjAwMC9zdmciPjwvc3ZnPg==\" style=\"--smush-placeholder-width: 1600px; --smush-placeholder-aspect-ratio: 1600\/1008;\" \/><\/figure>\n\n\n\n<p>CAPTCHAs are those little puzzles that ask you to identify traffic lights or bicycles. They&#8217;re annoying for humans but nearly impossible for most bots, making them perfect gatekeepers for important areas of your site.<\/p>\n\n\n\n<p><strong>How to Implement<\/strong><\/p>\n\n\n\n<ol class=\"wp-block-list\">\n<li>Sign up for <a target=\"_blank\" href=\"https:\/\/developers.google.com\/recaptcha\">Google&#8217;s reCAPTCHA (free)<\/a> or <a target=\"_blank\" href=\"https:\/\/www.hcaptcha.com\/\">hCaptcha<\/a>.<\/li>\n\n\n\n<li>Add the CAPTCHA code to your sensitive forms:\n<ul class=\"wp-block-list\">\n<li>Login pages<\/li>\n\n\n\n<li>Contact forms<\/li>\n\n\n\n<li>Checkout processes<\/li>\n\n\n\n<li>Comment sections<\/li>\n<\/ul>\n<\/li>\n<\/ol>\n\n\n\n<p>For WordPress users, plugins like<a target=\"_blank\" href=\"https:\/\/automattic.pxf.io\/daXbzW\"> Akismet<\/a> can handle this automatically for comments and form submissions.<\/p>\n\n\n\n<p><strong>Pro tip:<\/strong> Modern invisible CAPTCHAs (like reCAPTCHA v3) work behind the scenes for most visitors, only showing challenges to suspicious users. Use this method to gain protection without annoying legitimate customers.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">5. Consider the New llms.txt Standard<\/h3>\n\n\n\n<figure class=\"wp-block-image size-full\"><img decoding=\"async\" width=\"1600\" height=\"1150\" data-src=\"https:\/\/www.dreamhost.com\/blog\/wp-content\/uploads\/2025\/05\/04_consider_the_new_llms_txt_standard.webp\" alt=\"Zapier API documentation showing endpoints for Get Action Details, Get Prefill Choices, Search Actions, and Create AI Action with their respective HTTP methods and descriptions.\" class=\"wp-image-67304 lazyload\" data-srcset=\"https:\/\/www-dev.dreamhost.com\/blog\/wp-content\/uploads\/2025\/05\/04_consider_the_new_llms_txt_standard.webp 1600w, https:\/\/www-dev.dreamhost.com\/blog\/wp-content\/uploads\/2025\/05\/04_consider_the_new_llms_txt_standard-300x216.webp 300w, https:\/\/www-dev.dreamhost.com\/blog\/wp-content\/uploads\/2025\/05\/04_consider_the_new_llms_txt_standard-1024x736.webp 1024w, https:\/\/www-dev.dreamhost.com\/blog\/wp-content\/uploads\/2025\/05\/04_consider_the_new_llms_txt_standard-768x552.webp 768w, https:\/\/www-dev.dreamhost.com\/blog\/wp-content\/uploads\/2025\/05\/04_consider_the_new_llms_txt_standard-1536x1104.webp 1536w, https:\/\/www-dev.dreamhost.com\/blog\/wp-content\/uploads\/2025\/05\/04_consider_the_new_llms_txt_standard-600x431.webp 600w, https:\/\/www-dev.dreamhost.com\/blog\/wp-content\/uploads\/2025\/05\/04_consider_the_new_llms_txt_standard-1200x863.webp 1200w, https:\/\/www-dev.dreamhost.com\/blog\/wp-content\/uploads\/2025\/05\/04_consider_the_new_llms_txt_standard-730x525.webp 730w, https:\/\/www-dev.dreamhost.com\/blog\/wp-content\/uploads\/2025\/05\/04_consider_the_new_llms_txt_standard-1460x1049.webp 1460w, https:\/\/www-dev.dreamhost.com\/blog\/wp-content\/uploads\/2025\/05\/04_consider_the_new_llms_txt_standard-784x564.webp 784w, https:\/\/www-dev.dreamhost.com\/blog\/wp-content\/uploads\/2025\/05\/04_consider_the_new_llms_txt_standard-1568x1127.webp 1568w, https:\/\/www-dev.dreamhost.com\/blog\/wp-content\/uploads\/2025\/05\/04_consider_the_new_llms_txt_standard-877x630.webp 877w\" data-sizes=\"(max-width: 1600px) 100vw, 1600px\" src=\"data:image\/svg+xml;base64,PHN2ZyB3aWR0aD0iMSIgaGVpZ2h0PSIxIiB4bWxucz0iaHR0cDovL3d3dy53My5vcmcvMjAwMC9zdmciPjwvc3ZnPg==\" style=\"--smush-placeholder-width: 1600px; --smush-placeholder-aspect-ratio: 1600\/1150;\" \/><\/figure>\n\n\n\n<p>The <a target=\"_blank\" href=\"https:\/\/llmstxt.org\/\">llms.txt standard is a recent development<\/a> that controls how AI crawlers interact with your content.<\/p>\n\n\n\n<p>It&#8217;s like robots.txt but specifically for telling AI systems what information they can access and what they should avoid.<\/p>\n\n\n\n<p><strong>How to Implement<\/strong><\/p>\n\n\n\n<p>1. <strong>Create a markdown file named llms.txt with this content structure:<\/strong><\/p>\n\n\n\n<ol class=\"wp-block-list\"><\/ol>\n\n\n\n<pre class=\"wp-block-code\"><code># Your Website Name\n&gt; Brief description of your site\n\n## Main Content Areas\n- &#091;Product Pages](https:\/\/yoursite.com\/products): Information about products\n- &#091;Blog Articles](https:\/\/yoursite.com\/blog): Educational content\n\n## Restrictions\n- Please don't use our pricing information in training<\/code><\/pre>\n\n\n\n<p>2. <strong>Upload it to your root directory (at yourdomain.com\/llms.txt) \u2192<\/strong> Reach out to a developer if you don\u2019t have direct access to the server.\u00a0<\/p>\n\n\n\n<ol class=\"wp-block-list\"><\/ol>\n\n\n\n<p>Is llms.txt the official standard? Not yet.<\/p>\n\n\n\n<p>It\u2019s a standard <a target=\"_blank\" href=\"https:\/\/llmstxt.org\/#proposal\">proposed<\/a> in late 2024 by Jeremy Howard, which has been adopted by Zapier, Stripe, Cloudflare, and many other large companies. Here\u2019s a <a target=\"_blank\" href=\"https:\/\/github.com\/thedaviddias\/llms-txt-hub?tab=readme-ov-file\">growing list of websites adopting llms.txt<\/a>.<\/p>\n\n\n\n<p>So, if you want to jump on board, they have <a href=\"https:\/\/github.com\/AnswerDotAI\/llms-txt\" target=\"_blank\">official documentation on GitHub<\/a> with implementation guidelines.<\/p>\n\n\n\n<p><strong>Pro tip:<\/strong> Once implemented, see if ChatGPT (with web search enabled) can access and understand the llms.txt file.<\/p>\n\n\n\n<figure class=\"wp-block-image size-full\"><img decoding=\"async\" width=\"1600\" height=\"987\" data-src=\"https:\/\/www.dreamhost.com\/blog\/wp-content\/uploads\/2025\/05\/05_verify_that_the_llms_txt_is_accessible_with_chatgpt.webp\" alt=\"Screenshot of AI assistant interface explaining Zapier API documentation for AI Actions, including endpoints for retrieving action details, searching actions, and managing authentication.\" class=\"wp-image-67305 lazyload\" data-srcset=\"https:\/\/www-dev.dreamhost.com\/blog\/wp-content\/uploads\/2025\/05\/05_verify_that_the_llms_txt_is_accessible_with_chatgpt.webp 1600w, https:\/\/www-dev.dreamhost.com\/blog\/wp-content\/uploads\/2025\/05\/05_verify_that_the_llms_txt_is_accessible_with_chatgpt-300x185.webp 300w, https:\/\/www-dev.dreamhost.com\/blog\/wp-content\/uploads\/2025\/05\/05_verify_that_the_llms_txt_is_accessible_with_chatgpt-1024x632.webp 1024w, https:\/\/www-dev.dreamhost.com\/blog\/wp-content\/uploads\/2025\/05\/05_verify_that_the_llms_txt_is_accessible_with_chatgpt-768x474.webp 768w, https:\/\/www-dev.dreamhost.com\/blog\/wp-content\/uploads\/2025\/05\/05_verify_that_the_llms_txt_is_accessible_with_chatgpt-1536x948.webp 1536w, https:\/\/www-dev.dreamhost.com\/blog\/wp-content\/uploads\/2025\/05\/05_verify_that_the_llms_txt_is_accessible_with_chatgpt-600x370.webp 600w, https:\/\/www-dev.dreamhost.com\/blog\/wp-content\/uploads\/2025\/05\/05_verify_that_the_llms_txt_is_accessible_with_chatgpt-1200x740.webp 1200w, https:\/\/www-dev.dreamhost.com\/blog\/wp-content\/uploads\/2025\/05\/05_verify_that_the_llms_txt_is_accessible_with_chatgpt-730x450.webp 730w, https:\/\/www-dev.dreamhost.com\/blog\/wp-content\/uploads\/2025\/05\/05_verify_that_the_llms_txt_is_accessible_with_chatgpt-1460x901.webp 1460w, https:\/\/www-dev.dreamhost.com\/blog\/wp-content\/uploads\/2025\/05\/05_verify_that_the_llms_txt_is_accessible_with_chatgpt-784x484.webp 784w, https:\/\/www-dev.dreamhost.com\/blog\/wp-content\/uploads\/2025\/05\/05_verify_that_the_llms_txt_is_accessible_with_chatgpt-1568x967.webp 1568w, https:\/\/www-dev.dreamhost.com\/blog\/wp-content\/uploads\/2025\/05\/05_verify_that_the_llms_txt_is_accessible_with_chatgpt-877x541.webp 877w\" data-sizes=\"(max-width: 1600px) 100vw, 1600px\" src=\"data:image\/svg+xml;base64,PHN2ZyB3aWR0aD0iMSIgaGVpZ2h0PSIxIiB4bWxucz0iaHR0cDovL3d3dy53My5vcmcvMjAwMC9zdmciPjwvc3ZnPg==\" style=\"--smush-placeholder-width: 1600px; --smush-placeholder-aspect-ratio: 1600\/987;\" \/><\/figure>\n\n\n\n<p>Verify that the llms.txt is accessible to these bots by asking ChatGPT (or another LLM) to \u201cCheck if you can read this page\u201d or \u201cWhat does the page say.\u201d<\/p>\n\n\n\n<p>We can\u2019t know if the bots will respect llms.txt anytime soon. However, if the AI search can read and understand the llms.txt file now, they may start respecting it in the future, too.<\/p>\n\n\n\n<h2 id=\"h2_monitoring-and-maintaining-your-sites-bot-protection\" class=\"wp-block-heading\">Monitoring and Maintaining Your Site&#8217;s Bot Protection<\/h2>\n\n\n\n<p>So you&#8217;ve set up your bot defenses \u2014 awesome work!<\/p>\n\n\n\n<p>Just keep in mind that bot technology is always evolving, meaning bots come back with new tricks. Let&#8217;s make sure your site stays protected for the long haul.<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>Schedule regular security check-ups:<\/strong> Once a month, look at your server logs for anything fishy and make sure your robots.txt and llms.txt files are updated with any new page links that you\u2019d like the bots to access\/not access.<\/li>\n\n\n\n<li><strong>Keep your bot blocklist fresh:<\/strong> Bots keep changing their disguises. Follow security blogs (or let your hosting provider do it for you) and update your blocking rules at regular intervals.<\/li>\n\n\n\n<li><strong>Watch your speed:<\/strong> Bot protection that slows your site to a crawl isn&#8217;t doing you any favors. Keep an eye on your page load times and fine-tune your protection if things start getting sluggish. Remember, real humans are impatient creatures!<\/li>\n\n\n\n<li><strong>Consider going on autopilot:<\/strong> If all this sounds like too much work (we get it, you have a business to run!), look into automated solutions or <a target=\"_blank\" href=\"https:\/\/www.dreamhost.com\/wordpress\/managed-wp-hosting-2\/\">managed hosting<\/a> that handles security for you. Sometimes the best DIY is DIFM \u2014 Do It For Me!<\/li>\n<\/ul>\n\n\n\n<h2 id=\"h2_a-bot-free-website-while-you-sleep-yes-please\" class=\"wp-block-heading\">A Bot-Free Website While You Sleep? Yes, Please!<\/h2>\n\n\n\n<p>Pat yourself on the back. You&#8217;ve covered a lot of ground here!<\/p>\n\n\n\n<p>However, even with our step-by-step guidance, this stuff can get pretty technical. <em>(What exactly is an .htaccess file anyway?)<\/em><\/p>\n\n\n\n<p>And while DIY bot management is certainly possible, you mind find that your time is better spent running the business.<\/p>\n\n\n\n<p><a target=\"_blank\" href=\"https:\/\/www.dreamhost.com\/pro-services\/management\/dreamcare\/\">DreamCare<\/a> is the &#8220;we&#8217;ll handle it for you&#8221; button you\u2019re looking for.<\/p>\n\n\n\n<p>Our team keeps your site protected with:<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li>24\/7 monitoring that catches suspicious activity while you sleep<\/li>\n\n\n\n<li>Regular security reviews to stay ahead of emerging threats<\/li>\n\n\n\n<li>Automatic software updates that patch vulnerabilities before bots can exploit them<\/li>\n\n\n\n<li>Comprehensive malware scanning and removal if anything sneaks through<\/li>\n<\/ul>\n\n\n\n<p>See, bots are here to stay. And considering their rise in the last few years, we could see more bots than humans in the near future. No one knows.<\/p>\n\n\n\n<p>But, why lose sleep over it?<\/p>\n\n\n\n\n<div class=\"article-cta-shared article-cta-small article-cta--product\">\n\t<div class=\"tr-img-wrap-outer jsLoading\"><img decoding=\"async\" class=\"js-img-lazy \" src=\"https:\/\/www-dev.dreamhost.com\/blog\/wp-content\/themes\/blog2018\/assets\/img\/lazy-loading-transparent.webp\" data-srcset=\"https:\/\/www-dev.dreamhost.com\/blog\/wp-content\/uploads\/2024\/03\/product-cta-pro-services-website-management-877x586.webp 1x, https:\/\/www-dev.dreamhost.com\/blog\/wp-content\/uploads\/2024\/03\/product-cta-pro-services-website-management.webp 2x\"  \/><\/div>\n\n\t<a href='https:\/\/www.dreamhost.com\/pro-services\/management\/' class='link-top' target='_blank' rel='noopener noreferrer'>\n\t\t<span>Pro Services &#8211; Website Management<\/span>\n\t\t<svg xmlns=\"http:\/\/www.w3.org\/2000\/svg\" viewBox=\"0 0 384 512\" width=\"15\"><path d=\"M342.6 233.4c12.5 12.5 12.5 32.8 0 45.3l-192 192c-12.5 12.5-32.8 12.5-45.3 0s-12.5-32.8 0-45.3L274.7 256 105.4 86.6c-12.5-12.5-12.5-32.8 0-45.3s32.8-12.5 45.3 0l192 192z\"\/><\/svg>\n\t<\/a>\n\n\t<div class=\"content-btm\">\n\t\t<h2 class=\"h2--md\">\n\t\t\tWe&#8217;ll Handle the Technical Stuff\n\t\t<\/h2>\n\t\t<p class=\"p--md\">\n\t\t\tBring enterprise-grade performance and reliability to your website. Leave the backend to the experts \u2013 you focus on your business.\n\t\t<\/p>\n\n\t\t        <a\n            href=\"https:\/\/www.dreamhost.com\/pro-services\/management\/\"\n                        class=\"btn btn--white-outline btn--sm btn--round\"\n                                    target=\"_blank\"\n            rel=\"noopener noreferrer\"\n            >\n                            See More                    <\/a>\n\n\t<\/div>\n<\/div>\n\n\n<hr class=\"wp-block-separator has-alpha-channel-opacity\" \/>\n\n\n\n<p><em>This page contains affiliate links. This means we may earn a commission if you purchase services through our link without any extra cost to you.<\/em><\/p>\n","protected":false},"excerpt":{"rendered":"<p>Find out how to stop bad bots from draining your website\u2019s resources while keeping search engines happy. Our guide to stopping web crawlers shows you how.<\/p>\n","protected":false},"author":1058,"featured_media":67300,"comment_status":"closed","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"_yoast_wpseo_metadesc":"Find out how to stop bad bots from draining your website\u2019s resources while keeping search engines happy. Our guide to stopping web crawlers shows you how.","toc_headlines":"[[\"h-a-quick-refresher-on-bots\",\"A Quick Refresher on Bots\"],[\"h-good-bot-bad-bot-what-s-what\",\"Good Bot, Bad Bot: What\u2019s What?\"],[\"h2_are-bots-bogging-down-your-website-look-for-these-warning-signs\",\"Are Bots Bogging Down Your Website? Look for These Warning Signs\"],[\"h2_your-bot-busting-toolkit-5-simple-steps-to-take-back-control\",\"Your Bot-Busting Toolkit: 5 Simple Steps To Take Back Control\"],[\"h2_monitoring-and-maintaining-your-sites-bot-protection\",\"Monitoring and Maintaining Your Site's Bot Protection\"],[\"h2_a-bot-free-website-while-you-sleep-yes-please\",\"A Bot-Free Website While You Sleep? Yes, Please!\"]]","hide_toc":false,"footnotes":""},"categories":[804],"tags":[],"class_list":["post-67299","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-web-hosting"],"acf":[],"yoast_head":"<!-- This site is optimized with the Yoast SEO Premium plugin v26.3 (Yoast SEO v27.4) - https:\/\/yoast.com\/product\/yoast-seo-premium-wordpress\/ -->\n<title>How to Block Bad Bots &amp; Speed Up Your Site | DreamHost<\/title>\n<meta name=\"description\" content=\"Find out how to stop bad bots from draining your website\u2019s resources while keeping search engines happy. Our guide to stopping web crawlers shows you how.\" \/>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/www-dev.dreamhost.com\/blog\/block-web-crawlers\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Stop Bad Bots, Keep Your Website Fast\" \/>\n<meta property=\"og:description\" content=\"Over 51% of your website traffic could be bots. That\u2019s more bots than humans. And they\u2019re draining your resources. Learn which bots and crawlers to block, which to keep, and how to protect your site\u2019s speed.\" \/>\n<meta property=\"og:url\" content=\"https:\/\/www-dev.dreamhost.com\/blog\/block-web-crawlers\/\" \/>\n<meta property=\"og:site_name\" content=\"DreamHost Blog\" \/>\n<meta property=\"article:publisher\" content=\"https:\/\/www.facebook.com\/DreamHost\/\" \/>\n<meta property=\"article:published_time\" content=\"2025-05-12T14:00:00+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2025-05-26T18:19:20+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/www.dreamhost.com\/blog\/wp-content\/uploads\/2025\/05\/1220x628_ogimage_how_to_stop_web_crawlers_and_bots_from_slowing_down_your_website.webp\" \/>\n\t<meta property=\"og:image:width\" content=\"1200\" \/>\n\t<meta property=\"og:image:height\" content=\"628\" \/>\n\t<meta property=\"og:image:type\" content=\"image\/webp\" \/>\n<meta name=\"author\" content=\"Ian Hernandez\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:title\" content=\"Stop Bad Bots, Keep Your Website Fast\" \/>\n<meta name=\"twitter:description\" content=\"Over 51% of your website traffic could be bots. That\u2019s more bots than humans. And they\u2019re draining your resources. Learn which bots and crawlers to block, which to keep, and how to protect your site\u2019s speed.\" \/>\n<meta name=\"twitter:creator\" content=\"@dreamhost\" \/>\n<meta name=\"twitter:site\" content=\"@dreamhost\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"Ian Hernandez\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"11 minutes\" \/>\n<!-- \/ Yoast SEO Premium plugin. -->","yoast_head_json":{"title":"How to Block Bad Bots & Speed Up Your Site | DreamHost","description":"Find out how to stop bad bots from draining your website\u2019s resources while keeping search engines happy. Our guide to stopping web crawlers shows you how.","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/www-dev.dreamhost.com\/blog\/block-web-crawlers\/","og_locale":"en_US","og_type":"article","og_title":"Stop Bad Bots, Keep Your Website Fast","og_description":"Over 51% of your website traffic could be bots. That\u2019s more bots than humans. And they\u2019re draining your resources. Learn which bots and crawlers to block, which to keep, and how to protect your site\u2019s speed.","og_url":"https:\/\/www-dev.dreamhost.com\/blog\/block-web-crawlers\/","og_site_name":"DreamHost Blog","article_publisher":"https:\/\/www.facebook.com\/DreamHost\/","article_published_time":"2025-05-12T14:00:00+00:00","article_modified_time":"2025-05-26T18:19:20+00:00","og_image":[{"width":1200,"height":628,"url":"https:\/\/www.dreamhost.com\/blog\/wp-content\/uploads\/2025\/05\/1220x628_ogimage_how_to_stop_web_crawlers_and_bots_from_slowing_down_your_website.webp","type":"image\/webp"}],"author":"Ian Hernandez","twitter_card":"summary_large_image","twitter_title":"Stop Bad Bots, Keep Your Website Fast","twitter_description":"Over 51% of your website traffic could be bots. That\u2019s more bots than humans. And they\u2019re draining your resources. Learn which bots and crawlers to block, which to keep, and how to protect your site\u2019s speed.","twitter_creator":"@dreamhost","twitter_site":"@dreamhost","twitter_misc":{"Written by":"Ian Hernandez","Est. reading time":"11 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/www-dev.dreamhost.com\/blog\/block-web-crawlers\/#article","isPartOf":{"@id":"https:\/\/www-dev.dreamhost.com\/blog\/block-web-crawlers\/"},"author":{"name":"Ian Hernandez","@id":"https:\/\/www-dev.dreamhost.com\/blog\/#\/schema\/person\/220faf2464d5ae08e41cf5f3b28fdc96"},"headline":"How To Stop Web Crawlers and Bots From Slowing Down Your Website","datePublished":"2025-05-12T14:00:00+00:00","dateModified":"2025-05-26T18:19:20+00:00","mainEntityOfPage":{"@id":"https:\/\/www-dev.dreamhost.com\/blog\/block-web-crawlers\/"},"wordCount":2228,"publisher":{"@id":"https:\/\/www-dev.dreamhost.com\/blog\/#organization"},"image":{"@id":"https:\/\/www-dev.dreamhost.com\/blog\/block-web-crawlers\/#primaryimage"},"thumbnailUrl":"https:\/\/www-dev.dreamhost.com\/blog\/wp-content\/uploads\/2025\/05\/1460x1095_blog_hero_how_to_stop_web_crawlers_and_bots_from_slowing_down_your_website.webp","articleSection":["Web Hosting"],"inLanguage":"en-US"},{"@type":"WebPage","@id":"https:\/\/www-dev.dreamhost.com\/blog\/block-web-crawlers\/","url":"https:\/\/www-dev.dreamhost.com\/blog\/block-web-crawlers\/","name":"How to Block Bad Bots & Speed Up Your Site | DreamHost","isPartOf":{"@id":"https:\/\/www-dev.dreamhost.com\/blog\/#website"},"primaryImageOfPage":{"@id":"https:\/\/www-dev.dreamhost.com\/blog\/block-web-crawlers\/#primaryimage"},"image":{"@id":"https:\/\/www-dev.dreamhost.com\/blog\/block-web-crawlers\/#primaryimage"},"thumbnailUrl":"https:\/\/www-dev.dreamhost.com\/blog\/wp-content\/uploads\/2025\/05\/1460x1095_blog_hero_how_to_stop_web_crawlers_and_bots_from_slowing_down_your_website.webp","datePublished":"2025-05-12T14:00:00+00:00","dateModified":"2025-05-26T18:19:20+00:00","description":"Find out how to stop bad bots from draining your website\u2019s resources while keeping search engines happy. Our guide to stopping web crawlers shows you how.","breadcrumb":{"@id":"https:\/\/www-dev.dreamhost.com\/blog\/block-web-crawlers\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/www-dev.dreamhost.com\/blog\/block-web-crawlers\/"]}]},{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/www-dev.dreamhost.com\/blog\/block-web-crawlers\/#primaryimage","url":"https:\/\/www-dev.dreamhost.com\/blog\/wp-content\/uploads\/2025\/05\/1460x1095_blog_hero_how_to_stop_web_crawlers_and_bots_from_slowing_down_your_website.webp","contentUrl":"https:\/\/www-dev.dreamhost.com\/blog\/wp-content\/uploads\/2025\/05\/1460x1095_blog_hero_how_to_stop_web_crawlers_and_bots_from_slowing_down_your_website.webp","width":1460,"height":1095,"caption":"How To Stop Web Crawlers and Bots From Slowing Down Your Website"},{"@type":"BreadcrumbList","@id":"https:\/\/www-dev.dreamhost.com\/blog\/block-web-crawlers\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/dhblog.dream.press\/blog\/"},{"@type":"ListItem","position":2,"name":"How To Stop Web Crawlers and Bots From Slowing Down Your Website"}]},{"@type":"WebSite","@id":"https:\/\/www-dev.dreamhost.com\/blog\/#website","url":"https:\/\/www-dev.dreamhost.com\/blog\/","name":"DreamHost Blog","description":"","publisher":{"@id":"https:\/\/www-dev.dreamhost.com\/blog\/#organization"},"potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/www-dev.dreamhost.com\/blog\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"},{"@type":"Organization","@id":"https:\/\/www-dev.dreamhost.com\/blog\/#organization","name":"DreamHost","url":"https:\/\/www-dev.dreamhost.com\/blog\/","logo":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/www-dev.dreamhost.com\/blog\/#\/schema\/logo\/image\/","url":"https:\/\/dhblog.dream.press\/blog\/wp-content\/uploads\/2019\/01\/dh_logo-blue-2.png","contentUrl":"https:\/\/dhblog.dream.press\/blog\/wp-content\/uploads\/2019\/01\/dh_logo-blue-2.png","width":1200,"height":168,"caption":"DreamHost"},"image":{"@id":"https:\/\/www-dev.dreamhost.com\/blog\/#\/schema\/logo\/image\/"},"sameAs":["https:\/\/www.facebook.com\/DreamHost\/","https:\/\/x.com\/dreamhost","https:\/\/www.instagram.com\/dreamhost\/","https:\/\/www.linkedin.com\/company\/dreamhost\/","https:\/\/www.youtube.com\/user\/dreamhostusa"]},{"@type":"Person","@id":"https:\/\/www-dev.dreamhost.com\/blog\/#\/schema\/person\/220faf2464d5ae08e41cf5f3b28fdc96","name":"Ian Hernandez","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/www-dev.dreamhost.com\/blog\/wp-content\/uploads\/2022\/04\/ian-hernandez-dreamhost-150x150.png","url":"https:\/\/www-dev.dreamhost.com\/blog\/wp-content\/uploads\/2022\/04\/ian-hernandez-dreamhost-150x150.png","contentUrl":"https:\/\/www-dev.dreamhost.com\/blog\/wp-content\/uploads\/2022\/04\/ian-hernandez-dreamhost-150x150.png","caption":"Ian Hernandez"},"description":"Ian is a Product Designer based in Los Angeles, California. He is responsible for driving brand and product design at DreamHost, developing and maintaining our internal design system, and writing frontend code when he can. In his free time, he enjoys walking his dog, learning history, and discovering new music online and irl. Connect with him on LinkedIn: https:\/\/www.linkedin.com\/in\/ianhernandez23\/","url":"https:\/\/www-dev.dreamhost.com\/blog\/author\/ianh\/"}]}},"lang":"en","translations":{"en":67299,"es":67284,"it":68678,"fr":70927,"nl":70963,"uk":72301,"pt":72314,"pl":72342,"de":72890,"ru":72898},"pll_sync_post":[],"_links":{"self":[{"href":"https:\/\/www-dev.dreamhost.com\/blog\/wp-json\/wp\/v2\/posts\/67299","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www-dev.dreamhost.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www-dev.dreamhost.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www-dev.dreamhost.com\/blog\/wp-json\/wp\/v2\/users\/1058"}],"replies":[{"embeddable":true,"href":"https:\/\/www-dev.dreamhost.com\/blog\/wp-json\/wp\/v2\/comments?post=67299"}],"version-history":[{"count":7,"href":"https:\/\/www-dev.dreamhost.com\/blog\/wp-json\/wp\/v2\/posts\/67299\/revisions"}],"predecessor-version":[{"id":67348,"href":"https:\/\/www-dev.dreamhost.com\/blog\/wp-json\/wp\/v2\/posts\/67299\/revisions\/67348"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www-dev.dreamhost.com\/blog\/wp-json\/wp\/v2\/media\/67300"}],"wp:attachment":[{"href":"https:\/\/www-dev.dreamhost.com\/blog\/wp-json\/wp\/v2\/media?parent=67299"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www-dev.dreamhost.com\/blog\/wp-json\/wp\/v2\/categories?post=67299"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www-dev.dreamhost.com\/blog\/wp-json\/wp\/v2\/tags?post=67299"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}