{"id":893,"date":"2013-02-19T14:21:38","date_gmt":"2013-02-19T14:21:38","guid":{"rendered":"http:\/\/andrewpallant.ca\/wordpress\/?p=893"},"modified":"2013-11-26T11:47:02","modified_gmt":"2013-11-26T16:47:02","slug":"asp-bot-killer","status":"publish","type":"post","link":"http:\/\/andrewpallant.ca\/wordpress\/asp-bot-killer\/","title":{"rendered":"ASP – Bot Killer"},"content":{"rendered":"

Having troubles with bots that may be scraping, hammering on your sites or other inappropriate activities and you have a classic ASP site?<\/p>\n

This is a common issue that a lot of people go through. The trick is to filter out the good bots from the bad. To filter out the good from the bad, the first thing you need to do is identify the user agent and the easiest way is to create a function that returns true \/ false.<\/p>\n

function isGoodBot\r\n     ' get user agent from the request variables\r\n     userAgent = Request.ServerVariables(\"HTTP_USER_AGENT\")\r\n     isGoodBot = FALSE\r\n\r\n     ' Check user agent for likeable bot signatures\r\n     if instr(ucase(UserAgent), \"GOOGLE\") or instr(ucase(UserAgent), \"FACEBOOK\") or instr(ucase(UserAgent), \"YSEARCH\/SLURP\") _\r\n             or instr(ucase(UserAgent), \"MSNBOT\") or instr(ucase(UserAgent), \"BINGBOT\") then\r\n          isGoodBot = TRUE\r\n     end\r\nend function<\/pre>\n
The next thing to do is to figure out a tolerance level that you are comfortable with. The idea is to block people who are abusing your website, but allow those who are using your site properly to continue. I chose 20\/pages or more is a little more aggressive than I would like a user to be. I emailed myself every time I blocked so that I could research the IP address and I found that I was not blocking valid humans or bots until I was comfortable. \u00a0One site that I use to research IP addresses is: \u00a0http:\/\/www.ip-address.org\/lookup\/ip-locator.php<\/a><\/p>\n
Here is the logic I used was the following and only if not isGoodBot:<\/p>\n
    \n
  1. Log IP to a database table – this should be done for at least a day prior to the next steps. \u00a0Allows for some research.<\/li>\n
  2. Look up IP in the database, but only looking at the last 3 minutes.<\/li>\n
  3. Get average number of hits per minute during the last 3 minutes for the given IP address<\/li>\n
  4. If the number of hits per minute exceed the 20 pages per minute tolerance, then give 403 error<\/li>\n
  5. Instead of a 403 you can redirect to Google or any other site.<\/li>\n
  6. Create a SQL job to delete the old entries of the database anything older than a day ( the table will fill fast )<\/li>\n<\/ol>\n
    A 403 error in ASP is given like this:<\/p>\n
         Response.Status=\"403 Forbidden\"\r\n     Response.end<\/pre>\n
    I created this based on ASP because that was the project I was working in. It can easily be translated to PHP or any other language fairly easily.<\/p>\n","protected":false},"excerpt":{"rendered":"
    Having troubles with bots that may be scraping, hammering on your sites or other inappropriate activities and you have a classic ASP site? This is a common issue that a lot of people go through. The trick is to filter out the good bots from the bad.<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[146,36],"tags":[167],"yoast_head":"\r\nASP - Bot Killer - Software Developer In London<\/title>\r\n<meta name=\"description\" content=\"Having troubles with bots that may be scraping, hammering on your sites or other inappropriate activities and you have a classic ASP site?\" \/>\r\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\r\n<link rel=\"canonical\" href=\"http:\/\/andrewpallant.ca\/wordpress\/asp-bot-killer\/\" \/>\r\n<meta property=\"og:locale\" content=\"en_US\" \/>\r\n<meta property=\"og:type\" content=\"article\" \/>\r\n<meta property=\"og:title\" content=\"ASP - Bot Killer - Software Developer In London\" \/>\r\n<meta property=\"og:description\" content=\"Having troubles with bots that may be scraping, hammering on your sites or other inappropriate activities and you have a classic ASP site?\" \/>\r\n<meta property=\"og:url\" content=\"http:\/\/andrewpallant.ca\/wordpress\/asp-bot-killer\/\" \/>\r\n<meta property=\"og:site_name\" content=\"LDNDeveloper\" \/>\r\n<meta property=\"article:published_time\" content=\"2013-02-19T14:21:38+00:00\" \/>\r\n<meta property=\"article:modified_time\" content=\"2013-11-26T16:47:02+00:00\" \/>\r\n<meta name=\"author\" content=\"ldnDeveloper\" \/>\r\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\r\n<meta name=\"twitter:creator\" content=\"@LdnDeveloper\" \/>\r\n<meta name=\"twitter:site\" content=\"@LdnDeveloper\" \/>\r\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"ldnDeveloper\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"2 minutes\" \/>\r\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"Article\",\"@id\":\"http:\/\/andrewpallant.ca\/wordpress\/asp-bot-killer\/#article\",\"isPartOf\":{\"@id\":\"http:\/\/andrewpallant.ca\/wordpress\/asp-bot-killer\/\"},\"author\":{\"name\":\"ldnDeveloper\",\"@id\":\"https:\/\/andrewpallant.ca\/wordpress\/#\/schema\/person\/f6f5bb1ac3e0c5a54a8b5ce35fd67b84\"},\"headline\":\"ASP – Bot Killer\",\"datePublished\":\"2013-02-19T14:21:38+00:00\",\"dateModified\":\"2013-11-26T16:47:02+00:00\",\"mainEntityOfPage\":{\"@id\":\"http:\/\/andrewpallant.ca\/wordpress\/asp-bot-killer\/\"},\"wordCount\":343,\"commentCount\":1,\"publisher\":{\"@id\":\"https:\/\/andrewpallant.ca\/wordpress\/#\/schema\/person\/f6f5bb1ac3e0c5a54a8b5ce35fd67b84\"},\"keywords\":[\"Classic ASP\"],\"articleSection\":[\"ASP\",\"How To\"],\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"CommentAction\",\"name\":\"Comment\",\"target\":[\"http:\/\/andrewpallant.ca\/wordpress\/asp-bot-killer\/#respond\"]}]},{\"@type\":\"WebPage\",\"@id\":\"http:\/\/andrewpallant.ca\/wordpress\/asp-bot-killer\/\",\"url\":\"http:\/\/andrewpallant.ca\/wordpress\/asp-bot-killer\/\",\"name\":\"ASP - Bot Killer - Software Developer In London\",\"isPartOf\":{\"@id\":\"https:\/\/andrewpallant.ca\/wordpress\/#website\"},\"datePublished\":\"2013-02-19T14:21:38+00:00\",\"dateModified\":\"2013-11-26T16:47:02+00:00\",\"description\":\"Having troubles with bots that may be scraping, hammering on your sites or other inappropriate activities and you have a classic ASP site?\",\"breadcrumb\":{\"@id\":\"http:\/\/andrewpallant.ca\/wordpress\/asp-bot-killer\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"http:\/\/andrewpallant.ca\/wordpress\/asp-bot-killer\/\"]}]},{\"@type\":\"BreadcrumbList\",\"@id\":\"http:\/\/andrewpallant.ca\/wordpress\/asp-bot-killer\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\/\/andrewpallant.ca\/wordpress\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"ASP – Bot Killer\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\/\/andrewpallant.ca\/wordpress\/#website\",\"url\":\"https:\/\/andrewpallant.ca\/wordpress\/\",\"name\":\"LDNDeveloper\",\"description\":\"Learning, Growing and Sharing.\",\"publisher\":{\"@id\":\"https:\/\/andrewpallant.ca\/wordpress\/#\/schema\/person\/f6f5bb1ac3e0c5a54a8b5ce35fd67b84\"},\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\/\/andrewpallant.ca\/wordpress\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"en-US\"},{\"@type\":[\"Person\",\"Organization\"],\"@id\":\"https:\/\/andrewpallant.ca\/wordpress\/#\/schema\/person\/f6f5bb1ac3e0c5a54a8b5ce35fd67b84\",\"name\":\"ldnDeveloper\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/andrewpallant.ca\/wordpress\/#\/schema\/person\/image\/\",\"url\":\"https:\/\/andrewpallant.ca\/wordpress\/wp-content\/uploads\/2017\/05\/cropped-AAEAAQAAAAAAAAXQAAAAJDQxMGRlMzFjLWM4ODctNDk1NC05M2EyLWE1NDNhNTRiZjVlYw-2.jpg\",\"contentUrl\":\"https:\/\/andrewpallant.ca\/wordpress\/wp-content\/uploads\/2017\/05\/cropped-AAEAAQAAAAAAAAXQAAAAJDQxMGRlMzFjLWM4ODctNDk1NC05M2EyLWE1NDNhNTRiZjVlYw-2.jpg\",\"width\":512,\"height\":512,\"caption\":\"ldnDeveloper\"},\"logo\":{\"@id\":\"https:\/\/andrewpallant.ca\/wordpress\/#\/schema\/person\/image\/\"},\"description\":\"Andrew Pallant (@LdnDeveloper) has been a web, database and desktop developer for over 16 years. Andrew has worked on projects that ranged from factory automation to writing business applications. Most recently he has been heavily involved in various forms for ecommerce projects. Over the years Andrew has worn many hats: Project Manager, IT Manager, Lead Developer, Supervisor of Developers and many more - See more at: http:\/\/www.unlatched.com\/#sthash.8DiTkpKy.dpuf\",\"sameAs\":[\"http:\/\/www.andrewpallant.ca\",\"https:\/\/x.com\/LdnDeveloper\"],\"url\":\"http:\/\/andrewpallant.ca\/wordpress\/author\/ldndeveloper\/\"}]}<\/script>\r\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"ASP - Bot Killer - Software Developer In London","description":"Having troubles with bots that may be scraping, hammering on your sites or other inappropriate activities and you have a classic ASP site?","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"http:\/\/andrewpallant.ca\/wordpress\/asp-bot-killer\/","og_locale":"en_US","og_type":"article","og_title":"ASP - Bot Killer - Software Developer In London","og_description":"Having troubles with bots that may be scraping, hammering on your sites or other inappropriate activities and you have a classic ASP site?","og_url":"http:\/\/andrewpallant.ca\/wordpress\/asp-bot-killer\/","og_site_name":"LDNDeveloper","article_published_time":"2013-02-19T14:21:38+00:00","article_modified_time":"2013-11-26T16:47:02+00:00","author":"ldnDeveloper","twitter_card":"summary_large_image","twitter_creator":"@LdnDeveloper","twitter_site":"@LdnDeveloper","twitter_misc":{"Written by":"ldnDeveloper","Est. reading time":"2 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"http:\/\/andrewpallant.ca\/wordpress\/asp-bot-killer\/#article","isPartOf":{"@id":"http:\/\/andrewpallant.ca\/wordpress\/asp-bot-killer\/"},"author":{"name":"ldnDeveloper","@id":"https:\/\/andrewpallant.ca\/wordpress\/#\/schema\/person\/f6f5bb1ac3e0c5a54a8b5ce35fd67b84"},"headline":"ASP – Bot Killer","datePublished":"2013-02-19T14:21:38+00:00","dateModified":"2013-11-26T16:47:02+00:00","mainEntityOfPage":{"@id":"http:\/\/andrewpallant.ca\/wordpress\/asp-bot-killer\/"},"wordCount":343,"commentCount":1,"publisher":{"@id":"https:\/\/andrewpallant.ca\/wordpress\/#\/schema\/person\/f6f5bb1ac3e0c5a54a8b5ce35fd67b84"},"keywords":["Classic ASP"],"articleSection":["ASP","How To"],"inLanguage":"en-US","potentialAction":[{"@type":"CommentAction","name":"Comment","target":["http:\/\/andrewpallant.ca\/wordpress\/asp-bot-killer\/#respond"]}]},{"@type":"WebPage","@id":"http:\/\/andrewpallant.ca\/wordpress\/asp-bot-killer\/","url":"http:\/\/andrewpallant.ca\/wordpress\/asp-bot-killer\/","name":"ASP - Bot Killer - Software Developer In London","isPartOf":{"@id":"https:\/\/andrewpallant.ca\/wordpress\/#website"},"datePublished":"2013-02-19T14:21:38+00:00","dateModified":"2013-11-26T16:47:02+00:00","description":"Having troubles with bots that may be scraping, hammering on your sites or other inappropriate activities and you have a classic ASP site?","breadcrumb":{"@id":"http:\/\/andrewpallant.ca\/wordpress\/asp-bot-killer\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["http:\/\/andrewpallant.ca\/wordpress\/asp-bot-killer\/"]}]},{"@type":"BreadcrumbList","@id":"http:\/\/andrewpallant.ca\/wordpress\/asp-bot-killer\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/andrewpallant.ca\/wordpress\/"},{"@type":"ListItem","position":2,"name":"ASP – Bot Killer"}]},{"@type":"WebSite","@id":"https:\/\/andrewpallant.ca\/wordpress\/#website","url":"https:\/\/andrewpallant.ca\/wordpress\/","name":"LDNDeveloper","description":"Learning, Growing and Sharing.","publisher":{"@id":"https:\/\/andrewpallant.ca\/wordpress\/#\/schema\/person\/f6f5bb1ac3e0c5a54a8b5ce35fd67b84"},"potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/andrewpallant.ca\/wordpress\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"},{"@type":["Person","Organization"],"@id":"https:\/\/andrewpallant.ca\/wordpress\/#\/schema\/person\/f6f5bb1ac3e0c5a54a8b5ce35fd67b84","name":"ldnDeveloper","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/andrewpallant.ca\/wordpress\/#\/schema\/person\/image\/","url":"https:\/\/andrewpallant.ca\/wordpress\/wp-content\/uploads\/2017\/05\/cropped-AAEAAQAAAAAAAAXQAAAAJDQxMGRlMzFjLWM4ODctNDk1NC05M2EyLWE1NDNhNTRiZjVlYw-2.jpg","contentUrl":"https:\/\/andrewpallant.ca\/wordpress\/wp-content\/uploads\/2017\/05\/cropped-AAEAAQAAAAAAAAXQAAAAJDQxMGRlMzFjLWM4ODctNDk1NC05M2EyLWE1NDNhNTRiZjVlYw-2.jpg","width":512,"height":512,"caption":"ldnDeveloper"},"logo":{"@id":"https:\/\/andrewpallant.ca\/wordpress\/#\/schema\/person\/image\/"},"description":"Andrew Pallant (@LdnDeveloper) has been a web, database and desktop developer for over 16 years. Andrew has worked on projects that ranged from factory automation to writing business applications. Most recently he has been heavily involved in various forms for ecommerce projects. Over the years Andrew has worn many hats: Project Manager, IT Manager, Lead Developer, Supervisor of Developers and many more - See more at: http:\/\/www.unlatched.com\/#sthash.8DiTkpKy.dpuf","sameAs":["http:\/\/www.andrewpallant.ca","https:\/\/x.com\/LdnDeveloper"],"url":"http:\/\/andrewpallant.ca\/wordpress\/author\/ldndeveloper\/"}]}},"_links":{"self":[{"href":"http:\/\/andrewpallant.ca\/wordpress\/wp-json\/wp\/v2\/posts\/893"}],"collection":[{"href":"http:\/\/andrewpallant.ca\/wordpress\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"http:\/\/andrewpallant.ca\/wordpress\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"http:\/\/andrewpallant.ca\/wordpress\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"http:\/\/andrewpallant.ca\/wordpress\/wp-json\/wp\/v2\/comments?post=893"}],"version-history":[{"count":6,"href":"http:\/\/andrewpallant.ca\/wordpress\/wp-json\/wp\/v2\/posts\/893\/revisions"}],"predecessor-version":[{"id":900,"href":"http:\/\/andrewpallant.ca\/wordpress\/wp-json\/wp\/v2\/posts\/893\/revisions\/900"}],"wp:attachment":[{"href":"http:\/\/andrewpallant.ca\/wordpress\/wp-json\/wp\/v2\/media?parent=893"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"http:\/\/andrewpallant.ca\/wordpress\/wp-json\/wp\/v2\/categories?post=893"},{"taxonomy":"post_tag","embeddable":true,"href":"http:\/\/andrewpallant.ca\/wordpress\/wp-json\/wp\/v2\/tags?post=893"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}