Panscient Data Services
Panscient Data Services Pty Ltd is a Company in Australia. In the last few days their bot hit my bot trap on two different sites. The IP both times was 38.99.203.110 and the user agent was...
View ArticleRobots.txt & Webmaster Tools
Webmaster Tools has an excellent way for any webmaster to see how Google will respond to their robots.txt file. I was using the robots.txt checker today and discovered a couple of things I did not...
View ArticleA List of Some Disobedient Bots
All of these bots have got caught in one of my bot trap at least once. Bots like Panscient and Cyveillance have got caught on at least two of my sites. A few of the bots (spiders) listed are legitimate...
View ArticleProtecting your Site from Scrappers
Protect your Site from Scrappers by Preventing Archiving. Do not allow any site to cache your pages.
View ArticleOpt-in or Blacklist?
What do I mean by Opt-in? In short rather than always adding user agents, or IP addresses to your .htaccess file to blacklist them, only allow certain bots, and browsers to access your site. So rather...
View ArticleProject Honey Pot
I first heard about heard about installing Honey Pots (or Honey Traps) on web pages several months ago. I had no idea what that meant. In the context in which the references were made I could tell it...
View ArticleProcess1
Process1 has hit my bot trap 3 different times this month with different IPs so I started doing a little digging. Since October is not quite over I checked the logs for September for the two sites...
View ArticleList of Bad Bots
I’ve started something new. I’m publishing a list of bots that have disobeyed the robots.txt protocol on my sites. Because a bot is listed does not necessarily mean the bot is bad. It just means the...
View ArticleBot Traps
Every webmaster should be interested in protecting their website from scrapers and email harvesters. One tool for securing your site is using a bot trap. If your an experienced programmer you could...
View ArticleBlocked IPs
I have posted a list of IPs that have been blocked by my bot trap from July 2007. The list is a work in progress for two reasons. I’m updating the list almost daily. This is the easy part. Initially I...
View Article
