Перейти к публикации
Поиск в
  • Дополнительно...
Искать результаты, содержащие...
Искать результаты в...

Юзер агенты ботов


 Поделиться

Рекомендованные сообщения

Есть ли у кого-нибудь более содержательный список юзер агентов ботов, чем у меня? Просьба поделиться или дополнить мой.) 

Будет полезно, например, для модуля источников заказов или различных обработчиков логов.

Мой список под спойлером ниже:

Спойлер

YandexBot
YandexImages
YandexMetrika
YaDirectFetcher
YandexMobileBot
YandexMarket
YandexImageResizer
Yandex Market picture robot
YandexAccessibilityBot
YandexCatalog
Googlebot
AdsBot-Google
Mail.RU_Bot
bingbot
msnbot-media
Baiduspider-
Baiduspider
Yahoo! Slurp
curl
AhrefsBot
MJ12bot
SeznamBot
SemrushBot
price.ru-image-robot
ltx71.com
Riddler
Exabot
SurdotlyBot
ActivizmBot
glindahl-cocrawler
Twitterbot
facebookexternalhit
Virusdie crawler
Cliqzbot
statdom.ru/Bot
openstat.ru/Bot
linkdexbot
Linkbot
coccocbot-web
SMTBot
top100.rambler.ru crawler
WebMoney MegaStock robot
CommonCrawler Node
archive.org_bot
SEOkicks-Robot
oBot
TorgMailRuBot
Python-urllib/2.7
Yandex.Commerce.Pinger
BLEXBot
linux-gnu
Googlebot-Image
AdsBot-Google-Mobile
360Spider
Trident-не бот
YisouSpider
spider
PHP-Curl-Class
slurp
crawler
pinger
robot
WGet tools
FavIconizer
FaceBook bot
CFNetwork
NG 1.x (Exalead)
Phantom
RoadHouse Crawling System
SurveyBot
Perl tool
Sogou Spider
StackRambler
TITAN
Feedfetcher-Google
Netcraft
Fish search
RBSE Spider
Sift
W3C Validator
Konqueror
Yahoo Slurp
Mail.RU Bot
archive.org bot
Apache-HttpClient/
Flamingo_SearchEngine

 

Ссылка на комментарий
Поделиться на других сайтах

Ну гугл-ботов больше

 

Googlebot
Googlebot-Nes
Googlebot-Images
Googlebot-Video
Google-Mobile
Google-Mobile AdSense
Google-AdSense
Google-AdsBot

и вот еще парочка которых нет у тебя

Teoma
ia_archiver
Lycos
Aport
WebAlta

 

Изменено пользователем L0ky
Ссылка на комментарий
Поделиться на других сайтах


Внесу свои 5 копеек

Боты:

Спойлер
TinEye-bot/0.51 (see http://www.tineye.com/crawler.html)
  Whoismindbot/1.0 (+http://www.whoismind.com/bot.html)
  Mozilla/5.0 (compatible; MSIE 8.0; Windows NT 6.3; www.alertra.com)
  R6_FeedFetcher(www.radian6.com/crawler)
  CheckHost (http://check-host.net/)
  YandeG 1.03
  yacybot (/global; amd64 Linux 3.16-0.bpo.2-amd64; java 1.7.0_65; Europe/en) http://yacy.net/bot.html
  SEMrushBot
  Protopage/3.0 (http://www.protopage.com)
  yacybot (/global; amd64 Linux 3.16.0-4-amd64; java 1.7.0_75; Europe/en) http://yacy.net/bot.html
  StatoolsBot (+http://www.statools.com/bot.html)
  adidxbot/2.0 (+http://search.msn.com/msnbot.htm)
  Mozilla/5.0 (compatible; Mail.RU_Bot/2.0; +http://go.mail.ru/help/robots)
  Mozilla/5.0 (Linux; Android 5.0; Nexus 5 Build/LRX21O) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/46.0.2490.76 Mobile Safari/537.36 PTST/281
  TurnitinBot (https://turnitin.com/robot/crawlerinfo.html)
  Scrapy/1.0.5 (+http://scrapy.org)
  yacybot (/global; amd64 Linux 4.4.0-31-generic; java 1.8.0_91; Europe/en) http://yacy.net/bot.html
  XmlSitemapGenerator - http://xmlsitemapgenerator.org
  Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.0) Match by Siteimprove.com
  Mozilla/5.0 (compatible; SeznamBot/3.2; +http://napoveda.seznam.cz/en/seznambot-intro/)
  WatchMouse/18990 (http://watchmouse.com/ ; gab)
  WatchMouse/8.4.0.3 (http://watchmouse.com/ ; hkhkg02.watchmouse.net)
  Mozilla/5.0 (compatible; LinkpadBot/1.06; +http://www.linkpad.ru)
  Mozilla/5.0 (compatible; heritrix/1.14.2 +http://rjpower.org)
  yacybot (webportal/global; amd64 Linux 2.6.32-5-amd64; java 1.6.0_18; Europe/en) http://yacy.net/bot.html
  PercolateCrawler/4 ([email protected])
  msnbot-UDiscovery/2.0b (+http://search.msn.com/msnbot.htm)
  Mozilla/5.0 (compatible; MSIE or Firefox mutant; not on Windows server;) Daum 4.1
  Mozilla/5.0 (compatible; spbot/4.0.3; +http://www.seoprofiler.com/bot )
  Mozilla/5.0 (compatible; LoadTimeBot/0.9; +http://www.loadtime.net/bot.html)
  UnwindFetchor/1.0 (+http://www.gnip.com/)
  nrsbot/5.0(loopip.com/robot.html)
  SemrushBot/0.9
  Mozilla/5.0 (compatible; UASlinkChecker/2.1; +https://udger.com/support/UASlinkChecker)
  yacybot (/global; amd64 Linux 3.11.10-21-desktop; java 1.7.0_51; America/en) http://yacy.net/bot.html
  Netvibes (http://www.netvibes.com)
  Acoon v4.1.0 (www.acoon.de)
  msnbot/2.0b (+http://search.msn.com/msnbot.htm).
  Mozilla/5.0 (compatible; ltbot/3.2.0.10 +http://www.kdsl.tu-darmstadt.de/de/kdsl/research-program/crawling-and-semantic-structuring/)
  smart.apnoti.com Robot/v1.34 (http://smart.apnoti.com/en/aboutApnotiWebCrawler.html)
  HubSpot Links Crawler 2.0 http://www.hubspot.com/
  yacybot (/global; x86 Windows XP 5.1; java 1.7.0_55; Europe/de) http://yacy.net/bot.html
  Mozilla/5.0 (compatible; DCPbot/1.0; +http://domains.checkparams.com/)
  Mozilla/5.0 (compatible; Exabot/3.0 (BiggerBetter); +http://www.exabot.com/go/robot)
  WebDoc (abuse-webdoc at service.moquadv.com)
  coccoc
  Mozilla/5.0 (compatible; spbot/4.0b; +http://www.seoprofiler.com/bot )
  Mozilla/5.0 (compatible; ExchangleBot/3.0; +https://www.exchangle.com/exchangling)
  Mozilla/5.0 (compatible; Qwantify/2.0n; +https://www.qwant.com/)
  OOZBOT/0.20 ( Setooz výrazný ako say-th-uuz, znamená mosty. ; http://www.setooz.com/oozbot.html ; agentname at setooz dot_com )
  SpeedySpider - http://www.entireweb.com
  Mozilla/5.0 (compatible; Heurekabot/3.1; +http://sluzby.heureka.cz/)
  crawler for netopian (http://www.netopian.co.uk/)
  L.webis/0.51 (http://webalgo.iit.cnr.it/index.php?pg=lwebis)
  Influencebot/0.9; (Automatic classification of websites; http://www.influencebox.com/; [email protected])
  Baiduspider+(+http://www.baidu.com/search/spider.htm)
  Mozilla/5.0 (X11; compatible; semantic-visions.com crawler; HTTPClient 3.1)
  Mozilla/5.0 (Windows; U; Windows NT 6.0; en-US; rv:1.9.2.28) Gecko/20120306 Firefox/99.0 YottaaMonitor
  Scrapy/1.1.1 (+http://scrapy.org)
  Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/27.0.1453.116 Safari/537.36 HubSpot Marketing Grader
  RyzeCrawler/1.1.1 ( http://www.domain2day.nl/crawler/)
  eBot / v.1.0a (http://alfa.elchron.cz)
  Sogou News Spider/4.0(+http://www.sogou.com/docs/help/webmasters.htm#07)
  Crowsnest/0.5 (+http://www.crowsnest.tv/)
  Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.4 (KHTML, like Gecko; Google Page Speed Insights) Chrome/22.0.1229 Safari/537.4
  DoCoMo/2.0 N902iS(c100;TB;W24H12)(compatible; moba-crawler; http://crawler.dena.jp/)
  Yeti/1.1 (NHN Corp.; http://help.naver.com/robots/)
  Experibot_v1
  Nekstbot - http://www.ipipan.waw.pl/nekst/nekstbot/
  Mozilla/5.0 (compatible; MJ12bot/v1.4.5; http://www.majestic12.co.uk/bot.php?+)
  Mozilla/5.0 (X11; Linux i686) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/30.0.1599.101 Safari/537.36; SSL-Crawler: http://crawler.dcsec.uni-hannover.de
  CorpusCrawler 2.0.19 (http://corpora.fi.muni.cz/crawler/);Project:CzCorpus
  Mozilla/5.0 (compatible; Semager/1.4; http://www.semager.de/blog/semager-bots/)
  Mozilla/5.0 (compatible; dlcbot/0.1; +http://www.drlinkcheck.com/)
  yacybot (freeworld/global; amd64 Linux 3.11.10-21-desktop; java 1.7.0_51; Europe/de) http://yacy.net/bot.html
  A6-Indexer/1.0 (http://www.a6corp.com/a6-web-scraping-policy/)
  CopperEgg/RevealUptime/DallasTX(linode)
  Mozilla/5.0 (compatible; Qwantify/2.1dw; +https://www.qwant.com/)/*
  Curious George - www.analyticsseo.com
  GozaikBot (www.gozaik.com;[email protected];www.gozaik.com/gozaikbot.html)
  Pu_iN Crawler (+http://semanticjuice.com/)
  Mozilla/5.0 (compatible; OpenindexDeepSpider/Nutch-1.5-dev; +http://www.openindex.io/en/webmasters/spider.html; systemsATopenindexDOTio)
  yacybot (freeworld/global; i386 Linux 3.0.0-17-generic-pae; java 1.6.0_23; Europe/en) http://yacy.net/bot.html
  yacybot (-global; amd64 FreeBSD 9.2-RELEASE-p10; java 1.7.0_65; Europe/en) http://yacy.net/bot.html
  FAST Enteprise Crawler/6 (www dot fastsearch dot com)
  Mozilla/5.0 (compatible; parsijoo-update-crawler; +http://www.parsijoo.ir/; [email protected])
  yacybot (freeworld/global; amd64 Windows Server 2008 6.0; java 1.7.0_25; Europe/en) http://yacy.net/bot.html
  Inspingbot/1.0 (+https://www.insping.com/)
  OrgProbe/0.9.4 (+http://www.blocked.org.uk)
  Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; Trident/5.0); 360Spider
  Mozilla/5.0 (Windows NT 6.1; WOW64; Trident/7.0; rv:11.0) like Gecko PTST/276
  Mozilla/5.0 (Windows NT 6.3; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/42.0.2311.90 Safari/537.36 PTST/276
  Mozilla/5.0 (compatible; SemrushBot/0.97~bl; +http://www.semrush.com/bot.html)
  Feedspot http://www.feedspot.com
  Zookabot/2.0;++http://zookabot.com
  drupact/0.7; http://www.arocom.de/drupact
  RobotsChecker/0.6 (+http://www.blocked.org.uk)
  yacybot (/global; i386 Linux 3.13.0-37-generic; java 1.7.0_65; Europe/en) http://yacy.net/bot.html
  Scrapy/1.0.3 (+http://scrapy.org)
  Testomatobot/1.0 (Linux x86_64; +http://www.testomato.com/testomatobot) minicrawler/4.0.0~beta12
  Slack-ImgProxy 0.66 (+https://api.slack.com/robots)
  Mozilla/5.0 (compatible; WebCookies/1.0; +https://webcookies.org/faq/#agent)
  NalezenCzBot/1.0 (http://www.nalezen.cz)
  yacybot (freeworld/global; amd64 Windows Server 2008 6.0; java 1.7.0_03; Europe/en) http://yacy.net/bot.html
  Mozilla/5.0 (compatible; AhrefsBot/5.1; +http://ahrefs.com/robot/)
  yacybot (freeworld/global; amd64 Windows 8 6.2; java 1.7.0_51; Europe/de) http://yacy.net/bot.html
  yacybot (/global; amd64 Windows 7 6.1; java 1.8.0_65; Europe/de) http://yacy.net/bot.html
  Mozilla/4.0 compatible ZyBorg/1.0 ([email protected]; http://www.WISEnutbot.com)
  Mozilla/5.0 (en-us) AppleWebKit/537.36(KHTML, like Gecko; Google-Adwords-DisplayAds-WebRender;) Chrome/27.0.1453Safari/537.36
  Crawler powered by contentDetection (www.mindup.de)
  PagePeeker.com (info: http://pagepeeker.com/robots)
  Mozilla/5.0 (compatible; Linux x86_64; Mail.RU_Bot/Fast/2.0; +http://go.mail.ru/help/robots)
  coccoc/1.0 (http://help.coccoc.com/)
  Favicon downloader (+https://favico.be/bot.html)
  Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0) ([email protected])
  Mozilla/5.0 (compatible; DIY-SEOBot/0.1a; +http://www.upcity.com/bot.html)
  Mozilla/5.0 (Windows NT 6.3; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/49.0.2623.112 Safari/537.36 PTST/276
  Mozilla/5.0 (compatible; theoldreader.com)
  Mozilla/5.0+(compatible; UptimeRobot/2.0; http://www.uptimerobot.com/)
  hawkReader/1.8 (Link Parser; http://www.hawkreader.com/; Allow like Gecko) Build/f2b2566
  Mozilla/5.0 (compatible; heritrix/1.12.1 +http://www.webarchiv.cz)
  Mozilla/5.0 (compatible; YandexImages/3.0; +http://yandex.com/bots)
  yacybot (/global; amd64 Linux 3.14.32-xxxx-grs-ipv6-64; java 1.7.0_75; Europe/en) http://yacy.net/bot.html
  Sogou web spider/4.0(+http://www.sogou.com/docs/help/webmasters.htm#07)
  yacybot (/global; amd64 Windows 7 6.1; java 1.8.0_45; Europe/de) http://yacy.net/bot.html
  larbin_2.6.3 [email protected]
  yacybot (freeworld/global; amd64 Linux 3.10.17-gentoo; java 1.7.0_45; UTC/en) http://yacy.net/bot.html
  MeMoNewsBot/2.0 (http://www.memonews.com/en/crawler)
  Mozilla/5.0 (compatible; MSIE 10.0; Windows NT 6.1; WOW64; Trident/6.0) PTST/281
  Mozilla/5.0 (compatible; DCPbot/1.5; +http://domains.checkparams.com/)
  Mozilla/5.0 (compatible; SemrushBot/0.97; +http://www.semrush.com/bot.html)
  BlackBerry9000/4.6.0.167 Profile/MIDP-2.0 Configuration/CLDC-1.1 VendorID/102 ips-agent
  Mozilla/5.0 (compatible; Baiduspider-cpro; +http://www.baidu.com/search/spider.html)
  yacybot (/global; amd64 Linux 3.2.0-4-amd64; java 1.7.0_65; Europe/en) http://yacy.net/bot.html
  Mozilla/5.0 (compatible; electricmonk/3.1.1 +https://www.duedil.com/our-crawler/)
  Hatena Antenna/0.5 (http://a.hatena.ne.jp/help)
  Backlink-Ceck.de (+http://www.backlink-check.de/bot.html)
  Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; WOW64; Trident/5.0) commoncrawl.org/research//Nutch-1.7-SNAPSHOT
  Mozilla/5.0 (compatible; Steeler/3.5; http://www.tkl.iis.u-tokyo.ac.jp/~crawler/)
  LapozzBot/1.4 (+http://robot.lapozz.com)
  Mozilla/5.0 (WhatsMyIP.org HTTP_Compression_Test) http://whatsmyip.org/ua
  Mozilla/5.0 (compatible; MSIE or Firefox mutant; not on Windows server;) Daumoa/4.0 (Following Mediapartners-Google)
  Mozilla/5.0 (compatible; ExaleadCloudview/6;)
  ADmantX Platform Semantic Analyzer - ADmantX Inc. - www.admantx.com - [email protected]
  msnbot/1.0 (+http://search.msn.com/msnbot.htm)
  Mozilla/5.0 (Windows NT 5.1) AppleWebKit/537.4 (KHTML, like Gecko) Chrome/22.0.1229.79 Safari/537.4 LinkTiger 2.0
  Pixray-Seeker/1.1 (Pixray-Seeker; [email protected])
  mbot v.1.16
  MXT/Nutch-1.12-SNAPSHOT (http://t.co/GSRLLKex24; informatique at mixdata dot com)
  Testomatobot/1.0 (Linux x86_64; +http://www.testomato.com/testomatobot) minicrawler/3.0.0
  polybot 1.0 (http://cis.poly.edu/polybot/)
  Mozilla/5.0 (compatible; YandexMetrika/2.0; +http://yandex.com/bots mtweb01t.yandex.ru)
  Mozilla/5.0 (compatible; memoryBot/1.21.46 +http://internetmemory.org/en/)
  Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_1) AppleWebKit/600.2.5 (KHTML, like Gecko) Version/8.0.2 Safari/600.2.5 (Applebot/0.1)
  yacybot (amd64 Linux 2.6.26-2-amd64; java 1.6.0_0; Europe/en) http://yacy.net/bot.html
  yacybot (/global; x86 Windows 8.1 6.3; java 1.8.0_45; America/en) http://yacy.net/bot.html
  Mozilla/5.0 (compatible; startmebot/1.0; +http://www.start.me/bot)
  yacybot (amd64 Windows 7 6.1; java 1.6.0_21; Europe/fr) http://yacy.net/bot.html
  yacybot (/global; amd64 Windows 7 6.1; java 1.8.0_101; Asia/ru) http://yacy.net/bot.html
  yacybot (freeworld/global; amd64 Linux 3.0.0-17-generic; java 1.6.0_23; Europe/de) http://yacy.net/bot.html
  alexa v0.1.4 (http://www.openwebspider.org/)
  http://arachnode.net 1.4
  Photon/1.0
  NetpeakCheckerBot
  GIDBot/3.0 (+http://www.gidnetwork.com/tools/gzip-test.php)
  Yandex/1.01.001 (compatible; Win16; P)
  w3dt.net httphr/2.0
  yacybot (/global; amd64 Windows 7 6.1; java 1.8.0_51; Europe/de) http://yacy.net/bot.html
  iqdb/0.1 (+http://iqdb.org/)
  Mozilla/5.0 (compatible; GimmeUSAbot/1.0; +https://gimmeusa.com/pages/crawler)
  Motoricerca-Robots.txt-Checker/1.0 (http://tool.motoricerca.info/robots-checker.phtml)
  Mozilla/5.0 (compatible; Tagoobot/3.0; +http://www.tagoo.ru)
  Mozilla/5.0 (Windows NT 6.1; WOW64; rv:26.0) Gecko/20100101 Firefox/26.0 Evidon ([email protected])
  Mozilla/5.0 (compatible; Nmap Scripting Engine; http://nmap.org/book/nse.html)
  yacybot (i386 Linux 2.6.32-22-generic; java 1.6.0_20; Europe/de) http://yacy.net/bot.html
  Mozilla/5.0 (Linux; Android 4.4.3; HTC One Build/KTU84L) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/40.0.2125.111 Mobile Safari/537.36 DareBoost
  mindUpBot (datenbutler.de)
  Mozilla/5.0 (compatible; monitis - premium monitoring service; http://www.monitis.com)
  Mozilla/5.0 (compatible; RankSonicSiteAuditor/1.0; +https://ranksonic.com/ranksonic_sab.html)
  Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/40.0.2125.111 Safari/537.36 DareBoost
  Zookabot/2.1;++http://zookabot.com
  yacybot (freeworld/global; amd64 Windows 7 6.1; java 1.6.0_24; Europe/de) http://yacy.net/bot.html
  Mozilla/5.0 (compatible; Google-Structured-Data-Testing-Tool +http://developers.google.com/structured-data/testing-tool/)
  Mozilla/5.0 (compatible; Plukkie/1.2; http://www.botje.com/plukkie.htm)
  focusbot/1.1
  Mozilla/5.0 (compatible; idmarch Automatic.beta/1.2; +http://www.idmarch.org/bot.html)
  CorpusCrawler 2.0.14 (http://corpora.fi.muni.cz/crawler/)
  Mozilla/5.0 (compatible; XML Sitemaps Generator; https://www.xml-sitemaps.com) Gecko XML-Sitemaps/1.0
  ImplisenseBot 1.1
  Promotion_Tools_www.searchenginepromotionhelp.com
  Aboundex/0.2 (http://www.aboundex.com/crawler/)
  CorpusCrawler 2.0.22 (http://corpora.fi.muni.cz/crawler/);Project:CzCorpus
  yacybot (/global; amd64 no-os-name no-os-version; java no-java-version; Europe/en) http://yacy.net/bot.html
  LumpImageSearch/0.1 (+http://lump.co/about/bot)
  Mozilla/4.0 (compatible; NaverBot/1.0; http://help.naver.com/customer_webtxt_02.jsp)
  Mozilla/5.0 (compatible; UASlinkChecker/2.0; +http://udger.com/support/UASlinkChecker)
  GAChecker (+http://www.gachecker.com)
  yacybot (freeworld/global; x86 Windows 7 6.1; java 1.7.0_25; Europe/de) http://yacy.net/bot.html
  Influencebot/0.9; (Automatic classification of websites; http://www.influencebox.com/; [email protected])User-Agent: Mozilla/5.0 (X11; Linux i686; rv:9.0) Gecko/20100101 Firefox/9.0
  Experibot_v1 (https://dl.dropboxusercontent.com/u/8024465/site/Info.html)
  DNSPod-reporting(http://www.dnspod.cn/reporting)
  yacybot (freeworld/global; i386 Linux 3.2.0-23-generic; java 1.6.0_27; Europe/en) http://yacy.net/bot.html
  Mozilla/5.0 (Windows NT 6.3;compatible; Leikibot/1.0; +http://www.leiki.com)
  PostPost/1.0 (+http://postpo.st/crawlers)
  envolk/1.7 (+http://www.envolk.com/envolkspiderinfo.html)
  Snapbot/1.0 (Snap Shots, +http://www.snap.com)
  Mozilla/5.0 (Linux; Android 4.0.4; Galaxy Nexus Build/IMM76B) AppleWebKit/537.36 (KHTML, like Gecko; Google-Publisher-Plugin) Chrome/27.0.1453 Mobile Safari/537.36
  Mozilla/5.0 (compatible; Uptimebot/0.2.18; +http://www.uptime.com/uptimebot)
  MIA DEV/search:robot/0.0.1 (This is the MIA Bot - crawling for mia research project. If you feel unhappy and do not want to be visited by our crawler send an email to [email protected]; http://spider.neofonie.de; [email protected])
  Mozilla/5.0 (Windows NT 6.1; Trident/7.0; rv:11.0; PTST 2.386) like Gecko
  Jyxobot/1
  WebAlta Crawler/2.0 (http://www.webalta.net/ru/about_webmaster.html) (Windows; U; Windows NT 5.1; ru-RU)
  GC3pro+dir SEO Tools - Vers. 3.00b - For more informations: http://chkme.com/
  Mozilla/5.0 (X11; Linux x86_64; rv:45.0; GTmetrix https://gtmetrix.com/) Gecko/20100101 Firefox/45.0
  Mozilla/5.0 (Windows NT 6.1; WOW64; rv:18.0) Gecko/20100101 Firefox/18.0 AppEngine-Google; (+http://code.google.com/appengine; appid: s~aeshortener)
  Priceonomics Analysis Engine - Fetch/1.0
  Ruky-Roboter (Version: 1.06, powered by www.ruky.de +http://www.ruky.de/bot.html)
  Baiduspider+(+http://help.baidu.jp/system/05.html)
  Openstat/0.1
  Yandex/1.01.001 (compatible; Win16; m)
  Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_1) AppleWebKit/600.2.5 (KHTML, like Gecko) Version/8.0.2 Safari/600.2.5 (Applebot/0.1; +http://www.apple.com/go/applebot)
  Mozilla/5.0 (compatible; OpenindexSpider/Nutch-1.5-dev; +http://www.openindex.io/en/webmasters/spider.html)
  Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; Multiviewbot
  CERT.at-Statistics-Survey/1.0 (http://www.cert.at/about/consec/content.html)
  Mozilla/5.0 (compatible; pmoz.info ODP link checker; +http://pmoz.info/doc/botinfo.htm)
  yacybot (freeworld/global; x86_64 Mac OS X 10.6.8; java 1.6.0_29; Asia/ru) http://yacy.net/bot.html
  gonzo/1[P] (+http://www.suchen.de/faq.html)
  MixBot (+http://t.co/GSRLLKex24)
  yacybot (/global; amd64 Linux 3.2.0-4-amd64; java 1.7.0_60; Europe/en) http://yacy.net/bot.html
  Mozilla/5.0 (compatible; imbot/0.1 +http://internetmemory.org/en/
  Mozilla/5.0 (en-us) AppleWebKit/525.13 (KHTML, like Gecko; Google Web Preview) Version/3.1 Safari/525.13
  Mozilla/5.0 (compatible; Pi-Monster; https://pricepi.com/)
  ThumbSniper (http://thumbsniper.com)
  Shelob ([email protected])
  Mozilla/5.0 (Macintosh; Intel Mac OS X 10_7_3) AppleWebKit/537.36 (KHTML, like Gecko, Google-Publisher-Plugin) Chrome/27.0.1453 Safari/537.36
  Mozilla/5.0 (compatible; KaloogaBot; http://www.kalooga.com/info.html?page=crawler)
  yacybot (/global; arm Linux 4.1.13+; java 1.8.0_40-internal; Etc/de) http://yacy.net/bot.html
  CorporateNewsSearchEngine/Nutch-1.7 (http://pibs.co/news-search-engine)
  Mozilla/5.0 (compatible; MSIE 10.0; Windows NT 6.1; WOW64; Trident/6.0) PTST/276
  Yandex/1.01.001 (compatible; Win16; I)
  FlightDeckReportsBot/2.0 (http://www.flightdeckreports.com/pages/bot)
  Scrapy/0.24.4 (+http://scrapy.org)
  ADmantX Platform Semantic Analyzer US - Turn - ADmantX Inc. - www.admantx.com - [email protected]
  Kyoto-Tohoku-Crawler/v1 (Mozilla-compatible; [email protected]; http://nlp.ist.i.kyoto-u.ac.jp/?crawling-kt)
  Mozilla/5.0 (compatible; Scarlett/ 1.0; +http://www.ellerdale.com/crawler.html)
  Mozilla/5.0 (compatible; NetcraftSurveyAgent/1.0; [email protected])
  www.deadlinkchecker.com Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/46.0.2490.86 Safari/537.36
  http://arachnode.net 1.2
  Mozilla/5.0 (compatible; Plukkie/1.5; http://www.botje.com/plukkie.htm)
  yacybot (freeworld/global; arm Linux 4.4.11-v7+; java 1.7.0_101; Etc/en) http://yacy.net/bot.html
  Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.0.16) Gecko/2009121601 Ubuntu/9.04 (jaunty) Firefox/3.0.16 Specificfeeds- http://www.specificfeeds.com
  Mozilla/5.0 (compatible; evc-batch/2.0)
  Orbiter/1.2 (http://dailyorbit.com/)
  crawler4j (https://github.com/yasserg/crawler4j/)
  Mozilla/5.0 (compatible; SEOdiver/1.0; +http://www.seodiver.com/bot)
  Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/51.0.2704.79 Safari/537.36 (https://shrinktheweb.com)
  Scopia crawler 1.0 (+http://www.scopia.co)
  yacybot (i386 Linux 2.6.23; java 1.6.0_06; Europe/en) http://yacy.net/bot.html
  Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.1.2) Gecko/20090729 Firefox/3.5.2 (.NET CLR 3.5.30729; Diffbot/0.1; +http://www.diffbot.com)
  Mozilla/5.0 (compatible; LXRbot/1.0;http://www.lxrmarketplace.com/,[email protected])
  yacybot (freeworld/global; amd64 Linux 3.8.13-gentoo; java 1.7.0_21; Europe/de) http://yacy.net/bot.html
  Mozilla/5.0 (compatible; Sitemap Generator/1.3; +http://www.check-domains.com/sitemap/index.php)
  DuckDuckBot/1.1; (+http://duckduckgo.com/duckduckbot.html)
  Mozilla/5.0 (compatible; BLEXBot/1.0; +http://webmeup-crawler.com/)
  Slack-ImgProxy 0.59 (+https://api.slack.com/robots)
  Mozilla/5.0 (compatible; Ezooms/1.0; [email protected])
  Mozilla/5.0 (compatible; YandexPagechecker/2.0; +http://yandex.com/bots)
  CoinCornerBot/1.1 ( https://www.coincorner.com/BitcoinBot)
  yacybot (freeworld/global; amd64 Linux 3.8.0-21-generic; java 1.6.0_27; Pacific/en) http://yacy.net/bot.html
  ScreenerBot Crawler Beta 2.0 (+http://www.ScreenerBot.com)
  gonzo1[P] +http://www.suchen.de/faq.html
  Mozilla/5.0 (compatible; MSIE or Firefox mutant;) Daum 4.1
  Mozilla/5.0 (compatible; 008/0.83; http://www.80legs.com/spider.html;) Gecko/2008032620
  Sogou web spider/4.02525A
  Visbot/2.0 (+http://www.visvo.com/en/webmasters.jsp;[email protected])
  Mozilla/5.0 (compatible; AcoonBot/4.10.8; +http://www.acoon.de/robot.asp)
  WatchMouse/18990 (http://watchmouse.com/ ; d3.watchmouse.com)
  Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/534.34 (KHTML, like Gecko) Qt/4.8.1 Safari/534.34 ShoppimonAgent/1.0 ([email protected])
  webinatorbot 1.1; +http://www.webinator.de
  findlinks/2.0.5 (+http://wortschatz.uni-leipzig.de/findlinks/)
  WikiDo/1.1 (http://wikido.com; [email protected])
  yacybot (freeworld/global; amd64 Linux 3.5.0-27-generic; java 1.7.0_03; Europe/de) http://yacy.net/bot.html
  Mozilla/5.0 (compatible; Uptimebot/0.1.73; +http://www.uptime.com/uptimebot)
  Semantifire1/0.20 ( -- ; http://www.setooz.com/oozbot.html ; agentname at setooz dot_com )
  Speedy Spider (http://www.entireweb.com/about/search_tech/speedy_spider/)
  Mozilla/5.0 compatible; yelpspider/yelpspider-1.0 (Crawlerbot run by Yelp Inc; yelpbot at yelp dot com)
  Mozilla/5.0 (compatible; OpenHoseBot/2.1; +http://www.openhose.org/bot.html)
  Mozilla/5.0 (compatible; emefgebot/beta; +http://emefge.de/bot.html)
  Mozilla/5.0 (compatible; OpenindexShallowSpider/Nutch-1.5-dev; +http://www.openindex.io/en/webmasters/spider.html; systemsATopenindexDOTio)
  Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/49.0.2623.75 Safari/537.36 Google (+https://developers.google.com/+/web/snippet/)
  Mozilla/5.0 (compatible; MetaJobBot; http://www.metajob.de/crawler)
  CCBot/1.0 (+http://www.commoncrawl.org/bot.html)
  Lijit Crawler (+http://www.lijit.com/robot/crawler)
  baypup/1.1 (Baypup; http://www.baypup.com/; [email protected])
  Mozilla/5.0 (FauBot/0.1; +http://buzzvolume.com/fau/)
  Mozilla/5.0 (compatible; NLNZ_IAHarvester2016/3.3.0 +https://natlib.govt.nz/publishers-and-authors/web-harvesting/domain-harvest)
  yacybot (/global; amd64 Linux 3.12.1; java 1.7.0_65; Europe/en) http://yacy.net/bot.html
  yacybot (amd64 Windows 7 6.1; java 1.6.0_18; Europe/de) http://yacy.net/bot.html
  SEOENGWorldBot/1.0 (+http://www.seoengine.com/seoengbot.htm)
  Mozilla/5.0 (compatible; Finderbots finder bot; +http://wiki.github.com/bixo/bixo/bixocrawler; [email protected])
  Pompos/1.3 http://dir.com/pompos.html
  Mozilla/5.0 (compatible; Gimme60bot/1.0; +http://gimme60.com) Firefox/16.0
  Mozilla/5.0 (compatible; MJ12bot/v1.4.6; http://mj12bot.com/)
  Mozilla/5.0 (compatible; Sysomos/1.0; +http://www.sysomos.com/; Sysomos)
  urlfan-bot/1.0; +http://www.urlfan.com/site/bot/350.html
  al_viewer ([email protected])
  LoadImpactRload/3.1.1 (Load Impact; http://loadimpact.com);
  findlinks/2.0.2 (+http://wortschatz.uni-leipzig.de/findlinks/)
  Mozilla/5.0 (compatible; spbot/3.1; +http://www.seoprofiler.com/bot )
  Mozilla/5.0 (compatible; AcoonBot/4.11.0; +http://www.acoon.de/robot.asp)
  WillyBot/1.1 (http://www.willyfogg.com/info/willybot)
  Norton-Safeweb
  WMCAI-robot (http://www.topicmaster.jp/wmcai/crawler.html)
  rogerbot/1.1 (http://moz.com/help/guides/search-overview/crawl-diagnostics#more-help, [email protected])
  Szukacz/1.5 (robot; www.szukacz.pl/jakdzialarobot.html; [email protected])
  rogerbot/1.0 (http://www.seomoz.org/dp/rogerbot, [email protected])
  ROR Sitemap Generator (http://www.rorweb.com)
  http://domino.research.ibm.com/comm/research_projects.nsf/pages/sai-crawler.callingcard.html
  MozillaTest/5.0 (compatible; YodaoBot/1.0; http://www.yodao.com/help/webmaster/spider/; )
  Mozilla/5.0 (compatible; Peew/1.0; http://www.peew.de/crawler/)
  Mozilla/5.0 (compatible; Website Analyzer/1.1; +http://www.check-domains.com/website-analysis/website-analyzer.php)
  Mozilla/5.0 (compatible; Gluten Free Crawler/1.0; +http://glutenfreepleasure.com/)
  PagePeeker.com
  CorpusCrawler 2.0.10 (http://corpora.fi.muni.cz/crawler/)
  yacybot (/global; amd64 Linux 3.2.0-4-amd64; java 1.7.0_101; Europe/cs) http://yacy.net/bot.html
  yacybot (/global; amd64 Linux 4.0.7-1-ck; java 1.8.0_45; Europe/de) http://yacy.net/bot.html
  Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/51.0.2704.103 Safari/537.36 PTST/284
  yacybot (freeworld/global; i386 Linux 3.12-1-686-pae; java 1.7.0_21; Europe/fr) http://yacy.net/bot.html
  Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.1; Trident/4.0; PTST 2.295)
  CovarioIDS/1.1 (http://www.covario.com/ids; support at covario dot com)
  Mozilla/5.0 (compatible; heritrix/1.14.4 +http://parsijoo.ir)
  www.adressendeutschland.de
  DialogSearch.com Bot 1.0;http://dialogsearch.com/webmasters
  librabot/2.0 (+http://search.msn.com/msnbot.htm)
  Mozilla/5.0 (compatible; heritrix/3.1.1-SNAPSHOT-20120116.200628 +http://www.archive.org/details/archive.org_bot)
  wsAnalyzer/1.0; ++http://www.wsanalyzer.com/bot.html
  Mozilla/5.0 (compatible; websays; +http://wiki.github.com/bixo/bixo/bixocrawler; [email protected])
  facebookexternalhit/1.0 (+http://www.facebook.com/externalhit_uatext.php)
  yacybot (/global; amd64 Linux 4.4.5-1-ARCH; java 1.8.0_77; America/en) http://yacy.net/bot.html
  websitepulse checker/1.1 (compatible; MSIE 5.5; Netscape 4.75; Linux)
  Mozilla/5.0 (compatible; memoryBot/1.21.14 +http://mignify.com/bot.html)
  SalesIntelligent/v1.0
  larbin_2.6.2 [email protected]
  Mozilla/5.0 (compatible; SiteCondor; http://www.sitecondor.com)
  yacybot (freeworld/global; amd64 Windows Server 2012 6.2; java 1.7.0_51; Europe/de) http://yacy.net/bot.html
  Mozilla/5.0 (compatible; Crawlera/1.10.2; UID 43063)
  yacybot (-global; amd64 Linux 2.6.32-042stab111.11; java 1.7.0_79; Europe/en) http://yacy.net/bot.html
  Mozilla/5.0 (compatible; SemrushBot/0.96.2; +http://www.semrush.com/bot.html)
  Linguee Bot (http://www.linguee.com/bot)
  yacybot (/global; x86 Windows XP 5.1; java 1.7.0_51; Europe/de) http://yacy.net/bot.html
  ICC-Crawler(Mozilla-compatible; ; http://kc.nict.go.jp/project1/crawl.html)
  Mozilla/5.0 (X11; U; Linux i686 (x86_64); en-US; rv:1.8.1.11) Gecko/20080109 (Charlotte/0.9t; http://www.searchme.com/support/) (Charlotte/0.9t; http://www.searchme.com/support/)
  Mozilla/5.0 (compatible; Goodzer/2.0; [email protected])
  Acoon v4.10.5 (www.acoon.de)
  CorpusCrawler 2.0.20 (http://corpora.fi.muni.cz/crawler/);Project:CzCorpus
  Mozilla/5.0 (compatible; AMZNKAssocBot/4.0 +http://affiliate-program.amazon.com)
  Mozilla/5.0 (compatible; MojeekBot/0.5; http://www.mojeek.com/bot.html)
  yacybot (/global; amd64 Linux 4.2.0-22-generic; java 1.7.0_91; Europe/en) http://yacy.net/bot.html
  Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/534.34 (KHTML, like Gecko) Qt/4.8.3 Safari/534.34 https://linkpeek.com
  MetaTagRobot/0.2 (http://www.seocentro.com/tools/search-engines/metatag-analyzer.html)
  Mozilla/5.0 (compatible; MJ12bot/v1.3.1; http://www.majestic12.co.uk/bot.php?+)
  Mozilla/5.0 (compatible; Uptimebot/0.2.29; +http://www.uptime.com/uptimebot)
  Mozilla/5.0 (compatible; Pandeo Bot; +http://pandeo.de/bot.php)
  Pixray-Seeker/1.1 (Pixray-Seeker; http://www.pixray.com/pixraybot; [email protected])
  EasyBib AutoCite (http://autocite-info.citation-api.com/)
  Mozilla/5.0 (compatible; OptimizationCrawler/0.2; +http://www.domainoptima.com/robot)
  AboutUsBot/Harpy (Website Analysis; http://www.aboutus.org/Aboutus:Bot; [email protected])
  Gigabot/2.0
  DoCoMo/2.0 N905i(c100;TB;W24H16) (compatible; Googlebot-Mobile/2.1; +http://www.google.com/bot.html)
  Morning Paper 1.0 (robots.txt compliant!)
  Mozilla/5.0 (compatible; SurdotlyBot/1.0; +http://sur.ly/bot.html; Linux; Android 4; iPhone; CPU iPhone OS 6_0_1 like Mac OS X) AppleWebKit/536.26 (KHTML, like Gecko) Version/6.0 Mobile/10A523 Safari/8536.25
  Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.1; Trident/4.0; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; Media Center PC 6.0; .NET4.0C; .NET4.0E; GWX:RED; PTST 2.386)
  SuperPagesUrlVerifyBot/1.0
  CopperEgg/RevealUptime/LondonUK(linode)
  Mozilla/5.0 (compatible; www.monitor.us - free monitoring service; http://www.monitor.us)
  Mozilla/5.0 (compatible; LinkMarketbot/1.2; +http://www.linkmarket.com/)
  Mozilla/5.0 (Windows; U; Windows NT 5.1; zh-CN; ) Firefox/1.5.0.11; 360Spider
  wangling
  Mozilla/5.0 (compatible; linkdexbot/2.0; +http://www.linkdex.com/about/bots/)
  RSSMicro.com RSS/Atom Feed Robot
  GarlikCrawler/1.1 (http://garlik.com/, [email protected])
  al_org_viewer ([email protected])
  Mozilla/5.0 (compatible; JadynAveBot; +http://www.jadynave.com/robot)
  dj-research/Nutch-1.11 (analytics@@demandjump.com)
  Karneval-Bot (Version: 1.06, powered by www.karnevalsuchmaschine.de +http://www.karnevalsuchmaschine.de/bot.html)
  Baiduspider-image+(+http://www.baidu.com/search/spider.htm)\nReferer: http://image.baidu.com/i?ct=503316480&z=0&tn=baiduimagedetail
  SauceNAO/1.0 (+http://saucenao.com/)
  Mozilla/5.0 (compatible; JobdiggerSpider +http://www.jobdigger.nl/spider)
  Yepi/1.0 (NHN Corp.; http://help.naver.com/robots/)
  Mozilla/5.0 (compatible; coccoc/1.0; +http://help.coccoc.com/)
  SSL Labs (https://www.ssllabs.com/about/assessment.html)
  Mozilla/5.0 (compatible; http://alyze.info)
  GigablastOpenSource/1
  Mozilla/5.0 (Windows; U; Windows NT 5.1;fr;rv:1.8.1) VoilaBotCollector BETA 0.1 (http://www.voila.com/)
  Vorboss Web Crawler [[email protected]]/Nutch-2.3
  Mozilla/5.0 (compatible; SecretSerachEngineLabs.com-SBSearch/0.9; http://www.secretsearchenginelabs.com/secret-web-crawler.php)
  Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko; Google Search Console) Chrome/27.0.1453 Safari/537.36
  YahooCacheSystem
  search.KumKie.com
  yacybot (webportal-global; x86 Windows Vista 6.0; java 1.7.0_25; Europe/en) http://yacy.net/bot.html
  Embedly [email protected]
  stq_bot (+http://www.searchteq.de)
  MSRBOT
  Mozilla/5.0 (compatible; WBSearchBot/1.1; +http://www.warebay.com/bot.html)
  CorpusCrawler 2.0.17 (http://corpora.fi.muni.cz/crawler/);Project:CzCorpus
  Mozilla/5.0 (compatible; heritrix/3.1.0-RC1 +http://boston.lti.cs.cmu.edu/crawler_12/)
  Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; Trident/5.0) PTST/276
  Mozilla/5.0 (compatible; WebCookies/1.0; +http://webcookies.org/faq/#agent)
  L.webis/0.44 (http://webalgo.iit.cnr.it/index.php?pg=lwebis)
  yacybot (freeworld/global; amd64 Linux 3.0.0-17-generic; java 1.6.0_23; America/en) http://yacy.net/bot.html
  Mozilla/5.0 (Windows NT 6.1; WOW64; rv:46.0) Gecko/20100101 Firefox/46.0 PTST/279
  Mozilla/5.0 (compatible; MSIE or Firefox mutant; not on Windows server;) Daumoa/4.0
  Mozilla/5.0 (compatible; YandexMetrika/2.0; +http://yandex.com/bots mtmon01i.yandex.ru)
  Y!J-BSC/1.0 crawler (http://help.yahoo.co.jp/help/jp/blog-search/)
  Mozilla/5.0 (compatible; UptimeRobot/1.0; http://www.uptimerobot.com/)
  NG-Search/0.86 (+http://www.ng-search.com)
  ichiro/3.0 (http://help.goo.ne.jp/help/article/1142)
  TwengaBot/1.1 (+http://www.twenga.com/bot.html)
  WebImages 0.3 ( http://herbert.groot.jebbink.nl/?app=WebImages )
  Mozilla/5.0 (compatible; aiHitBot/2.8; +http://endb-consolidated.aihit.com/)
  RyzeCrawler/1.1.1 (+http://www.domain2day.nl/crawler/)
  Mozilla/5.0 (Nekstbot; http://www.ipipan.waw.pl/nekst/nekstbot/)
  Mozilla/5.0 (compatible; adidxbot/2.0; http://www.bing.com/bingbot.htm)
  Mozilla/5.0 (compatible; SWEBot/1.0; +http://swebot-crawler.net)
  WatchMouse/8.4.0.3 (http://watchmouse.com/ ; gblon01.watchmouse.net)
  yacybot (/global; amd64 Windows 8.1 6.3; java 1.7.0_55; Europe/de) http://yacy.net/bot.html
  Page Analyzer v4.0 ( http://www.ranks.nl/ )
  web_bh ([email protected])
  findlinks/1.1.6-beta3 (+http://wortschatz.uni-leipzig.de/findlinks/)
  Mozilla/5.0 (compatible; Findxbot/1.0; +http://www.findxbot.com)
  findlinks/2.0 (+http://wortschatz.uni-leipzig.de/findlinks/)
  Mozilla/5.0 (compatible; imagecoccoc/1.0; +http://help.coccoc.com/)
  PagesInventory (robot http://www.pagesinventory.com)
  Mozilla/5.0 (compatible; aiHitBot/1.0-DS; +http://www.aihit.com/)
  tagSeoBot/1.0 (http://www.tagseoblog.de/tools)
  Mozilla/5.0 (en-us) AppleWebKit/537.36 (KHTML, like Gecko; Google PP Default) Chrome/27.0.1453 Safari/537.36
  404 Checker [http://www.404checker.com/user-agent]
  CopperEgg/RevealUptime/
  Mozilla/5.0 (compatible; adidxbot/2.0; +http://www.bing.com/bingbot.htm)
  Mozilla/5.0 (compatible; ZumBot/1.0; http://help.zum.com/inquiry)
  CopperEgg/RevealUptime/TokyoJapan
  Mozilla/5.0 (compatible; MojeekBot/0.2; http://www.mojeek.com/bot.html)
  GoSquared-Status-Checker/0.2
  Mozilla/5.0 (Windows NT 6.0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/99.0 Safari/537.36 YottaaMonitor
  WatchMouse/18990 (http://watchmouse.com/ ; bc.watchmouse.com)
  yacybot (freeworld/global; amd64 Linux 3.2.1-gentoo-r2; java 1.6.0_24; Europe/de) http://yacy.net/bot.html
  Mozilla/5.0 (compatible; adbeat_bot; [email protected]; [email protected])
  linkdexbot/Nutch-1.0-dev (http://www.linkdex.com/; crawl at linkdex dot com)
  Heurekabot-Feed/1.0 (+http://sluzby.heureka.cz/napoveda/heurekabot/)
  Mozilla/5.0 (compatible; Charlotte/1.1; http://www.searchme.com/support/)
  yacybot (/global; amd64 Linux 3.10.0-229.7.2.el7.x86_64; java 1.8.0_45; Europe/en) http://yacy.net/bot.html
  LSSRocketCrawler/1.0 LightspeedSystems
  Mozilla/5.0 (X11; U; Linux Core i7-4980HQ; de; rv:32.0; compatible; Jobboerse.com; http://www.xn--jobbrse-d1a.com) Gecko/20100401 Firefox/24.0
  findlinks/2.2 (+http://wortschatz.uni-leipzig.de/findlinks/)
  Mozilla/5.0 (compatible; kulturarw3 +http://www.kb.se/om/projekt/Svenska-webbsidor---Kulturarw3/)
  Mozilla/5.0 (compatible; CloudFlare-AlwaysOnline/1.0; +http://www.cloudflare.com/always-online) AppleWebKit/534.34
  Mozilla/5.0 (compatible; SemrushBot/0.96.4; +http://www.semrush.com/bot.html)
  woobot/2.0
  GarlikCrawler/1.2 (http://garlik.com/, [email protected])
  yacybot (webportal-global; amd64 Linux 3.2.0-4-amd64; java 1.7.0_67; Europe/en) http://yacy.net/bot.html
  Mozilla/5.0 (compatible; AboutUsBot Johnny5/2.0; +http://www.AboutUs.org/)
  yacybot (/global; amd64 Linux 3.10.0-327.22.2.el7.x86_64; java 1.7.0_101; Etc/en) http://yacy.net/bot.html
  Mozilla/5.0 (Linux; Android 4.1.2; Galaxy Nexus Build/JZO54K; GTmetrix http://gtmetrix.com/) AppleWebKit/537.22 (KHTML, like Gecko) Chrome/26.0.1410.58 Mobile Safari/537.22
  Nuhk/2.4 ( http://www.neti.ee/cgi-bin/abi/Otsing/Nuhk/)
  Grahambot/0.1 (+http://www.sunaga-lab.com/graham-bot)
  yacybot (/global; amd64 Linux 3.16.0-4-amd64; java 1.7.0_91; Europe/de) http://yacy.net/bot.html
  SEO Consulting; Redirect Checker Tool V.02; IP:
  Mozilla/5.0 (compatible; Pro Sitemaps Generator; https://pro-sitemaps.com) Gecko Pro-Sitemaps/1.0
  yacybot (/global; amd64 Linux 4.4.10-antix.1-amd64-smp; java 1.8.0_101; Europe/en) http://yacy.net/bot.html
  Mozilla/5.0 (Windows NT 6.2; WOW64) Runet-Research-Crawler (itrack.ru/research/cmsrate; [email protected])
  Mozilla/5.0 (compatible; Crawler/0.9; http://linkfluence.net/)
  ADmantX Platform Semantic Analyzer US Async - ADmantX Inc. - www.admantx.com - [email protected]
  GetProxi.es-bot/1.1 (http://getproxi.es/spiderinfo/)
  Mozilla/5.0 (Windows NT 6.3; WOW64; rv:46.0) Gecko/20100101 Firefox/46.0 PTST/277
  Pinterest/0.2 (+http://www.pinterest.com/)
  CopperEgg/RevealUptime/AtlantaGA(linode)
  OdklBot/1.0 ([email protected])
  Mozilla/5.0 (compatible; Exabot-Images/3.0; +http://www.exabot.com/go/robot)
  yacybot (freeworld/global; amd64 Linux 3.2.1-gentoo-r2; java 1.6.0_22; Europe/de) http://yacy.net/bot.html
  MojeekBot/0.2 (archi; http://www.mojeek.com/bot.html)
  Sogou web spider/4.05252A
  Mozilla/5.0 (Windows NT 6.3; WOW64; rv:45.0) Gecko/20100101 Firefox/45.0 PTST/276
  Mozilla/5.0 (compatible; MotoMinerBot/1.0; +https://motominer.com/Bot)
  NG/2.0
  Mozilla/5.0 (compatible; heritrix/1.14.2 +http://www.webarchiv.cz)
  StackRambler/2.0 (MSIE incompatible)
  Baiduspider+(+http://www.baidu.jp/spider/)
  yacybot (freeworld/global; amd64 Windows 7 6.1; java 1.6.0_25; Europe/de) http://yacy.net/bot.html
  Mozilla/5.0 (compatible; houzzbot; +http://www.houzz.com/)
  Woko robot 3.0
  Mozilla/5.0 (compatible; Qwantify/2.0; +https://www.qwant.com/)
  yacybot (/global; amd64 Linux 4.2.0-27-generic; java 1.8.0_66-internal; America/en) http://yacy.net/bot.html
  ADmantX Platform Semantic Analyzer - APAC - ADmantX Inc. - www.admantx.com - [email protected]
  Mozilla/5.0 (compatible; Uptimebot/0.2.40; +http://www.uptime.com/uptimebot)
  Mozilla/5.0 (compatible; ExpertSearchSpider +http://www.expertsearch.nl/spider)
  Mozilla/5.0 (compatible; coccocbot-web/1.0; +http://help.coccoc.com/searchengine)
  Acoon v4.10.4 (www.acoon.de)
  Mozilla/5.0 (compatible; memoryBot/1.20.210 +http://internetmemory.org/en/)
  Readability/740ec9 - http://readability.com/about/
  Mozilla/5.0 (compatible; Apercite; +http://www.apercite.fr/robot/index.html)
  yacybot (i386 Linux 2.6.28-gentoo-r5; java 1.5.0_18; Europe/en) http://yacy.net/bot.html
  Mozilla/5.0 (compatible; suggybot v0.01a, http://blog.suggy.com/was-ist-suggy/suggy-webcrawler/)
  yacybot (amd64 Windows 7 6.1; java 1.6.0_14; Europe/de) http://yacy.net/bot.html
  yacybot (freeworld/global; amd64 Linux 3.3.4-1-ARCH; java 1.6.0_24; Europe/en) http://yacy.net/bot.html
  Mozilla/5.0 (compatible; AportWorm/3.2; +http://www.aport.ru/help)
  Mozilla/5.0 (compatible; memoryBot/1.20.235 +http://internetmemory.org/en/)
  findlinks/2.6 (+http://wortschatz.uni-leipzig.de/findlinks/)
  Mozilla/5.0 (compatible; Hailoobot/1.2; +http://www.hailoo.com/spider.html)
  eCommerceBot (http://www.ehandel.se/botinfo.html)
  Mozilla/5.0(compatible;Sosospider/2.0;+http://help.soso.com/webspider.htm)
  yacybot (/global; amd64 Linux 4.3.0-gentoo-ARCH; java 1.7.0_85; Europe/en) http://yacy.net/bot.html
  Nymesis/2.0 (http://nymesis.com)
  CopperEgg/RevealUptime/OregonUSA
  uclassify.com/1.0
  Mozilla/5.0 (compatible; Butterfly/1.0; +http://labs.topsy.com/butterfly.html) Gecko/2009032608 Firefox/3.0.8
  Mozilla/5.0 (compatible; Prlog/1.0; +http://prlog.ru/)
  Slack-ImgProxy 1.106 (+https://api.slack.com/robots)
  AdnormCrawler www.adnorm.com/crawler
  Mozilla/5.0 (compatible; YandexZakladki/3.0; +http://yandex.com/bots)
  Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/49.0.2623.75 Safari/537.36 Google Favicon
  Mozilla/5.0 (compatible; Sonic/1.0; http://www.yama.info.waseda.ac.jp/~crawler/info.html)
  Mozilla/5.0 (iPhone; U; CPU iPhone OS 4_1 like Mac OS X; en-us) AppleWebKit/532.9 (KHTML, like Gecko) Version/4.0.5 Mobile/8B117 Safari/6531.22.7 (compatible; Googlebot-Mobile/2.1; +http://www.google.com/bot.html)
  ICC-Crawler/2.0 (Mozilla-compatible; ; http://kc.nict.go.jp/project1/crawl.html)
  Mozilla/4.0 ([email protected])
  bot-pge.chlooe.com/1.0.0 (+http://www.chlooe.com/)
  Mozilla/5.0 (compatible; GroupHigh/1.0; +http://www.grouphigh.com/)
  Mozilla/5.0 (compatible; Webmaster tools +http://sitexy.com/)
  yacybot (/global; amd64 Windows 8.1 6.3; java 1.8.0_40; Europe/de) http://yacy.net/bot.html
  Mozilla/5.0 (compatible; MJ12bot/v1.4.1; http://www.majestic12.co.uk/bot.php?+)
  Mozilla/5.0 (compatible; spbot/4.0.6; +http://www.seoprofiler.com/bot )
  Mozilla/5.0 (compatible; EuripBot/2.0; +http://www.eurip.com)
  findlinks/2.1 (+http://wortschatz.uni-leipzig.de/findlinks/)
  Sogou web spider/4.025251
  SETOOZBOT/5.0 ( http://www.setooz.com/bot.html )
  Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0) ([email protected])
  Mozilla/5.0 (compatible; spbot/2.0.4; +http://www.seoprofiler.com/bot )
  Mozilla/5.0 (TweetmemeBot/4.0; +http://datasift.com/bot.html) Gecko/20100101 Firefox/31.0
  mozilla/5.0 ([email protected])
  Mozilla/5.0 (compatible; spbot/2.1; +http://www.seoprofiler.com/bot )
  Mozilla/5.0 (Windows NT 6.2; WOW64) AppleWebKit/537.4 (KHTML, like Gecko) Chrome/98 Safari/537.4 (StatusCake SSL Monitor)
  Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; Trident/5.0; PTST 2.385)
  Mozilla/5.0 (compatible; evc-batch/2.0.20160608212921)
  Mozilla/5.0 (compatible; Mail.RU_Bot/2.0)
  seebot/2.0 (+http://www.seegnify.com/bot)
  bl.uk_lddc_bot/3.3.0-LBS-2016-02 (+http://www.bl.uk/aboutus/legaldeposit/websites/websites/faqswebmaster/index.html)
  CommaFeed/2.3.0-SNAPSHOT (https://www.commafeed.com)
  OmniExplorer_Bot/5.91c (+http://www.omni-explorer.com) WorldIndexer
  hledejLevne.cz/2.0
  page_verifier (http://www.securecomputing.com/goto/pv)
  url_test ([email protected])
  Mozilla/5.0 (X11; Linux x86_64; rv:10.0.12) Gecko/20100101 Firefox/21.0 WordPress.com mShots
  Mozilla/5.0 (compatible; Linux x86_64; Mail.RU_Bot/2.0; +http://go.mail.ru/help/robots)
  Is is up? (+http://isitup.org)
  Metaspinner/0.01 (Metaspinner; http://www.meta-spinner.de/; [email protected]/)
  TwengaBot-2.0 Champigny (+http://www.twenga.com/bot.html)
  LivelapBot/0.2 (http://site.livelap.com/crawler)
  HubSpot Crawler 1.0 http://www.hubspot.com/
  Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/45.0.2454.101 Safari/537.36 TinEye/1.0 (via http://www.tineye.com/)
  yacybot (/global; x86 Windows 10 10.0; java 1.8.0_73; Europe/de) http://yacy.net/bot.html
  Scrapy/0.24.6 (+http://scrapy.org)
  FAST-WebCrawler/3.6/FirstPage (atw-crawler at fast dot no;http://fast.no/support/crawler.asp)
  Baiduspider+(+http://www.baidu.com/search/spider_jp.html)
  Mozilla/5.0 (compatible; seplinkbot/1.0 )
  Mozilla/5.0 (compatible; Falconsbot; +http://ws.nju.edu.cn/falcons/)
  Mozilla/5.0 (Windows NT 6.1; WOW64; rv:46.0) Gecko/20100101 Firefox/46.0 PTST/277
  Mozilla/4.0 (Toread-Crawler/1.1; +http://news.toread.cc/crawler.php)
  TinEye-bot/0.02 (see http://www.tineye.com/crawler.html)
  yacybot (freeworld/global; i386 Linux 2.6.32-39-generic-pae; java 1.6.0_20; Europe/en) http://yacy.net/bot.html
  Mozilla/5.0 (compatible; parsijoo; +http://www.parsijoo.ir/; [email protected])
  Mozilla/5.0 (compatible; spbot/4.4.0; +http://OpenLinkProfiler.org/bot )
  MaxPoint Bot (+http://www.maxpoint.com)
  Mozilla/5.0 (compatible; Infohelfer/1.2.0; +http://www.infohelfer.de/crawler.php)
  ExB Language Crawler 2.1.1 (+http://www.exb.de/crawler)
  cg-eye interactive
  ZumBot/1.0 (ZUM Search; http://help.zum.com/inquiry)
  rogerbot/1.1 (http://moz.com/help/guides/search-overview/crawl-diagnostics#more-help, [email protected])
  Mozilla/5.0 (compatible; Crawlera/1.10.2; UID 47129)
  Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:14.0) Gecko/20100101 Firefox/14.0.1 AppEngine-Google; (+http://code.google.com/appengine; appid: s~metacdn-hr)
  Mozilla/5.0 (compatible; WoTBoT; +https://www.wslta.com/WoTBoT.html)
  HolmesBot (http://holmes.ge)
  Baiduspider-image+(+http://www.baidu.com/search/spider.htm)
  Mozilla/5.0 (compatible; alexa site audit/1.0; +http://www.alexa.com/help/webmasters; [email protected])
  Mediapartners-Google
  Mozilla/5.0 (compatible; MFGPagesBot/2.1; http://www.mfgpages.com)
  larbin_2.6.2 [email protected]
  Mozilla/5.0 (Windows; U; Windows NT 6.0; en-GB; rv:1.0; trendictionbot0.5.0; trendiction search; http://www.trendiction.de/bot; please let us know of any problems; web at trendiction.com) Gecko/20071127 Firefox/3.0.0.11
  BlogPulseLive ([email protected])
  WeSEE:Search/0.1 (Alpha, http://www.wesee.com/en/support/bot/)
  yacybot (freeworld/global; i386 Linux 3.0.0-17-generic; java 1.6.0_23; America/en) http://yacy.net/bot.html
  ImplisenseBot 1.0
  Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; WOW64; Trident/5.0; PTST 2.386)
  gonzo2[P] +http://www.suchen.de/faq.html
  Mozilla/5.0 (compatible; LXRbot/1.0; http://lxrseo.com/, [email protected])
  Mozilla/5.0 (compatible; Arachnophilia/1.0; +http://arachnys.com/)
  Mozilla/5.0 (compatible; CloudServerMarketSpider/1.0; +http://cloudservermarket.com/spider.html)
  kalooga/KaloogaBot (Kalooga; http://www.kalooga.com/info.html?page=crawler)
  yacybot (webportal/global; x86_64 Mac OS X 10.9.2; java 1.6.0_65; Europe/de) http://yacy.net/bot.html
  Mozilla/5.0 (compatible; DotBot/1.1; http://www.dotnetdotcom.org/, [email protected])
  yacybot (/global; amd64 Linux 4.1.19-gentoo; java 1.7.0_95; Europe/pl) http://yacy.net/bot.html
  Mozilla/5.0 (Windows NT 6.3; WOW64; Trident/7.0; rv:11.0) like Gecko PTST/276
  Mozilla/5.0 (compatible; RankActiveLinkBot; +https://rankactive.com/resources/rankactive-linkbot)
  audisto.com full crawler 3.26.431 (refer to in robots.txt as audisto, see https://audisto.com/bot)
  Mozilla/5.0 (compatible; spbot/4.0.1; +http://www.seoprofiler.com/bot )
  Mozilla/5.0 (compatible; OpenindexShallowSpider/Nutch-1.5-dev; +http://www.openindex.io/en/webmasters/spider.html)
  mozilla/5.0 (compatible; discobot/1.1; +http://discoveryengine.com/discobot.html)
  gonzo2[p] (+http://www.suchen.de/faq.html)
  Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; Trident/5.0); 360Spider(compatible; HaosouSpider; http://www.haosou.com/help/help_3_2.html)
  Mozilla/5.0 (compatible; spbot/2.0.1; +http://www.seoprofiler.com/bot/ )
  WatchMouse/18990 (http://watchmouse.com/ ; uk)
  yacybot (freeworld/global; amd64 Linux 3.1.10-hardened; java 1.7.0_03-icedtea; Europe/en) http://yacy.net/bot.html
  VeBot (+http://www.veinteractive.com/vebot)
  Mozilla/5.0 (compatible; NLNZ_IAHarvester2013 +http://natlib.govt.nz/about-us/current-initiatives/web-harvest-2012)
  findlinks/1.1.3-beta8 (+http://wortschatz.uni-leipzig.de/findlinks/)
  Mozilla/5.0 (Linux; U; Android 2.3.4; generic) AppleWebKit/537.36 (KHTML, like Gecko; Google Web Preview) Version/4.0 Mobile Safari/537.36
  DialogSearch.com Bot 1.4;http://dialogsearch.com/webmasters
  Mozilla/5.0 (compatible; GurujiBot/1.0; +http://www.guruji.com/en/WebmasterFAQ.html)
  Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9) AppleWebKit/537.71 (KHTML, like Gecko) Version/7.0 Safari/537.71 (Rival IQ, rivaliq.com)
  wscheck.com/1.0.0 (+http://wscheck.com/)
  Mozilla/5.0 (Windows NT 6.3; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/30.0.1599.69 Safari/537.36 Webthumb/2.0
  Mozilla/5.0 (compatible; FlipboardProxy/1.1; +http://flipboard.com/browserproxy)
  WeSEE:Ads/PictureBot (http://www.wesee.com/bot/)
  Mozilla/5.0 (compatible; Dataprovider/6.101; +https://www.dataprovider.com/)
  Kyoto-Crawler/n1.0 (Mozilla-compatible; [email protected]; http://nlp.ist.i.kyoto-u.ac.jp/?crawling)
  Mozilla/5.0 (compatible; SEOkicks-Robot; +http://www.seokicks.de/robot.html)
  Mozilla/5.0 (compatible; spbot/4.0.9; +http://OpenLinkProfiler.org/bot )
  Mozilla/5.0 (compatible; spbot/4.2.0; +http://OpenLinkProfiler.org/bot )
  seo-nastroj.cz
  LoadImpactPageAnalyzer/1.3.0 (Load Impact; http://loadimpact.com/)
  CSS Certificate Spider (http://www.css-security.com/certificatespider/)
  MetaGeneratorCrawler/1.3.2 (www.metagenerator.info)
  Testomatobot/1.0 (Linux x86_64; +http://www.testomato.com/testomatobot) minicrawler/4.0.0~beta8
  BLEXBot
  Mozilla/2.0 (compatible; Ask Jeeves/Teoma)
  Testomatobot/1.0 (Linux x86_64; +http://www.testomato.com/testomatobot) minicrawler/4.0.0~beta7
  Mozilla/5.0 (compatible; XoviBot/2.0; +http://www.xovibot.net/)
  Mozilla/5.0 (compatible; spbot/4.4.1; +http://OpenLinkProfiler.org/bot )
  ia_archiver (+http://www.alexa.com/site/help/webmasters; [email protected])
  HeartRails Robot/0.1 (http://www.heartrails.com)
  Mozilla/5.0 (compatible; Faveeo/1.0; +http://www.faveeo.com)
  yacybot (/global; amd64 Linux 3.16.0-49-generic; java 1.7.0_79; Europe/en) http://yacy.net/bot.html
  Castabot/0.1 (+http://topixtream.com/)
  Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.9.0.5) Gecko/2010033101 Gentoo Firefox/3.0.5 (Dot TK - spider 3.0)
  Mozilla/5.0 (compatible; MSIE or Firefox mutant; not on Windows server; +http://ws.daum.net/aboutWebSearch.html) Daumoa/2.0
  istellabot-nutch/Nutch-1.10
  Mail.RU/2.0
  ichiro/2.0 (http://help.goo.ne.jp/door/crawler.html)
  LexxeBot/1.0 ([email protected])
  Mozilla/5.0 (iPhone; CPU iPhone OS 8_1 like Mac OS X) AppleWebKit/600.1.4 (KHTML, like Gecko) Version/8.0 Mobile/12B410 Safari/600.1.4 (Applebot/0.1; +http://www.apple.com/go/applebot)
  Mozilla/5.0 (compatible; FlipboardRSS/1.1; +http://flipboard.com/browserproxy)
  Mozilla/5.0 (compatible; Linux x86_64; Mail.RU_Bot/Robots/2.0; +http://go.mail.ru/help/robots)
  yacybot (-global; amd64 Linux 3.10.0-229.4.2.el7.x86_64; java 1.7.0_79; Europe/en) http://yacy.net/bot.html
  it2media-domain-crawler/1.0 on crawler-prod.it2media.de
  yacybot (/global; amd64 Windows 8.1 6.3; java 1.8.0_25; Europe/de) http://yacy.net/bot.html
  yacybot (freeworld-global; amd64 Linux 3.16.0-4-amd64; java 1.7.0_79; Europe/de) http://yacy.net/bot.html
  Mozilla/5.0 (WhatsMyIP.org HTTP_Headers) http://whatsmyip.org/ua
  checkgzipcompression.com robot
  Mozilla/5.0 eCairn-Grabber/1.0 (+http://ecairn.com/grabber)
  RankurBot/3.3 (+http://rankur.com)
  L.webis/0.50 (http://webalgo.iit.cnr.it/index.php?pg=lwebis)
  Speedy Spider (Submit your site at http://www.entireweb.com/free_submission/)
  oBot
  Snappy/2.0 ( http://www.urltrends.com/ )
  Mozilla/5.0 (compatible; MSIE 10.0; Windows NT 6.2; WOW64; Trident/6.0) CrawlerProcess (http://www.PowerMapper.com) /5.23.770.0
  Mozilla/5.0 (compatible; alexa site audit/1.0; +http://www.alexa.com/help/webmasters; )
  flatlandbot/baypup (Flatland Industries Web Spider; http://www.flatlandindustries.com/flatlandbot; [email protected])
  istellabot/Nutch-1.11
  GetintentCrawler getintent.com
  Covario-IDS/1.0 (Covario; http://www.covario.com/ids; support at covario dot com)
  Mozilla/5.0 (compatible; FatBot 2.0; http://www.thefind.com/crawler)
  Mozilla/5.0 (compatible; MegaIndex.com/2.0; +http://megaindex.com/crawler)
  Microsearch.ru/1.0; http://microsearch.ru/webmasters
  TurnitinBot/3.0 (http://www.turnitin.com/robot/crawlerinfo.html)
  BacklinkCrawler V (http://www.backlinktest.com/crawler.html)
  FeedlyBot/1.0 (http://feedly.com)
  Clickagy Intelligence Bot v2
  GetURLInfo/1.0
  DoCoMo/2.0 P900i(c100;TB;W24H11)(compatible; ichiro/mobile goo;+http://help.goo.ne.jp/door/crawler.html)
  Scrapy/1.1.0 (+http://scrapy.org)
  yacybot (/global; amd64 Linux 3.16.0-4-amd64; java 1.7.0_75; America/en) http://yacy.net/bot.html
  Mozilla/5.0 (compatible; VSAgent/1.2)
  Experibot_v1 [goo.gl/n6zrAf]
  Mozilla/5.0 (compatible; DNS-Digger-Explorer/1.0; +http://www.dnsdigger.com)
  boitho.com-robot/1.0
  yacybot (/global; amd64 Windows Server 2012 6.2; java 1.7.0_51; Europe/de) http://yacy.net/bot.html
  RelateIQ Crawler www.relateiq.com
  Mozilla/4.0 (compatible; Netcraft Web Server Survey)
  Mozilla/5.0 (X11; U; Linux Core i7-4980HQ; de; rv:32.0; compatible; JobboerseBot; https://www.jobboerse.com/bot.htm) Gecko/20100101 Firefox/38.0
  yacybot (i386 Linux 2.6.9-023stab046.2-smp; java 1.6.0_05; Europe/en) http://yacy.net/bot.html
  Scrubby/2.2 (http://www.scrubtheweb.com/)
  Mozilla/5.0 (Yahoo-MMCrawler/4.0; mailto:[email protected])
  Fetch/2.0a (CMS Detection/Web/SEO analysis tool, see http://guess.scritch.org)
  bitlybot/3.0 (+http://bit.ly/)
  Mozilla/5.0 (compatible; heritrix/1.14.4 +http://www.exif-search.com)
  yacybot (freeworld/global; amd64 Windows Server 2012 6.2; java 1.7.0_25; Europe/de) http://yacy.net/bot.html
  agentslug.com - website monitoring tool
  SafeDNSBot (https://www.safedns.com/searchbot)
  Feedbin
  Mozilla/5.0 (compatible; proximic; +http://www.proximic.com)
  Mozilla/5.0 (compatible; AcoonBot/4.11.1; +http://www.acoon.de/robot.asp)
  Mozilla/5.0 (compatible; MagiBot/3.4.3; +http://magi.peak-labs.com/robots.txt)
  Mozilla/5.0 (compatible; XML Sitemaps Generator; http://www.xml-sitemaps.com) Gecko XML-Sitemaps/1.0
  baypup/colbert (Baypup; http://sf.baypup.com/webmasters; [email protected])
  MergeFlow-PageReader/0.91;+(+http://mergeflow.net/info/pagereader) Mozilla/5.0 (Windows) compatible
  Mozilla/5.0 (Mobile; rv:18.0) Gecko/18.0 Firefox/18.0 commoncrawl.org/research//Nutch-1.7-SNAPSHOT
  Mozilla/5.0 (compatible; spbot/4.1.0; +http://OpenLinkProfiler.org/bot )
  Mozilla/5.0 (compatible; OsO; http://oso.octopodus.com/abot.html)
  Mozilla/5.0 (compatible; gofind; +http://govid.mobi/bot.php)
  CatchBot/2.0; +http://www.catchbot.com
  Mozilla/5.0 (compatible; BusinessSeek.biz_Spider; http://www.businessseek.biz/)
  Quora Link Preview/1.0 (http://www.quora.com)
  radian6_default_(www.radian6.com/crawler)
  DWDS-Crawler +http://odo.dwds.de/dwds-crawler.html
  Mozilla/5.0 (compatible; DuckDuckGo-Favicons-Bot/1.0; +http://duckduckgo.com)
  sogou spider
  Mozilla/5.0 (compatible; GigablastOpenSource/1.0)
  Mozilla/5.0 (compatible; HomeTags/1.0; +http://www.hometags.nl/bot)
  CorpusCrawler 2.0.24 (http://corpora.fi.muni.cz/crawler/);Project:CzCorpus
  Scooter/3.3
  Zookabot/2.5;++http://zookabot.com
  Mozilla/5.0 (iPhone; CPU iPhone OS 8_1 like Mac OS X) AppleWebKit/600.1.4 (KHTML, like Gecko) Version/8.0 Mobile/12B411 Safari/600.1.4 (compatible; YandexMobileBot/3.0; +http://yandex.com/bots)
  Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/534.34 (KHTML, like Gecko) Safari/534.34; +http://sniptracker.com
  Mozilla/5.0 (compatible; archive.org_bot; Wayback Machine Live Record; +http://archive.org/details/archive.org_bot)
  es_com_viewer ([email protected])
  Mozilla/5.0 (compatible; Uptimebot/0.2.14; +http://www.uptime.com/uptimebot)
  yacybot (freeworld/global; amd64 Linux 3.0.0-14-generic; java 1.6.0_23; Europe/en) http://yacy.net/bot.html
  Mozilla/5.0 (X11; Linux x86_64; rv:35.0) Gecko/20100101 Firefox/35.0 DareBoost
  yacybot (freeworld/global; amd64 Linux 2.6.32-41-server; java 1.6.0_26; Europe/de) http://yacy.net/bot.html
  yacybot (amd64 Linux 2.6.28-18-generic; java 1.6.0_16; GMT/en) http://yacy.net/bot.html
  Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.1; Trident/4.0; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; Media Center PC 6.0; .NET4.0C; .NET4.0E; PTST 2.386)
  KD Bot
  Mozilla/5.0 (X11; U; Linux i686 (x86_64); en-US; rv:1.9.2.19) Gecko WebThumb/1.0
  Mozilla/5.0 (compatible; Yahoo! Slurp/3.0; http://help.yahoo.com/help/us/ysearch/slurp)
  rogerbot/1.0 (http://moz.com/help/pro/what-is-rogerbot-, [email protected])
  MJ12bot/v1.2.0 (http://majestic12.co.uk/bot.php?+)
  Superarama.com-Tarama-Botu-v.01
  Mozilla/5.0 (compatible; Urlfilebot/2.2; +http://urlfile.com/bot.html)
  Vegi bot (we follow your robots.txt settings before crawling, you can slow down the bot by change the Crawl-Delay parameter in the settings.if you have an enquiry, please email to: [email protected])
  MnoGoSearch/3.3.9
  Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/534+ (KHTML, like Gecko) BingPreview/1.0b
  JyxobotRSS/0.06
  Mozilla/5.0 (compatible; BigBozz/2.2.1; +http://www.bigbozz.com/)
  Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/534.57.2 (KHTML, like Gecko) Version/5.1.7 Safari/534.57.2 PTST/277
  KiNShooboT (compatible; KiNShooboT/1.0.C; +http://www.kinshoo.com/bot.html)
  SentiBot www.sentibot.eu (compatible with Googlebot)
  Topicbot/1.0 (Mozilla;I;+http://92.42.190.57/)
  IDG/IT (http://spaziodati.eu/)
  LinkAider (http://linkaider.com/crawler/)
  Mozilla/5.0 (compatible; coccoc/1.0; +http://help.coccoc.com/searchengine)
  yacybot (/global; amd64 Linux 2.6.32-573.3.1.el6.x86_64; java 1.7.0_85; Europe/en) http://yacy.net/bot.html
  bitlybot
  Mozilla/5.0 (compatible; Mp3Bot/0.7; +http://mp3realm.org/mp3bot/)
  WWW::LayeredExtractor::Handler::Feed/0.01
  Mozilla/5.0 (compatible; SWEBot/1.0; +http://swebot.net)
  Mozzila/5.0 (compatible; Sonic/1.0; http://www.yama.info.waseda.ac.jp/~crawler/info.html)
  MetaTagRobot/2.1 (http://www.seocentro.com/tools/search-engines/metatag-analyzer.html)
  Mozilla/5.0 (compatible; Goodzer/1.0)
  yacybot (/global; amd64 Linux 3.16.0-4-amd64; java 1.8.0_40; Europe/en) http://yacy.net/bot.html
  Mozilla/5.0 (Windows; U; Windows NT 6.1; en-GB; rv:1.9.2.3) Gecko/20100401 Firefox/3.6.3 (NetShelter ContentScan)
  Facebot/1.0
  SafeAds.xyz bot
  Mozilla/5.0 (compatible; Yeti/1.1; +http://help.naver.com/robots/)
  yacybot (/global; amd64 Windows 7 6.1; java 1.7.0_55; Europe/en) http://yacy.net/bot.html
  scrapyproject (+http://www.profound.net)
  Mozilla/5.0 (compatible; Online Domain Tools - Online Website Link Checker/1.2; +http://website-link-checker.online-domain-tools.com)
  Mozilla/5.0 (compatible; heritrix/2.0.2 +http://aihit.com)
  Mozilla/5.0 (compatible; Scrubby/3.1; +http://www.scrubtheweb.com/help/technology.html)
  Semantifire1/0.20 ( http://www.setooz.com/oozbot.html ; agentname at setooz dot_com )
  Mozilla/5.0 (X11; Ubuntu; Linux i686; rv:14.0; ips-agent) Gecko/20100101 Firefox/14.0.1
  UnisterBot (Mozilla/5.0 compatible; [email protected])
  Mozilla/5.0 (compatible; Ezooms/1.0; [email protected])
  Mozilla/5.0 (compatible; YandexVideo/3.0; +http://yandex.com/bots)
  Mozilla/5.0 (compatible; spbot/4.4.2; +http://OpenLinkProfiler.org/bot )
  findlinks/2.1.3 (+http://wortschatz.uni-leipzig.de/findlinks/)
  Mozilla/5.0 (compatible; MJ12bot/v1.2.5; http://www.majestic12.co.uk/bot.php?+)
  Speedy Spider (Entireweb; Beta/1.2; http://www.entireweb.com/about/search_tech/speedyspider/)
  Mozilla/5.0 (compatible; kazbtbot/0.1; +http://kazbt.com/)
  Orgbybot/OrgbyBot v1.3 (Spider; http://orgby.com/bot/ ; Orgby.com Search Engine)
  Mozilla/5.0 (compatible; aiHitBot-DM/2.0.2 +http://www.aihit.com)
  YahooSeeker-Testing/v3.9 (compatible; Mozilla 4.0; MSIE 5.5; http://search.yahoo.com/)
  Mozilla/5.0 (compatible; Crawlera/1.10.2; UID 40409)
  Mozilla/5.0 (compatible; image.coccoc/1.0; +http://help.coccoc.com/)
  yacybot (/global; amd64 Linux 3.19.2-1-ARCH; java 1.8.0_40; Europe/de) http://yacy.net/bot.html
  Mozilla/5.0 (compatible; evc-batch/2.0.20161017175820)
  datagnionbot (+http://www.datagnion.com/bot.html)
  CopperEgg/RevealUptime/TokyoJP(linode)
  MB-SiteCrawler
  WatchMouse/18990 (http://watchmouse.com/ ; liz)
  HybridBot (hybrid.ru/about. If our bot caused problems please contact us. Contact email: [email protected])
  Mozilla/5.0 (compatible; Crawlera/1.10.2; UID 24522)
  yacybot (/global; amd64 Linux 3.13.0-74-generic; java 1.7.0_91; Europe/en) http://yacy.net/bot.html
  findlinks/1.1.6-beta5 (+http://wortschatz.uni-leipzig.de/findlinks/)
  yacybot (freeworld/global; amd64 Linux 3.13.0-24-generic; java 1.7.0_55; Europe/de) http://yacy.net/bot.html
  Mozilla/5.0 (compatible; OpenfosBot/2.4; +http://www.openfos.com)
  Nusearch Spider (www.nusearch.com)
  WatchMouse/8.4.0.3 (http://watchmouse.com/ ; sesto02.watchmouse.net)
  ICC-Crawler/2.0 (Mozilla-compatible; ; http://www.nict.go.jp/en/univ-com/plan/crawl.html)
  Mozilla/5.0 (compatible; spbot/1.2; +http://www.seoprofiler.com/bot/ )
  Orbiter/1.3 (http://dailyorbit.com/)
  CCBot/2.0
  Mozilla/5.0 (compatible; linkdexbot/2.1; +http://www.linkdex.com/about/bots/)
  Mozilla/5.0 (compatible; DNS-Digger/1.0; +http://www.dnsdigger.com)
  AppEngine-Google; (+http://code.google.com/appengine; appid: s~feedly-social)
  yacybot (/global; x86 Windows 7 6.1; java 1.8.0_71; Europe/de) http://yacy.net/bot.html
  Mozilla/5.0 (Windows Phone 8.1; ARM; Trident/7.0; Touch; rv:11.0; IEMobile/11.0; NOKIA; Lumia 530) like Gecko (compatible; adidxbot/2.0; +http://www.bing.com/bingbot.htm)
  Dex Social Bot
  yacybot (/global; amd64 Linux 4.7.6-200.fc24.x86_64; java 1.8.0_102; Etc/en) http://yacy.net/bot.html
  Mozilla/5.0 (compatible; heritrix/1.14.3 +http://archive.org)
  CRAZYWEBCRAWLER 0.9.10, http://www.crazywebcrawler.com
  Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/50.0.2661.102 Safari/537.36 PTST/277
  Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.1.3; ips-agent) Gecko/20090824 Fedora/1.0.7-1.1.fc4 Firefox/3.5.3
  SafeSearch microdata crawler (https://safesearch.avira.com, [email protected])
  Zemanta Aggregator/0.9 +http://www.zemanta.com
  Mozilla/5.0 (compatible; special_archiver/3.2.0 +http://www.loc.gov/webarchiving/notice_to_webmasters.html)
  Mozilla/5.0 (compatible; GimmeUSAbot/1.0; +https://gimmeusa.com/crawler.html)
  yacybot (freeworld/global; amd64 Windows 7 6.1; java 1.7.0_45; Europe/de) http://yacy.net/bot.html
  Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.1 (KHTML, like Gecko) Chrome/21.0.1180.89 Safari/537.1; 360Spider
  FyberSpider/1.3 (http://www.fybersearch.com/fyberspider.php)
  yacybot (/global; amd64 FreeBSD 10.3-RELEASE; java 1.8.0_77; GMT/en) http://yacy.net/bot.html
  Searchie/1.0 (a Storm-based crawler; https://www.searchie.org; [email protected])
  yacybot (freeworld/global; amd64 Linux 3.1.10-1-desktop; java 1.6.0_22; Europe/de) http://yacy.net/bot.html
  holmes/3.12.4 (http://morfeo.centrum.cz/bot)
  WatchMouse/18990 (http://watchmouse.com/ ; se.watchmouse.com)
  MXT/Nutch-1.10 (http://t.co/GSRLLKex24; informatique at mixdata dot com)
  Mozilla/5.0 (compatible; Shareaholicbot/1.0; +http://www.shareaholic.com/bot)
  ZoomInformation Bot
  Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 5.1; www.alertra.com)
  Mozilla/5.0 (compatible; HomeTags/1.0; http://www.hometags.nl/bot)
  Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.2.17) Gecko/20110515 HeartRails_Capture/1.0.4 (+http://capture.heartrails.com/) Namoroka/3.6.17
  OmniExplorer_Bot/5.20 (+http://www.omni-explorer.com) WorldIndexer
  WatchMouse/18990 (http://watchmouse.com/ ; it)
  Mozilla/5.0 (compatible; imrbot/1.10.8 +http://www.mignify.com)
  Mozilla/5.0 (compatible; spbot/4.0.2; +http://www.seoprofiler.com/bot )
  Readability/1900e6 - http://readability.com/about/
  Mozilla/5.0 (compatible; Dataprovider/6.92; +https://www.dataprovider.com/)
  Mozilla/5.0 (iPhone; CPU iPhone OS 6_0_1 like Mac OS X) AppleWebKit/537.36 (KHTML, like Gecko; Google Page Speed Insights) Version/6.0 Mobile/10A525 Safari/8536.25
  VegeBot (we follow your robots.txt settings before crawling, you can slow down the bot by change the Crawl-Delay parameter in the settings.if you have an enquiry, please email to: [email protected])
  yacybot (webportal-global; amd64 Linux 3.2.0-4-amd64; java 1.7.0_65; Europe/en) http://yacy.net/bot.html
  FeedBucket/1.0 (+http://www.feedbucket.com)
  topster.de HTTP-Header 1.0
  MaxPointCrawler/Nutch-1.10 (maxpoint.crawler at maxpointinteractive dot com)
  Mozilla/5.0 (Windows NT 6.0; rv:45.0) Gecko/20100101 Firefox/45.0 PTST/276
  yacybot (/global; amd64 Windows 7 6.1; java 1.8.0_05; Europe/es) http://yacy.net/bot.html
  NETCRAFT
  Mozilla/5.0 (compatible; WbSrch/1.0; +https://wbsrch.com)
  WebCookies/1.0 (+http://webcookies.info/faq/#agent)
  netEstate Impressumscrawler (+http://www.netestate.de/De/Loesungen/Impressumscrawler)
  msnbot-media/2.0b (+http://search.msn.com/msnbot.htm)
  CopperEgg/RevealUptime/DublinIEUSA
  StormCrawler/1.0 (a Storm-based crawler; https://github.com/DigitalPebble/storm-crawler; [email protected])
  WatchMouse/8.4.0.3 (http://watchmouse.com/ ; uschi02.watchmouse.net)
  Mozilla/5.0 (Linux; Android 5.0.2; SM-G920T Build/LRX22G) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/40.0.2125.111 Mobile Safari/537.36 DareBoost
  Mozilla/5.0 (compatible; DeuSu/0.1.0; +https://deusu.org)
  Mozilla/5.0 (compatible; MJ12bot/v1.4.4 (domain ownership verifier); http://www.majestic12.co.uk/bot.php?+)
  Mozilla/5.0 (compatible; SemrushBot/0.99~bl; +http://www.semrush.com/bot.html)
  WebWatch/Robot_txtChecker
  Mozilla/5.0 (compatible; linkdexbot/2.2; +http://www.linkdex.com/bots/)
  Feedly/1.0 (+http://www.feedly.com/fetcher.html; like FeedFetcher-Google)
  Toweyabot: toweya.com
  Mozilla/5.0 (compatible; Infohelfer/1.4.3; +http://www.infohelfer.de/crawler.php)
  Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.2.1; aggregator:Spinn3r (Spinn3r 3.1); http://spinn3r.com/robot) Gecko/20021130
  FeedCatBot/3.0 (+http://www.feedcat.net/)
  LinkedInBot/1.0 (compatible; Mozilla/5.0; Apache-HttpClient +http://www.linkedin.com), libot/Nutch-1.9 (http://www.linkedin.com; [email protected])
  R6_CommentReader(www.radian6.com/crawler)
  Mozilla/5.0 (compatible; Crawlera/1.10.2; UID 70350)
  Mozilla/5.0 (compatible; MJ12bot/v1.2.3; http://www.majestic12.co.uk/bot.php?+)
  Domain Re-Animator Bot (http://domainreanimator.com) - [email protected]
  Riddler (http://riddler.io/about.html)
  Mozilla/5.0 (compatible; Esribot/1.0; http://www.esrihu.hu/)
  Favicon downloader (http://favicon.netk6.com/)
  findlinks/1.1.5-beta7 (+http://wortschatz.uni-leipzig.de/findlinks/)
  ConveraCrawler/0.9d (+http://www.authoritativeweb.com/crawl)
  Mozilla/5.0 (Windows NT 6.1; rv:6.0) Gecko/20110814 Firefox/6.0 Google favicon
  GetFoundBot (+http://www.getfound.cz/getfoundbot/)
  WatchMouse/8.4.0.3 (http://watchmouse.com/ ; usdal02.watchmouse.net)
  Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/48.0.2564.97 Safari/537.36 Yandex.Translate
  Mozilla/5.0 (compatible; NLNZ_IAHarvester2013 +http://natlib.govt.nz/about-us/current-initiatives/web-harvest-2013)
  COMODOSpider/Nutch-1.2
  Mozilla/5.0 (compatible; DomainMacroCrawler/0.1; +http://domainmacro.com)
  Mozilla/5.0 (compatible; heritrix/3.2.0 +http://www.exif-search.com)
  Mozilla/5.0 (compatible; MagiBot/3.6.2; +http://magi.peak-labs.com/robots.txt)
  Mozilla/5.0 (compatible; Alexabot/1.0; +http://www.alexa.com/help/certifyscan; [email protected])
  psbot-image (+http://www.picsearch.com/bot.html)
  Mozilla/5.0 (Macintosh; Intel Mac OS X 10.9; rv:28.0) Gecko/20100101 Firefox/28.0 (FlipboardProxy/1.6; +http://flipboard.com/browserproxy)
  larbin_2.6.4 ([email protected])
  yacybot (/global; amd64 Windows 7 6.1; java 1.7.0_55; Europe/ru) http://yacy.net/bot.html
  BacklinkCrawler (http://www.backlinktest.com/crawler.html)
  Mozilla/5.0 (Windows NT 6.1; WOW64) adbeat.com/policy AppleWebKit/537.36 (KHTML, like Gecko) Chrome/48.0.2564.116 Safari/537.36
  Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/534.57.2 (KHTML, like Gecko) Version/5.1.7 Safari/534.57.2 PTST/276
  Tools4noobs.com/1.0 Spider
  yacybot (freeworld/global; amd64 Linux 2.6.32-40-generic; java 1.6.0_20; Europe/de) http://yacy.net/bot.html
  Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US) Speedy Spider (http://www.entireweb.com/about/search_tech/speedy_spider/)
  istellabot/t.1
  Mozilla/5.0 (compatible; Lipperhey-Kaus-Australis/5.0; +https://www.lipperhey.com/en/about/)
  Mozilla/5.0 (compatible; Veoozbot/1.0; +http://www.veooz.com/veoozbot.html)
  SemrushBot/Nutch-1.5-SNAPSHOT
  Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)
  Mozilla/5.0 (X11; Linux x86_64; rv:47.0; GTmetrix https://gtmetrix.com/) Gecko/20100101 Firefox/47.0
  CopperEgg/RevealUptime/Oregon(aws)
  yacybot (/global; amd64 Linux 3.16.0-4-amd64; java 1.8.0_102; Europe/en) http://yacy.net/bot.html
  rogerbot/1.0 (http://www.seomoz.org/dp/rogerbot, [email protected])
  yacybot (amd64 Linux 2.6.26-2-amd64; java 1.6.0_20; Europe/en) http://yacy.net/bot.html
  Mozilla/5.0 (Windows NT 6.3; WOW64; Trident/7.0; rv:11.0; BingPreview/1.0b) like Gecko
  Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 5.1; Trident/4.0; .NET CLR 1.1.4322; .NET CLR 2.0.50727; .NET CLR 3.0.04506.30; .NET CLR 3.0.4506.2152; .NET CLR 3.5.30729; MDDS; PTST 2.386)
  Mozilla/5.0 (compatible; RavenCrawler/2.0; +https://raventools.com/seo-website-auditor/)
  Mozilla/5.0 (compatible; YandexMetrika/2.0; +http://yandex.com/bots)
  Mozilla/5.0 (compatible; Scrubby/3.2; +http://seotools.scrubtheweb.com/webpage-analyzer.html)
  dubaiindex (adressendeutschland.de)
  Mozilla/5.0 (compatible; spbot/5.0.2; +http://OpenLinkProfiler.org/bot )
  Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; Netcraft SSL Server Survey - contact [email protected])
  Mozilla/5.0 (compatible; CompSpyBot/1.0; +http://www.compspy.com/spider.html)
  Mozilla/5.0 (compatible; LA2; +http://www.zeerch.com/zeerch2/bot.php)
  yacybot (/global; amd64 Linux 2.6.32-042stab108.8; java 1.7.0_91; America/en) http://yacy.net/bot.html
  updated/0.1-alpha (updated crawler; http://www.updated.com; [email protected])
  CopperEgg/RevealUptime/FrankfurtGermany
  Scrapy/1.1.2 (+http://scrapy.org)
  yrspider (Mozilla/5.0 (compatible; YRSpider; +http://www.yunrang.com/yrspider.html))
  Mozilla/5.0 (compatible; YandexAntivirus/2.0; +http://yandex.com/bots)
  yacybot (freeworld/global; amd64 Windows 7 6.1; java 1.6.0_31; Europe/de) http://yacy.net/bot.html
  Surphace Scout&v4.0 - scout at surphace dot com
  yacybot (freeworld/global; amd64 Linux 2.6.32-5-xen-amd64; java 1.6.0_18; Europe/fr) http://yacy.net/bot.html
  MetaGeneratorCrawler/1.1 (www.metagenerator.info)
  Porkbun/Mustache (Website Analysis; http://porkbun.com; [email protected])
  Mozilla/5.0 (compatible; ScoutJet; +http://www.scoutjet.com/)
  Ruby, link_thumbnailer
  yacybot (freeworld/global; amd64 Windows Server 2008 R2 6.1; java 1.7.0_25; Europe/de) http://yacy.net/bot.html
  yacybot (/global; amd64 Linux 3.13.0-042stab093.4; java 1.7.0_79; Europe/de) http://yacy.net/bot.html
  Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.11 (KHTML, like Gecko) Chrome/23.0.1271.64 Safari/537.11 GotSiteMonitor.com
  Mozilla/5.0 (Linux; Android 4.4; Nexus 5 Build/KRT16M) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/40.0.2125.111 Mobile Safari/537.36 DareBoost
  Mozilla/5.0 (KeepRight OpenStreetMap Checker; http://keepright.at) Gecko/20100101 Firefox/22.0
  Mozilla/5.0 (Windows NT 6.1; WOW64; Trident/7.0; rv:11.0; topster.de Linkchecker 5.0) like Gecko (194.228.205.74)
  Mozilla/5.0 AppleWebkit/537.36 (KHTML, like Gecko) Trident/7.0; rv:11.0, Chrome/42.0.2311.135, Edge/12.10136, http://www.shrinktheweb.com/, Webshot/0.9
  topicbot/1.0 (Mozilla;I;+http://www.topic.bot/contact_page.html)
  DataparkSearch/4.53 (+http://dataparksearch.org/bot)
  Mozilla/5.0 (compatible; Linux; InfegyAtlas/1.0; en-US; [email protected])
  Mozilla/5.0 (X11; Linux x86_64; rv:41.0; GTmetrix https://gtmetrix.com/) Gecko/20100101 Firefox/41.0
  Mozilla/5.0 (compatible; EasouSpider; +http://www.easou.com/search/spider.html)
  Mozilla/5.0 (compatible; parsijoo-update-crawler; +http://www.parsijoo.ir/; [email protected])
  Abrave Spider v4 Robot 1 (http://robot.abrave.co.uk)
  Mozilla/5.0 (compatible; memoryBot/1.21.24 +http://internetmemory.org/en/)
  Mozilla/5.0 (compatible; STINGbot/1.0; +http://136.186.231.16)
  domainsbot (+http://www.domainsbot.com)
  WEPA/3.1 (http://www.wepa.com/; [email protected])
  Mozilla/5.0 (compatible; linkdexbot/2.1; +http://www.linkdex.com/bots/)
  Mozilla/5.0 (compatible; forensiq; +http://www.forensiq.com)
  192.comAgent
  JUST-CRAWLER(+http://www.justsystems.com/jp/tech/crawler/)
  Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm) SitemapProbe
  HuaweiSymantecSpider/[email protected]+(compatible; MSIE 7.0; Windows NT 5.1; Trident/4.0; .NET CLR 2.0.50727; .NET CLR 3.0.4506.2152; .NET CLR ; http://www.huaweisymantec.com/en/IRL/spider)
  Mozilla/5.0 ( Macintosh; Intel Mac OS X 10_10_1 ) AppleWebKit/600.2.5 ( KHTML, like Gecko ) Version/8.0.2 Safari/600.2.5 ( compatible; CloudServerMarketSpider/1.0; +http://cloudservermarket.com/spider.html )
  Mozilla/5.0 (compatible; MJ12bot/v1.3.0; http://www.majestic12.co.uk/bot.php?+)
  Mozilla/5.0 (compatible; AcoonBot/4.10.6; +http://www.acoon.de/robot.asp)
  Scrapy/1.0.1 (+http://scrapy.org)
  seegnifybot/1.0.0 (http://www.seegnify.com/bot)
  omgili/0.5 +http://omgili.com
  Slack-ImgProxy 149 (+https://api.slack.com/robots)
  yacybot (/global; amd64 Linux 3.16.0-4-amd64; java 1.7.0_79; Europe/de) http://yacy.net/bot.html
  www.integromedb.org/Crawler
  Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm
  Mozilla/5.0 (compatible) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/47.0.2526.73 Safari/537.36 [email protected]
  Mozilla/5.0 (compatible; IstellaBot/1.18.81 +http://www.tiscali.it/)
  Gigabot/1.0
  page_test ([email protected])
  istellabot/Nutch-1.10
  Mozilla/5.0 (iPhone; U; CPU iPhone OS 4_1 like Mac OS X; en-us) AppleWebKit/532.9 (KHTML, like Gecko) Version/4.0.5 Mobile/8B117 Safari/6531.22.7 (compatible; Mediapartners-Google/2.1; +http://www.google.com/bot.html)
  ldspider (http://code.google.com/p/ldspider/wiki/Robots)
  yacybot (/global; amd64 Windows 8.1 6.3; java 1.8.0_45; Europe/ru) http://yacy.net/bot.html
  facebookplatform/1.0 (+http://developers.facebook.com)
  Mozilla/5.0 (compatible; RTGI; http://linkfluence.net/)
  yacybot (freeworld/global; amd64 Linux 3.12.43-52.6-default; java 1.8.0_40; Europe/en) http://yacy.net/bot.html
  Mozilla/5.0 (compatible; YandexBot/3.0; MirrorDetector; +http://yandex.com/bots)
  Mozilla/5.0 (compatible; DCPbot/1.2; +http://domains.checkparams.com/)
  WatchMouse/18990 (http://watchmouse.com/ ; d2.watchmouse.com)
  MnoGoSearch/3.3.15
  Mozilla/5.0 (compatible; AboutUsBot/0.9; +http://www.aboutus.org/AboutUsBot)
  Web-sniffer/1.1.0 (+http://web-sniffer.net/)
  Mozilla/4.0 (compatible; Fooooo_Web_Video_Crawl http://fooooo.com/bot.html)
  50.nu/0.01 ( +http://50.nu/bot.html )
  Mozilla/4.0 (compatible; MSIE 7.0; Windows; Windows NT 5.1) BrokenLinkCheck.com/1.1
  Comodo-Certificates-Spider
  Iframely/0.9.8 (+https://iframely.com/;)
  MnoGoSearch/3.3.12
  CopperEgg/RevealUptime/FrankfurtEU
  Python-urllib/2.7 (+http://ella.juls.savba.sk/aranea_about)
  Mozilla/5.0 (compatible; JobKereso; +http://www.kozvetlen-allasok.hu/robot.jsp [email protected])
  Mozilla/5.0 (Windows NT 5.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/49.0.2623.112 Safari/537.36 PTST/201
  Mozilla/5.0 (compatible; Google Keyword Tool; +http://adwords.google.com/select/KeywordToolExternal)
  WinWebBot/1.0; (Balaena Ltd, UK); http://www.balaena.com/winwebbot.html; [email protected];)
  AppleNewsBot
  classbot (+http://allclasses.com)
  yacybot (freeworld/global; amd64 Windows Server 2008 R2 6.1; java 1.6.0_31; America/pt) http://yacy.net/bot.html
  Mozilla/5.0 (iPhone; CPU iPhone OS 7_0 like Mac OS X) AppleWebKit/537.51.1 (KHTML, like Gecko) Version/7.0 Mobile/11A465 Safari/9537.53 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)
  yacybot (/global; arm Linux 3.12.28+; java 1.7.0_71; Europe/en) http://yacy.net/bot.html
  Mozilla/5.0 (compatible; YandexMetrika/2.0; +http://yandex.com/bots mtmon01e.yandex.ru)
  CopperEgg/RevealUptime/TokyoJP
  Mozilla/5.0 (compatible; AhrefsBot/3.1; +http://ahrefs.com/robot/)
  CJNetworkQuality; http://www.cj.com/networkquality
  psbot-page (+http://www.picsearch.com/bot.html)
  DealGates Bot/1.1 by Luc Michalski (http://spider.dealgates.com/bot.html)
  Pingdom GIGRIB (http://www.pingdom.com)
  Mozilla/5.0 (compatible; SEOlyticsCrawler/3.0; +http://crawler.seolytics.net/)
  Mozilla/5.0 (compatible; EventGuruBot/1.0; +http://www.eventguru.com/spider.html)
  Python-urllib/2.7 (+http://aranea.juls.savba.sk/aranea_about.html)
  yacybot (/global; amd64 Windows Vista 6.0; java 1.7.0_55; America/en) http://yacy.net/bot.html
  The Lemur Web Crawler/Nutch-1.3 (Lemur Web Crawler; http://boston.lti.cs.cmu.edu/crawler_12/; [email protected])
  SAMSUNG-SGH-E250/1.0 Profile/MIDP-2.0 Configuration/CLDC-1.1 UP.Browser/6.2.3.3.c.1.101 (GUI) MMP/2.0 (compatible; Googlebot-Mobile/2.1; +http://www.google.com/bot.html)
  Feedfetcher-Nuesbyte; (+http://www.nuesbyte.com)
  copyright sheriff (+http://www.copyrightsheriff.com/)
  yacybot (/global; amd64 Linux 3.2.0-4-amd64; java 1.7.0_75; Europe/en) http://yacy.net/bot.html
  Touche (+http://www.touche.com.ve)
  Panopta v1.1
  Mozilla/5.0 (compatible; parsijoo-bot; +http://www.parsijoo.ir/; [email protected])
  Zookabot/2.2;++http://zookabot.com
  yacybot (/global; amd64 Linux 2.6.32-042stab093.4; java 1.7.0_65; Etc/en) http://yacy.net/bot.html
  it2media-domain-crawler/2.0
  yacybot (/global; amd64 Linux 4.2.0-27-generic; java 1.7.0_95; Europe/en) http://yacy.net/bot.html
  Mozilla/5.0 (Android; Mobile; rv:21.0) Gecko/21.0 Firefox/21.0 commoncrawl.org/research//Nutch-1.7-SNAPSHOT
  Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 5.2; Trident/4.0; Data Center; .NET CLR 1.1.4322; .NET CLR 2.0.50727; .NET CLR 3.0.04506.648; .NET CLR 3.5.21022; .NET CLR 3.0.4506.2152; .NET CLR 3.5.30729; PTST 2.386)
  Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/99.0 Safari/537.36 YottaaMonitor
  CopperEgg/RevealUptime/AtlantaGAUSA
  Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
  yacybot (/global; amd64 Windows 7 6.1; java 1.7.0_55; Europe/fr) http://yacy.net/bot.html
  Mozilla/5.0 (compatible; MJ12bot/v1.2.4; http://www.majestic12.co.uk/bot.php?+)
  Mozilla/5.0 (compatible; Applebot/0.3; +http://www.apple.com/go/applebot)
  ICC-Crawler/2.0 (Mozilla-compatible; ; http://www.nict.go.jp/en/ucri/plan/crawl.html)
  Mozilla/5.0 (compatible; YYSpider; +http://www.yunyun.com/spider.html)
  Mozilla/5.0 (iPhone; CPU iPhone OS 5_0_1 like Mac OS X) (compatible; Yeti-Mobile/0.1; +http://help.naver.com/robots/)
  Mozilla/5.0 (WhatsMyIP.org Text_to_Code_Ratio_Tool) http://whatsmyip.org/ua
  Thumbnail.CZ robot 1.1 (http://thumbnail.cz/why-no-robots-txt.html)
  Google-Calendar-Importer
  Mozilla/5.0 (Windows NT 6.3; WOW64) AppleWebKit/534.34 (KHTML, like Gecko) Websnapr/3.0 Safari/534.34
  Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/50.0.2661.75 Safari/537.36 PTST/276
  WeSEE
  librabot/2.0 (+http://academic.research.microsoft.com/)
  Testomatobot/1.0 (Linux x86_64; +http://www.testomato.com/testomatobot) minicrawler/3.0.1
  Mozilla/5.0 (compatible; MSIE 10.0; Windows NT 6.1; Trident/6.0) PTST/276
  Cliqzbot/0.1 (+http://cliqz.com [email protected])
  Scrubby/3.0 (+http://www.scrubtheweb.com/help/technology.html)
  Pinterest/0.1 +http://pinterest.com/
  Mozilla/5.0 (compatible; Crawlera/1.10.2; UID 52327)
  Comodo-Webinspector-Crawler 2.2.0, http://www.comodorobot.com
  Iframely/0.6.0 (+http://iframely.com/;)
  Mozilla/5.0 (compatible; PaperLiBot/2.1; http://support.paper.li/entries/20023257-what-is-paper-li)
  Mozilla/5.0 (compatible; SiteExplorer/1.0b; +http://siteexplorer.info/)
  Y!J-BSC/1.0 (http://help.yahoo.co.jp/help/jp/blog-search/)
  Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/49.0.2623.112 Safari/537.36 PTST/276
  Mozilla/5.0 (compatible; U; Hootsuite-WebFeed/1.0; mailto:[email protected])
  RankFlex.com Webspider
  TipTop http://feeltiptop.com
  vebidoobot
  Mozilla/5.0 (compatible; Dow Jones Searchbot)
  Mozilla/5.0 (compatible; OrangeBot-Collector/2.0; [email protected])
  Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/534+ (KHTML, like Gecko) MsnBot-Media /1.0b
  Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1.6) Gecko/20070725 Firefox/2.0.0.6 Ara.com.tr AraBot 1.0
  Mozilla/5.0 (iPhone; CPU iPhone OS 9_1 like Mac OS X) AppleWebKit/601.1.46 (KHTML, like Gecko) Version/9.0 Mobile/13B143 Safari/601.1 (compatible; AdsBot-Google-Mobile; +http://www.google.com/mobile/adsbot.html)
  Mozilla/5.0 (compatible; YandexMetrika/2.0; +http://yandex.com/bots mtmon01g.yandex.ru)
  BUbiNG (+http://law.di.unimi.it/BUbiNG.html)
  downnotifier.com monitoring
  A6-Indexer
  Mozilla/5.0 (compatible; htrix/1.14.4 ++https://webarchiv.onb.ac.at/robot.html)
  MetaURI API/2.0 +metauri.com
  Mozilla/5.0 (compatible; MixrankBot; [email protected])
  Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/44.0.2403.155 Safari/537.36 TinEye/1.0 (via http://www.tineye.com/)
  Mozilla/5.0 Project 25499 (project25499.com)
  COMODOspider/Nutch-1.0
  Mozilla/5.0 (compatible; CareerBot/1.1; +http://www.career-x.de/bot.html)
  Uptime.com_(http://uptime.com/)
  AcoonBot/4.10.5 (+http://www.acoon.de)
  BDFetch
  20090717_aa_page ([email protected])
  Metaspinner/1.0 (Metaspinner Search Engine; http://www.meta-spinner.de/; [email protected])
  Mozilla/5.0 (compatible; DBLBot/1.0; +http://www.dontbuylists.com/)
  Mozilla/5.0 (compatible; spbot/2.0; +http://www.seoprofiler.com/bot/ )
  LYT.SR v1.5 +http://lyt.sr/ (running on http://lyt.sr)
  Putin (+http://semanticjuice.com/)
  Mozilla/5.0 (compatible; LA1; +http://www.zeerch.com/bot.php)
  WatchMouse/18990 (http://watchmouse.com/ ; ny)
  bitlybot/2.0
  yacybot (/global; amd64 Linux 3.13.0-45-generic; java 1.7.0_75; Europe/en) http://yacy.net/bot.html
  Webshot/0.9 ( http://www.shrinktheweb.com/ )
  MetaURI API +metauri.com
  Mozilla/5.0 (compatible; ProductoDownloadUrlBot/1.0; +http://www.producto.de/)
  Google-Adwords-Instant (+http://www.google.com/adsbot.html)
  yacybot (freeworld/global; amd64 Linux 3.8.0-19-generic; java 1.7.0_25; Europe/en) http://yacy.net/bot.html
  yacybot (freeworld/global; i386 Linux 2.6.32-5-686; java 1.6.0_18; Europe/en) http://yacy.net/bot.html
  Mozilla/5.0 (Windows NT 6.3; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/42.0.2311.90 Safari/537.36 PTST/277
  Mozilla/5.0 (compatible; DomainSigmaCrawler/0.1; +http://domainsigma.com/robot)
  HyperZbozi.cz Feeder/3.1
  Mozilla/5.0 (Windows; U; Windows NT 5.1; en; rv:1.9.0.13) Gecko/2009073022 Firefox/3.5.2 (.NET CLR 3.5.30729) SurveyBot/2.3 (DomainTools)
  yacybot (/global; amd64 Windows Server 2012 R2 6.3; java 1.7.0_55; Europe/de) http://yacy.net/bot.html
  Google-AdWords-Express
  Mozilla/5.0 (compatible; MJ12bot/v1.4.0; http://www.majestic12.co.uk/bot.php?+)
  SMTBot (similartech.com/smtbot)
  Mozilla/5.0 (compatible; Lipperhey SEO Service; http://www.lipperhey.com/)
  MetaInspector/4.7.2 (+https://github.com/jaimeiniesta/metainspector)
  Mozilla/5.0 (Compatible; Vedma/0.91Beta; +http://www.vedma.ru/bot.htm)
  Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.21 (KHTML, like Gecko) web-capture.net/0.9 Safari/537.21
  Nokia6680/1.0 (4.04.07) SymbianOS/8.0 Series60/2.6 Profile/MIDP-2.0 Configuration/CLDC-1.1 (botmobi find.mobi/bot.html [email protected])
  Mozilla/5.0 (compatible; Exploratodo/1.0; +http://www.exploratodo.com
  iZSearch.com
  Mozilla/5.0 (compatible; YandexFavicons/1.0; +http://yandex.com/bots)
  bl.uk_lddc_bot/3.3.0-SNAPSHOT-2014-10-07T09:33:31Z (+http://www.bl.uk/aboutus/legaldeposit/websites/websites/faqswebmaster/index.html)
  yacybot (/global; amd64 Linux 3.14-0.bpo.1-amd64; java 1.7.0_55; Europe/de) http://yacy.net/bot.html
  HuaweiSymantecSpider/[email protected]+(compatible; MSIE 7.0; Windows NT 5.1; Trident/4.0; .NET CLR 2.0.50727; .NET CLR 3.0.4506.2152; .NET CLR ; http://www.huaweisymantec.com/cn/IRL/spider)
  FAST-WebCrawler/3.7/FirstPage (atw-crawler at fast dot no;http://fast.no/support/crawler.asp)
  Mozilla/5.0 (compatible; DripfeedBot/2.0; +http://dripfeedbookmark.com/bot.html
  Mozilla/5.0 (compatible; heritrix/2.0.2 +http://seekda.com)
  yacybot (freeworld/global; i386 Linux 3.2.0-23-generic-pae; java 1.7.0_03; Europe/en) http://yacy.net/bot.html
  hawkReader/0.9b (Feed Parser; http://www.hawkreader.com; Allow like Gecko)
  ruijie ([email protected])
  CorpusCrawler 2.0.0 (http://corpora.fi.muni.cz/crawler/)
  ExactSeek Crawler (http://www.exactseek.com/)/Nutch-1.4
  WordPress.com; https://botd.wordpress.com
  Mozilla/5.0 (compatible; Qwantify/2.0n; +https://www.qwant.com/)/*
  Mozilla/5.0 (compatible; memoryBot/1.20.268 +http://internetmemory.org/en/)
  ShowyouBot (http://showyou.com/crawler)
  Yoleo Consumer v0.2
  Mozilla/5.0 (compatible; IstellaBot/1.10.2 +http://www.tiscali.it/)
  Mozilla/5.0 (compatible; MSIE or Firefox mutant; not on Windows server; +http://ws.daum.net/aboutWebSearch.html) Daumoa/3.0
  Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/46.0.2490.86 YaBrowser/15.12.0.6151 Safari/537.36 Yandex.Translate
  Mozilla/5.0 (compatible; TeeRaidBot; +https://teeraid.com/bot/)
  Mozilla/5.0 (compatible; Scopia Crawler 1.1; +http://www.scopia.co)
  CopperEgg/RevealUptime/DublinIE(aws)
  yacybot (freeworld/global; amd64 Windows 7 6.1; java 1.7.0_04; Asia/ja) http://yacy.net/bot.html
  RankurBot/Rankur2.1 (http://rankur.com; info at rankur dot com)
  Gaisbot/3.0+([email protected];+http://gais.cs.ccu.edu.tw/robot.php)
  Mozilla/5.0 (compatible; zitebot support [at] zite [dot] com +http://zite.com)
  MetaTagRobot/2.0 (http://www.check-domains.com/website-analysis/website-analyzer.php)
  Virusdie crawler/2.1
  findlinks/2.0.4 (+http://wortschatz.uni-leipzig.de/findlinks/)
  Mozilla/5.0 (compatible; AcoonBot/4.10.7; +http://www.acoon.de/robot.asp)
  Mozilla/5.0 (compatible; DeuSu/5.0.2; +https://deusu.de/robot.html)
  yacybot (-global; amd64 Linux 3.19.0-15-generic; java 1.8.0_45-internal; Europe/de) http://yacy.net/bot.html
  Google_Analytics_Snippet_Validator
  SurcentroBot
  Mozilla/5.0 (compatible; Diffbot/0.1; +http://www.diffbot.com)
  findlinks/1.1.3-beta9 (+http://wortschatz.uni-leipzig.de/findlinks/)
  Mozilla/5.0 (compatible; memoryBot/1.20.199 +http://internetmemory.org/en/)
  Y!J-ASR/0.1 crawler (http://www.yahoo-help.jp/app/answers/detail/p/595/a_id/42716/)
  TwengaBot-Discover (http://www.twenga.fr/bot-discover.html)
  Mozilla/5.0 (bl.uk_lddc_bot; Linux x86_64) PhantomJS/1.9.7 (+http://www.bl.uk/aboutus/legaldeposit/websites/websites/faqswebmaster/index.html)
  panscient.com
  Mozilla/5.0 (compatible; TestCrawler; +http://wiki.github.com/bixo/bixo/bixocrawler; [email protected])
  DataparkSearch/4.37-23012006 ( http://www.dataparksearch.org/)
  GOFORITBOT ( http://www.goforit.com/about/ )
  CopperEgg/RevealUptime/SydneyAustralia
  msnbot-media/1.0 (+http://search.msn.com/msnbot.htm)
  Mozilla/5.0 (compatible; MegaIndex.ru/2.0; +https://www.megaindex.ru/?tab=linkAnalyze)
  MetaCommentBot; http://metacomment.io/about
  Mozilla/5.0 (compatible; YesupBot/1.0; +http://www.yesup.net/bot.html)
  Twikle/1.0 , http://twikle.com , [email protected]
  yacybot (/global; amd64 Windows Server 2008 R2 6.1; java 1.8.0_60; Europe/de) http://yacy.net/bot.html
  Mozilla/4.0 (compatible; Vagabondo/4.0Beta; webcrawler at wise-guys dot nl; http://webagent.wise-guys.nl/; http://www.wise-guys.nl/)
  Comodo-Webinspector-Crawler 2.1
  Mozilla/5.0 (Macintosh; Intel Mac OS X 10_7_5) AppleWebKit/537.11 (KHTML, like Gecko) Chrome/23.0.1271.64 Safari/537.11 KimonoLabs/0.2
  Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1 + FairShare-http://fairshare.cc)
  Scrapy/0.24.5 (+http://scrapy.org)
  Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/534.34 (KHTML, like Gecko) Site-Shot/2.1 (http://www.site-shot.com/) Safari/534.34
  Port Monitor check service 1.0 | http://www.port-monitor.com
  Mozilla/5.0 (compatible; Uptimebot/0.1.43; +http://www.uptime.com/uptimebot)
  yacybot (freeworld/global; amd64 Linux 2.6.32-5-amd64; java 1.6.0_18; Europe/en) http://yacy.net/bot.html
  SolomonoBot/1.04 (http://www.solomono.ru)
  Easy-Thumb (https://www.easy-thumb.net/)
  Mozilla/5.0 (Windows NT 6.0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/49.0.2623.112 Safari/537.36 PTST/276
  nic.it crawler v.1.221115 (7.35.0)
  Mozilla/5.0 (compatible; memorybot/1.24.12 +https://internetmemory.net/en/bots)
  Mozilla/5.0 (compatible; inoreader.com-like FeedFetcher-Google)
  yacybot (/global; amd64 Windows 7 6.1; java 1.7.0_72; Europe/de) http://yacy.net/bot.html
  WatchMouse/8.4.0.3 (http://watchmouse.com/ ; demuc02.watchmouse.net)
  Mozilla/5.0 (X11; Linux x86_64; rv:43.0; GTmetrix https://gtmetrix.com/) Gecko/20100101 Firefox/43.0
  ShopWiki/1.0 ( +http://www.shopwiki.com/wiki/Help:Bot)
  Mozilla/5.0 (compatible; UnisterBot; http://www.bluekiwi.de/misc/imprint)
  Web-sniffer/1.0.31 (+http://web-sniffer.net/)
  woobot
  Mozilla/5.0 (X11; U; Linux Core i7-4980HQ; de; rv:32.0; compatible; JobboerseBot; https://www.jobboerse.com/bot.htm) Gecko/20100401 Firefox/24.0
  AntBot/Ant-Nutch-1.1 (Ant Nutch Crawler; http://www.ant.com; [email protected])
  Mozilla/5.0 (compatible; YandexMedia/3.0; +http://yandex.com/bots)
  Mozilla/5.0 (compatible; Evrinid Iudex 1.0.0; +http://www.evri.com/evrinid)
  adidxbot/1.1 (+http://search.msn.com/msnbot.htm)
  Mozilla/5.0 (compatible; WebThumbnail/2.2; Website Thumbnail Generator; +http://webthumbnail.org)
  Mozilla/5.0 (compatible; Embedly/0.2; +http://support.embed.ly/)
  EasyBib AutoCite (http://content.easybib.com/autocite/)
  yacybot (-global; amd64 Linux 3.2.0-4-amd64; java 1.7.0_65; Europe/en) http://yacy.net/bot.html
  Mozilla/5.0 (compatible; woriobot +http://worio.com)
  Mozilla/4.0 (compatible; Vagabondo/4.0; http://webagent.wise-guys.nl/)
  SniffRSS/0.5beta (+http://www.blogator.com/)
  woobot/1.1
  Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) WordPress.com mShots Safari/537.36
  ADmantX Platform Semantic Analyzer - Turn - ADmantX Inc. - www.admantx.com - [email protected]
  Mozilla/5.0 (compatible; UnisterBot; [email protected])
  Mozilla/5.0 (compatible; electricmonk/3.2.0 +https://www.duedil.com/our-crawler/)
  UptimeDog Robot (www.uptimedog.com)
  Googlebot/2.1 (+http://www.google.com/bot.html)
  Mozilla/5.0 (Windows NT 6.3; WOW64; rv:36.0) Gecko/20100101 Firefox/36.0 (NetShelter ContentScan, contact [email protected] for information)
  Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.0; Trident/4.0; SLCC1; .NET CLR 2.0.50727; Media Center PC 5.0; .NET CLR 3.5.30729; .NET4.0C; .NET CLR 3.0.30729; .NET4.0E) PTST/276
  ServiceUptime.robot
  Mozilla/4.0 (compatible;HostTracker/2.0;+http://www.host-tracker.com/)
  SEO Browser
  Leikibot/1.0 (+http://www.leiki.com)
  yacybot (webportal-global; amd64 Windows 7 6.1; java 1.7.0_04; America/en) http://yacy.net/bot.html
  AppEngine-Google; (+http://code.google.com/appengine; appid: downforeveryoneorjustme)
  Mozilla/5.0 (Windows NT 5.1) BrokenLinkCheck.com/1.1
  Seobility (SEO-Check; http://bit.ly/1dJuuzs)
  Mozilla/5.0 (seoanalyzer; compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)
  Mozilla/5.0 (compatible; Digincore bot; https://www.digincore.com/crawler.html for rules and instructions.
  Site24x7
  websitepulse checker/3.0 (compatible; MSIE 5.5; Netscape 4.75; Linux)
  Mozilla/5.0 (compatible; SemrushBot-SA/0.97; +http://www.semrush.com/bot.html)
  Mozilla/5.0 (compatible; NerdByNature.Bot; http://www.nerdbynature.net/bot)
  Abrave Spider v4 Robot 2 (http://robot.abrave.co.uk)
  HubPages V0.2.2 (http://hubpages.com/help/crawlingpolicy)
  lmspider ([email protected])
  magpie-crawler/1.1 (U; Linux amd64; en-GB; +http://www.brandwatch.net)
  Mozilla/5.0 (compatible; WbSrch/1.1 +http://wbsrch.com
  Mozilla/5.0 (Windows; U; Windows NT 5.1; de; rv:1.9.0.7; Google-SearchByImage) Gecko/2009021910 Firefox/3.0.7
  Mozilla/5.0 (compatible; Najdi.si/3.1)
  larbin_2.6.3 ([email protected])
  Mozilla/5.0 (compatible; aiHitBot/1.1; +http://www.aihit.com/)
  Mozilla/5.0 (Windows NT 6.3; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/50.0.2661.102 Safari/537.36 PTST/277
  Mozilla/5.0 (YahooYSMcm/3.0.0; http://help.yahoo.com)
  yacybot (/global; amd64 Linux 3.16.0-4-amd64; java 1.7.0_79; Europe/en) http://yacy.net/bot.html
  OOZBOT/0.20 ( http://www.setooz.com/oozbot.html ; agentname at setooz dot_com )
  findlinks/2.0.9 (+http://wortschatz.uni-leipzig.de/findlinks/)
  Mozilla/5.0 (compatible; Lipperhey Link Explorer; http://www.lipperhey.com/)
  Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.6; en-US; rv:1.9.2) Gecko/20100115 Firefox/3.6 (FlipboardProxy/1.1; +http://flipboard.com/browserproxy)
  Mozilla/5.0 (Twiceler-0.9 http://www.cuil.com/twiceler/robot.html)
  Mozilla/5.0 (compatible; memoryBot/1.24.11 +http://internetmemory.org/en/)
  RED/1 (https://redbot.org/)
  Twitterbot/1.0
  Mozilla/5.0 (compatible; ptd-crawler; +http://bixolabs.com/crawler/ptd/; [email protected])
  Open Web Analytics Bot 1.5.4
  JikeSpider Mozilla/5.0 (compatible; JikeSpider; +http://shoulu.jike.com/spider.html)
  Mozilla/5.0 (compatible; vkShare; +http://vk.com/dev/Share)
  Mozilla/5.0 (compatible; oBot/2.3.1; +http://www-935.ibm.com/services/us/index.wss/detail/iss/a1029077?cntxt=a1027244)
  elefent/Elefent 1.2 (A friendly web elefent.; http://elefent.eu/; [email protected])
  Genderanalyzer/1.0
  elefent/1.2 (Web Crawler; https://crawler.elefent.net; webmaster at elefent dot net)
  Yandex/1.01.001 (compatible; Win16; H)
  crawler4j for XQuery
  Mozilla/5.0 (compatible; Plukkie/1.4; http://www.botje.com/plukkie.htm)
  LinguaBot/v0.001-dev (MultiLinual Sarch Engine v0.001; LinguaSeek; admin at linguaseek dot com)
  MrakyPraceCzBot/1.0 (http://www.mrakyprace.cz/about-crawler)
  Y!J-BRJ/YATS crawler (http://help.yahoo.co.jp/help/jp/search/indexing/indexing-15.html)
  WatchMouse/8.4.0.3 (http://watchmouse.com/ ; plgdn01.watchmouse.net)
  Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2272.96 Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
  AskQuickly v2 (http://askquickly.org/)
  Mozilla/5.0 (compatible; SISTRIX Crawler; http://crawler.sistrix.net/)
  Miniflux (http://miniflux.net)
  Mozilla/5.0 (compatible; spbot/5.0.1; +http://OpenLinkProfiler.org/bot )
  Mozilla/5.0 (compatible; wmtips.com/1.0; +http://www.wmtips.com/tools/)
  Mozilla/5.0 (compatible; linkdexbot/2.0; +http://www.linkdex.com/bots/)
  Mozilla/2.0 (compatible; Ask Jeeves/Teoma; +http://sp.ask.com/docs/about/tech_crawling.html)
  Google favicon
  Mozilla/5.0(compatible; Sosospider/2.0; +http://help.soso.com/webspider.htm)
  MiaDev/0.0.1 (MIA Bot for research project MIA (www.MIA-marktplatz.de); http://www.mia-marktplatz.de/spider; [email protected])
  Mozilla/5.0 (compatible; proximic; +http://www.proximic.com/info/spider.php)
  Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/33.0.1750.146 Safari/537.36 Specificfeeds- http://www.specificfeeds.com
  Mozilla/5.0 (compatible; socialbm_bot/1.0; +http://spider.socialbm.net)
  wp.com feedbot/1.0 (+https://wp.com)
  Mozilla/5.0 (compatible; CligooRobot/2.0; +http://www.cligoo.de/wk/technik.php)
  Slack-ImgProxy (+https://api.slack.com/robots)
  Mozilla/5.0 (compatible; Online Domain Tools - HTTP Headers Online/1.0; +http://http-headers.online-domain-tools.com)
  Mozilla/5.0 (compatible; SurdotlyBot/1.0; +http://sur.ly/bot.html)
  WGETbot/1.0 (+http://wget.alanreed.org)
  yacybot (/global; amd64 Linux 4.0.9-2-ck; java 1.8.0_51; Europe/de) http://yacy.net/bot.html
  Y!J-BRI/0.0.1 crawler ( http://help.yahoo.co.jp/help/jp/search/indexing/indexing-15.html )
  dlvr.it/1.0 (+http://dlvr.it/) Mozilla/5.0
  yacybot (/global; arm Linux 4.1.13-v7+; java 1.8.0; Etc/de) http://yacy.net/bot.html
  Mozilla/5.0 (compatible; GimmeUSAbot/1.0; +http://gimmeusa.com/crawler.html)
  yacybot (freeworld/global; amd64 Windows 7 6.1; java 1.6.0_29; Europe/fr) http://yacy.net/bot.html
  Link Valet Online 1.2
  Mozilla/5.0 (compatible; METASpider; +http://meta.ua/spider)
  NeutrinoAPI/2.0.1
  thumbshots-de-bot (+http://www.thumbshots.de/)
  Mozilla/5.0 (iPhone; CPU iPhone OS 6_0 like Mac OS X) AppleWebKit/536.26 (KHTML, like Gecko) Version/6.0 Mobile/10A5376e Safari/8536.25 (compatible; Applebot/0.3; +http://www.apple.com/go/applebot)
  argenfybot
  MergadoBot
  Ant/Ant-Nutch-1.1 (Ant Nutch Crawler; http://www.ant.com; [email protected])
  CopperEgg/RevealUptime/SaoPauloBrazil
  Mozilla/4.0 (compatible; Vagabondo/4.0/EU; http://webagent.wise-guys.nl/)
  Mozilla/4.0 (compatible; DAWINCI ANTIPLAG SPIDER)
  Mozilla/5.0 (Windows NT 6.1; WOW64; rv:45.0) Gecko/20100101 Firefox/45.0 PTST/276
  HetrixTools.COM Uptime Monitoring Bot. https://hetrixtools.com/uptime-monitoring-bot.html
  yacybot (/global; amd64 Linux 3.2.0-4-amd64; java 1.7.0_79; Europe/en) http://yacy.net/bot.html
  Mozilla/5.0 (Windows NT 6.1) (compatible; SMTBot/1.0; +http://www.similartech.com/smtbot)
  Mozilla/5.0 (compatible; InbyBot/2.1; +http://inbyapp.com)
  all_web1 ([email protected])
  Mozilla/5.0 (compatible; BusinessSeek.biz_Directory; http://www.businessseek.biz/)
  Wotbox/2.0 ([email protected]; http://www.wotbox.com)
  Mediatoolkitbot ([email protected])
  OpenWebSpider v0.1.2.B (http://www.openwebspider.org/)
  Mozilla/5.0 (compatible; WebCorp/5.0; +http://www.webcorp.org.uk)
  Testomatobot/1.0 (Linux x86_64; +http://www.testomato.com/testomatobot) minicrawler/4.0.0
  fastbot crawler beta 2.0 (+http://www.fastbot.de)
  Sogou Web Spider
  DoCoMo/2.0 P900i(c100;TB;W24H11) (compatible; ichiro/mobile goo;+http://search.goo.ne.jp/option/use/sub4/sub4-1/)
  Mozilla/5.0 (compatible; imagecoccoc/1.0; +http://help.coccoc.com/searchengine)
  XoviBot/1.0
  Mozilla/5.0 (compatible; Online Domain Tools - Online Website Link Checker/1.1; +http://website-link-checker.online-domain-tools.com)
  WatchMouse/8.4.0.3 (http://watchmouse.com/ ; decgn02.watchmouse.net)
  yacybot (i386 Linux 2.6.31-21-generic; java 1.6.0_0; Europe/en) http://yacy.net/bot.html
  ZnajdzFoto/ImageBot 2.0b
  Mozilla/5.0 (compatible; heritrix/3.0.0-SNAPSHOT-20091120.021634 +http://crawler.archive.org)
  Mozilla/5.0 (compatible; ProCogSEOBot/1.0; +http://www.procog.com/ )
  Mozilla/5.0 (compatible; Plukkie/1.6; http://www.botje.com/plukkie.htm)
  yacybot (freeworld/global; amd64 Linux 3.2.0-21-generic; java 1.7.0_03-icedtea; America/en) http://yacy.net/bot.html
  Mozilla/5.0 (iPhone; CPU iPhone OS 8_3 like Mac OS X) AppleWebKit/600.1.4 (KHTML, like Gecko) Version/8.0 Mobile/12F70 Safari/600.1.4 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
  Mozilla/5.0 (compatible; MSIE or Firefox mutant; not on Windows server; + http://tab.search.daum.net/aboutWebSearch.html) Daumoa/3.0
  Mozilla/5.0 (compatible; NLNZ_IAHarvester2014 +http://natlib.govt.nz/about-us/current-initiatives/web-harvest-2014)
  Mozilla/5.0 (compatible; archive.org_bot +http://archive.org/details/archive.org_bot)
  Mozilla/5.0 (compatible; special_archiver/3.3.0 +http://www.loc.gov/webarchiving/notice_to_webmasters.html)
  yacybot (freeworld/global; i386 Linux 3.4.2-linode44; java 1.6.0_27; America/en) http://yacy.net/bot.html
  Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/47.0.2526.111 Safari/537.36 Yandex.Translate
  yacybot (webportal-global; amd64 Linux 3.2.0-4-amd64; java 1.7.0_25; Europe/en) http://yacy.net/bot.html
  FreeWebMonitoring SiteChecker/0.1 (+http://www.freewebmonitoring.com)
  Mozilla/5.0 (compatible; Infohelfer/1.3.0; +http://www.infohelfer.de/crawler.php)
  Kemvibot/1.0 (http://kemvi.com)
  Mozilla/4.0 (compatible; HostTracker.com/1.0;+http://host-tracker.com/)
  AdsBot-Google-Mobile (+http://www.google.com/mobile/adsbot.html) Mozilla (iPhone; U; CPU iPhone OS 3 0 like Mac OS X) AppleWebKit (KHTML, like Gecko) Mobile Safari
  Mozilla/5.0 (compatible; GeliyooBot/1.0; +http://www.geliyoo.com/)
  yacybot (x86 Windows XP 5.1; java 1.6.0_21; Europe/de) http://yacy.net/bot.html
  Mozilla/5.0 (compatible; archive.org_bot/3.1.2 +http://natlib.govt.nz/about-us/current-initiatives/web-harvest-2012)
  Mozilla/5.0 (compatible; StorygizeBot; http://www.storygize.com)
  infohelfer/1.1.0 (http://www.infohelfer.de/)
  yacybot (/global; amd64 Linux 3.16.0-41-generic; java 1.7.0_80; Europe/de) http://yacy.net/bot.html
  SeoCheckBot (FischerNetzDesign Seo Checker, [email protected])
  netEstate NE Crawler (+http://www.website-datenbank.de/)
  Mail.Ru/1.0
  binlar_2.6.3 ([email protected])
  LinkedInBot/1.0 (compatible; Mozilla/5.0; Jakarta Commons-HttpClient/3.1 +http://www.linkedin.com)
  bot.wsowner.com/1.0.0 (+http://wsowner.com/)
  Ronzoobot/1.5 (http://www.ronzoo.com/about/)
  Mozilla/5.0 (compatible; heritrix/3.2.0 +http://www.mixdata.com)
  Mozilla/5.0 (compatible; Yahoo! Slurp China; http://misc.yahoo.com.cn/help.html)
  findlinks/1.1.6-beta6 (+http://wortschatz.uni-leipzig.de/findlinks/)
  Mozilla/5.0 (compatible; LoadTimeBot/0.7; +http://www.load-time.com/bot.html)
  Mozilla/5.0 (compatible; MJ12bot/v1.4.2; http://www.majestic12.co.uk/bot.php?+)
  Mozilla/4.0 (compatible; MSIE 5.01; GomezAgent 2.0; Windows NT)
  Mozilla/5.0 (compatible; MJ12bot/v1.3.2; http://www.majestic12.co.uk/bot.php?+)
  SEOENGBot/1.2 (+http://learn.seoeng.com/seoengbot.htm)
  yacybot (i386 Linux 2.6.31-18-generic; java 1.6.0_0; Europe/en) http://yacy.net/bot.html
  Online Virus Scanner: http://tools.geek-tools.org
  DwnldBot (+http://dwnld.me)
  Mozilla/5.0 (compatible; OpenfosBot/2.1; http://www.openfos.com)
  taptubot *** please read http://www.taptu.com/corp/taptubot ***
  yacybot (freeworld/global; amd64 Linux 3.9.3-1-ARCH; java 1.7.0_21; America/en) http://yacy.net/bot.html
  Testomatobot/1.0 (Linux x86_64; +http://www.testomato.com/testomatobot) minicrawler/4.0.0~beta10
  rogerbot/1.0 (http://www.moz.com/dp/rogerbot, [email protected])
  Mozilla/5.0 (Windows NT 6.1; Win64; x64) KomodiaBot/1.0
  Mozilla/5.0 (compatible; spbot/3.0; +http://www.seoprofiler.com/bot )
  Mozilla/5.0 (compatible; DomainTunoCrawler/0.1; +https://www.domaintuno.com/robot)
  Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Googlebot/2.1; +http://www.google.com/bot.html) Safari/537.36
  yacybot (amd64 Linux 2.6.32-gentoo; java 1.6.0_17; Europe/en) http://yacy.net/bot.html
  Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko; Google Page Speed Insights) Chrome/27.0.1453 Mobile Safari/537.36
  Mozilla/5.0 (compatible; GimmeUSAbot/1.0; +http://gimmeusa-update.com/crawler)
  Cloud mapping experiment. Contact [email protected]
  emefgebot/beta (+http://emefge.de/bot.html)
  DoCoMo/2.0 D903i(c100;TB;W28H20) (compatible; Hatena-Mobile-Gateway/1.2; +http://mgw.hatena.ne.jp/help)
  ichiro/3.0 (http://help.goo.ne.jp/door/crawler.html)
  Mozilla/5.0 (compatible; SecurityResearch.bot; +http://besome1.info/securityresearchbot.html)
  WPScan v2.9 (http://wpscan.org)
  Ronzoobot/1.6 (http://www.ronzoo.com/about/)
  livedoor ScreenShot/0.10
  Mozilla/5.0 (compatible; MSIE 8.0; Windows NT 5.1) KomodiaBot/1.0
  Mozilla/5.0 (compatible; memoryBot/1.20.289 +http://internetmemory.org/en/)
  Mozilla/5.0 (compatible; DomainTunoCrawler/0.1; +http://www.domaintuno.com/robot)
  yacybot (freeworld/global; amd64 Linux 2.6.26-2-amd64; java 1.6.0_18; Europe/de) http://yacy.net/bot.html
  Mozilla/5.0 (compatible; GimmeUSAbot/1.0; +http://gimmeusa.com/crawler)
  Mozilla/5.0 (compatible; spbot/5.0; +http://OpenLinkProfiler.org/bot )
  Mozilla/5.0 (compatible; Webauskunft.org; http://www.webauskunft.org)
  CCResearchBot/1.0 commoncrawl.org/research//Nutch-1.7-SNAPSHOT
  Mozilla/4.0 (compatible; MSIE 5.5; Windows NT 4.0; obot)
  linkdex.com/v2.0
  Mozilla/5.0 (compatible; Crawlera/1.10.2; UID 73393)
  Mozilla/5.0 (compatible; SEODat/0.1 http://crawler.seodat.com)
  yacybot (freeworld/global; amd64 Linux 3.2.0-4-amd64; java 1.6.0_27; Europe/en) http://yacy.net/bot.html
  CorpusCrawler 2.0.12 (http://corpora.fi.muni.cz/crawler/)
  Mozilla/5.0 (compatible; spbot/4.3.0; +http://OpenLinkProfiler.org/bot )
  Mozilla/5.0 (compatible; aiHitBot/2.7; +http://www.aihit.com/)
  Mozilla/5.0 (compatible; SiteAnalyzerBot/4.0; +https://www.site-analyzer.com/)
  DataparkSearch/4.40.1 (+http://www.dataparksearch.org/)
  yacybot (freeworld-global; amd64 Linux 3.0.0-21-generic; java 1.6.0_26; America/en) http://yacy.net/bot.html
  findlinks/1.1.6-beta1 (+http://wortschatz.uni-leipzig.de/findlinks/; YaCy 0.1; yacy.net)
  CopperEgg/RevealUptime/SydneyAU(aws)
  OpenAcoon v4.1.0 (www.openacoon.de)
  semanticbot ([email protected])
  Mozilla/5.0 (compatible; Crawlera/1.10.2; UID 75425)
  Mozilla/5.0 (compatible; Crawlera/1.10.2; UID 11912)
  Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp)
  Mozilla/5.0 (Windows NT 5.1; U; Win64; fr; rv:1.8.1) VoilaBot BETA 1.2 ([email protected])
  yacybot (-global; amd64 Linux 3.13.0-61-generic; java 1.7.0_79; Europe/en) http://yacy.net/bot.html
  CorpusCrawler 2.0.25 (http://corpora.fi.muni.cz/crawler/);Project:CzCorpus
  ichiro/3.0 (http://search.goo.ne.jp/option/use/sub4/sub4-1/)
  R6_FeedFetcher_(www.radian6.com/crawler)
  sosoimagespider+(+http://help.soso.com/soso-image-spider.htm)
  Mozilla/5.0 (X11; Linux x86_64; rv:39.0; GTmetrix http://gtmetrix.com/) Gecko/20100101 Firefox/39.0
  yacybot (freeworld/global; amd64 Linux 3.2.0-24-generic; java 1.6.0_24; Europe/en) http://yacy.net/bot.html
  Mozilla/4.0 (compatible; Vagabondo/4.0; webcrawler at wise-guys dot nl; http://webagent.wise-guys.nl/)
  CommaFeed/2.4.0-SNAPSHOT (https://github.com/Athou/commafeed)
  Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; Google Web Preview Analytics) Chrome/27.0.1453 Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
  yacybot (/global; x86 Windows 7 6.1; java 1.8.0_31; Europe/de) http://yacy.net/bot.html
  Mozilla/5.0 (compatible; woriobot support [at] zite [dot] com +http://zite.com)
  Mozilla/5.0 (compatible; MetaJobBot; http://www.metajob.at/crawler)
  bl.uk_lddc_bot/3.3.0-SNAPSHOT-20140519-0844 (+http://www.bl.uk/aboutus/legaldeposit/websites/websites/faqswebmaster/index.html)
  FFbot/1.0; +http://feedfury.com
  Mozzila/5.0 (compatible; YamanaLab-Robot/1.0; http://www.yama.info.waseda.ac.jp/~crawler/info.html)
  LinqiaRSSBot/1.0 ([email protected])
  FreeWebMonitoring SiteChecker/0.2 (+http://www.freewebmonitoring.com/bot.html)
  Mozilla/5.0 (compatible; Web-Monitoring/1.0; +http://monoid.nic.ru/)
  Mozilla/5.0 (compatible; heritrix/1.12.1b +http://netarkivet.dk/website/info.html)
  Mozilla/5.0 (compatible; SpiderLing (a SPIDER for LINGustic research); http://nlp.fi.muni.cz/projects/biwec/)
  UrlTrends Ranking Updater/2.0 ( http://www.urltrends.com/ )
  yacybot (/global; amd64 Linux 3.16.1-gentoo; java 1.7.0_55; Europe/en) http://yacy.net/bot.html
  Mozilla/5.0 (compatible; gimme60bot/1.0 ; +http://gimme60.com)
  CatchBot/5.0; +http://www.catchbot.com
  elefent/Elefent 1.1 (Friendly web walker.; http://elefent.eu/; [email protected])
  findlinks/2.0.1 (+http://wortschatz.uni-leipzig.de/findlinks/)
  Googlebot-Video/1.0
  Mozilla/5.0 (compatible; XmarksFetch/1.0; +http://www.xmarks.com/about/crawler; [email protected])
  XRL/3.00 (Linux; i686; en-us) (+http://metamark.net/about)
  Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/50.0.2661.87 Safari/537.36 PTST/276
  CorpusCrawler 2.0.8 (http://corpora.fi.muni.cz/crawler/)
  Mozilla/5.0 (compatible; archive.org_bot +http://www.archive.org/details/archive.org_bot)
  Mozilla/5.0 (compatible; OpenindexDeepSpider/Nutch-1.5-dev; +http://openindex.io/spider.html; systemsATopenindexDOTio)
  Ronzoobot/1.3 (http://www.ronzoo.com/about.php)
  MetaInspector/2.2.1 (+https://github.com/jaimeiniesta/metainspector)
  Mozilla/5.0 (compatible; WASALive-Bot ; http://blog.wasalive.com/wasalive-bots/)
  Mozilla/5.0 (compatible; YioopBot; +http://www.yioop.com/bot.php)
  Mozilla/5.0 (compatible; Mozilla/5.0; +http://wiki.github.com/bixo/bixo/bixocrawler; [email protected])
  Mozilla/5.0 (compatible; Kraken/0.1; http://linkfluence.net/; [email protected])
  Mozilla/5.0 (compatible; Qseero; +http://www.q0.com)
  yacybot (-global; amd64 Linux 3.10.0-229.4.2.el7.x86_64; java 1.8.0_45; Europe/en) http://yacy.net/bot.html
  Mozilla/5.0 (compatible; ProCogBot/1.0; +http://www.procog.com/spider.html)
  Mozilla/5.0 (compatible; MJ12bot/v1.4.3; http://www.majestic12.co.uk/bot.php?+)
  Mozilla/5.0 (Anturis Agent)
  Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/534.34 (KHTML, like Gecko) Chromium/31.0.1650.63 Site-Shot/2.1 (http://www.site-shot.com/) Safari/534.34
  webnumbrFetcher/1.0 (http://webnumbr.com/)
  Mozilla/5.0 (compatible; meanpathbot/1.0; +http://www.meanpath.com/meanpathbot.html)
  NIF/1.1 (http://www.newsisfree.com/robot.php users:)
  Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/51.0.2704.103 Safari/537.36 PTST/284
  DoCoMo/2.0 P901i(c100;TB;W24H11) (compatible; ichiro/mobile goo; +http://help.goo.ne.jp/door/crawler.html)
  Iframely/0.8.8 (+https://iframely.com/;)
  yacybot (ppc Mac OS X 10.5.2; java 1.5.0_13; Europe/de) http://yacy.net/bot.html
  Mozilla/5.0 (compatible; UptimeRobot/2.0; http://www.uptimerobot.com/)
  yacybot (/global; amd64 Linux 3.13.0-85-generic; java 1.7.0_95; Europe/en) http://yacy.net/bot.html
  Mozilla/5.0 (compatible; FeedBooster; +http://feeds.qsensei.com)
  yacybot (freeworld/global; amd64 Linux 2.6.32-5-amd64; java 1.6.0_18; Europe/de) http://yacy.net/bot.html
  Mozilla/5.0 (compatible; SputnikFaviconBot/1.2; +http://corp.sputnik.ru/webmaster)
  Mozilla/5.0 (compatible; heritrix/3.1.1; UniLeipzigASV +http://corpora.informatik.uni-leipzig.de/crawler_faq.html)
  Mozilla/5.0 (compatible; OrangeBot/2.0; [email protected])
  Yeti-FeedItemCrawler/1.0 (NHN Corp.; http://help.naver.com/robots/)
  Mozilla/5.0 (compatible; SeznamBot/3.2-test1-1; +http://napoveda.seznam.cz/en/seznambot-intro/)
  Mozilla/5.0 (compatible; heritrix/3.2.0 +http://www.crim.ca)
  yacybot (/global; amd64 Linux 4.0.5-gentoo; java 1.8.0_51; Europe/en) http://yacy.net/bot.html
  pr-cy.ru Screenshot Bot
  GentleSource Short URL Checker ( http://www.gentlesource.com/ )
  crazywebcrawler 0.9.0, http://www.crazywebcrawler.com
  a.pr-cy.ru
  Mozilla/5.0 (iPhone; CPU iPhone OS 7_0 like Mac OS X) AppleWebKit/537.51.1 (KHTML, like Gecko) Version/7.0 Mobile/11A465 Safari/9537.53 (compatible; bingbot/2.0; http://www.bing.com/bingbot.htm)
  findlinks/1.1.6-beta1 (+http://wortschatz.uni-leipzig.de/findlinks/)
  ichiro/5.0 (http://help.goo.ne.jp/door/crawler.html)
  yacybot (/global; amd64 Windows 7 6.1; java 1.8.0_25; Europe/en) http://yacy.net/bot.html
  Yeti/1.1 (Naver Corp.; http://help.naver.com/robots/)
  Technoratibot/8.0
  Setooz/Nutch-1.0 (http://www.setooz.com)
  Mozilla/5.0 (Windows NT 5.2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/49.0.2623.112 Safari/537.36 PTST/201
  semanticbot
  Mozilla/5.0 (compatible; JikeSpider; +http://shoulu.jike.com/spider.html)
  Mozilla/5.0 (compatible; MetamojiCrawler/1.0; +http://www.metamoji.com/jp/crawler.html
  Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1.6) Gecko/20070725 Firefox/2.0.0.6 - James BOT - WebCrawler http://cognitiveseo.com/bot.html
  Mozilla/5.0 (Windows NT 6.1; compatible; BDCbot/1.0; +http://ecommerce.bigdatacorp.com.br/faq.aspx) ppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2272.118 Safari/537.36
  Mozilla/5.0 (compatible; CloudServerMarketSpider/1.0; +http://www.cloudservermarket.com/spider.html)
  Mozilla/5.0 (Windows NT 6.1; WOW64; rv:38.0; Neustar WPM) Gecko/20100101 Firefox/38.0
  Mozilla/5.0 (compatible;+ParchBot/1.0;++http://www.parchmenthill.com/search.htm)
  Sogou-Test-Spider/4.0 (compatible; MSIE 5.5; Windows 98)
  MetaHeadersBot (+http://www.metaheaders.com/bot.html)
  Mozilla/5.0 (Windows; U; Windows NT 6.0; en-GB; rv:1.0; trendictionbot0.4.2; trendiction media ssppiiddeerr; http://www.trendiction.com/bot/; please let us know of any problems; ssppiiddeerr at trendiction.com) Gecko/20071127 Firefox/2.0.0.11
  voyager/2.0 (http://www.kosmix.com/crawler.html)
  Mozilla/5.0 (compatible; YodaoBot/1.0; http://www.yodao.com/help/webmaster/spider/; )
  hivaBot/hivaBot-1.0 (Iranian Search Engine bot; http://yooz.ir; [email protected])
  yacybot (freeworld/global; amd64 Linux 3.2.0-4-amd64; java 1.7.0_55; Europe/de) http://yacy.net/bot.html
  AppEngine-Google; (+http://code.google.com/appengine; appid: s~closure-compiler-hrd)
  larbin_2.5.0 ([email protected])
  Scrapy/0.22.2 (+http://scrapy.org)
  Ronzoobot/1.2 (http://www.ronzoo.com/about.php)
  gooblog/2.0 (http://help.goo.ne.jp/contact/)
  Pingdom.com_bot_version_1.4_(http://www.pingdom.com/)
  Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/535.21 (KHTML, like Gecko) Chrome/19.0.1042.0 Safari/535.21 PagePeeker/2.1; +http://pagepeeker.com/robots/
  WeSEE:Ads/PageBot (http://www.wesee.com/bot/)
  yacybot (amd64 Linux 2.6.18-164.el5; java 1.6.0; Europe/de) http://yacy.net/bot.html
  SemanticScholarBot/1.0 (+http://s2.allenai.org/bot.html)
  Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/535. Safari/535.22+; (+http://immediatenet.com/thumbnail_api.html; free thumbnails)
  Mozilla/5.0 (compatible; Crawlera/1.10.2; UID 85747)
  SBIder/Nutch-1.0-dev (http://www.sitesell.com/sbider.html)
  Scrapy/1.0.4 (+http://scrapy.org)
  SanszBot/1.7(http://www.sansz.org/sanszbot, [email protected]) ([email protected])
  Mozilla/5.0 (compatible; KHTML, like Gecko) Chrome/43.0.2357.81 Safari/537.36 [email protected]
  Mozilla/5.0 (iPhone; U; CPU iPhone OS 4_3 like Mac OS X; en-GB) adbeat.com/policy AppleWebKit/533.17.9 (KHTML, like Gecko) Version/5.0.2 Mobile/8F190 Safari/6533.18.5
  Curious George - www.analyticsseo.com/crawler
  Grammarly/1.0 (http://www.grammarly.com)
  OpenCalaisSemanticProxy
  yacybot (freeworld/global; amd64 Linux 2.6.32-49-server; java 1.6.0_27; Europe/en) http://yacy.net/bot.html
  netEstate RSS crawler (+http://www.rss-directory.info/)
  ipv6-test.com validator
  Mozilla/5.0 (compatible; Qualidator.com Bot 1.0;)
  ichiro/2.0 ([email protected])
  yacybot (freeworld/global; amd64 Linux 2.6.38-14-generic; java 1.6.0_22; Europe/en) http://yacy.net/bot.html
  cg-eye live
  Mozilla/5.0 (compatible; OpenindexSpider; +http://www.openindex.io/en/webmasters/spider.html)
  Mozilla/5.0 (Windows; U; MSIE 9.0; Windows NT 9.0; en-US) AppEngine-Google; (+http://code.google.com/appengine; appid: s~virustotalcloud)
  Mozilla/5.0 (compatible; heritrix/1.14.3.r6601 +http://www.buddybuzz.net/yptrino)
  MetaGeneratorCrawler/1.3.4 (www.metagenerator.info)
  Mozilla/5.0 (Linux; Android 5.0; LG-D855 Build/LRX21R.A1422018487) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/40.0.2214.109 Mobile Safari/537.36 DareBoost
  Mozilla/5.0 (compatible; Nigma.ru/3.0; [email protected])
  Mozilla/5.0 (compatible; Wappalyzer; +https://github.com/AliasIO/Wappalyzer)
  yacybot (/global; x86 Windows XP 5.1; java 1.7.0_71; Europe/de) http://yacy.net/bot.html
  Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/50.0.2661.102 Safari/537.36 PTST/277
  OpenWebSpider v0.1.4 (http://www.openwebspider.org/)
  HubSpot Connect 1.0 (http://dev.hubspot.com/)
  Page Valet/4.1pre5
  msnbot/2.0b (+http://search.msn.com/msnbot.htm)
  amibot - http://www.amidalla.de - [email protected] libwww-perl/5.831
  SETOOZBOT/5.0 ( compatible; SETOOZBOT/0.30 ; http://www.setooz.com/bot.html )
  Mozilla/5.0 (compatible; YoudaoBot/1.0; http://www.youdao.com/help/webmaster/spider/; )
  Mozilla/5.0 (compatible; MSIE or Firefox mutant; not on Windows server;) Daumoa 4.0
  Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.0) LinkCheck by Siteimprove.com
  EdisterBot (http://www.edister.com/bot.html)
  Mozilla/5.0 (iPhone; CPU iPhone OS 6_0 like Mac OS X) AppleWebKit/536.26 (KHTML, like Gecko) Version/6.0 Mobile/10A5376e Safari/8536.25 (compatible; SMTBot/1.0; +http://www.similartech.com/smtbot)
  Tarantula/BETA-0.5 (Tarantula de la Enlle, el mejor buscador en la red; http://www.enlle.com; [email protected])
  Mozilla/5.0 (Windows; U; Windows NT 5.1; fr; rv:1.8.1) VoilaBot BETA 1.2 ([email protected])
  yacybot (freeworld/global; amd64 Linux 2.6.38-8-generic; java 1.6.0_22; Europe/en) http://yacy.net/bot.html
  Mozilla/5.0 (compatible; SemrushBot/0.95; +http://www.semrush.com/bot.html)
  CatchBot/3.0; +http://www.catchbot.com
  Mozilla/5.0 (compatible; uMBot-LN/1.0; mailto: [email protected])
  Mozilla/5.0 (compatible; RukiCrawler/1.0; +http://ruki.rezko.net)
  adbeat_bot
  Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/27.0.1453.116 Safari/537.36 HubSpot Webcrawler
  Mozilla/5.0 (compatible; YandexCatalog/3.0; +http://yandex.com/bots)
  NutchCVS/0.7.1 (Nutch; http://lucene.apache.org/nutch/bot.html; [email protected])
  Statastico/4.0
  facebookexternalhit/1.1 (+http://www.facebook.com/externalhit_uatext.php)
  Mozilla/5.0 (X11; U; Linux x86_64; en-US) AppleWebKit/534.16 (KHTML, like Gecko, Google Keyword Suggestion) Chrome/10.0.648.127 Safari/534.16
  Mozilla/5.0 (iPhone; CPU iPhone OS 7_0 like Mac OS X) AppleWebKit/537.51.1 (KHTML, like Gecko) Version/7.0 Mobile/11A465 Safari/9537.53 BingPreview/1.0b
  EuripBot/2.0 (+http://www.eurip.com) GetRobots
  Mozilla/5.0 (compatible; MJ12bot/v1.2.1; http://www.majestic12.co.uk/bot.php?+)
  Braintree-Webhooks
  Nutch Checksem/Nutch-1.10
  Mozilla/5.0 (compatible; HyperCrawl/0.2; +http://www.seograph.net/bot.html)
  quickobot/quickobot-1 (Quicko Labs; http://quicko.co; robot at quicko dot co)
  Whibse
  Yahoo! Site Explorer Feed Validator http://help.yahoo.com/l/us/yahoo/search/siteexplorer/manage/
  Mozilla/5.0 (compatible; AhrefsBot/5.0; +http://ahrefs.com/robot/)
  Siteshooter B0t 2.1
  GingerCrawler/1.0 (Language Assistant for Dyslexics; www.gingersoftware.com/crawler_agent.htm; support at ginger software dot com)
  Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko; Google Page Speed Insights) Chrome/27.0.1453 Safari/537.36
  eBot / v.0.1e (http://alfa.elchron.cz)
  Mozilla/5.0 (compatible; memoryBot/1.20.247 +http://internetmemory.org/en/)
  yacybot (x86 Windows 2003 5.2; java 1.6.0_20; America/de) http://yacy.net/bot.html
  YandexSomething/1.0
  Scoop
  Mozilla/5.0 (compatible; MegaIndex.ru/2.0; +http://megaindex.com/crawler)
  Mozilla/5.0 (compatible; SputnikBot/2.3; +http://corp.sputnik.ru/webmaster)
  TurnitinBot/2.1 (http://www.turnitin.com/robot/crawlerinfo.html)
  Mozilla/5.0 (compatible; Ask Jeeves/Teoma; +http://about.ask.com/en/docs/about/webmasters.shtml)
  Mozilla/5.0 (compatible; CrawlForMe/3.0.2; [email protected])
  Seologies/0.1 Bot
  www.deadlinkchecker.com XMLHTTP/1.0 Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/46.0.2490.86 Safari/537.36
  dubaiindex (addressendeutschland.de)
  yacybot (/global; x86 Windows 7 6.1; java 1.8.0_40; America/es) http://yacy.net/bot.html
  Mozilla/5.0+(compatible;+PiplBot;++http://www.pipl.com/bot/)
  LinkStats Bot
  Mozilla/5.0 (compatible; adbeat-publisher-description-fetcher; [email protected]; [email protected])
  CISPA Vulnerability Notification (https://notify.mmci.uni-saarland.de)
  librabot/1.0 (+http://search.msn.com/msnbot.htm)
  SeoCheckBot (Seomira Seo Check, http://www.kfsw.de/bot.html)
  yacybot (x86 Windows 2003 5.2; java 1.6.0_16; Europe/de) http://yacy.net/bot.html
  WatchMouse/8.4.0.3 (http://watchmouse.com/ ; itpda01.watchmouse.net)
  yacybot (/global; amd64 Linux 3.13.0-83-generic; java 1.7.0_95; Europe/en) http://yacy.net/bot.html
  BrainbruBot/1.0 (+http://www.brainbru.com/)
  Docoloc
  Mozilla/5.0 (Unknown; Linux x86_64) AppleWebKit/538.1 (KHTML, like Gecko) PhantomJS/2.1.1 Safari/538.1 Prerender (+https://github.com/prerender/prerender)
  dlvr.it/1.0 (+http://dlvr.it/)
  Mozilla/5.0 (compatible; parsijoo-bot; +http://www.parsijoo.ir/; [email protected])
  DataparkSearch/4.35-02122005 ( http://www.dataparksearch.org/)
  Pixray-Seeker/2.0 (Pixray-Seeker; +http://www.pixray.com/pixraybot; [email protected])
  Abrave v5.5 (http://robot.abrave.co.uk)
  Mozilla/5.0 (X11; U; Linux i686 (x86_64); en-US; rv:1.8.1.20) Gecko/20090429 HeartRails_Capture/0.6 (+http://capture.heartrails.com/) BonEcho/2.0.0.20
  yacybot (/global; amd64 Linux 3.19.0-26-generic; java 1.7.0_95; Europe/en) http://yacy.net/bot.html
  MLBot (www.metadatalabs.com/mlbot)
  yacybot (freeworld/global; amd64 Linux 2.6.32-34-server; java 1.6.0_26; Europe/de) http://yacy.net/bot.html
  Mozilla/5.0 (compatible; LinkpadBot/1.08; +http://www.linkpad.ru)
  Mozilla/5.0 (compatible; SearchmetricsBot; http://www.searchmetrics.com/en/searchmetrics-bot/)
  msnbot-NewsBlogs/2.0b (+http://search.msn.com/msnbot.htm)
  ZnajdzFoto/Image 2.0
  Mozilla/5.0 (compatible; Magibot/4.0.0; +http://magi.peak-labs.com/robots.txt)
  Mozilla/5.0 (compatible; spbot/4.0.8; +http://OpenLinkProfiler.org/bot )
  Eurobot/1.2 (http://eurobot.ayell.eu)
  Mozilla/5.0 (compatible; Embedly/0.2; snap; +http://support.embed.ly/)
  Yaanb/1.5.001 (compatible; Win64;+http://www.yaanb.com/company/bot.hmtl)
  yacybot (i386 Linux 2.6.31-16-generic; java 1.6.0_15; Europe/en) http://yacy.net/bot.html
  Mozilla/5.0 (compatible; YioopBot; +http://173.13.143.74/bot.php)
  CopperEgg/RevealUptime/Singapore(aws)
  Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/534.24 (KHTML, like Gecko; Google Web Preview) Chrome/11.0.696 Safari/534.24
  yacybot (freeworld/global; amd64 Windows 8 6.2; java 1.7.0_25; Europe/de) http://yacy.net/bot.html
  Server Density Service Monitoring v2
  Testomatobot/1.0 (Linux x86_64; +http://www.testomato.com/testomatobot) minicrawler/3.0~beta
  TinEye/1.0; +http://www.tineye.com/
  Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.9.0.19; aggregator:Spinn3r (Spinn3r 3.1); http://spinn3r.com/robot) Gecko/2010040121 Firefox/3.0.19
  GigablastOpenSource/1.0
  Googlebot (gocrawl v0.4)
  CSmartCrawler (email: abuse[at]adsadv[dot]com)
  Testomatobot/1.0 (Linux x86_64; +http://www.testomato.com/testomatobot) minicrawler/4.0.0~beta9
  KOCMOHABT (+http://kozmonavt.tk/) AppEngine-Google; (+http://code.google.com/appengine; appid: s~web-explorer)
  Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.2.18) Gecko/20110622 HeartRails_Capture/1.0.5 (+http://capture.heartrails.com/) Namoroka/3.6.18
  PingSpot/1.0 (http://www.pingspot.org/)
  AppEngine-Google; (+http://code.google.com/appengine; appid: s~buzzstream-tools-hrd)
  AddThis.com (http://support.addthis.com/)
  Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko; Google Web Preview) Chrome/27.0.1453 Safari/537.36
  ExactSeekCrawler/1.0
  Yeti/1.0 (NHN Corp.; http://help.naver.com/robots/)
  Mozilla/5.0 (compatible; jpg-newsbot/2.0; +http://vipnytt.no/bot.html)
  g2reader-bot/1.0 (+http://www.g2reader.com/)
  MetaGeneratorCrawler/1.3.12 (www.metagenerator.info)
  QuerySeekerSpider ( http://queryseeker.com/bot.html )
  KDDI-CA31 UP.Browser/6.2.0.7.3.129 (GUI) MMP/2.0 (compatible; ichiro/mobile goo; +http://help.goo.ne.jp/help/article/1142/)
  Mozilla/5.0 (compatible; GrapeshotCrawler/2.0; +http://www.grapeshot.co.uk/crawler.php)
  Mozilla/5.0 (compatible; MSIE 10.0; Windows NT 6.2; Trident/6.0; WOW64; Trident/6.0; BingPreview/1.0b)
  Mozilla/5.0 (compatible; firmilybot/0.3; +http://www.firmily.com/bot.php
  Mozilla/5.0 (X11; U; Linux Core i7-4980HQ; de; rv:32.0; compatible; JobboerseBot; http://www.jobboerse.com/bot.htm) Gecko/20100101 Firefox/38.0
  Mozilla/5.0 (compatible; metager2-verification-bot; +http://metager2.de/technology.php)
  crawler4j (http://code.google.com/p/crawler4j/)
  CirrusExplorer/1.1 (http://www.cireu.com/explorer.php)
  Mozilla/5.0 (compatible; RankvalBot/1.0; +https://rankval.com)
  Mozilla/5.0 (compatible; heritrix/1.14.3 +http://www.accelobot.com)
  Mozilla/5.0 (compatible; BLEXBotTest/1.0; +http://webmeup.com/crawler.html)
  nWormFeedFinder (http://www.nworm.com)
  netEstate NE Crawler (+http://www.sengine.info/)
  Mozila/5.0 (compatible; StudioFACA Search; +http://www.studiofaca.com/)
  Mozilla/5.0 (compatible; Crawlera/1.10.2; UID 92751)
  wikiwix-bot-3.0
  SEOCentro Page Keyword Analyzer v1.2
  yacybot (amd64 Linux 2.6.18-164.15.1.el5xen; java 1.6.0_0; Europe/en) http://yacy.net/bot.html
  Mozilla/5.0 (compatible; bingbot/2.0; http://www.bing.com/bingbot.htm)
  Automattic Analytics Crawler/0.1; http://wordpress.com/crawler/
  SemrushBot/0.92
  http://www.uni-koblenz.de/~flocke/robot-info.txt
  Mozilla/5.0 (compatible; idmarch Automatic.beta/1.3; +http://www.idmarch.org/bot.html)
  Mozilla/5.0 (X11; Linux i686 on x86_64; rv:25.0; GTmetrix http://gtmetrix.com/) Gecko/20100101 Firefox/25.0
  msnbot/2.0b (+http://search.msn.com/msnbot.htm)._
  Stroke.cz (http://stroke.cz)
  Mozilla (+http://web.nli.org.il/sites/NLI/English/collections/israel-collection/internet_archive/Pages/default.aspx)
  audisto.com full crawler 4.33.470 (refer to in robots.txt as audisto, see https://audisto.com/bot)
  http://arachnode.net 2.5
  SurveyBot/2.3 (Whois Source)
  ant.com/Nutch-1.7 (http://ant.com)
  CorpusCrawler 2.0.13 (http://corpora.fi.muni.cz/crawler/)
  Seo Servis - Analyza zdrojoveho kodu
  SEOENGBot/1.2 (+http://www.seoengine.com/seoengbot.htm)
  Mozilla/5.0 (compatible; Genieo/1.0 http://www.genieo.com/webfilter.html)
  Mozilla/5.0 (compatible; evc-batch/2.0.20160708184126)
  HubSpot Marketing Grader
  Mozilla/5.0 (compatible; MSIE 10.0; Windows NT 6.1; Trident/6.0) LinkCheck by Siteimprove.com
  nrsbot/6.0(loopip.com/robot.html)
  Mozilla/5.0 (compatible; BLEXBot/1.0; +http://webmeup.com/crawler.html)
  yacybot (freeworld-global; amd64 Windows 7 6.1; java 1.7.0_02-ea; America/en) http://yacy.net/bot.html
  Mozilla/5.0 (compatible; Blekkobot; ScoutJet; +http://blekko.com/about/blekkobot)
  Mozilla/5.0 (compatible; Topicbot/12.0rc-2; +http://topicbot.awardspace.us/)
  Mozilla/5.0 (compatible; Gimme60bot/1.0; +http://gimme60.com)
  yacybot (i386 Linux 2.6.28-11-generic; java 1.6.0_13; Europe/de) http://yacy.net/bot.html
  Mozilla/5.0 (compatible; DotBot/1.1; http://www.opensiteexplorer.org/dotbot, [email protected])
  Kemvibot/1.0 (http://kemvi.com, [email protected])
  CorpusCrawler 2.0.15 (http://corpora.fi.muni.cz/crawler/)
  dotbot
  Mozilla/5.0 (iPhone; CPU iPhone OS 7_0 like Mac OS X) AppleWebKit/537.51.1 (KHTML, like Gecko) Version/7.0 Mobile/11A465 Safari/9537.53 (compatible; adidxbot/2.0; +http://www.bing.com/bingbot.htm)
  page_viewer ([email protected])
  HootSuite Image proxy ([email protected])
  Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US) Speedy Spider for SpeedyAds (http://www.entireweb.com/about/search_tech/speedy_spider/)
  PINGOMETER_BOT_(HTTPS://PINGOMETER.COM)
  Mozilla/5.0 (compatible; Vagabondo/2.1; webcrawler at wise-guys dot nl; http://webagent.wise-guys.nl/)
  VSE/1.0 ([email protected])
  Scrapy/0.24.0 (+http://scrapy.org)
  Speedy Spider (http://www.entireweb.com)
  Mozilla/5.0 (X11; Linux x86_64; GTmetrix https://gtmetrix.com/) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/51.0.2704.106 Safari/537.36
  WatchMouse/18990 (http://watchmouse.com/ ; hk)
  Mozilla/5.0 (compatible; yoozBot-2.2; http://yooz.ir; [email protected])
  Mozilla/5.0 (compatible; Crawlera/1.10.2; UID 87214)
  Mozilla/5.0 (compatible; discobot/2.0; +http://discoveryengine.com/discobot.html)
  yacybot (freeworld/global; amd64 Windows 7 6.1; java 1.8.0_11; Europe/de) http://yacy.net/bot.html
  Testomatobot/1.0 (Linux x86_64; +http://www.testomato.com/testomatobot) minicrawler/4.0.0~beta6
  yacybot (freeworld/global; amd64 Linux 3.0.0-15-server; java 1.6.0_23; Europe/en) http://yacy.net/bot.html
  Mozilla/5.0 (Macintosh; Intel Mac OS X 10.9; rv:28.0) Gecko/20100101 Firefox/28.0 (FlipboardProxy/1.1; +http://flipboard.com/browserproxy)
  YisouSpider
  Slack-ImgProxy 1.138 (+https://api.slack.com/robots)
  mozilla/5.0 (compatible; webmastercoffee/0.7; +http://webmastercoffee.com/about)
  Mozilla/5.0 (compatible; Uptimebot/0.2.22; +http://www.uptime.com/uptimebot)
  Mozilla/5.0 (compatible; AhrefsBot/1.0; +http://ahrefs.com/robot/)
  Mozilla/5.0 (compatible; Feedspotbot/1.0; +http://www.feedspot.com/fs/bot)
  sg-Orbiter/1.0 (+http://searchgears.de/uber-uns/crawling-faq.html)
  Site24x7 Tools
  NextGenSearchBot 1 (for information visit http://www.zoominfo.com/About/misc/NextGenSearchBot.aspx)
  Aboundex/0.3 (http://www.aboundex.com/crawler/)
  yacybot (freeworld/global; i386 Linux 2.6.37.6-0.5-desktop; java 1.6.0_20; Europe/en) http://yacy.net/bot.html
  Mozilla/5.0 (Windows NT 6.2; WOW64) AppleWebKit/537.4 (KHTML, like Gecko) Chrome/98 Safari/537.4 (StatusCake)
  Mozilla/5.0 (compatible; DomainSONOCrawler/0.1; +http://domainsono.com)
  Mozilla/5.0 (compatible; Online Domain Tools - Server Monitor/1.0; +http://server-monitoring.online-domain-tools.com)
  Mozilla/5.0 (compatible; Konqueror/3.5; Linux) KHTML/3.5.5 (like Gecko) (Exabot-Thumbnails)
  Mozilla/5.0 (compatible; spbot/4.0.4; +http://www.seoprofiler.com/bot )
  Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/27.0.1453 Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
  Factbot 1.09 (see http://www.factbites.com/webmasters.php)
  CopperEgg/RevealUptime/SaoPauloBR(aws)
  CRIM Crawler/Nutch-2.3 (Crawler du Centre de Recherche Informatique de Montr\xc3\xa9al (CRIM))
  Mozilla/5.0 (compatible; heritrix/1.14.3 +http://www.webarchiv.cz)
  Mozilla/5.0 (compatible; WormlyBot; +http://wormly.com)
  yacybot (i386 Linux 2.6.23; java 1.6.0_17; Europe/en) http://yacy.net/bot.html
  Mozilla/5.0 (compatible; Cliqzbot/0.1 +http://cliqz.com/company/cliqzbot)
  URLitor.com
  RavenCrawler
  Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.2.18) Gecko/20130331 HeartRails_Capture/1.0.5 (+http://capture.heartrails.com/) Namoroka/3.6.18
  Mozilla/5.0 (compatible; WebThumbnail/3.x; Website Thumbnail Generator; +http://webthumbnail.org)
  Kyoto-Crawler/2.0 (Mozilla-compatible; kyoto-crawler-contact(at)nlp(dot)kuee(dot)kyoto-u(dot)ac(dot)jp; http://nlp.ist.i.kyoto-u.ac.jp/)
  Mozilla/5.0 (compatible; SputnikImageBot/2.3; +http://corp.sputnik.ru/webmaster)
  WatchMouse/8.4.0.3 (http://watchmouse.com/ ; lvps91-250-96-109.dedicated.hosteurope.de)
  MetaGeneratorCrawler/1.3.3 (www.metagenerator.info)
  LongURL API
  Mozilla/5.0 ( compatible; SETOOZBOT/0.30 ; http://www.setooz.com/bot.html ; agentname at setooz dot_com )
  Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.6; en-US; rv:1.9.2) Gecko/20100115 Firefox/3.6 (FlipboardProxy/2.0; +http://flipboard.com/browserproxy)
  http://tools.geek-tools.org/link-counter/
  SCFCrawler/Nutch-1.8 (Image Crawler for StolenCameraFinder.com; http://www.stolencamerafinder.com/; [email protected])
  Mozilla/5.0 (compatible; Uptimebot/0.2.41; +http://www.uptime.com/uptimebot)
  Mozilla/5.0 (compatible; DomainAppender /1.0; +http://www.profound.net/domainappender)
  Mozilla/5.0 (compatible; hypestat/1.0; +http://www.hypestat.com/bot)
  yacybot (/global; amd64 Linux 3.13.0-042stab093.4; java 1.7.0_79; Europe/en) http://yacy.net/bot.html
  PagesInventory (robot +http://www.pagesinventory.com)
  Sosospider+(+http://help.soso.com/webspider.htm)
  Mozilla/5.0 (compatible; AntBot/1.0; +http://www.ant.com/)
  PagesInventory (robot http://www.pagesinvenotry.com)
  Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; http://www.changedetection.com/bot.html )
  Wotbox/2.01 (+http://www.wotbox.com/bot/)
  Mozilla/5.0 (compatible; bixolabs/1.0; +http://bixolabs.com/crawler/general; [email protected])
  Mozilla/5.0 (compatible; SpiderLing (a SPIDER for LINGustic research); +http://nlp.fi.muni.cz/projects/biwec/)
  CopperEgg/RevealUptime/DallasTXUSA
  yacybot (freeworld/global; amd64 Linux 2.6.32-custom; java 1.6.0_26; Europe/en) http://yacy.net/bot.html
  Bad-Neighborhood Link Analyzer (http://www.bad-neighborhood.com/)
  Mozilla/5.0 (compatible; oBot/2.3.1; +http://filterdb.iss.net/crawler/)
  Pixray-Seeker/2.0 (Pixray-Seeker; http://www.pixray.com/pixraybot; [email protected])
  Cityreview Robot (+http://www.cityreview.org/crawler/)
  yacybot (/global; amd64 Linux 3.16.0-4-amd64; java 1.7.0_95; Europe/en) http://yacy.net/bot.html
  Dlvr.it/1.0 (http://dlvr.it/)
  Zookabot/2.4;++http://zookabot.com
  AddThis.com robot [email protected]
  Mozilla/5.0 (compatible; discoverybot/2.0; +http://discoveryengine.com/discoverybot.html)
  TwengaBot
  Mozilla/5.0 (compatible; 4SeoHuntBot; +http://4seohunt.biz/about.html)
  Web-sniffer.me/1.0.0 (+http://web-sniffer.me/)
  KDDI-CA31 UP.Browser/6.2.0.7.3.129 (GUI) MMP/2.0 (compatible; ichiro/mobile goo; +http://search.goo.ne.jp/option/use/sub4/sub4-1/)
  gosquared-thumbnailer/1.0
  yacybot (/global; amd64 Windows 7 6.1; java 1.7.0_55; Asia/zh) http://yacy.net/bot.html
  BPImageWalker/2.0 (www.bdbrandprotect.com)
  Acoon v4.10.3 (www.acoon.de)
  Mozilla/4.0 (CMS Crawler: http://www.cmscrawler.com)
  WP Engine Install Performance API
  Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.2.24; ips-agent) Gecko/20111107 Ubuntu/10.04 (lucid) Firefox/3.6.24
  Mozilla/5.0 (Windows NT 6.1; WOW64; Trident/7.0; rv:11.0) like Gecko PTST/277
  CamontSpider/1.0 +http://epweb2.ph.bham.ac.uk/user/slater/camont/info.html
  SEOENGBot/1.2 (+http://learn.seoengine.com/seoengbot.htm)
  Y!J-BRJ/YATS crawler (http://listing.yahoo.co.jp/support/faq/int/other/other_001.html)
  CopperEgg/RevealUptime/N.Virginia(aws)
  Mozilla/5.0, Google-AdWords-Express
  Mozilla/5.0 (compatible;YodaoBot-Image/1.0;http://www.youdao.com/help/webmaster/spider/;)
  yacybot (/global; amd64 Linux 3.16.0-4-amd64; java 1.8.0_91; Europe/en) http://yacy.net/bot.html
  yacybot (freeworld/global; amd64 Windows 7 6.1; java 1.7.0_09; Europe/de) http://yacy.net/bot.html
  Acoon v4.10.1 (www.acoon.de)
  Mediatoolkitbot ([email protected])
  Mozilla/5.0 (compatible; Speedy Spider; http://www.entireweb.com/about/search_tech/speedy_spider/)
  Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.2.3) SPEng
  2Bone_LinkChkr/1.0 libwww-perl/6.03
  Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.6; en-US; rv:1.9.2) Gecko/20100115 Firefox/3.6 (FlipboardBrowserProxy/0.0.5; +http://flipboard.com/browserproxy)
  Mozilla/5.0 (compatible; SeznamBot/3.2-test1; +http://napoveda.seznam.cz/en/seznambot-intro/)
  Comodo SSL Checker
  Comodo Spider 1.2
  Who.is Bot
  Mozilla/5.0 (compatible; online-webceo-bot/1.0; +http://online.webceo.com)
  yacybot (freeworld/global; amd64 Windows Server 2008 R2 6.1; java 1.6.0_29; Europe/en) http://yacy.net/bot.html
  psbot/0.1 (+http://www.picsearch.com/bot.html)
  IlTrovatore-Setaccio/1.2 (http://www.iltrovatore.it/aiuto/faq.html)
  Mozilla/5.0 (compatible; Twingly Recon; twingly.com)
  Mozilla/5.0 Moreover/5.1 (+http://www.moreover.com; [email protected])
  Orgbybot/OrgbyBot v1.2 (Spidering the net for Orgby; http://www.orgby.com/ ; Orgby.com Search Engine)
  Browsershots
  AdsBot-Google (+http://www.google.com/adsbot.html)
  Qseero v1.0.0
  Mozilla/5.0 (compatible; ltbot/0.3.6 +http://www.kdsl.tu-darmstadt.de/de/kdsl/research-program/crawling-and-semantic-structuring/)
  yacybot (x86 Windows XP 5.1; java 1.6.0_18; Europe/de) http://yacy.net/bot.html
  visionutils/0.2
  Mozilla/5.0 (compatible; Googlebot/2.1; https://www.deepcrawl.com/bot)
  Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.4 (KHTML, like Gecko; Google Web Preview) Chrome/22.0.1229 Safari/537.4
  Mozilla/5.0 (compatible; Crawlera/1.10.2; UID 8942)
  Mozilla/4.0 (compatible; LinkMarket-Bot v1.2)
  Buzzbot/1.0 (Buzzbot; http://www.buzzstream.com; [email protected])
  WordPress.com; https://public-api.wordpress.com
  Mozilla/5.0 ( compatible; SETOOZBOT/0.30 ; http://www.setooz.com/bot.html )
  CopperEgg/RevealUptime/FremontCAUSA
  Mozilla/5.0 (Windows; U; Windows NT 6.0; en-GB; rv:1.0; trendictionbot0.4.5; trendiction search; http://www.trendiction.de/bot; please let us know of any problems; web at trendiction.com) Gecko/20071127 Firefox/3.0.0.11
  yacybot (/global; amd64 Linux 2.6.32-042stab094.8; java 1.7.0_79; America/en) http://yacy.net/bot.html
  Mozilla/5.0 (compatible; Windows NT 6.1?; ZumBot/1.0; http://help.zum.com/inquiry)
  Mozilla/5.0 (Windows NT 6.1; WOW64; Trident/7.0; rv:11.0; topster.de Linkchecker 5.0) like Gecko (90.177.52.133)
  Mozilla/5.0+(compatible; Monitority/1.0; http://www.monitority.com/)
  yacybot (freeworld/global; x86 Windows 7 6.1; java 1.8.0_25; Europe/de) http://yacy.net/bot.html
  Mozilla/5.0 (compatible; abby/1.0; +http://www.ellerdale.com/crawler.html)
  Mozilla/5.0 (compatible; SemrushBot/1.1~bl; +http://www.semrush.com/bot.html)
  YowedoBot/Yowedo 1.0 (Search Engine crawler for yowedo.com; http://yowedo.com/en/partners.html; [email protected])
  Mozilla/5.0 (compatible; oBot/2.3.1; http://filterdb.iss.net/crawler/)
  Nokia6820/2.0 (4.83) Profile/MIDP-1.0 Configuration/CLDC-1.0 (compatible; Googlebot-Mobile/2.1; +http://www.google.com/bot.html)
  Mozilla/5.0 (compatible; IstellaBot/1.23.15 +http://www.tiscali.it/)
  GetIntent Crawler (http://getintent.com/bot.html)
  Mozilla/5.0 (compatible; Uptimebot/0.1.89; +http://www.uptime.com/uptimebot)
  Cliqzbot
  AboutUsBot
  yacybot (-global; amd64 Linux 3.2.0-4-amd64; java 1.7.0_67; Europe/en) http://yacy.net/bot.html
  Sufog/Nutch-2.2.1 (www.sufog.com; www.sufog.com)
  Mozilla/5.0 (compatible; imbot/0.1 +http://internetmemory.org/en/)
  KOCMOHABT (+http://kozmonavt.tk/) Mozilla/5.0 (compatible; Web Explorer 1) AppEngine-Google; (+http://code.google.com/appengine; appid: s~web-explorer)
  yacybot (/global; amd64 Linux 3.16.0-53-generic; java 1.7.0_85; Europe/en) http://yacy.net/bot.html
  L.webis/0.53 (http://webalgo.iit.cnr.it/index.php?pg=lwebis)
  Mozilla/5.0 (compatible; memoryBot/1.21.23 +http://internetmemory.org/en/)
  ExB Language Crawler 2.1.5 (+http://www.exb.de/crawler)
  Peeplo Screenshot Bot/0.20 ( abuse at peeplo dot_com )
  Mozilla/5.0 (compatible; Qwantify/2.1w; +https://www.qwant.com/)/*
  Experibot_v1 [bit.ly/2bLMJdw]
  WikioFeedBot 1.0 (http://www.wikio.com)
  SalesIntelligent/v1.1 (https://www.salesintelligent.com)
  Mozilla/5.0 (compatible; ExaleadCloudView/5;)
  Mozilla/5.0 (compatible; rogerBot/1.0; UrlCrawler; http://www.seomoz.org/dp/rogerbot)
  Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/44.0.2403.157 Safari/537.36 TinEye/1.0 (via http://www.tineye.com/)
  Mozilla/5.0 (compatible; Search17Bot/1.1; http://www.search17.com/bot.php)
  FollowSite Bot ( http://www.followsite.com/bot.html )
  Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/27.0.1453.116 Safari/537.36 HubSpot Webcrawler - [email protected]
  Mozilla/5.0 (compatible; BlinkaCrawler/1.0; +http://www.blinka.jp/crawler/)
  Mozilla/5.0 (compatible; Qwantify/2.2w; +https://www.qwant.com/)/*
  findlinks/1.1.6-beta2 (+http://wortschatz.uni-leipzig.de/findlinks/)
  yacybot (/global; amd64 Linux 4.4.0-28-generic; java 1.8.0_91; Europe/pl) http://yacy.net/bot.html
  yacybot (/global; amd64 Windows Server 2008 R2 6.1; java 1.8.0_40; Europe/de) http://yacy.net/bot.html
  Mozilla/5.0 (compatible; 200PleaseBot/1.0; +http://www.200please.com/bot)
  WatchMouse/8.4.0.3 (http://watchmouse.com/ ; chzrh01.watchmouse.net)
  Iframely/0.8.8 (+http://iframely.com/;)
  Mozilla/5.0 (compatible; URLAppendBot/1.0; +http://www.profound.net/urlappendbot.html)
  LoadImpactRload/3.2.0 (Load Impact; http://loadimpact.com);
  RSSingBot (http://www.rssing.com)
  RADaR-Bot/Nutch-1.3 (http://radar-bot.com/)
  Ocelli/1.4 (http://www.globalspec.com/Ocelli)
  Mozilla/5.0 (compatible; BuzzSumo; +http://www.buzzsumo.com/bot.html)
  Mozilla/4.0 (compatible; IDwhois/2.0alpha-10102619; Linux; en-US; +http://idwhois.info/about)
  omgilibot/0.4 +http://omgili.com
  yacybot (freeworld/global; amd64 Windows 7 6.1; java 1.6.0_23; Europe/de) http://yacy.net/bot.html
  Y!J-BRW/1.0 crawler (http://help.yahoo.co.jp/help/jp/search/indexing/indexing-15.html)
  Mozilla/5.0 (compatible; AhrefsBot/2.0; +http://ahrefs.com/robot/)
  Mozilla/5.0 (compatible; VideoSurf_bot +http://www.videosurf.com/bot.html)
  Symfony Spider (http://symfony.com/spider)
  Mozilla/5.0 (compatible; Sosospider/2.0; +http://help.soso.com/webspider.htm)
  Mozilla/5.0 (compatible; OpenindexDeepSpider/Nutch-1.5-dev; +http://www.openindex.io/en/webmasters/spider.html)
  ADmantX Platform Semantic Analyzer - ADform - ADmantX Inc. - www.admantx.com - [email protected]
  Mozilla/5.0 (compatible; MJ12bot/v1.3.3; http://www.majestic12.co.uk/bot.php?+)
  cuwhois/1.0 (+http://www.cuwhois.com/)
  Mozilla/5.0 (compatible; memoryBot/1.20.311 +http://internetmemory.org/en/)
  Mozilla/4.0 (compatible; Vagabondo/4.0; webcrawler at wise-guys dot nl; http://webagent.wise-guys.nl/; http://www.wise-guys.nl/)
  yacybot (-global; amd64 Linux 3.16.0-4-amd64; java 1.7.0_75; Europe/en) http://yacy.net/bot.html
  Mozilla/5.0 (compatible; Dazoobot/1.0; http://dazoo.fr)
  Mozilla/5.0 (compatible; Semager/1.4c; +http://www.semager.de/blog/semager-bots/)
  EuripBot/1.1 (+http://www.eurip.com) GetRobots
  yacybot (freeworld/global; amd64 Linux 3.8.0-23-generic; java 1.6.0_27; Pacific/en) http://yacy.net/bot.html
  yacybot (amd64 Linux 2.6.24-23-generic; java 1.6.0_07; Europe/de) http://yacy.net/bot.html
  yacybot (freeworld/global; x86 Windows 2003 5.2; java 1.6.0_32; Europe/de) http://yacy.net/bot.html
  yacybot (freeworld/global; amd64 Linux 3.2.0-4-amd64; java 1.7.0_25; Europe/de) http://yacy.net/bot.html
  socialbm_bot http://spider.socialbm.net
  Mozilla/5.0 (compatible; 007ac9 Crawler; http://crawler.007ac9.net/)
  webinatorbot 1.0; +http://www.webinator.de
  WorldBrewBot/2.1 (+http://www.marketbrew.com/)
  BotOnParade, http://www.bots-on-para.de/bot.html
  ltx71 - (http://ltx71.com/)
  VSB-TUO/1.0 (+http://www.vsb.cz)
  yacybot (/global; amd64 Linux 3.13.0-85-generic; java 1.7.0_101; Europe/en) http://yacy.net/bot.html
  CopperEgg/RevealUptime/TokyoJP(aws)
  Mozilla/5.0 (compatible; etoolsbot/1.0; +https://etools.io/bots/)
  Comodo-Webinspector-Crawler 2.2.2, http://www.comodorobot.com
  Mozilla/5.0 (compatible; TWMBot/0.1; +http://thewebminer.com)
  Mozilla/5.0 (compatible; Plukkie/1.3; http://www.botje.com/plukkie.htm)
  Autonomy
  yacybot (i386 Linux 2.6.30-2-686; java 1.6.0_0; SystemV/en) http://yacy.net/bot.html
  Testomatobot/1.0 (Linux x86_64; +http://www.testomato.com/testomatobot) minicrawler/4.1.0
  CatchBot/1.0; +http://www.catchbot.com
  Sitedomain-Bot(Sitedomain-Bot 1.0, http://www.sitedomain.de/sitedomain-bot/)
  Mozilla/5.0 (compatible; Uptimebot/0.2.42; +http://www.uptime.com/uptimebot)
  yacybot (/global; amd64 Linux 3.2.0-4-amd64; java 1.7.0_03; Europe/en) http://yacy.net/bot.html
  SiteTruth.com site rating system
  SiteUptime.com
  yacybot (i386 Linux 2.6.24-23-generic; java 1.6.0_16; Europe/en) http://yacy.net/bot.html
  UrlTrends Ranking Updater/2.0
  yacybot (/global; amd64 Windows 7 6.1; java 1.7.0_55; Europe/de) http://yacy.net/bot.html
  yacybot (freeworld/global; amd64 Linux 3.5.0-27-generic; java 1.7.0_25; Europe/en) http://yacy.net/bot.html
  GarlikCrawler/1.1 (http://garlik.com/, [email protected])
  Mozilla/5.0 (compatible; ParsijooBot; +http://www.parsijoo.ir/; [email protected])
  Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.1 (KHTML, like Gecko) Chrome/21.0.1180.89 Safari/537.1; 360Spider(compatible; HaosouSpider; http://www.haosou.com/help/help_3_2.html)
  Willow Internet Crawler by Twotrees V2.1
  KrOWLer/0.1.0, matentzn at cs dot man dot ac dot uk
  SimplyFast.info Headers
  yacybot (freeworld/global; amd64 Linux 2.6.32-431.29.2.el6.x86_64; java 1.7.0_65; Europe/en) http://yacy.net/bot.html
  Mozilla/5.0 (compatible; YandexBlogs/0.99; robot; +http://yandex.com/bots)
  msnbot-media/1.1 (+http://search.msn.com/msnbot.htm)
  Mozilla/5.0 (Windows NT 6.1; WOW64; rv:46.0) Gecko/20100101 Firefox/46.0 PTST/278
  Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/534.57.2 (KHTML, like Gecko) Version/5.1.7 Safari/534.57.2 PTST/278
  gipo-crawler/Nutch-1.10 (Global Internet Policy Observatory crawler)
  Mozilla/5.0 (compatible; sukibot_heritrix/3.1.1 +http://suki.ling.helsinki.fi/eng/webmasters.html)
  PayPal IPN ( https://www.paypal.com/ipn )
  Mozilla/5.0 (compatible; KaloogaBot; http://kalooga.com/crawler)
  eCairn-Grabber/1.0 (+http://ecairn.com/grabber) curl/7.15
  Mozilla/5.0 (Windows NT 6.2; WOW64) AppleWebKit/537.15 (KHTML like Gecko; +http://www.diffbot.com) Chrome/24.0.1295.0 Safari/537.15
  ImageEngine/1.0
  DER-bot (+https://goo.gl/9ZibHG)
  yacybot (freeworld/global; amd64 Windows 7 6.1; java 1.7.0_60; Europe/de) http://yacy.net/bot.html
  Mozilla/5.0 (compatible; SemrushBot/0.96.3; +http://www.semrush.com/bot.html)
  Mozilla/5.0 (compatible; Exabot/3.0; +http://www.exabot.com/go/robot)
  Mozilla/5.0 (compatible; YandexDirect/3.0; +http://yandex.com/bots)
  Mozilla/5.0 (compatible; aiHitBot/2.9; +https://www.aihitdata.com/about)
  Linguee Bot (http://www.linguee.com/bot; [email protected])
  MetaGeneratorCrawler/1.3.9 (www.metagenerator.info)
  TelegramBot (like TwitterBot)
  Mozilla/5.0 (compatible; Siteliner/1.0; +http://www.siteliner.com/bot)
  calculon spider/0.1.0 mailto:[email protected]
  yacybot (/global; amd64 Linux 3.13.0-83-generic; java 1.7.0_95; Europe/de) http://yacy.net/bot.html
  PocketParser/2.0 (+https://getpocket.com/pocketparser_ua)
  montastic-monitor http://www.montastic.com
  Mozilla/5.0 (compatible; trovitBot 1.0; +http://www.trovit.com/bot.html)
  Mozilla/5.0 (Windows NT 6.1)+(compatible; SMTBot/1.0; +http://www.similartech.com/smtbot)
  Yaanb/1.5.001 (compatible; Win64;)
  Mozilla/5.0 (compatible; Crawlera/1.10.2; UID 70054)
  Mozilla/5.0 (compatible; spbot/2.0.2; +http://www.seoprofiler.com/bot/ )
  yacybot (freeworld-global; amd64 Linux 3.2.0-4-amd64; java 1.6.0_24; Europe/en) http://yacy.net/bot.html
  Mozilla/5.0 (compatible; ImageFetcher/5.6; +http://images.weserv.nl/)
  CopperEgg/RevealUptime/N.VirginiaUSA
  Huaweisymantecspider (compatible; MSIE 8.0; [email protected])
  OrangeCrawler/Nutch-1.0 ([email protected])
  CopperEgg/RevealUptime/AmsterdamHolland
  Mozilla/5.0 (compatible; SemrushBot-SI/0.97; +http://www.semrush.com/bot.html)
  Mozilla/5.0 (X11; Linux x86_64; GTmetrix https://gtmetrix.com/) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/50.0.2661.75 Safari/537.36
  Mozilla/5.0 (compatible; Genieo/1.0 http://www.genieo.com/webfilter.html) AppEngine-Google; (+http://code.google.com/appengine; appid: s~natmacdevice)
  yacybot (/global; amd64 Linux 3.2.41-042stab094.7; java 1.7.0_75; Europe/en) http://yacy.net/bot.html
  yacybot (x86 Windows Vista 6.1; java 1.6.0_13; Europe/de) http://yacy.net/bot.html
  SmabblerBot/1.0 (http://www.smabbler.com/)
  Mozilla/5.0 (compatible; grapeFX/0.9; [email protected]
  Mozilla/5.0 (compatible; SemrushBot/1~bl; +http://www.semrush.com/bot.html)
  MetaGeneratorCrawler/1.3.14 (www.metagenerator.info)
  oku-taka-lab-bot (Mozilla-compatible; [email protected]; http://lr-www.pi.titech.ac.jp/wp/)
  Mozilla/5.0 (Linux; Android 4.1.2; Galaxy Nexus Build/JZO54K; GTmetrix https://gtmetrix.com/) AppleWebKit/537.22 (KHTML, like Gecko) Chrome/26.0.1410.58 Mobile Safari/537.22
  Mozilla/5.0 (compatible; Feedage/2.0; +http://www.feedage.com/bot.php)
  yacybot (amd64 Linux 2.6.26-2-openvz-amd64; java 1.6.0_12; UTC/en) http://yacy.net/bot.html
  yacybot (-global; amd64 Linux 2.6.32-042stab116.1; java 1.7.0_79; Europe/en) http://yacy.net/bot.html
  Mozilla/5.0 (compatible; DKIMRepBot/1.0; +http://www.dkim-reputation.org)
  Mozilla/5.0 (Windows; U; Windows NT 5.1; zh-CN; rv:1.8.0.11) Firefox/1.5.0.11; 360Spider
  Mozilla/5.0 (compatible; pub-crawler; +http://wiki.github.com/bixo/bixo/bixocrawler; [email protected])
  mfibot/1.1 (http://www.mfisoft.ru/analyst/; <[email protected]>; en-RU)
  Visited by http://tools.geek-tools.org
  peerindex
  Mozilla/5.0 (compatible; janforman.com/2.4; +http://janforman.com/)
  http://www.amagit.com/
  yacybot (freeworld/global; amd64 Linux 2.6.32-40-server; java 1.6.0_20; Europe/en) http://yacy.net/bot.html
  Mozilla/5.0 (iPhone; CPU iPhone OS 8_3 like Mac OS X) AppleWebKit/537.36 (KHTML, like Gecko) Version/8.0 Mobile/12F70 Safari/600.1.4 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
  Mozilla/5.0 (compatible; NLNZ_IAHarvester2014 +https://natlib.govt.nz/publishers-and-authors/web-harvesting/domain-harvest)
  Mozilla/5.0 (compatible; YandexImageResizer/2.0; +http://yandex.com/bots)
  mahonie, neofonie search:robot/search:robot/0.0.1 (This is the MIA Bot - crawling for mia research project. If you feel unhappy and do not want to be visited by our crawler send an email to [email protected]; http://spider.neofonie.de; [email protected])
  Mozilla/5.0 (Linux; Android 5.0; Nexus 5 Build/LRX21O) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/46.0.2490.76 Mobile Safari/537.36 PTST/276
  Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; Trident/5.0; WOW64; Trident/5.0; BingPreview/1.0b)
  Mozilla/5.0 (compatible; OptimizationCrawler/0.2; +http://www.domainoptima.com/bot.html)
  Mozilla/5.0 (compatible; Cloudinary/1.0)
  rogerbot/1.0 (http://moz.com/help/pro/what-is-rogerbot-, [email protected])
  Iframely/0.9.8 (+http://iframely.com/;)
  yacybot (webportal-global; amd64 Linux 2.6.32-5-amd64; java 1.6.0_18; Europe/en) http://yacy.net/bot.html
  Interdose AntiSpamBot/2.0 (+http://www.piep.net)
  CheckMarkNetwork/1.0 (+http://www.checkmarknetwork.com/spider.html)
  Mozilla/5.0 (iPhone; CPU iPhone OS 8_3 like Mac OS X) AppleWebKit/537.36 (KHTML, like Gecko) Version/8.0 Mobile/12F70 Safari/600.1.4 (compatible; Google Search Console)
  rogerbot/1.0 (http://www.seomoz.org/dp/rogerbot, [email protected])
  findlinks/2.5 (+http://wortschatz.uni-leipzig.de/findlinks/)
  Mozilla/5.0 (compatible; MSIE 10.0; Windows NT 6.1; WOW64; Trident/6.0) PTST/277
  yacybot (/global; x86 Windows 2003 5.2; java 1.7.0_67; Europe/de) http://yacy.net/bot.html
  NeutrinoAPI/2.0.3
  Mozilla/5.0 (compatible; spbot/5.0.3; +http://OpenLinkProfiler.org/bot )
  Mozilla/5.0 (compatible; MJ12bot/v1.4.4; http://www.majestic12.co.uk/bot.php?+)
  Site24x7 Agent
  Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.2; WOW64; Trident/6.0; .NET4.0E; .NET4.0C; .NET CLR 3.5.30729; .NET CLR 2.0.50727; .NET CLR 3.0.30729) CrawlerProcess (http://www.PowerMapper.com) /5.7.704.0
  Mozilla/5.0 (compatible; Falconsbot; +http://iws.seu.edu.cn/services/falcons/contact_us.jsp)
  Super Monitoring
  Seznam-Zbozi-robot/3.0
  LinkedInBot/1.0 (compatible; Mozilla/5.0; Jakarta Commons-HttpClient/4.3 +http://www.linkedin.com)
  Mozilla/5.0 (compatible; EveryoneSocialBot/1.0; [email protected] http://everyonesocial.com/)
  Mozilla/5.0 (compatible; IstellaBot/1.01.18 +http://www.tiscali.it/)
  Mozilla/5.0 (compatible; ToutiaoSpider/1.0; http://web.toutiao.com/media_cooperation/;)
  Mozilla/5.0 (compatible; BegunAdvertising/3.0; +http://begun.ru/begun/technology/indexer/)
  Netcraft SSL Server Survey - contact [email protected]
  Mozilla/5.0 (compatible; Qualidator.com SiteAnalyzer 1.0;)
  Mozilla/5.0 (compatible; OrangeBot/2.0; [email protected])
  Mozilla/5.0 (compatible; CloudFlare-AlwaysOnline/1.0; +http://www.cloudflare.com/always-online)
  seebot/1.0.0 (http://www.seegnify.com/bot)
  Slackbot-LinkExpanding 1.0 (+https://api.slack.com/robots)
  Eurobot/1.1 (http://eurobot.ayell.eu)
  Mozilla/5.0 (compatible; IXEbot; +http://medialab.di.unipi.it/IXEbot.html)
  Mozilla/5.0 (GetLinkInfo.com - http://www.getlinkinfo.com)
  Mozilla/5.0 (compatible; spbot/4.0.5; +http://www.seoprofiler.com/bot )
  yacybot (freeworld/global; amd64 Windows 7 6.1; java 1.6.0_31; America/en) http://yacy.net/bot.html
  Mozilla/5.0 (Windows NT 6.1; WOW64; rv:48.0) Gecko/20100101 Firefox/48.0 PTST/314
  Mozilla/5.0 (compatible; AhrefsBot/4.0; +http://ahrefs.com/robot/)
  UXCrawlerBot
  Pixray-Seeker/2.0 (http://www.pixray.com/pixraybot; [email protected])
  Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/50.0.2661.94 Safari/537.36 PTST/276
  FAST-WebCrawler/3.7 (atw-crawler at fast dot no; http://fast.no/support/crawler.asp)
  yacybot (freeworld/global; amd64 Windows 10 10.0; java 1.8.0_60; Europe/de) http://yacy.net/bot.html
  Technoratibot/7.0
  Mozilla/4.0 (compatible; MSIE 5.01; Windows NT 5.0; DomainDB-1.1; http://domaindb.com/crawler/)
  Abrave v6.0 (http://robot.abrave.com)
  Testomatobot/1.0 (Linux x86_64; +http://www.testomato.com/testomatobot) minicrawler/4.0.0~beta11
  DowntimeDetector/1.0 (+http://downforeveryoneorjustme.com)
  Comodo Spider 1.1
  Link Valet Online 1.1
  yacybot (freeworld/global; amd64 Windows XP 5.2; java 1.7.0_04; America/en) http://yacy.net/bot.html
  li_viewer ([email protected])
  Mozilla/5.0 (compatible; Mail.RU/2.0c)
  Mozilla/5.0 (compatible; aiHitBot/1.0; +http://www.aihit.com/)
  Mozilla/5.0 (compatible; Uptimebot/0.2.19; +http://www.uptime.com/uptimebot)
  Mozilla/5.0 (compatible; YandexBot/3.0; +http://yandex.com/bots)
  Mozilla/5.0 (iPhone; CPU iPhone OS 6_0 like Mac OS X) AppleWebKit/536.26 (KHTML, like Gecko) Version/6.0 Mobile/10A5376e Safari/8536.25 ( compatible; CloudServerMarketSpider/1.0; +http://cloudservermarket.com/spider.html )
  yacybot (i386 Linux 2.6.28-13-generic; java 1.6.0_13; Europe/en) http://yacy.net/bot.html
  linguatools-bot/Nutch-1.6 (searching for translated pages; http://www.linguatools.de/linguatoolsbot.html; peter dot kolb at linguatools dot org)
  Mozilla/5.0 (compatible; Lippershey/5.2; +https://www.lipperhey.com/en/about/)
  NextGenSearchBot 69 (for information visit http://www.zoominfo.com/business/nextgensearchbot.html)
  Mozilla/5.0 (compatible; YandexWebmaster/2.0; +http://yandex.com/bots)
  Mozilla/5.0 (compatible; Baiduspider/2.0; +http://www.baidu.com/search/spider.html)
  yacybot (freeworld-global; amd64 Linux 3.2.0-35-generic; java 1.7.0_09; Europe/en) http://yacy.net/bot.html
  peerindex/0.1 (http://www.peerindex.com/; crawler AT peerindex DOT com)
  iqdb/0.2 (+http://iqdb.harry.lu/)
  Mozilla/5.0 (compatible; LuminateBot/1.0; +http://www.luminate.com/bot/)
  Mozilla/5.0 (X11; Linux x86_64) adbeat.com/policy AppleWebKit/537.21 (KHTML, like Gecko) Firefox/34.0 Safari/537.21
  AntBot/1.0 (http://www.ant.com)
  Mozilla/5.0 (compatible; Thumbshots.ru; +http://thumbshots.ru/bot) Firefox/3
  Mozilla/5.0 (compatible; value/1.1; +http://www.aihit.com/)
  Digg Deeper/v1 (http://digg.com/about)
  Elmer, the Thinglink ImageBot (http://www.thinglink.com/help/ThinglinkImageBot)
  rogerbot/1.0 (http://moz.com/help/pro/what-is-rogerbot-, [email protected])
  Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.1 (KHTML, like Gecko) Chrome/21.0.1180.89 Safari/537.1; http://spyonweb.com [email protected]
  BabalooSpider/1.3 (BabalooSpider; http://www.babaloo.si; [email protected])
  Mozilla/5.0 (compatible; dotSemantic/1.0; +http://www.dotsemantic.org)
  TinEye/1.1 (http://tineye.com/crawler.html)
  Mozilla/5.0 (compatible; SemrushBot/0.98~bl; +http://www.semrush.com/bot.html)
  CorpusCrawler 2.0.9 (http://corpora.fi.muni.cz/crawler/)
  aboutthedomain
  Cliqzbot/0.1 (+http://cliqz.com/company/cliqzbot)
  Mozilla/5.0 (compatible; Uptimebot/0.2.43; +http://www.uptime.com/uptimebot)
  LinqiaMetadataDownloaderBot/1.0 ([email protected])
  yacybot (freeworld/global; amd64 Linux 3.2.0-32-generic; java 1.6.0_24; Europe/en) http://yacy.net/bot.html
  NeumobBot/0.2.16 (+http://www.neumob.com/bot)
  Mozilla/5.0 (compatible; Mp3Bot/0.4; +http://mp3realm.org/mp3bot/)
  Miniflux (https://miniflux.net)
  Mozilla/5.0 (iPhone; CPU iPhone OS 8_3 like Mac OS X) AppleWebKit/537.36 (KHTML, like Gecko; Google Page Speed Insights) Version/8.0 Mobile/12F70 Safari/600.1.4
  Digincore crawler bot. See https://www.digincore.com/crawler.html for rules and instructions.
  Mozilla/5.0 (iPhone; CPU iPhone OS 7_0 like Mac OS X) AppleWebKit/537.51.1 (KHTML, like Gecko) Version/7.0 Mobile/11A465 Safari/9537.53 (compatible; adidxbot/2.0; http://www.bing.com/bingbot.htm)
  NerdyBot
  Nutch/2.2.1 (page scorer; http://integralads.com/site-indexing-policy/)
  Mozilla/5.0 (compatible; 008/0.83; http://www.80legs.com/webcrawler.html;) Gecko/2008032620
  Mozilla/4.0 (compatible; LinkMarket-Bot)
  SeoCheckBot (Seo-Check, http://www.kfsw.de/bot.html)
  Mozilla/5.0 (compatible; Whynder Magnet; http://whynder.com/magnet)
  IdeelaborPlagiaat/1
  x28-job-bot; +http://x28.ch/bot.html
  Mozilla/5.0 (compatible; spbot/4.0.7; +https://www.seoprofiler.com/bot )
  Mozilla/5.0 (compatible; RelateIQ Crawler/0.4; +www.relateiq.com; [email protected])
  WebRankSpider/1.37 (+http://ulm191.server4you.de/crawler/)
  Mozilla/5.0 (iPhone; U; CPU iPhone OS 10_0 like Mac OS X; en-us) AppleWebKit/602.1.38 (KHTML, like Gecko) Version/10.0 Mobile/14A5297c Safari/602.1 (compatible; Mediapartners-Google/2.1; +http://www.google.com/bot.html)
  LinkWalker/3.0 (http://www.brandprotect.com)
  Mozilla/5.0 (compatible; discobot/1.0; +http://discoveryengine.com/discobot.html)
  ia_archiver-web.archive.org
  Mozilla/5.0 (compatible; Online Domain Tools - SEO Checker/1.0; +http://online-domain-tools.com)
  Mozilla/5.0 (compatible; AhrefsBot/3.0; +http://ahrefs.com/robot/)
  Mozilla/5.0 (Macintosh; Intel Mac OS X 10_6_8) AppleWebKit/537.13+ (KHTML, like Gecko) Version/5.1.7 Safari/534.57.2; Shoppimon Analyzer (http://www.shoppimon.com/)
  Mozilla/5.0 (compatible; NewShareCounts.com/1.0; +http://newsharecounts.com/crawler)
  Mozilla/5.0 (compatible; Mozilla; +http://wiki.github.com/bixo/bixo/bixocrawler; [email protected])
  Mozilla/5.0 (compatible; CukBot; Not a spammer; http://www.companiesintheuk.co.uk/bot.html)
  yacybot (freeworld/global; amd64 Linux 3.1.10-1.9-default; java 1.6.0_24; Europe/de) http://yacy.net/bot.html
  Nuhk/2.4 (+http://www.neti.ee/cgi-bin/abi/otsing.html)
  elefent/Elefent 1.1 (Friendly web walker.; http://www.elefent.eu/; [email protected])
  HubSpot Links Crawler 1.0 http://www.hubspot.com/
  Mozilla/5.0 (compatible; Siteluxbot/1.0; +https://sitelux.co.uk/)
  CopperEgg/RevealUptime/AmsterdamNL(softlayer)
  Mozilla/5.0 (compatible; spbot/4.0.7; +http://OpenLinkProfiler.org/bot )
  Mediumbot-MetaTagFetcher/0.1 (+https://medium.com/)
  Mozilla/2.0 (compatible; Ask Jeeves/Teoma; +http://about.ask.com/en/docs/about/webmasters.shtml)
  Mozilla/5.0 (compatible; 008/0.85; http://www.80legs.com/webcrawler.html) Gecko/2008032620
  yacybot (/global; amd64 Linux 4.4.0ct; java 1.7.0_91; Europe/de) http://yacy.net/bot.html
  Mozilla/5.0 (compatible; Scrubby/2.2; http://www.scrubtheweb.com/)
  yacybot (/global; amd64 Linux 3.2.0-4-amd64; java 1.8.0_05; Europe/en) http://yacy.net/bot.html
  Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.2.18) Gecko/20110622 HeartRails_Capture/1.0.4 (+http://capture.heartrails.com/) Namoroka/3.6.18
  Mozilla/5.0 (X11; U; Linux i686; de; rv:1.9.0.1; compatible; iCjobs Stellenangebote Jobs; http://www.icjobs.de) Gecko/20100401 iCjobs/3.2.3
  KOCMOHABT (+http://www.KOCMOHABT.tk/) AppEngine-Google; (+http://code.google.com/appengine; appid: s~web-explorer)
  yacybot (freeworld/global; x86 Windows 7 6.1; java 1.6.0_31; Europe/de) http://yacy.net/bot.html
  Mozilla/5.0 (compatible; MSIE 10.0; Windows NT 6.2; WOW64; Trident/6.0) CrawlerProcess (http://www.PowerMapper.com) /5.21.754.0
  Mozilla/5.0 (compatible; Linux x86_64; Mail.RU_Bot/Robots; +http://go.mail.ru/help/robots)
  Mozilla/5.0 (compatible; Googlebot/2.1; +http://import.io)
  ownCloud Server Crawler
  Mozilla/5.0 (compatible; Qwantify/2.1n; +https://www.qwant.com/)/*
  LoadImpactRload/3.1.5 (Load Impact; http://loadimpact.com);
  Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 4.0; Girafabot; girafabot at girafa dot com; http://www.girafa.com)
  OmniExplorer_Bot/6.65a (+http://www.omni-explorer.com) WorldIndexer
  Mozilla/5.0 (compatible; SMTBot/1.0; +http://www.similartech.com/smtbot)
  anonymous coward/1.0 (a Storm-based crawler; https://github.com/DigitalPebble/storm-crawler; [email protected])
  Mozilla/5.0 (compatible; Uptimebot/1.0; +http://www.uptime.com/uptimebot)
  Mozilla/5.0 (compatible; FoundSeoTool/1.0; +http://www.found.co.uk/seo-tool/)
  findlinks/2.1.5 (+http://wortschatz.uni-leipzig.de/findlinks/)
  yacybot (freeworld/global; amd64 Windows NT (unknown) 6.2; java 1.7.0_05; Africa/en) http://yacy.net/bot.html
  Mozilla/5.0 (compatible; memorybot/1.20.71 +http://archivethe.net/en/index.php/about/internet_memory1 on behalf of DNB)
  ownCloud News/5.3.6 (+https://owncloud.org/; 1 subscriber;)
  YottaBot Spider/Nutch-2.3
  Mozilla/5.0 (compatible; WeViKaBot/1.0; +http://www.wevika.de/)
  ThumbShots-Bot (+http://thumbshots.in/bot.html)
  Mnogosearch-3.1.21
  Favicon downloader (+https://favico.be/)
  Mozilla/5.0 (compatible; Page2RSS/0.7; +http://page2rss.com/)
  ExB Language Crawler 2.1.2 (+http://www.exb.de/crawler)
  PrivateSearch/0.1.0 (Polite Nutch Crawler; grierforensics.com)
  nodestackbot/0.1 ([email protected] http://nodestack.com/bot.html)
  iskanie (+http://www.iskanie.com)
  yacybot (/global; amd64 Windows 7 6.1; java 1.7.0_67; Europe/de) http://yacy.net/bot.html
  Mozilla/5.0 (compatible; Dataprovider Site Explorer; http://www.dataprovider.com/)
  Mozilla/5.0 (compatible; akula/12.0rc-2; +http://k311.fd.cvut.cz/)
  Mozilla/5.0 (compatible; bnf.fr_bot; +http://www.bnf.fr/fr/outils/a.dl_web_capture_robot.html)
  Mozilla/5.0 (iPhone; CPU iPhone OS 6_0 like Mac OS X) AppleWebKit/536.26 (KHTML, like Gecko) Version/6.0 Mobile/10A5376e Safari/8536.25 (compatible; Googlebot-Mobile/2.1; +http://www.google.com/bot.html)
  Mozilla/5.0 (compatible; MSIE 10.0; Windows NT 6.1; WOW64; Trident/6.0) PTST/284
  Mozilla/5.0 (compatible; Infohelfer/1.3.3; +http://www.infohelfer.de/crawler.php)
  www.deadlinkchecker.com XMLHTTP/1.0
  Mozilla/5.0 (compatible; TweetmemeBot/3.0; +http://tweetmeme.com/)
  Chrome/49.0.2623.87 (compatible; Wappalyzer; +https://github.com/AliasIO/Wappalyzer)
  Mozilla/5.0 (compatible; MSIE 10.0; Windows NT 6.2; WOW64; Trident/6.0) CrawlerProcess (http://www.PowerMapper.com) /5.11.740.0
  XING-contenttabreceiver/2.0
  Mozilla/5.0 (iPhone; CPU iPhone OS 6_0 like Mac OS X) AppleWebKit/536.26 (KHTML, like Gecko) Version/6.0 Mobile/10A5376e Safari/8536.25 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
  Mozilla/5.0 (compatible; MojeekBot/0.2; http://www.mojeek.com/bot.html#relaunch)
  Pingoscope
  Mozilla/5.0 (compatible; um-LN/1.0; mailto: [email protected])
  Mozilla/5.0 (compatible; Pagespeed/1.1 Fetcher; +http://www.pagespeed.de)
  HTTP-Header-Abfrage/1.0 (http://www.internalscripts.de/werkzeuge/http-header-abfrage.php)
  Mozilla/5.0 (compatible; 80bot/0.71; http://www.80legs.com/spider.html;) Gecko/2008032620
  LoadImpactRload/3.0.3 (Load Impact; http://loadimpact.com);
  Mozilla/5.0 (compatible; Peepowbot/1.0; +http://www.peepow.com/bot.php)
  yacybot (freeworld/global; amd64 Linux 2.6.32-5-amd64; java 1.6.0_26; Atlantic/en) http://yacy.net/bot.html
  WatchMouse/8.4.0.3 (http://watchmouse.com/ ; grath01.watchmouse.net)
  Mozilla/5.0 (compatible; +http://servernfo.com/)
  Mozilla/5.0 (compatible; special_archiver/3.1.1 +http://www.archive.org/details/archive.org_bot)
  CopperEgg/RevealUptime/NewarkNJUSA
  Mozilla/5.0 (compatible; archive.org_bot/3.3.0 +http://www.archive.org/details/archive.org_bot)
  Mozilla/5.0 (compatible; Plukkie/1.1; http://www.botje.com/plukkie.htm)
  Mozilla/5.0 (compatible; musobot/1.0; [email protected]; +http://www.muso.com)
  yacybot (freeworld/global; amd64 Linux 3.10.17-gentooxen; java 1.6.0_31; Europe/nb) http://yacy.net/bot.html
  ResponseCodeTest/1.1
  Mozilla/5.0 (compatible; MSIE 10.0; Windows NT 6.1; Trident/6.0) SiteCheck-sitecrawl by Siteimprove.com
  Acoon v4.9.5 (www.acoon.de)
  Mozilla/5.0 (Windows NT 6.1; WOW64) SkypeUriPreview Preview/0.5
  SiteGuardian/2.0 (Internet Monitoring)
  Mozilla/5.0 (compatible; DCPbot/1.4; +http://domains.checkparams.com/)
  BoardReader Favicon Fetcher /1.0 [email protected]
  baypup/colbert (Baypup; http://www.baypup.com/webmasters; [email protected])
  Mozilla/5.0 (compatible; MonTools.com)
  yacybot (freeworld/global; amd64 Linux 3.2.0-4-amd64; java 1.7.0_03; Etc/de) http://yacy.net/bot.html
  Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/534.51 (KHTML, like Gecko; Google Web Preview) Chrome/12.0.742 Safari/534.51
  linkapediabot (+http://www.linkapedia.com)
  Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.7.12; ips-agent) Gecko/20050922 Fedora/1.0.7-1.1.fc4 Firefox/1.0.7
  BCKLINKS 1.0
  CorpusCrawler 2.0.21 (http://corpora.fi.muni.cz/crawler/);Project:CzCorpus
  Mozilla/5.0 (compatible; Yeti/1.1; +http://help.naver.com/support/robots.html)
  Mozilla/5.0 (compatible; pingbot/2.0; +http://www.pingdom.com/)
  Mozilla/5.0 (compatible; memoryBot/1.20.275 +http://internetmemory.org/en/)
  Mozilla/5.0 (compatible; spbot/1.1; +http://www.seoprofiler.com/bot/ )
  yacybot (/global; amd64 Linux 3.13.0-88-generic; java 1.7.0_101; Europe/en) http://yacy.net/bot.html
  BlogSearch/2 +http://www.icerocket.com/
  Mozilla/5.0 (compatible; LoadTimeBot/0.81; +http://www.load-time.com/bot.html)
  Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9) Gecko/2008052906 Firefox/3.0 (BrowserSpyBot/1.0) (http://browserspy.dk/)
  Mozilla/5.0 (compatible; MojeekBot/0.6; +https://www.mojeek.com/bot.html)
  CopperEgg/RevealUptime/NewarkNJ(linode)
  Automattic Analytics Crawler/0.2; http://wordpress.com/crawler/
  larbin_2.6.3 [email protected]
  HatenaScreenshot
  CopperEgg/RevealUptime/LondonUK
  Mozilla/5.0 (compatible; MojeekBot/0.6; http://www.mojeek.com/bot.html)
  Mozilla/5.0 (compatible; evc-batch/2.0.20160902205649)
  yacybot (freeworld/global; amd64 Linux 3.2.0-23-generic; java 1.6.0_24; Europe/en) http://yacy.net/bot.html
  WordPress.com;
  Mozilla/5.0 (Windows NT 6.1; rv:6.0) Gecko/20110814 Firefox/6.0 Google (+https://developers.google.com/+/web/snippet/)
  Mozilla/5.0 (compatible; ScoutJet; http://www.scoutjet.com/)
  Mozilla/5.0 (compatible; Uptimebot/0.1.97; +http://www.uptime.com/uptimebot)
  WebTarantula.com Crawler
  WatchMouse/8.4.0.3 (http://watchmouse.com/ ; itmil01.watchmouse.net)
  LinqiaScrapeBot/1.0 ([email protected])
  Orbiter (+http://www.dailyorbit.com/bot.htm)
  Mozilla/5.0 (compatible; janforman.com/2.5; +http://janforman.com/)
  InternetSeer.com
  Mozilla/5.0 (compatible; aiHitBot/2.9; +http://www.aihitdata.com/about)
  Mozilla/5.0 (compatible; WebNL; +http://www.web.nl/webmasters/spider.html; [email protected])
  coccoc/1.0 (http://help.coccoc.vn/)
  Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/53.0.2785.116 Safari/537.36 PTST/314
  Mozilla/5.0 (compatible; Googlebot/2.1; startmebot/1.0; +https://start.me/bot)
  Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 5.2; Trident/4.0; .NET CLR 1.1.4322; .NET CLR 2.0.50727; .NET CLR 3.0.4506.2152; .NET CLR 3.5.30729; .NET4.0C; .NET4.0E; PTST 2.386)
  Mozilla/5.0 (compatible; PrivacyAwareBot/1.1; +http://www.privacyaware.org)
  Mozilla/5.0 (compatible; idmarch Automatic.beta/1.4; +http://www.idmarch.org/bot.html)
  Mozilla/5.0 (compatible; tldstat.com/Bot; +http://tldstat.com/bot.html)
  yacybot (/global; amd64 Linux 3.16.0-4-amd64; java 1.7.0_111; Europe/de) http://yacy.net/bot.html
  Mozilla/5.0 (compatible; Crawlera/1.10.2; UID 84846)
  Mozilla/5.0 (compatible; spbot/4.0a; +http://www.seoprofiler.com/bot )
  Mozilla/5.0 (Windows NT 6.1; Win64; x64; +http://www.komodia.com/newwiki/index.php/URL_server_crawler) KomodiaBot/1.0
  WebarooBot (Webaroo Bot; http://64.124.122.252/feedback.html)
  OOZBOT/0.20 ( -- ; http://www.setooz.com/oozbot.html ; agentname at setooz dot_com )
  Mozilla/5.0 (compatible; memoryBot/1.21.16 +http://internetmemory.org/en/)
  FAST-WebCrawler/3.8
  Googlebot-Image/1.0
  enlle punto com/Nutch-1.9
  OpenAcoon v4.10.5 (www.openacoon.de)
  yacybot (freeworld/global; amd64 Linux 3.0.0-21-generic; java 1.6.0_26; America/en) http://yacy.net/bot.html
  Microsearch.ru Bot 1.3;http://microsearch.ru/webmasters
  Mozilla/5.0 (compatible; Google-Site-Verification/1.0)
  Mozilla/5.0 (compatible; Qwantify/2.3w; +https://www.qwant.com/)/2.3w
  rogerbot/1.1 (http://moz.com/help/guides/search-overview/crawl-diagnostics#more-help, [email protected])
  HubSpot Connect 2.0 (http://dev.hubspot.com/)
  Robozilla/1.0
  Mozilla/5.0 (compatible; spbot/2.0.3; +http://www.seoprofiler.com/bot/ )
  Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/53.0.2785.116 Safari/537.36 PTST/314
  facebookexternalhit/1.1
  PercolateCrawler/3.1.30 ([email protected])
  Mozilla/5.0 (compatible; Nmap Scripting Engine; https://nmap.org/book/nse.html)
  Mozilla/5.0 (compatible; pdffillerbot/1.0; +http://www.pdffiller.com/en/support.htm)
  fastbot crawler beta 4.0 (+http://www.fastbot.de)
  SafeDNS search bot/Nutch-1.9 (https://www.safedns.com/searchbot; support [at] safedns [dot] com)
  Mozilla/5.0 (Nekstbot; http://nekst.ipipan.waw.pl/nekstbot/)
  yacybot (freeworld/global; amd64 Linux 2.6.32-5-amd64; java 1.6.0_18; US/en) http://yacy.net/bot.html
  yacybot (freeworld/global; amd64 Linux 3.0.0-12-generic; java 1.6.0_26; Europe/en) http://yacy.net/bot.html
  coccoc/1.0 ()
  larbin_2.6.3 ([email protected])
  yacybot (i386 Linux 2.6.26-2-686; java 1.6.0_0; Europe/de) http://yacy.net/bot.html
  Mozilla/5.0 (Windows NT 6.1; compatible; BDCbot/1.0; +http://ecommerce.bigdatacorp.com.br/faq.aspx) AppleWebKit/535.1 (KHTML, like Gecko) Chrome/13.0.782.107 Safari/535.1
  yacybot (amd64 Linux 2.6.31-22-server; java 1.6.0_18; Asia/en) http://yacy.net/bot.html
  yacybot (/global; amd64 Windows Server 2012 6.2; java 1.8.0_91; Europe/de) http://yacy.net/bot.html
  Mozilla/5.0 (compatible; Netseer crawler/2.0; +http://www.netseer.com/crawler.html; [email protected])
  Mozilla/5.0 (compatible; heritrix/3.1.1 +http://www.mixdata.com)
  Scrapy/0.25.1 (+http://scrapy.org)
  Mozilla/5.0 (compatible; Scopia Crawler 1.2; +http://www.scopia.co)
  cn_ab_viewer ([email protected])
  Mozilla/5.0 (Windows NT 6.3; Trident/7.0; rv 11.0) like Gecko (compatible; Zombiebot/2.1; +http://www.zombiedomain.net/robot/)
  Mozilla/5.0 (compatible; Uptimebot/0.1.38; +http://www.uptime.com/uptimebot)
  yacybot (freeworld/global; amd64 Windows 7 6.1; java 1.7.0_25; Europe/de) http://yacy.net/bot.html
  Python-urllib/2.7 (+http://sketch.juls.savba.sk/aranea_about)
  Mozilla/5.0 (compatible; spbot/4.0; +http://www.seoprofiler.com/bot )
  yacybot (-global; amd64 Windows 8 6.2; java 1.7.0_55; Europe/de) http://yacy.net/bot.html
  yacybot (/global; amd64 Linux 3.2.41-042stab094.7; java 1.7.0_79; Europe/en) http://yacy.net/bot.html
  Mozilla/5.0 (compatible; UASlinkChecker/1.0; +http://user-agent-string.info/UASlinkChecker)
  Y!J-BRO/YFSJ crawler (compatible; Mozilla 4.0; MSIE 5.5; http://help.yahoo.co.jp/help/jp/search/indexing/indexing-15.html; YahooFeedSeekerJp/2.0)
  yacybot (freeworld/global; amd64 Linux 3.10.15-1-MANJARO; java 1.7.0_40; Asia/en) http://yacy.net/bot.html
  Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.2; WOW64; Trident/6.0; .NET4.0E; .NET4.0C; .NET CLR 3.5.30729; .NET CLR 2.0.50727; .NET CLR 3.0.30729) CrawlerProcess (http://www.PowerMapper.com) /5.9.726.0
  Scrubby/2.1 (http://www.scrubtheweb.com/)
  Mozilla/5.0 (compatible; JobKeresoBot; +http://www.kozvetlen-allasok.hu/help.jsp; [email protected]; 5.0
  yacybot (/global; amd64 Windows 7 6.1; java 1.8.0_66; Europe/de) http://yacy.net/bot.html
  Mozilla/5.0 (X11; Linux x86_64 UXCrawlerBot http://zium.co)
  Slackbot-LinkExpanding (+https://api.slack.com/robots)
  yacybot (freeworld/global; amd64 Linux 3.12.15-gentoo; java 1.7.0_55; Europe/en) http://yacy.net/bot.html
  Mozilla/5.0 (compatible; LinkpadBot/1.07; +http://www.linkpad.ru)
  Mozilla/5.0 (compatible; ZB-1; +http://www.zeerch.com/bot.php)
  Mozilla/5.0 (compatible; WbSrch/1.1 +http://wbsrch.com)
  WeSEE:Search
  Mozilla/5.0 (compatible; DCPbot/1.1; +http://domains.checkparams.com/)
  yacybot (webportal-global; amd64 Linux 2.6.32-5-amd64; java 1.6.0_45; Europe/en) http://yacy.net/bot.html
  WordPress.com mShots; http://support.wordpress.com/contact/
  Mozilla/4.0 (compatible; MSIE 5.01; Windows NT 5.0; FeedFinder-2.0; http://bloggz.se/crawler)
  Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.6; en-US; rv:1.9.2) Gecko/20100115 Firefox/3.6 (FlipboardProxy/0.0.5; +http://flipboard.com/browserproxy)
  Mozilla/5.0 (compatible; coccocbot-image/1.0; +http://help.coccoc.com/searchengine)
  Mozilla/5.0 (X11; Linux x86_64; GTmetrix http://gtmetrix.com/) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/31.0.1650.63 Safari/537.36
  https_page_test ([email protected])
  Mozilla/5.0 (Windows; U; Windows NT 5.1; zh-CN; rv:1.8.0.11) Firefox/1.5.0.11 360Spider;
  yacybot (freeworld/global; amd64 Linux 3.2.0-2-amd64; java 1.6.0_24; Europe/en) http://yacy.net/bot.html
  Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_2) AppleWebKit/600.1.4 (KHTML, like Gecko) Safari/600.1.4 (compatible; Princetonbot/1.0; +http://http://tigress-web.princeton.edu/~fy/bot.html)
  Mozilla/5.0 (Windows; U; Windows NT 5.1; zh-CN; rv:1.8.0.11) Gecko/20070312 Firefox/1.5.0.11; 360Spider
  KeywordDensityRobot/0.8 (http://www.seocentro.com/tools/search-engines/keyword-density.html)
  Mozilla/5.0 (compatible; aiHitBot-BP/1.1; +http://www.aihit.com/)
  findlinks/1.1.6-beta4 (+http://wortschatz.uni-leipzig.de/findlinks/)
  Mozilla/5.0 (compatible; Butterfly/1.0; +http://labs.topsy.com/butterfly/) Gecko/2009032608 Firefox/3.0.8
  CopperEgg/RevealUptime/FremontCA(linode)
  Mozilla/5.0 (compatible; NTENTbot; +http://www.ntent.com/ntentbot)
  link_thumbnailer
  KeywordDensityRobot/0.9 (http://www.seocentro.com/tools/search-engines/keyword-density.html)
  Mozilla/5.0 (compatible;WI Job Roboter Spider Version 3;+http://www.webintegration.at)
  Mozilla/5.0 (compatible; Mail.RU/2.0)
  CopperEgg/RevealUptime/N.California(aws)
  Searchie Bot/1.0 (a Storm-based crawler; https://www.searchie.org; [email protected])
  ^Nail (http://CaretNail.com)
  MarketBrewBot/2.1 (+http://www.marketbrew.com/)
  Updownerbot (+http://www.updowner.com/bot)
  yacybot (-global; amd64 Windows 8.1 6.3; java 1.7.0_55; Europe/de) http://yacy.net/bot.html
  Yanga WorldSearch Bot v1.1/beta (http://www.yanga.co.uk/)
  Mozilla/5.0 (compatible; bixolabs/1.0; +http://bixolabs.com/crawler/general; [email protected])
  Mozilla/5.0 (compatible; Scopia Crawler 2.0; +http://www.scopia.co)
  asafaweb.com
  KrOWLer/0.0.1, matentzn at cs dot man dot ac dot uk
  Mozilla/5.0 (compatible; alexa site audit/1.0; +http://www.alexa.com/help/webmasters; [email protected])
  findlinks/1.0 (+http://wortschatz.uni-leipzig.de/findlinks/)
  SygolBot http://www.sygol.com
  voltron
  Mozilla/5.0 (compatible; uMBot-FC/1.0; mailto: [email protected])
  Comodo-Webinspector-Crawler 2.2.1, http://www.comodorobot.com
  Superarama.com-Tarama-Botu-v.0.1
  yacybot (freeworld/global; amd64 Linux 3.2.0-3-amd64; java 1.6.0_24; Europe/en) http://yacy.net/bot.html
  Mozilla/5.0 (compatible; SSLBot/1.0; http://www.sslstats.com/sslbot)
  PagesInventory.com (robot http://www.pagesinvenotry.com)
  Mozilla/5.0 (iPhone; U; CPU iPhone OS 4_3 like Mac OS X; en-US) adbeat.com/policy AppleWebKit/533.17.9 (KHTML, like Gecko) Version/5.0.2 Mobile/8F190 Safari/6533.18.5
  WordPress.com Reader
  Mozilla/5.0 (compatible; SEOkicks-Robot +http://www.seokicks.de/robot.html)
  Woko 3.0
  Gigabot/3.0 (http://www.gigablast.com/spider.html)
  ichiro/4.0 (http://help.goo.ne.jp/door/crawler.html)
  bl.uk_lddc_bot/3.1.1 (+http://www.bl.uk/aboutus/legaldeposit/websites/websites/faqswebmaster/index.html)
  Urlstat (http://bit.ly/1dJuuzs)
  yacybot (/global; amd64 Windows 8.1 6.3; java 1.7.0_55; America/ru) http://yacy.net/bot.html
  Bad Neighborhood Header Detector (http://www.bad-neighborhood.com/header_detector.php)
  Mozilla/5.0 (compatible; GeliyooBot/1.0beta; +http://www.geliyoo.com/)
  NalezenCzBot/1.0 (http://www.nalezen.cz/about-crawler)
  WeSEE_Bot:we_help_monitize_your_site (http://www.wesee.com/bot/)
  yacybot (freeworld/global; x86 Windows XP 5.1; java 1.7.0_55; Asia/en) http://yacy.net/bot.html
  studylib download bot, see http://studylib.es/public/bot for additional info
  SemrushBot/0.91
  Jabse.com Crawler v.1.0 www.jabse.com/crawler.php//imagecrawler
  Mozilla/4.0 (compatible; MSIE 5.0; Windows NT; Girafabot; girafabot at girafa dot com; http://www.girafa.com)
  BizwikiBot/0.3 (http://www.bizwiki.com/bizwikibot.htm)
  Mozilla/5.0 (compatible; BIXOCRAWLER; +http://wiki.github.com/bixo/bixo/bixocrawler; [email protected])
  PostPost/1.0 (+http://postpost.com/crawlers)
  Mozilla/5.0 (nomore404.com robot/1.1; +https://nomore404.com/)
  Riddler (http://riddler.io/about)
  Mozilla/5.0 (compatible; socketcrawler; http://nlp.fi.muni.cz/projects/biwec/)
  internetVista monitor (Mozilla compatible)
  2Bone_LinkChecker/1.0 libwww-perl/6.03
  MaxPointCrawler/Nutch-1.1 (maxpoint.crawler at maxpointinteractive dot com)
  DomainStatsBot/1.0 (http://domainstats.io/our-bot)
  KDDI-CA31 UP.Browser/6.2.0.7.3.129 (GUI) MMP/2.0 (compatible; ichiro/mobile goo;+http://search.goo.ne.jp/option/use/sub4/sub4-1/)
  Mergadobot/3.0.2 (+http://mergado.cz)
  DoCoMo/2.0 P900i(c100;TB;W24H11) (compatible; ichiro/mobile goo; +http://search.goo.ne.jp/option/use/sub4/sub4-1/)
  SalesIntelligent/v1.1
  SiteAnalyzerBot
  DoCoMo/2.0 P900i(c100;TB;W24H11) (compatible; ichiro/mobile goo; +http://help.goo.ne.jp/help/article/1142/)
  yacybot (/global; i386 Linux 3.16.3-gentoo; java 1.7.0_55; Australia/en) http://yacy.net/bot.html
  Mozilla/5.0 (compatible; Linux x86_64; Mail.RU_Bot/Img/2.0; +http://go.mail.ru/help/robots)
  Cliqz Bot (+http://www.cliqz.com)
  SalesIntelligent (+https://www.salesintelligent.com)
  Mozilla/5.0 (compatible; memoryBot/1.20.243 +http://internetmemory.org/en/)
  gimme60 (Gimme60 Store ID Bot; gimme60.com)
  Orbiter/1.0 (http://dailyorbit.com/)
  msnbot/1.1 (+http://search.msn.com/msnbot.htm)
  Mozilla/5.0 (compatible; memoryBot/1.20.241 +http://internetmemory.org/en/)
  Mozilla/5.0 (compatible; akula/k311; +http://k311.fd.cvut.cz/)
  Irokez.cz monitoring v1.2 - (http://www.irokez.cz, Irokez.cz, crawl)
  CCBot/2.0 (http://commoncrawl.org/faq/)
  Mozilla/5.0 (compatible; openstat.ru/Bot)
  Mozilla/5.0 (compatible; GigaBot/1.0; +http://www.gigablast.com/ )
  Readability/6a54d4 - http://readability.com/about/
  percbotspider <[email protected]>
  Mozilla/5.0 (compatible; spbot/1.0; +http://www.seoprofiler.com/bot/ )
  HatenaScreenshot/1.0 (checker)
  Mozilla/5.0 (compatible; AcoonBot/4.12.1; +http://www.acoon.de/robot.asp)
  CRAZYWEBCRAWLER 0.9.2, http://www.crazywebcrawler.com
  Mozilla/5.0 (compatible; Online Domain Tools - Online Sitemap Generator/1.1; +http://sitemap-generator.online-domain-tools.com)
  Mozilla/5.0 (compatible; Google-Structured-Data-Testing-Tool +https://search.google.com/structured-data/testing-tool)
  OnPageBot (compatible; Googlebot 2.1; +https://bot.onpage.org/)
  Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/48.0.2564.103 Safari/537.36 Yandex.Translate
  Sogou web spider/4.0
  Mozilla/5.0 (compatible; YandexMetrika/2.0; +http://yandex.com/bots DEV)
  LinkWalker/2.0
  SeoCheck (FischerNetzDesign Seo Checker, [email protected])
  yacybot (freeworld/global; amd64 Linux 3.2.13-1-ARCH; java 1.7.0_03-icedtea; Europe/fr) http://yacy.net/bot.html
  SiteCheck - http://downforeveryoneorjustme.com
  Robots_Tester_http_www.searchenginepromotionhelp.com
  CRAZYWEBCRAWLER 0.9.4, http://www.crazywebcrawler.com
  Mozilla/5.0 (compatible; heritrix/3.1.1 +http://places.tomtom.com/crawlerinfo)
  Mozilla/5.0 (Windows; U; Windows NT 5.0; en-US; rv:1.7.10) Gecko/20050716 Thunderbird/1.0.6 - WebCrawler http://cognitiveseo.com/bot.html
  Mozilla/5.0 (compatible; Cliqzbot/1.0 +http://cliqz.com/company/cliqzbot)
  yacybot (/global; amd64 Linux 4.2.0-gentoo-r1; java 1.8.0_45; Australia/en) http://yacy.net/bot.html
  DomainsDB.net MetaCrawler v.0.9.7c (http://domainsdb.net/)
  Mozilla/4.0 (compatible; MSIE 5.5; Windows NT 5.0; mowser; http://www.mowser.com)
  www.probethenet.com scanner
  Mozilla/5.0 (compatible; Crawlera/1.10.2; UID 6573)
  LCC (+http://corpora.informatik.uni-leipzig.de/crawler_faq.html)
  Imagga Downloader/v0.14.3
  SeopultContentAnalyzer/1.0
  Mozilla/5.0 (compatible; +http://www.hosterstats.com/newgtld-websurvey.php)
  Mozilla/5.0 (compatible; GoogleDocs; apps-spreadsheets; +http://docs.google.com)
  Mozilla/5.0 (compatible;acapbot/0.1)
  Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1;Alibaba.Security.Heimdall)
  HTTPClient/1.0 (2.8.0, ruby 2.0.0 (2014-05-08))
  Jetty/9.3.z-SNAPSHOT
  Jetty/9.3.z-SNAPSHOT, SMUrlExpander
  Jetty/9.3.z-SNAPSHOT, SMViewer
  Liferea/0.x.x (Linux; en_US.UTF-8; http://liferea.sf.net/)
  SynHttpClient-Built/5825
  asynchttp
  SMUrlExpander
  KickFire
  HAAARTLAND http client
  PEAR HTTPRequest class ( http://pear.php.net/ )
  Mozilla/4.0 (vBSEO; http://www.vbseo.com)
  HAARTLAND http client
  JS-Kit URL Resolver http://js-kit.com/
  http://www.checkprivacy.or.kr:6600/RS/PRIVACY_ENFAQ.jsp
  Kml-Google; (+http://code.google.com/apis/kml) gzip
  gvfs/1.20.3
  request.js
  Microsoft URL Control - 6.01.9782
  git/2.1.4
  IPS/1.0
  OxfordCloudService/1.0
  Mozilla/5.0 (compatible; informatique at mixdata dot com/0.6; +http://t.co/GSRLLKex24; informatique at mixdata dot com)
  Mozilla/5.0 (compatible; Feedwind/2.0; +http://feed.mikle.com/support/description/)
  Mozilla/5.0 (compatible; Feedwind/3.0; +http://feed.mikle.com/support/description/)
  Mozilla/5.0 (Macintosh; Intel Mac OS X 1068) AppleWebKit/534.50 (KHTML like Gecko) Version/5.1 Instapaper/4.0 (+http://www.instapaper.com/)
  Mozilla/5.0 (compatible; Keywords Research; http://www.check-domains.com/keyword-research/keywords-adwords.php)
  CAAM/0.1 (CAAM; caam dot crwlr at gmail dot com)
  i2kconnect/"1.7.0"
  resolver/0.1.12 (+http://github.com/andris9/resolver)
  DangDang
  Mozilla/5.0 (compatible; Tetrahedron/1.0; +https://compute.info/tet)
  PritTorrent/0.1
  FeedBunch/1.0 (+http://www.feedbunch.com)
  Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.21 (KHTML, like Gecko) www.HomepageScreenshot.de - Free ScreenShotService/1.0 Safari/537.21
  Mozilla/5.0 (compatible; Licorne Image Snapshot 1.0; +http://www.jednorozec.cz)
  Pinterest/0.2 (+https://www.pinterest.com/bot.html)
  Pinterest/0.1 (+https://www.pinterest.com/bot.html)
  Mozilla/5.0 (iplabel; Windows NT 6.3; WOW64; rv:33.0) Gecko/20100101 Firefox/33.0
  Mozilla/5.0 zgrab/0.x
  Zend_Http_Client
  Zend\Http\Client
  Zao/0.1 (http://www.kototoi.org/zao/)
  YOURLS v1.7.1 +http://yourls.org/ (running on http://flnx.co)
  yoogliFetchAgent/0.1
  Yo-yo
  Mozilla/5.0 (compatible; Yahoo Link Preview; https://help.yahoo.com/kb/mail/yahoo-link-preview-SLN23615.html)
  Mozilla/5.0 (compatible; Yahoo Ad monitoring; https://help.yahoo.com/kb/yahoo-ad-monitoring-SLN24857.html)
  Mozilla/5.0 (iPhone; CPU iPhone OS 7_1 like Mac OS X) AppleWebKit (KHTML, like Gecko) Mobile (compatible; Yahoo Ad monitoring; https://help.yahoo.com/kb/yahoo-ad-monitoring-SLN24857.html)
  xpymep.exe
  xpymep2.exe
  xpymep3.exe
  Xenu Link Sleuth 1.2b
  Xenu Link Sleuth 1.2c
  Xenu Link Sleuth 1.2d
  Xenu Link Sleuth 1.2e
  Xenu Link Sleuth 1.2f
  Xenu Link Sleuth 1.2g
  Xenu Link Sleuth 1.2h
  Xenu Link Sleuth 1.2i
  Xenu Link Sleuth/1.3.7
  Xenu Link Sleuth/1.3.8
  Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/40.0.2214.93 Safari/537.36 XaxisSemanticsClassifier/1.0 http://crystalsemantics.com
  WWW-Mechanize/1.73
  WordPress/4.0.1; http://bestcomp.hol.es
  WordPress/4.3.3; http://homebusinessreviews.online
  WordPress/4.3.3; http://legithomejobs.club
  WordPress/4.4.2; http://cheapautoinsuranceinva.com
  WordPress/4.4.2; http://citycollegeinmiami.com
  WordPress/4.4.2; http://cookingideabook.com
  WordPress/4.4.2; http://oldmanhealth.com
  WordPress/4.4.2; http://sapcrmarchitecture.com
  WordPress/4.4.2; http://www.spar3d.com
  WordPress/4.5.2; http://1.we-careu.xyz
  WordPress/x.x.x.x PHP/4.x.xxMozilla/5.0 (Unknown; Linux x86_64) AppleWebKit/534.34 (KHTML, like Gecko) PhantomJS/1.6.1 Safari/534.34
  Mozilla/5.0 (Macintosh; Intel Mac OS X) Word/14.0.0
  Mozilla/5.0 (Macintosh; Intel Mac OS X) Word/14.25.0
  Mozilla/5.0 (Macintosh; Intel Mac OS X) Word/14.47.0
  Mozilla/5.0 (Macintosh; Intel Mac OS X) Word/14.48.0
  Mozilla/5.0 (Macintosh; Intel Mac OS X) Word/14.50.0
  Mozilla/5.0 (Macintosh; Intel Mac OS X) Word/14.56.0
  Mozilla/5.0 (Macintosh; Intel Mac OS X) Word/14.61.0
  Mozilla/5.0 (Macintosh; Intel Mac OS X) Word/14.62.0
  Mozilla/5.0 (Macintosh; Intel Mac OS X) Word/14.63.0
  Mozilla/5.0 (Macintosh; Intel Mac OS X) Word/14.64.0
  WomlpeFactory/0.1 (+http://www.Womple.com/bot.html)
  Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/534.34 (KHTML, like Gecko) wkhtmltoimage Safari/534.34
  Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/534.34 (KHTML, like Gecko) wkhtmltopdf Safari/534.34
  Mozilla/4.0 (compatible; Win32; WinHttp.WinHttpRequest.5)
  Windows-RSS-Platform/2.0 (IE 11.0; Windows NT 10.0)
  WhatsApp
  WhatsApp/2.12.15/i
  WhatsApp/2.12.16/i
  WhatsApp/2.12.17/i
  WhatsApp/2.12.449 A
  WhatsApp/2.12.453 A
  WhatsApp/2.12.510 A
  WhatsApp/2.12.540 A
  WhatsApp/2.12.548 A
  WhatsApp/2.12.555 A
  WhatsApp/2.12.556 A
  WhatsApp/2.16.1/i
  WhatsApp/2.16.13 A
  WhatsApp/2.16.2/i
  WhatsApp/2.16.42 A
  WhatsApp/2.16.57 A
  WhatWeb/0.4.8-dev
  http://www.almaden.ibm.com/cs/crawler [wf84]
  WeLikeLinks - WeAreNotEvil
  Websquash.com (Add url robot)
  WeCrawlForThePeace - We are not Evil
  Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.21 (KHTML, like Gecko) webscreenie Safari/537.21
  Mozilla/3.0 (compatible; WebMon 1.0.11; Windows XP)
  Mozilla/5.0 (X11; BSD Four) AppleWebKit/534.34 (KHTML, like Gecko) webkit2png Safari/534.34
  WebFetch
  WebIndex
  webcollage/1.114
  webcollage/1.117
  webcollage/1.125
  webcollage/1.129
  webcollage/1.93
  Mozilla/3.0 (compatible; WebCapture 2.0; Auto; Windows)
  W3C_Unicorn/1.0
  W3C_I18n-Checker/1.0
  W3C-mobileOK/DDC-1.0
  W3C-checklink
  W3C-checklink/2.90 libwww-perl/5.64
  W3C-checklink/3.6.2.3 libwww-perl/5.64
  W3C-checklink/4.2 [4.20] libwww-perl/5.803
  W3C-checklink/4.2.1 [4.21] libwww-perl/5.803
  W3C-checklink/4.3 [4.42] libwww-perl/5.805
  W3C-checklink/4.3 [4.42] libwww-perl/5.808
  W3C-checklink/4.3 [4.42] libwww-perl/5.820
  W3C-checklink/4.5 [4.154] libwww-perl/5.823
  W3C-checklink/4.5 [4.160] libwww-perl/5.823
  w3af.org
  VYU2 (GNU; OpenRISC)
  VORTEX/1.2 ( http://marty.anstey.ca/robots/vortex/)
  Vortex/2.2 ( http://marty.anstey.ca/robots/vortex/)
  Vortex/2.2 (+http://marty.anstey.ca/robots/vortex/)
  Mozilla/5.0 (Windows NT 5.1; rv:11.0) Gecko Firefox/11.0 (via ggpht.com GoogleImageProxy)
  URLChecker
  urlresolver
  Upflow/1.0
  Typhoeus - https://github.com/typhoeus/typhoeus
  UdmSearch/3.1.x
  Mozilla/5.0 Tweetminster
  Mozilla/5.0 (compatible; +http://tweetedtimes.com)
  truwoGPS/1.0 (GNU/Linux; U; i686; en-US; +http://www.lan4lano.net/browser.html )
  Traackr.com
  More-Testing (+http://www.theinternetrules.com)
  The Expert HTML Source Viewer (http://www.expert-html.com)
  htmlayout 3.3; above-Windows-7; www.terrainformatica.com )
  WebFuck V2.1 T0PHackTeam www.t0p.xyz
  Symfony2 BrowserKit
  Mozilla/5.0 (compatible; SwiteScraper)
  Summify (Summify/1.0.1; +http://summify.com)
  Stratagems Kumo
  Sqworm/2.9.85-BETA (beta_release; 20011115-775; i686-pc-linux-gnu)
  Sprinklr 2.0
  spray-can/1.3.3
  SortSite/4.4.500.0
  Snoopy v2.0.0
  SMRF URL Expander
  silk/1.0 (+http://www.slider.com/silk.htm)/3.7
  SiteBar/3.3.5 (Bookmark Server; http://sitebar.org/)
  SiteBar/3.3.8 (Bookmark Server; http://sitebar.org/)
  SimplePie/1.3.1 (Feed Parser; http://simplepie.org; Allow like Gecko) Build/20160226060242
  ShortLinkTranslate
  Mozilla/5.0 (compatible; Seznam screenshot-generator 2.1; +http://fulltext.sblog.cz/screenshot/)
  SEOstats 2.0.9 https://github.com/eyecatchup/SEOstats
  semanticdiscovery/0.1
  SearchSight/2.0 (http://SearchSight.com/)
  ScoutURLMonitor/5.9.8
  Robosourcer/1.0
  Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; RetrevoPageAnalyzer; +http://www.retrevo.com/content/about-us)
  RestSharp/105.2.3.0
  Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_5) AppleWebKit/537.78.2 (KHTML, like Gecko) Version/7.0.1(2.5.3) Safari(ReederForMac)
  redback/v0-570-g26f8c96
  RebelMouse/0.1 Mozilla/5.0 (compatible; http://rebelmouse.com) Gecko/20100101 Firefox/7.0.1
  RealPlayer%20Downloader/1569 CFNetwork/720.3.13 Darwin/14.3.0 (x86_64)
  RealPlayer%20Downloader/1701 CFNetwork/760.5.1 Darwin/15.5.0 (x86_64)
  RealPlayer%20Downloader/1725 CFNetwork/720.3.13 Darwin/14.3.0 (x86_64)
  RealPlayer%20Downloader/1750 CFNetwork/720.5.7 Darwin/14.5.0 (x86_64)
  RealPlayer%20Downloader/1750 CFNetwork/760.0.5 Darwin/15.0.0 (x86_64)
  RealPlayer%20Downloader/1750 CFNetwork/760.4.2 Darwin/15.4.0 (x86_64)
  Qirina Hurdler v. 1.05 10.11.01 (+http://www.qirina.com/hurdler.html)
  python-requests/1.1.0 CPython/2.7.6 Linux/3.13.0-58-generic
  python-requests/1.2.0 CPython/2.7.3 Linux/3.2.0-29-virtual
  python-requests/2.1.0 CPython/2.7.3 Linux/3.2.0-4-amd64
  python-requests/2.10.0
  python-requests/2.2.1 CPython/2.7.6 Linux/3.13.0-48-generic
  python-requests/2.2.1 CPython/2.7.6 Linux/3.13.0-83-generic
  python-requests/2.3.0 CPython/2.7.6 Linux/3.13.0-57-generic
  python-requests/2.5.0 CPython/2.7.9 Linux/3.14.20-20.44.amzn1.x86_64
  python-requests/2.6.0 CPython/2.7.3 Linux/3.2.0-4-amd64
  python-requests/2.6.0 CPython/2.7.5 Linux/3.10.0-327.3.1.el7.x86_64
  python-requests/2.7.0 CPython/2.7.6 Linux/3.13.0-85-generic
  python-requests/2.8.0
  python-requests/2.8.1
  python-requests/2.9.1
  Python-httplib2/0.7.4 (gzip)
  Pulsepoint XT3 web scraper
  Mozilla/5.0 (compatible; ProspectB2B/2.0; http://prospectb2b.com)
  Mozilla/5.0 (Macintosh; Intel Mac OS X) PowerPoint/14.39.0
  PostRank/2.0 (postrank.com; 1 subscribers)
  PostRank/2.0 (postrank.com)
  postano
  Ploetz + Zeller (http://www.ploetz-zeller.de) Link Validator v1.0 ([email protected]) for ARIS Business Architect
  Pizilla++ ver 2.45
  phpcrawl
  Pearltrees
  Pcore-HTTP/v0.23.20
  Mozilla/5.0 (Windows; U; Windows NT 6.0; en-US; rv:1.9.0.4) Gecko/2008102920 http://ow.ly web crawler (.NET CLR 3.5.30729)
  Mozilla/5.0 (compatible; Optimizer)
  Mozilla/5.0 [en] (X11, U; OpenVAS 7.0.10)
  JetBrains Omea Reader 1.0.x (http://www.jetbrains.com/omea_reader/)
  JetBrains Omea Reader 2.0 Release Candidate 1 (http://www.jetbrains.com/omea_reader/)
  okhttp/2.5.0
  oegp v. 1.3.0
  Nuzzel
  Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/43.0.2357.134 Safari/537.36 http://notifyninja.com/monitoring
  Notifixious/LinkChecker (http://notifixio.us)
  http://www.nominet.org.uk/privacypolicy
  node-superagent/1.8.3
  node.io
  Kimengi/nineconnections.com
  newspaper/0.0.9.8
  newspaper/0.1.7
  newsme/1.0; [email protected]
  NewsGator/2.0 Bot (http://www.newsgator.com)
  NewsGator/2.5 (http://www.newsgator.com; Microsoft Windows NT 5.1.2600.0; .NET CLR 1.1.4322.2032)
  NetTrack Anonymous Web Statistics https://nettrack.info/support.php
  NetTrack.info Crawler
  NetResearchServer(http://www.look.com)
  NetResearchServer/2.5(loopimprovements.com/robot.html)
  NetResearchServer/2.7(loopimprovements.com/robot.html)
  NetResearchServer/2.8(loopimprovements.com/robot.html)
  NetResearchServer/3.5(loopimprovements.com/robot.html)
  NetResearchServer/4.0(loopimprovements.com/robot.html)
  NetLyzer FastProbe (See http://netlyzer.com/report/www.domain.com for info)
  check_http/v2.0.3 (nagios-plugins 2.0.3)
  MVAClient
  Link Validity Check From: http://www.w3dir.com/cgi-bin (Using: Hot Links SQL by Mrcgiguy.com)
  Mojolicious (Perl)
  mogimogi/1.0
  Mindjet
  Mindjet MindManager
  Microsoft Windows Network Diagnostics
  Microsoft Office Excel 2013
  Microsoft Office Excel 2014
  Microsoft Office Excel 2014 (16.0.6528) Windows NT 10.0
  Microsoft Office Existence Discovery
  Microsoft Office Mobile/15.0
  Microsoft Office OneNote 2013
  Microsoft Office PowerPoint 2013
  Microsoft Office PowerPoint 2013 (15.0.4420) Windows NT 6.2
  Microsoft Office PowerPoint 2013 (15.0.4641) Windows NT 6.2
  Microsoft Office PowerPoint 2013 (15.0.4797) Windows NT 10.0
  Microsoft Office PowerPoint 2013 (15.0.4797) Windows NT 6.2
  Microsoft Office PowerPoint 2013 (15.0.4805) Windows NT 6.1
  Microsoft Office PowerPoint 2013 (15.0.4805) Windows NT 6.2
  Microsoft Office PowerPoint 2013 (15.0.4815) Windows NT 10.0
  Microsoft Office PowerPoint 2014
  Microsoft Office Word 2013
  Microsoft Office Word 2013 (15.0.4623) Windows NT 6.2
  Microsoft Office Word 2013 (15.0.4701) Windows NT 6.1
  Microsoft Office Word 2013 (15.0.4797) Windows NT 10.0
  Microsoft Office Word 2013 (15.0.4805) Windows NT 10.0
  Microsoft Office Word 2013 (15.0.4805) Windows NT 6.1
  Microsoft Office Word 2013 (15.0.4823) Windows NT 10.0
  Microsoft Office Word 2014
  Melvil Rawi/1.0
  marketinggrader.com
  Mandrill-Webhook/1.0
  MailChimp.com
  MagpieRSS/0.72 (+http://magpierss.sf.net)
  http://www.mabontland.com
  lwp-trivial/1.33
  lwp-trivial/1.35
  lwp-trivial/1.36
  lwp-trivial/1.38
  lwp-trivial/1.41
  looksystems.net
  LinkExaminer/1.01 (Windows)
  VLC/2.2.4 LibVLC/2.2.4
  [email protected]
  Mozilla/5.0 (compatible; Jobrapido/1.1; +http://www.jobrapido.com)
  JoBo/1.x (http://www.matuschek.net/jobo.html)
  Jigsaw/2.3.0 W3C_CSS_Validator_JFouffa/2.0
  Mozilla/5.0 (compatible; IODC-Internet Engineering Odysseus 001; +http://iodc.co.uk)
  Mozilla/5.0 (compatible; IODC-Internet Engineering Odysseus 005; +http://iodc.co.uk)
  Mozilla/5.0 (compatible; IODC-Internet Engineering Odysseus 006; +http://iodc.co.uk)
  Mozilla/5.0 (compatible; IODC-Internet Engineering Odysseus 007; +http://iodc.co.uk)
  Mozilla/5.0 (compatible; IODC-Internet Engineering Odysseus 011; +http://iodc.co.uk)
  Mozilla/5.0 (compatible; IODC-Internet Engineering Odysseus 013; +http://iodc.co.uk)
  Mozilla/5.0 (compatible; IODC-Odysseus Survey 29691-100-070315021100-72; +http://iodc.co.uk)
  Mozilla/5.0 (compatible; IODC-Odysseus Survey 29692-100-060415034231-192; +http://iodc.co.uk)
  Mozilla/5.0 (compatible; IODC-Odysseus Survey 31193-100-080615161452-372; +http://iodc.co.uk)
  Mozilla/5.0 (compatible; IODC-Odysseus Survey 37044-100-110516191742-025; +https://iodc.co.uk)
  Mozilla/5.0 (compatible; IODC-Odysseus Survey 41437-100-090715214314-122; +http://iodc.co.uk)
  Integrity/6
  InfoWizards Reciprocal Link System PRO - (http://www.infowizards.com)
  InDesign%20CC/11300 CFNetwork/720.4.4 Darwin/14.4.0 (x86_64)
  inbound.li parser
  InAGist URL Resolver (http://inagist.com)
  igdeSpyder (compatible; igde.ru; +http://igde.ru/doc/tech.html)
  IDG Twitter Links Resolver
  Mozilla/4.5 (compatible; HTTrack 3.0x; Windows 98)
  Jersey/2.19 (HttpUrlConnection 1.7.0_85)
  httpunit/1.x
  httpssites_power/1 CFNetwork/758.3.15 Darwin/15.4.0
  httpscheck (unknown version) CFNetwork/760.5.1 Darwin/15.5.0 (x86_64)
  HttpComponents/1.1
  http_requester/0.1
  HTTP_Request2/@package_version@ (http://pear.php.net/package/http_request2) PHP/5.5.9-1ubuntu4.14
  HTTP_Request2/2.2.0 (http://pear.php.net/package/http_request2) PHP/5.3.29
  HTTP_Request2/2.2.1 (http://pear.php.net/package/http_request2) PHP/5.3.29
  HTTP_Request2/2.2.1 (http://pear.php.net/package/http_request2) PHP/5.5.33
  HTTP_Request2/2.2.1 (http://pear.php.net/package/http_request2) PHP/5.5.35
  HTTP_Request2/2.3.0 (http://pear.php.net/package/http_request2) PHP/5.3.29
  HTTP-Tiny/0.054
  http-kit/2.0
  HTMLParser/2.0
  htdig/3.1.5
  htdig/3.1.5 ([email protected])
  htdig/3.1.5 (root@localhost)
  htdig/3.1.5 ([email protected])
  htdig/3.1.6 ([email protected])
  htdig/3.1.6 ([email protected])
  ht://check/2.0.0-rc1 (Linux 2.6.32-573.18.1.el6.x86_64 x86_64)
  HEADMasterSEO
  Mozilla/4.0 (compatible; grub-client-0.3.0; Crawl your own stuff with http://grub.org)
  Mozilla/4.0 (compatible; grub-client-1.0.4; Crawl your own stuff with http://grub.org)
  Mozilla/4.0 (compatible; grub-client-1.0.5; Crawl your own stuff with http://grub.org)
  Mozilla/4.0 (compatible; grub-client-1.0.6; Crawl your own stuff with http://grub.org)
  Mozilla/4.0 (compatible; grub-client-1.0.7; Crawl your own stuff with http://grub.org)
  Mozilla/4.0 (compatible; grub-client-1.1.1; Crawl your own stuff with http://grub.org)
  Mozilla/4.0 (compatible; grub-client-1.2.1; Crawl your own stuff with http://grub.org)
  Mozilla/4.0 (compatible; grub-client-1.3.1; Crawl your own stuff with http://grub.org)
  Mozilla/4.0 (compatible; grub-client-1.3.7; Crawl your own stuff with http://grub.org)
  Mozilla/4.0 (compatible; grub-client-1.4.3; Crawl your own stuff with http://grub.org)
  Mozilla/4.0 (compatible; grub-client-1.5.3; Crawl your own stuff with http://grub.org)
  Mozilla/4.0 (compatible; grub-client-2.3)
  Mozilla/4.0 (compatible; grub-client-2.6.0)
  Mozilla/4.0 (compatible; grub-client-2.6.1)
  GoSpotCheck 1.0.6 (iPhone; iPhone OS 5.1.1; en_US)
  GoScraper
  GoogleProducer; (+http://goo.gl/7y4SX)
  GoogleHC/
  Google-HTTP-Java-Client/1.17.0-rc (gzip)
  Google-HTTP-Java-Client/1.9.0-alpha
  Mozilla/5.0 (compatible; Google-Apps-Script)
  Go-http-client/1.1
  Go 1.1 package http
  Mozilla/5.0 (Macintosh; Intel Mac OS X 10_6_6) AppleWebKit/534.24 (KHTML, like Gecko) (Contact: [email protected])
  Mozilla/5.0 (compatible; Funnelback) RPT-HTTPClient/0.3-3E
  Mozilla/5.0 (compatible; Snacktory; +https://flynxapp.com)
  Mozilla/3.0 (compatible; Fluffy the spider; http://www.searchhippo.com/; [email protected])
  Flamingo_SearchEngine (+http://www.flamingosearch.com/bot)
  Mozilla/4.0 (compatible; MSIE 5.5; Windows NT 5.0) Fetch API Request
  FeedChecker-Zocle/1.0 (+https://zocle.com/zoclechecker)
  FeedBurner/1.0 (http://www.FeedBurner.com)
  FavOrg
  Faraday v0.9.1
  Mozilla/5.0 (Macintosh; Intel Mac OS X) Excel/14.10.0
  Mozilla/5.0 (Macintosh; Intel Mac OS X) Excel/14.62.0
  ExactSearch
  EventMachine HttpClient
  Mozilla/5.0 (compatible; MSIE 7.0 +http://www.europarchive.org)
  Embed PHP Library
  ECCP/1.2.1+([email protected])
  ec2linkfinder
  Mozilla/5.0 (compatible; EARTHCOM/2.2; +http://enter4u.eu)
  Drupal (+http://drupal.org/)
  Dragonfly File Reader
  DNS-Tools Header-Analyzer
  Dispatch/0.11.1-SNAPSHOT
  [email protected]
  CyberPatrol SiteCat Webbot (http://www.cyberpatrol.com/cyberpatrolcrawler.asp)
  curb
  Mozilla/4.0 (compatible; cron-job.org; http://cron-job.org/abuse/)
  cosmos/0.9_([email protected])
  Mozilla/5.0/Firefox/42.0 (contactbigdatafr at gmail.com)
  Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; .NET CLR 2.0.50727; MAXTHON 2.0); Connect Us: [email protected]
  Pattern/2.6 +http://www.clips.ua.ac.be/pattern
  Mozilla/4.0 (compatible; Cerberian Drtrs Version-3.2-Build-0)
  Mozilla/4.0 (compatible; Cerberian Drtrs Version-3.2-Build-1)
  CC Metadata Scaper http://wiki.creativecommons.org/Metadata_Scraper
  cb crawl (+http://www.companybooknetworking.com)
  CapsuleChecker (http://www.capsulink.com/)
  CakePHP
  Branch-Passthrough
  Branch Metrics API
  boitho.com-dc/0.79 ( http://www.boitho.com/dcbot.html )
  boitho.com-dc/0.81 ( http://www.boitho.com/dcbot.html )
  boitho.com-dc/0.82 ( http://www.boitho.com/dcbot.html )
  boitho.com-dc/0.83 ( http://www.boitho.com/dcbot.html )
  boitho.com-dc/0.85 ( http://www.boitho.com/dcbot.html )
  boitho.com-robot/1.1
  Blackboard Safeassign
  Mozilla/5.0 (Windows NT 6.3; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/42.0.2311.152 Safari/537.36 (BingLocalSearch)
  BIGLOTRON (Beta 2;GNU/Linux)
  Mozilla/5.0 (compatible; bnf.fr_bot; +http://bibnum.bnf.fr/robot/bnf.html)
  Mozilla/4.0 (compatible; B-l-i-t-z-B-O-T)
  Astute SRM
  aria2/1.18.8
  Mozilla/5.0 (compatible; archive-com.com/1.1; +http://archive-com.com/bot)
  Mozilla/4.0 (compatible; Arachmo)
  Apache-HttpClient/4.2.3 (java 1.5)
  Apache-HttpClient/4.2.5 (java 1.5)
  Apache-HttpClient/4.3.1 (java 1.5)
  Apache-HttpClient/4.3.3 (java 1.5)
  Apache-HttpClient/4.3.5 (java 1.5)
  Apache-HttpClient/4.5 (Java/1.8.0_51)
  Apache-HttpClient/4.5 (Java/1.8.0_72)
  Apache-HttpClient/4.5 (Java/1.8.0_74)
  Apache-HttpClient/4.5.1 (Java/1.7.0_79)
  Apache-HttpClient/4.5.1 (Java/1.8.0_40)
  Apache-HttpClient/4.5.2 (Java/1.8.0_72)
  Apache-HttpClient/UNAVAILABLE (java 1.4)
  Mozilla/5.0 (compatible; U; AnyEvent-HTTP/2.22; +http://software.schmorp.de/pkg/AnyEvent)
  Mozilla/5.0 (compatible; U; AnyEvent-HTTP/2.23; +http://software.schmorp.de/pkg/AnyEvent)
  Anemone/0.7.2
  AndroidDownloadManager/6.0.1 (Linux; U; Android 6.0.1; SM-G900F Build/MMB29M)
  AndroidDownloadManager/4.4.2 (Linux; U; Android 4.4.2; SM-T310 Build/KOT49H)
  AndroidDownloadManager/4.4.2 (Linux; U; Android 4.4.2; SM-T210 Build/KOT49H)
  Airmail 1.3.3 rv:237 (Macintosh; Mac OS X 10.9.3; sk_SK)
  Python/3.5 aiohttp/0.21.6
  AHC/1.0
  Mozilla/4.0 (compatible; Advanced Email Extractor v2.61)
  Pulp/1.5.2 (iPad; http://www.acrylicapps.com/pulp/)
  Accoona-AI-Agent/1.1.1 (crawler at accoona dot com)
  Accoona-AI-Agent/1.1.2
  Accoona-AI-Agent/1.1.2 (aicrawler at accoonabot dot com)
  Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1) [email protected]
  scrutiny/6
  RMA/1.0 (compatible; RealMedia)
  PHP/5.5
  NING/1.0
  Mget/0.1.9
  Java/1.4.1_04
  Java/1.6.0
  Java/1.6.0_04
  Java/1.6.0_22
  Java/1.6.0_24
  Java/1.6.0_26
  Java/1.6.0_31
  Java/1.6.0_32
  Java/1.6.0_34
  Java/1.6.0_37
  Java/1.7.0_101
  Java/1.7.0_45
  Java/1.7.0_51
  Java/1.7.0_65
  Java/1.7.0_80
  Java/1.8.0_05
  Java/1.8.0_20
  Java/1.8.0_31
  Java/1.8.0_40
  Java/1.8.0_45
  Java/1.8.0_51
  Java/1.8.0_60
  Java/1.8.0_65
  Java/1.8.0_66
  Java/1.8.0_73
  Java/1.8.0_77
  Java/1.8.0_91
  Java/1.8.0_92
  Goose/1.0.25
  FDM 3.x
  bluefish 0.6 HTML editor
  2ip.ru CMS Detector (http://2ip.ru/cms/)
  masscan/1.0 (https://github.com/robertdavidgraham/masscan)
  VidibleScraper/1.0
  WhereGoes? Redirect Tracer/1.0
  unfurlist (https://github.com/Doist/unfurlist)
  pshtt, https scanning
  gofetch
  COMODO DCV
  CATExplorador/1.0beta (sistemes at domini dot cat; http://domini.cat/catexplorador.html)
  ZmEu
  RSSOwl/2.2.1.201312301314 (Windows; U; en)
  WebClient/1.0
  Microsoft-WebDAV-MiniRedir/10.0.10586
  Microsoft-WebDAV-MiniRedir/10.0.14393
  Microsoft-WebDAV-MiniRedir/5.1.2600
  Microsoft-WebDAV-MiniRedir/6.0.6001
  Microsoft-WebDAV-MiniRedir/6.1.7601
  Microsoft-WebDAV-MiniRedir/6.3.9600
  Microsoft Data Access Internet Publishing Provider Protocol Discovery
  Test Certificate Info
  Needle/1.4.2 (Node.js v6.9.2; linux x64)
  DavClnt
  Mozilla/5.0 (iPhone; CPU iPhone OS 6_0 like Mac OS X) AppleWebKit/536.26 (KHTML, like Gecko) Version/6.0 Mobile/10A5376e Safari/8536.25 mixed-content-scan
  Mozilla/5.0 (compatible; Semiocast HTTP client; http://semiocast.com/)
  Mozilla/5.0 (compatible; The Drop Reaper; +http://tag1consulting.com/dropreaper.html)
  /n software IPWorks HTTP/S Component - www.nsoftware.com
  safe-agent-scanner
  Mozilla/5.0 (compatible; ACHE/Unknown Version; +https://github.com/ViDA-NYU/ache; )
  ANAM (github.com/dutchcoders/anam)
  Dolphin http client/11.2.8(385) (Android)
  Dolphin http client/11.3.4(405) (Android)
  Calypso v/0.01
  -Mozilla/4.0 (compatible; http://search.thunderstone.com/texis/websearch/about.html)
  Mozilla/5.0 (compatible; Tweezler/2.0; http://tweezler.com)
  Jeode/1.x.x
  KeyCDN Perf Test
  Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) YLT Chrome/27.0.1453.110 Safari/537.36
  Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/51.0.2704.84 Safari/537.36 DMBrowser-UV
  Taringa UGC www.taringa.net/shared-link.php
  Mozilla/5.0 (Unknown; Linux x86_64) AppleWebKit/534.34 (KHTML, like Gecko) PingdomTMS/0.8.5 Safari/534.34
  ganarvisitas/1.0 (+http://www.ganarvisitas.com/)
  FeedFetcher-Google; (+http://www.google.com/feedfetcher.html)
  Mozilla/5.0 (compatible) Feedfetcher-Google; (+http://www.google.com/feedfetcher.html)
  Mozilla/5.0 (compatible; Exabot PyExalead/3.0; +http://www.exabot.com/go/robot)
  Mozilla/5.0 PhantomJS (compatible; Seznam screenshot-generator 2.1; +http://fulltext.sblog.cz/screenshot/)
  Mozilla/5.0 (compatible; SeznamBot/3.2-test4; +http://napoveda.seznam.cz/en/seznambot-intro/)
  Mozilla/5.0 (compatible; SeznamBot/3.2-test2; +http://napoveda.seznam.cz/en/seznambot-intro/)
  Mozilla/5.0 (compatible; SeznamBot/3.2-test1; +http://fulltext.sblog.cz/)
  Mozilla/5.0 (compatible; SeznamBot/3.2-test4; +http://fulltext.sblog.cz/)
  Mozilla/5.0 (compatible; SeznamBot/3.2; +http://fulltext.sblog.cz/)
  Seznam-Zbozi-robot/3.2.1
  SeznamBot/2.0 (+http://fulltext.sblog.cz/robot/)
  Seznam-Zbozi-robot/3.2.2
  Mozilla/5.0 (compatible; SeznamBot/3.2-test2; +http://fulltext.sblog.cz/)
  Mozilla/5.0 (compatible; SeznamBot/3.2-test3; +http://fulltext.sblog.cz/)
  Seznam-Zbozi-robot/3.3
  SklikBot/2.0 ([email protected];+http://napoveda.sklik.cz/)
  SeznamBot/3.0 (+http://fulltext.sblog.cz/)
  Mozilla/5.0 (compatible; SeznamBot/3.1-test1; +http://fulltext.sblog.cz/)
  Mozilla/5.0 (Linux; U; Android 4.1.2; cs-cz; Seznam screenshot-generator Build/Q3) AppleWebKit/534.30 (KHTML, like Gecko) Version/4.0 Mobile Safari/534.30
  SeznamBot/3.0-test (+http://fulltext.sblog.cz/), I
  SeznamBot/3.0 (HaF+http://fulltext.sblog.cz/)
  SeznamBot/3.0-test (+http://fulltext.sblog.cz/)
  SeznamBot/3.0-beta (+http://fulltext.sblog.cz/), I
  SeznamBot/3.0-beta (+http://fulltext.sblog.cz/)
  SeznamBot/3.0-alpha (+http://fulltext.sblog.cz/)
  SeznamBot/2.0 (+http://fulltext.seznam.cz/)
  SeznamBot/2.0-Test (+http://fulltext.sblog.cz/robot/)
  Mozilla/5.0 (compatible; Seznam screenshot-generator 2.0; +http://fulltext.sblog.cz/screenshot/)
  Mozilla/5.0 (compatible; FlipboardRSS/1.2; +http://flipboard.com/browserproxy)
  Mozilla/5.0 (Macintosh; Intel Mac OS X 10.11; rv:49.0) Gecko/20100101 Firefox/49.0 (FlipboardProxy/1.2; +http://flipboard.com/browserproxy)
  Mozilla/5.0 (compatible; MJ12bot/v1.4.7; http://mj12bot.com/)
  Mozilla/5.0 (Macintosh; Intel Mac OS X 10_7_2) AppleWebKit/535.7 (KHTML, like Gecko) Chrome/16.0.912.77 Safari/535.7 AppEngine-Google; (+http://code.google.com/appengine; appid: s~feedly-social)
  Mozilla/5.0 (Macintosh; Intel Mac OS X 10_7_2) AppleWebKit/535.7 (KHTML, like Gecko) Chrome/23.0.912.77 Safari/535.7 AppEngine-Google; (+http://code.google.com/appengine; appid: s~feedly-nikon3)
  Mozilla/5.0 (compatible; SemrushBot/1.2~bl; +http://www.semrush.com/bot.html)
  Mozilla/5.0 (compatible; YandexMetrika/2.0; +http://yandex.com/bots mtmon01it.mtrs.yandex.ru)
  Mozilla/5.0 (iPhone; CPU iPhone OS 8_1 like Mac OS X) AppleWebKit/600.1.4 (KHTML, like Gecko) Version/8.0 Mobile/12B411 Safari/600.1.4 (compatible; YandexBot/3.0; +http://yandex.com/bots)
  Mozilla/5.0 (compatible; memoryBot/1.24.61 +http://internetmemory.org/en/)
  Mozilla/5.0 (compatible; memoryBot/1.24.54 +http://internetmemory.org/en/)
  BUbiNG - Research at Brno University of Technology - KNOT group - http://knot.fit.vutbr.cz/crawling/ - Stop? http://law.di.unimi.it/BUbiNG.html
  Mozilla/5.0 (compatible; MSIE 9.0; Firefox/50.1) Daum/4.1
  Mozilla/5.0 (Unknown; Linux x86_64) AppleWebKit/534.34 (KHTML, like Gecko) PhantomJS/1.9.7 Daum Raam Edition b20140820 Safari/534.34
  Sogou Pic Spider/3.0(+http://www.sogou.com/docs/help/webmasters.htm#07)
  yacybot (/global; amd64 Linux 4.4.0-59-generic; java 1.8.0_121; Europe/de) http://yacy.net/bot.html
  yacybot (-global; amd64 Linux 4.4.0-57-generic; java 9-internal; Europe/en) http://yacy.net/bot.html
  yacybot (/global; amd64 Windows 8.1 6.3; java 1.8.0_111; Europe/de) http://yacy.net/bot.html
  yacybot (/global; amd64 Linux 3.16.0-4-amd64; java 1.8.0_111; Europe/en) http://yacy.net/bot.html
  yacybot (/global; amd64 FreeBSD 10.3-RELEASE-p7; java 1.7.0_95; GMT/en) http://yacy.net/bot.html
  yacybot (webportal-global; amd64 Linux 4.4.0-46-generic; java 1.8.0_91; Europe/en) http://yacy.net/bot.html
  yacybot (-global; amd64 Linux 3.14.32-xxxx-grs-ipv6-64; java 1.8.0_111; Europe/de) http://yacy.net/bot.html
  Mozilla/5.0 (compatible; special_archiver/3.3.0 [email protected] +https://archive.org/details/archive.org_bot)
  Mozilla/5.0 (compatible; heritrix/3.3.0-SNAPSHOT-20140702-2247 +http://archive.org/details/archive.org_bot)
  Mozilla/5.0 (compatible; archive.org_bot/heritrix-1.15.4 +http://www.archive.org)
  SeobilityBot (SEO-Check; http://bit.ly/1dJuuzs)
  Testomatobot/1.0 (Linux x86_64; +http://www.testomato.com/testomatobot) minicrawler/5.0.4
  Testomatobot/1.0 (Linux x86_64; +http://www.testomato.com/testomatobot) minicrawler/5.0.3
  Testomatobot/1.0 (Linux x86_64; +http://www.testomato.com/testomatobot) minicrawler/5.0.2
  Testomatobot/1.0 (Linux x86_64; +http://www.testomato.com/testomatobot) minicrawler/5.0.1
  Mozilla/5.0 (compatible; AhrefsBot/5.2; +http://ahrefs.com/robot/)
  Heurekabot-Feed/1.0 (+https://sluzby.heureka.cz/napoveda/heurekabot/)
  Heurekabot-ImageFullText/1.0 (+https://sluzby.heureka.cz/napoveda/heurekabot/)
  Heurekabot-ImageFullText/1.0 (+http://sluzby.heureka.cz/napoveda/heurekabot/)
  Mozilla/5.0 (Windows NT 6.2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/31.0.1650.63 Safari/537.36 QIHU 360SE; 360Spider
  ZoomBot (Linkbot 1.0 http://suite.seozoom.it/bot.html)
  Mozilla/5.0 (compatible; SOLOFIELD/1.0 +http://solofield.net/bot.html)
  Qwantify/1.0
  Mozilla/5.0 (compatible; Yeti/1.1; +http://naver.me/bot)
  Scrapy/1.3.0 (+http://scrapy.org)
  Scrapy/1.0.5.post4+g4b324a8 (+http://scrapy.org)
  Scrapy/1.2.1 (+http://scrapy.org)
  Scrapy/0.16.5 (+http://scrapy.org)
  Mozilla/5.0 (compatible; Dataprovider.com;)
  Mozilla/5.0 (compatible; Dataprovider; https://www.dataprovider.com/spider/)
  Mozilla/5.0 (compatible; SiteExplorer/1.1b; +http://siteexplorer.info/Backlink-Checker-Spider/)
  ICC-Crawler/2.0 (Mozilla-compatible; ; http://ucri.nict.go.jp/en/icccrawler.html)
  DuckDuckBot/1.0; (+http://duckduckgo.com/duckduckbot.html)
  Mozilla/5.0 (compatible; SMTBot/1.0; http://www.similartech.com/smtbot)
  Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/538.1 (KHTML, like Gecko) Chromium/31.0.1650.63 Site-Shot/2.1 (http://www.site-shot.com/) Safari/538.1
  Mozilla/5.0 (Windows NT 6.1; WOW64; Neustar WPM) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/48.0.2564.82 Safari/537.36
  Mozilla/5.0 (iPad; CPU OS 6_1_2 like Mac OS X; Neustar WPM) AppleWebKit/536.26 (KHTML, like Gecko) Version/6.0 Mobile/10A5376e Safari/8536.25
  Mozilla/5.0 (Windows NT 6.0; Neustar WPM) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/48.0.2564.82 Safari/537.36
  Mozilla/5.0 (Windows NT 6.1; Neustar WPM) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/48.0.2564.82 Safari/537.36
  Mozilla/5.0 (Windows NT 6.0; rv:38.0; Neustar WPM) Gecko/20100101 Firefox/38.0
  Mozilla/5.0 (Windows NT 6.1; rv:38.0; Neustar WPM) Gecko/20100101 Firefox/38.0
  Mozilla/5.0 (Windows NT 6.1; Neustar WPM) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/52.0.2743.82 Safari/537.36
  Mozilla/5.0 (Linux; Android 4.1.1; HTC6435LVW Build/JRO03C; Neustar WPM) AppleWebKit/537.22 (KHTML, like Gecko) Chrome/25.0.1364.169 Mobile Safari/537.22
  Mozilla/5.0 (compatible; Whoiswebsitebot/0.1; +http://www.whoiswebsite.net)
  Mozilla/5.0 (Linux; Android 5.0.2; SM-G920T Build/LRX22G) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/54.0.2840 Mobile Safari/537.36 DareBoost
  Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/54.0.2840 Safari/537.36 DareBoost
  Mozilla/5.0 (Linux; Android 5.0; LG-D855 Build/LRX21R.A1422018487) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/54.0.2840 Mobile Safari/537.36 DareBoost
  Mozilla/5.0 (iPhone; CPU iPhone OS 10_0_1 like Mac OS X) AppleWebKit/602.1.50 (KHTML, like Gecko) Version/10.0 Mobile/14A403 Safari/602.1 DareBoost
  Mozilla/5.0 (Linux; Android 4.4.3; HTC One Build/KTU84L) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/54.0.2840 Mobile Safari/537.36 DareBoost
  Mozilla/5.0 (Linux; Android 4.4; Nexus 5 Build/KRT16M) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/54.0.2840 Mobile Safari/537.36 DareBoost
  HubSpot Website Grader ([email protected])
  HubSpot Marketing Grader, HubSpot Marketing Grader
  com.hubspot.spooks.core.UserAgent@753535c0, HubSpot Marketing Grader
  Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/51.0.2704.79 Safari/537.36 (http://www.shrinktheweb.com)
  scrapy-redis (+https://github.com/rolando/scrapy-redis)
  Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/55.0.2883.87 Safari/537.36 PTST/348
  Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/54.0.2840.99 Safari/537.36 PTST/345
  TeamSpeak3-ImageFetcher-1.0
  wf_crawler (http://www.websitefigures.com)
  wf_crawler (http://websitefigures.com)
  Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/55.0.2883.87 Safari/537.36 TinEye/1.0 (via http://www.tineye.com/)
  TinEye-bot/0.61 (see http://www.tineye.com/crawler.html)
  Mozilla/5.0 (compatible; Googlebot/2.1; https://deepcrawl.com/bot)
  Googlebot deepcrawl
  Mozilla/5.0 (compatible; evc-batch/2.0.20170110125844)
  Mozilla/5.0 (compatible; evc-batch/2.0.20161220184139)
  Mozilla/5.0 (compatible; evc-batch/2.0.20161122223033)
  Mozilla/5.0 (compatible; evc-batch/2.0.20161026010401)
  Mozilla/5.0 (X11; Linux x86_64; rv:49.0; GTmetrix https://gtmetrix.com/) Gecko/20100101 Firefox/49.0
  Mozilla/5.0 (Linux; Android 4.3; Galaxy Nexus Build/JWR66Y; GTmetrix https://gtmetrix.com/) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/54.0.2840.68 Mobile Safari/537.36
  Mozilla/5.0 (X11; Linux x86_64; GTmetrix https://gtmetrix.com/) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/54.0.2840.71 Safari/537.36
  Mediumbot-MetaTagFetcher/0.3 (+https://medium.com/)
  Mediumbot-ProxyFetcher/0.1 (+https://medium.com/)
  YOOBI.de - WebBot 1.0
  YOOBI - WebBot 1.0
  Mozilla/5.0 (compatible; heritrix/3.3.0-SNAPSHOT-20160309-0050; UniLeipzigASV +http://corpora.informatik.uni-leipzig.de/crawler_faq.html)
  IAS crawler (page scorer; http://integralads.com/site-indexing-policy/)
  Mozilla/5.0 (compatible; JobKeresoBot; +https://www.kozvetlen-allasok.hu/help.jsp; [email protected]; 5.0
  Mozilla/5.0 (compatible; JobKereso; +https://www.kozvetlen-allasok.hu/robot.jsp [email protected])
  pyspider/0.3.9 (+http://pyspider.org/)
  Mozilla/5.0 (X11; U; Linux x86_64; de-DE; rv:1.9.0.19) Gecko/2010120923 ThumbShotsBot (KFSW 3.0.6-3)
  Mozilla/5.0 (compatible; Discordbot/1.0; +https://discordapp.com)
  Mozilla/5.0 (compatible; Dead Link Checker; http://www.dead-link-checker.com/)
  Mozilla/5.0 (compatible; MSIE 10.0; Windows NT 6.2; WOW64; Trident/6.0) CrawlerProcess (http://www.PowerMapper.com) /5.24.776.0
  Mozilla/4.0 (compatible; BOTW Spider; +http://botw.org)
  TwengaBot-2.0
  Iframely/1.0.0 (+https://iframely.com/;)
  Iframely/0.8.5 (+http://iframely.com/;)
  rogerbot/1.2 (http://moz.com/help/pro/what-is-rogerbot-, [email protected])
  rogerbot/1.0 (http://moz.com/help/pro/what-is-rogerbot-, [email protected]
  Mozilla/5.0 (compatible; Sitemap Generator/1.3; http://www.check-domains.com/sitemap/index.php) Gecko Check-domains)
  mindUpBot
  Mozilla/5.0 (Windows NT 6.1; compatible; BDCbot/1.0; +http://bigweb.bigdatacorp.com.br/faq.aspx) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2272.118 Safari/537.36
  Contacts Crawler (+http://www.scrapinghub.com)
  TA SEO Crawler v0.1 URLData.pm ([email protected])
  Mozilla/5.0 (compatible; scanbot/1.0; +http://dazzlepod.com/ip/)
  ArielisBot/1.1 (+http://arielis.com/
  RedesScrapy/0.24.1 (+http://g2pi.tsc.uc3m.es/es)
  Blackboard Safeassign/0.1 (a Storm-based Blackboard Safeassign web-crawler; https://github.com/DigitalPebble/storm-crawler; [email protected])
  Mozilla/5.0 (Macintosh; Intel Mac OS X 10.11; rv:48.0) Gecko/20100101 Firefox/48.0 glindahl-cocrawler/0.01 (+http://www.pbm.com/~lindahl/glindahl-cocrawler.html)
  botrobin/Nutch-1.13-SNAPSHOT (http://smarter.codes/bot-robin/; [email protected])
  Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/51.0.2704.63 Safari/537.36 (compatible; aylienbot/0.2; +http://www.aylien.com/bot.html)
  GloomarBot/1.1 (https://www.gloomar.com/bot)
  Hatena::Bookmark/2.00 (Hatena::Bookmark; master;)
  Opendi Screenshot Bot [email protected]
  Mozilla/5.0 (compatible; YaK/1.0; http://linkfluence.com/; [email protected])
  Mozilla/5.0 (compatible; KAZ.KZ_Bot/3.0)
  BuckyOHare/1.3 (Googlebot/2.1; +https://hypefactors.com/webcrawler)
  uipbot/1.0 ([email protected])
  gMozilla/5.0 (compatible; Infohelfer/1.4.3; +http://www.infohelfer.de/crawler.php)
  Mozilla/5.0 (compatible; lincobot/0.3; http://www.linksco.com/about/robot.html) AppEngine-Google; (+http://code.google.com/appengine)
  Mozilla/5.0 (compatible; MSIE 10.0; Windows NT 6.2; WOW64; Trident/6.0) MapperCmd/5.24.780.0
  Seeker v.1 http://LookSeek.com/seeker/
  AppManager RPT-HTTPClient/0.3-3E
  LinkAlarm/3.1
  Mozilla/5.0 (X11; U; Linux i686) Firefox/2.0.0.11 wprecon.com survey
  Jeffrey's Exif Viewer (http://regex.info/exif)
  Untiny
  Mozilla/5.0 (Grabify.link - https://grabify.link/expander)
  biNu image cacher ([email protected])
  Mozilla/5.0 (compatible; Google-Youtube-Links)
  ping.blo.gs/2.0
  Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; QQDownload 1.7; .NET CLR 2.0.50727)
  Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.1; Trident/4.0; QQDownload 663; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; .NET4.0C; .NET CLR 1.1.4322)
  Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 5.1; Trident/4.0; QQDownload 760; .NET CLR 2.0.50727; .NET CLR 3.0.4506.2152; .NET CLR 3.5.30729; InfoPath.2; .NET4.0E; .NET4.0C; .NET CLR 1.1.4322)
  Mozilla/5.0 (X11; Linux x86_64; rv:52.0.2) Gecko/20100101 Firefox/52.0.2 Ghost Inspector
  Mozilla/5.0 (X11; Linux x86_64; rv:52.0) Gecko/20100101 Firefox/52.0 Ghost Inspector
  Mozilla/5.0 (X11; Linux x86_64; rv:50.1.0) Gecko/20100101 Firefox/50.1.0 Ghost Inspector
  Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/538.1 (KHTML, like Gecko) QtWebEngine/5.3.0 Safari/538.1 Ghost Inspector
  Mozilla/5.0 (X11; Linux x86_64; rv:45.0.1) Gecko/20100101 Firefox/45.0.1 Ghost Inspector
  Mozilla/5.0 (X11; Linux x86_64; rv:46.0.1) Gecko/20100101 Firefox/46.0.1 Ghost Inspector
  Mozilla/5.0 (X11; Linux x86_64; rv:47.0.2) Gecko/20100101 Firefox/47.0.2 Ghost Inspector
  Mozilla/5.0 (X11; Linux x86_64; rv:48.0.2) Gecko/20100101 Firefox/48.0.2 Ghost Inspector
  Mozilla/5.0 (X11; Linux x86_64; rv:49.0.2) Gecko/20100101 Firefox/49.0.2 Ghost Inspector
  Netpursual/1.0
  Kaspersky Lab CFR link resolver [email protected]
  Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/27.0.1453 Safari/537.36 (compatible; Google-HotelAdsVerifier/2.0)
  b0t
  "echocrawl 2.0"
  masscan/1.0
  Mozilla/5.0 (iPhone; CPU iPhone OS 8_1 like Mac OS X) AppleWebKit/600.1.4 (KHTML, like Gecko) Version/8.0 Mobile/12B411 Safari/600.1.4 (compatible; YandexBot/3.0; +http://yandex.com/bots)
  Mozilla/5.0 (compatible; YandexAccessibilityBot/3.0; +http://yandex.com/bots)
  Mozilla/5.0 (iPhone; CPU iPhone OS 8_1 like Mac OS X) AppleWebKit/600.1.4 (KHTML, like Gecko) Version/8.0 Mobile/12B411 Safari/600.1.4 (compatible; YandexMobileBot/3.0; +http://yandex.com/bots)
  Mozilla/5.0 (compatible; YandexDirectDyn/1.0; +http://yandex.com/bots
  Mozilla/5.0 (compatible; YandexImages/3.0; +http://yandex.com/bots)
  Mozilla/5.0 (compatible; YandexVideo/3.0; +http://yandex.com/bots)
  Mozilla/5.0 (compatible; YandexMedia/3.0; +http://yandex.com/bots)
  Mozilla/5.0 (compatible; YandexBlogs/0.99; robot; +http://yandex.com/bots)
  Mozilla/5.0 (compatible; YandexFavicons/1.0; +http://yandex.com/bots)
  Mozilla/5.0 (compatible; YandexWebmaster/2.0; +http://yandex.com/bots)
  Mozilla/5.0 (compatible; YandexPagechecker/1.0; +http://yandex.com/bots)
  Mozilla/5.0 (compatible; YandexImageResizer/2.0; +http://yandex.com/bots)
  Mozilla/5.0 (compatible; YaDirectFetcher/1.0; Dyatel; +http://yandex.com/bots)
  Mozilla/5.0 (compatible; YandexCalendar/1.0; +http://yandex.com/bots)
  Mozilla/5.0 (compatible; YandexSitelinks; Dyatel; +http://yandex.com/bots)
  Mozilla/5.0 (compatible; YandexMetrika/3.0; +http://yandex.com/bots)
  Mozilla/5.0 (compatible; YandexAntivirus/2.0; +http://yandex.com/bots)
  Mozilla/5.0 (compatible; YandexVertis/3.0; +http://yandex.com/bots)
  Mozilla/5.0 (compatible; YandexBot/3.0; MirrorDetector; +http://yandex.com/bots)
  jetmon/1.0 (Jetpack Site Uptime Monitor by WordPress.com)
  Mozilla/5.0 (compatible; Abonti/0.8 - http://www.abonti.com)
  CloudEndure Scanner ([email protected])
  AFB/3.0 (+http://allloadin.com)
  TLSProbe/1.0 (+https://scan.trustnet.venafi.com/)
  Sens.ai http://sovereign.ai/contact
  eContext/1.0 (eContext Classification Engine)
  Owler
  Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Slack/1.2.6 Chrome/45.0.2454.85 AtomShell/0.34.3 Safari/537.36 Slack_SSB/1.2.6
  Mozilla/5.0 (compatible; ChangesMeter/1.9.1; http://intuiware.com/apps/changes-meter)
  Mozilla/5.0 (compatible; Gookey.co/1.0; +http://gookey.co/)
  TrapitAgent/0.1 (feed processor; +http://trapit.com/about)
  sqlmap/1.0-dev-nongit-201612050a8c (http://sqlmap.org)
  UniversalFeedParser/3.3 +http://feedparser.org/
  TulipChain/5.xx (http://ostermiller.org/tulipchain/) Java/1.x.1_0x (http://apple.com/) Mac_OS_X/10.2.8
  scalaj-http/1.0
  HggH PhantomJS Screenshoter
  fasthttp
  fasthttp, Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/53.0.2785.101 Safari/537.36
  lua-resty-http/0.10 (Lua) ngx_lua/10000
  GIS-LABS:CertMon
  GuzzleHttp/6.2.1 PHP/7.0.15-0ubuntu0.16.04.4
  http-request/v0.7.0 (http://git.io/tl_S2w) node.js/v0.10.29
  intraVnews/1.x
  MailChimp
  G-i-g-a-b-o-t
  Symfony BrowserKit
  VCPP71 http-get 1.0a
  Jaunt/1.2
  Mozilla/5.0 (compatible; Owler/0.4; +; )
  MovableType/x.x
  OMSC
  websitepulse agent/0.1 (compatible; MSIE 5.5; Netscape 4.75; Linux)
  Mozilla/5.0 (compatible; Uptimia; www.uptimia.com)
  ssl-tools.net
  Mozilla/5.0 (compatible; MSIE 10.0; Windows NT 6.1; Trident/6.0; .NET CLR 1.1.4322; .NET CLR 1.0.3705; .NET CLR 2.0.50727; .NET CLR 3.0.04506.648; .NET CLR 3.5.21022; .NET CLR 3.0.4506.2152; .NET CLR 3.5.30729) DMBrowser/2.1
  keycdn-tools/perf
  keycdn-tools/br
  keycdn-tools/h2
  keycdn-tools/perf
  keycdn-tools/br
  keycdn-tools/h2
  KOCMOHABT (+https://kozmonavt.tk/) Mozilla/5.0 (compatible; Web Explorer 2)
  Donuts Content Explorer (www.donuts.domains)
  Netsparker
  Mozilla/5.0+(compatible; MxToolbox/Beta7; http://www.mxtoolbox.com/)
  Apache-HttpAsyncClient/4.1.3 (Java/1.8.0_121)
  Tenon.io Automated Web Accessibility Tester
  ApacheBench/2.3
  Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; Clarsentia)
  Mozilla/5.0 (compatible; historious/2.0; +http://historio.us/)
  Mozilla/5.0 (compatible; uCrawlr/1.0 ; +https://blog.ucoz.ru/upolicy)
  Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)
  Brodie/1.0 (+http://www.15miles.com/)
  Jobg8 URL Monitor
  Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.21 (KHTML, like Gecko) wpif Safari/537.21
  Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.21 (KHTML, like Gecko) imgsizer Safari/537.21
  Mozilla/5.0 (compatible; AppBeat/1.0; +http://www.appbeat.io/)
  Mozilla/5.0 (compatible; Chirp/1.0; +http://www.binarycanary.com)
  Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; WOW64; Trident/5.0; SiteMonitor)
  Mixnode/3.0.1 (+https://www.mixnode.com/; Speed analysis; no data is stored)
  Pingability.com Monitoring Service

Тоже, но с учетом регулярных выражений:

Спойлер

.*Java.*outbrain

008\/

192\.comAgent

2ip\.ru

404checker

^bluefish

^Calypso v\/

^COMODO DCV

^DangDang

^DavClnt

^FDM

^git\/

^Goose\/

^HTTPClient\/

^Java\/

^Jeode\/

^Jetty\/

^Mget

^Microsoft URL Control

^NG\/[0-9\.]

^NING\/

^PHP\/[0-9]

^RMA\/

^Ruby|Ruby\/[0-9]

^scrutiny\/

^VSE\/[0-9]

^WordPress\.com

^XRL\/[0-9]

^ZmEu

a3logics\.in

A6-Indexer

a\.pr-cy\.ru

Abonti\/

Aboundex

aboutthedomain

Accoona-AI-Agent

acoon

acrylicapps\.com\/pulp

adbeat

AddThis

ADmantX

adressendeutschland

Advanced Email Extractor v

agentslug

AHC

aihit

aiohttp\/

Airmail

akula\/

alertra

alexa site audit

Alibaba\.Security\.Heimdall

allloadin\.com

alyze\.info

amagit

AndroidDownloadManager

Anemone

Ant\.com

Anturis Agent

AnyEvent-HTTP\/

ApacheBench\/

Apache-HttpAsyncClient\/

Apache-HttpClient\/

AportWorm\/[0-9]

AppBeat\/[0-9]

AppEngine-Google

Arachmo

arachnode

Arachnophilia

aria2

asafaweb.com

AskQuickly

Astute

asynchttp

autocite

Autonomy

B-l-i-t-z-B-O-T

^b0t$

Backlink-Ceck\.de

Bad-Neighborhood

baidu\.com

baypup\/[0-9]

baypup\/colbert

BazQux

BCKLINKS

BDFetch

BegunAdvertising\/

BigBozz

biglotron

BingLocalSearch

BingPreview

binlar

biNu image cacher

biz_Directory

Blackboard Safeassign

Bloglovin

BlogPulseLive

BlogSearch

Blogtrottr

boitho\.com-dc

BPImageWalker

Braintree-Webhooks

Branch Metrics API

Branch-Passthrough

Brodie\/

Browsershots

BUbiNG

Butterfly\/

BuzzSumo

CAAM\/[0-9]

CakePHP

CapsuleChecker

CaretNail

catexplorador

cb crawl

CC Metadata Scaper

Cerberian Drtrs

CERT\.at-Statistics-Survey

cg-eye

changedetection

ChangesMeter\/

Charlotte

CheckHost

checkprivacy

chkme\.com

Chirp\/[0-9]

CirrusExplorer\/

CISPA Vulnerability Notification

CJNetworkQuality

Clarsentia

clips\.ua\.ac\.be

Cloud mapping experiment

CloudEndure

CloudFlare-AlwaysOnline

Cloudinary\/[0-9]

cmcm\.com

coccoc

CommaFeed

Commons-HttpClient

Comodo SSL Checker

contactbigdatafr

convera

copyright sheriff

Covario-IDS

CrawlForMe\/[0-9]

cron-job\.org

Crowsnest

curb

Curious George

curl

cuwhois\/[0-9]

cybo\.com

DareBoost

DataparkSearch

dataprovider

Daum(oa)?[ \/][0-9]

DeuSu

developers\.google\.com\/\+\/web\/snippet\/

Digg

Dispatch\/

dlvr

DMBrowser

DNS-Tools Header-Analyzer

DNSPod-reporting

docoloc

Dolphin http client\/

DomainAppender

Donuts Content Explorer

dotSemantic

downforeveryoneorjustme

downnotifier\.com

DowntimeDetector

Dragonfly File Reader

drupact

Drupal \(\+http:\/\/drupal\.org\/\)

dubaiindex

EARTHCOM

Easy-Thumb

ec2linkfinder

eCairn-Grabber

ECCP

echocrawl

eContext\/

ElectricMonk

elefent

EMail Exractor

EmailWolf

Embed PHP Library

Embedly

europarchive\.org

evc-batch\/[0-9]

EventMachine HttpClient

Evidon

Evrinid

ExactSearch

ExaleadCloudview

Excel\/

Exif Viewer

Exploratodo

ezooms

facebookexternalhit

facebookplatform

fairshare

Faraday v

fasthttp

Faveeo

Favicon downloader

FavOrg

Feed Wrangler

Feedbin

FeedBooster

FeedBucket

FeedBunch\/[0-9]

FeedBurner

FeedChecker

Feedly

Feedspot

Feedwind\/[0-9]

feeltiptop

Fetch API

Fetch\/[0-9]

Fever\/[0-9]

findlink

findthatfile

FlipboardBrowserProxy

FlipboardProxy

FlipboardRSS

fluffy

flynxapp

forensiq

FoundSeoTool\/[0-9]

free thumbnails

FreeWebMonitoring SiteChecker

Funnelback

G-i-g-a-b-o-t

g00g1e\.net

GAChecker

ganarvisitas\/[0-9]

geek-tools

Genderanalyzer

Genieo

GentleSource

GetLinkInfo

getprismatic\.com

GetURLInfo\/[0-9]

Ghost Inspector

GigablastOpenSource

GIS-LABS

github\.com\/

Go [\d\.]* package http

Go-http-client

gofetch

GomezAgent

gooblog

Goodzer\/[0-9]

Google favicon

Google Keyword Suggestion

Google Keyword Tool

Google Page Speed Insights

Google PP Default

Google Search Console

Google Web Preview

Google-Adwords

Google-Apps-Script

Google-Calendar-Importer

Google-HotelAdsVerifier

Google-HTTP-Java-Client

Google-Publisher-Plugin

Google-SearchByImage

Google-Site-Verification

Google-Structured-Data-Testing-Tool

Google-Youtube-Links

google_partner_monitoring

GoogleDocs

GoogleHC\/

GoogleProducer

Gookey

GoScraper

GoSpotCheck

GoSquared-Status-Checker

gosquared-thumbnailer

grabify

Grammarly

grouphigh

grub-client

GTmetrix

GuzzleHttp

gvfs\/

HAA(A)?RTLAND http client

Hatena

hawkReader

HEADMasterSEO

HeartRails_Capture

heritrix

historious\/

hledejLevne\.cz\/[0-9]

Holmes

HootSuite Image proxy

Hootsuite-WebFeed\/[0-9]

HostTracker

ht:\/\/check

htdig

HTMLParser\/

http-get

HTTP-Header-Abfrage

http-kit

http-request\/

HTTP-Tiny

HTTP_Compression_Test

http_request2

http_requester

HttpComponents

httphr

HTTPMon

PEAR HTTPRequest

httpscheck

httpssites_power

httpunit

HttpUrlConnection

httrack

hosterstats

huaweisymantec

HubPages.*crawlingpolicy

HubSpot Connect

HubSpot Marketing Grader

HyperZbozi.cz Feeder

i2kconnect\/

ichiro

IdeelaborPlagiaat

IDG Twitter Links Resolver

IDwhois\/[0-9]

Iframely

igdeSpyder

IlTrovatore

ImageEngine\/

Imagga

imgsizer

InAGist

inbound\.li parser

InDesign%20CC

infegy

infohelfer

InfoWizards Reciprocal Link System PRO

Instapaper

inpwrd\.com

Integrity

integromedb

internet_archive

InternetSeer

internetVista monitor

intraVnews

IODC

IOI

iplabel

IPS\/[0-9]

ips-agent

IPWorks HTTP\/S Component

iqdb\/

Irokez

isitup\.org

iskanie

iZSearch

janforman

Jaunt\/

Jigsaw

Jobboerse

Jobg8 URL Monitor

jobo

Jobrapido

JS-Kit

Kaspersky Lab CFR link resolver

KeepRight OpenStreetMap Checker

KeyCDN

Keywords Research

KickFire

KimonoLabs\/

Kml-Google

knows\.is

KOCMOHABT

kouio

kulturarw3

KumKie

L\.webis

Larbin

LayeredExtractor

letsencrypt

LibVLC

libwww

Licorne Image Snapshot

Liferea\/

link checker

Link Valet

link_thumbnailer

LinkAlarm\/

linkCheck

linkdex

LinkExaminer

linkfluence

linkpeek

LinkTiger

LinkWalker

Lipperhey

livedoor ScreenShot

LoadImpactPageAnalyzer

LoadImpactRload

LongURL API

looksystems\.net

ltx71

lua-resty-http

lwp-trivial

lycos

LYT\.SR

mabontland

MagpieRSS

Mail.Ru

MailChimp

Mandrill

MapperCmd

marketinggrader

masscan\/[0-9]

Mediapartners-Google

MegaIndex\.ru

Melvil Rawi\/

MergeFlow-PageReader

Metaspinner

MetaURI

Microsearch

Microsoft-WebDAV-MiniRedir

Microsoft Data Access Internet Publishing Provider Protocol

Microsoft Office

Microsoft Windows Network Diagnostics

Mindjet

Miniflux

mixdata dot com

mixed-content-scan

mixnode

Mnogosearch

mogimogi

Mojolicious \(Perl\)

monitis

Monitority\/[0-9]

montastic

MonTools

Moreover

Morning Paper

mowser

MovableType

Mrcgiguy

mShots

MxToolbox\/

MVAClient

nagios

Najdi\.si\/

Needle\/

NETCRAFT

NetLyzer FastProbe

Netpursual

netresearch

NetShelter ContentScan

Netsparker

NetTrack

Netvibes

Neustar WPM

NeutrinoAPI

NewsBlur .*Finder

NewsGator

newsme

newspaper\/

NG-Search

nineconnections\.com

NLNZ_IAHarvester

Nmap Scripting Engine

node-superagent

node\.io

nominet\.org\.uk

Norton-Safeweb

Notifixious

notifyninja

nuhk

nutch

Nuzzel

nWormFeedFinder

Nymesis

Ocelli\/[0-9]

oegp

okhttp

Omea Reader

omgili

OMSC

Online Domain Tools

OpenCalaisSemanticProxy

Openstat\/

OpenVAS

Optimizer

Orbiter

OrgProbe\/[0-9]

Owler

ow\.ly

ownCloud News

OxfordCloudService\/[0-9]

Page Analyzer

Page Valet

page2rss

page_verifier

PagePeeker

Pagespeed\/[0-9]

Panopta

panscient

parsijoo

PayPal IPN

Pcore-HTTP

Pearltrees

peerindex

Peew

PhantomJS\/

PhantomJS Screenshoter

Photon\/

phpcrawl

phpservermon

Pi-Monster

ping\.blo\.gs\/

Pingability

Pingdom

Pingoscope

PingSpot

pinterest\.com

Pizilla

Ploetz \+ Zeller

Plukkie

PocketParser

Pompos

Porkbun

Port Monitor

postano

PostPost

postrank

PowerPoint\/

Priceonomics Analysis Engine

PritTorrent\/[0-9]

Prlog

probethenet

Project 25499

Promotion_Tools_www.searchenginepromotionhelp.com

prospectb2b

Protopage

proximic

pshtt, https scanning

PTST

PTST\/[0-9]+

Pulsepoint XT3 web scraper

Python-httplib2

python-requests

Python-urllib

Qirina Hurdler

QQDownload

Qseero

Qualidator.com SiteAnalyzer

Quora Link Preview

Qwantify

Radian6

RankSonicSiteAuditor

Readability

RealPlayer%20Downloader

RebelMouse

redback\/

Redirect Checker Tool

ReederForMac

request\.js

ResponseCodeTest\/[0-9]

RestSharp

RetrevoPageAnalyzer

Riddler

Rival IQ

Robosourcer

Robozilla\/[0-9]

ROI Hunter

RPT-HTTPClient

RSSOwl

safe-agent-scanner

SalesIntelligent

SauceNAO

SBIder

scalaj-http

Scoop

scooter

ScoutJet

ScoutURLMonitor

Scrapy

ScreenShotService\/[0-9]

Scrubby

search\.thunderstone

SearchSight

Seeker

semanticdiscovery

semanticjuice

Semiocast HTTP client

SEO Browser

Seo Servis

seo-nastroj.cz

Seobility

SEOCentro

SeoCheck

SeopultContentAnalyzer

Server Density Service Monitoring

servernfo\.com

Seznam screenshot-generator

Shelob

Shoppimon Analyzer

ShoppimonAgent\/[0-9]

ShopWiki

ShortLinkTranslate

shrinktheweb

SilverReader

SimplePie

SimplyFast

Site-Shot\/

Site24x7

SiteBar

SiteCondor

siteexplorer\.info

SiteGuardian

Siteimprove\.com

Sitemap(s)? Generator

SiteMonitor

Siteshooter B0t

SiteTruth

sitexy\.com

SkypeUriPreview

Slack\/

slider\.com

slurp

SMRF URL Expander

SMUrlExpander

Snappy

SniffRSS

sniptracker

Snoopy

sogou web

SortSite

sovereign\.ai

spaziodati

Specificfeeds

speedy

SPEng

Spinn3r

spray-can

Sprinklr

sqlmap

spyonweb

Sqworm

SSL Labs

ssl-tools

StackRambler

Statastico\/

StatusCake

Stratagems Kumo

Stroke.cz

StudioFACA

suchen

summify

Super Monitoring

Surphace Scout

SwiteScraper

Symfony2 BrowserKit

SynHttpClient-Built

Sysomos

Symfony BrowserKit

T0PHackTeam

Tarantula\/

Taringa UGC

Tenon\.io

teoma

terrainformatica\.com

Test Certificate Info

Tetrahedron\/[0-9]

The Drop Reaper

The Expert HTML Source Viewer

theinternetrules

theoldreader\.com

Thumbshots

ThumbSniper

TinEye

Tiny Tiny RSS

TLSProbe\/

topster

touche.com

Traackr.com

TrapitAgent

truwoGPS

TulipChain

tweetedtimes\.com

Tweetminster

Tweezler\/

Twikle

Twingly

ubermetrics-technologies

uclassify

uCrawlr\/

UdmSearch

UniversalFeedParser

Untiny

UnwindFetchor

updated

Upflow

Uptimia

URLChecker

URLitor.com

urlresolver

Urlstat

UrlTrends Ranking Updater

Vagabondo

vBSEO

via ggpht\.com GoogleImageProxy

VidibleScraper\/

visionutils

vkShare

voltron

voyager\/

VSAgent\/[0-9]

VSB-TUO\/[0-9]

VYU2

w3af\.org

W3C-checklink

W3C-mobileOK

W3C_I18n-Checker

W3C_Unicorn

wangling

WatchMouse

WbSrch\/

web-capture\.net

Web-Monitoring

Web-sniffer

Webauskunft

WebCapture

WebClient\/

webcollage

WebCookies

WebCorp

WebDoc

WebFetch

WebImages

WebIndex

webkit2png

webmastercoffee

webmon

webscreenie

Webshot

Website Analyzer\/

websitepulse agent

websitepulse[+ ]checker

Websnapr\/

Webthumb\/[0-9]

WebThumbnail

WeCrawlForThePeace

WeLikeLinks

WEPA

WeSEE

wf84

wget

WhatsApp

WhatsMyIP

WhatWeb

WhereGoes\?

Whibse

Whynder Magnet

Windows-RSS-Platform

WinHttpRequest

wkhtmlto

wmtips

Woko

Word\/

WordPress\/

wotbox

WP Engine Install Performance API

wpif

wprecon\.com survey

WPScan

wscheck

WWW-Mechanize

www\.monitor\.us

XaxisSemanticsClassifier

Xenu Link Sleuth

XING-contenttabreceiver\/[0-9]

XmlSitemapGenerator

xpymep([0-9]?)\.exe

Y!J-(ASR|BSC)

Yaanb

yacy

Yahoo Ad monitoring

Yahoo Link Preview

YahooCacheSystem

YahooYSMcm

YandeG

Yandex(?!Search)

yanga

yeti

YLT

Yo-yo

Yoleo Consumer

yoogliFetchAgent

YottaaMonitor

yourls\.org

Zao

Zemanta Aggregator

Zend\\Http\\Client

Zend_Http_Client

zgrab

ZnajdzFoto

ZyBorg

[a-z0-9\-_]*((?<!cu)bot|crawler|archiver|transcoder|spider|uptime|validator|fetcher)

Источник здесь https://github.com/JayBizzle/Crawler-Detect .

  • +1 1
Ссылка на комментарий
Поделиться на других сайтах


6 часов назад, agefremov сказал:

Внесу свои 5 копеек

Боты:

  Показать контент
TinEye-bot/0.51 (see http://www.tineye.com/crawler.html)
  Whoismindbot/1.0 (+http://www.whoismind.com/bot.html)
  Mozilla/5.0 (compatible; MSIE 8.0; Windows NT 6.3; www.alertra.com)
  R6_FeedFetcher(www.radian6.com/crawler)
  CheckHost (http://check-host.net/)
  YandeG 1.03
  yacybot (/global; amd64 Linux 3.16-0.bpo.2-amd64; java 1.7.0_65; Europe/en) http://yacy.net/bot.html
  SEMrushBot
  Protopage/3.0 (http://www.protopage.com)
  yacybot (/global; amd64 Linux 3.16.0-4-amd64; java 1.7.0_75; Europe/en) http://yacy.net/bot.html
  StatoolsBot (+http://www.statools.com/bot.html)
  adidxbot/2.0 (+http://search.msn.com/msnbot.htm)
  Mozilla/5.0 (compatible; Mail.RU_Bot/2.0; +http://go.mail.ru/help/robots)
  Mozilla/5.0 (Linux; Android 5.0; Nexus 5 Build/LRX21O) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/46.0.2490.76 Mobile Safari/537.36 PTST/281
  TurnitinBot (https://turnitin.com/robot/crawlerinfo.html)
  Scrapy/1.0.5 (+http://scrapy.org)
  yacybot (/global; amd64 Linux 4.4.0-31-generic; java 1.8.0_91; Europe/en) http://yacy.net/bot.html
  XmlSitemapGenerator - http://xmlsitemapgenerator.org
  Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.0) Match by Siteimprove.com
  Mozilla/5.0 (compatible; SeznamBot/3.2; +http://napoveda.seznam.cz/en/seznambot-intro/)
  WatchMouse/18990 (http://watchmouse.com/ ; gab)
  WatchMouse/8.4.0.3 (http://watchmouse.com/ ; hkhkg02.watchmouse.net)
  Mozilla/5.0 (compatible; LinkpadBot/1.06; +http://www.linkpad.ru)
  Mozilla/5.0 (compatible; heritrix/1.14.2 +http://rjpower.org)
  yacybot (webportal/global; amd64 Linux 2.6.32-5-amd64; java 1.6.0_18; Europe/en) http://yacy.net/bot.html
  PercolateCrawler/4 ([email protected])
  msnbot-UDiscovery/2.0b (+http://search.msn.com/msnbot.htm)
  Mozilla/5.0 (compatible; MSIE or Firefox mutant; not on Windows server;) Daum 4.1
  Mozilla/5.0 (compatible; spbot/4.0.3; +http://www.seoprofiler.com/bot )
  Mozilla/5.0 (compatible; LoadTimeBot/0.9; +http://www.loadtime.net/bot.html)
  UnwindFetchor/1.0 (+http://www.gnip.com/)
  nrsbot/5.0(loopip.com/robot.html)
  SemrushBot/0.9
  Mozilla/5.0 (compatible; UASlinkChecker/2.1; +https://udger.com/support/UASlinkChecker)
  yacybot (/global; amd64 Linux 3.11.10-21-desktop; java 1.7.0_51; America/en) http://yacy.net/bot.html
  Netvibes (http://www.netvibes.com)
  Acoon v4.1.0 (www.acoon.de)
  msnbot/2.0b (+http://search.msn.com/msnbot.htm).
  Mozilla/5.0 (compatible; ltbot/3.2.0.10 +http://www.kdsl.tu-darmstadt.de/de/kdsl/research-program/crawling-and-semantic-structuring/)
  smart.apnoti.com Robot/v1.34 (http://smart.apnoti.com/en/aboutApnotiWebCrawler.html)
  HubSpot Links Crawler 2.0 http://www.hubspot.com/
  yacybot (/global; x86 Windows XP 5.1; java 1.7.0_55; Europe/de) http://yacy.net/bot.html
  Mozilla/5.0 (compatible; DCPbot/1.0; +http://domains.checkparams.com/)
  Mozilla/5.0 (compatible; Exabot/3.0 (BiggerBetter); +http://www.exabot.com/go/robot)
  WebDoc (abuse-webdoc at service.moquadv.com)
  coccoc
  Mozilla/5.0 (compatible; spbot/4.0b; +http://www.seoprofiler.com/bot )
  Mozilla/5.0 (compatible; ExchangleBot/3.0; +https://www.exchangle.com/exchangling)
  Mozilla/5.0 (compatible; Qwantify/2.0n; +https://www.qwant.com/)
  OOZBOT/0.20 ( Setooz výrazný ako say-th-uuz, znamená mosty. ; http://www.setooz.com/oozbot.html ; agentname at setooz dot_com )
  SpeedySpider - http://www.entireweb.com
  Mozilla/5.0 (compatible; Heurekabot/3.1; +http://sluzby.heureka.cz/)
  crawler for netopian (http://www.netopian.co.uk/)
  L.webis/0.51 (http://webalgo.iit.cnr.it/index.php?pg=lwebis)
  Influencebot/0.9; (Automatic classification of websites; http://www.influencebox.com/; [email protected])
  Baiduspider+(+http://www.baidu.com/search/spider.htm)
  Mozilla/5.0 (X11; compatible; semantic-visions.com crawler; HTTPClient 3.1)
  Mozilla/5.0 (Windows; U; Windows NT 6.0; en-US; rv:1.9.2.28) Gecko/20120306 Firefox/99.0 YottaaMonitor
  Scrapy/1.1.1 (+http://scrapy.org)
  Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/27.0.1453.116 Safari/537.36 HubSpot Marketing Grader
  RyzeCrawler/1.1.1 ( http://www.domain2day.nl/crawler/)
  eBot / v.1.0a (http://alfa.elchron.cz)
  Sogou News Spider/4.0(+http://www.sogou.com/docs/help/webmasters.htm#07)
  Crowsnest/0.5 (+http://www.crowsnest.tv/)
  Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.4 (KHTML, like Gecko; Google Page Speed Insights) Chrome/22.0.1229 Safari/537.4
  DoCoMo/2.0 N902iS(c100;TB;W24H12)(compatible; moba-crawler; http://crawler.dena.jp/)
  Yeti/1.1 (NHN Corp.; http://help.naver.com/robots/)
  Experibot_v1
  Nekstbot - http://www.ipipan.waw.pl/nekst/nekstbot/
  Mozilla/5.0 (compatible; MJ12bot/v1.4.5; http://www.majestic12.co.uk/bot.php?+)
  Mozilla/5.0 (X11; Linux i686) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/30.0.1599.101 Safari/537.36; SSL-Crawler: http://crawler.dcsec.uni-hannover.de
  CorpusCrawler 2.0.19 (http://corpora.fi.muni.cz/crawler/);Project:CzCorpus
  Mozilla/5.0 (compatible; Semager/1.4; http://www.semager.de/blog/semager-bots/)
  Mozilla/5.0 (compatible; dlcbot/0.1; +http://www.drlinkcheck.com/)
  yacybot (freeworld/global; amd64 Linux 3.11.10-21-desktop; java 1.7.0_51; Europe/de) http://yacy.net/bot.html
  A6-Indexer/1.0 (http://www.a6corp.com/a6-web-scraping-policy/)
  CopperEgg/RevealUptime/DallasTX(linode)
  Mozilla/5.0 (compatible; Qwantify/2.1dw; +https://www.qwant.com/)/*
  Curious George - www.analyticsseo.com
  GozaikBot (www.gozaik.com;[email protected];www.gozaik.com/gozaikbot.html)
  Pu_iN Crawler (+http://semanticjuice.com/)
  Mozilla/5.0 (compatible; OpenindexDeepSpider/Nutch-1.5-dev; +http://www.openindex.io/en/webmasters/spider.html; systemsATopenindexDOTio)
  yacybot (freeworld/global; i386 Linux 3.0.0-17-generic-pae; java 1.6.0_23; Europe/en) http://yacy.net/bot.html
  yacybot (-global; amd64 FreeBSD 9.2-RELEASE-p10; java 1.7.0_65; Europe/en) http://yacy.net/bot.html
  FAST Enteprise Crawler/6 (www dot fastsearch dot com)
  Mozilla/5.0 (compatible; parsijoo-update-crawler; +http://www.parsijoo.ir/; [email protected])
  yacybot (freeworld/global; amd64 Windows Server 2008 6.0; java 1.7.0_25; Europe/en) http://yacy.net/bot.html
  Inspingbot/1.0 (+https://www.insping.com/)
  OrgProbe/0.9.4 (+http://www.blocked.org.uk)
  Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; Trident/5.0); 360Spider
  Mozilla/5.0 (Windows NT 6.1; WOW64; Trident/7.0; rv:11.0) like Gecko PTST/276
  Mozilla/5.0 (Windows NT 6.3; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/42.0.2311.90 Safari/537.36 PTST/276
  Mozilla/5.0 (compatible; SemrushBot/0.97~bl; +http://www.semrush.com/bot.html)
  Feedspot http://www.feedspot.com
  Zookabot/2.0;++http://zookabot.com
  drupact/0.7; http://www.arocom.de/drupact
  RobotsChecker/0.6 (+http://www.blocked.org.uk)
  yacybot (/global; i386 Linux 3.13.0-37-generic; java 1.7.0_65; Europe/en) http://yacy.net/bot.html
  Scrapy/1.0.3 (+http://scrapy.org)
  Testomatobot/1.0 (Linux x86_64; +http://www.testomato.com/testomatobot) minicrawler/4.0.0~beta12
  Slack-ImgProxy 0.66 (+https://api.slack.com/robots)
  Mozilla/5.0 (compatible; WebCookies/1.0; +https://webcookies.org/faq/#agent)
  NalezenCzBot/1.0 (http://www.nalezen.cz)
  yacybot (freeworld/global; amd64 Windows Server 2008 6.0; java 1.7.0_03; Europe/en) http://yacy.net/bot.html
  Mozilla/5.0 (compatible; AhrefsBot/5.1; +http://ahrefs.com/robot/)
  yacybot (freeworld/global; amd64 Windows 8 6.2; java 1.7.0_51; Europe/de) http://yacy.net/bot.html
  yacybot (/global; amd64 Windows 7 6.1; java 1.8.0_65; Europe/de) http://yacy.net/bot.html
  Mozilla/4.0 compatible ZyBorg/1.0 ([email protected]; http://www.WISEnutbot.com)
  Mozilla/5.0 (en-us) AppleWebKit/537.36(KHTML, like Gecko; Google-Adwords-DisplayAds-WebRender;) Chrome/27.0.1453Safari/537.36
  Crawler powered by contentDetection (www.mindup.de)
  PagePeeker.com (info: http://pagepeeker.com/robots)
  Mozilla/5.0 (compatible; Linux x86_64; Mail.RU_Bot/Fast/2.0; +http://go.mail.ru/help/robots)
  coccoc/1.0 (http://help.coccoc.com/)
  Favicon downloader (+https://favico.be/bot.html)
  Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0) ([email protected])
  Mozilla/5.0 (compatible; DIY-SEOBot/0.1a; +http://www.upcity.com/bot.html)
  Mozilla/5.0 (Windows NT 6.3; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/49.0.2623.112 Safari/537.36 PTST/276
  Mozilla/5.0 (compatible; theoldreader.com)
  Mozilla/5.0+(compatible; UptimeRobot/2.0; http://www.uptimerobot.com/)
  hawkReader/1.8 (Link Parser; http://www.hawkreader.com/; Allow like Gecko) Build/f2b2566
  Mozilla/5.0 (compatible; heritrix/1.12.1 +http://www.webarchiv.cz)
  Mozilla/5.0 (compatible; YandexImages/3.0; +http://yandex.com/bots)
  yacybot (/global; amd64 Linux 3.14.32-xxxx-grs-ipv6-64; java 1.7.0_75; Europe/en) http://yacy.net/bot.html
  Sogou web spider/4.0(+http://www.sogou.com/docs/help/webmasters.htm#07)
  yacybot (/global; amd64 Windows 7 6.1; java 1.8.0_45; Europe/de) http://yacy.net/bot.html
  larbin_2.6.3 [email protected]
  yacybot (freeworld/global; amd64 Linux 3.10.17-gentoo; java 1.7.0_45; UTC/en) http://yacy.net/bot.html
  MeMoNewsBot/2.0 (http://www.memonews.com/en/crawler)
  Mozilla/5.0 (compatible; MSIE 10.0; Windows NT 6.1; WOW64; Trident/6.0) PTST/281
  Mozilla/5.0 (compatible; DCPbot/1.5; +http://domains.checkparams.com/)
  Mozilla/5.0 (compatible; SemrushBot/0.97; +http://www.semrush.com/bot.html)
  BlackBerry9000/4.6.0.167 Profile/MIDP-2.0 Configuration/CLDC-1.1 VendorID/102 ips-agent
  Mozilla/5.0 (compatible; Baiduspider-cpro; +http://www.baidu.com/search/spider.html)
  yacybot (/global; amd64 Linux 3.2.0-4-amd64; java 1.7.0_65; Europe/en) http://yacy.net/bot.html
  Mozilla/5.0 (compatible; electricmonk/3.1.1 +https://www.duedil.com/our-crawler/)
  Hatena Antenna/0.5 (http://a.hatena.ne.jp/help)
  Backlink-Ceck.de (+http://www.backlink-check.de/bot.html)
  Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; WOW64; Trident/5.0) commoncrawl.org/research//Nutch-1.7-SNAPSHOT
  Mozilla/5.0 (compatible; Steeler/3.5; http://www.tkl.iis.u-tokyo.ac.jp/~crawler/)
  LapozzBot/1.4 (+http://robot.lapozz.com)
  Mozilla/5.0 (WhatsMyIP.org HTTP_Compression_Test) http://whatsmyip.org/ua
  Mozilla/5.0 (compatible; MSIE or Firefox mutant; not on Windows server;) Daumoa/4.0 (Following Mediapartners-Google)
  Mozilla/5.0 (compatible; ExaleadCloudview/6;)
  ADmantX Platform Semantic Analyzer - ADmantX Inc. - www.admantx.com - [email protected]
  msnbot/1.0 (+http://search.msn.com/msnbot.htm)
  Mozilla/5.0 (Windows NT 5.1) AppleWebKit/537.4 (KHTML, like Gecko) Chrome/22.0.1229.79 Safari/537.4 LinkTiger 2.0
  Pixray-Seeker/1.1 (Pixray-Seeker; [email protected])
  mbot v.1.16
  MXT/Nutch-1.12-SNAPSHOT (http://t.co/GSRLLKex24; informatique at mixdata dot com)
  Testomatobot/1.0 (Linux x86_64; +http://www.testomato.com/testomatobot) minicrawler/3.0.0
  polybot 1.0 (http://cis.poly.edu/polybot/)
  Mozilla/5.0 (compatible; YandexMetrika/2.0; +http://yandex.com/bots mtweb01t.yandex.ru)
  Mozilla/5.0 (compatible; memoryBot/1.21.46 +http://internetmemory.org/en/)
  Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_1) AppleWebKit/600.2.5 (KHTML, like Gecko) Version/8.0.2 Safari/600.2.5 (Applebot/0.1)
  yacybot (amd64 Linux 2.6.26-2-amd64; java 1.6.0_0; Europe/en) http://yacy.net/bot.html
  yacybot (/global; x86 Windows 8.1 6.3; java 1.8.0_45; America/en) http://yacy.net/bot.html
  Mozilla/5.0 (compatible; startmebot/1.0; +http://www.start.me/bot)
  yacybot (amd64 Windows 7 6.1; java 1.6.0_21; Europe/fr) http://yacy.net/bot.html
  yacybot (/global; amd64 Windows 7 6.1; java 1.8.0_101; Asia/ru) http://yacy.net/bot.html
  yacybot (freeworld/global; amd64 Linux 3.0.0-17-generic; java 1.6.0_23; Europe/de) http://yacy.net/bot.html
  alexa v0.1.4 (http://www.openwebspider.org/)
  http://arachnode.net 1.4
  Photon/1.0
  NetpeakCheckerBot
  GIDBot/3.0 (+http://www.gidnetwork.com/tools/gzip-test.php)
  Yandex/1.01.001 (compatible; Win16; P)
  w3dt.net httphr/2.0
  yacybot (/global; amd64 Windows 7 6.1; java 1.8.0_51; Europe/de) http://yacy.net/bot.html
  iqdb/0.1 (+http://iqdb.org/)
  Mozilla/5.0 (compatible; GimmeUSAbot/1.0; +https://gimmeusa.com/pages/crawler)
  Motoricerca-Robots.txt-Checker/1.0 (http://tool.motoricerca.info/robots-checker.phtml)
  Mozilla/5.0 (compatible; Tagoobot/3.0; +http://www.tagoo.ru)
  Mozilla/5.0 (Windows NT 6.1; WOW64; rv:26.0) Gecko/20100101 Firefox/26.0 Evidon ([email protected])
  Mozilla/5.0 (compatible; Nmap Scripting Engine; http://nmap.org/book/nse.html)
  yacybot (i386 Linux 2.6.32-22-generic; java 1.6.0_20; Europe/de) http://yacy.net/bot.html
  Mozilla/5.0 (Linux; Android 4.4.3; HTC One Build/KTU84L) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/40.0.2125.111 Mobile Safari/537.36 DareBoost
  mindUpBot (datenbutler.de)
  Mozilla/5.0 (compatible; monitis - premium monitoring service; http://www.monitis.com)
  Mozilla/5.0 (compatible; RankSonicSiteAuditor/1.0; +https://ranksonic.com/ranksonic_sab.html)
  Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/40.0.2125.111 Safari/537.36 DareBoost
  Zookabot/2.1;++http://zookabot.com
  yacybot (freeworld/global; amd64 Windows 7 6.1; java 1.6.0_24; Europe/de) http://yacy.net/bot.html
  Mozilla/5.0 (compatible; Google-Structured-Data-Testing-Tool +http://developers.google.com/structured-data/testing-tool/)
  Mozilla/5.0 (compatible; Plukkie/1.2; http://www.botje.com/plukkie.htm)
  focusbot/1.1
  Mozilla/5.0 (compatible; idmarch Automatic.beta/1.2; +http://www.idmarch.org/bot.html)
  CorpusCrawler 2.0.14 (http://corpora.fi.muni.cz/crawler/)
  Mozilla/5.0 (compatible; XML Sitemaps Generator; https://www.xml-sitemaps.com) Gecko XML-Sitemaps/1.0
  ImplisenseBot 1.1
  Promotion_Tools_www.searchenginepromotionhelp.com
  Aboundex/0.2 (http://www.aboundex.com/crawler/)
  CorpusCrawler 2.0.22 (http://corpora.fi.muni.cz/crawler/);Project:CzCorpus
  yacybot (/global; amd64 no-os-name no-os-version; java no-java-version; Europe/en) http://yacy.net/bot.html
  LumpImageSearch/0.1 (+http://lump.co/about/bot)
  Mozilla/4.0 (compatible; NaverBot/1.0; http://help.naver.com/customer_webtxt_02.jsp)
  Mozilla/5.0 (compatible; UASlinkChecker/2.0; +http://udger.com/support/UASlinkChecker)
  GAChecker (+http://www.gachecker.com)
  yacybot (freeworld/global; x86 Windows 7 6.1; java 1.7.0_25; Europe/de) http://yacy.net/bot.html
  Influencebot/0.9; (Automatic classification of websites; http://www.influencebox.com/; [email protected])User-Agent: Mozilla/5.0 (X11; Linux i686; rv:9.0) Gecko/20100101 Firefox/9.0
  Experibot_v1 (https://dl.dropboxusercontent.com/u/8024465/site/Info.html)
  DNSPod-reporting(http://www.dnspod.cn/reporting)
  yacybot (freeworld/global; i386 Linux 3.2.0-23-generic; java 1.6.0_27; Europe/en) http://yacy.net/bot.html
  Mozilla/5.0 (Windows NT 6.3;compatible; Leikibot/1.0; +http://www.leiki.com)
  PostPost/1.0 (+http://postpo.st/crawlers)
  envolk/1.7 (+http://www.envolk.com/envolkspiderinfo.html)
  Snapbot/1.0 (Snap Shots, +http://www.snap.com)
  Mozilla/5.0 (Linux; Android 4.0.4; Galaxy Nexus Build/IMM76B) AppleWebKit/537.36 (KHTML, like Gecko; Google-Publisher-Plugin) Chrome/27.0.1453 Mobile Safari/537.36
  Mozilla/5.0 (compatible; Uptimebot/0.2.18; +http://www.uptime.com/uptimebot)
  MIA DEV/search:robot/0.0.1 (This is the MIA Bot - crawling for mia research project. If you feel unhappy and do not want to be visited by our crawler send an email to [email protected]; http://spider.neofonie.de; [email protected])
  Mozilla/5.0 (Windows NT 6.1; Trident/7.0; rv:11.0; PTST 2.386) like Gecko
  Jyxobot/1
  WebAlta Crawler/2.0 (http://www.webalta.net/ru/about_webmaster.html) (Windows; U; Windows NT 5.1; ru-RU)
  GC3pro+dir SEO Tools - Vers. 3.00b - For more informations: http://chkme.com/
  Mozilla/5.0 (X11; Linux x86_64; rv:45.0; GTmetrix https://gtmetrix.com/) Gecko/20100101 Firefox/45.0
  Mozilla/5.0 (Windows NT 6.1; WOW64; rv:18.0) Gecko/20100101 Firefox/18.0 AppEngine-Google; (+http://code.google.com/appengine; appid: s~aeshortener)
  Priceonomics Analysis Engine - Fetch/1.0
  Ruky-Roboter (Version: 1.06, powered by www.ruky.de +http://www.ruky.de/bot.html)
  Baiduspider+(+http://help.baidu.jp/system/05.html)
  Openstat/0.1
  Yandex/1.01.001 (compatible; Win16; m)
  Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_1) AppleWebKit/600.2.5 (KHTML, like Gecko) Version/8.0.2 Safari/600.2.5 (Applebot/0.1; +http://www.apple.com/go/applebot)
  Mozilla/5.0 (compatible; OpenindexSpider/Nutch-1.5-dev; +http://www.openindex.io/en/webmasters/spider.html)
  Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; Multiviewbot
  CERT.at-Statistics-Survey/1.0 (http://www.cert.at/about/consec/content.html)
  Mozilla/5.0 (compatible; pmoz.info ODP link checker; +http://pmoz.info/doc/botinfo.htm)
  yacybot (freeworld/global; x86_64 Mac OS X 10.6.8; java 1.6.0_29; Asia/ru) http://yacy.net/bot.html
  gonzo/1[P] (+http://www.suchen.de/faq.html)
  MixBot (+http://t.co/GSRLLKex24)
  yacybot (/global; amd64 Linux 3.2.0-4-amd64; java 1.7.0_60; Europe/en) http://yacy.net/bot.html
  Mozilla/5.0 (compatible; imbot/0.1 +http://internetmemory.org/en/
  Mozilla/5.0 (en-us) AppleWebKit/525.13 (KHTML, like Gecko; Google Web Preview) Version/3.1 Safari/525.13
  Mozilla/5.0 (compatible; Pi-Monster; https://pricepi.com/)
  ThumbSniper (http://thumbsniper.com)
  Shelob ([email protected])
  Mozilla/5.0 (Macintosh; Intel Mac OS X 10_7_3) AppleWebKit/537.36 (KHTML, like Gecko, Google-Publisher-Plugin) Chrome/27.0.1453 Safari/537.36
  Mozilla/5.0 (compatible; KaloogaBot; http://www.kalooga.com/info.html?page=crawler)
  yacybot (/global; arm Linux 4.1.13+; java 1.8.0_40-internal; Etc/de) http://yacy.net/bot.html
  CorporateNewsSearchEngine/Nutch-1.7 (http://pibs.co/news-search-engine)
  Mozilla/5.0 (compatible; MSIE 10.0; Windows NT 6.1; WOW64; Trident/6.0) PTST/276
  Yandex/1.01.001 (compatible; Win16; I)
  FlightDeckReportsBot/2.0 (http://www.flightdeckreports.com/pages/bot)
  Scrapy/0.24.4 (+http://scrapy.org)
  ADmantX Platform Semantic Analyzer US - Turn - ADmantX Inc. - www.admantx.com - [email protected]
  Kyoto-Tohoku-Crawler/v1 (Mozilla-compatible; [email protected]; http://nlp.ist.i.kyoto-u.ac.jp/?crawling-kt)
  Mozilla/5.0 (compatible; Scarlett/ 1.0; +http://www.ellerdale.com/crawler.html)
  Mozilla/5.0 (compatible; NetcraftSurveyAgent/1.0; [email protected])
  www.deadlinkchecker.com Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/46.0.2490.86 Safari/537.36
  http://arachnode.net 1.2
  Mozilla/5.0 (compatible; Plukkie/1.5; http://www.botje.com/plukkie.htm)
  yacybot (freeworld/global; arm Linux 4.4.11-v7+; java 1.7.0_101; Etc/en) http://yacy.net/bot.html
  Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.0.16) Gecko/2009121601 Ubuntu/9.04 (jaunty) Firefox/3.0.16 Specificfeeds- http://www.specificfeeds.com
  Mozilla/5.0 (compatible; evc-batch/2.0)
  Orbiter/1.2 (http://dailyorbit.com/)
  crawler4j (https://github.com/yasserg/crawler4j/)
  Mozilla/5.0 (compatible; SEOdiver/1.0; +http://www.seodiver.com/bot)
  Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/51.0.2704.79 Safari/537.36 (https://shrinktheweb.com)
  Scopia crawler 1.0 (+http://www.scopia.co)
  yacybot (i386 Linux 2.6.23; java 1.6.0_06; Europe/en) http://yacy.net/bot.html
  Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.1.2) Gecko/20090729 Firefox/3.5.2 (.NET CLR 3.5.30729; Diffbot/0.1; +http://www.diffbot.com)
  Mozilla/5.0 (compatible; LXRbot/1.0;http://www.lxrmarketplace.com/,[email protected])
  yacybot (freeworld/global; amd64 Linux 3.8.13-gentoo; java 1.7.0_21; Europe/de) http://yacy.net/bot.html
  Mozilla/5.0 (compatible; Sitemap Generator/1.3; +http://www.check-domains.com/sitemap/index.php)
  DuckDuckBot/1.1; (+http://duckduckgo.com/duckduckbot.html)
  Mozilla/5.0 (compatible; BLEXBot/1.0; +http://webmeup-crawler.com/)
  Slack-ImgProxy 0.59 (+https://api.slack.com/robots)
  Mozilla/5.0 (compatible; Ezooms/1.0; [email protected])
  Mozilla/5.0 (compatible; YandexPagechecker/2.0; +http://yandex.com/bots)
  CoinCornerBot/1.1 ( https://www.coincorner.com/BitcoinBot)
  yacybot (freeworld/global; amd64 Linux 3.8.0-21-generic; java 1.6.0_27; Pacific/en) http://yacy.net/bot.html
  ScreenerBot Crawler Beta 2.0 (+http://www.ScreenerBot.com)
  gonzo1[P] +http://www.suchen.de/faq.html
  Mozilla/5.0 (compatible; MSIE or Firefox mutant;) Daum 4.1
  Mozilla/5.0 (compatible; 008/0.83; http://www.80legs.com/spider.html;) Gecko/2008032620
  Sogou web spider/4.02525A
  Visbot/2.0 (+http://www.visvo.com/en/webmasters.jsp;[email protected])
  Mozilla/5.0 (compatible; AcoonBot/4.10.8; +http://www.acoon.de/robot.asp)
  WatchMouse/18990 (http://watchmouse.com/ ; d3.watchmouse.com)
  Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/534.34 (KHTML, like Gecko) Qt/4.8.1 Safari/534.34 ShoppimonAgent/1.0 ([email protected])
  webinatorbot 1.1; +http://www.webinator.de
  findlinks/2.0.5 (+http://wortschatz.uni-leipzig.de/findlinks/)
  WikiDo/1.1 (http://wikido.com; [email protected])
  yacybot (freeworld/global; amd64 Linux 3.5.0-27-generic; java 1.7.0_03; Europe/de) http://yacy.net/bot.html
  Mozilla/5.0 (compatible; Uptimebot/0.1.73; +http://www.uptime.com/uptimebot)
  Semantifire1/0.20 ( -- ; http://www.setooz.com/oozbot.html ; agentname at setooz dot_com )
  Speedy Spider (http://www.entireweb.com/about/search_tech/speedy_spider/)
  Mozilla/5.0 compatible; yelpspider/yelpspider-1.0 (Crawlerbot run by Yelp Inc; yelpbot at yelp dot com)
  Mozilla/5.0 (compatible; OpenHoseBot/2.1; +http://www.openhose.org/bot.html)
  Mozilla/5.0 (compatible; emefgebot/beta; +http://emefge.de/bot.html)
  Mozilla/5.0 (compatible; OpenindexShallowSpider/Nutch-1.5-dev; +http://www.openindex.io/en/webmasters/spider.html; systemsATopenindexDOTio)
  Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/49.0.2623.75 Safari/537.36 Google (+https://developers.google.com/+/web/snippet/)
  Mozilla/5.0 (compatible; MetaJobBot; http://www.metajob.de/crawler)
  CCBot/1.0 (+http://www.commoncrawl.org/bot.html)
  Lijit Crawler (+http://www.lijit.com/robot/crawler)
  baypup/1.1 (Baypup; http://www.baypup.com/; [email protected])
  Mozilla/5.0 (FauBot/0.1; +http://buzzvolume.com/fau/)
  Mozilla/5.0 (compatible; NLNZ_IAHarvester2016/3.3.0 +https://natlib.govt.nz/publishers-and-authors/web-harvesting/domain-harvest)
  yacybot (/global; amd64 Linux 3.12.1; java 1.7.0_65; Europe/en) http://yacy.net/bot.html
  yacybot (amd64 Windows 7 6.1; java 1.6.0_18; Europe/de) http://yacy.net/bot.html
  SEOENGWorldBot/1.0 (+http://www.seoengine.com/seoengbot.htm)
  Mozilla/5.0 (compatible; Finderbots finder bot; +http://wiki.github.com/bixo/bixo/bixocrawler; [email protected])
  Pompos/1.3 http://dir.com/pompos.html
  Mozilla/5.0 (compatible; Gimme60bot/1.0; +http://gimme60.com) Firefox/16.0
  Mozilla/5.0 (compatible; MJ12bot/v1.4.6; http://mj12bot.com/)
  Mozilla/5.0 (compatible; Sysomos/1.0; +http://www.sysomos.com/; Sysomos)
  urlfan-bot/1.0; +http://www.urlfan.com/site/bot/350.html
  al_viewer ([email protected])
  LoadImpactRload/3.1.1 (Load Impact; http://loadimpact.com);
  findlinks/2.0.2 (+http://wortschatz.uni-leipzig.de/findlinks/)
  Mozilla/5.0 (compatible; spbot/3.1; +http://www.seoprofiler.com/bot )
  Mozilla/5.0 (compatible; AcoonBot/4.11.0; +http://www.acoon.de/robot.asp)
  WillyBot/1.1 (http://www.willyfogg.com/info/willybot)
  Norton-Safeweb
  WMCAI-robot (http://www.topicmaster.jp/wmcai/crawler.html)
  rogerbot/1.1 (http://moz.com/help/guides/search-overview/crawl-diagnostics#more-help, [email protected])
  Szukacz/1.5 (robot; www.szukacz.pl/jakdzialarobot.html; [email protected])
  rogerbot/1.0 (http://www.seomoz.org/dp/rogerbot, [email protected])
  ROR Sitemap Generator (http://www.rorweb.com)
  http://domino.research.ibm.com/comm/research_projects.nsf/pages/sai-crawler.callingcard.html
  MozillaTest/5.0 (compatible; YodaoBot/1.0; http://www.yodao.com/help/webmaster/spider/; )
  Mozilla/5.0 (compatible; Peew/1.0; http://www.peew.de/crawler/)
  Mozilla/5.0 (compatible; Website Analyzer/1.1; +http://www.check-domains.com/website-analysis/website-analyzer.php)
  Mozilla/5.0 (compatible; Gluten Free Crawler/1.0; +http://glutenfreepleasure.com/)
  PagePeeker.com
  CorpusCrawler 2.0.10 (http://corpora.fi.muni.cz/crawler/)
  yacybot (/global; amd64 Linux 3.2.0-4-amd64; java 1.7.0_101; Europe/cs) http://yacy.net/bot.html
  yacybot (/global; amd64 Linux 4.0.7-1-ck; java 1.8.0_45; Europe/de) http://yacy.net/bot.html
  Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/51.0.2704.103 Safari/537.36 PTST/284
  yacybot (freeworld/global; i386 Linux 3.12-1-686-pae; java 1.7.0_21; Europe/fr) http://yacy.net/bot.html
  Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.1; Trident/4.0; PTST 2.295)
  CovarioIDS/1.1 (http://www.covario.com/ids; support at covario dot com)
  Mozilla/5.0 (compatible; heritrix/1.14.4 +http://parsijoo.ir)
  www.adressendeutschland.de
  DialogSearch.com Bot 1.0;http://dialogsearch.com/webmasters
  librabot/2.0 (+http://search.msn.com/msnbot.htm)
  Mozilla/5.0 (compatible; heritrix/3.1.1-SNAPSHOT-20120116.200628 +http://www.archive.org/details/archive.org_bot)
  wsAnalyzer/1.0; ++http://www.wsanalyzer.com/bot.html
  Mozilla/5.0 (compatible; websays; +http://wiki.github.com/bixo/bixo/bixocrawler; [email protected])
  facebookexternalhit/1.0 (+http://www.facebook.com/externalhit_uatext.php)
  yacybot (/global; amd64 Linux 4.4.5-1-ARCH; java 1.8.0_77; America/en) http://yacy.net/bot.html
  websitepulse checker/1.1 (compatible; MSIE 5.5; Netscape 4.75; Linux)
  Mozilla/5.0 (compatible; memoryBot/1.21.14 +http://mignify.com/bot.html)
  SalesIntelligent/v1.0
  larbin_2.6.2 [email protected]
  Mozilla/5.0 (compatible; SiteCondor; http://www.sitecondor.com)
  yacybot (freeworld/global; amd64 Windows Server 2012 6.2; java 1.7.0_51; Europe/de) http://yacy.net/bot.html
  Mozilla/5.0 (compatible; Crawlera/1.10.2; UID 43063)
  yacybot (-global; amd64 Linux 2.6.32-042stab111.11; java 1.7.0_79; Europe/en) http://yacy.net/bot.html
  Mozilla/5.0 (compatible; SemrushBot/0.96.2; +http://www.semrush.com/bot.html)
  Linguee Bot (http://www.linguee.com/bot)
  yacybot (/global; x86 Windows XP 5.1; java 1.7.0_51; Europe/de) http://yacy.net/bot.html
  ICC-Crawler(Mozilla-compatible; ; http://kc.nict.go.jp/project1/crawl.html)
  Mozilla/5.0 (X11; U; Linux i686 (x86_64); en-US; rv:1.8.1.11) Gecko/20080109 (Charlotte/0.9t; http://www.searchme.com/support/) (Charlotte/0.9t; http://www.searchme.com/support/)
  Mozilla/5.0 (compatible; Goodzer/2.0; [email protected])
  Acoon v4.10.5 (www.acoon.de)
  CorpusCrawler 2.0.20 (http://corpora.fi.muni.cz/crawler/);Project:CzCorpus
  Mozilla/5.0 (compatible; AMZNKAssocBot/4.0 +http://affiliate-program.amazon.com)
  Mozilla/5.0 (compatible; MojeekBot/0.5; http://www.mojeek.com/bot.html)
  yacybot (/global; amd64 Linux 4.2.0-22-generic; java 1.7.0_91; Europe/en) http://yacy.net/bot.html
  Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/534.34 (KHTML, like Gecko) Qt/4.8.3 Safari/534.34 https://linkpeek.com
  MetaTagRobot/0.2 (http://www.seocentro.com/tools/search-engines/metatag-analyzer.html)
  Mozilla/5.0 (compatible; MJ12bot/v1.3.1; http://www.majestic12.co.uk/bot.php?+)
  Mozilla/5.0 (compatible; Uptimebot/0.2.29; +http://www.uptime.com/uptimebot)
  Mozilla/5.0 (compatible; Pandeo Bot; +http://pandeo.de/bot.php)
  Pixray-Seeker/1.1 (Pixray-Seeker; http://www.pixray.com/pixraybot; [email protected])
  EasyBib AutoCite (http://autocite-info.citation-api.com/)
  Mozilla/5.0 (compatible; OptimizationCrawler/0.2; +http://www.domainoptima.com/robot)
  AboutUsBot/Harpy (Website Analysis; http://www.aboutus.org/Aboutus:Bot; [email protected])
  Gigabot/2.0
  DoCoMo/2.0 N905i(c100;TB;W24H16) (compatible; Googlebot-Mobile/2.1; +http://www.google.com/bot.html)
  Morning Paper 1.0 (robots.txt compliant!)
  Mozilla/5.0 (compatible; SurdotlyBot/1.0; +http://sur.ly/bot.html; Linux; Android 4; iPhone; CPU iPhone OS 6_0_1 like Mac OS X) AppleWebKit/536.26 (KHTML, like Gecko) Version/6.0 Mobile/10A523 Safari/8536.25
  Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.1; Trident/4.0; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; Media Center PC 6.0; .NET4.0C; .NET4.0E; GWX:RED; PTST 2.386)
  SuperPagesUrlVerifyBot/1.0
  CopperEgg/RevealUptime/LondonUK(linode)
  Mozilla/5.0 (compatible; www.monitor.us - free monitoring service; http://www.monitor.us)
  Mozilla/5.0 (compatible; LinkMarketbot/1.2; +http://www.linkmarket.com/)
  Mozilla/5.0 (Windows; U; Windows NT 5.1; zh-CN; ) Firefox/1.5.0.11; 360Spider
  wangling
  Mozilla/5.0 (compatible; linkdexbot/2.0; +http://www.linkdex.com/about/bots/)
  RSSMicro.com RSS/Atom Feed Robot
  GarlikCrawler/1.1 (http://garlik.com/, [email protected])
  al_org_viewer ([email protected])
  Mozilla/5.0 (compatible; JadynAveBot; +http://www.jadynave.com/robot)
  dj-research/Nutch-1.11 (analytics@@demandjump.com)
  Karneval-Bot (Version: 1.06, powered by www.karnevalsuchmaschine.de +http://www.karnevalsuchmaschine.de/bot.html)
  Baiduspider-image+(+http://www.baidu.com/search/spider.htm)\nReferer: http://image.baidu.com/i?ct=503316480&z=0&tn=baiduimagedetail
  SauceNAO/1.0 (+http://saucenao.com/)
  Mozilla/5.0 (compatible; JobdiggerSpider +http://www.jobdigger.nl/spider)
  Yepi/1.0 (NHN Corp.; http://help.naver.com/robots/)
  Mozilla/5.0 (compatible; coccoc/1.0; +http://help.coccoc.com/)
  SSL Labs (https://www.ssllabs.com/about/assessment.html)
  Mozilla/5.0 (compatible; http://alyze.info)
  GigablastOpenSource/1
  Mozilla/5.0 (Windows; U; Windows NT 5.1;fr;rv:1.8.1) VoilaBotCollector BETA 0.1 (http://www.voila.com/)
  Vorboss Web Crawler [[email protected]]/Nutch-2.3
  Mozilla/5.0 (compatible; SecretSerachEngineLabs.com-SBSearch/0.9; http://www.secretsearchenginelabs.com/secret-web-crawler.php)
  Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko; Google Search Console) Chrome/27.0.1453 Safari/537.36
  YahooCacheSystem
  search.KumKie.com
  yacybot (webportal-global; x86 Windows Vista 6.0; java 1.7.0_25; Europe/en) http://yacy.net/bot.html
  Embedly [email protected]
  stq_bot (+http://www.searchteq.de)
  MSRBOT
  Mozilla/5.0 (compatible; WBSearchBot/1.1; +http://www.warebay.com/bot.html)
  CorpusCrawler 2.0.17 (http://corpora.fi.muni.cz/crawler/);Project:CzCorpus
  Mozilla/5.0 (compatible; heritrix/3.1.0-RC1 +http://boston.lti.cs.cmu.edu/crawler_12/)
  Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; Trident/5.0) PTST/276
  Mozilla/5.0 (compatible; WebCookies/1.0; +http://webcookies.org/faq/#agent)
  L.webis/0.44 (http://webalgo.iit.cnr.it/index.php?pg=lwebis)
  yacybot (freeworld/global; amd64 Linux 3.0.0-17-generic; java 1.6.0_23; America/en) http://yacy.net/bot.html
  Mozilla/5.0 (Windows NT 6.1; WOW64; rv:46.0) Gecko/20100101 Firefox/46.0 PTST/279
  Mozilla/5.0 (compatible; MSIE or Firefox mutant; not on Windows server;) Daumoa/4.0
  Mozilla/5.0 (compatible; YandexMetrika/2.0; +http://yandex.com/bots mtmon01i.yandex.ru)
  Y!J-BSC/1.0 crawler (http://help.yahoo.co.jp/help/jp/blog-search/)
  Mozilla/5.0 (compatible; UptimeRobot/1.0; http://www.uptimerobot.com/)
  NG-Search/0.86 (+http://www.ng-search.com)
  ichiro/3.0 (http://help.goo.ne.jp/help/article/1142)
  TwengaBot/1.1 (+http://www.twenga.com/bot.html)
  WebImages 0.3 ( http://herbert.groot.jebbink.nl/?app=WebImages )
  Mozilla/5.0 (compatible; aiHitBot/2.8; +http://endb-consolidated.aihit.com/)
  RyzeCrawler/1.1.1 (+http://www.domain2day.nl/crawler/)
  Mozilla/5.0 (Nekstbot; http://www.ipipan.waw.pl/nekst/nekstbot/)
  Mozilla/5.0 (compatible; adidxbot/2.0; http://www.bing.com/bingbot.htm)
  Mozilla/5.0 (compatible; SWEBot/1.0; +http://swebot-crawler.net)
  WatchMouse/8.4.0.3 (http://watchmouse.com/ ; gblon01.watchmouse.net)
  yacybot (/global; amd64 Windows 8.1 6.3; java 1.7.0_55; Europe/de) http://yacy.net/bot.html
  Page Analyzer v4.0 ( http://www.ranks.nl/ )
  web_bh ([email protected])
  findlinks/1.1.6-beta3 (+http://wortschatz.uni-leipzig.de/findlinks/)
  Mozilla/5.0 (compatible; Findxbot/1.0; +http://www.findxbot.com)
  findlinks/2.0 (+http://wortschatz.uni-leipzig.de/findlinks/)
  Mozilla/5.0 (compatible; imagecoccoc/1.0; +http://help.coccoc.com/)
  PagesInventory (robot http://www.pagesinventory.com)
  Mozilla/5.0 (compatible; aiHitBot/1.0-DS; +http://www.aihit.com/)
  tagSeoBot/1.0 (http://www.tagseoblog.de/tools)
  Mozilla/5.0 (en-us) AppleWebKit/537.36 (KHTML, like Gecko; Google PP Default) Chrome/27.0.1453 Safari/537.36
  404 Checker [http://www.404checker.com/user-agent]
  CopperEgg/RevealUptime/
  Mozilla/5.0 (compatible; adidxbot/2.0; +http://www.bing.com/bingbot.htm)
  Mozilla/5.0 (compatible; ZumBot/1.0; http://help.zum.com/inquiry)
  CopperEgg/RevealUptime/TokyoJapan
  Mozilla/5.0 (compatible; MojeekBot/0.2; http://www.mojeek.com/bot.html)
  GoSquared-Status-Checker/0.2
  Mozilla/5.0 (Windows NT 6.0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/99.0 Safari/537.36 YottaaMonitor
  WatchMouse/18990 (http://watchmouse.com/ ; bc.watchmouse.com)
  yacybot (freeworld/global; amd64 Linux 3.2.1-gentoo-r2; java 1.6.0_24; Europe/de) http://yacy.net/bot.html
  Mozilla/5.0 (compatible; adbeat_bot; [email protected]; [email protected])
  linkdexbot/Nutch-1.0-dev (http://www.linkdex.com/; crawl at linkdex dot com)
  Heurekabot-Feed/1.0 (+http://sluzby.heureka.cz/napoveda/heurekabot/)
  Mozilla/5.0 (compatible; Charlotte/1.1; http://www.searchme.com/support/)
  yacybot (/global; amd64 Linux 3.10.0-229.7.2.el7.x86_64; java 1.8.0_45; Europe/en) http://yacy.net/bot.html
  LSSRocketCrawler/1.0 LightspeedSystems
  Mozilla/5.0 (X11; U; Linux Core i7-4980HQ; de; rv:32.0; compatible; Jobboerse.com; http://www.xn--jobbrse-d1a.com) Gecko/20100401 Firefox/24.0
  findlinks/2.2 (+http://wortschatz.uni-leipzig.de/findlinks/)
  Mozilla/5.0 (compatible; kulturarw3 +http://www.kb.se/om/projekt/Svenska-webbsidor---Kulturarw3/)
  Mozilla/5.0 (compatible; CloudFlare-AlwaysOnline/1.0; +http://www.cloudflare.com/always-online) AppleWebKit/534.34
  Mozilla/5.0 (compatible; SemrushBot/0.96.4; +http://www.semrush.com/bot.html)
  woobot/2.0
  GarlikCrawler/1.2 (http://garlik.com/, [email protected])
  yacybot (webportal-global; amd64 Linux 3.2.0-4-amd64; java 1.7.0_67; Europe/en) http://yacy.net/bot.html
  Mozilla/5.0 (compatible; AboutUsBot Johnny5/2.0; +http://www.AboutUs.org/)
  yacybot (/global; amd64 Linux 3.10.0-327.22.2.el7.x86_64; java 1.7.0_101; Etc/en) http://yacy.net/bot.html
  Mozilla/5.0 (Linux; Android 4.1.2; Galaxy Nexus Build/JZO54K; GTmetrix http://gtmetrix.com/) AppleWebKit/537.22 (KHTML, like Gecko) Chrome/26.0.1410.58 Mobile Safari/537.22
  Nuhk/2.4 ( http://www.neti.ee/cgi-bin/abi/Otsing/Nuhk/)
  Grahambot/0.1 (+http://www.sunaga-lab.com/graham-bot)
  yacybot (/global; amd64 Linux 3.16.0-4-amd64; java 1.7.0_91; Europe/de) http://yacy.net/bot.html
  SEO Consulting; Redirect Checker Tool V.02; IP:
  Mozilla/5.0 (compatible; Pro Sitemaps Generator; https://pro-sitemaps.com) Gecko Pro-Sitemaps/1.0
  yacybot (/global; amd64 Linux 4.4.10-antix.1-amd64-smp; java 1.8.0_101; Europe/en) http://yacy.net/bot.html
  Mozilla/5.0 (Windows NT 6.2; WOW64) Runet-Research-Crawler (itrack.ru/research/cmsrate; [email protected])
  Mozilla/5.0 (compatible; Crawler/0.9; http://linkfluence.net/)
  ADmantX Platform Semantic Analyzer US Async - ADmantX Inc. - www.admantx.com - [email protected]
  GetProxi.es-bot/1.1 (http://getproxi.es/spiderinfo/)
  Mozilla/5.0 (Windows NT 6.3; WOW64; rv:46.0) Gecko/20100101 Firefox/46.0 PTST/277
  Pinterest/0.2 (+http://www.pinterest.com/)
  CopperEgg/RevealUptime/AtlantaGA(linode)
  OdklBot/1.0 ([email protected])
  Mozilla/5.0 (compatible; Exabot-Images/3.0; +http://www.exabot.com/go/robot)
  yacybot (freeworld/global; amd64 Linux 3.2.1-gentoo-r2; java 1.6.0_22; Europe/de) http://yacy.net/bot.html
  MojeekBot/0.2 (archi; http://www.mojeek.com/bot.html)
  Sogou web spider/4.05252A
  Mozilla/5.0 (Windows NT 6.3; WOW64; rv:45.0) Gecko/20100101 Firefox/45.0 PTST/276
  Mozilla/5.0 (compatible; MotoMinerBot/1.0; +https://motominer.com/Bot)
  NG/2.0
  Mozilla/5.0 (compatible; heritrix/1.14.2 +http://www.webarchiv.cz)
  StackRambler/2.0 (MSIE incompatible)
  Baiduspider+(+http://www.baidu.jp/spider/)
  yacybot (freeworld/global; amd64 Windows 7 6.1; java 1.6.0_25; Europe/de) http://yacy.net/bot.html
  Mozilla/5.0 (compatible; houzzbot; +http://www.houzz.com/)
  Woko robot 3.0
  Mozilla/5.0 (compatible; Qwantify/2.0; +https://www.qwant.com/)
  yacybot (/global; amd64 Linux 4.2.0-27-generic; java 1.8.0_66-internal; America/en) http://yacy.net/bot.html
  ADmantX Platform Semantic Analyzer - APAC - ADmantX Inc. - www.admantx.com - [email protected]
  Mozilla/5.0 (compatible; Uptimebot/0.2.40; +http://www.uptime.com/uptimebot)
  Mozilla/5.0 (compatible; ExpertSearchSpider +http://www.expertsearch.nl/spider)
  Mozilla/5.0 (compatible; coccocbot-web/1.0; +http://help.coccoc.com/searchengine)
  Acoon v4.10.4 (www.acoon.de)
  Mozilla/5.0 (compatible; memoryBot/1.20.210 +http://internetmemory.org/en/)
  Readability/740ec9 - http://readability.com/about/
  Mozilla/5.0 (compatible; Apercite; +http://www.apercite.fr/robot/index.html)
  yacybot (i386 Linux 2.6.28-gentoo-r5; java 1.5.0_18; Europe/en) http://yacy.net/bot.html
  Mozilla/5.0 (compatible; suggybot v0.01a, http://blog.suggy.com/was-ist-suggy/suggy-webcrawler/)
  yacybot (amd64 Windows 7 6.1; java 1.6.0_14; Europe/de) http://yacy.net/bot.html
  yacybot (freeworld/global; amd64 Linux 3.3.4-1-ARCH; java 1.6.0_24; Europe/en) http://yacy.net/bot.html
  Mozilla/5.0 (compatible; AportWorm/3.2; +http://www.aport.ru/help)
  Mozilla/5.0 (compatible; memoryBot/1.20.235 +http://internetmemory.org/en/)
  findlinks/2.6 (+http://wortschatz.uni-leipzig.de/findlinks/)
  Mozilla/5.0 (compatible; Hailoobot/1.2; +http://www.hailoo.com/spider.html)
  eCommerceBot (http://www.ehandel.se/botinfo.html)
  Mozilla/5.0(compatible;Sosospider/2.0;+http://help.soso.com/webspider.htm)
  yacybot (/global; amd64 Linux 4.3.0-gentoo-ARCH; java 1.7.0_85; Europe/en) http://yacy.net/bot.html
  Nymesis/2.0 (http://nymesis.com)
  CopperEgg/RevealUptime/OregonUSA
  uclassify.com/1.0
  Mozilla/5.0 (compatible; Butterfly/1.0; +http://labs.topsy.com/butterfly.html) Gecko/2009032608 Firefox/3.0.8
  Mozilla/5.0 (compatible; Prlog/1.0; +http://prlog.ru/)
  Slack-ImgProxy 1.106 (+https://api.slack.com/robots)
  AdnormCrawler www.adnorm.com/crawler
  Mozilla/5.0 (compatible; YandexZakladki/3.0; +http://yandex.com/bots)
  Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/49.0.2623.75 Safari/537.36 Google Favicon
  Mozilla/5.0 (compatible; Sonic/1.0; http://www.yama.info.waseda.ac.jp/~crawler/info.html)
  Mozilla/5.0 (iPhone; U; CPU iPhone OS 4_1 like Mac OS X; en-us) AppleWebKit/532.9 (KHTML, like Gecko) Version/4.0.5 Mobile/8B117 Safari/6531.22.7 (compatible; Googlebot-Mobile/2.1; +http://www.google.com/bot.html)
  ICC-Crawler/2.0 (Mozilla-compatible; ; http://kc.nict.go.jp/project1/crawl.html)
  Mozilla/4.0 ([email protected])
  bot-pge.chlooe.com/1.0.0 (+http://www.chlooe.com/)
  Mozilla/5.0 (compatible; GroupHigh/1.0; +http://www.grouphigh.com/)
  Mozilla/5.0 (compatible; Webmaster tools +http://sitexy.com/)
  yacybot (/global; amd64 Windows 8.1 6.3; java 1.8.0_40; Europe/de) http://yacy.net/bot.html
  Mozilla/5.0 (compatible; MJ12bot/v1.4.1; http://www.majestic12.co.uk/bot.php?+)
  Mozilla/5.0 (compatible; spbot/4.0.6; +http://www.seoprofiler.com/bot )
  Mozilla/5.0 (compatible; EuripBot/2.0; +http://www.eurip.com)
  findlinks/2.1 (+http://wortschatz.uni-leipzig.de/findlinks/)
  Sogou web spider/4.025251
  SETOOZBOT/5.0 ( http://www.setooz.com/bot.html )
  Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0) ([email protected])
  Mozilla/5.0 (compatible; spbot/2.0.4; +http://www.seoprofiler.com/bot )
  Mozilla/5.0 (TweetmemeBot/4.0; +http://datasift.com/bot.html) Gecko/20100101 Firefox/31.0
  mozilla/5.0 ([email protected])
  Mozilla/5.0 (compatible; spbot/2.1; +http://www.seoprofiler.com/bot )
  Mozilla/5.0 (Windows NT 6.2; WOW64) AppleWebKit/537.4 (KHTML, like Gecko) Chrome/98 Safari/537.4 (StatusCake SSL Monitor)
  Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; Trident/5.0; PTST 2.385)
  Mozilla/5.0 (compatible; evc-batch/2.0.20160608212921)
  Mozilla/5.0 (compatible; Mail.RU_Bot/2.0)
  seebot/2.0 (+http://www.seegnify.com/bot)
  bl.uk_lddc_bot/3.3.0-LBS-2016-02 (+http://www.bl.uk/aboutus/legaldeposit/websites/websites/faqswebmaster/index.html)
  CommaFeed/2.3.0-SNAPSHOT (https://www.commafeed.com)
  OmniExplorer_Bot/5.91c (+http://www.omni-explorer.com) WorldIndexer
  hledejLevne.cz/2.0
  page_verifier (http://www.securecomputing.com/goto/pv)
  url_test ([email protected])
  Mozilla/5.0 (X11; Linux x86_64; rv:10.0.12) Gecko/20100101 Firefox/21.0 WordPress.com mShots
  Mozilla/5.0 (compatible; Linux x86_64; Mail.RU_Bot/2.0; +http://go.mail.ru/help/robots)
  Is is up? (+http://isitup.org)
  Metaspinner/0.01 (Metaspinner; http://www.meta-spinner.de/; [email protected]/)
  TwengaBot-2.0 Champigny (+http://www.twenga.com/bot.html)
  LivelapBot/0.2 (http://site.livelap.com/crawler)
  HubSpot Crawler 1.0 http://www.hubspot.com/
  Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/45.0.2454.101 Safari/537.36 TinEye/1.0 (via http://www.tineye.com/)
  yacybot (/global; x86 Windows 10 10.0; java 1.8.0_73; Europe/de) http://yacy.net/bot.html
  Scrapy/0.24.6 (+http://scrapy.org)
  FAST-WebCrawler/3.6/FirstPage (atw-crawler at fast dot no;http://fast.no/support/crawler.asp)
  Baiduspider+(+http://www.baidu.com/search/spider_jp.html)
  Mozilla/5.0 (compatible; seplinkbot/1.0 )
  Mozilla/5.0 (compatible; Falconsbot; +http://ws.nju.edu.cn/falcons/)
  Mozilla/5.0 (Windows NT 6.1; WOW64; rv:46.0) Gecko/20100101 Firefox/46.0 PTST/277
  Mozilla/4.0 (Toread-Crawler/1.1; +http://news.toread.cc/crawler.php)
  TinEye-bot/0.02 (see http://www.tineye.com/crawler.html)
  yacybot (freeworld/global; i386 Linux 2.6.32-39-generic-pae; java 1.6.0_20; Europe/en) http://yacy.net/bot.html
  Mozilla/5.0 (compatible; parsijoo; +http://www.parsijoo.ir/; [email protected])
  Mozilla/5.0 (compatible; spbot/4.4.0; +http://OpenLinkProfiler.org/bot )
  MaxPoint Bot (+http://www.maxpoint.com)
  Mozilla/5.0 (compatible; Infohelfer/1.2.0; +http://www.infohelfer.de/crawler.php)
  ExB Language Crawler 2.1.1 (+http://www.exb.de/crawler)
  cg-eye interactive
  ZumBot/1.0 (ZUM Search; http://help.zum.com/inquiry)
  rogerbot/1.1 (http://moz.com/help/guides/search-overview/crawl-diagnostics#more-help, [email protected])
  Mozilla/5.0 (compatible; Crawlera/1.10.2; UID 47129)
  Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:14.0) Gecko/20100101 Firefox/14.0.1 AppEngine-Google; (+http://code.google.com/appengine; appid: s~metacdn-hr)
  Mozilla/5.0 (compatible; WoTBoT; +https://www.wslta.com/WoTBoT.html)
  HolmesBot (http://holmes.ge)
  Baiduspider-image+(+http://www.baidu.com/search/spider.htm)
  Mozilla/5.0 (compatible; alexa site audit/1.0; +http://www.alexa.com/help/webmasters; [email protected])
  Mediapartners-Google
  Mozilla/5.0 (compatible; MFGPagesBot/2.1; http://www.mfgpages.com)
  larbin_2.6.2 [email protected]
  Mozilla/5.0 (Windows; U; Windows NT 6.0; en-GB; rv:1.0; trendictionbot0.5.0; trendiction search; http://www.trendiction.de/bot; please let us know of any problems; web at trendiction.com) Gecko/20071127 Firefox/3.0.0.11
  BlogPulseLive ([email protected])
  WeSEE:Search/0.1 (Alpha, http://www.wesee.com/en/support/bot/)
  yacybot (freeworld/global; i386 Linux 3.0.0-17-generic; java 1.6.0_23; America/en) http://yacy.net/bot.html
  ImplisenseBot 1.0
  Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; WOW64; Trident/5.0; PTST 2.386)
  gonzo2[P] +http://www.suchen.de/faq.html
  Mozilla/5.0 (compatible; LXRbot/1.0; http://lxrseo.com/, [email protected])
  Mozilla/5.0 (compatible; Arachnophilia/1.0; +http://arachnys.com/)
  Mozilla/5.0 (compatible; CloudServerMarketSpider/1.0; +http://cloudservermarket.com/spider.html)
  kalooga/KaloogaBot (Kalooga; http://www.kalooga.com/info.html?page=crawler)
  yacybot (webportal/global; x86_64 Mac OS X 10.9.2; java 1.6.0_65; Europe/de) http://yacy.net/bot.html
  Mozilla/5.0 (compatible; DotBot/1.1; http://www.dotnetdotcom.org/, [email protected])
  yacybot (/global; amd64 Linux 4.1.19-gentoo; java 1.7.0_95; Europe/pl) http://yacy.net/bot.html
  Mozilla/5.0 (Windows NT 6.3; WOW64; Trident/7.0; rv:11.0) like Gecko PTST/276
  Mozilla/5.0 (compatible; RankActiveLinkBot; +https://rankactive.com/resources/rankactive-linkbot)
  audisto.com full crawler 3.26.431 (refer to in robots.txt as audisto, see https://audisto.com/bot)
  Mozilla/5.0 (compatible; spbot/4.0.1; +http://www.seoprofiler.com/bot )
  Mozilla/5.0 (compatible; OpenindexShallowSpider/Nutch-1.5-dev; +http://www.openindex.io/en/webmasters/spider.html)
  mozilla/5.0 (compatible; discobot/1.1; +http://discoveryengine.com/discobot.html)
  gonzo2[p] (+http://www.suchen.de/faq.html)
  Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; Trident/5.0); 360Spider(compatible; HaosouSpider; http://www.haosou.com/help/help_3_2.html)
  Mozilla/5.0 (compatible; spbot/2.0.1; +http://www.seoprofiler.com/bot/ )
  WatchMouse/18990 (http://watchmouse.com/ ; uk)
  yacybot (freeworld/global; amd64 Linux 3.1.10-hardened; java 1.7.0_03-icedtea; Europe/en) http://yacy.net/bot.html
  VeBot (+http://www.veinteractive.com/vebot)
  Mozilla/5.0 (compatible; NLNZ_IAHarvester2013 +http://natlib.govt.nz/about-us/current-initiatives/web-harvest-2012)
  findlinks/1.1.3-beta8 (+http://wortschatz.uni-leipzig.de/findlinks/)
  Mozilla/5.0 (Linux; U; Android 2.3.4; generic) AppleWebKit/537.36 (KHTML, like Gecko; Google Web Preview) Version/4.0 Mobile Safari/537.36
  DialogSearch.com Bot 1.4;http://dialogsearch.com/webmasters
  Mozilla/5.0 (compatible; GurujiBot/1.0; +http://www.guruji.com/en/WebmasterFAQ.html)
  Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9) AppleWebKit/537.71 (KHTML, like Gecko) Version/7.0 Safari/537.71 (Rival IQ, rivaliq.com)
  wscheck.com/1.0.0 (+http://wscheck.com/)
  Mozilla/5.0 (Windows NT 6.3; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/30.0.1599.69 Safari/537.36 Webthumb/2.0
  Mozilla/5.0 (compatible; FlipboardProxy/1.1; +http://flipboard.com/browserproxy)
  WeSEE:Ads/PictureBot (http://www.wesee.com/bot/)
  Mozilla/5.0 (compatible; Dataprovider/6.101; +https://www.dataprovider.com/)
  Kyoto-Crawler/n1.0 (Mozilla-compatible; [email protected]; http://nlp.ist.i.kyoto-u.ac.jp/?crawling)
  Mozilla/5.0 (compatible; SEOkicks-Robot; +http://www.seokicks.de/robot.html)
  Mozilla/5.0 (compatible; spbot/4.0.9; +http://OpenLinkProfiler.org/bot )
  Mozilla/5.0 (compatible; spbot/4.2.0; +http://OpenLinkProfiler.org/bot )
  seo-nastroj.cz
  LoadImpactPageAnalyzer/1.3.0 (Load Impact; http://loadimpact.com/)
  CSS Certificate Spider (http://www.css-security.com/certificatespider/)
  MetaGeneratorCrawler/1.3.2 (www.metagenerator.info)
  Testomatobot/1.0 (Linux x86_64; +http://www.testomato.com/testomatobot) minicrawler/4.0.0~beta8
  BLEXBot
  Mozilla/2.0 (compatible; Ask Jeeves/Teoma)
  Testomatobot/1.0 (Linux x86_64; +http://www.testomato.com/testomatobot) minicrawler/4.0.0~beta7
  Mozilla/5.0 (compatible; XoviBot/2.0; +http://www.xovibot.net/)
  Mozilla/5.0 (compatible; spbot/4.4.1; +http://OpenLinkProfiler.org/bot )
  ia_archiver (+http://www.alexa.com/site/help/webmasters; [email protected])
  HeartRails Robot/0.1 (http://www.heartrails.com)
  Mozilla/5.0 (compatible; Faveeo/1.0; +http://www.faveeo.com)
  yacybot (/global; amd64 Linux 3.16.0-49-generic; java 1.7.0_79; Europe/en) http://yacy.net/bot.html
  Castabot/0.1 (+http://topixtream.com/)
  Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.9.0.5) Gecko/2010033101 Gentoo Firefox/3.0.5 (Dot TK - spider 3.0)
  Mozilla/5.0 (compatible; MSIE or Firefox mutant; not on Windows server; +http://ws.daum.net/aboutWebSearch.html) Daumoa/2.0
  istellabot-nutch/Nutch-1.10
  Mail.RU/2.0
  ichiro/2.0 (http://help.goo.ne.jp/door/crawler.html)
  LexxeBot/1.0 ([email protected])
  Mozilla/5.0 (iPhone; CPU iPhone OS 8_1 like Mac OS X) AppleWebKit/600.1.4 (KHTML, like Gecko) Version/8.0 Mobile/12B410 Safari/600.1.4 (Applebot/0.1; +http://www.apple.com/go/applebot)
  Mozilla/5.0 (compatible; FlipboardRSS/1.1; +http://flipboard.com/browserproxy)
  Mozilla/5.0 (compatible; Linux x86_64; Mail.RU_Bot/Robots/2.0; +http://go.mail.ru/help/robots)
  yacybot (-global; amd64 Linux 3.10.0-229.4.2.el7.x86_64; java 1.7.0_79; Europe/en) http://yacy.net/bot.html
  it2media-domain-crawler/1.0 on crawler-prod.it2media.de
  yacybot (/global; amd64 Windows 8.1 6.3; java 1.8.0_25; Europe/de) http://yacy.net/bot.html
  yacybot (freeworld-global; amd64 Linux 3.16.0-4-amd64; java 1.7.0_79; Europe/de) http://yacy.net/bot.html
  Mozilla/5.0 (WhatsMyIP.org HTTP_Headers) http://whatsmyip.org/ua
  checkgzipcompression.com robot
  Mozilla/5.0 eCairn-Grabber/1.0 (+http://ecairn.com/grabber)
  RankurBot/3.3 (+http://rankur.com)
  L.webis/0.50 (http://webalgo.iit.cnr.it/index.php?pg=lwebis)
  Speedy Spider (Submit your site at http://www.entireweb.com/free_submission/)
  oBot
  Snappy/2.0 ( http://www.urltrends.com/ )
  Mozilla/5.0 (compatible; MSIE 10.0; Windows NT 6.2; WOW64; Trident/6.0) CrawlerProcess (http://www.PowerMapper.com) /5.23.770.0
  Mozilla/5.0 (compatible; alexa site audit/1.0; +http://www.alexa.com/help/webmasters; )
  flatlandbot/baypup (Flatland Industries Web Spider; http://www.flatlandindustries.com/flatlandbot; [email protected])
  istellabot/Nutch-1.11
  GetintentCrawler getintent.com
  Covario-IDS/1.0 (Covario; http://www.covario.com/ids; support at covario dot com)
  Mozilla/5.0 (compatible; FatBot 2.0; http://www.thefind.com/crawler)
  Mozilla/5.0 (compatible; MegaIndex.com/2.0; +http://megaindex.com/crawler)
  Microsearch.ru/1.0; http://microsearch.ru/webmasters
  TurnitinBot/3.0 (http://www.turnitin.com/robot/crawlerinfo.html)
  BacklinkCrawler V (http://www.backlinktest.com/crawler.html)
  FeedlyBot/1.0 (http://feedly.com)
  Clickagy Intelligence Bot v2
  GetURLInfo/1.0
  DoCoMo/2.0 P900i(c100;TB;W24H11)(compatible; ichiro/mobile goo;+http://help.goo.ne.jp/door/crawler.html)
  Scrapy/1.1.0 (+http://scrapy.org)
  yacybot (/global; amd64 Linux 3.16.0-4-amd64; java 1.7.0_75; America/en) http://yacy.net/bot.html
  Mozilla/5.0 (compatible; VSAgent/1.2)
  Experibot_v1 [goo.gl/n6zrAf]
  Mozilla/5.0 (compatible; DNS-Digger-Explorer/1.0; +http://www.dnsdigger.com)
  boitho.com-robot/1.0
  yacybot (/global; amd64 Windows Server 2012 6.2; java 1.7.0_51; Europe/de) http://yacy.net/bot.html
  RelateIQ Crawler www.relateiq.com
  Mozilla/4.0 (compatible; Netcraft Web Server Survey)
  Mozilla/5.0 (X11; U; Linux Core i7-4980HQ; de; rv:32.0; compatible; JobboerseBot; https://www.jobboerse.com/bot.htm) Gecko/20100101 Firefox/38.0
  yacybot (i386 Linux 2.6.9-023stab046.2-smp; java 1.6.0_05; Europe/en) http://yacy.net/bot.html
  Scrubby/2.2 (http://www.scrubtheweb.com/)
  Mozilla/5.0 (Yahoo-MMCrawler/4.0; mailto:[email protected])
  Fetch/2.0a (CMS Detection/Web/SEO analysis tool, see http://guess.scritch.org)
  bitlybot/3.0 (+http://bit.ly/)
  Mozilla/5.0 (compatible; heritrix/1.14.4 +http://www.exif-search.com)
  yacybot (freeworld/global; amd64 Windows Server 2012 6.2; java 1.7.0_25; Europe/de) http://yacy.net/bot.html
  agentslug.com - website monitoring tool
  SafeDNSBot (https://www.safedns.com/searchbot)
  Feedbin
  Mozilla/5.0 (compatible; proximic; +http://www.proximic.com)
  Mozilla/5.0 (compatible; AcoonBot/4.11.1; +http://www.acoon.de/robot.asp)
  Mozilla/5.0 (compatible; MagiBot/3.4.3; +http://magi.peak-labs.com/robots.txt)
  Mozilla/5.0 (compatible; XML Sitemaps Generator; http://www.xml-sitemaps.com) Gecko XML-Sitemaps/1.0
  baypup/colbert (Baypup; http://sf.baypup.com/webmasters; [email protected])
  MergeFlow-PageReader/0.91;+(+http://mergeflow.net/info/pagereader) Mozilla/5.0 (Windows) compatible
  Mozilla/5.0 (Mobile; rv:18.0) Gecko/18.0 Firefox/18.0 commoncrawl.org/research//Nutch-1.7-SNAPSHOT
  Mozilla/5.0 (compatible; spbot/4.1.0; +http://OpenLinkProfiler.org/bot )
  Mozilla/5.0 (compatible; OsO; http://oso.octopodus.com/abot.html)
  Mozilla/5.0 (compatible; gofind; +http://govid.mobi/bot.php)
  CatchBot/2.0; +http://www.catchbot.com
  Mozilla/5.0 (compatible; BusinessSeek.biz_Spider; http://www.businessseek.biz/)
  Quora Link Preview/1.0 (http://www.quora.com)
  radian6_default_(www.radian6.com/crawler)
  DWDS-Crawler +http://odo.dwds.de/dwds-crawler.html
  Mozilla/5.0 (compatible; DuckDuckGo-Favicons-Bot/1.0; +http://duckduckgo.com)
  sogou spider
  Mozilla/5.0 (compatible; GigablastOpenSource/1.0)
  Mozilla/5.0 (compatible; HomeTags/1.0; +http://www.hometags.nl/bot)
  CorpusCrawler 2.0.24 (http://corpora.fi.muni.cz/crawler/);Project:CzCorpus
  Scooter/3.3
  Zookabot/2.5;++http://zookabot.com
  Mozilla/5.0 (iPhone; CPU iPhone OS 8_1 like Mac OS X) AppleWebKit/600.1.4 (KHTML, like Gecko) Version/8.0 Mobile/12B411 Safari/600.1.4 (compatible; YandexMobileBot/3.0; +http://yandex.com/bots)
  Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/534.34 (KHTML, like Gecko) Safari/534.34; +http://sniptracker.com
  Mozilla/5.0 (compatible; archive.org_bot; Wayback Machine Live Record; +http://archive.org/details/archive.org_bot)
  es_com_viewer ([email protected])
  Mozilla/5.0 (compatible; Uptimebot/0.2.14; +http://www.uptime.com/uptimebot)
  yacybot (freeworld/global; amd64 Linux 3.0.0-14-generic; java 1.6.0_23; Europe/en) http://yacy.net/bot.html
  Mozilla/5.0 (X11; Linux x86_64; rv:35.0) Gecko/20100101 Firefox/35.0 DareBoost
  yacybot (freeworld/global; amd64 Linux 2.6.32-41-server; java 1.6.0_26; Europe/de) http://yacy.net/bot.html
  yacybot (amd64 Linux 2.6.28-18-generic; java 1.6.0_16; GMT/en) http://yacy.net/bot.html
  Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.1; Trident/4.0; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; Media Center PC 6.0; .NET4.0C; .NET4.0E; PTST 2.386)
  KD Bot
  Mozilla/5.0 (X11; U; Linux i686 (x86_64); en-US; rv:1.9.2.19) Gecko WebThumb/1.0
  Mozilla/5.0 (compatible; Yahoo! Slurp/3.0; http://help.yahoo.com/help/us/ysearch/slurp)
  rogerbot/1.0 (http://moz.com/help/pro/what-is-rogerbot-, [email protected])
  MJ12bot/v1.2.0 (http://majestic12.co.uk/bot.php?+)
  Superarama.com-Tarama-Botu-v.01
  Mozilla/5.0 (compatible; Urlfilebot/2.2; +http://urlfile.com/bot.html)
  Vegi bot (we follow your robots.txt settings before crawling, you can slow down the bot by change the Crawl-Delay parameter in the settings.if you have an enquiry, please email to: [email protected])
  MnoGoSearch/3.3.9
  Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/534+ (KHTML, like Gecko) BingPreview/1.0b
  JyxobotRSS/0.06
  Mozilla/5.0 (compatible; BigBozz/2.2.1; +http://www.bigbozz.com/)
  Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/534.57.2 (KHTML, like Gecko) Version/5.1.7 Safari/534.57.2 PTST/277
  KiNShooboT (compatible; KiNShooboT/1.0.C; +http://www.kinshoo.com/bot.html)
  SentiBot www.sentibot.eu (compatible with Googlebot)
  Topicbot/1.0 (Mozilla;I;+http://92.42.190.57/)
  IDG/IT (http://spaziodati.eu/)
  LinkAider (http://linkaider.com/crawler/)
  Mozilla/5.0 (compatible; coccoc/1.0; +http://help.coccoc.com/searchengine)
  yacybot (/global; amd64 Linux 2.6.32-573.3.1.el6.x86_64; java 1.7.0_85; Europe/en) http://yacy.net/bot.html
  bitlybot
  Mozilla/5.0 (compatible; Mp3Bot/0.7; +http://mp3realm.org/mp3bot/)
  WWW::LayeredExtractor::Handler::Feed/0.01
  Mozilla/5.0 (compatible; SWEBot/1.0; +http://swebot.net)
  Mozzila/5.0 (compatible; Sonic/1.0; http://www.yama.info.waseda.ac.jp/~crawler/info.html)
  MetaTagRobot/2.1 (http://www.seocentro.com/tools/search-engines/metatag-analyzer.html)
  Mozilla/5.0 (compatible; Goodzer/1.0)
  yacybot (/global; amd64 Linux 3.16.0-4-amd64; java 1.8.0_40; Europe/en) http://yacy.net/bot.html
  Mozilla/5.0 (Windows; U; Windows NT 6.1; en-GB; rv:1.9.2.3) Gecko/20100401 Firefox/3.6.3 (NetShelter ContentScan)
  Facebot/1.0
  SafeAds.xyz bot
  Mozilla/5.0 (compatible; Yeti/1.1; +http://help.naver.com/robots/)
  yacybot (/global; amd64 Windows 7 6.1; java 1.7.0_55; Europe/en) http://yacy.net/bot.html
  scrapyproject (+http://www.profound.net)
  Mozilla/5.0 (compatible; Online Domain Tools - Online Website Link Checker/1.2; +http://website-link-checker.online-domain-tools.com)
  Mozilla/5.0 (compatible; heritrix/2.0.2 +http://aihit.com)
  Mozilla/5.0 (compatible; Scrubby/3.1; +http://www.scrubtheweb.com/help/technology.html)
  Semantifire1/0.20 ( http://www.setooz.com/oozbot.html ; agentname at setooz dot_com )
  Mozilla/5.0 (X11; Ubuntu; Linux i686; rv:14.0; ips-agent) Gecko/20100101 Firefox/14.0.1
  UnisterBot (Mozilla/5.0 compatible; [email protected])
  Mozilla/5.0 (compatible; Ezooms/1.0; [email protected])
  Mozilla/5.0 (compatible; YandexVideo/3.0; +http://yandex.com/bots)
  Mozilla/5.0 (compatible; spbot/4.4.2; +http://OpenLinkProfiler.org/bot )
  findlinks/2.1.3 (+http://wortschatz.uni-leipzig.de/findlinks/)
  Mozilla/5.0 (compatible; MJ12bot/v1.2.5; http://www.majestic12.co.uk/bot.php?+)
  Speedy Spider (Entireweb; Beta/1.2; http://www.entireweb.com/about/search_tech/speedyspider/)
  Mozilla/5.0 (compatible; kazbtbot/0.1; +http://kazbt.com/)
  Orgbybot/OrgbyBot v1.3 (Spider; http://orgby.com/bot/ ; Orgby.com Search Engine)
  Mozilla/5.0 (compatible; aiHitBot-DM/2.0.2 +http://www.aihit.com)
  YahooSeeker-Testing/v3.9 (compatible; Mozilla 4.0; MSIE 5.5; http://search.yahoo.com/)
  Mozilla/5.0 (compatible; Crawlera/1.10.2; UID 40409)
  Mozilla/5.0 (compatible; image.coccoc/1.0; +http://help.coccoc.com/)
  yacybot (/global; amd64 Linux 3.19.2-1-ARCH; java 1.8.0_40; Europe/de) http://yacy.net/bot.html
  Mozilla/5.0 (compatible; evc-batch/2.0.20161017175820)
  datagnionbot (+http://www.datagnion.com/bot.html)
  CopperEgg/RevealUptime/TokyoJP(linode)
  MB-SiteCrawler
  WatchMouse/18990 (http://watchmouse.com/ ; liz)
  HybridBot (hybrid.ru/about. If our bot caused problems please contact us. Contact email: [email protected])
  Mozilla/5.0 (compatible; Crawlera/1.10.2; UID 24522)
  yacybot (/global; amd64 Linux 3.13.0-74-generic; java 1.7.0_91; Europe/en) http://yacy.net/bot.html
  findlinks/1.1.6-beta5 (+http://wortschatz.uni-leipzig.de/findlinks/)
  yacybot (freeworld/global; amd64 Linux 3.13.0-24-generic; java 1.7.0_55; Europe/de) http://yacy.net/bot.html
  Mozilla/5.0 (compatible; OpenfosBot/2.4; +http://www.openfos.com)
  Nusearch Spider (www.nusearch.com)
  WatchMouse/8.4.0.3 (http://watchmouse.com/ ; sesto02.watchmouse.net)
  ICC-Crawler/2.0 (Mozilla-compatible; ; http://www.nict.go.jp/en/univ-com/plan/crawl.html)
  Mozilla/5.0 (compatible; spbot/1.2; +http://www.seoprofiler.com/bot/ )
  Orbiter/1.3 (http://dailyorbit.com/)
  CCBot/2.0
  Mozilla/5.0 (compatible; linkdexbot/2.1; +http://www.linkdex.com/about/bots/)
  Mozilla/5.0 (compatible; DNS-Digger/1.0; +http://www.dnsdigger.com)
  AppEngine-Google; (+http://code.google.com/appengine; appid: s~feedly-social)
  yacybot (/global; x86 Windows 7 6.1; java 1.8.0_71; Europe/de) http://yacy.net/bot.html
  Mozilla/5.0 (Windows Phone 8.1; ARM; Trident/7.0; Touch; rv:11.0; IEMobile/11.0; NOKIA; Lumia 530) like Gecko (compatible; adidxbot/2.0; +http://www.bing.com/bingbot.htm)
  Dex Social Bot
  yacybot (/global; amd64 Linux 4.7.6-200.fc24.x86_64; java 1.8.0_102; Etc/en) http://yacy.net/bot.html
  Mozilla/5.0 (compatible; heritrix/1.14.3 +http://archive.org)
  CRAZYWEBCRAWLER 0.9.10, http://www.crazywebcrawler.com
  Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/50.0.2661.102 Safari/537.36 PTST/277
  Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.1.3; ips-agent) Gecko/20090824 Fedora/1.0.7-1.1.fc4 Firefox/3.5.3
  SafeSearch microdata crawler (https://safesearch.avira.com, [email protected])
  Zemanta Aggregator/0.9 +http://www.zemanta.com
  Mozilla/5.0 (compatible; special_archiver/3.2.0 +http://www.loc.gov/webarchiving/notice_to_webmasters.html)
  Mozilla/5.0 (compatible; GimmeUSAbot/1.0; +https://gimmeusa.com/crawler.html)
  yacybot (freeworld/global; amd64 Windows 7 6.1; java 1.7.0_45; Europe/de) http://yacy.net/bot.html
  Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.1 (KHTML, like Gecko) Chrome/21.0.1180.89 Safari/537.1; 360Spider
  FyberSpider/1.3 (http://www.fybersearch.com/fyberspider.php)
  yacybot (/global; amd64 FreeBSD 10.3-RELEASE; java 1.8.0_77; GMT/en) http://yacy.net/bot.html
  Searchie/1.0 (a Storm-based crawler; https://www.searchie.org; [email protected])
  yacybot (freeworld/global; amd64 Linux 3.1.10-1-desktop; java 1.6.0_22; Europe/de) http://yacy.net/bot.html
  holmes/3.12.4 (http://morfeo.centrum.cz/bot)
  WatchMouse/18990 (http://watchmouse.com/ ; se.watchmouse.com)
  MXT/Nutch-1.10 (http://t.co/GSRLLKex24; informatique at mixdata dot com)
  Mozilla/5.0 (compatible; Shareaholicbot/1.0; +http://www.shareaholic.com/bot)
  ZoomInformation Bot
  Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 5.1; www.alertra.com)
  Mozilla/5.0 (compatible; HomeTags/1.0; http://www.hometags.nl/bot)
  Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.2.17) Gecko/20110515 HeartRails_Capture/1.0.4 (+http://capture.heartrails.com/) Namoroka/3.6.17
  OmniExplorer_Bot/5.20 (+http://www.omni-explorer.com) WorldIndexer
  WatchMouse/18990 (http://watchmouse.com/ ; it)
  Mozilla/5.0 (compatible; imrbot/1.10.8 +http://www.mignify.com)
  Mozilla/5.0 (compatible; spbot/4.0.2; +http://www.seoprofiler.com/bot )
  Readability/1900e6 - http://readability.com/about/
  Mozilla/5.0 (compatible; Dataprovider/6.92; +https://www.dataprovider.com/)
  Mozilla/5.0 (iPhone; CPU iPhone OS 6_0_1 like Mac OS X) AppleWebKit/537.36 (KHTML, like Gecko; Google Page Speed Insights) Version/6.0 Mobile/10A525 Safari/8536.25
  VegeBot (we follow your robots.txt settings before crawling, you can slow down the bot by change the Crawl-Delay parameter in the settings.if you have an enquiry, please email to: [email protected])
  yacybot (webportal-global; amd64 Linux 3.2.0-4-amd64; java 1.7.0_65; Europe/en) http://yacy.net/bot.html
  FeedBucket/1.0 (+http://www.feedbucket.com)
  topster.de HTTP-Header 1.0
  MaxPointCrawler/Nutch-1.10 (maxpoint.crawler at maxpointinteractive dot com)
  Mozilla/5.0 (Windows NT 6.0; rv:45.0) Gecko/20100101 Firefox/45.0 PTST/276
  yacybot (/global; amd64 Windows 7 6.1; java 1.8.0_05; Europe/es) http://yacy.net/bot.html
  NETCRAFT
  Mozilla/5.0 (compatible; WbSrch/1.0; +https://wbsrch.com)
  WebCookies/1.0 (+http://webcookies.info/faq/#agent)
  netEstate Impressumscrawler (+http://www.netestate.de/De/Loesungen/Impressumscrawler)
  msnbot-media/2.0b (+http://search.msn.com/msnbot.htm)
  CopperEgg/RevealUptime/DublinIEUSA
  StormCrawler/1.0 (a Storm-based crawler; https://github.com/DigitalPebble/storm-crawler; [email protected])
  WatchMouse/8.4.0.3 (http://watchmouse.com/ ; uschi02.watchmouse.net)
  Mozilla/5.0 (Linux; Android 5.0.2; SM-G920T Build/LRX22G) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/40.0.2125.111 Mobile Safari/537.36 DareBoost
  Mozilla/5.0 (compatible; DeuSu/0.1.0; +https://deusu.org)
  Mozilla/5.0 (compatible; MJ12bot/v1.4.4 (domain ownership verifier); http://www.majestic12.co.uk/bot.php?+)
  Mozilla/5.0 (compatible; SemrushBot/0.99~bl; +http://www.semrush.com/bot.html)
  WebWatch/Robot_txtChecker
  Mozilla/5.0 (compatible; linkdexbot/2.2; +http://www.linkdex.com/bots/)
  Feedly/1.0 (+http://www.feedly.com/fetcher.html; like FeedFetcher-Google)
  Toweyabot: toweya.com
  Mozilla/5.0 (compatible; Infohelfer/1.4.3; +http://www.infohelfer.de/crawler.php)
  Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.2.1; aggregator:Spinn3r (Spinn3r 3.1); http://spinn3r.com/robot) Gecko/20021130
  FeedCatBot/3.0 (+http://www.feedcat.net/)
  LinkedInBot/1.0 (compatible; Mozilla/5.0; Apache-HttpClient +http://www.linkedin.com), libot/Nutch-1.9 (http://www.linkedin.com; [email protected])
  R6_CommentReader(www.radian6.com/crawler)
  Mozilla/5.0 (compatible; Crawlera/1.10.2; UID 70350)
  Mozilla/5.0 (compatible; MJ12bot/v1.2.3; http://www.majestic12.co.uk/bot.php?+)
  Domain Re-Animator Bot (http://domainreanimator.com) - [email protected]
  Riddler (http://riddler.io/about.html)
  Mozilla/5.0 (compatible; Esribot/1.0; http://www.esrihu.hu/)
  Favicon downloader (http://favicon.netk6.com/)
  findlinks/1.1.5-beta7 (+http://wortschatz.uni-leipzig.de/findlinks/)
  ConveraCrawler/0.9d (+http://www.authoritativeweb.com/crawl)
  Mozilla/5.0 (Windows NT 6.1; rv:6.0) Gecko/20110814 Firefox/6.0 Google favicon
  GetFoundBot (+http://www.getfound.cz/getfoundbot/)
  WatchMouse/8.4.0.3 (http://watchmouse.com/ ; usdal02.watchmouse.net)
  Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/48.0.2564.97 Safari/537.36 Yandex.Translate
  Mozilla/5.0 (compatible; NLNZ_IAHarvester2013 +http://natlib.govt.nz/about-us/current-initiatives/web-harvest-2013)
  COMODOSpider/Nutch-1.2
  Mozilla/5.0 (compatible; DomainMacroCrawler/0.1; +http://domainmacro.com)
  Mozilla/5.0 (compatible; heritrix/3.2.0 +http://www.exif-search.com)
  Mozilla/5.0 (compatible; MagiBot/3.6.2; +http://magi.peak-labs.com/robots.txt)
  Mozilla/5.0 (compatible; Alexabot/1.0; +http://www.alexa.com/help/certifyscan; [email protected])
  psbot-image (+http://www.picsearch.com/bot.html)
  Mozilla/5.0 (Macintosh; Intel Mac OS X 10.9; rv:28.0) Gecko/20100101 Firefox/28.0 (FlipboardProxy/1.6; +http://flipboard.com/browserproxy)
  larbin_2.6.4 ([email protected])
  yacybot (/global; amd64 Windows 7 6.1; java 1.7.0_55; Europe/ru) http://yacy.net/bot.html
  BacklinkCrawler (http://www.backlinktest.com/crawler.html)
  Mozilla/5.0 (Windows NT 6.1; WOW64) adbeat.com/policy AppleWebKit/537.36 (KHTML, like Gecko) Chrome/48.0.2564.116 Safari/537.36
  Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/534.57.2 (KHTML, like Gecko) Version/5.1.7 Safari/534.57.2 PTST/276
  Tools4noobs.com/1.0 Spider
  yacybot (freeworld/global; amd64 Linux 2.6.32-40-generic; java 1.6.0_20; Europe/de) http://yacy.net/bot.html
  Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US) Speedy Spider (http://www.entireweb.com/about/search_tech/speedy_spider/)
  istellabot/t.1
  Mozilla/5.0 (compatible; Lipperhey-Kaus-Australis/5.0; +https://www.lipperhey.com/en/about/)
  Mozilla/5.0 (compatible; Veoozbot/1.0; +http://www.veooz.com/veoozbot.html)
  SemrushBot/Nutch-1.5-SNAPSHOT
  Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)
  Mozilla/5.0 (X11; Linux x86_64; rv:47.0; GTmetrix https://gtmetrix.com/) Gecko/20100101 Firefox/47.0
  CopperEgg/RevealUptime/Oregon(aws)
  yacybot (/global; amd64 Linux 3.16.0-4-amd64; java 1.8.0_102; Europe/en) http://yacy.net/bot.html
  rogerbot/1.0 (http://www.seomoz.org/dp/rogerbot, [email protected])
  yacybot (amd64 Linux 2.6.26-2-amd64; java 1.6.0_20; Europe/en) http://yacy.net/bot.html
  Mozilla/5.0 (Windows NT 6.3; WOW64; Trident/7.0; rv:11.0; BingPreview/1.0b) like Gecko
  Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 5.1; Trident/4.0; .NET CLR 1.1.4322; .NET CLR 2.0.50727; .NET CLR 3.0.04506.30; .NET CLR 3.0.4506.2152; .NET CLR 3.5.30729; MDDS; PTST 2.386)
  Mozilla/5.0 (compatible; RavenCrawler/2.0; +https://raventools.com/seo-website-auditor/)
  Mozilla/5.0 (compatible; YandexMetrika/2.0; +http://yandex.com/bots)
  Mozilla/5.0 (compatible; Scrubby/3.2; +http://seotools.scrubtheweb.com/webpage-analyzer.html)
  dubaiindex (adressendeutschland.de)
  Mozilla/5.0 (compatible; spbot/5.0.2; +http://OpenLinkProfiler.org/bot )
  Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; Netcraft SSL Server Survey - contact [email protected])
  Mozilla/5.0 (compatible; CompSpyBot/1.0; +http://www.compspy.com/spider.html)
  Mozilla/5.0 (compatible; LA2; +http://www.zeerch.com/zeerch2/bot.php)
  yacybot (/global; amd64 Linux 2.6.32-042stab108.8; java 1.7.0_91; America/en) http://yacy.net/bot.html
  updated/0.1-alpha (updated crawler; http://www.updated.com; [email protected])
  CopperEgg/RevealUptime/FrankfurtGermany
  Scrapy/1.1.2 (+http://scrapy.org)
  yrspider (Mozilla/5.0 (compatible; YRSpider; +http://www.yunrang.com/yrspider.html))
  Mozilla/5.0 (compatible; YandexAntivirus/2.0; +http://yandex.com/bots)
  yacybot (freeworld/global; amd64 Windows 7 6.1; java 1.6.0_31; Europe/de) http://yacy.net/bot.html
  Surphace Scout&v4.0 - scout at surphace dot com
  yacybot (freeworld/global; amd64 Linux 2.6.32-5-xen-amd64; java 1.6.0_18; Europe/fr) http://yacy.net/bot.html
  MetaGeneratorCrawler/1.1 (www.metagenerator.info)
  Porkbun/Mustache (Website Analysis; http://porkbun.com; [email protected])
  Mozilla/5.0 (compatible; ScoutJet; +http://www.scoutjet.com/)
  Ruby, link_thumbnailer
  yacybot (freeworld/global; amd64 Windows Server 2008 R2 6.1; java 1.7.0_25; Europe/de) http://yacy.net/bot.html
  yacybot (/global; amd64 Linux 3.13.0-042stab093.4; java 1.7.0_79; Europe/de) http://yacy.net/bot.html
  Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.11 (KHTML, like Gecko) Chrome/23.0.1271.64 Safari/537.11 GotSiteMonitor.com
  Mozilla/5.0 (Linux; Android 4.4; Nexus 5 Build/KRT16M) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/40.0.2125.111 Mobile Safari/537.36 DareBoost
  Mozilla/5.0 (KeepRight OpenStreetMap Checker; http://keepright.at) Gecko/20100101 Firefox/22.0
  Mozilla/5.0 (Windows NT 6.1; WOW64; Trident/7.0; rv:11.0; topster.de Linkchecker 5.0) like Gecko (194.228.205.74)
  Mozilla/5.0 AppleWebkit/537.36 (KHTML, like Gecko) Trident/7.0; rv:11.0, Chrome/42.0.2311.135, Edge/12.10136, http://www.shrinktheweb.com/, Webshot/0.9
  topicbot/1.0 (Mozilla;I;+http://www.topic.bot/contact_page.html)
  DataparkSearch/4.53 (+http://dataparksearch.org/bot)
  Mozilla/5.0 (compatible; Linux; InfegyAtlas/1.0; en-US; [email protected])
  Mozilla/5.0 (X11; Linux x86_64; rv:41.0; GTmetrix https://gtmetrix.com/) Gecko/20100101 Firefox/41.0
  Mozilla/5.0 (compatible; EasouSpider; +http://www.easou.com/search/spider.html)
  Mozilla/5.0 (compatible; parsijoo-update-crawler; +http://www.parsijoo.ir/; [email protected])
  Abrave Spider v4 Robot 1 (http://robot.abrave.co.uk)
  Mozilla/5.0 (compatible; memoryBot/1.21.24 +http://internetmemory.org/en/)
  Mozilla/5.0 (compatible; STINGbot/1.0; +http://136.186.231.16)
  domainsbot (+http://www.domainsbot.com)
  WEPA/3.1 (http://www.wepa.com/; [email protected])
  Mozilla/5.0 (compatible; linkdexbot/2.1; +http://www.linkdex.com/bots/)
  Mozilla/5.0 (compatible; forensiq; +http://www.forensiq.com)
  192.comAgent
  JUST-CRAWLER(+http://www.justsystems.com/jp/tech/crawler/)
  Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm) SitemapProbe
  HuaweiSymantecSpider/[email protected]+(compatible; MSIE 7.0; Windows NT 5.1; Trident/4.0; .NET CLR 2.0.50727; .NET CLR 3.0.4506.2152; .NET CLR ; http://www.huaweisymantec.com/en/IRL/spider)
  Mozilla/5.0 ( Macintosh; Intel Mac OS X 10_10_1 ) AppleWebKit/600.2.5 ( KHTML, like Gecko ) Version/8.0.2 Safari/600.2.5 ( compatible; CloudServerMarketSpider/1.0; +http://cloudservermarket.com/spider.html )
  Mozilla/5.0 (compatible; MJ12bot/v1.3.0; http://www.majestic12.co.uk/bot.php?+)
  Mozilla/5.0 (compatible; AcoonBot/4.10.6; +http://www.acoon.de/robot.asp)
  Scrapy/1.0.1 (+http://scrapy.org)
  seegnifybot/1.0.0 (http://www.seegnify.com/bot)
  omgili/0.5 +http://omgili.com
  Slack-ImgProxy 149 (+https://api.slack.com/robots)
  yacybot (/global; amd64 Linux 3.16.0-4-amd64; java 1.7.0_79; Europe/de) http://yacy.net/bot.html
  www.integromedb.org/Crawler
  Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm
  Mozilla/5.0 (compatible) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/47.0.2526.73 Safari/537.36 [email protected]
  Mozilla/5.0 (compatible; IstellaBot/1.18.81 +http://www.tiscali.it/)
  Gigabot/1.0
  page_test ([email protected])
  istellabot/Nutch-1.10
  Mozilla/5.0 (iPhone; U; CPU iPhone OS 4_1 like Mac OS X; en-us) AppleWebKit/532.9 (KHTML, like Gecko) Version/4.0.5 Mobile/8B117 Safari/6531.22.7 (compatible; Mediapartners-Google/2.1; +http://www.google.com/bot.html)
  ldspider (http://code.google.com/p/ldspider/wiki/Robots)
  yacybot (/global; amd64 Windows 8.1 6.3; java 1.8.0_45; Europe/ru) http://yacy.net/bot.html
  facebookplatform/1.0 (+http://developers.facebook.com)
  Mozilla/5.0 (compatible; RTGI; http://linkfluence.net/)
  yacybot (freeworld/global; amd64 Linux 3.12.43-52.6-default; java 1.8.0_40; Europe/en) http://yacy.net/bot.html
  Mozilla/5.0 (compatible; YandexBot/3.0; MirrorDetector; +http://yandex.com/bots)
  Mozilla/5.0 (compatible; DCPbot/1.2; +http://domains.checkparams.com/)
  WatchMouse/18990 (http://watchmouse.com/ ; d2.watchmouse.com)
  MnoGoSearch/3.3.15
  Mozilla/5.0 (compatible; AboutUsBot/0.9; +http://www.aboutus.org/AboutUsBot)
  Web-sniffer/1.1.0 (+http://web-sniffer.net/)
  Mozilla/4.0 (compatible; Fooooo_Web_Video_Crawl http://fooooo.com/bot.html)
  50.nu/0.01 ( +http://50.nu/bot.html )
  Mozilla/4.0 (compatible; MSIE 7.0; Windows; Windows NT 5.1) BrokenLinkCheck.com/1.1
  Comodo-Certificates-Spider
  Iframely/0.9.8 (+https://iframely.com/;)
  MnoGoSearch/3.3.12
  CopperEgg/RevealUptime/FrankfurtEU
  Python-urllib/2.7 (+http://ella.juls.savba.sk/aranea_about)
  Mozilla/5.0 (compatible; JobKereso; +http://www.kozvetlen-allasok.hu/robot.jsp [email protected])
  Mozilla/5.0 (Windows NT 5.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/49.0.2623.112 Safari/537.36 PTST/201
  Mozilla/5.0 (compatible; Google Keyword Tool; +http://adwords.google.com/select/KeywordToolExternal)
  WinWebBot/1.0; (Balaena Ltd, UK); http://www.balaena.com/winwebbot.html; [email protected];)
  AppleNewsBot
  classbot (+http://allclasses.com)
  yacybot (freeworld/global; amd64 Windows Server 2008 R2 6.1; java 1.6.0_31; America/pt) http://yacy.net/bot.html
  Mozilla/5.0 (iPhone; CPU iPhone OS 7_0 like Mac OS X) AppleWebKit/537.51.1 (KHTML, like Gecko) Version/7.0 Mobile/11A465 Safari/9537.53 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)
  yacybot (/global; arm Linux 3.12.28+; java 1.7.0_71; Europe/en) http://yacy.net/bot.html
  Mozilla/5.0 (compatible; YandexMetrika/2.0; +http://yandex.com/bots mtmon01e.yandex.ru)
  CopperEgg/RevealUptime/TokyoJP
  Mozilla/5.0 (compatible; AhrefsBot/3.1; +http://ahrefs.com/robot/)
  CJNetworkQuality; http://www.cj.com/networkquality
  psbot-page (+http://www.picsearch.com/bot.html)
  DealGates Bot/1.1 by Luc Michalski (http://spider.dealgates.com/bot.html)
  Pingdom GIGRIB (http://www.pingdom.com)
  Mozilla/5.0 (compatible; SEOlyticsCrawler/3.0; +http://crawler.seolytics.net/)
  Mozilla/5.0 (compatible; EventGuruBot/1.0; +http://www.eventguru.com/spider.html)
  Python-urllib/2.7 (+http://aranea.juls.savba.sk/aranea_about.html)
  yacybot (/global; amd64 Windows Vista 6.0; java 1.7.0_55; America/en) http://yacy.net/bot.html
  The Lemur Web Crawler/Nutch-1.3 (Lemur Web Crawler; http://boston.lti.cs.cmu.edu/crawler_12/; [email protected])
  SAMSUNG-SGH-E250/1.0 Profile/MIDP-2.0 Configuration/CLDC-1.1 UP.Browser/6.2.3.3.c.1.101 (GUI) MMP/2.0 (compatible; Googlebot-Mobile/2.1; +http://www.google.com/bot.html)
  Feedfetcher-Nuesbyte; (+http://www.nuesbyte.com)
  copyright sheriff (+http://www.copyrightsheriff.com/)
  yacybot (/global; amd64 Linux 3.2.0-4-amd64; java 1.7.0_75; Europe/en) http://yacy.net/bot.html
  Touche (+http://www.touche.com.ve)
  Panopta v1.1
  Mozilla/5.0 (compatible; parsijoo-bot; +http://www.parsijoo.ir/; [email protected])
  Zookabot/2.2;++http://zookabot.com
  yacybot (/global; amd64 Linux 2.6.32-042stab093.4; java 1.7.0_65; Etc/en) http://yacy.net/bot.html
  it2media-domain-crawler/2.0
  yacybot (/global; amd64 Linux 4.2.0-27-generic; java 1.7.0_95; Europe/en) http://yacy.net/bot.html
  Mozilla/5.0 (Android; Mobile; rv:21.0) Gecko/21.0 Firefox/21.0 commoncrawl.org/research//Nutch-1.7-SNAPSHOT
  Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 5.2; Trident/4.0; Data Center; .NET CLR 1.1.4322; .NET CLR 2.0.50727; .NET CLR 3.0.04506.648; .NET CLR 3.5.21022; .NET CLR 3.0.4506.2152; .NET CLR 3.5.30729; PTST 2.386)
  Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/99.0 Safari/537.36 YottaaMonitor
  CopperEgg/RevealUptime/AtlantaGAUSA
  Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
  yacybot (/global; amd64 Windows 7 6.1; java 1.7.0_55; Europe/fr) http://yacy.net/bot.html
  Mozilla/5.0 (compatible; MJ12bot/v1.2.4; http://www.majestic12.co.uk/bot.php?+)
  Mozilla/5.0 (compatible; Applebot/0.3; +http://www.apple.com/go/applebot)
  ICC-Crawler/2.0 (Mozilla-compatible; ; http://www.nict.go.jp/en/ucri/plan/crawl.html)
  Mozilla/5.0 (compatible; YYSpider; +http://www.yunyun.com/spider.html)
  Mozilla/5.0 (iPhone; CPU iPhone OS 5_0_1 like Mac OS X) (compatible; Yeti-Mobile/0.1; +http://help.naver.com/robots/)
  Mozilla/5.0 (WhatsMyIP.org Text_to_Code_Ratio_Tool) http://whatsmyip.org/ua
  Thumbnail.CZ robot 1.1 (http://thumbnail.cz/why-no-robots-txt.html)
  Google-Calendar-Importer
  Mozilla/5.0 (Windows NT 6.3; WOW64) AppleWebKit/534.34 (KHTML, like Gecko) Websnapr/3.0 Safari/534.34
  Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/50.0.2661.75 Safari/537.36 PTST/276
  WeSEE
  librabot/2.0 (+http://academic.research.microsoft.com/)
  Testomatobot/1.0 (Linux x86_64; +http://www.testomato.com/testomatobot) minicrawler/3.0.1
  Mozilla/5.0 (compatible; MSIE 10.0; Windows NT 6.1; Trident/6.0) PTST/276
  Cliqzbot/0.1 (+http://cliqz.com [email protected])
  Scrubby/3.0 (+http://www.scrubtheweb.com/help/technology.html)
  Pinterest/0.1 +http://pinterest.com/
  Mozilla/5.0 (compatible; Crawlera/1.10.2; UID 52327)
  Comodo-Webinspector-Crawler 2.2.0, http://www.comodorobot.com
  Iframely/0.6.0 (+http://iframely.com/;)
  Mozilla/5.0 (compatible; PaperLiBot/2.1; http://support.paper.li/entries/20023257-what-is-paper-li)
  Mozilla/5.0 (compatible; SiteExplorer/1.0b; +http://siteexplorer.info/)
  Y!J-BSC/1.0 (http://help.yahoo.co.jp/help/jp/blog-search/)
  Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/49.0.2623.112 Safari/537.36 PTST/276
  Mozilla/5.0 (compatible; U; Hootsuite-WebFeed/1.0; mailto:[email protected])
  RankFlex.com Webspider
  TipTop http://feeltiptop.com
  vebidoobot
  Mozilla/5.0 (compatible; Dow Jones Searchbot)
  Mozilla/5.0 (compatible; OrangeBot-Collector/2.0; [email protected])
  Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/534+ (KHTML, like Gecko) MsnBot-Media /1.0b
  Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1.6) Gecko/20070725 Firefox/2.0.0.6 Ara.com.tr AraBot 1.0
  Mozilla/5.0 (iPhone; CPU iPhone OS 9_1 like Mac OS X) AppleWebKit/601.1.46 (KHTML, like Gecko) Version/9.0 Mobile/13B143 Safari/601.1 (compatible; AdsBot-Google-Mobile; +http://www.google.com/mobile/adsbot.html)
  Mozilla/5.0 (compatible; YandexMetrika/2.0; +http://yandex.com/bots mtmon01g.yandex.ru)
  BUbiNG (+http://law.di.unimi.it/BUbiNG.html)
  downnotifier.com monitoring
  A6-Indexer
  Mozilla/5.0 (compatible; htrix/1.14.4 ++https://webarchiv.onb.ac.at/robot.html)
  MetaURI API/2.0 +metauri.com
  Mozilla/5.0 (compatible; MixrankBot; [email protected])
  Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/44.0.2403.155 Safari/537.36 TinEye/1.0 (via http://www.tineye.com/)
  Mozilla/5.0 Project 25499 (project25499.com)
  COMODOspider/Nutch-1.0
  Mozilla/5.0 (compatible; CareerBot/1.1; +http://www.career-x.de/bot.html)
  Uptime.com_(http://uptime.com/)
  AcoonBot/4.10.5 (+http://www.acoon.de)
  BDFetch
  20090717_aa_page ([email protected])
  Metaspinner/1.0 (Metaspinner Search Engine; http://www.meta-spinner.de/; [email protected])
  Mozilla/5.0 (compatible; DBLBot/1.0; +http://www.dontbuylists.com/)
  Mozilla/5.0 (compatible; spbot/2.0; +http://www.seoprofiler.com/bot/ )
  LYT.SR v1.5 +http://lyt.sr/ (running on http://lyt.sr)
  Putin (+http://semanticjuice.com/)
  Mozilla/5.0 (compatible; LA1; +http://www.zeerch.com/bot.php)
  WatchMouse/18990 (http://watchmouse.com/ ; ny)
  bitlybot/2.0
  yacybot (/global; amd64 Linux 3.13.0-45-generic; java 1.7.0_75; Europe/en) http://yacy.net/bot.html
  Webshot/0.9 ( http://www.shrinktheweb.com/ )
  MetaURI API +metauri.com
  Mozilla/5.0 (compatible; ProductoDownloadUrlBot/1.0; +http://www.producto.de/)
  Google-Adwords-Instant (+http://www.google.com/adsbot.html)
  yacybot (freeworld/global; amd64 Linux 3.8.0-19-generic; java 1.7.0_25; Europe/en) http://yacy.net/bot.html
  yacybot (freeworld/global; i386 Linux 2.6.32-5-686; java 1.6.0_18; Europe/en) http://yacy.net/bot.html
  Mozilla/5.0 (Windows NT 6.3; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/42.0.2311.90 Safari/537.36 PTST/277
  Mozilla/5.0 (compatible; DomainSigmaCrawler/0.1; +http://domainsigma.com/robot)
  HyperZbozi.cz Feeder/3.1
  Mozilla/5.0 (Windows; U; Windows NT 5.1; en; rv:1.9.0.13) Gecko/2009073022 Firefox/3.5.2 (.NET CLR 3.5.30729) SurveyBot/2.3 (DomainTools)
  yacybot (/global; amd64 Windows Server 2012 R2 6.3; java 1.7.0_55; Europe/de) http://yacy.net/bot.html
  Google-AdWords-Express
  Mozilla/5.0 (compatible; MJ12bot/v1.4.0; http://www.majestic12.co.uk/bot.php?+)
  SMTBot (similartech.com/smtbot)
  Mozilla/5.0 (compatible; Lipperhey SEO Service; http://www.lipperhey.com/)
  MetaInspector/4.7.2 (+https://github.com/jaimeiniesta/metainspector)
  Mozilla/5.0 (Compatible; Vedma/0.91Beta; +http://www.vedma.ru/bot.htm)
  Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.21 (KHTML, like Gecko) web-capture.net/0.9 Safari/537.21
  Nokia6680/1.0 (4.04.07) SymbianOS/8.0 Series60/2.6 Profile/MIDP-2.0 Configuration/CLDC-1.1 (botmobi find.mobi/bot.html [email protected])
  Mozilla/5.0 (compatible; Exploratodo/1.0; +http://www.exploratodo.com
  iZSearch.com
  Mozilla/5.0 (compatible; YandexFavicons/1.0; +http://yandex.com/bots)
  bl.uk_lddc_bot/3.3.0-SNAPSHOT-2014-10-07T09:33:31Z (+http://www.bl.uk/aboutus/legaldeposit/websites/websites/faqswebmaster/index.html)
  yacybot (/global; amd64 Linux 3.14-0.bpo.1-amd64; java 1.7.0_55; Europe/de) http://yacy.net/bot.html
  HuaweiSymantecSpider/[email protected]+(compatible; MSIE 7.0; Windows NT 5.1; Trident/4.0; .NET CLR 2.0.50727; .NET CLR 3.0.4506.2152; .NET CLR ; http://www.huaweisymantec.com/cn/IRL/spider)
  FAST-WebCrawler/3.7/FirstPage (atw-crawler at fast dot no;http://fast.no/support/crawler.asp)
  Mozilla/5.0 (compatible; DripfeedBot/2.0; +http://dripfeedbookmark.com/bot.html
  Mozilla/5.0 (compatible; heritrix/2.0.2 +http://seekda.com)
  yacybot (freeworld/global; i386 Linux 3.2.0-23-generic-pae; java 1.7.0_03; Europe/en) http://yacy.net/bot.html
  hawkReader/0.9b (Feed Parser; http://www.hawkreader.com; Allow like Gecko)
  ruijie ([email protected])
  CorpusCrawler 2.0.0 (http://corpora.fi.muni.cz/crawler/)
  ExactSeek Crawler (http://www.exactseek.com/)/Nutch-1.4
  WordPress.com; https://botd.wordpress.com
  Mozilla/5.0 (compatible; Qwantify/2.0n; +https://www.qwant.com/)/*
  Mozilla/5.0 (compatible; memoryBot/1.20.268 +http://internetmemory.org/en/)
  ShowyouBot (http://showyou.com/crawler)
  Yoleo Consumer v0.2
  Mozilla/5.0 (compatible; IstellaBot/1.10.2 +http://www.tiscali.it/)
  Mozilla/5.0 (compatible; MSIE or Firefox mutant; not on Windows server; +http://ws.daum.net/aboutWebSearch.html) Daumoa/3.0
  Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/46.0.2490.86 YaBrowser/15.12.0.6151 Safari/537.36 Yandex.Translate
  Mozilla/5.0 (compatible; TeeRaidBot; +https://teeraid.com/bot/)
  Mozilla/5.0 (compatible; Scopia Crawler 1.1; +http://www.scopia.co)
  CopperEgg/RevealUptime/DublinIE(aws)
  yacybot (freeworld/global; amd64 Windows 7 6.1; java 1.7.0_04; Asia/ja) http://yacy.net/bot.html
  RankurBot/Rankur2.1 (http://rankur.com; info at rankur dot com)
  Gaisbot/3.0+([email protected];+http://gais.cs.ccu.edu.tw/robot.php)
  Mozilla/5.0 (compatible; zitebot support [at] zite [dot] com +http://zite.com)
  MetaTagRobot/2.0 (http://www.check-domains.com/website-analysis/website-analyzer.php)
  Virusdie crawler/2.1
  findlinks/2.0.4 (+http://wortschatz.uni-leipzig.de/findlinks/)
  Mozilla/5.0 (compatible; AcoonBot/4.10.7; +http://www.acoon.de/robot.asp)
  Mozilla/5.0 (compatible; DeuSu/5.0.2; +https://deusu.de/robot.html)
  yacybot (-global; amd64 Linux 3.19.0-15-generic; java 1.8.0_45-internal; Europe/de) http://yacy.net/bot.html
  Google_Analytics_Snippet_Validator
  SurcentroBot
  Mozilla/5.0 (compatible; Diffbot/0.1; +http://www.diffbot.com)
  findlinks/1.1.3-beta9 (+http://wortschatz.uni-leipzig.de/findlinks/)
  Mozilla/5.0 (compatible; memoryBot/1.20.199 +http://internetmemory.org/en/)
  Y!J-ASR/0.1 crawler (http://www.yahoo-help.jp/app/answers/detail/p/595/a_id/42716/)
  TwengaBot-Discover (http://www.twenga.fr/bot-discover.html)
  Mozilla/5.0 (bl.uk_lddc_bot; Linux x86_64) PhantomJS/1.9.7 (+http://www.bl.uk/aboutus/legaldeposit/websites/websites/faqswebmaster/index.html)
  panscient.com
  Mozilla/5.0 (compatible; TestCrawler; +http://wiki.github.com/bixo/bixo/bixocrawler; [email protected])
  DataparkSearch/4.37-23012006 ( http://www.dataparksearch.org/)
  GOFORITBOT ( http://www.goforit.com/about/ )
  CopperEgg/RevealUptime/SydneyAustralia
  msnbot-media/1.0 (+http://search.msn.com/msnbot.htm)
  Mozilla/5.0 (compatible; MegaIndex.ru/2.0; +https://www.megaindex.ru/?tab=linkAnalyze)
  MetaCommentBot; http://metacomment.io/about
  Mozilla/5.0 (compatible; YesupBot/1.0; +http://www.yesup.net/bot.html)
  Twikle/1.0 , http://twikle.com , [email protected]
  yacybot (/global; amd64 Windows Server 2008 R2 6.1; java 1.8.0_60; Europe/de) http://yacy.net/bot.html
  Mozilla/4.0 (compatible; Vagabondo/4.0Beta; webcrawler at wise-guys dot nl; http://webagent.wise-guys.nl/; http://www.wise-guys.nl/)
  Comodo-Webinspector-Crawler 2.1
  Mozilla/5.0 (Macintosh; Intel Mac OS X 10_7_5) AppleWebKit/537.11 (KHTML, like Gecko) Chrome/23.0.1271.64 Safari/537.11 KimonoLabs/0.2
  Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1 + FairShare-http://fairshare.cc)
  Scrapy/0.24.5 (+http://scrapy.org)
  Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/534.34 (KHTML, like Gecko) Site-Shot/2.1 (http://www.site-shot.com/) Safari/534.34
  Port Monitor check service 1.0 | http://www.port-monitor.com
  Mozilla/5.0 (compatible; Uptimebot/0.1.43; +http://www.uptime.com/uptimebot)
  yacybot (freeworld/global; amd64 Linux 2.6.32-5-amd64; java 1.6.0_18; Europe/en) http://yacy.net/bot.html
  SolomonoBot/1.04 (http://www.solomono.ru)
  Easy-Thumb (https://www.easy-thumb.net/)
  Mozilla/5.0 (Windows NT 6.0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/49.0.2623.112 Safari/537.36 PTST/276
  nic.it crawler v.1.221115 (7.35.0)
  Mozilla/5.0 (compatible; memorybot/1.24.12 +https://internetmemory.net/en/bots)
  Mozilla/5.0 (compatible; inoreader.com-like FeedFetcher-Google)
  yacybot (/global; amd64 Windows 7 6.1; java 1.7.0_72; Europe/de) http://yacy.net/bot.html
  WatchMouse/8.4.0.3 (http://watchmouse.com/ ; demuc02.watchmouse.net)
  Mozilla/5.0 (X11; Linux x86_64; rv:43.0; GTmetrix https://gtmetrix.com/) Gecko/20100101 Firefox/43.0
  ShopWiki/1.0 ( +http://www.shopwiki.com/wiki/Help:Bot)
  Mozilla/5.0 (compatible; UnisterBot; http://www.bluekiwi.de/misc/imprint)
  Web-sniffer/1.0.31 (+http://web-sniffer.net/)
  woobot
  Mozilla/5.0 (X11; U; Linux Core i7-4980HQ; de; rv:32.0; compatible; JobboerseBot; https://www.jobboerse.com/bot.htm) Gecko/20100401 Firefox/24.0
  AntBot/Ant-Nutch-1.1 (Ant Nutch Crawler; http://www.ant.com; [email protected])
  Mozilla/5.0 (compatible; YandexMedia/3.0; +http://yandex.com/bots)
  Mozilla/5.0 (compatible; Evrinid Iudex 1.0.0; +http://www.evri.com/evrinid)
  adidxbot/1.1 (+http://search.msn.com/msnbot.htm)
  Mozilla/5.0 (compatible; WebThumbnail/2.2; Website Thumbnail Generator; +http://webthumbnail.org)
  Mozilla/5.0 (compatible; Embedly/0.2; +http://support.embed.ly/)
  EasyBib AutoCite (http://content.easybib.com/autocite/)
  yacybot (-global; amd64 Linux 3.2.0-4-amd64; java 1.7.0_65; Europe/en) http://yacy.net/bot.html
  Mozilla/5.0 (compatible; woriobot +http://worio.com)
  Mozilla/4.0 (compatible; Vagabondo/4.0; http://webagent.wise-guys.nl/)
  SniffRSS/0.5beta (+http://www.blogator.com/)
  woobot/1.1
  Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) WordPress.com mShots Safari/537.36
  ADmantX Platform Semantic Analyzer - Turn - ADmantX Inc. - www.admantx.com - [email protected]
  Mozilla/5.0 (compatible; UnisterBot; [email protected])
  Mozilla/5.0 (compatible; electricmonk/3.2.0 +https://www.duedil.com/our-crawler/)
  UptimeDog Robot (www.uptimedog.com)
  Googlebot/2.1 (+http://www.google.com/bot.html)
  Mozilla/5.0 (Windows NT 6.3; WOW64; rv:36.0) Gecko/20100101 Firefox/36.0 (NetShelter ContentScan, contact [email protected] for information)
  Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.0; Trident/4.0; SLCC1; .NET CLR 2.0.50727; Media Center PC 5.0; .NET CLR 3.5.30729; .NET4.0C; .NET CLR 3.0.30729; .NET4.0E) PTST/276
  ServiceUptime.robot
  Mozilla/4.0 (compatible;HostTracker/2.0;+http://www.host-tracker.com/)
  SEO Browser
  Leikibot/1.0 (+http://www.leiki.com)
  yacybot (webportal-global; amd64 Windows 7 6.1; java 1.7.0_04; America/en) http://yacy.net/bot.html
  AppEngine-Google; (+http://code.google.com/appengine; appid: downforeveryoneorjustme)
  Mozilla/5.0 (Windows NT 5.1) BrokenLinkCheck.com/1.1
  Seobility (SEO-Check; http://bit.ly/1dJuuzs)
  Mozilla/5.0 (seoanalyzer; compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)
  Mozilla/5.0 (compatible; Digincore bot; https://www.digincore.com/crawler.html for rules and instructions.
  Site24x7
  websitepulse checker/3.0 (compatible; MSIE 5.5; Netscape 4.75; Linux)
  Mozilla/5.0 (compatible; SemrushBot-SA/0.97; +http://www.semrush.com/bot.html)
  Mozilla/5.0 (compatible; NerdByNature.Bot; http://www.nerdbynature.net/bot)
  Abrave Spider v4 Robot 2 (http://robot.abrave.co.uk)
  HubPages V0.2.2 (http://hubpages.com/help/crawlingpolicy)
  lmspider ([email protected])
  magpie-crawler/1.1 (U; Linux amd64; en-GB; +http://www.brandwatch.net)
  Mozilla/5.0 (compatible; WbSrch/1.1 +http://wbsrch.com
  Mozilla/5.0 (Windows; U; Windows NT 5.1; de; rv:1.9.0.7; Google-SearchByImage) Gecko/2009021910 Firefox/3.0.7
  Mozilla/5.0 (compatible; Najdi.si/3.1)
  larbin_2.6.3 ([email protected])
  Mozilla/5.0 (compatible; aiHitBot/1.1; +http://www.aihit.com/)
  Mozilla/5.0 (Windows NT 6.3; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/50.0.2661.102 Safari/537.36 PTST/277
  Mozilla/5.0 (YahooYSMcm/3.0.0; http://help.yahoo.com)
  yacybot (/global; amd64 Linux 3.16.0-4-amd64; java 1.7.0_79; Europe/en) http://yacy.net/bot.html
  OOZBOT/0.20 ( http://www.setooz.com/oozbot.html ; agentname at setooz dot_com )
  findlinks/2.0.9 (+http://wortschatz.uni-leipzig.de/findlinks/)
  Mozilla/5.0 (compatible; Lipperhey Link Explorer; http://www.lipperhey.com/)
  Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.6; en-US; rv:1.9.2) Gecko/20100115 Firefox/3.6 (FlipboardProxy/1.1; +http://flipboard.com/browserproxy)
  Mozilla/5.0 (Twiceler-0.9 http://www.cuil.com/twiceler/robot.html)
  Mozilla/5.0 (compatible; memoryBot/1.24.11 +http://internetmemory.org/en/)
  RED/1 (https://redbot.org/)
  Twitterbot/1.0
  Mozilla/5.0 (compatible; ptd-crawler; +http://bixolabs.com/crawler/ptd/; [email protected])
  Open Web Analytics Bot 1.5.4
  JikeSpider Mozilla/5.0 (compatible; JikeSpider; +http://shoulu.jike.com/spider.html)
  Mozilla/5.0 (compatible; vkShare; +http://vk.com/dev/Share)
  Mozilla/5.0 (compatible; oBot/2.3.1; +http://www-935.ibm.com/services/us/index.wss/detail/iss/a1029077?cntxt=a1027244)
  elefent/Elefent 1.2 (A friendly web elefent.; http://elefent.eu/; [email protected])
  Genderanalyzer/1.0
  elefent/1.2 (Web Crawler; https://crawler.elefent.net; webmaster at elefent dot net)
  Yandex/1.01.001 (compatible; Win16; H)
  crawler4j for XQuery
  Mozilla/5.0 (compatible; Plukkie/1.4; http://www.botje.com/plukkie.htm)
  LinguaBot/v0.001-dev (MultiLinual Sarch Engine v0.001; LinguaSeek; admin at linguaseek dot com)
  MrakyPraceCzBot/1.0 (http://www.mrakyprace.cz/about-crawler)
  Y!J-BRJ/YATS crawler (http://help.yahoo.co.jp/help/jp/search/indexing/indexing-15.html)
  WatchMouse/8.4.0.3 (http://watchmouse.com/ ; plgdn01.watchmouse.net)
  Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2272.96 Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
  AskQuickly v2 (http://askquickly.org/)
  Mozilla/5.0 (compatible; SISTRIX Crawler; http://crawler.sistrix.net/)
  Miniflux (http://miniflux.net)
  Mozilla/5.0 (compatible; spbot/5.0.1; +http://OpenLinkProfiler.org/bot )
  Mozilla/5.0 (compatible; wmtips.com/1.0; +http://www.wmtips.com/tools/)
  Mozilla/5.0 (compatible; linkdexbot/2.0; +http://www.linkdex.com/bots/)
  Mozilla/2.0 (compatible; Ask Jeeves/Teoma; +http://sp.ask.com/docs/about/tech_crawling.html)
  Google favicon
  Mozilla/5.0(compatible; Sosospider/2.0; +http://help.soso.com/webspider.htm)
  MiaDev/0.0.1 (MIA Bot for research project MIA (www.MIA-marktplatz.de); http://www.mia-marktplatz.de/spider; [email protected])
  Mozilla/5.0 (compatible; proximic; +http://www.proximic.com/info/spider.php)
  Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/33.0.1750.146 Safari/537.36 Specificfeeds- http://www.specificfeeds.com
  Mozilla/5.0 (compatible; socialbm_bot/1.0; +http://spider.socialbm.net)
  wp.com feedbot/1.0 (+https://wp.com)
  Mozilla/5.0 (compatible; CligooRobot/2.0; +http://www.cligoo.de/wk/technik.php)
  Slack-ImgProxy (+https://api.slack.com/robots)
  Mozilla/5.0 (compatible; Online Domain Tools - HTTP Headers Online/1.0; +http://http-headers.online-domain-tools.com)
  Mozilla/5.0 (compatible; SurdotlyBot/1.0; +http://sur.ly/bot.html)
  WGETbot/1.0 (+http://wget.alanreed.org)
  yacybot (/global; amd64 Linux 4.0.9-2-ck; java 1.8.0_51; Europe/de) http://yacy.net/bot.html
  Y!J-BRI/0.0.1 crawler ( http://help.yahoo.co.jp/help/jp/search/indexing/indexing-15.html )
  dlvr.it/1.0 (+http://dlvr.it/) Mozilla/5.0
  yacybot (/global; arm Linux 4.1.13-v7+; java 1.8.0; Etc/de) http://yacy.net/bot.html
  Mozilla/5.0 (compatible; GimmeUSAbot/1.0; +http://gimmeusa.com/crawler.html)
  yacybot (freeworld/global; amd64 Windows 7 6.1; java 1.6.0_29; Europe/fr) http://yacy.net/bot.html
  Link Valet Online 1.2
  Mozilla/5.0 (compatible; METASpider; +http://meta.ua/spider)
  NeutrinoAPI/2.0.1
  thumbshots-de-bot (+http://www.thumbshots.de/)
  Mozilla/5.0 (iPhone; CPU iPhone OS 6_0 like Mac OS X) AppleWebKit/536.26 (KHTML, like Gecko) Version/6.0 Mobile/10A5376e Safari/8536.25 (compatible; Applebot/0.3; +http://www.apple.com/go/applebot)
  argenfybot
  MergadoBot
  Ant/Ant-Nutch-1.1 (Ant Nutch Crawler; http://www.ant.com; [email protected])
  CopperEgg/RevealUptime/SaoPauloBrazil
  Mozilla/4.0 (compatible; Vagabondo/4.0/EU; http://webagent.wise-guys.nl/)
  Mozilla/4.0 (compatible; DAWINCI ANTIPLAG SPIDER)
  Mozilla/5.0 (Windows NT 6.1; WOW64; rv:45.0) Gecko/20100101 Firefox/45.0 PTST/276
  HetrixTools.COM Uptime Monitoring Bot. https://hetrixtools.com/uptime-monitoring-bot.html
  yacybot (/global; amd64 Linux 3.2.0-4-amd64; java 1.7.0_79; Europe/en) http://yacy.net/bot.html
  Mozilla/5.0 (Windows NT 6.1) (compatible; SMTBot/1.0; +http://www.similartech.com/smtbot)
  Mozilla/5.0 (compatible; InbyBot/2.1; +http://inbyapp.com)
  all_web1 ([email protected])
  Mozilla/5.0 (compatible; BusinessSeek.biz_Directory; http://www.businessseek.biz/)
  Wotbox/2.0 ([email protected]; http://www.wotbox.com)
  Mediatoolkitbot ([email protected])
  OpenWebSpider v0.1.2.B (http://www.openwebspider.org/)
  Mozilla/5.0 (compatible; WebCorp/5.0; +http://www.webcorp.org.uk)
  Testomatobot/1.0 (Linux x86_64; +http://www.testomato.com/testomatobot) minicrawler/4.0.0
  fastbot crawler beta 2.0 (+http://www.fastbot.de)
  Sogou Web Spider
  DoCoMo/2.0 P900i(c100;TB;W24H11) (compatible; ichiro/mobile goo;+http://search.goo.ne.jp/option/use/sub4/sub4-1/)
  Mozilla/5.0 (compatible; imagecoccoc/1.0; +http://help.coccoc.com/searchengine)
  XoviBot/1.0
  Mozilla/5.0 (compatible; Online Domain Tools - Online Website Link Checker/1.1; +http://website-link-checker.online-domain-tools.com)
  WatchMouse/8.4.0.3 (http://watchmouse.com/ ; decgn02.watchmouse.net)
  yacybot (i386 Linux 2.6.31-21-generic; java 1.6.0_0; Europe/en) http://yacy.net/bot.html
  ZnajdzFoto/ImageBot 2.0b
  Mozilla/5.0 (compatible; heritrix/3.0.0-SNAPSHOT-20091120.021634 +http://crawler.archive.org)
  Mozilla/5.0 (compatible; ProCogSEOBot/1.0; +http://www.procog.com/ )
  Mozilla/5.0 (compatible; Plukkie/1.6; http://www.botje.com/plukkie.htm)
  yacybot (freeworld/global; amd64 Linux 3.2.0-21-generic; java 1.7.0_03-icedtea; America/en) http://yacy.net/bot.html
  Mozilla/5.0 (iPhone; CPU iPhone OS 8_3 like Mac OS X) AppleWebKit/600.1.4 (KHTML, like Gecko) Version/8.0 Mobile/12F70 Safari/600.1.4 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
  Mozilla/5.0 (compatible; MSIE or Firefox mutant; not on Windows server; + http://tab.search.daum.net/aboutWebSearch.html) Daumoa/3.0
  Mozilla/5.0 (compatible; NLNZ_IAHarvester2014 +http://natlib.govt.nz/about-us/current-initiatives/web-harvest-2014)
  Mozilla/5.0 (compatible; archive.org_bot +http://archive.org/details/archive.org_bot)
  Mozilla/5.0 (compatible; special_archiver/3.3.0 +http://www.loc.gov/webarchiving/notice_to_webmasters.html)
  yacybot (freeworld/global; i386 Linux 3.4.2-linode44; java 1.6.0_27; America/en) http://yacy.net/bot.html
  Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/47.0.2526.111 Safari/537.36 Yandex.Translate
  yacybot (webportal-global; amd64 Linux 3.2.0-4-amd64; java 1.7.0_25; Europe/en) http://yacy.net/bot.html
  FreeWebMonitoring SiteChecker/0.1 (+http://www.freewebmonitoring.com)
  Mozilla/5.0 (compatible; Infohelfer/1.3.0; +http://www.infohelfer.de/crawler.php)
  Kemvibot/1.0 (http://kemvi.com)
  Mozilla/4.0 (compatible; HostTracker.com/1.0;+http://host-tracker.com/)
  AdsBot-Google-Mobile (+http://www.google.com/mobile/adsbot.html) Mozilla (iPhone; U; CPU iPhone OS 3 0 like Mac OS X) AppleWebKit (KHTML, like Gecko) Mobile Safari
  Mozilla/5.0 (compatible; GeliyooBot/1.0; +http://www.geliyoo.com/)
  yacybot (x86 Windows XP 5.1; java 1.6.0_21; Europe/de) http://yacy.net/bot.html
  Mozilla/5.0 (compatible; archive.org_bot/3.1.2 +http://natlib.govt.nz/about-us/current-initiatives/web-harvest-2012)
  Mozilla/5.0 (compatible; StorygizeBot; http://www.storygize.com)
  infohelfer/1.1.0 (http://www.infohelfer.de/)
  yacybot (/global; amd64 Linux 3.16.0-41-generic; java 1.7.0_80; Europe/de) http://yacy.net/bot.html
  SeoCheckBot (FischerNetzDesign Seo Checker, [email protected])
  netEstate NE Crawler (+http://www.website-datenbank.de/)
  Mail.Ru/1.0
  binlar_2.6.3 ([email protected])
  LinkedInBot/1.0 (compatible; Mozilla/5.0; Jakarta Commons-HttpClient/3.1 +http://www.linkedin.com)
  bot.wsowner.com/1.0.0 (+http://wsowner.com/)
  Ronzoobot/1.5 (http://www.ronzoo.com/about/)
  Mozilla/5.0 (compatible; heritrix/3.2.0 +http://www.mixdata.com)
  Mozilla/5.0 (compatible; Yahoo! Slurp China; http://misc.yahoo.com.cn/help.html)
  findlinks/1.1.6-beta6 (+http://wortschatz.uni-leipzig.de/findlinks/)
  Mozilla/5.0 (compatible; LoadTimeBot/0.7; +http://www.load-time.com/bot.html)
  Mozilla/5.0 (compatible; MJ12bot/v1.4.2; http://www.majestic12.co.uk/bot.php?+)
  Mozilla/4.0 (compatible; MSIE 5.01; GomezAgent 2.0; Windows NT)
  Mozilla/5.0 (compatible; MJ12bot/v1.3.2; http://www.majestic12.co.uk/bot.php?+)
  SEOENGBot/1.2 (+http://learn.seoeng.com/seoengbot.htm)
  yacybot (i386 Linux 2.6.31-18-generic; java 1.6.0_0; Europe/en) http://yacy.net/bot.html
  Online Virus Scanner: http://tools.geek-tools.org
  DwnldBot (+http://dwnld.me)
  Mozilla/5.0 (compatible; OpenfosBot/2.1; http://www.openfos.com)
  taptubot *** please read http://www.taptu.com/corp/taptubot ***
  yacybot (freeworld/global; amd64 Linux 3.9.3-1-ARCH; java 1.7.0_21; America/en) http://yacy.net/bot.html
  Testomatobot/1.0 (Linux x86_64; +http://www.testomato.com/testomatobot) minicrawler/4.0.0~beta10
  rogerbot/1.0 (http://www.moz.com/dp/rogerbot, [email protected])
  Mozilla/5.0 (Windows NT 6.1; Win64; x64) KomodiaBot/1.0
  Mozilla/5.0 (compatible; spbot/3.0; +http://www.seoprofiler.com/bot )
  Mozilla/5.0 (compatible; DomainTunoCrawler/0.1; +https://www.domaintuno.com/robot)
  Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Googlebot/2.1; +http://www.google.com/bot.html) Safari/537.36
  yacybot (amd64 Linux 2.6.32-gentoo; java 1.6.0_17; Europe/en) http://yacy.net/bot.html
  Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko; Google Page Speed Insights) Chrome/27.0.1453 Mobile Safari/537.36
  Mozilla/5.0 (compatible; GimmeUSAbot/1.0; +http://gimmeusa-update.com/crawler)
  Cloud mapping experiment. Contact [email protected]
  emefgebot/beta (+http://emefge.de/bot.html)
  DoCoMo/2.0 D903i(c100;TB;W28H20) (compatible; Hatena-Mobile-Gateway/1.2; +http://mgw.hatena.ne.jp/help)
  ichiro/3.0 (http://help.goo.ne.jp/door/crawler.html)
  Mozilla/5.0 (compatible; SecurityResearch.bot; +http://besome1.info/securityresearchbot.html)
  WPScan v2.9 (http://wpscan.org)
  Ronzoobot/1.6 (http://www.ronzoo.com/about/)
  livedoor ScreenShot/0.10
  Mozilla/5.0 (compatible; MSIE 8.0; Windows NT 5.1) KomodiaBot/1.0
  Mozilla/5.0 (compatible; memoryBot/1.20.289 +http://internetmemory.org/en/)
  Mozilla/5.0 (compatible; DomainTunoCrawler/0.1; +http://www.domaintuno.com/robot)
  yacybot (freeworld/global; amd64 Linux 2.6.26-2-amd64; java 1.6.0_18; Europe/de) http://yacy.net/bot.html
  Mozilla/5.0 (compatible; GimmeUSAbot/1.0; +http://gimmeusa.com/crawler)
  Mozilla/5.0 (compatible; spbot/5.0; +http://OpenLinkProfiler.org/bot )
  Mozilla/5.0 (compatible; Webauskunft.org; http://www.webauskunft.org)
  CCResearchBot/1.0 commoncrawl.org/research//Nutch-1.7-SNAPSHOT
  Mozilla/4.0 (compatible; MSIE 5.5; Windows NT 4.0; obot)
  linkdex.com/v2.0
  Mozilla/5.0 (compatible; Crawlera/1.10.2; UID 73393)
  Mozilla/5.0 (compatible; SEODat/0.1 http://crawler.seodat.com)
  yacybot (freeworld/global; amd64 Linux 3.2.0-4-amd64; java 1.6.0_27; Europe/en) http://yacy.net/bot.html
  CorpusCrawler 2.0.12 (http://corpora.fi.muni.cz/crawler/)
  Mozilla/5.0 (compatible; spbot/4.3.0; +http://OpenLinkProfiler.org/bot )
  Mozilla/5.0 (compatible; aiHitBot/2.7; +http://www.aihit.com/)
  Mozilla/5.0 (compatible; SiteAnalyzerBot/4.0; +https://www.site-analyzer.com/)
  DataparkSearch/4.40.1 (+http://www.dataparksearch.org/)
  yacybot (freeworld-global; amd64 Linux 3.0.0-21-generic; java 1.6.0_26; America/en) http://yacy.net/bot.html
  findlinks/1.1.6-beta1 (+http://wortschatz.uni-leipzig.de/findlinks/; YaCy 0.1; yacy.net)
  CopperEgg/RevealUptime/SydneyAU(aws)
  OpenAcoon v4.1.0 (www.openacoon.de)
  semanticbot ([email protected])
  Mozilla/5.0 (compatible; Crawlera/1.10.2; UID 75425)
  Mozilla/5.0 (compatible; Crawlera/1.10.2; UID 11912)
  Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp)
  Mozilla/5.0 (Windows NT 5.1; U; Win64; fr; rv:1.8.1) VoilaBot BETA 1.2 ([email protected])
  yacybot (-global; amd64 Linux 3.13.0-61-generic; java 1.7.0_79; Europe/en) http://yacy.net/bot.html
  CorpusCrawler 2.0.25 (http://corpora.fi.muni.cz/crawler/);Project:CzCorpus
  ichiro/3.0 (http://search.goo.ne.jp/option/use/sub4/sub4-1/)
  R6_FeedFetcher_(www.radian6.com/crawler)
  sosoimagespider+(+http://help.soso.com/soso-image-spider.htm)
  Mozilla/5.0 (X11; Linux x86_64; rv:39.0; GTmetrix http://gtmetrix.com/) Gecko/20100101 Firefox/39.0
  yacybot (freeworld/global; amd64 Linux 3.2.0-24-generic; java 1.6.0_24; Europe/en) http://yacy.net/bot.html
  Mozilla/4.0 (compatible; Vagabondo/4.0; webcrawler at wise-guys dot nl; http://webagent.wise-guys.nl/)
  CommaFeed/2.4.0-SNAPSHOT (https://github.com/Athou/commafeed)
  Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; Google Web Preview Analytics) Chrome/27.0.1453 Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
  yacybot (/global; x86 Windows 7 6.1; java 1.8.0_31; Europe/de) http://yacy.net/bot.html
  Mozilla/5.0 (compatible; woriobot support [at] zite [dot] com +http://zite.com)
  Mozilla/5.0 (compatible; MetaJobBot; http://www.metajob.at/crawler)
  bl.uk_lddc_bot/3.3.0-SNAPSHOT-20140519-0844 (+http://www.bl.uk/aboutus/legaldeposit/websites/websites/faqswebmaster/index.html)
  FFbot/1.0; +http://feedfury.com
  Mozzila/5.0 (compatible; YamanaLab-Robot/1.0; http://www.yama.info.waseda.ac.jp/~crawler/info.html)
  LinqiaRSSBot/1.0 ([email protected])
  FreeWebMonitoring SiteChecker/0.2 (+http://www.freewebmonitoring.com/bot.html)
  Mozilla/5.0 (compatible; Web-Monitoring/1.0; +http://monoid.nic.ru/)
  Mozilla/5.0 (compatible; heritrix/1.12.1b +http://netarkivet.dk/website/info.html)
  Mozilla/5.0 (compatible; SpiderLing (a SPIDER for LINGustic research); http://nlp.fi.muni.cz/projects/biwec/)
  UrlTrends Ranking Updater/2.0 ( http://www.urltrends.com/ )
  yacybot (/global; amd64 Linux 3.16.1-gentoo; java 1.7.0_55; Europe/en) http://yacy.net/bot.html
  Mozilla/5.0 (compatible; gimme60bot/1.0 ; +http://gimme60.com)
  CatchBot/5.0; +http://www.catchbot.com
  elefent/Elefent 1.1 (Friendly web walker.; http://elefent.eu/; [email protected])
  findlinks/2.0.1 (+http://wortschatz.uni-leipzig.de/findlinks/)
  Googlebot-Video/1.0
  Mozilla/5.0 (compatible; XmarksFetch/1.0; +http://www.xmarks.com/about/crawler; [email protected])
  XRL/3.00 (Linux; i686; en-us) (+http://metamark.net/about)
  Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/50.0.2661.87 Safari/537.36 PTST/276
  CorpusCrawler 2.0.8 (http://corpora.fi.muni.cz/crawler/)
  Mozilla/5.0 (compatible; archive.org_bot +http://www.archive.org/details/archive.org_bot)
  Mozilla/5.0 (compatible; OpenindexDeepSpider/Nutch-1.5-dev; +http://openindex.io/spider.html; systemsATopenindexDOTio)
  Ronzoobot/1.3 (http://www.ronzoo.com/about.php)
  MetaInspector/2.2.1 (+https://github.com/jaimeiniesta/metainspector)
  Mozilla/5.0 (compatible; WASALive-Bot ; http://blog.wasalive.com/wasalive-bots/)
  Mozilla/5.0 (compatible; YioopBot; +http://www.yioop.com/bot.php)
  Mozilla/5.0 (compatible; Mozilla/5.0; +http://wiki.github.com/bixo/bixo/bixocrawler; [email protected])
  Mozilla/5.0 (compatible; Kraken/0.1; http://linkfluence.net/; [email protected])
  Mozilla/5.0 (compatible; Qseero; +http://www.q0.com)
  yacybot (-global; amd64 Linux 3.10.0-229.4.2.el7.x86_64; java 1.8.0_45; Europe/en) http://yacy.net/bot.html
  Mozilla/5.0 (compatible; ProCogBot/1.0; +http://www.procog.com/spider.html)
  Mozilla/5.0 (compatible; MJ12bot/v1.4.3; http://www.majestic12.co.uk/bot.php?+)
  Mozilla/5.0 (Anturis Agent)
  Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/534.34 (KHTML, like Gecko) Chromium/31.0.1650.63 Site-Shot/2.1 (http://www.site-shot.com/) Safari/534.34
  webnumbrFetcher/1.0 (http://webnumbr.com/)
  Mozilla/5.0 (compatible; meanpathbot/1.0; +http://www.meanpath.com/meanpathbot.html)
  NIF/1.1 (http://www.newsisfree.com/robot.php users:)
  Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/51.0.2704.103 Safari/537.36 PTST/284
  DoCoMo/2.0 P901i(c100;TB;W24H11) (compatible; ichiro/mobile goo; +http://help.goo.ne.jp/door/crawler.html)
  Iframely/0.8.8 (+https://iframely.com/;)
  yacybot (ppc Mac OS X 10.5.2; java 1.5.0_13; Europe/de) http://yacy.net/bot.html
  Mozilla/5.0 (compatible; UptimeRobot/2.0; http://www.uptimerobot.com/)
  yacybot (/global; amd64 Linux 3.13.0-85-generic; java 1.7.0_95; Europe/en) http://yacy.net/bot.html
  Mozilla/5.0 (compatible; FeedBooster; +http://feeds.qsensei.com)
  yacybot (freeworld/global; amd64 Linux 2.6.32-5-amd64; java 1.6.0_18; Europe/de) http://yacy.net/bot.html
  Mozilla/5.0 (compatible; SputnikFaviconBot/1.2; +http://corp.sputnik.ru/webmaster)
  Mozilla/5.0 (compatible; heritrix/3.1.1; UniLeipzigASV +http://corpora.informatik.uni-leipzig.de/crawler_faq.html)
  Mozilla/5.0 (compatible; OrangeBot/2.0; [email protected])
  Yeti-FeedItemCrawler/1.0 (NHN Corp.; http://help.naver.com/robots/)
  Mozilla/5.0 (compatible; SeznamBot/3.2-test1-1; +http://napoveda.seznam.cz/en/seznambot-intro/)
  Mozilla/5.0 (compatible; heritrix/3.2.0 +http://www.crim.ca)
  yacybot (/global; amd64 Linux 4.0.5-gentoo; java 1.8.0_51; Europe/en) http://yacy.net/bot.html
  pr-cy.ru Screenshot Bot
  GentleSource Short URL Checker ( http://www.gentlesource.com/ )
  crazywebcrawler 0.9.0, http://www.crazywebcrawler.com
  a.pr-cy.ru
  Mozilla/5.0 (iPhone; CPU iPhone OS 7_0 like Mac OS X) AppleWebKit/537.51.1 (KHTML, like Gecko) Version/7.0 Mobile/11A465 Safari/9537.53 (compatible; bingbot/2.0; http://www.bing.com/bingbot.htm)
  findlinks/1.1.6-beta1 (+http://wortschatz.uni-leipzig.de/findlinks/)
  ichiro/5.0 (http://help.goo.ne.jp/door/crawler.html)
  yacybot (/global; amd64 Windows 7 6.1; java 1.8.0_25; Europe/en) http://yacy.net/bot.html
  Yeti/1.1 (Naver Corp.; http://help.naver.com/robots/)
  Technoratibot/8.0
  Setooz/Nutch-1.0 (http://www.setooz.com)
  Mozilla/5.0 (Windows NT 5.2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/49.0.2623.112 Safari/537.36 PTST/201
  semanticbot
  Mozilla/5.0 (compatible; JikeSpider; +http://shoulu.jike.com/spider.html)
  Mozilla/5.0 (compatible; MetamojiCrawler/1.0; +http://www.metamoji.com/jp/crawler.html
  Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1.6) Gecko/20070725 Firefox/2.0.0.6 - James BOT - WebCrawler http://cognitiveseo.com/bot.html
  Mozilla/5.0 (Windows NT 6.1; compatible; BDCbot/1.0; +http://ecommerce.bigdatacorp.com.br/faq.aspx) ppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2272.118 Safari/537.36
  Mozilla/5.0 (compatible; CloudServerMarketSpider/1.0; +http://www.cloudservermarket.com/spider.html)
  Mozilla/5.0 (Windows NT 6.1; WOW64; rv:38.0; Neustar WPM) Gecko/20100101 Firefox/38.0
  Mozilla/5.0 (compatible;+ParchBot/1.0;++http://www.parchmenthill.com/search.htm)
  Sogou-Test-Spider/4.0 (compatible; MSIE 5.5; Windows 98)
  MetaHeadersBot (+http://www.metaheaders.com/bot.html)
  Mozilla/5.0 (Windows; U; Windows NT 6.0; en-GB; rv:1.0; trendictionbot0.4.2; trendiction media ssppiiddeerr; http://www.trendiction.com/bot/; please let us know of any problems; ssppiiddeerr at trendiction.com) Gecko/20071127 Firefox/2.0.0.11
  voyager/2.0 (http://www.kosmix.com/crawler.html)
  Mozilla/5.0 (compatible; YodaoBot/1.0; http://www.yodao.com/help/webmaster/spider/; )
  hivaBot/hivaBot-1.0 (Iranian Search Engine bot; http://yooz.ir; [email protected])
  yacybot (freeworld/global; amd64 Linux 3.2.0-4-amd64; java 1.7.0_55; Europe/de) http://yacy.net/bot.html
  AppEngine-Google; (+http://code.google.com/appengine; appid: s~closure-compiler-hrd)
  larbin_2.5.0 ([email protected])
  Scrapy/0.22.2 (+http://scrapy.org)
  Ronzoobot/1.2 (http://www.ronzoo.com/about.php)
  gooblog/2.0 (http://help.goo.ne.jp/contact/)
  Pingdom.com_bot_version_1.4_(http://www.pingdom.com/)
  Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/535.21 (KHTML, like Gecko) Chrome/19.0.1042.0 Safari/535.21 PagePeeker/2.1; +http://pagepeeker.com/robots/
  WeSEE:Ads/PageBot (http://www.wesee.com/bot/)
  yacybot (amd64 Linux 2.6.18-164.el5; java 1.6.0; Europe/de) http://yacy.net/bot.html
  SemanticScholarBot/1.0 (+http://s2.allenai.org/bot.html)
  Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/535. Safari/535.22+; (+http://immediatenet.com/thumbnail_api.html; free thumbnails)
  Mozilla/5.0 (compatible; Crawlera/1.10.2; UID 85747)
  SBIder/Nutch-1.0-dev (http://www.sitesell.com/sbider.html)
  Scrapy/1.0.4 (+http://scrapy.org)
  SanszBot/1.7(http://www.sansz.org/sanszbot, [email protected]) ([email protected])
  Mozilla/5.0 (compatible; KHTML, like Gecko) Chrome/43.0.2357.81 Safari/537.36 [email protected]
  Mozilla/5.0 (iPhone; U; CPU iPhone OS 4_3 like Mac OS X; en-GB) adbeat.com/policy AppleWebKit/533.17.9 (KHTML, like Gecko) Version/5.0.2 Mobile/8F190 Safari/6533.18.5
  Curious George - www.analyticsseo.com/crawler
  Grammarly/1.0 (http://www.grammarly.com)
  OpenCalaisSemanticProxy
  yacybot (freeworld/global; amd64 Linux 2.6.32-49-server; java 1.6.0_27; Europe/en) http://yacy.net/bot.html
  netEstate RSS crawler (+http://www.rss-directory.info/)
  ipv6-test.com validator
  Mozilla/5.0 (compatible; Qualidator.com Bot 1.0;)
  ichiro/2.0 ([email protected])
  yacybot (freeworld/global; amd64 Linux 2.6.38-14-generic; java 1.6.0_22; Europe/en) http://yacy.net/bot.html
  cg-eye live
  Mozilla/5.0 (compatible; OpenindexSpider; +http://www.openindex.io/en/webmasters/spider.html)
  Mozilla/5.0 (Windows; U; MSIE 9.0; Windows NT 9.0; en-US) AppEngine-Google; (+http://code.google.com/appengine; appid: s~virustotalcloud)
  Mozilla/5.0 (compatible; heritrix/1.14.3.r6601 +http://www.buddybuzz.net/yptrino)
  MetaGeneratorCrawler/1.3.4 (www.metagenerator.info)
  Mozilla/5.0 (Linux; Android 5.0; LG-D855 Build/LRX21R.A1422018487) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/40.0.2214.109 Mobile Safari/537.36 DareBoost
  Mozilla/5.0 (compatible; Nigma.ru/3.0; [email protected])
  Mozilla/5.0 (compatible; Wappalyzer; +https://github.com/AliasIO/Wappalyzer)
  yacybot (/global; x86 Windows XP 5.1; java 1.7.0_71; Europe/de) http://yacy.net/bot.html
  Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/50.0.2661.102 Safari/537.36 PTST/277
  OpenWebSpider v0.1.4 (http://www.openwebspider.org/)
  HubSpot Connect 1.0 (http://dev.hubspot.com/)
  Page Valet/4.1pre5
  msnbot/2.0b (+http://search.msn.com/msnbot.htm)
  amibot - http://www.amidalla.de - [email protected] libwww-perl/5.831
  SETOOZBOT/5.0 ( compatible; SETOOZBOT/0.30 ; http://www.setooz.com/bot.html )
  Mozilla/5.0 (compatible; YoudaoBot/1.0; http://www.youdao.com/help/webmaster/spider/; )
  Mozilla/5.0 (compatible; MSIE or Firefox mutant; not on Windows server;) Daumoa 4.0
  Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.0) LinkCheck by Siteimprove.com
  EdisterBot (http://www.edister.com/bot.html)
  Mozilla/5.0 (iPhone; CPU iPhone OS 6_0 like Mac OS X) AppleWebKit/536.26 (KHTML, like Gecko) Version/6.0 Mobile/10A5376e Safari/8536.25 (compatible; SMTBot/1.0; +http://www.similartech.com/smtbot)
  Tarantula/BETA-0.5 (Tarantula de la Enlle, el mejor buscador en la red; http://www.enlle.com; [email protected])
  Mozilla/5.0 (Windows; U; Windows NT 5.1; fr; rv:1.8.1) VoilaBot BETA 1.2 ([email protected])
  yacybot (freeworld/global; amd64 Linux 2.6.38-8-generic; java 1.6.0_22; Europe/en) http://yacy.net/bot.html
  Mozilla/5.0 (compatible; SemrushBot/0.95; +http://www.semrush.com/bot.html)
  CatchBot/3.0; +http://www.catchbot.com
  Mozilla/5.0 (compatible; uMBot-LN/1.0; mailto: [email protected])
  Mozilla/5.0 (compatible; RukiCrawler/1.0; +http://ruki.rezko.net)
  adbeat_bot
  Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/27.0.1453.116 Safari/537.36 HubSpot Webcrawler
  Mozilla/5.0 (compatible; YandexCatalog/3.0; +http://yandex.com/bots)
  NutchCVS/0.7.1 (Nutch; http://lucene.apache.org/nutch/bot.html; [email protected])
  Statastico/4.0
  facebookexternalhit/1.1 (+http://www.facebook.com/externalhit_uatext.php)
  Mozilla/5.0 (X11; U; Linux x86_64; en-US) AppleWebKit/534.16 (KHTML, like Gecko, Google Keyword Suggestion) Chrome/10.0.648.127 Safari/534.16
  Mozilla/5.0 (iPhone; CPU iPhone OS 7_0 like Mac OS X) AppleWebKit/537.51.1 (KHTML, like Gecko) Version/7.0 Mobile/11A465 Safari/9537.53 BingPreview/1.0b
  EuripBot/2.0 (+http://www.eurip.com) GetRobots
  Mozilla/5.0 (compatible; MJ12bot/v1.2.1; http://www.majestic12.co.uk/bot.php?+)
  Braintree-Webhooks
  Nutch Checksem/Nutch-1.10
  Mozilla/5.0 (compatible; HyperCrawl/0.2; +http://www.seograph.net/bot.html)
  quickobot/quickobot-1 (Quicko Labs; http://quicko.co; robot at quicko dot co)
  Whibse
  Yahoo! Site Explorer Feed Validator http://help.yahoo.com/l/us/yahoo/search/siteexplorer/manage/
  Mozilla/5.0 (compatible; AhrefsBot/5.0; +http://ahrefs.com/robot/)
  Siteshooter B0t 2.1
  GingerCrawler/1.0 (Language Assistant for Dyslexics; www.gingersoftware.com/crawler_agent.htm; support at ginger software dot com)
  Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko; Google Page Speed Insights) Chrome/27.0.1453 Safari/537.36
  eBot / v.0.1e (http://alfa.elchron.cz)
  Mozilla/5.0 (compatible; memoryBot/1.20.247 +http://internetmemory.org/en/)
  yacybot (x86 Windows 2003 5.2; java 1.6.0_20; America/de) http://yacy.net/bot.html
  YandexSomething/1.0
  Scoop
  Mozilla/5.0 (compatible; MegaIndex.ru/2.0; +http://megaindex.com/crawler)
  Mozilla/5.0 (compatible; SputnikBot/2.3; +http://corp.sputnik.ru/webmaster)
  TurnitinBot/2.1 (http://www.turnitin.com/robot/crawlerinfo.html)
  Mozilla/5.0 (compatible; Ask Jeeves/Teoma; +http://about.ask.com/en/docs/about/webmasters.shtml)
  Mozilla/5.0 (compatible; CrawlForMe/3.0.2; [email protected])
  Seologies/0.1 Bot
  www.deadlinkchecker.com XMLHTTP/1.0 Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/46.0.2490.86 Safari/537.36
  dubaiindex (addressendeutschland.de)
  yacybot (/global; x86 Windows 7 6.1; java 1.8.0_40; America/es) http://yacy.net/bot.html
  Mozilla/5.0+(compatible;+PiplBot;++http://www.pipl.com/bot/)
  LinkStats Bot
  Mozilla/5.0 (compatible; adbeat-publisher-description-fetcher; [email protected]; [email protected])
  CISPA Vulnerability Notification (https://notify.mmci.uni-saarland.de)
  librabot/1.0 (+http://search.msn.com/msnbot.htm)
  SeoCheckBot (Seomira Seo Check, http://www.kfsw.de/bot.html)
  yacybot (x86 Windows 2003 5.2; java 1.6.0_16; Europe/de) http://yacy.net/bot.html
  WatchMouse/8.4.0.3 (http://watchmouse.com/ ; itpda01.watchmouse.net)
  yacybot (/global; amd64 Linux 3.13.0-83-generic; java 1.7.0_95; Europe/en) http://yacy.net/bot.html
  BrainbruBot/1.0 (+http://www.brainbru.com/)
  Docoloc
  Mozilla/5.0 (Unknown; Linux x86_64) AppleWebKit/538.1 (KHTML, like Gecko) PhantomJS/2.1.1 Safari/538.1 Prerender (+https://github.com/prerender/prerender)
  dlvr.it/1.0 (+http://dlvr.it/)
  Mozilla/5.0 (compatible; parsijoo-bot; +http://www.parsijoo.ir/; [email protected])
  DataparkSearch/4.35-02122005 ( http://www.dataparksearch.org/)
  Pixray-Seeker/2.0 (Pixray-Seeker; +http://www.pixray.com/pixraybot; [email protected])
  Abrave v5.5 (http://robot.abrave.co.uk)
  Mozilla/5.0 (X11; U; Linux i686 (x86_64); en-US; rv:1.8.1.20) Gecko/20090429 HeartRails_Capture/0.6 (+http://capture.heartrails.com/) BonEcho/2.0.0.20
  yacybot (/global; amd64 Linux 3.19.0-26-generic; java 1.7.0_95; Europe/en) http://yacy.net/bot.html
  MLBot (www.metadatalabs.com/mlbot)
  yacybot (freeworld/global; amd64 Linux 2.6.32-34-server; java 1.6.0_26; Europe/de) http://yacy.net/bot.html
  Mozilla/5.0 (compatible; LinkpadBot/1.08; +http://www.linkpad.ru)
  Mozilla/5.0 (compatible; SearchmetricsBot; http://www.searchmetrics.com/en/searchmetrics-bot/)
  msnbot-NewsBlogs/2.0b (+http://search.msn.com/msnbot.htm)
  ZnajdzFoto/Image 2.0
  Mozilla/5.0 (compatible; Magibot/4.0.0; +http://magi.peak-labs.com/robots.txt)
  Mozilla/5.0 (compatible; spbot/4.0.8; +http://OpenLinkProfiler.org/bot )
  Eurobot/1.2 (http://eurobot.ayell.eu)
  Mozilla/5.0 (compatible; Embedly/0.2; snap; +http://support.embed.ly/)
  Yaanb/1.5.001 (compatible; Win64;+http://www.yaanb.com/company/bot.hmtl)
  yacybot (i386 Linux 2.6.31-16-generic; java 1.6.0_15; Europe/en) http://yacy.net/bot.html
  Mozilla/5.0 (compatible; YioopBot; +http://173.13.143.74/bot.php)
  CopperEgg/RevealUptime/Singapore(aws)
  Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/534.24 (KHTML, like Gecko; Google Web Preview) Chrome/11.0.696 Safari/534.24
  yacybot (freeworld/global; amd64 Windows 8 6.2; java 1.7.0_25; Europe/de) http://yacy.net/bot.html
  Server Density Service Monitoring v2
  Testomatobot/1.0 (Linux x86_64; +http://www.testomato.com/testomatobot) minicrawler/3.0~beta
  TinEye/1.0; +http://www.tineye.com/
  Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.9.0.19; aggregator:Spinn3r (Spinn3r 3.1); http://spinn3r.com/robot) Gecko/2010040121 Firefox/3.0.19
  GigablastOpenSource/1.0
  Googlebot (gocrawl v0.4)
  CSmartCrawler (email: abuse[at]adsadv[dot]com)
  Testomatobot/1.0 (Linux x86_64; +http://www.testomato.com/testomatobot) minicrawler/4.0.0~beta9
  KOCMOHABT (+http://kozmonavt.tk/) AppEngine-Google; (+http://code.google.com/appengine; appid: s~web-explorer)
  Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.2.18) Gecko/20110622 HeartRails_Capture/1.0.5 (+http://capture.heartrails.com/) Namoroka/3.6.18
  PingSpot/1.0 (http://www.pingspot.org/)
  AppEngine-Google; (+http://code.google.com/appengine; appid: s~buzzstream-tools-hrd)
  AddThis.com (http://support.addthis.com/)
  Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko; Google Web Preview) Chrome/27.0.1453 Safari/537.36
  ExactSeekCrawler/1.0
  Yeti/1.0 (NHN Corp.; http://help.naver.com/robots/)
  Mozilla/5.0 (compatible; jpg-newsbot/2.0; +http://vipnytt.no/bot.html)
  g2reader-bot/1.0 (+http://www.g2reader.com/)
  MetaGeneratorCrawler/1.3.12 (www.metagenerator.info)
  QuerySeekerSpider ( http://queryseeker.com/bot.html )
  KDDI-CA31 UP.Browser/6.2.0.7.3.129 (GUI) MMP/2.0 (compatible; ichiro/mobile goo; +http://help.goo.ne.jp/help/article/1142/)
  Mozilla/5.0 (compatible; GrapeshotCrawler/2.0; +http://www.grapeshot.co.uk/crawler.php)
  Mozilla/5.0 (compatible; MSIE 10.0; Windows NT 6.2; Trident/6.0; WOW64; Trident/6.0; BingPreview/1.0b)
  Mozilla/5.0 (compatible; firmilybot/0.3; +http://www.firmily.com/bot.php
  Mozilla/5.0 (X11; U; Linux Core i7-4980HQ; de; rv:32.0; compatible; JobboerseBot; http://www.jobboerse.com/bot.htm) Gecko/20100101 Firefox/38.0
  Mozilla/5.0 (compatible; metager2-verification-bot; +http://metager2.de/technology.php)
  crawler4j (http://code.google.com/p/crawler4j/)
  CirrusExplorer/1.1 (http://www.cireu.com/explorer.php)
  Mozilla/5.0 (compatible; RankvalBot/1.0; +https://rankval.com)
  Mozilla/5.0 (compatible; heritrix/1.14.3 +http://www.accelobot.com)
  Mozilla/5.0 (compatible; BLEXBotTest/1.0; +http://webmeup.com/crawler.html)
  nWormFeedFinder (http://www.nworm.com)
  netEstate NE Crawler (+http://www.sengine.info/)
  Mozila/5.0 (compatible; StudioFACA Search; +http://www.studiofaca.com/)
  Mozilla/5.0 (compatible; Crawlera/1.10.2; UID 92751)
  wikiwix-bot-3.0
  SEOCentro Page Keyword Analyzer v1.2
  yacybot (amd64 Linux 2.6.18-164.15.1.el5xen; java 1.6.0_0; Europe/en) http://yacy.net/bot.html
  Mozilla/5.0 (compatible; bingbot/2.0; http://www.bing.com/bingbot.htm)
  Automattic Analytics Crawler/0.1; http://wordpress.com/crawler/
  SemrushBot/0.92
  http://www.uni-koblenz.de/~flocke/robot-info.txt
  Mozilla/5.0 (compatible; idmarch Automatic.beta/1.3; +http://www.idmarch.org/bot.html)
  Mozilla/5.0 (X11; Linux i686 on x86_64; rv:25.0; GTmetrix http://gtmetrix.com/) Gecko/20100101 Firefox/25.0
  msnbot/2.0b (+http://search.msn.com/msnbot.htm)._
  Stroke.cz (http://stroke.cz)
  Mozilla (+http://web.nli.org.il/sites/NLI/English/collections/israel-collection/internet_archive/Pages/default.aspx)
  audisto.com full crawler 4.33.470 (refer to in robots.txt as audisto, see https://audisto.com/bot)
  http://arachnode.net 2.5
  SurveyBot/2.3 (Whois Source)
  ant.com/Nutch-1.7 (http://ant.com)
  CorpusCrawler 2.0.13 (http://corpora.fi.muni.cz/crawler/)
  Seo Servis - Analyza zdrojoveho kodu
  SEOENGBot/1.2 (+http://www.seoengine.com/seoengbot.htm)
  Mozilla/5.0 (compatible; Genieo/1.0 http://www.genieo.com/webfilter.html)
  Mozilla/5.0 (compatible; evc-batch/2.0.20160708184126)
  HubSpot Marketing Grader
  Mozilla/5.0 (compatible; MSIE 10.0; Windows NT 6.1; Trident/6.0) LinkCheck by Siteimprove.com
  nrsbot/6.0(loopip.com/robot.html)
  Mozilla/5.0 (compatible; BLEXBot/1.0; +http://webmeup.com/crawler.html)
  yacybot (freeworld-global; amd64 Windows 7 6.1; java 1.7.0_02-ea; America/en) http://yacy.net/bot.html
  Mozilla/5.0 (compatible; Blekkobot; ScoutJet; +http://blekko.com/about/blekkobot)
  Mozilla/5.0 (compatible; Topicbot/12.0rc-2; +http://topicbot.awardspace.us/)
  Mozilla/5.0 (compatible; Gimme60bot/1.0; +http://gimme60.com)
  yacybot (i386 Linux 2.6.28-11-generic; java 1.6.0_13; Europe/de) http://yacy.net/bot.html
  Mozilla/5.0 (compatible; DotBot/1.1; http://www.opensiteexplorer.org/dotbot, [email protected])
  Kemvibot/1.0 (http://kemvi.com, [email protected])
  CorpusCrawler 2.0.15 (http://corpora.fi.muni.cz/crawler/)
  dotbot
  Mozilla/5.0 (iPhone; CPU iPhone OS 7_0 like Mac OS X) AppleWebKit/537.51.1 (KHTML, like Gecko) Version/7.0 Mobile/11A465 Safari/9537.53 (compatible; adidxbot/2.0; +http://www.bing.com/bingbot.htm)
  page_viewer ([email protected])
  HootSuite Image proxy ([email protected])
  Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US) Speedy Spider for SpeedyAds (http://www.entireweb.com/about/search_tech/speedy_spider/)
  PINGOMETER_BOT_(HTTPS://PINGOMETER.COM)
  Mozilla/5.0 (compatible; Vagabondo/2.1; webcrawler at wise-guys dot nl; http://webagent.wise-guys.nl/)
  VSE/1.0 ([email protected])
  Scrapy/0.24.0 (+http://scrapy.org)
  Speedy Spider (http://www.entireweb.com)
  Mozilla/5.0 (X11; Linux x86_64; GTmetrix https://gtmetrix.com/) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/51.0.2704.106 Safari/537.36
  WatchMouse/18990 (http://watchmouse.com/ ; hk)
  Mozilla/5.0 (compatible; yoozBot-2.2; http://yooz.ir; [email protected])
  Mozilla/5.0 (compatible; Crawlera/1.10.2; UID 87214)
  Mozilla/5.0 (compatible; discobot/2.0; +http://discoveryengine.com/discobot.html)
  yacybot (freeworld/global; amd64 Windows 7 6.1; java 1.8.0_11; Europe/de) http://yacy.net/bot.html
  Testomatobot/1.0 (Linux x86_64; +http://www.testomato.com/testomatobot) minicrawler/4.0.0~beta6
  yacybot (freeworld/global; amd64 Linux 3.0.0-15-server; java 1.6.0_23; Europe/en) http://yacy.net/bot.html
  Mozilla/5.0 (Macintosh; Intel Mac OS X 10.9; rv:28.0) Gecko/20100101 Firefox/28.0 (FlipboardProxy/1.1; +http://flipboard.com/browserproxy)
  YisouSpider
  Slack-ImgProxy 1.138 (+https://api.slack.com/robots)
  mozilla/5.0 (compatible; webmastercoffee/0.7; +http://webmastercoffee.com/about)
  Mozilla/5.0 (compatible; Uptimebot/0.2.22; +http://www.uptime.com/uptimebot)
  Mozilla/5.0 (compatible; AhrefsBot/1.0; +http://ahrefs.com/robot/)
  Mozilla/5.0 (compatible; Feedspotbot/1.0; +http://www.feedspot.com/fs/bot)
  sg-Orbiter/1.0 (+http://searchgears.de/uber-uns/crawling-faq.html)
  Site24x7 Tools
  NextGenSearchBot 1 (for information visit http://www.zoominfo.com/About/misc/NextGenSearchBot.aspx)
  Aboundex/0.3 (http://www.aboundex.com/crawler/)
  yacybot (freeworld/global; i386 Linux 2.6.37.6-0.5-desktop; java 1.6.0_20; Europe/en) http://yacy.net/bot.html
  Mozilla/5.0 (Windows NT 6.2; WOW64) AppleWebKit/537.4 (KHTML, like Gecko) Chrome/98 Safari/537.4 (StatusCake)
  Mozilla/5.0 (compatible; DomainSONOCrawler/0.1; +http://domainsono.com)
  Mozilla/5.0 (compatible; Online Domain Tools - Server Monitor/1.0; +http://server-monitoring.online-domain-tools.com)
  Mozilla/5.0 (compatible; Konqueror/3.5; Linux) KHTML/3.5.5 (like Gecko) (Exabot-Thumbnails)
  Mozilla/5.0 (compatible; spbot/4.0.4; +http://www.seoprofiler.com/bot )
  Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/27.0.1453 Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
  Factbot 1.09 (see http://www.factbites.com/webmasters.php)
  CopperEgg/RevealUptime/SaoPauloBR(aws)
  CRIM Crawler/Nutch-2.3 (Crawler du Centre de Recherche Informatique de Montr\xc3\xa9al (CRIM))
  Mozilla/5.0 (compatible; heritrix/1.14.3 +http://www.webarchiv.cz)
  Mozilla/5.0 (compatible; WormlyBot; +http://wormly.com)
  yacybot (i386 Linux 2.6.23; java 1.6.0_17; Europe/en) http://yacy.net/bot.html
  Mozilla/5.0 (compatible; Cliqzbot/0.1 +http://cliqz.com/company/cliqzbot)
  URLitor.com
  RavenCrawler
  Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.2.18) Gecko/20130331 HeartRails_Capture/1.0.5 (+http://capture.heartrails.com/) Namoroka/3.6.18
  Mozilla/5.0 (compatible; WebThumbnail/3.x; Website Thumbnail Generator; +http://webthumbnail.org)
  Kyoto-Crawler/2.0 (Mozilla-compatible; kyoto-crawler-contact(at)nlp(dot)kuee(dot)kyoto-u(dot)ac(dot)jp; http://nlp.ist.i.kyoto-u.ac.jp/)
  Mozilla/5.0 (compatible; SputnikImageBot/2.3; +http://corp.sputnik.ru/webmaster)
  WatchMouse/8.4.0.3 (http://watchmouse.com/ ; lvps91-250-96-109.dedicated.hosteurope.de)
  MetaGeneratorCrawler/1.3.3 (www.metagenerator.info)
  LongURL API
  Mozilla/5.0 ( compatible; SETOOZBOT/0.30 ; http://www.setooz.com/bot.html ; agentname at setooz dot_com )
  Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.6; en-US; rv:1.9.2) Gecko/20100115 Firefox/3.6 (FlipboardProxy/2.0; +http://flipboard.com/browserproxy)
  http://tools.geek-tools.org/link-counter/
  SCFCrawler/Nutch-1.8 (Image Crawler for StolenCameraFinder.com; http://www.stolencamerafinder.com/; [email protected])
  Mozilla/5.0 (compatible; Uptimebot/0.2.41; +http://www.uptime.com/uptimebot)
  Mozilla/5.0 (compatible; DomainAppender /1.0; +http://www.profound.net/domainappender)
  Mozilla/5.0 (compatible; hypestat/1.0; +http://www.hypestat.com/bot)
  yacybot (/global; amd64 Linux 3.13.0-042stab093.4; java 1.7.0_79; Europe/en) http://yacy.net/bot.html
  PagesInventory (robot +http://www.pagesinventory.com)
  Sosospider+(+http://help.soso.com/webspider.htm)
  Mozilla/5.0 (compatible; AntBot/1.0; +http://www.ant.com/)
  PagesInventory (robot http://www.pagesinvenotry.com)
  Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; http://www.changedetection.com/bot.html )
  Wotbox/2.01 (+http://www.wotbox.com/bot/)
  Mozilla/5.0 (compatible; bixolabs/1.0; +http://bixolabs.com/crawler/general; [email protected])
  Mozilla/5.0 (compatible; SpiderLing (a SPIDER for LINGustic research); +http://nlp.fi.muni.cz/projects/biwec/)
  CopperEgg/RevealUptime/DallasTXUSA
  yacybot (freeworld/global; amd64 Linux 2.6.32-custom; java 1.6.0_26; Europe/en) http://yacy.net/bot.html
  Bad-Neighborhood Link Analyzer (http://www.bad-neighborhood.com/)
  Mozilla/5.0 (compatible; oBot/2.3.1; +http://filterdb.iss.net/crawler/)
  Pixray-Seeker/2.0 (Pixray-Seeker; http://www.pixray.com/pixraybot; [email protected])
  Cityreview Robot (+http://www.cityreview.org/crawler/)
  yacybot (/global; amd64 Linux 3.16.0-4-amd64; java 1.7.0_95; Europe/en) http://yacy.net/bot.html
  Dlvr.it/1.0 (http://dlvr.it/)
  Zookabot/2.4;++http://zookabot.com
  AddThis.com robot [email protected]
  Mozilla/5.0 (compatible; discoverybot/2.0; +http://discoveryengine.com/discoverybot.html)
  TwengaBot
  Mozilla/5.0 (compatible; 4SeoHuntBot; +http://4seohunt.biz/about.html)
  Web-sniffer.me/1.0.0 (+http://web-sniffer.me/)
  KDDI-CA31 UP.Browser/6.2.0.7.3.129 (GUI) MMP/2.0 (compatible; ichiro/mobile goo; +http://search.goo.ne.jp/option/use/sub4/sub4-1/)
  gosquared-thumbnailer/1.0
  yacybot (/global; amd64 Windows 7 6.1; java 1.7.0_55; Asia/zh) http://yacy.net/bot.html
  BPImageWalker/2.0 (www.bdbrandprotect.com)
  Acoon v4.10.3 (www.acoon.de)
  Mozilla/4.0 (CMS Crawler: http://www.cmscrawler.com)
  WP Engine Install Performance API
  Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.2.24; ips-agent) Gecko/20111107 Ubuntu/10.04 (lucid) Firefox/3.6.24
  Mozilla/5.0 (Windows NT 6.1; WOW64; Trident/7.0; rv:11.0) like Gecko PTST/277
  CamontSpider/1.0 +http://epweb2.ph.bham.ac.uk/user/slater/camont/info.html
  SEOENGBot/1.2 (+http://learn.seoengine.com/seoengbot.htm)
  Y!J-BRJ/YATS crawler (http://listing.yahoo.co.jp/support/faq/int/other/other_001.html)
  CopperEgg/RevealUptime/N.Virginia(aws)
  Mozilla/5.0, Google-AdWords-Express
  Mozilla/5.0 (compatible;YodaoBot-Image/1.0;http://www.youdao.com/help/webmaster/spider/;)
  yacybot (/global; amd64 Linux 3.16.0-4-amd64; java 1.8.0_91; Europe/en) http://yacy.net/bot.html
  yacybot (freeworld/global; amd64 Windows 7 6.1; java 1.7.0_09; Europe/de) http://yacy.net/bot.html
  Acoon v4.10.1 (www.acoon.de)
  Mediatoolkitbot ([email protected])
  Mozilla/5.0 (compatible; Speedy Spider; http://www.entireweb.com/about/search_tech/speedy_spider/)
  Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.2.3) SPEng
  2Bone_LinkChkr/1.0 libwww-perl/6.03
  Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.6; en-US; rv:1.9.2) Gecko/20100115 Firefox/3.6 (FlipboardBrowserProxy/0.0.5; +http://flipboard.com/browserproxy)
  Mozilla/5.0 (compatible; SeznamBot/3.2-test1; +http://napoveda.seznam.cz/en/seznambot-intro/)
  Comodo SSL Checker
  Comodo Spider 1.2
  Who.is Bot
  Mozilla/5.0 (compatible; online-webceo-bot/1.0; +http://online.webceo.com)
  yacybot (freeworld/global; amd64 Windows Server 2008 R2 6.1; java 1.6.0_29; Europe/en) http://yacy.net/bot.html
  psbot/0.1 (+http://www.picsearch.com/bot.html)
  IlTrovatore-Setaccio/1.2 (http://www.iltrovatore.it/aiuto/faq.html)
  Mozilla/5.0 (compatible; Twingly Recon; twingly.com)
  Mozilla/5.0 Moreover/5.1 (+http://www.moreover.com; [email protected])
  Orgbybot/OrgbyBot v1.2 (Spidering the net for Orgby; http://www.orgby.com/ ; Orgby.com Search Engine)
  Browsershots
  AdsBot-Google (+http://www.google.com/adsbot.html)
  Qseero v1.0.0
  Mozilla/5.0 (compatible; ltbot/0.3.6 +http://www.kdsl.tu-darmstadt.de/de/kdsl/research-program/crawling-and-semantic-structuring/)
  yacybot (x86 Windows XP 5.1; java 1.6.0_18; Europe/de) http://yacy.net/bot.html
  visionutils/0.2
  Mozilla/5.0 (compatible; Googlebot/2.1; https://www.deepcrawl.com/bot)
  Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.4 (KHTML, like Gecko; Google Web Preview) Chrome/22.0.1229 Safari/537.4
  Mozilla/5.0 (compatible; Crawlera/1.10.2; UID 8942)
  Mozilla/4.0 (compatible; LinkMarket-Bot v1.2)
  Buzzbot/1.0 (Buzzbot; http://www.buzzstream.com; [email protected])
  WordPress.com; https://public-api.wordpress.com
  Mozilla/5.0 ( compatible; SETOOZBOT/0.30 ; http://www.setooz.com/bot.html )
  CopperEgg/RevealUptime/FremontCAUSA
  Mozilla/5.0 (Windows; U; Windows NT 6.0; en-GB; rv:1.0; trendictionbot0.4.5; trendiction search; http://www.trendiction.de/bot; please let us know of any problems; web at trendiction.com) Gecko/20071127 Firefox/3.0.0.11
  yacybot (/global; amd64 Linux 2.6.32-042stab094.8; java 1.7.0_79; America/en) http://yacy.net/bot.html
  Mozilla/5.0 (compatible; Windows NT 6.1?; ZumBot/1.0; http://help.zum.com/inquiry)
  Mozilla/5.0 (Windows NT 6.1; WOW64; Trident/7.0; rv:11.0; topster.de Linkchecker 5.0) like Gecko (90.177.52.133)
  Mozilla/5.0+(compatible; Monitority/1.0; http://www.monitority.com/)
  yacybot (freeworld/global; x86 Windows 7 6.1; java 1.8.0_25; Europe/de) http://yacy.net/bot.html
  Mozilla/5.0 (compatible; abby/1.0; +http://www.ellerdale.com/crawler.html)
  Mozilla/5.0 (compatible; SemrushBot/1.1~bl; +http://www.semrush.com/bot.html)
  YowedoBot/Yowedo 1.0 (Search Engine crawler for yowedo.com; http://yowedo.com/en/partners.html; [email protected])
  Mozilla/5.0 (compatible; oBot/2.3.1; http://filterdb.iss.net/crawler/)
  Nokia6820/2.0 (4.83) Profile/MIDP-1.0 Configuration/CLDC-1.0 (compatible; Googlebot-Mobile/2.1; +http://www.google.com/bot.html)
  Mozilla/5.0 (compatible; IstellaBot/1.23.15 +http://www.tiscali.it/)
  GetIntent Crawler (http://getintent.com/bot.html)
  Mozilla/5.0 (compatible; Uptimebot/0.1.89; +http://www.uptime.com/uptimebot)
  Cliqzbot
  AboutUsBot
  yacybot (-global; amd64 Linux 3.2.0-4-amd64; java 1.7.0_67; Europe/en) http://yacy.net/bot.html
  Sufog/Nutch-2.2.1 (www.sufog.com; www.sufog.com)
  Mozilla/5.0 (compatible; imbot/0.1 +http://internetmemory.org/en/)
  KOCMOHABT (+http://kozmonavt.tk/) Mozilla/5.0 (compatible; Web Explorer 1) AppEngine-Google; (+http://code.google.com/appengine; appid: s~web-explorer)
  yacybot (/global; amd64 Linux 3.16.0-53-generic; java 1.7.0_85; Europe/en) http://yacy.net/bot.html
  L.webis/0.53 (http://webalgo.iit.cnr.it/index.php?pg=lwebis)
  Mozilla/5.0 (compatible; memoryBot/1.21.23 +http://internetmemory.org/en/)
  ExB Language Crawler 2.1.5 (+http://www.exb.de/crawler)
  Peeplo Screenshot Bot/0.20 ( abuse at peeplo dot_com )
  Mozilla/5.0 (compatible; Qwantify/2.1w; +https://www.qwant.com/)/*
  Experibot_v1 [bit.ly/2bLMJdw]
  WikioFeedBot 1.0 (http://www.wikio.com)
  SalesIntelligent/v1.1 (https://www.salesintelligent.com)
  Mozilla/5.0 (compatible; ExaleadCloudView/5;)
  Mozilla/5.0 (compatible; rogerBot/1.0; UrlCrawler; http://www.seomoz.org/dp/rogerbot)
  Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/44.0.2403.157 Safari/537.36 TinEye/1.0 (via http://www.tineye.com/)
  Mozilla/5.0 (compatible; Search17Bot/1.1; http://www.search17.com/bot.php)
  FollowSite Bot ( http://www.followsite.com/bot.html )
  Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/27.0.1453.116 Safari/537.36 HubSpot Webcrawler - [email protected]
  Mozilla/5.0 (compatible; BlinkaCrawler/1.0; +http://www.blinka.jp/crawler/)
  Mozilla/5.0 (compatible; Qwantify/2.2w; +https://www.qwant.com/)/*
  findlinks/1.1.6-beta2 (+http://wortschatz.uni-leipzig.de/findlinks/)
  yacybot (/global; amd64 Linux 4.4.0-28-generic; java 1.8.0_91; Europe/pl) http://yacy.net/bot.html
  yacybot (/global; amd64 Windows Server 2008 R2 6.1; java 1.8.0_40; Europe/de) http://yacy.net/bot.html
  Mozilla/5.0 (compatible; 200PleaseBot/1.0; +http://www.200please.com/bot)
  WatchMouse/8.4.0.3 (http://watchmouse.com/ ; chzrh01.watchmouse.net)
  Iframely/0.8.8 (+http://iframely.com/;)
  Mozilla/5.0 (compatible; URLAppendBot/1.0; +http://www.profound.net/urlappendbot.html)
  LoadImpactRload/3.2.0 (Load Impact; http://loadimpact.com);
  RSSingBot (http://www.rssing.com)
  RADaR-Bot/Nutch-1.3 (http://radar-bot.com/)
  Ocelli/1.4 (http://www.globalspec.com/Ocelli)
  Mozilla/5.0 (compatible; BuzzSumo; +http://www.buzzsumo.com/bot.html)
  Mozilla/4.0 (compatible; IDwhois/2.0alpha-10102619; Linux; en-US; +http://idwhois.info/about)
  omgilibot/0.4 +http://omgili.com
  yacybot (freeworld/global; amd64 Windows 7 6.1; java 1.6.0_23; Europe/de) http://yacy.net/bot.html
  Y!J-BRW/1.0 crawler (http://help.yahoo.co.jp/help/jp/search/indexing/indexing-15.html)
  Mozilla/5.0 (compatible; AhrefsBot/2.0; +http://ahrefs.com/robot/)
  Mozilla/5.0 (compatible; VideoSurf_bot +http://www.videosurf.com/bot.html)
  Symfony Spider (http://symfony.com/spider)
  Mozilla/5.0 (compatible; Sosospider/2.0; +http://help.soso.com/webspider.htm)
  Mozilla/5.0 (compatible; OpenindexDeepSpider/Nutch-1.5-dev; +http://www.openindex.io/en/webmasters/spider.html)
  ADmantX Platform Semantic Analyzer - ADform - ADmantX Inc. - www.admantx.com - [email protected]
  Mozilla/5.0 (compatible; MJ12bot/v1.3.3; http://www.majestic12.co.uk/bot.php?+)
  cuwhois/1.0 (+http://www.cuwhois.com/)
  Mozilla/5.0 (compatible; memoryBot/1.20.311 +http://internetmemory.org/en/)
  Mozilla/4.0 (compatible; Vagabondo/4.0; webcrawler at wise-guys dot nl; http://webagent.wise-guys.nl/; http://www.wise-guys.nl/)
  yacybot (-global; amd64 Linux 3.16.0-4-amd64; java 1.7.0_75; Europe/en) http://yacy.net/bot.html
  Mozilla/5.0 (compatible; Dazoobot/1.0; http://dazoo.fr)
  Mozilla/5.0 (compatible; Semager/1.4c; +http://www.semager.de/blog/semager-bots/)
  EuripBot/1.1 (+http://www.eurip.com) GetRobots
  yacybot (freeworld/global; amd64 Linux 3.8.0-23-generic; java 1.6.0_27; Pacific/en) http://yacy.net/bot.html
  yacybot (amd64 Linux 2.6.24-23-generic; java 1.6.0_07; Europe/de) http://yacy.net/bot.html
  yacybot (freeworld/global; x86 Windows 2003 5.2; java 1.6.0_32; Europe/de) http://yacy.net/bot.html
  yacybot (freeworld/global; amd64 Linux 3.2.0-4-amd64; java 1.7.0_25; Europe/de) http://yacy.net/bot.html
  socialbm_bot http://spider.socialbm.net
  Mozilla/5.0 (compatible; 007ac9 Crawler; http://crawler.007ac9.net/)
  webinatorbot 1.0; +http://www.webinator.de
  WorldBrewBot/2.1 (+http://www.marketbrew.com/)
  BotOnParade, http://www.bots-on-para.de/bot.html
  ltx71 - (http://ltx71.com/)
  VSB-TUO/1.0 (+http://www.vsb.cz)
  yacybot (/global; amd64 Linux 3.13.0-85-generic; java 1.7.0_101; Europe/en) http://yacy.net/bot.html
  CopperEgg/RevealUptime/TokyoJP(aws)
  Mozilla/5.0 (compatible; etoolsbot/1.0; +https://etools.io/bots/)
  Comodo-Webinspector-Crawler 2.2.2, http://www.comodorobot.com
  Mozilla/5.0 (compatible; TWMBot/0.1; +http://thewebminer.com)
  Mozilla/5.0 (compatible; Plukkie/1.3; http://www.botje.com/plukkie.htm)
  Autonomy
  yacybot (i386 Linux 2.6.30-2-686; java 1.6.0_0; SystemV/en) http://yacy.net/bot.html
  Testomatobot/1.0 (Linux x86_64; +http://www.testomato.com/testomatobot) minicrawler/4.1.0
  CatchBot/1.0; +http://www.catchbot.com
  Sitedomain-Bot(Sitedomain-Bot 1.0, http://www.sitedomain.de/sitedomain-bot/)
  Mozilla/5.0 (compatible; Uptimebot/0.2.42; +http://www.uptime.com/uptimebot)
  yacybot (/global; amd64 Linux 3.2.0-4-amd64; java 1.7.0_03; Europe/en) http://yacy.net/bot.html
  SiteTruth.com site rating system
  SiteUptime.com
  yacybot (i386 Linux 2.6.24-23-generic; java 1.6.0_16; Europe/en) http://yacy.net/bot.html
  UrlTrends Ranking Updater/2.0
  yacybot (/global; amd64 Windows 7 6.1; java 1.7.0_55; Europe/de) http://yacy.net/bot.html
  yacybot (freeworld/global; amd64 Linux 3.5.0-27-generic; java 1.7.0_25; Europe/en) http://yacy.net/bot.html
  GarlikCrawler/1.1 (http://garlik.com/, [email protected])
  Mozilla/5.0 (compatible; ParsijooBot; +http://www.parsijoo.ir/; [email protected])
  Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.1 (KHTML, like Gecko) Chrome/21.0.1180.89 Safari/537.1; 360Spider(compatible; HaosouSpider; http://www.haosou.com/help/help_3_2.html)
  Willow Internet Crawler by Twotrees V2.1
  KrOWLer/0.1.0, matentzn at cs dot man dot ac dot uk
  SimplyFast.info Headers
  yacybot (freeworld/global; amd64 Linux 2.6.32-431.29.2.el6.x86_64; java 1.7.0_65; Europe/en) http://yacy.net/bot.html
  Mozilla/5.0 (compatible; YandexBlogs/0.99; robot; +http://yandex.com/bots)
  msnbot-media/1.1 (+http://search.msn.com/msnbot.htm)
  Mozilla/5.0 (Windows NT 6.1; WOW64; rv:46.0) Gecko/20100101 Firefox/46.0 PTST/278
  Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/534.57.2 (KHTML, like Gecko) Version/5.1.7 Safari/534.57.2 PTST/278
  gipo-crawler/Nutch-1.10 (Global Internet Policy Observatory crawler)
  Mozilla/5.0 (compatible; sukibot_heritrix/3.1.1 +http://suki.ling.helsinki.fi/eng/webmasters.html)
  PayPal IPN ( https://www.paypal.com/ipn )
  Mozilla/5.0 (compatible; KaloogaBot; http://kalooga.com/crawler)
  eCairn-Grabber/1.0 (+http://ecairn.com/grabber) curl/7.15
  Mozilla/5.0 (Windows NT 6.2; WOW64) AppleWebKit/537.15 (KHTML like Gecko; +http://www.diffbot.com) Chrome/24.0.1295.0 Safari/537.15
  ImageEngine/1.0
  DER-bot (+https://goo.gl/9ZibHG)
  yacybot (freeworld/global; amd64 Windows 7 6.1; java 1.7.0_60; Europe/de) http://yacy.net/bot.html
  Mozilla/5.0 (compatible; SemrushBot/0.96.3; +http://www.semrush.com/bot.html)
  Mozilla/5.0 (compatible; Exabot/3.0; +http://www.exabot.com/go/robot)
  Mozilla/5.0 (compatible; YandexDirect/3.0; +http://yandex.com/bots)
  Mozilla/5.0 (compatible; aiHitBot/2.9; +https://www.aihitdata.com/about)
  Linguee Bot (http://www.linguee.com/bot; [email protected])
  MetaGeneratorCrawler/1.3.9 (www.metagenerator.info)
  TelegramBot (like TwitterBot)
  Mozilla/5.0 (compatible; Siteliner/1.0; +http://www.siteliner.com/bot)
  calculon spider/0.1.0 mailto:[email protected]
  yacybot (/global; amd64 Linux 3.13.0-83-generic; java 1.7.0_95; Europe/de) http://yacy.net/bot.html
  PocketParser/2.0 (+https://getpocket.com/pocketparser_ua)
  montastic-monitor http://www.montastic.com
  Mozilla/5.0 (compatible; trovitBot 1.0; +http://www.trovit.com/bot.html)
  Mozilla/5.0 (Windows NT 6.1)+(compatible; SMTBot/1.0; +http://www.similartech.com/smtbot)
  Yaanb/1.5.001 (compatible; Win64;)
  Mozilla/5.0 (compatible; Crawlera/1.10.2; UID 70054)
  Mozilla/5.0 (compatible; spbot/2.0.2; +http://www.seoprofiler.com/bot/ )
  yacybot (freeworld-global; amd64 Linux 3.2.0-4-amd64; java 1.6.0_24; Europe/en) http://yacy.net/bot.html
  Mozilla/5.0 (compatible; ImageFetcher/5.6; +http://images.weserv.nl/)
  CopperEgg/RevealUptime/N.VirginiaUSA
  Huaweisymantecspider (compatible; MSIE 8.0; [email protected])
  OrangeCrawler/Nutch-1.0 ([email protected])
  CopperEgg/RevealUptime/AmsterdamHolland
  Mozilla/5.0 (compatible; SemrushBot-SI/0.97; +http://www.semrush.com/bot.html)
  Mozilla/5.0 (X11; Linux x86_64; GTmetrix https://gtmetrix.com/) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/50.0.2661.75 Safari/537.36
  Mozilla/5.0 (compatible; Genieo/1.0 http://www.genieo.com/webfilter.html) AppEngine-Google; (+http://code.google.com/appengine; appid: s~natmacdevice)
  yacybot (/global; amd64 Linux 3.2.41-042stab094.7; java 1.7.0_75; Europe/en) http://yacy.net/bot.html
  yacybot (x86 Windows Vista 6.1; java 1.6.0_13; Europe/de) http://yacy.net/bot.html
  SmabblerBot/1.0 (http://www.smabbler.com/)
  Mozilla/5.0 (compatible; grapeFX/0.9; [email protected]
  Mozilla/5.0 (compatible; SemrushBot/1~bl; +http://www.semrush.com/bot.html)
  MetaGeneratorCrawler/1.3.14 (www.metagenerator.info)
  oku-taka-lab-bot (Mozilla-compatible; [email protected]; http://lr-www.pi.titech.ac.jp/wp/)
  Mozilla/5.0 (Linux; Android 4.1.2; Galaxy Nexus Build/JZO54K; GTmetrix https://gtmetrix.com/) AppleWebKit/537.22 (KHTML, like Gecko) Chrome/26.0.1410.58 Mobile Safari/537.22
  Mozilla/5.0 (compatible; Feedage/2.0; +http://www.feedage.com/bot.php)
  yacybot (amd64 Linux 2.6.26-2-openvz-amd64; java 1.6.0_12; UTC/en) http://yacy.net/bot.html
  yacybot (-global; amd64 Linux 2.6.32-042stab116.1; java 1.7.0_79; Europe/en) http://yacy.net/bot.html
  Mozilla/5.0 (compatible; DKIMRepBot/1.0; +http://www.dkim-reputation.org)
  Mozilla/5.0 (Windows; U; Windows NT 5.1; zh-CN; rv:1.8.0.11) Firefox/1.5.0.11; 360Spider
  Mozilla/5.0 (compatible; pub-crawler; +http://wiki.github.com/bixo/bixo/bixocrawler; [email protected])
  mfibot/1.1 (http://www.mfisoft.ru/analyst/; <[email protected]>; en-RU)
  Visited by http://tools.geek-tools.org
  peerindex
  Mozilla/5.0 (compatible; janforman.com/2.4; +http://janforman.com/)
  http://www.amagit.com/
  yacybot (freeworld/global; amd64 Linux 2.6.32-40-server; java 1.6.0_20; Europe/en) http://yacy.net/bot.html
  Mozilla/5.0 (iPhone; CPU iPhone OS 8_3 like Mac OS X) AppleWebKit/537.36 (KHTML, like Gecko) Version/8.0 Mobile/12F70 Safari/600.1.4 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
  Mozilla/5.0 (compatible; NLNZ_IAHarvester2014 +https://natlib.govt.nz/publishers-and-authors/web-harvesting/domain-harvest)
  Mozilla/5.0 (compatible; YandexImageResizer/2.0; +http://yandex.com/bots)
  mahonie, neofonie search:robot/search:robot/0.0.1 (This is the MIA Bot - crawling for mia research project. If you feel unhappy and do not want to be visited by our crawler send an email to [email protected]; http://spider.neofonie.de; [email protected])
  Mozilla/5.0 (Linux; Android 5.0; Nexus 5 Build/LRX21O) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/46.0.2490.76 Mobile Safari/537.36 PTST/276
  Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; Trident/5.0; WOW64; Trident/5.0; BingPreview/1.0b)
  Mozilla/5.0 (compatible; OptimizationCrawler/0.2; +http://www.domainoptima.com/bot.html)
  Mozilla/5.0 (compatible; Cloudinary/1.0)
  rogerbot/1.0 (http://moz.com/help/pro/what-is-rogerbot-, [email protected])
  Iframely/0.9.8 (+http://iframely.com/;)
  yacybot (webportal-global; amd64 Linux 2.6.32-5-amd64; java 1.6.0_18; Europe/en) http://yacy.net/bot.html
  Interdose AntiSpamBot/2.0 (+http://www.piep.net)
  CheckMarkNetwork/1.0 (+http://www.checkmarknetwork.com/spider.html)
  Mozilla/5.0 (iPhone; CPU iPhone OS 8_3 like Mac OS X) AppleWebKit/537.36 (KHTML, like Gecko) Version/8.0 Mobile/12F70 Safari/600.1.4 (compatible; Google Search Console)
  rogerbot/1.0 (http://www.seomoz.org/dp/rogerbot, [email protected])
  findlinks/2.5 (+http://wortschatz.uni-leipzig.de/findlinks/)
  Mozilla/5.0 (compatible; MSIE 10.0; Windows NT 6.1; WOW64; Trident/6.0) PTST/277
  yacybot (/global; x86 Windows 2003 5.2; java 1.7.0_67; Europe/de) http://yacy.net/bot.html
  NeutrinoAPI/2.0.3
  Mozilla/5.0 (compatible; spbot/5.0.3; +http://OpenLinkProfiler.org/bot )
  Mozilla/5.0 (compatible; MJ12bot/v1.4.4; http://www.majestic12.co.uk/bot.php?+)
  Site24x7 Agent
  Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.2; WOW64; Trident/6.0; .NET4.0E; .NET4.0C; .NET CLR 3.5.30729; .NET CLR 2.0.50727; .NET CLR 3.0.30729) CrawlerProcess (http://www.PowerMapper.com) /5.7.704.0
  Mozilla/5.0 (compatible; Falconsbot; +http://iws.seu.edu.cn/services/falcons/contact_us.jsp)
  Super Monitoring
  Seznam-Zbozi-robot/3.0
  LinkedInBot/1.0 (compatible; Mozilla/5.0; Jakarta Commons-HttpClient/4.3 +http://www.linkedin.com)
  Mozilla/5.0 (compatible; EveryoneSocialBot/1.0; [email protected] http://everyonesocial.com/)
  Mozilla/5.0 (compatible; IstellaBot/1.01.18 +http://www.tiscali.it/)
  Mozilla/5.0 (compatible; ToutiaoSpider/1.0; http://web.toutiao.com/media_cooperation/;)
  Mozilla/5.0 (compatible; BegunAdvertising/3.0; +http://begun.ru/begun/technology/indexer/)
  Netcraft SSL Server Survey - contact [email protected]
  Mozilla/5.0 (compatible; Qualidator.com SiteAnalyzer 1.0;)
  Mozilla/5.0 (compatible; OrangeBot/2.0; [email protected])
  Mozilla/5.0 (compatible; CloudFlare-AlwaysOnline/1.0; +http://www.cloudflare.com/always-online)
  seebot/1.0.0 (http://www.seegnify.com/bot)
  Slackbot-LinkExpanding 1.0 (+https://api.slack.com/robots)
  Eurobot/1.1 (http://eurobot.ayell.eu)
  Mozilla/5.0 (compatible; IXEbot; +http://medialab.di.unipi.it/IXEbot.html)
  Mozilla/5.0 (GetLinkInfo.com - http://www.getlinkinfo.com)
  Mozilla/5.0 (compatible; spbot/4.0.5; +http://www.seoprofiler.com/bot )
  yacybot (freeworld/global; amd64 Windows 7 6.1; java 1.6.0_31; America/en) http://yacy.net/bot.html
  Mozilla/5.0 (Windows NT 6.1; WOW64; rv:48.0) Gecko/20100101 Firefox/48.0 PTST/314
  Mozilla/5.0 (compatible; AhrefsBot/4.0; +http://ahrefs.com/robot/)
  UXCrawlerBot
  Pixray-Seeker/2.0 (http://www.pixray.com/pixraybot; [email protected])
  Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/50.0.2661.94 Safari/537.36 PTST/276
  FAST-WebCrawler/3.7 (atw-crawler at fast dot no; http://fast.no/support/crawler.asp)
  yacybot (freeworld/global; amd64 Windows 10 10.0; java 1.8.0_60; Europe/de) http://yacy.net/bot.html
  Technoratibot/7.0
  Mozilla/4.0 (compatible; MSIE 5.01; Windows NT 5.0; DomainDB-1.1; http://domaindb.com/crawler/)
  Abrave v6.0 (http://robot.abrave.com)
  Testomatobot/1.0 (Linux x86_64; +http://www.testomato.com/testomatobot) minicrawler/4.0.0~beta11
  DowntimeDetector/1.0 (+http://downforeveryoneorjustme.com)
  Comodo Spider 1.1
  Link Valet Online 1.1
  yacybot (freeworld/global; amd64 Windows XP 5.2; java 1.7.0_04; America/en) http://yacy.net/bot.html
  li_viewer ([email protected])
  Mozilla/5.0 (compatible; Mail.RU/2.0c)
  Mozilla/5.0 (compatible; aiHitBot/1.0; +http://www.aihit.com/)
  Mozilla/5.0 (compatible; Uptimebot/0.2.19; +http://www.uptime.com/uptimebot)
  Mozilla/5.0 (compatible; YandexBot/3.0; +http://yandex.com/bots)
  Mozilla/5.0 (iPhone; CPU iPhone OS 6_0 like Mac OS X) AppleWebKit/536.26 (KHTML, like Gecko) Version/6.0 Mobile/10A5376e Safari/8536.25 ( compatible; CloudServerMarketSpider/1.0; +http://cloudservermarket.com/spider.html )
  yacybot (i386 Linux 2.6.28-13-generic; java 1.6.0_13; Europe/en) http://yacy.net/bot.html
  linguatools-bot/Nutch-1.6 (searching for translated pages; http://www.linguatools.de/linguatoolsbot.html; peter dot kolb at linguatools dot org)
  Mozilla/5.0 (compatible; Lippershey/5.2; +https://www.lipperhey.com/en/about/)
  NextGenSearchBot 69 (for information visit http://www.zoominfo.com/business/nextgensearchbot.html)
  Mozilla/5.0 (compatible; YandexWebmaster/2.0; +http://yandex.com/bots)
  Mozilla/5.0 (compatible; Baiduspider/2.0; +http://www.baidu.com/search/spider.html)
  yacybot (freeworld-global; amd64 Linux 3.2.0-35-generic; java 1.7.0_09; Europe/en) http://yacy.net/bot.html
  peerindex/0.1 (http://www.peerindex.com/; crawler AT peerindex DOT com)
  iqdb/0.2 (+http://iqdb.harry.lu/)
  Mozilla/5.0 (compatible; LuminateBot/1.0; +http://www.luminate.com/bot/)
  Mozilla/5.0 (X11; Linux x86_64) adbeat.com/policy AppleWebKit/537.21 (KHTML, like Gecko) Firefox/34.0 Safari/537.21
  AntBot/1.0 (http://www.ant.com)
  Mozilla/5.0 (compatible; Thumbshots.ru; +http://thumbshots.ru/bot) Firefox/3
  Mozilla/5.0 (compatible; value/1.1; +http://www.aihit.com/)
  Digg Deeper/v1 (http://digg.com/about)
  Elmer, the Thinglink ImageBot (http://www.thinglink.com/help/ThinglinkImageBot)
  rogerbot/1.0 (http://moz.com/help/pro/what-is-rogerbot-, [email protected])
  Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.1 (KHTML, like Gecko) Chrome/21.0.1180.89 Safari/537.1; http://spyonweb.com [email protected]
  BabalooSpider/1.3 (BabalooSpider; http://www.babaloo.si; [email protected])
  Mozilla/5.0 (compatible; dotSemantic/1.0; +http://www.dotsemantic.org)
  TinEye/1.1 (http://tineye.com/crawler.html)
  Mozilla/5.0 (compatible; SemrushBot/0.98~bl; +http://www.semrush.com/bot.html)
  CorpusCrawler 2.0.9 (http://corpora.fi.muni.cz/crawler/)
  aboutthedomain
  Cliqzbot/0.1 (+http://cliqz.com/company/cliqzbot)
  Mozilla/5.0 (compatible; Uptimebot/0.2.43; +http://www.uptime.com/uptimebot)
  LinqiaMetadataDownloaderBot/1.0 ([email protected])
  yacybot (freeworld/global; amd64 Linux 3.2.0-32-generic; java 1.6.0_24; Europe/en) http://yacy.net/bot.html
  NeumobBot/0.2.16 (+http://www.neumob.com/bot)
  Mozilla/5.0 (compatible; Mp3Bot/0.4; +http://mp3realm.org/mp3bot/)
  Miniflux (https://miniflux.net)
  Mozilla/5.0 (iPhone; CPU iPhone OS 8_3 like Mac OS X) AppleWebKit/537.36 (KHTML, like Gecko; Google Page Speed Insights) Version/8.0 Mobile/12F70 Safari/600.1.4
  Digincore crawler bot. See https://www.digincore.com/crawler.html for rules and instructions.
  Mozilla/5.0 (iPhone; CPU iPhone OS 7_0 like Mac OS X) AppleWebKit/537.51.1 (KHTML, like Gecko) Version/7.0 Mobile/11A465 Safari/9537.53 (compatible; adidxbot/2.0; http://www.bing.com/bingbot.htm)
  NerdyBot
  Nutch/2.2.1 (page scorer; http://integralads.com/site-indexing-policy/)
  Mozilla/5.0 (compatible; 008/0.83; http://www.80legs.com/webcrawler.html;) Gecko/2008032620
  Mozilla/4.0 (compatible; LinkMarket-Bot)
  SeoCheckBot (Seo-Check, http://www.kfsw.de/bot.html)
  Mozilla/5.0 (compatible; Whynder Magnet; http://whynder.com/magnet)
  IdeelaborPlagiaat/1
  x28-job-bot; +http://x28.ch/bot.html
  Mozilla/5.0 (compatible; spbot/4.0.7; +https://www.seoprofiler.com/bot )
  Mozilla/5.0 (compatible; RelateIQ Crawler/0.4; +www.relateiq.com; [email protected])
  WebRankSpider/1.37 (+http://ulm191.server4you.de/crawler/)
  Mozilla/5.0 (iPhone; U; CPU iPhone OS 10_0 like Mac OS X; en-us) AppleWebKit/602.1.38 (KHTML, like Gecko) Version/10.0 Mobile/14A5297c Safari/602.1 (compatible; Mediapartners-Google/2.1; +http://www.google.com/bot.html)
  LinkWalker/3.0 (http://www.brandprotect.com)
  Mozilla/5.0 (compatible; discobot/1.0; +http://discoveryengine.com/discobot.html)
  ia_archiver-web.archive.org
  Mozilla/5.0 (compatible; Online Domain Tools - SEO Checker/1.0; +http://online-domain-tools.com)
  Mozilla/5.0 (compatible; AhrefsBot/3.0; +http://ahrefs.com/robot/)
  Mozilla/5.0 (Macintosh; Intel Mac OS X 10_6_8) AppleWebKit/537.13+ (KHTML, like Gecko) Version/5.1.7 Safari/534.57.2; Shoppimon Analyzer (http://www.shoppimon.com/)
  Mozilla/5.0 (compatible; NewShareCounts.com/1.0; +http://newsharecounts.com/crawler)
  Mozilla/5.0 (compatible; Mozilla; +http://wiki.github.com/bixo/bixo/bixocrawler; [email protected])
  Mozilla/5.0 (compatible; CukBot; Not a spammer; http://www.companiesintheuk.co.uk/bot.html)
  yacybot (freeworld/global; amd64 Linux 3.1.10-1.9-default; java 1.6.0_24; Europe/de) http://yacy.net/bot.html
  Nuhk/2.4 (+http://www.neti.ee/cgi-bin/abi/otsing.html)
  elefent/Elefent 1.1 (Friendly web walker.; http://www.elefent.eu/; [email protected])
  HubSpot Links Crawler 1.0 http://www.hubspot.com/
  Mozilla/5.0 (compatible; Siteluxbot/1.0; +https://sitelux.co.uk/)
  CopperEgg/RevealUptime/AmsterdamNL(softlayer)
  Mozilla/5.0 (compatible; spbot/4.0.7; +http://OpenLinkProfiler.org/bot )
  Mediumbot-MetaTagFetcher/0.1 (+https://medium.com/)
  Mozilla/2.0 (compatible; Ask Jeeves/Teoma; +http://about.ask.com/en/docs/about/webmasters.shtml)
  Mozilla/5.0 (compatible; 008/0.85; http://www.80legs.com/webcrawler.html) Gecko/2008032620
  yacybot (/global; amd64 Linux 4.4.0ct; java 1.7.0_91; Europe/de) http://yacy.net/bot.html
  Mozilla/5.0 (compatible; Scrubby/2.2; http://www.scrubtheweb.com/)
  yacybot (/global; amd64 Linux 3.2.0-4-amd64; java 1.8.0_05; Europe/en) http://yacy.net/bot.html
  Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.2.18) Gecko/20110622 HeartRails_Capture/1.0.4 (+http://capture.heartrails.com/) Namoroka/3.6.18
  Mozilla/5.0 (X11; U; Linux i686; de; rv:1.9.0.1; compatible; iCjobs Stellenangebote Jobs; http://www.icjobs.de) Gecko/20100401 iCjobs/3.2.3
  KOCMOHABT (+http://www.KOCMOHABT.tk/) AppEngine-Google; (+http://code.google.com/appengine; appid: s~web-explorer)
  yacybot (freeworld/global; x86 Windows 7 6.1; java 1.6.0_31; Europe/de) http://yacy.net/bot.html
  Mozilla/5.0 (compatible; MSIE 10.0; Windows NT 6.2; WOW64; Trident/6.0) CrawlerProcess (http://www.PowerMapper.com) /5.21.754.0
  Mozilla/5.0 (compatible; Linux x86_64; Mail.RU_Bot/Robots; +http://go.mail.ru/help/robots)
  Mozilla/5.0 (compatible; Googlebot/2.1; +http://import.io)
  ownCloud Server Crawler
  Mozilla/5.0 (compatible; Qwantify/2.1n; +https://www.qwant.com/)/*
  LoadImpactRload/3.1.5 (Load Impact; http://loadimpact.com);
  Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 4.0; Girafabot; girafabot at girafa dot com; http://www.girafa.com)
  OmniExplorer_Bot/6.65a (+http://www.omni-explorer.com) WorldIndexer
  Mozilla/5.0 (compatible; SMTBot/1.0; +http://www.similartech.com/smtbot)
  anonymous coward/1.0 (a Storm-based crawler; https://github.com/DigitalPebble/storm-crawler; [email protected])
  Mozilla/5.0 (compatible; Uptimebot/1.0; +http://www.uptime.com/uptimebot)
  Mozilla/5.0 (compatible; FoundSeoTool/1.0; +http://www.found.co.uk/seo-tool/)
  findlinks/2.1.5 (+http://wortschatz.uni-leipzig.de/findlinks/)
  yacybot (freeworld/global; amd64 Windows NT (unknown) 6.2; java 1.7.0_05; Africa/en) http://yacy.net/bot.html
  Mozilla/5.0 (compatible; memorybot/1.20.71 +http://archivethe.net/en/index.php/about/internet_memory1 on behalf of DNB)
  ownCloud News/5.3.6 (+https://owncloud.org/; 1 subscriber;)
  YottaBot Spider/Nutch-2.3
  Mozilla/5.0 (compatible; WeViKaBot/1.0; +http://www.wevika.de/)
  ThumbShots-Bot (+http://thumbshots.in/bot.html)
  Mnogosearch-3.1.21
  Favicon downloader (+https://favico.be/)
  Mozilla/5.0 (compatible; Page2RSS/0.7; +http://page2rss.com/)
  ExB Language Crawler 2.1.2 (+http://www.exb.de/crawler)
  PrivateSearch/0.1.0 (Polite Nutch Crawler; grierforensics.com)
  nodestackbot/0.1 ([email protected] http://nodestack.com/bot.html)
  iskanie (+http://www.iskanie.com)
  yacybot (/global; amd64 Windows 7 6.1; java 1.7.0_67; Europe/de) http://yacy.net/bot.html
  Mozilla/5.0 (compatible; Dataprovider Site Explorer; http://www.dataprovider.com/)
  Mozilla/5.0 (compatible; akula/12.0rc-2; +http://k311.fd.cvut.cz/)
  Mozilla/5.0 (compatible; bnf.fr_bot; +http://www.bnf.fr/fr/outils/a.dl_web_capture_robot.html)
  Mozilla/5.0 (iPhone; CPU iPhone OS 6_0 like Mac OS X) AppleWebKit/536.26 (KHTML, like Gecko) Version/6.0 Mobile/10A5376e Safari/8536.25 (compatible; Googlebot-Mobile/2.1; +http://www.google.com/bot.html)
  Mozilla/5.0 (compatible; MSIE 10.0; Windows NT 6.1; WOW64; Trident/6.0) PTST/284
  Mozilla/5.0 (compatible; Infohelfer/1.3.3; +http://www.infohelfer.de/crawler.php)
  www.deadlinkchecker.com XMLHTTP/1.0
  Mozilla/5.0 (compatible; TweetmemeBot/3.0; +http://tweetmeme.com/)
  Chrome/49.0.2623.87 (compatible; Wappalyzer; +https://github.com/AliasIO/Wappalyzer)
  Mozilla/5.0 (compatible; MSIE 10.0; Windows NT 6.2; WOW64; Trident/6.0) CrawlerProcess (http://www.PowerMapper.com) /5.11.740.0
  XING-contenttabreceiver/2.0
  Mozilla/5.0 (iPhone; CPU iPhone OS 6_0 like Mac OS X) AppleWebKit/536.26 (KHTML, like Gecko) Version/6.0 Mobile/10A5376e Safari/8536.25 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
  Mozilla/5.0 (compatible; MojeekBot/0.2; http://www.mojeek.com/bot.html#relaunch)
  Pingoscope
  Mozilla/5.0 (compatible; um-LN/1.0; mailto: [email protected])
  Mozilla/5.0 (compatible; Pagespeed/1.1 Fetcher; +http://www.pagespeed.de)
  HTTP-Header-Abfrage/1.0 (http://www.internalscripts.de/werkzeuge/http-header-abfrage.php)
  Mozilla/5.0 (compatible; 80bot/0.71; http://www.80legs.com/spider.html;) Gecko/2008032620
  LoadImpactRload/3.0.3 (Load Impact; http://loadimpact.com);
  Mozilla/5.0 (compatible; Peepowbot/1.0; +http://www.peepow.com/bot.php)
  yacybot (freeworld/global; amd64 Linux 2.6.32-5-amd64; java 1.6.0_26; Atlantic/en) http://yacy.net/bot.html
  WatchMouse/8.4.0.3 (http://watchmouse.com/ ; grath01.watchmouse.net)
  Mozilla/5.0 (compatible; +http://servernfo.com/)
  Mozilla/5.0 (compatible; special_archiver/3.1.1 +http://www.archive.org/details/archive.org_bot)
  CopperEgg/RevealUptime/NewarkNJUSA
  Mozilla/5.0 (compatible; archive.org_bot/3.3.0 +http://www.archive.org/details/archive.org_bot)
  Mozilla/5.0 (compatible; Plukkie/1.1; http://www.botje.com/plukkie.htm)
  Mozilla/5.0 (compatible; musobot/1.0; [email protected]; +http://www.muso.com)
  yacybot (freeworld/global; amd64 Linux 3.10.17-gentooxen; java 1.6.0_31; Europe/nb) http://yacy.net/bot.html
  ResponseCodeTest/1.1
  Mozilla/5.0 (compatible; MSIE 10.0; Windows NT 6.1; Trident/6.0) SiteCheck-sitecrawl by Siteimprove.com
  Acoon v4.9.5 (www.acoon.de)
  Mozilla/5.0 (Windows NT 6.1; WOW64) SkypeUriPreview Preview/0.5
  SiteGuardian/2.0 (Internet Monitoring)
  Mozilla/5.0 (compatible; DCPbot/1.4; +http://domains.checkparams.com/)
  BoardReader Favicon Fetcher /1.0 [email protected]
  baypup/colbert (Baypup; http://www.baypup.com/webmasters; [email protected])
  Mozilla/5.0 (compatible; MonTools.com)
  yacybot (freeworld/global; amd64 Linux 3.2.0-4-amd64; java 1.7.0_03; Etc/de) http://yacy.net/bot.html
  Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/534.51 (KHTML, like Gecko; Google Web Preview) Chrome/12.0.742 Safari/534.51
  linkapediabot (+http://www.linkapedia.com)
  Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.7.12; ips-agent) Gecko/20050922 Fedora/1.0.7-1.1.fc4 Firefox/1.0.7
  BCKLINKS 1.0
  CorpusCrawler 2.0.21 (http://corpora.fi.muni.cz/crawler/);Project:CzCorpus
  Mozilla/5.0 (compatible; Yeti/1.1; +http://help.naver.com/support/robots.html)
  Mozilla/5.0 (compatible; pingbot/2.0; +http://www.pingdom.com/)
  Mozilla/5.0 (compatible; memoryBot/1.20.275 +http://internetmemory.org/en/)
  Mozilla/5.0 (compatible; spbot/1.1; +http://www.seoprofiler.com/bot/ )
  yacybot (/global; amd64 Linux 3.13.0-88-generic; java 1.7.0_101; Europe/en) http://yacy.net/bot.html
  BlogSearch/2 +http://www.icerocket.com/
  Mozilla/5.0 (compatible; LoadTimeBot/0.81; +http://www.load-time.com/bot.html)
  Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9) Gecko/2008052906 Firefox/3.0 (BrowserSpyBot/1.0) (http://browserspy.dk/)
  Mozilla/5.0 (compatible; MojeekBot/0.6; +https://www.mojeek.com/bot.html)
  CopperEgg/RevealUptime/NewarkNJ(linode)
  Automattic Analytics Crawler/0.2; http://wordpress.com/crawler/
  larbin_2.6.3 [email protected]
  HatenaScreenshot
  CopperEgg/RevealUptime/LondonUK
  Mozilla/5.0 (compatible; MojeekBot/0.6; http://www.mojeek.com/bot.html)
  Mozilla/5.0 (compatible; evc-batch/2.0.20160902205649)
  yacybot (freeworld/global; amd64 Linux 3.2.0-23-generic; java 1.6.0_24; Europe/en) http://yacy.net/bot.html
  WordPress.com;
  Mozilla/5.0 (Windows NT 6.1; rv:6.0) Gecko/20110814 Firefox/6.0 Google (+https://developers.google.com/+/web/snippet/)
  Mozilla/5.0 (compatible; ScoutJet; http://www.scoutjet.com/)
  Mozilla/5.0 (compatible; Uptimebot/0.1.97; +http://www.uptime.com/uptimebot)
  WebTarantula.com Crawler
  WatchMouse/8.4.0.3 (http://watchmouse.com/ ; itmil01.watchmouse.net)
  LinqiaScrapeBot/1.0 ([email protected])
  Orbiter (+http://www.dailyorbit.com/bot.htm)
  Mozilla/5.0 (compatible; janforman.com/2.5; +http://janforman.com/)
  InternetSeer.com
  Mozilla/5.0 (compatible; aiHitBot/2.9; +http://www.aihitdata.com/about)
  Mozilla/5.0 (compatible; WebNL; +http://www.web.nl/webmasters/spider.html; [email protected])
  coccoc/1.0 (http://help.coccoc.vn/)
  Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/53.0.2785.116 Safari/537.36 PTST/314
  Mozilla/5.0 (compatible; Googlebot/2.1; startmebot/1.0; +https://start.me/bot)
  Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 5.2; Trident/4.0; .NET CLR 1.1.4322; .NET CLR 2.0.50727; .NET CLR 3.0.4506.2152; .NET CLR 3.5.30729; .NET4.0C; .NET4.0E; PTST 2.386)
  Mozilla/5.0 (compatible; PrivacyAwareBot/1.1; +http://www.privacyaware.org)
  Mozilla/5.0 (compatible; idmarch Automatic.beta/1.4; +http://www.idmarch.org/bot.html)
  Mozilla/5.0 (compatible; tldstat.com/Bot; +http://tldstat.com/bot.html)
  yacybot (/global; amd64 Linux 3.16.0-4-amd64; java 1.7.0_111; Europe/de) http://yacy.net/bot.html
  Mozilla/5.0 (compatible; Crawlera/1.10.2; UID 84846)
  Mozilla/5.0 (compatible; spbot/4.0a; +http://www.seoprofiler.com/bot )
  Mozilla/5.0 (Windows NT 6.1; Win64; x64; +http://www.komodia.com/newwiki/index.php/URL_server_crawler) KomodiaBot/1.0
  WebarooBot (Webaroo Bot; http://64.124.122.252/feedback.html)
  OOZBOT/0.20 ( -- ; http://www.setooz.com/oozbot.html ; agentname at setooz dot_com )
  Mozilla/5.0 (compatible; memoryBot/1.21.16 +http://internetmemory.org/en/)
  FAST-WebCrawler/3.8
  Googlebot-Image/1.0
  enlle punto com/Nutch-1.9
  OpenAcoon v4.10.5 (www.openacoon.de)
  yacybot (freeworld/global; amd64 Linux 3.0.0-21-generic; java 1.6.0_26; America/en) http://yacy.net/bot.html
  Microsearch.ru Bot 1.3;http://microsearch.ru/webmasters
  Mozilla/5.0 (compatible; Google-Site-Verification/1.0)
  Mozilla/5.0 (compatible; Qwantify/2.3w; +https://www.qwant.com/)/2.3w
  rogerbot/1.1 (http://moz.com/help/guides/search-overview/crawl-diagnostics#more-help, [email protected])
  HubSpot Connect 2.0 (http://dev.hubspot.com/)
  Robozilla/1.0
  Mozilla/5.0 (compatible; spbot/2.0.3; +http://www.seoprofiler.com/bot/ )
  Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/53.0.2785.116 Safari/537.36 PTST/314
  facebookexternalhit/1.1
  PercolateCrawler/3.1.30 ([email protected])
  Mozilla/5.0 (compatible; Nmap Scripting Engine; https://nmap.org/book/nse.html)
  Mozilla/5.0 (compatible; pdffillerbot/1.0; +http://www.pdffiller.com/en/support.htm)
  fastbot crawler beta 4.0 (+http://www.fastbot.de)
  SafeDNS search bot/Nutch-1.9 (https://www.safedns.com/searchbot; support [at] safedns [dot] com)
  Mozilla/5.0 (Nekstbot; http://nekst.ipipan.waw.pl/nekstbot/)
  yacybot (freeworld/global; amd64 Linux 2.6.32-5-amd64; java 1.6.0_18; US/en) http://yacy.net/bot.html
  yacybot (freeworld/global; amd64 Linux 3.0.0-12-generic; java 1.6.0_26; Europe/en) http://yacy.net/bot.html
  coccoc/1.0 ()
  larbin_2.6.3 ([email protected])
  yacybot (i386 Linux 2.6.26-2-686; java 1.6.0_0; Europe/de) http://yacy.net/bot.html
  Mozilla/5.0 (Windows NT 6.1; compatible; BDCbot/1.0; +http://ecommerce.bigdatacorp.com.br/faq.aspx) AppleWebKit/535.1 (KHTML, like Gecko) Chrome/13.0.782.107 Safari/535.1
  yacybot (amd64 Linux 2.6.31-22-server; java 1.6.0_18; Asia/en) http://yacy.net/bot.html
  yacybot (/global; amd64 Windows Server 2012 6.2; java 1.8.0_91; Europe/de) http://yacy.net/bot.html
  Mozilla/5.0 (compatible; Netseer crawler/2.0; +http://www.netseer.com/crawler.html; [email protected])
  Mozilla/5.0 (compatible; heritrix/3.1.1 +http://www.mixdata.com)
  Scrapy/0.25.1 (+http://scrapy.org)
  Mozilla/5.0 (compatible; Scopia Crawler 1.2; +http://www.scopia.co)
  cn_ab_viewer ([email protected])
  Mozilla/5.0 (Windows NT 6.3; Trident/7.0; rv 11.0) like Gecko (compatible; Zombiebot/2.1; +http://www.zombiedomain.net/robot/)
  Mozilla/5.0 (compatible; Uptimebot/0.1.38; +http://www.uptime.com/uptimebot)
  yacybot (freeworld/global; amd64 Windows 7 6.1; java 1.7.0_25; Europe/de) http://yacy.net/bot.html
  Python-urllib/2.7 (+http://sketch.juls.savba.sk/aranea_about)
  Mozilla/5.0 (compatible; spbot/4.0; +http://www.seoprofiler.com/bot )
  yacybot (-global; amd64 Windows 8 6.2; java 1.7.0_55; Europe/de) http://yacy.net/bot.html
  yacybot (/global; amd64 Linux 3.2.41-042stab094.7; java 1.7.0_79; Europe/en) http://yacy.net/bot.html
  Mozilla/5.0 (compatible; UASlinkChecker/1.0; +http://user-agent-string.info/UASlinkChecker)
  Y!J-BRO/YFSJ crawler (compatible; Mozilla 4.0; MSIE 5.5; http://help.yahoo.co.jp/help/jp/search/indexing/indexing-15.html; YahooFeedSeekerJp/2.0)
  yacybot (freeworld/global; amd64 Linux 3.10.15-1-MANJARO; java 1.7.0_40; Asia/en) http://yacy.net/bot.html
  Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.2; WOW64; Trident/6.0; .NET4.0E; .NET4.0C; .NET CLR 3.5.30729; .NET CLR 2.0.50727; .NET CLR 3.0.30729) CrawlerProcess (http://www.PowerMapper.com) /5.9.726.0
  Scrubby/2.1 (http://www.scrubtheweb.com/)
  Mozilla/5.0 (compatible; JobKeresoBot; +http://www.kozvetlen-allasok.hu/help.jsp; [email protected]; 5.0
  yacybot (/global; amd64 Windows 7 6.1; java 1.8.0_66; Europe/de) http://yacy.net/bot.html
  Mozilla/5.0 (X11; Linux x86_64 UXCrawlerBot http://zium.co)
  Slackbot-LinkExpanding (+https://api.slack.com/robots)
  yacybot (freeworld/global; amd64 Linux 3.12.15-gentoo; java 1.7.0_55; Europe/en) http://yacy.net/bot.html
  Mozilla/5.0 (compatible; LinkpadBot/1.07; +http://www.linkpad.ru)
  Mozilla/5.0 (compatible; ZB-1; +http://www.zeerch.com/bot.php)
  Mozilla/5.0 (compatible; WbSrch/1.1 +http://wbsrch.com)
  WeSEE:Search
  Mozilla/5.0 (compatible; DCPbot/1.1; +http://domains.checkparams.com/)
  yacybot (webportal-global; amd64 Linux 2.6.32-5-amd64; java 1.6.0_45; Europe/en) http://yacy.net/bot.html
  WordPress.com mShots; http://support.wordpress.com/contact/
  Mozilla/4.0 (compatible; MSIE 5.01; Windows NT 5.0; FeedFinder-2.0; http://bloggz.se/crawler)
  Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.6; en-US; rv:1.9.2) Gecko/20100115 Firefox/3.6 (FlipboardProxy/0.0.5; +http://flipboard.com/browserproxy)
  Mozilla/5.0 (compatible; coccocbot-image/1.0; +http://help.coccoc.com/searchengine)
  Mozilla/5.0 (X11; Linux x86_64; GTmetrix http://gtmetrix.com/) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/31.0.1650.63 Safari/537.36
  https_page_test ([email protected])
  Mozilla/5.0 (Windows; U; Windows NT 5.1; zh-CN; rv:1.8.0.11) Firefox/1.5.0.11 360Spider;
  yacybot (freeworld/global; amd64 Linux 3.2.0-2-amd64; java 1.6.0_24; Europe/en) http://yacy.net/bot.html
  Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_2) AppleWebKit/600.1.4 (KHTML, like Gecko) Safari/600.1.4 (compatible; Princetonbot/1.0; +http://http://tigress-web.princeton.edu/~fy/bot.html)
  Mozilla/5.0 (Windows; U; Windows NT 5.1; zh-CN; rv:1.8.0.11) Gecko/20070312 Firefox/1.5.0.11; 360Spider
  KeywordDensityRobot/0.8 (http://www.seocentro.com/tools/search-engines/keyword-density.html)
  Mozilla/5.0 (compatible; aiHitBot-BP/1.1; +http://www.aihit.com/)
  findlinks/1.1.6-beta4 (+http://wortschatz.uni-leipzig.de/findlinks/)
  Mozilla/5.0 (compatible; Butterfly/1.0; +http://labs.topsy.com/butterfly/) Gecko/2009032608 Firefox/3.0.8
  CopperEgg/RevealUptime/FremontCA(linode)
  Mozilla/5.0 (compatible; NTENTbot; +http://www.ntent.com/ntentbot)
  link_thumbnailer
  KeywordDensityRobot/0.9 (http://www.seocentro.com/tools/search-engines/keyword-density.html)
  Mozilla/5.0 (compatible;WI Job Roboter Spider Version 3;+http://www.webintegration.at)
  Mozilla/5.0 (compatible; Mail.RU/2.0)
  CopperEgg/RevealUptime/N.California(aws)
  Searchie Bot/1.0 (a Storm-based crawler; https://www.searchie.org; [email protected])
  ^Nail (http://CaretNail.com)
  MarketBrewBot/2.1 (+http://www.marketbrew.com/)
  Updownerbot (+http://www.updowner.com/bot)
  yacybot (-global; amd64 Windows 8.1 6.3; java 1.7.0_55; Europe/de) http://yacy.net/bot.html
  Yanga WorldSearch Bot v1.1/beta (http://www.yanga.co.uk/)
  Mozilla/5.0 (compatible; bixolabs/1.0; +http://bixolabs.com/crawler/general; [email protected])
  Mozilla/5.0 (compatible; Scopia Crawler 2.0; +http://www.scopia.co)
  asafaweb.com
  KrOWLer/0.0.1, matentzn at cs dot man dot ac dot uk
  Mozilla/5.0 (compatible; alexa site audit/1.0; +http://www.alexa.com/help/webmasters; [email protected])
  findlinks/1.0 (+http://wortschatz.uni-leipzig.de/findlinks/)
  SygolBot http://www.sygol.com
  voltron
  Mozilla/5.0 (compatible; uMBot-FC/1.0; mailto: [email protected])
  Comodo-Webinspector-Crawler 2.2.1, http://www.comodorobot.com
  Superarama.com-Tarama-Botu-v.0.1
  yacybot (freeworld/global; amd64 Linux 3.2.0-3-amd64; java 1.6.0_24; Europe/en) http://yacy.net/bot.html
  Mozilla/5.0 (compatible; SSLBot/1.0; http://www.sslstats.com/sslbot)
  PagesInventory.com (robot http://www.pagesinvenotry.com)
  Mozilla/5.0 (iPhone; U; CPU iPhone OS 4_3 like Mac OS X; en-US) adbeat.com/policy AppleWebKit/533.17.9 (KHTML, like Gecko) Version/5.0.2 Mobile/8F190 Safari/6533.18.5
  WordPress.com Reader
  Mozilla/5.0 (compatible; SEOkicks-Robot +http://www.seokicks.de/robot.html)
  Woko 3.0
  Gigabot/3.0 (http://www.gigablast.com/spider.html)
  ichiro/4.0 (http://help.goo.ne.jp/door/crawler.html)
  bl.uk_lddc_bot/3.1.1 (+http://www.bl.uk/aboutus/legaldeposit/websites/websites/faqswebmaster/index.html)
  Urlstat (http://bit.ly/1dJuuzs)
  yacybot (/global; amd64 Windows 8.1 6.3; java 1.7.0_55; America/ru) http://yacy.net/bot.html
  Bad Neighborhood Header Detector (http://www.bad-neighborhood.com/header_detector.php)
  Mozilla/5.0 (compatible; GeliyooBot/1.0beta; +http://www.geliyoo.com/)
  NalezenCzBot/1.0 (http://www.nalezen.cz/about-crawler)
  WeSEE_Bot:we_help_monitize_your_site (http://www.wesee.com/bot/)
  yacybot (freeworld/global; x86 Windows XP 5.1; java 1.7.0_55; Asia/en) http://yacy.net/bot.html
  studylib download bot, see http://studylib.es/public/bot for additional info
  SemrushBot/0.91
  Jabse.com Crawler v.1.0 www.jabse.com/crawler.php//imagecrawler
  Mozilla/4.0 (compatible; MSIE 5.0; Windows NT; Girafabot; girafabot at girafa dot com; http://www.girafa.com)
  BizwikiBot/0.3 (http://www.bizwiki.com/bizwikibot.htm)
  Mozilla/5.0 (compatible; BIXOCRAWLER; +http://wiki.github.com/bixo/bixo/bixocrawler; [email protected])
  PostPost/1.0 (+http://postpost.com/crawlers)
  Mozilla/5.0 (nomore404.com robot/1.1; +https://nomore404.com/)
  Riddler (http://riddler.io/about)
  Mozilla/5.0 (compatible; socketcrawler; http://nlp.fi.muni.cz/projects/biwec/)
  internetVista monitor (Mozilla compatible)
  2Bone_LinkChecker/1.0 libwww-perl/6.03
  MaxPointCrawler/Nutch-1.1 (maxpoint.crawler at maxpointinteractive dot com)
  DomainStatsBot/1.0 (http://domainstats.io/our-bot)
  KDDI-CA31 UP.Browser/6.2.0.7.3.129 (GUI) MMP/2.0 (compatible; ichiro/mobile goo;+http://search.goo.ne.jp/option/use/sub4/sub4-1/)
  Mergadobot/3.0.2 (+http://mergado.cz)
  DoCoMo/2.0 P900i(c100;TB;W24H11) (compatible; ichiro/mobile goo; +http://search.goo.ne.jp/option/use/sub4/sub4-1/)
  SalesIntelligent/v1.1
  SiteAnalyzerBot
  DoCoMo/2.0 P900i(c100;TB;W24H11) (compatible; ichiro/mobile goo; +http://help.goo.ne.jp/help/article/1142/)
  yacybot (/global; i386 Linux 3.16.3-gentoo; java 1.7.0_55; Australia/en) http://yacy.net/bot.html
  Mozilla/5.0 (compatible; Linux x86_64; Mail.RU_Bot/Img/2.0; +http://go.mail.ru/help/robots)
  Cliqz Bot (+http://www.cliqz.com)
  SalesIntelligent (+https://www.salesintelligent.com)
  Mozilla/5.0 (compatible; memoryBot/1.20.243 +http://internetmemory.org/en/)
  gimme60 (Gimme60 Store ID Bot; gimme60.com)
  Orbiter/1.0 (http://dailyorbit.com/)
  msnbot/1.1 (+http://search.msn.com/msnbot.htm)
  Mozilla/5.0 (compatible; memoryBot/1.20.241 +http://internetmemory.org/en/)
  Mozilla/5.0 (compatible; akula/k311; +http://k311.fd.cvut.cz/)
  Irokez.cz monitoring v1.2 - (http://www.irokez.cz, Irokez.cz, crawl)
  CCBot/2.0 (http://commoncrawl.org/faq/)
  Mozilla/5.0 (compatible; openstat.ru/Bot)
  Mozilla/5.0 (compatible; GigaBot/1.0; +http://www.gigablast.com/ )
  Readability/6a54d4 - http://readability.com/about/
  percbotspider <[email protected]>
  Mozilla/5.0 (compatible; spbot/1.0; +http://www.seoprofiler.com/bot/ )
  HatenaScreenshot/1.0 (checker)
  Mozilla/5.0 (compatible; AcoonBot/4.12.1; +http://www.acoon.de/robot.asp)
  CRAZYWEBCRAWLER 0.9.2, http://www.crazywebcrawler.com
  Mozilla/5.0 (compatible; Online Domain Tools - Online Sitemap Generator/1.1; +http://sitemap-generator.online-domain-tools.com)
  Mozilla/5.0 (compatible; Google-Structured-Data-Testing-Tool +https://search.google.com/structured-data/testing-tool)
  OnPageBot (compatible; Googlebot 2.1; +https://bot.onpage.org/)
  Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/48.0.2564.103 Safari/537.36 Yandex.Translate
  Sogou web spider/4.0
  Mozilla/5.0 (compatible; YandexMetrika/2.0; +http://yandex.com/bots DEV)
  LinkWalker/2.0
  SeoCheck (FischerNetzDesign Seo Checker, [email protected])
  yacybot (freeworld/global; amd64 Linux 3.2.13-1-ARCH; java 1.7.0_03-icedtea; Europe/fr) http://yacy.net/bot.html
  SiteCheck - http://downforeveryoneorjustme.com
  Robots_Tester_http_www.searchenginepromotionhelp.com
  CRAZYWEBCRAWLER 0.9.4, http://www.crazywebcrawler.com
  Mozilla/5.0 (compatible; heritrix/3.1.1 +http://places.tomtom.com/crawlerinfo)
  Mozilla/5.0 (Windows; U; Windows NT 5.0; en-US; rv:1.7.10) Gecko/20050716 Thunderbird/1.0.6 - WebCrawler http://cognitiveseo.com/bot.html
  Mozilla/5.0 (compatible; Cliqzbot/1.0 +http://cliqz.com/company/cliqzbot)
  yacybot (/global; amd64 Linux 4.2.0-gentoo-r1; java 1.8.0_45; Australia/en) http://yacy.net/bot.html
  DomainsDB.net MetaCrawler v.0.9.7c (http://domainsdb.net/)
  Mozilla/4.0 (compatible; MSIE 5.5; Windows NT 5.0; mowser; http://www.mowser.com)
  www.probethenet.com scanner
  Mozilla/5.0 (compatible; Crawlera/1.10.2; UID 6573)
  LCC (+http://corpora.informatik.uni-leipzig.de/crawler_faq.html)
  Imagga Downloader/v0.14.3
  SeopultContentAnalyzer/1.0
  Mozilla/5.0 (compatible; +http://www.hosterstats.com/newgtld-websurvey.php)
  Mozilla/5.0 (compatible; GoogleDocs; apps-spreadsheets; +http://docs.google.com)
  Mozilla/5.0 (compatible;acapbot/0.1)
  Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1;Alibaba.Security.Heimdall)
  HTTPClient/1.0 (2.8.0, ruby 2.0.0 (2014-05-08))
  Jetty/9.3.z-SNAPSHOT
  Jetty/9.3.z-SNAPSHOT, SMUrlExpander
  Jetty/9.3.z-SNAPSHOT, SMViewer
  Liferea/0.x.x (Linux; en_US.UTF-8; http://liferea.sf.net/)
  SynHttpClient-Built/5825
  asynchttp
  SMUrlExpander
  KickFire
  HAAARTLAND http client
  PEAR HTTPRequest class ( http://pear.php.net/ )
  Mozilla/4.0 (vBSEO; http://www.vbseo.com)
  HAARTLAND http client
  JS-Kit URL Resolver http://js-kit.com/
  http://www.checkprivacy.or.kr:6600/RS/PRIVACY_ENFAQ.jsp
  Kml-Google; (+http://code.google.com/apis/kml) gzip
  gvfs/1.20.3
  request.js
  Microsoft URL Control - 6.01.9782
  git/2.1.4
  IPS/1.0
  OxfordCloudService/1.0
  Mozilla/5.0 (compatible; informatique at mixdata dot com/0.6; +http://t.co/GSRLLKex24; informatique at mixdata dot com)
  Mozilla/5.0 (compatible; Feedwind/2.0; +http://feed.mikle.com/support/description/)
  Mozilla/5.0 (compatible; Feedwind/3.0; +http://feed.mikle.com/support/description/)
  Mozilla/5.0 (Macintosh; Intel Mac OS X 1068) AppleWebKit/534.50 (KHTML like Gecko) Version/5.1 Instapaper/4.0 (+http://www.instapaper.com/)
  Mozilla/5.0 (compatible; Keywords Research; http://www.check-domains.com/keyword-research/keywords-adwords.php)
  CAAM/0.1 (CAAM; caam dot crwlr at gmail dot com)
  i2kconnect/"1.7.0"
  resolver/0.1.12 (+http://github.com/andris9/resolver)
  DangDang
  Mozilla/5.0 (compatible; Tetrahedron/1.0; +https://compute.info/tet)
  PritTorrent/0.1
  FeedBunch/1.0 (+http://www.feedbunch.com)
  Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.21 (KHTML, like Gecko) www.HomepageScreenshot.de - Free ScreenShotService/1.0 Safari/537.21
  Mozilla/5.0 (compatible; Licorne Image Snapshot 1.0; +http://www.jednorozec.cz)
  Pinterest/0.2 (+https://www.pinterest.com/bot.html)
  Pinterest/0.1 (+https://www.pinterest.com/bot.html)
  Mozilla/5.0 (iplabel; Windows NT 6.3; WOW64; rv:33.0) Gecko/20100101 Firefox/33.0
  Mozilla/5.0 zgrab/0.x
  Zend_Http_Client
  Zend\Http\Client
  Zao/0.1 (http://www.kototoi.org/zao/)
  YOURLS v1.7.1 +http://yourls.org/ (running on http://flnx.co)
  yoogliFetchAgent/0.1
  Yo-yo
  Mozilla/5.0 (compatible; Yahoo Link Preview; https://help.yahoo.com/kb/mail/yahoo-link-preview-SLN23615.html)
  Mozilla/5.0 (compatible; Yahoo Ad monitoring; https://help.yahoo.com/kb/yahoo-ad-monitoring-SLN24857.html)
  Mozilla/5.0 (iPhone; CPU iPhone OS 7_1 like Mac OS X) AppleWebKit (KHTML, like Gecko) Mobile (compatible; Yahoo Ad monitoring; https://help.yahoo.com/kb/yahoo-ad-monitoring-SLN24857.html)
  xpymep.exe
  xpymep2.exe
  xpymep3.exe
  Xenu Link Sleuth 1.2b
  Xenu Link Sleuth 1.2c
  Xenu Link Sleuth 1.2d
  Xenu Link Sleuth 1.2e
  Xenu Link Sleuth 1.2f
  Xenu Link Sleuth 1.2g
  Xenu Link Sleuth 1.2h
  Xenu Link Sleuth 1.2i
  Xenu Link Sleuth/1.3.7
  Xenu Link Sleuth/1.3.8
  Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/40.0.2214.93 Safari/537.36 XaxisSemanticsClassifier/1.0 http://crystalsemantics.com
  WWW-Mechanize/1.73
  WordPress/4.0.1; http://bestcomp.hol.es
  WordPress/4.3.3; http://homebusinessreviews.online
  WordPress/4.3.3; http://legithomejobs.club
  WordPress/4.4.2; http://cheapautoinsuranceinva.com
  WordPress/4.4.2; http://citycollegeinmiami.com
  WordPress/4.4.2; http://cookingideabook.com
  WordPress/4.4.2; http://oldmanhealth.com
  WordPress/4.4.2; http://sapcrmarchitecture.com
  WordPress/4.4.2; http://www.spar3d.com
  WordPress/4.5.2; http://1.we-careu.xyz
  WordPress/x.x.x.x PHP/4.x.xxMozilla/5.0 (Unknown; Linux x86_64) AppleWebKit/534.34 (KHTML, like Gecko) PhantomJS/1.6.1 Safari/534.34
  Mozilla/5.0 (Macintosh; Intel Mac OS X) Word/14.0.0
  Mozilla/5.0 (Macintosh; Intel Mac OS X) Word/14.25.0
  Mozilla/5.0 (Macintosh; Intel Mac OS X) Word/14.47.0
  Mozilla/5.0 (Macintosh; Intel Mac OS X) Word/14.48.0
  Mozilla/5.0 (Macintosh; Intel Mac OS X) Word/14.50.0
  Mozilla/5.0 (Macintosh; Intel Mac OS X) Word/14.56.0
  Mozilla/5.0 (Macintosh; Intel Mac OS X) Word/14.61.0
  Mozilla/5.0 (Macintosh; Intel Mac OS X) Word/14.62.0
  Mozilla/5.0 (Macintosh; Intel Mac OS X) Word/14.63.0
  Mozilla/5.0 (Macintosh; Intel Mac OS X) Word/14.64.0
  WomlpeFactory/0.1 (+http://www.Womple.com/bot.html)
  Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/534.34 (KHTML, like Gecko) wkhtmltoimage Safari/534.34
  Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/534.34 (KHTML, like Gecko) wkhtmltopdf Safari/534.34
  Mozilla/4.0 (compatible; Win32; WinHttp.WinHttpRequest.5)
  Windows-RSS-Platform/2.0 (IE 11.0; Windows NT 10.0)
  WhatsApp
  WhatsApp/2.12.15/i
  WhatsApp/2.12.16/i
  WhatsApp/2.12.17/i
  WhatsApp/2.12.449 A
  WhatsApp/2.12.453 A
  WhatsApp/2.12.510 A
  WhatsApp/2.12.540 A
  WhatsApp/2.12.548 A
  WhatsApp/2.12.555 A
  WhatsApp/2.12.556 A
  WhatsApp/2.16.1/i
  WhatsApp/2.16.13 A
  WhatsApp/2.16.2/i
  WhatsApp/2.16.42 A
  WhatsApp/2.16.57 A
  WhatWeb/0.4.8-dev
  http://www.almaden.ibm.com/cs/crawler [wf84]
  WeLikeLinks - WeAreNotEvil
  Websquash.com (Add url robot)
  WeCrawlForThePeace - We are not Evil
  Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.21 (KHTML, like Gecko) webscreenie Safari/537.21
  Mozilla/3.0 (compatible; WebMon 1.0.11; Windows XP)
  Mozilla/5.0 (X11; BSD Four) AppleWebKit/534.34 (KHTML, like Gecko) webkit2png Safari/534.34
  WebFetch
  WebIndex
  webcollage/1.114
  webcollage/1.117
  webcollage/1.125
  webcollage/1.129
  webcollage/1.93
  Mozilla/3.0 (compatible; WebCapture 2.0; Auto; Windows)
  W3C_Unicorn/1.0
  W3C_I18n-Checker/1.0
  W3C-mobileOK/DDC-1.0
  W3C-checklink
  W3C-checklink/2.90 libwww-perl/5.64
  W3C-checklink/3.6.2.3 libwww-perl/5.64
  W3C-checklink/4.2 [4.20] libwww-perl/5.803
  W3C-checklink/4.2.1 [4.21] libwww-perl/5.803
  W3C-checklink/4.3 [4.42] libwww-perl/5.805
  W3C-checklink/4.3 [4.42] libwww-perl/5.808
  W3C-checklink/4.3 [4.42] libwww-perl/5.820
  W3C-checklink/4.5 [4.154] libwww-perl/5.823
  W3C-checklink/4.5 [4.160] libwww-perl/5.823
  w3af.org
  VYU2 (GNU; OpenRISC)
  VORTEX/1.2 ( http://marty.anstey.ca/robots/vortex/)
  Vortex/2.2 ( http://marty.anstey.ca/robots/vortex/)
  Vortex/2.2 (+http://marty.anstey.ca/robots/vortex/)
  Mozilla/5.0 (Windows NT 5.1; rv:11.0) Gecko Firefox/11.0 (via ggpht.com GoogleImageProxy)
  URLChecker
  urlresolver
  Upflow/1.0
  Typhoeus - https://github.com/typhoeus/typhoeus
  UdmSearch/3.1.x
  Mozilla/5.0 Tweetminster
  Mozilla/5.0 (compatible; +http://tweetedtimes.com)
  truwoGPS/1.0 (GNU/Linux; U; i686; en-US; +http://www.lan4lano.net/browser.html )
  Traackr.com
  More-Testing (+http://www.theinternetrules.com)
  The Expert HTML Source Viewer (http://www.expert-html.com)
  htmlayout 3.3; above-Windows-7; www.terrainformatica.com )
  WebFuck V2.1 T0PHackTeam www.t0p.xyz
  Symfony2 BrowserKit
  Mozilla/5.0 (compatible; SwiteScraper)
  Summify (Summify/1.0.1; +http://summify.com)
  Stratagems Kumo
  Sqworm/2.9.85-BETA (beta_release; 20011115-775; i686-pc-linux-gnu)
  Sprinklr 2.0
  spray-can/1.3.3
  SortSite/4.4.500.0
  Snoopy v2.0.0
  SMRF URL Expander
  silk/1.0 (+http://www.slider.com/silk.htm)/3.7
  SiteBar/3.3.5 (Bookmark Server; http://sitebar.org/)
  SiteBar/3.3.8 (Bookmark Server; http://sitebar.org/)
  SimplePie/1.3.1 (Feed Parser; http://simplepie.org; Allow like Gecko) Build/20160226060242
  ShortLinkTranslate
  Mozilla/5.0 (compatible; Seznam screenshot-generator 2.1; +http://fulltext.sblog.cz/screenshot/)
  SEOstats 2.0.9 https://github.com/eyecatchup/SEOstats
  semanticdiscovery/0.1
  SearchSight/2.0 (http://SearchSight.com/)
  ScoutURLMonitor/5.9.8
  Robosourcer/1.0
  Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; RetrevoPageAnalyzer; +http://www.retrevo.com/content/about-us)
  RestSharp/105.2.3.0
  Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_5) AppleWebKit/537.78.2 (KHTML, like Gecko) Version/7.0.1(2.5.3) Safari(ReederForMac)
  redback/v0-570-g26f8c96
  RebelMouse/0.1 Mozilla/5.0 (compatible; http://rebelmouse.com) Gecko/20100101 Firefox/7.0.1
  RealPlayer%20Downloader/1569 CFNetwork/720.3.13 Darwin/14.3.0 (x86_64)
  RealPlayer%20Downloader/1701 CFNetwork/760.5.1 Darwin/15.5.0 (x86_64)
  RealPlayer%20Downloader/1725 CFNetwork/720.3.13 Darwin/14.3.0 (x86_64)
  RealPlayer%20Downloader/1750 CFNetwork/720.5.7 Darwin/14.5.0 (x86_64)
  RealPlayer%20Downloader/1750 CFNetwork/760.0.5 Darwin/15.0.0 (x86_64)
  RealPlayer%20Downloader/1750 CFNetwork/760.4.2 Darwin/15.4.0 (x86_64)
  Qirina Hurdler v. 1.05 10.11.01 (+http://www.qirina.com/hurdler.html)
  python-requests/1.1.0 CPython/2.7.6 Linux/3.13.0-58-generic
  python-requests/1.2.0 CPython/2.7.3 Linux/3.2.0-29-virtual
  python-requests/2.1.0 CPython/2.7.3 Linux/3.2.0-4-amd64
  python-requests/2.10.0
  python-requests/2.2.1 CPython/2.7.6 Linux/3.13.0-48-generic
  python-requests/2.2.1 CPython/2.7.6 Linux/3.13.0-83-generic
  python-requests/2.3.0 CPython/2.7.6 Linux/3.13.0-57-generic
  python-requests/2.5.0 CPython/2.7.9 Linux/3.14.20-20.44.amzn1.x86_64
  python-requests/2.6.0 CPython/2.7.3 Linux/3.2.0-4-amd64
  python-requests/2.6.0 CPython/2.7.5 Linux/3.10.0-327.3.1.el7.x86_64
  python-requests/2.7.0 CPython/2.7.6 Linux/3.13.0-85-generic
  python-requests/2.8.0
  python-requests/2.8.1
  python-requests/2.9.1
  Python-httplib2/0.7.4 (gzip)
  Pulsepoint XT3 web scraper
  Mozilla/5.0 (compatible; ProspectB2B/2.0; http://prospectb2b.com)
  Mozilla/5.0 (Macintosh; Intel Mac OS X) PowerPoint/14.39.0
  PostRank/2.0 (postrank.com; 1 subscribers)
  PostRank/2.0 (postrank.com)
  postano
  Ploetz + Zeller (http://www.ploetz-zeller.de) Link Validator v1.0 ([email protected]) for ARIS Business Architect
  Pizilla++ ver 2.45
  phpcrawl
  Pearltrees
  Pcore-HTTP/v0.23.20
  Mozilla/5.0 (Windows; U; Windows NT 6.0; en-US; rv:1.9.0.4) Gecko/2008102920 http://ow.ly web crawler (.NET CLR 3.5.30729)
  Mozilla/5.0 (compatible; Optimizer)
  Mozilla/5.0 [en] (X11, U; OpenVAS 7.0.10)
  JetBrains Omea Reader 1.0.x (http://www.jetbrains.com/omea_reader/)
  JetBrains Omea Reader 2.0 Release Candidate 1 (http://www.jetbrains.com/omea_reader/)
  okhttp/2.5.0
  oegp v. 1.3.0
  Nuzzel
  Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/43.0.2357.134 Safari/537.36 http://notifyninja.com/monitoring
  Notifixious/LinkChecker (http://notifixio.us)
  http://www.nominet.org.uk/privacypolicy
  node-superagent/1.8.3
  node.io
  Kimengi/nineconnections.com
  newspaper/0.0.9.8
  newspaper/0.1.7
  newsme/1.0; [email protected]
  NewsGator/2.0 Bot (http://www.newsgator.com)
  NewsGator/2.5 (http://www.newsgator.com; Microsoft Windows NT 5.1.2600.0; .NET CLR 1.1.4322.2032)
  NetTrack Anonymous Web Statistics https://nettrack.info/support.php
  NetTrack.info Crawler
  NetResearchServer(http://www.look.com)
  NetResearchServer/2.5(loopimprovements.com/robot.html)
  NetResearchServer/2.7(loopimprovements.com/robot.html)
  NetResearchServer/2.8(loopimprovements.com/robot.html)
  NetResearchServer/3.5(loopimprovements.com/robot.html)
  NetResearchServer/4.0(loopimprovements.com/robot.html)
  NetLyzer FastProbe (See http://netlyzer.com/report/www.domain.com for info)
  check_http/v2.0.3 (nagios-plugins 2.0.3)
  MVAClient
  Link Validity Check From: http://www.w3dir.com/cgi-bin (Using: Hot Links SQL by Mrcgiguy.com)
  Mojolicious (Perl)
  mogimogi/1.0
  Mindjet
  Mindjet MindManager
  Microsoft Windows Network Diagnostics
  Microsoft Office Excel 2013
  Microsoft Office Excel 2014
  Microsoft Office Excel 2014 (16.0.6528) Windows NT 10.0
  Microsoft Office Existence Discovery
  Microsoft Office Mobile/15.0
  Microsoft Office OneNote 2013
  Microsoft Office PowerPoint 2013
  Microsoft Office PowerPoint 2013 (15.0.4420) Windows NT 6.2
  Microsoft Office PowerPoint 2013 (15.0.4641) Windows NT 6.2
  Microsoft Office PowerPoint 2013 (15.0.4797) Windows NT 10.0
  Microsoft Office PowerPoint 2013 (15.0.4797) Windows NT 6.2
  Microsoft Office PowerPoint 2013 (15.0.4805) Windows NT 6.1
  Microsoft Office PowerPoint 2013 (15.0.4805) Windows NT 6.2
  Microsoft Office PowerPoint 2013 (15.0.4815) Windows NT 10.0
  Microsoft Office PowerPoint 2014
  Microsoft Office Word 2013
  Microsoft Office Word 2013 (15.0.4623) Windows NT 6.2
  Microsoft Office Word 2013 (15.0.4701) Windows NT 6.1
  Microsoft Office Word 2013 (15.0.4797) Windows NT 10.0
  Microsoft Office Word 2013 (15.0.4805) Windows NT 10.0
  Microsoft Office Word 2013 (15.0.4805) Windows NT 6.1
  Microsoft Office Word 2013 (15.0.4823) Windows NT 10.0
  Microsoft Office Word 2014
  Melvil Rawi/1.0
  marketinggrader.com
  Mandrill-Webhook/1.0
  MailChimp.com
  MagpieRSS/0.72 (+http://magpierss.sf.net)
  http://www.mabontland.com
  lwp-trivial/1.33
  lwp-trivial/1.35
  lwp-trivial/1.36
  lwp-trivial/1.38
  lwp-trivial/1.41
  looksystems.net
  LinkExaminer/1.01 (Windows)
  VLC/2.2.4 LibVLC/2.2.4
  [email protected]
  Mozilla/5.0 (compatible; Jobrapido/1.1; +http://www.jobrapido.com)
  JoBo/1.x (http://www.matuschek.net/jobo.html)
  Jigsaw/2.3.0 W3C_CSS_Validator_JFouffa/2.0
  Mozilla/5.0 (compatible; IODC-Internet Engineering Odysseus 001; +http://iodc.co.uk)
  Mozilla/5.0 (compatible; IODC-Internet Engineering Odysseus 005; +http://iodc.co.uk)
  Mozilla/5.0 (compatible; IODC-Internet Engineering Odysseus 006; +http://iodc.co.uk)
  Mozilla/5.0 (compatible; IODC-Internet Engineering Odysseus 007; +http://iodc.co.uk)
  Mozilla/5.0 (compatible; IODC-Internet Engineering Odysseus 011; +http://iodc.co.uk)
  Mozilla/5.0 (compatible; IODC-Internet Engineering Odysseus 013; +http://iodc.co.uk)
  Mozilla/5.0 (compatible; IODC-Odysseus Survey 29691-100-070315021100-72; +http://iodc.co.uk)
  Mozilla/5.0 (compatible; IODC-Odysseus Survey 29692-100-060415034231-192; +http://iodc.co.uk)
  Mozilla/5.0 (compatible; IODC-Odysseus Survey 31193-100-080615161452-372; +http://iodc.co.uk)
  Mozilla/5.0 (compatible; IODC-Odysseus Survey 37044-100-110516191742-025; +https://iodc.co.uk)
  Mozilla/5.0 (compatible; IODC-Odysseus Survey 41437-100-090715214314-122; +http://iodc.co.uk)
  Integrity/6
  InfoWizards Reciprocal Link System PRO - (http://www.infowizards.com)
  InDesign%20CC/11300 CFNetwork/720.4.4 Darwin/14.4.0 (x86_64)
  inbound.li parser
  InAGist URL Resolver (http://inagist.com)
  igdeSpyder (compatible; igde.ru; +http://igde.ru/doc/tech.html)
  IDG Twitter Links Resolver
  Mozilla/4.5 (compatible; HTTrack 3.0x; Windows 98)
  Jersey/2.19 (HttpUrlConnection 1.7.0_85)
  httpunit/1.x
  httpssites_power/1 CFNetwork/758.3.15 Darwin/15.4.0
  httpscheck (unknown version) CFNetwork/760.5.1 Darwin/15.5.0 (x86_64)
  HttpComponents/1.1
  http_requester/0.1
  HTTP_Request2/@package_version@ (http://pear.php.net/package/http_request2) PHP/5.5.9-1ubuntu4.14
  HTTP_Request2/2.2.0 (http://pear.php.net/package/http_request2) PHP/5.3.29
  HTTP_Request2/2.2.1 (http://pear.php.net/package/http_request2) PHP/5.3.29
  HTTP_Request2/2.2.1 (http://pear.php.net/package/http_request2) PHP/5.5.33
  HTTP_Request2/2.2.1 (http://pear.php.net/package/http_request2) PHP/5.5.35
  HTTP_Request2/2.3.0 (http://pear.php.net/package/http_request2) PHP/5.3.29
  HTTP-Tiny/0.054
  http-kit/2.0
  HTMLParser/2.0
  htdig/3.1.5
  htdig/3.1.5 ([email protected])
  htdig/3.1.5 (root@localhost)
  htdig/3.1.5 ([email protected])
  htdig/3.1.6 ([email protected])
  htdig/3.1.6 ([email protected])
  ht://check/2.0.0-rc1 (Linux 2.6.32-573.18.1.el6.x86_64 x86_64)
  HEADMasterSEO
  Mozilla/4.0 (compatible; grub-client-0.3.0; Crawl your own stuff with http://grub.org)
  Mozilla/4.0 (compatible; grub-client-1.0.4; Crawl your own stuff with http://grub.org)
  Mozilla/4.0 (compatible; grub-client-1.0.5; Crawl your own stuff with http://grub.org)
  Mozilla/4.0 (compatible; grub-client-1.0.6; Crawl your own stuff with http://grub.org)
  Mozilla/4.0 (compatible; grub-client-1.0.7; Crawl your own stuff with http://grub.org)
  Mozilla/4.0 (compatible; grub-client-1.1.1; Crawl your own stuff with http://grub.org)
  Mozilla/4.0 (compatible; grub-client-1.2.1; Crawl your own stuff with http://grub.org)
  Mozilla/4.0 (compatible; grub-client-1.3.1; Crawl your own stuff with http://grub.org)
  Mozilla/4.0 (compatible; grub-client-1.3.7; Crawl your own stuff with http://grub.org)
  Mozilla/4.0 (compatible; grub-client-1.4.3; Crawl your own stuff with http://grub.org)
  Mozilla/4.0 (compatible; grub-client-1.5.3; Crawl your own stuff with http://grub.org)
  Mozilla/4.0 (compatible; grub-client-2.3)
  Mozilla/4.0 (compatible; grub-client-2.6.0)
  Mozilla/4.0 (compatible; grub-client-2.6.1)
  GoSpotCheck 1.0.6 (iPhone; iPhone OS 5.1.1; en_US)
  GoScraper
  GoogleProducer; (+http://goo.gl/7y4SX)
  GoogleHC/
  Google-HTTP-Java-Client/1.17.0-rc (gzip)
  Google-HTTP-Java-Client/1.9.0-alpha
  Mozilla/5.0 (compatible; Google-Apps-Script)
  Go-http-client/1.1
  Go 1.1 package http
  Mozilla/5.0 (Macintosh; Intel Mac OS X 10_6_6) AppleWebKit/534.24 (KHTML, like Gecko) (Contact: [email protected])
  Mozilla/5.0 (compatible; Funnelback) RPT-HTTPClient/0.3-3E
  Mozilla/5.0 (compatible; Snacktory; +https://flynxapp.com)
  Mozilla/3.0 (compatible; Fluffy the spider; http://www.searchhippo.com/; [email protected])
  Flamingo_SearchEngine (+http://www.flamingosearch.com/bot)
  Mozilla/4.0 (compatible; MSIE 5.5; Windows NT 5.0) Fetch API Request
  FeedChecker-Zocle/1.0 (+https://zocle.com/zoclechecker)
  FeedBurner/1.0 (http://www.FeedBurner.com)
  FavOrg
  Faraday v0.9.1
  Mozilla/5.0 (Macintosh; Intel Mac OS X) Excel/14.10.0
  Mozilla/5.0 (Macintosh; Intel Mac OS X) Excel/14.62.0
  ExactSearch
  EventMachine HttpClient
  Mozilla/5.0 (compatible; MSIE 7.0 +http://www.europarchive.org)
  Embed PHP Library
  ECCP/1.2.1+([email protected])
  ec2linkfinder
  Mozilla/5.0 (compatible; EARTHCOM/2.2; +http://enter4u.eu)
  Drupal (+http://drupal.org/)
  Dragonfly File Reader
  DNS-Tools Header-Analyzer
  Dispatch/0.11.1-SNAPSHOT
  [email protected]
  CyberPatrol SiteCat Webbot (http://www.cyberpatrol.com/cyberpatrolcrawler.asp)
  curb
  Mozilla/4.0 (compatible; cron-job.org; http://cron-job.org/abuse/)
  cosmos/0.9_([email protected])
  Mozilla/5.0/Firefox/42.0 (contactbigdatafr at gmail.com)
  Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; .NET CLR 2.0.50727; MAXTHON 2.0); Connect Us: [email protected]
  Pattern/2.6 +http://www.clips.ua.ac.be/pattern
  Mozilla/4.0 (compatible; Cerberian Drtrs Version-3.2-Build-0)
  Mozilla/4.0 (compatible; Cerberian Drtrs Version-3.2-Build-1)
  CC Metadata Scaper http://wiki.creativecommons.org/Metadata_Scraper
  cb crawl (+http://www.companybooknetworking.com)
  CapsuleChecker (http://www.capsulink.com/)
  CakePHP
  Branch-Passthrough
  Branch Metrics API
  boitho.com-dc/0.79 ( http://www.boitho.com/dcbot.html )
  boitho.com-dc/0.81 ( http://www.boitho.com/dcbot.html )
  boitho.com-dc/0.82 ( http://www.boitho.com/dcbot.html )
  boitho.com-dc/0.83 ( http://www.boitho.com/dcbot.html )
  boitho.com-dc/0.85 ( http://www.boitho.com/dcbot.html )
  boitho.com-robot/1.1
  Blackboard Safeassign
  Mozilla/5.0 (Windows NT 6.3; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/42.0.2311.152 Safari/537.36 (BingLocalSearch)
  BIGLOTRON (Beta 2;GNU/Linux)
  Mozilla/5.0 (compatible; bnf.fr_bot; +http://bibnum.bnf.fr/robot/bnf.html)
  Mozilla/4.0 (compatible; B-l-i-t-z-B-O-T)
  Astute SRM
  aria2/1.18.8
  Mozilla/5.0 (compatible; archive-com.com/1.1; +http://archive-com.com/bot)
  Mozilla/4.0 (compatible; Arachmo)
  Apache-HttpClient/4.2.3 (java 1.5)
  Apache-HttpClient/4.2.5 (java 1.5)
  Apache-HttpClient/4.3.1 (java 1.5)
  Apache-HttpClient/4.3.3 (java 1.5)
  Apache-HttpClient/4.3.5 (java 1.5)
  Apache-HttpClient/4.5 (Java/1.8.0_51)
  Apache-HttpClient/4.5 (Java/1.8.0_72)
  Apache-HttpClient/4.5 (Java/1.8.0_74)
  Apache-HttpClient/4.5.1 (Java/1.7.0_79)
  Apache-HttpClient/4.5.1 (Java/1.8.0_40)
  Apache-HttpClient/4.5.2 (Java/1.8.0_72)
  Apache-HttpClient/UNAVAILABLE (java 1.4)
  Mozilla/5.0 (compatible; U; AnyEvent-HTTP/2.22; +http://software.schmorp.de/pkg/AnyEvent)
  Mozilla/5.0 (compatible; U; AnyEvent-HTTP/2.23; +http://software.schmorp.de/pkg/AnyEvent)
  Anemone/0.7.2
  AndroidDownloadManager/6.0.1 (Linux; U; Android 6.0.1; SM-G900F Build/MMB29M)
  AndroidDownloadManager/4.4.2 (Linux; U; Android 4.4.2; SM-T310 Build/KOT49H)
  AndroidDownloadManager/4.4.2 (Linux; U; Android 4.4.2; SM-T210 Build/KOT49H)
  Airmail 1.3.3 rv:237 (Macintosh; Mac OS X 10.9.3; sk_SK)
  Python/3.5 aiohttp/0.21.6
  AHC/1.0
  Mozilla/4.0 (compatible; Advanced Email Extractor v2.61)
  Pulp/1.5.2 (iPad; http://www.acrylicapps.com/pulp/)
  Accoona-AI-Agent/1.1.1 (crawler at accoona dot com)
  Accoona-AI-Agent/1.1.2
  Accoona-AI-Agent/1.1.2 (aicrawler at accoonabot dot com)
  Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1) [email protected]
  scrutiny/6
  RMA/1.0 (compatible; RealMedia)
  PHP/5.5
  NING/1.0
  Mget/0.1.9
  Java/1.4.1_04
  Java/1.6.0
  Java/1.6.0_04
  Java/1.6.0_22
  Java/1.6.0_24
  Java/1.6.0_26
  Java/1.6.0_31
  Java/1.6.0_32
  Java/1.6.0_34
  Java/1.6.0_37
  Java/1.7.0_101
  Java/1.7.0_45
  Java/1.7.0_51
  Java/1.7.0_65
  Java/1.7.0_80
  Java/1.8.0_05
  Java/1.8.0_20
  Java/1.8.0_31
  Java/1.8.0_40
  Java/1.8.0_45
  Java/1.8.0_51
  Java/1.8.0_60
  Java/1.8.0_65
  Java/1.8.0_66
  Java/1.8.0_73
  Java/1.8.0_77
  Java/1.8.0_91
  Java/1.8.0_92
  Goose/1.0.25
  FDM 3.x
  bluefish 0.6 HTML editor
  2ip.ru CMS Detector (http://2ip.ru/cms/)
  masscan/1.0 (https://github.com/robertdavidgraham/masscan)
  VidibleScraper/1.0
  WhereGoes? Redirect Tracer/1.0
  unfurlist (https://github.com/Doist/unfurlist)
  pshtt, https scanning
  gofetch
  COMODO DCV
  CATExplorador/1.0beta (sistemes at domini dot cat; http://domini.cat/catexplorador.html)
  ZmEu
  RSSOwl/2.2.1.201312301314 (Windows; U; en)
  WebClient/1.0
  Microsoft-WebDAV-MiniRedir/10.0.10586
  Microsoft-WebDAV-MiniRedir/10.0.14393
  Microsoft-WebDAV-MiniRedir/5.1.2600
  Microsoft-WebDAV-MiniRedir/6.0.6001
  Microsoft-WebDAV-MiniRedir/6.1.7601
  Microsoft-WebDAV-MiniRedir/6.3.9600
  Microsoft Data Access Internet Publishing Provider Protocol Discovery
  Test Certificate Info
  Needle/1.4.2 (Node.js v6.9.2; linux x64)
  DavClnt
  Mozilla/5.0 (iPhone; CPU iPhone OS 6_0 like Mac OS X) AppleWebKit/536.26 (KHTML, like Gecko) Version/6.0 Mobile/10A5376e Safari/8536.25 mixed-content-scan
  Mozilla/5.0 (compatible; Semiocast HTTP client; http://semiocast.com/)
  Mozilla/5.0 (compatible; The Drop Reaper; +http://tag1consulting.com/dropreaper.html)
  /n software IPWorks HTTP/S Component - www.nsoftware.com
  safe-agent-scanner
  Mozilla/5.0 (compatible; ACHE/Unknown Version; +https://github.com/ViDA-NYU/ache; )
  ANAM (github.com/dutchcoders/anam)
  Dolphin http client/11.2.8(385) (Android)
  Dolphin http client/11.3.4(405) (Android)
  Calypso v/0.01
  -Mozilla/4.0 (compatible; http://search.thunderstone.com/texis/websearch/about.html)
  Mozilla/5.0 (compatible; Tweezler/2.0; http://tweezler.com)
  Jeode/1.x.x
  KeyCDN Perf Test
  Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) YLT Chrome/27.0.1453.110 Safari/537.36
  Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/51.0.2704.84 Safari/537.36 DMBrowser-UV
  Taringa UGC www.taringa.net/shared-link.php
  Mozilla/5.0 (Unknown; Linux x86_64) AppleWebKit/534.34 (KHTML, like Gecko) PingdomTMS/0.8.5 Safari/534.34
  ganarvisitas/1.0 (+http://www.ganarvisitas.com/)
  FeedFetcher-Google; (+http://www.google.com/feedfetcher.html)
  Mozilla/5.0 (compatible) Feedfetcher-Google; (+http://www.google.com/feedfetcher.html)
  Mozilla/5.0 (compatible; Exabot PyExalead/3.0; +http://www.exabot.com/go/robot)
  Mozilla/5.0 PhantomJS (compatible; Seznam screenshot-generator 2.1; +http://fulltext.sblog.cz/screenshot/)
  Mozilla/5.0 (compatible; SeznamBot/3.2-test4; +http://napoveda.seznam.cz/en/seznambot-intro/)
  Mozilla/5.0 (compatible; SeznamBot/3.2-test2; +http://napoveda.seznam.cz/en/seznambot-intro/)
  Mozilla/5.0 (compatible; SeznamBot/3.2-test1; +http://fulltext.sblog.cz/)
  Mozilla/5.0 (compatible; SeznamBot/3.2-test4; +http://fulltext.sblog.cz/)
  Mozilla/5.0 (compatible; SeznamBot/3.2; +http://fulltext.sblog.cz/)
  Seznam-Zbozi-robot/3.2.1
  SeznamBot/2.0 (+http://fulltext.sblog.cz/robot/)
  Seznam-Zbozi-robot/3.2.2
  Mozilla/5.0 (compatible; SeznamBot/3.2-test2; +http://fulltext.sblog.cz/)
  Mozilla/5.0 (compatible; SeznamBot/3.2-test3; +http://fulltext.sblog.cz/)
  Seznam-Zbozi-robot/3.3
  SklikBot/2.0 ([email protected];+http://napoveda.sklik.cz/)
  SeznamBot/3.0 (+http://fulltext.sblog.cz/)
  Mozilla/5.0 (compatible; SeznamBot/3.1-test1; +http://fulltext.sblog.cz/)
  Mozilla/5.0 (Linux; U; Android 4.1.2; cs-cz; Seznam screenshot-generator Build/Q3) AppleWebKit/534.30 (KHTML, like Gecko) Version/4.0 Mobile Safari/534.30
  SeznamBot/3.0-test (+http://fulltext.sblog.cz/), I
  SeznamBot/3.0 (HaF+http://fulltext.sblog.cz/)
  SeznamBot/3.0-test (+http://fulltext.sblog.cz/)
  SeznamBot/3.0-beta (+http://fulltext.sblog.cz/), I
  SeznamBot/3.0-beta (+http://fulltext.sblog.cz/)
  SeznamBot/3.0-alpha (+http://fulltext.sblog.cz/)
  SeznamBot/2.0 (+http://fulltext.seznam.cz/)
  SeznamBot/2.0-Test (+http://fulltext.sblog.cz/robot/)
  Mozilla/5.0 (compatible; Seznam screenshot-generator 2.0; +http://fulltext.sblog.cz/screenshot/)
  Mozilla/5.0 (compatible; FlipboardRSS/1.2; +http://flipboard.com/browserproxy)
  Mozilla/5.0 (Macintosh; Intel Mac OS X 10.11; rv:49.0) Gecko/20100101 Firefox/49.0 (FlipboardProxy/1.2; +http://flipboard.com/browserproxy)
  Mozilla/5.0 (compatible; MJ12bot/v1.4.7; http://mj12bot.com/)
  Mozilla/5.0 (Macintosh; Intel Mac OS X 10_7_2) AppleWebKit/535.7 (KHTML, like Gecko) Chrome/16.0.912.77 Safari/535.7 AppEngine-Google; (+http://code.google.com/appengine; appid: s~feedly-social)
  Mozilla/5.0 (Macintosh; Intel Mac OS X 10_7_2) AppleWebKit/535.7 (KHTML, like Gecko) Chrome/23.0.912.77 Safari/535.7 AppEngine-Google; (+http://code.google.com/appengine; appid: s~feedly-nikon3)
  Mozilla/5.0 (compatible; SemrushBot/1.2~bl; +http://www.semrush.com/bot.html)
  Mozilla/5.0 (compatible; YandexMetrika/2.0; +http://yandex.com/bots mtmon01it.mtrs.yandex.ru)
  Mozilla/5.0 (iPhone; CPU iPhone OS 8_1 like Mac OS X) AppleWebKit/600.1.4 (KHTML, like Gecko) Version/8.0 Mobile/12B411 Safari/600.1.4 (compatible; YandexBot/3.0; +http://yandex.com/bots)
  Mozilla/5.0 (compatible; memoryBot/1.24.61 +http://internetmemory.org/en/)
  Mozilla/5.0 (compatible; memoryBot/1.24.54 +http://internetmemory.org/en/)
  BUbiNG - Research at Brno University of Technology - KNOT group - http://knot.fit.vutbr.cz/crawling/ - Stop? http://law.di.unimi.it/BUbiNG.html
  Mozilla/5.0 (compatible; MSIE 9.0; Firefox/50.1) Daum/4.1
  Mozilla/5.0 (Unknown; Linux x86_64) AppleWebKit/534.34 (KHTML, like Gecko) PhantomJS/1.9.7 Daum Raam Edition b20140820 Safari/534.34
  Sogou Pic Spider/3.0(+http://www.sogou.com/docs/help/webmasters.htm#07)
  yacybot (/global; amd64 Linux 4.4.0-59-generic; java 1.8.0_121; Europe/de) http://yacy.net/bot.html
  yacybot (-global; amd64 Linux 4.4.0-57-generic; java 9-internal; Europe/en) http://yacy.net/bot.html
  yacybot (/global; amd64 Windows 8.1 6.3; java 1.8.0_111; Europe/de) http://yacy.net/bot.html
  yacybot (/global; amd64 Linux 3.16.0-4-amd64; java 1.8.0_111; Europe/en) http://yacy.net/bot.html
  yacybot (/global; amd64 FreeBSD 10.3-RELEASE-p7; java 1.7.0_95; GMT/en) http://yacy.net/bot.html
  yacybot (webportal-global; amd64 Linux 4.4.0-46-generic; java 1.8.0_91; Europe/en) http://yacy.net/bot.html
  yacybot (-global; amd64 Linux 3.14.32-xxxx-grs-ipv6-64; java 1.8.0_111; Europe/de) http://yacy.net/bot.html
  Mozilla/5.0 (compatible; special_archiver/3.3.0 [email protected] +https://archive.org/details/archive.org_bot)
  Mozilla/5.0 (compatible; heritrix/3.3.0-SNAPSHOT-20140702-2247 +http://archive.org/details/archive.org_bot)
  Mozilla/5.0 (compatible; archive.org_bot/heritrix-1.15.4 +http://www.archive.org)
  SeobilityBot (SEO-Check; http://bit.ly/1dJuuzs)
  Testomatobot/1.0 (Linux x86_64; +http://www.testomato.com/testomatobot) minicrawler/5.0.4
  Testomatobot/1.0 (Linux x86_64; +http://www.testomato.com/testomatobot) minicrawler/5.0.3
  Testomatobot/1.0 (Linux x86_64; +http://www.testomato.com/testomatobot) minicrawler/5.0.2
  Testomatobot/1.0 (Linux x86_64; +http://www.testomato.com/testomatobot) minicrawler/5.0.1
  Mozilla/5.0 (compatible; AhrefsBot/5.2; +http://ahrefs.com/robot/)
  Heurekabot-Feed/1.0 (+https://sluzby.heureka.cz/napoveda/heurekabot/)
  Heurekabot-ImageFullText/1.0 (+https://sluzby.heureka.cz/napoveda/heurekabot/)
  Heurekabot-ImageFullText/1.0 (+http://sluzby.heureka.cz/napoveda/heurekabot/)
  Mozilla/5.0 (Windows NT 6.2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/31.0.1650.63 Safari/537.36 QIHU 360SE; 360Spider
  ZoomBot (Linkbot 1.0 http://suite.seozoom.it/bot.html)
  Mozilla/5.0 (compatible; SOLOFIELD/1.0 +http://solofield.net/bot.html)
  Qwantify/1.0
  Mozilla/5.0 (compatible; Yeti/1.1; +http://naver.me/bot)
  Scrapy/1.3.0 (+http://scrapy.org)
  Scrapy/1.0.5.post4+g4b324a8 (+http://scrapy.org)
  Scrapy/1.2.1 (+http://scrapy.org)
  Scrapy/0.16.5 (+http://scrapy.org)
  Mozilla/5.0 (compatible; Dataprovider.com;)
  Mozilla/5.0 (compatible; Dataprovider; https://www.dataprovider.com/spider/)
  Mozilla/5.0 (compatible; SiteExplorer/1.1b; +http://siteexplorer.info/Backlink-Checker-Spider/)
  ICC-Crawler/2.0 (Mozilla-compatible; ; http://ucri.nict.go.jp/en/icccrawler.html)
  DuckDuckBot/1.0; (+http://duckduckgo.com/duckduckbot.html)
  Mozilla/5.0 (compatible; SMTBot/1.0; http://www.similartech.com/smtbot)
  Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/538.1 (KHTML, like Gecko) Chromium/31.0.1650.63 Site-Shot/2.1 (http://www.site-shot.com/) Safari/538.1
  Mozilla/5.0 (Windows NT 6.1; WOW64; Neustar WPM) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/48.0.2564.82 Safari/537.36
  Mozilla/5.0 (iPad; CPU OS 6_1_2 like Mac OS X; Neustar WPM) AppleWebKit/536.26 (KHTML, like Gecko) Version/6.0 Mobile/10A5376e Safari/8536.25
  Mozilla/5.0 (Windows NT 6.0; Neustar WPM) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/48.0.2564.82 Safari/537.36
  Mozilla/5.0 (Windows NT 6.1; Neustar WPM) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/48.0.2564.82 Safari/537.36
  Mozilla/5.0 (Windows NT 6.0; rv:38.0; Neustar WPM) Gecko/20100101 Firefox/38.0
  Mozilla/5.0 (Windows NT 6.1; rv:38.0; Neustar WPM) Gecko/20100101 Firefox/38.0
  Mozilla/5.0 (Windows NT 6.1; Neustar WPM) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/52.0.2743.82 Safari/537.36
  Mozilla/5.0 (Linux; Android 4.1.1; HTC6435LVW Build/JRO03C; Neustar WPM) AppleWebKit/537.22 (KHTML, like Gecko) Chrome/25.0.1364.169 Mobile Safari/537.22
  Mozilla/5.0 (compatible; Whoiswebsitebot/0.1; +http://www.whoiswebsite.net)
  Mozilla/5.0 (Linux; Android 5.0.2; SM-G920T Build/LRX22G) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/54.0.2840 Mobile Safari/537.36 DareBoost
  Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/54.0.2840 Safari/537.36 DareBoost
  Mozilla/5.0 (Linux; Android 5.0; LG-D855 Build/LRX21R.A1422018487) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/54.0.2840 Mobile Safari/537.36 DareBoost
  Mozilla/5.0 (iPhone; CPU iPhone OS 10_0_1 like Mac OS X) AppleWebKit/602.1.50 (KHTML, like Gecko) Version/10.0 Mobile/14A403 Safari/602.1 DareBoost
  Mozilla/5.0 (Linux; Android 4.4.3; HTC One Build/KTU84L) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/54.0.2840 Mobile Safari/537.36 DareBoost
  Mozilla/5.0 (Linux; Android 4.4; Nexus 5 Build/KRT16M) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/54.0.2840 Mobile Safari/537.36 DareBoost
  HubSpot Website Grader ([email protected])
  HubSpot Marketing Grader, HubSpot Marketing Grader
  com.hubspot.spooks.core.UserAgent@753535c0, HubSpot Marketing Grader
  Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/51.0.2704.79 Safari/537.36 (http://www.shrinktheweb.com)
  scrapy-redis (+https://github.com/rolando/scrapy-redis)
  Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/55.0.2883.87 Safari/537.36 PTST/348
  Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/54.0.2840.99 Safari/537.36 PTST/345
  TeamSpeak3-ImageFetcher-1.0
  wf_crawler (http://www.websitefigures.com)
  wf_crawler (http://websitefigures.com)
  Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/55.0.2883.87 Safari/537.36 TinEye/1.0 (via http://www.tineye.com/)
  TinEye-bot/0.61 (see http://www.tineye.com/crawler.html)
  Mozilla/5.0 (compatible; Googlebot/2.1; https://deepcrawl.com/bot)
  Googlebot deepcrawl
  Mozilla/5.0 (compatible; evc-batch/2.0.20170110125844)
  Mozilla/5.0 (compatible; evc-batch/2.0.20161220184139)
  Mozilla/5.0 (compatible; evc-batch/2.0.20161122223033)
  Mozilla/5.0 (compatible; evc-batch/2.0.20161026010401)
  Mozilla/5.0 (X11; Linux x86_64; rv:49.0; GTmetrix https://gtmetrix.com/) Gecko/20100101 Firefox/49.0
  Mozilla/5.0 (Linux; Android 4.3; Galaxy Nexus Build/JWR66Y; GTmetrix https://gtmetrix.com/) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/54.0.2840.68 Mobile Safari/537.36
  Mozilla/5.0 (X11; Linux x86_64; GTmetrix https://gtmetrix.com/) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/54.0.2840.71 Safari/537.36
  Mediumbot-MetaTagFetcher/0.3 (+https://medium.com/)
  Mediumbot-ProxyFetcher/0.1 (+https://medium.com/)
  YOOBI.de - WebBot 1.0
  YOOBI - WebBot 1.0
  Mozilla/5.0 (compatible; heritrix/3.3.0-SNAPSHOT-20160309-0050; UniLeipzigASV +http://corpora.informatik.uni-leipzig.de/crawler_faq.html)
  IAS crawler (page scorer; http://integralads.com/site-indexing-policy/)
  Mozilla/5.0 (compatible; JobKeresoBot; +https://www.kozvetlen-allasok.hu/help.jsp; [email protected]; 5.0
  Mozilla/5.0 (compatible; JobKereso; +https://www.kozvetlen-allasok.hu/robot.jsp [email protected])
  pyspider/0.3.9 (+http://pyspider.org/)
  Mozilla/5.0 (X11; U; Linux x86_64; de-DE; rv:1.9.0.19) Gecko/2010120923 ThumbShotsBot (KFSW 3.0.6-3)
  Mozilla/5.0 (compatible; Discordbot/1.0; +https://discordapp.com)
  Mozilla/5.0 (compatible; Dead Link Checker; http://www.dead-link-checker.com/)
  Mozilla/5.0 (compatible; MSIE 10.0; Windows NT 6.2; WOW64; Trident/6.0) CrawlerProcess (http://www.PowerMapper.com) /5.24.776.0
  Mozilla/4.0 (compatible; BOTW Spider; +http://botw.org)
  TwengaBot-2.0
  Iframely/1.0.0 (+https://iframely.com/;)
  Iframely/0.8.5 (+http://iframely.com/;)
  rogerbot/1.2 (http://moz.com/help/pro/what-is-rogerbot-, [email protected])
  rogerbot/1.0 (http://moz.com/help/pro/what-is-rogerbot-, [email protected]
  Mozilla/5.0 (compatible; Sitemap Generator/1.3; http://www.check-domains.com/sitemap/index.php) Gecko Check-domains)
  mindUpBot
  Mozilla/5.0 (Windows NT 6.1; compatible; BDCbot/1.0; +http://bigweb.bigdatacorp.com.br/faq.aspx) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2272.118 Safari/537.36
  Contacts Crawler (+http://www.scrapinghub.com)
  TA SEO Crawler v0.1 URLData.pm ([email protected])
  Mozilla/5.0 (compatible; scanbot/1.0; +http://dazzlepod.com/ip/)
  ArielisBot/1.1 (+http://arielis.com/
  RedesScrapy/0.24.1 (+http://g2pi.tsc.uc3m.es/es)
  Blackboard Safeassign/0.1 (a Storm-based Blackboard Safeassign web-crawler; https://github.com/DigitalPebble/storm-crawler; [email protected])
  Mozilla/5.0 (Macintosh; Intel Mac OS X 10.11; rv:48.0) Gecko/20100101 Firefox/48.0 glindahl-cocrawler/0.01 (+http://www.pbm.com/~lindahl/glindahl-cocrawler.html)
  botrobin/Nutch-1.13-SNAPSHOT (http://smarter.codes/bot-robin/; [email protected])
  Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/51.0.2704.63 Safari/537.36 (compatible; aylienbot/0.2; +http://www.aylien.com/bot.html)
  GloomarBot/1.1 (https://www.gloomar.com/bot)
  Hatena::Bookmark/2.00 (Hatena::Bookmark; master;)
  Opendi Screenshot Bot [email protected]
  Mozilla/5.0 (compatible; YaK/1.0; http://linkfluence.com/; [email protected])
  Mozilla/5.0 (compatible; KAZ.KZ_Bot/3.0)
  BuckyOHare/1.3 (Googlebot/2.1; +https://hypefactors.com/webcrawler)
  uipbot/1.0 ([email protected])
  gMozilla/5.0 (compatible; Infohelfer/1.4.3; +http://www.infohelfer.de/crawler.php)
  Mozilla/5.0 (compatible; lincobot/0.3; http://www.linksco.com/about/robot.html) AppEngine-Google; (+http://code.google.com/appengine)
  Mozilla/5.0 (compatible; MSIE 10.0; Windows NT 6.2; WOW64; Trident/6.0) MapperCmd/5.24.780.0
  Seeker v.1 http://LookSeek.com/seeker/
  AppManager RPT-HTTPClient/0.3-3E
  LinkAlarm/3.1
  Mozilla/5.0 (X11; U; Linux i686) Firefox/2.0.0.11 wprecon.com survey
  Jeffrey's Exif Viewer (http://regex.info/exif)
  Untiny
  Mozilla/5.0 (Grabify.link - https://grabify.link/expander)
  biNu image cacher ([email protected])
  Mozilla/5.0 (compatible; Google-Youtube-Links)
  ping.blo.gs/2.0
  Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; QQDownload 1.7; .NET CLR 2.0.50727)
  Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.1; Trident/4.0; QQDownload 663; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; .NET4.0C; .NET CLR 1.1.4322)
  Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 5.1; Trident/4.0; QQDownload 760; .NET CLR 2.0.50727; .NET CLR 3.0.4506.2152; .NET CLR 3.5.30729; InfoPath.2; .NET4.0E; .NET4.0C; .NET CLR 1.1.4322)
  Mozilla/5.0 (X11; Linux x86_64; rv:52.0.2) Gecko/20100101 Firefox/52.0.2 Ghost Inspector
  Mozilla/5.0 (X11; Linux x86_64; rv:52.0) Gecko/20100101 Firefox/52.0 Ghost Inspector
  Mozilla/5.0 (X11; Linux x86_64; rv:50.1.0) Gecko/20100101 Firefox/50.1.0 Ghost Inspector
  Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/538.1 (KHTML, like Gecko) QtWebEngine/5.3.0 Safari/538.1 Ghost Inspector
  Mozilla/5.0 (X11; Linux x86_64; rv:45.0.1) Gecko/20100101 Firefox/45.0.1 Ghost Inspector
  Mozilla/5.0 (X11; Linux x86_64; rv:46.0.1) Gecko/20100101 Firefox/46.0.1 Ghost Inspector
  Mozilla/5.0 (X11; Linux x86_64; rv:47.0.2) Gecko/20100101 Firefox/47.0.2 Ghost Inspector
  Mozilla/5.0 (X11; Linux x86_64; rv:48.0.2) Gecko/20100101 Firefox/48.0.2 Ghost Inspector
  Mozilla/5.0 (X11; Linux x86_64; rv:49.0.2) Gecko/20100101 Firefox/49.0.2 Ghost Inspector
  Netpursual/1.0
  Kaspersky Lab CFR link resolver [email protected]
  Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/27.0.1453 Safari/537.36 (compatible; Google-HotelAdsVerifier/2.0)
  b0t
  "echocrawl 2.0"
  masscan/1.0
  Mozilla/5.0 (iPhone; CPU iPhone OS 8_1 like Mac OS X) AppleWebKit/600.1.4 (KHTML, like Gecko) Version/8.0 Mobile/12B411 Safari/600.1.4 (compatible; YandexBot/3.0; +http://yandex.com/bots)
  Mozilla/5.0 (compatible; YandexAccessibilityBot/3.0; +http://yandex.com/bots)
  Mozilla/5.0 (iPhone; CPU iPhone OS 8_1 like Mac OS X) AppleWebKit/600.1.4 (KHTML, like Gecko) Version/8.0 Mobile/12B411 Safari/600.1.4 (compatible; YandexMobileBot/3.0; +http://yandex.com/bots)
  Mozilla/5.0 (compatible; YandexDirectDyn/1.0; +http://yandex.com/bots
  Mozilla/5.0 (compatible; YandexImages/3.0; +http://yandex.com/bots)
  Mozilla/5.0 (compatible; YandexVideo/3.0; +http://yandex.com/bots)
  Mozilla/5.0 (compatible; YandexMedia/3.0; +http://yandex.com/bots)
  Mozilla/5.0 (compatible; YandexBlogs/0.99; robot; +http://yandex.com/bots)
  Mozilla/5.0 (compatible; YandexFavicons/1.0; +http://yandex.com/bots)
  Mozilla/5.0 (compatible; YandexWebmaster/2.0; +http://yandex.com/bots)
  Mozilla/5.0 (compatible; YandexPagechecker/1.0; +http://yandex.com/bots)
  Mozilla/5.0 (compatible; YandexImageResizer/2.0; +http://yandex.com/bots)
  Mozilla/5.0 (compatible; YaDirectFetcher/1.0; Dyatel; +http://yandex.com/bots)
  Mozilla/5.0 (compatible; YandexCalendar/1.0; +http://yandex.com/bots)
  Mozilla/5.0 (compatible; YandexSitelinks; Dyatel; +http://yandex.com/bots)
  Mozilla/5.0 (compatible; YandexMetrika/3.0; +http://yandex.com/bots)
  Mozilla/5.0 (compatible; YandexAntivirus/2.0; +http://yandex.com/bots)
  Mozilla/5.0 (compatible; YandexVertis/3.0; +http://yandex.com/bots)
  Mozilla/5.0 (compatible; YandexBot/3.0; MirrorDetector; +http://yandex.com/bots)
  jetmon/1.0 (Jetpack Site Uptime Monitor by WordPress.com)
  Mozilla/5.0 (compatible; Abonti/0.8 - http://www.abonti.com)
  CloudEndure Scanner ([email protected])
  AFB/3.0 (+http://allloadin.com)
  TLSProbe/1.0 (+https://scan.trustnet.venafi.com/)
  Sens.ai http://sovereign.ai/contact
  eContext/1.0 (eContext Classification Engine)
  Owler
  Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Slack/1.2.6 Chrome/45.0.2454.85 AtomShell/0.34.3 Safari/537.36 Slack_SSB/1.2.6
  Mozilla/5.0 (compatible; ChangesMeter/1.9.1; http://intuiware.com/apps/changes-meter)
  Mozilla/5.0 (compatible; Gookey.co/1.0; +http://gookey.co/)
  TrapitAgent/0.1 (feed processor; +http://trapit.com/about)
  sqlmap/1.0-dev-nongit-201612050a8c (http://sqlmap.org)
  UniversalFeedParser/3.3 +http://feedparser.org/
  TulipChain/5.xx (http://ostermiller.org/tulipchain/) Java/1.x.1_0x (http://apple.com/) Mac_OS_X/10.2.8
  scalaj-http/1.0
  HggH PhantomJS Screenshoter
  fasthttp
  fasthttp, Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/53.0.2785.101 Safari/537.36
  lua-resty-http/0.10 (Lua) ngx_lua/10000
  GIS-LABS:CertMon
  GuzzleHttp/6.2.1 PHP/7.0.15-0ubuntu0.16.04.4
  http-request/v0.7.0 (http://git.io/tl_S2w) node.js/v0.10.29
  intraVnews/1.x
  MailChimp
  G-i-g-a-b-o-t
  Symfony BrowserKit
  VCPP71 http-get 1.0a
  Jaunt/1.2
  Mozilla/5.0 (compatible; Owler/0.4; +; )
  MovableType/x.x
  OMSC
  websitepulse agent/0.1 (compatible; MSIE 5.5; Netscape 4.75; Linux)
  Mozilla/5.0 (compatible; Uptimia; www.uptimia.com)
  ssl-tools.net
  Mozilla/5.0 (compatible; MSIE 10.0; Windows NT 6.1; Trident/6.0; .NET CLR 1.1.4322; .NET CLR 1.0.3705; .NET CLR 2.0.50727; .NET CLR 3.0.04506.648; .NET CLR 3.5.21022; .NET CLR 3.0.4506.2152; .NET CLR 3.5.30729) DMBrowser/2.1
  keycdn-tools/perf
  keycdn-tools/br
  keycdn-tools/h2
  keycdn-tools/perf
  keycdn-tools/br
  keycdn-tools/h2
  KOCMOHABT (+https://kozmonavt.tk/) Mozilla/5.0 (compatible; Web Explorer 2)
  Donuts Content Explorer (www.donuts.domains)
  Netsparker
  Mozilla/5.0+(compatible; MxToolbox/Beta7; http://www.mxtoolbox.com/)
  Apache-HttpAsyncClient/4.1.3 (Java/1.8.0_121)
  Tenon.io Automated Web Accessibility Tester
  ApacheBench/2.3
  Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; Clarsentia)
  Mozilla/5.0 (compatible; historious/2.0; +http://historio.us/)
  Mozilla/5.0 (compatible; uCrawlr/1.0 ; +https://blog.ucoz.ru/upolicy)
  Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)
  Brodie/1.0 (+http://www.15miles.com/)
  Jobg8 URL Monitor
  Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.21 (KHTML, like Gecko) wpif Safari/537.21
  Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.21 (KHTML, like Gecko) imgsizer Safari/537.21
  Mozilla/5.0 (compatible; AppBeat/1.0; +http://www.appbeat.io/)
  Mozilla/5.0 (compatible; Chirp/1.0; +http://www.binarycanary.com)
  Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; WOW64; Trident/5.0; SiteMonitor)
  Mixnode/3.0.1 (+https://www.mixnode.com/; Speed analysis; no data is stored)
  Pingability.com Monitoring Service

Тоже, но с учетом регулярных выражений:

  Скрыть контент

.*Java.*outbrain

008\/

192\.comAgent

2ip\.ru

404checker

^bluefish

^Calypso v\/

^COMODO DCV

^DangDang

^DavClnt

^FDM

^git\/

^Goose\/

^HTTPClient\/

^Java\/

^Jeode\/

^Jetty\/

^Mget

^Microsoft URL Control

^NG\/[0-9\.]

^NING\/

^PHP\/[0-9]

^RMA\/

^Ruby|Ruby\/[0-9]

^scrutiny\/

^VSE\/[0-9]

^WordPress\.com

^XRL\/[0-9]

^ZmEu

a3logics\.in

A6-Indexer

a\.pr-cy\.ru

Abonti\/

Aboundex

aboutthedomain

Accoona-AI-Agent

acoon

acrylicapps\.com\/pulp

adbeat

AddThis

ADmantX

adressendeutschland

Advanced Email Extractor v

agentslug

AHC

aihit

aiohttp\/

Airmail

akula\/

alertra

alexa site audit

Alibaba\.Security\.Heimdall

allloadin\.com

alyze\.info

amagit

AndroidDownloadManager

Anemone

Ant\.com

Anturis Agent

AnyEvent-HTTP\/

ApacheBench\/

Apache-HttpAsyncClient\/

Apache-HttpClient\/

AportWorm\/[0-9]

AppBeat\/[0-9]

AppEngine-Google

Arachmo

arachnode

Arachnophilia

aria2

asafaweb.com

AskQuickly

Astute

asynchttp

autocite

Autonomy

B-l-i-t-z-B-O-T

^b0t$

Backlink-Ceck\.de

Bad-Neighborhood

baidu\.com

baypup\/[0-9]

baypup\/colbert

BazQux

BCKLINKS

BDFetch

BegunAdvertising\/

BigBozz

biglotron

BingLocalSearch

BingPreview

binlar

biNu image cacher

biz_Directory

Blackboard Safeassign

Bloglovin

BlogPulseLive

BlogSearch

Blogtrottr

boitho\.com-dc

BPImageWalker

Braintree-Webhooks

Branch Metrics API

Branch-Passthrough

Brodie\/

Browsershots

BUbiNG

Butterfly\/

BuzzSumo

CAAM\/[0-9]

CakePHP

CapsuleChecker

CaretNail

catexplorador

cb crawl

CC Metadata Scaper

Cerberian Drtrs

CERT\.at-Statistics-Survey

cg-eye

changedetection

ChangesMeter\/

Charlotte

CheckHost

checkprivacy

chkme\.com

Chirp\/[0-9]

CirrusExplorer\/

CISPA Vulnerability Notification

CJNetworkQuality

Clarsentia

clips\.ua\.ac\.be

Cloud mapping experiment

CloudEndure

CloudFlare-AlwaysOnline

Cloudinary\/[0-9]

cmcm\.com

coccoc

CommaFeed

Commons-HttpClient

Comodo SSL Checker

contactbigdatafr

convera

copyright sheriff

Covario-IDS

CrawlForMe\/[0-9]

cron-job\.org

Crowsnest

curb

Curious George

curl

cuwhois\/[0-9]

cybo\.com

DareBoost

DataparkSearch

dataprovider

Daum(oa)?[ \/][0-9]

DeuSu

developers\.google\.com\/\+\/web\/snippet\/

Digg

Dispatch\/

dlvr

DMBrowser

DNS-Tools Header-Analyzer

DNSPod-reporting

docoloc

Dolphin http client\/

DomainAppender

Donuts Content Explorer

dotSemantic

downforeveryoneorjustme

downnotifier\.com

DowntimeDetector

Dragonfly File Reader

drupact

Drupal \(\+http:\/\/drupal\.org\/\)

dubaiindex

EARTHCOM

Easy-Thumb

ec2linkfinder

eCairn-Grabber

ECCP

echocrawl

eContext\/

ElectricMonk

elefent

EMail Exractor

EmailWolf

Embed PHP Library

Embedly

europarchive\.org

evc-batch\/[0-9]

EventMachine HttpClient

Evidon

Evrinid

ExactSearch

ExaleadCloudview

Excel\/

Exif Viewer

Exploratodo

ezooms

facebookexternalhit

facebookplatform

fairshare

Faraday v

fasthttp

Faveeo

Favicon downloader

FavOrg

Feed Wrangler

Feedbin

FeedBooster

FeedBucket

FeedBunch\/[0-9]

FeedBurner

FeedChecker

Feedly

Feedspot

Feedwind\/[0-9]

feeltiptop

Fetch API

Fetch\/[0-9]

Fever\/[0-9]

findlink

findthatfile

FlipboardBrowserProxy

FlipboardProxy

FlipboardRSS

fluffy

flynxapp

forensiq

FoundSeoTool\/[0-9]

free thumbnails

FreeWebMonitoring SiteChecker

Funnelback

G-i-g-a-b-o-t

g00g1e\.net

GAChecker

ganarvisitas\/[0-9]

geek-tools

Genderanalyzer

Genieo

GentleSource

GetLinkInfo

getprismatic\.com

GetURLInfo\/[0-9]

Ghost Inspector

GigablastOpenSource

GIS-LABS

github\.com\/

Go [\d\.]* package http

Go-http-client

gofetch

GomezAgent

gooblog

Goodzer\/[0-9]

Google favicon

Google Keyword Suggestion

Google Keyword Tool

Google Page Speed Insights

Google PP Default

Google Search Console

Google Web Preview

Google-Adwords

Google-Apps-Script

Google-Calendar-Importer

Google-HotelAdsVerifier

Google-HTTP-Java-Client

Google-Publisher-Plugin

Google-SearchByImage

Google-Site-Verification

Google-Structured-Data-Testing-Tool

Google-Youtube-Links

google_partner_monitoring

GoogleDocs

GoogleHC\/

GoogleProducer

Gookey

GoScraper

GoSpotCheck

GoSquared-Status-Checker

gosquared-thumbnailer

grabify

Grammarly

grouphigh

grub-client

GTmetrix

GuzzleHttp

gvfs\/

HAA(A)?RTLAND http client

Hatena

hawkReader

HEADMasterSEO

HeartRails_Capture

heritrix

historious\/

hledejLevne\.cz\/[0-9]

Holmes

HootSuite Image proxy

Hootsuite-WebFeed\/[0-9]

HostTracker

ht:\/\/check

htdig

HTMLParser\/

http-get

HTTP-Header-Abfrage

http-kit

http-request\/

HTTP-Tiny

HTTP_Compression_Test

http_request2

http_requester

HttpComponents

httphr

HTTPMon

PEAR HTTPRequest

httpscheck

httpssites_power

httpunit

HttpUrlConnection

httrack

hosterstats

huaweisymantec

HubPages.*crawlingpolicy

HubSpot Connect

HubSpot Marketing Grader

HyperZbozi.cz Feeder

i2kconnect\/

ichiro

IdeelaborPlagiaat

IDG Twitter Links Resolver

IDwhois\/[0-9]

Iframely

igdeSpyder

IlTrovatore

ImageEngine\/

Imagga

imgsizer

InAGist

inbound\.li parser

InDesign%20CC

infegy

infohelfer

InfoWizards Reciprocal Link System PRO

Instapaper

inpwrd\.com

Integrity

integromedb

internet_archive

InternetSeer

internetVista monitor

intraVnews

IODC

IOI

iplabel

IPS\/[0-9]

ips-agent

IPWorks HTTP\/S Component

iqdb\/

Irokez

isitup\.org

iskanie

iZSearch

janforman

Jaunt\/

Jigsaw

Jobboerse

Jobg8 URL Monitor

jobo

Jobrapido

JS-Kit

Kaspersky Lab CFR link resolver

KeepRight OpenStreetMap Checker

KeyCDN

Keywords Research

KickFire

KimonoLabs\/

Kml-Google

knows\.is

KOCMOHABT

kouio

kulturarw3

KumKie

L\.webis

Larbin

LayeredExtractor

letsencrypt

LibVLC

libwww

Licorne Image Snapshot

Liferea\/

link checker

Link Valet

link_thumbnailer

LinkAlarm\/

linkCheck

linkdex

LinkExaminer

linkfluence

linkpeek

LinkTiger

LinkWalker

Lipperhey

livedoor ScreenShot

LoadImpactPageAnalyzer

LoadImpactRload

LongURL API

looksystems\.net

ltx71

lua-resty-http

lwp-trivial

lycos

LYT\.SR

mabontland

MagpieRSS

Mail.Ru

MailChimp

Mandrill

MapperCmd

marketinggrader

masscan\/[0-9]

Mediapartners-Google

MegaIndex\.ru

Melvil Rawi\/

MergeFlow-PageReader

Metaspinner

MetaURI

Microsearch

Microsoft-WebDAV-MiniRedir

Microsoft Data Access Internet Publishing Provider Protocol

Microsoft Office

Microsoft Windows Network Diagnostics

Mindjet

Miniflux

mixdata dot com

mixed-content-scan

mixnode

Mnogosearch

mogimogi

Mojolicious \(Perl\)

monitis

Monitority\/[0-9]

montastic

MonTools

Moreover

Morning Paper

mowser

MovableType

Mrcgiguy

mShots

MxToolbox\/

MVAClient

nagios

Najdi\.si\/

Needle\/

NETCRAFT

NetLyzer FastProbe

Netpursual

netresearch

NetShelter ContentScan

Netsparker

NetTrack

Netvibes

Neustar WPM

NeutrinoAPI

NewsBlur .*Finder

NewsGator

newsme

newspaper\/

NG-Search

nineconnections\.com

NLNZ_IAHarvester

Nmap Scripting Engine

node-superagent

node\.io

nominet\.org\.uk

Norton-Safeweb

Notifixious

notifyninja

nuhk

nutch

Nuzzel

nWormFeedFinder

Nymesis

Ocelli\/[0-9]

oegp

okhttp

Omea Reader

omgili

OMSC

Online Domain Tools

OpenCalaisSemanticProxy

Openstat\/

OpenVAS

Optimizer

Orbiter

OrgProbe\/[0-9]

Owler

ow\.ly

ownCloud News

OxfordCloudService\/[0-9]

Page Analyzer

Page Valet

page2rss

page_verifier

PagePeeker

Pagespeed\/[0-9]

Panopta

panscient

parsijoo

PayPal IPN

Pcore-HTTP

Pearltrees

peerindex

Peew

PhantomJS\/

PhantomJS Screenshoter

Photon\/

phpcrawl

phpservermon

Pi-Monster

ping\.blo\.gs\/

Pingability

Pingdom

Pingoscope

PingSpot

pinterest\.com

Pizilla

Ploetz \+ Zeller

Plukkie

PocketParser

Pompos

Porkbun

Port Monitor

postano

PostPost

postrank

PowerPoint\/

Priceonomics Analysis Engine

PritTorrent\/[0-9]

Prlog

probethenet

Project 25499

Promotion_Tools_www.searchenginepromotionhelp.com

prospectb2b

Protopage

proximic

pshtt, https scanning

PTST

PTST\/[0-9]+

Pulsepoint XT3 web scraper

Python-httplib2

python-requests

Python-urllib

Qirina Hurdler

QQDownload

Qseero

Qualidator.com SiteAnalyzer

Quora Link Preview

Qwantify

Radian6

RankSonicSiteAuditor

Readability

RealPlayer%20Downloader

RebelMouse

redback\/

Redirect Checker Tool

ReederForMac

request\.js

ResponseCodeTest\/[0-9]

RestSharp

RetrevoPageAnalyzer

Riddler

Rival IQ

Robosourcer

Robozilla\/[0-9]

ROI Hunter

RPT-HTTPClient

RSSOwl

safe-agent-scanner

SalesIntelligent

SauceNAO

SBIder

scalaj-http

Scoop

scooter

ScoutJet

ScoutURLMonitor

Scrapy

ScreenShotService\/[0-9]

Scrubby

search\.thunderstone

SearchSight

Seeker

semanticdiscovery

semanticjuice

Semiocast HTTP client

SEO Browser

Seo Servis

seo-nastroj.cz

Seobility

SEOCentro

SeoCheck

SeopultContentAnalyzer

Server Density Service Monitoring

servernfo\.com

Seznam screenshot-generator

Shelob

Shoppimon Analyzer

ShoppimonAgent\/[0-9]

ShopWiki

ShortLinkTranslate

shrinktheweb

SilverReader

SimplePie

SimplyFast

Site-Shot\/

Site24x7

SiteBar

SiteCondor

siteexplorer\.info

SiteGuardian

Siteimprove\.com

Sitemap(s)? Generator

SiteMonitor

Siteshooter B0t

SiteTruth

sitexy\.com

SkypeUriPreview

Slack\/

slider\.com

slurp

SMRF URL Expander

SMUrlExpander

Snappy

SniffRSS

sniptracker

Snoopy

sogou web

SortSite

sovereign\.ai

spaziodati

Specificfeeds

speedy

SPEng

Spinn3r

spray-can

Sprinklr

sqlmap

spyonweb

Sqworm

SSL Labs

ssl-tools

StackRambler

Statastico\/

StatusCake

Stratagems Kumo

Stroke.cz

StudioFACA

suchen

summify

Super Monitoring

Surphace Scout

SwiteScraper

Symfony2 BrowserKit

SynHttpClient-Built

Sysomos

Symfony BrowserKit

T0PHackTeam

Tarantula\/

Taringa UGC

Tenon\.io

teoma

terrainformatica\.com

Test Certificate Info

Tetrahedron\/[0-9]

The Drop Reaper

The Expert HTML Source Viewer

theinternetrules

theoldreader\.com

Thumbshots

ThumbSniper

TinEye

Tiny Tiny RSS

TLSProbe\/

topster

touche.com

Traackr.com

TrapitAgent

truwoGPS

TulipChain

tweetedtimes\.com

Tweetminster

Tweezler\/

Twikle

Twingly

ubermetrics-technologies

uclassify

uCrawlr\/

UdmSearch

UniversalFeedParser

Untiny

UnwindFetchor

updated

Upflow

Uptimia

URLChecker

URLitor.com

urlresolver

Urlstat

UrlTrends Ranking Updater

Vagabondo

vBSEO

via ggpht\.com GoogleImageProxy

VidibleScraper\/

visionutils

vkShare

voltron

voyager\/

VSAgent\/[0-9]

VSB-TUO\/[0-9]

VYU2

w3af\.org

W3C-checklink

W3C-mobileOK

W3C_I18n-Checker

W3C_Unicorn

wangling

WatchMouse

WbSrch\/

web-capture\.net

Web-Monitoring

Web-sniffer

Webauskunft

WebCapture

WebClient\/

webcollage

WebCookies

WebCorp

WebDoc

WebFetch

WebImages

WebIndex

webkit2png

webmastercoffee

webmon

webscreenie

Webshot

Website Analyzer\/

websitepulse agent

websitepulse[+ ]checker

Websnapr\/

Webthumb\/[0-9]

WebThumbnail

WeCrawlForThePeace

WeLikeLinks

WEPA

WeSEE

wf84

wget

WhatsApp

WhatsMyIP

WhatWeb

WhereGoes\?

Whibse

Whynder Magnet

Windows-RSS-Platform

WinHttpRequest

wkhtmlto

wmtips

Woko

Word\/

WordPress\/

wotbox

WP Engine Install Performance API

wpif

wprecon\.com survey

WPScan

wscheck

WWW-Mechanize

www\.monitor\.us

XaxisSemanticsClassifier

Xenu Link Sleuth

XING-contenttabreceiver\/[0-9]

XmlSitemapGenerator

xpymep([0-9]?)\.exe

Y!J-(ASR|BSC)

Yaanb

yacy

Yahoo Ad monitoring

Yahoo Link Preview

YahooCacheSystem

YahooYSMcm

YandeG

Yandex(?!Search)

yanga

yeti

YLT

Yo-yo

Yoleo Consumer

yoogliFetchAgent

YottaaMonitor

yourls\.org

Zao

Zemanta Aggregator

Zend\\Http\\Client

Zend_Http_Client

zgrab

ZnajdzFoto

ZyBorg

[a-z0-9\-_]*((?<!cu)bot|crawler|archiver|transcoder|spider|uptime|validator|fetcher)

Источник здесь https://github.com/JayBizzle/Crawler-Detect .

 

Супер, спасибо

Ссылка на комментарий
Поделиться на других сайтах

Создайте аккаунт или войдите в него для комментирования

Вы должны быть пользователем, чтобы оставить комментарий

Создать аккаунт

Зарегистрируйтесь для получения аккаунта. Это просто!

Зарегистрировать аккаунт

Войти

Уже зарегистрированы? Войдите здесь.

Войти сейчас
 Поделиться

  • Сейчас на странице   0 пользователей

    • Нет пользователей, просматривающих эту страницу.
×
×
  • Создать...

Важная информация

На нашем сайте используются файлы cookie и происходит обработка некоторых персональных данных пользователей, чтобы улучшить пользовательский интерфейс. Чтобы узнать для чего и какие персональные данные мы обрабатываем перейдите по ссылке. Если Вы нажмете «Я даю согласие», это означает, что Вы понимаете и принимаете все условия, указанные в этом Уведомлении о Конфиденциальности.