Перейти к содержанию

Рекомендуемые сообщения

Есть ли у кого-нибудь более содержательный список юзер агентов ботов, чем у меня? Просьба поделиться или дополнить мой.) 

Будет полезно, например, для модуля источников заказов или различных обработчиков логов.

Мой список под спойлером ниже:

Спойлер

YandexBot
YandexImages
YandexMetrika
YaDirectFetcher
YandexMobileBot
YandexMarket
YandexImageResizer
Yandex Market picture robot
YandexAccessibilityBot
YandexCatalog
Googlebot
AdsBot-Google
Mail.RU_Bot
bingbot
msnbot-media
Baiduspider-
Baiduspider
Yahoo! Slurp
curl
AhrefsBot
MJ12bot
SeznamBot
SemrushBot
price.ru-image-robot
ltx71.com
Riddler
Exabot
SurdotlyBot
ActivizmBot
glindahl-cocrawler
Twitterbot
facebookexternalhit
Virusdie crawler
Cliqzbot
statdom.ru/Bot
openstat.ru/Bot
linkdexbot
Linkbot
coccocbot-web
SMTBot
top100.rambler.ru crawler
WebMoney MegaStock robot
CommonCrawler Node
archive.org_bot
SEOkicks-Robot
oBot
TorgMailRuBot
Python-urllib/2.7
Yandex.Commerce.Pinger
BLEXBot
linux-gnu
Googlebot-Image
AdsBot-Google-Mobile
360Spider
Trident-не бот
YisouSpider
spider
PHP-Curl-Class
slurp
crawler
pinger
robot
WGet tools
FavIconizer
FaceBook bot
CFNetwork
NG 1.x (Exalead)
Phantom
RoadHouse Crawling System
SurveyBot
Perl tool
Sogou Spider
StackRambler
TITAN
Feedfetcher-Google
Netcraft
Fish search
RBSE Spider
Sift
W3C Validator
Konqueror
Yahoo Slurp
Mail.RU Bot
archive.org bot
Apache-HttpClient/
Flamingo_SearchEngine

 

Поделиться сообщением


Ссылка на сообщение
Поделиться на другие сайты

Ну гугл-ботов больше

 

Googlebot
Googlebot-Nes
Googlebot-Images
Googlebot-Video
Google-Mobile
Google-Mobile AdSense
Google-AdSense
Google-AdsBot

и вот еще парочка которых нет у тебя

Teoma
ia_archiver
Lycos
Aport
WebAlta

 

Изменено пользователем L0ky

Поделиться сообщением


Ссылка на сообщение
Поделиться на другие сайты

Внесу свои 5 копеек

Боты:

Спойлер
TinEye-bot/0.51 (see http://www.tineye.com/crawler.html)
  Whoismindbot/1.0 (+http://www.whoismind.com/bot.html)
  Mozilla/5.0 (compatible; MSIE 8.0; Windows NT 6.3; www.alertra.com)
  R6_FeedFetcher(www.radian6.com/crawler)
  CheckHost (http://check-host.net/)
  YandeG 1.03
  yacybot (/global; amd64 Linux 3.16-0.bpo.2-amd64; java 1.7.0_65; Europe/en) http://yacy.net/bot.html
  SEMrushBot
  Protopage/3.0 (http://www.protopage.com)
  yacybot (/global; amd64 Linux 3.16.0-4-amd64; java 1.7.0_75; Europe/en) http://yacy.net/bot.html
  StatoolsBot (+http://www.statools.com/bot.html)
  adidxbot/2.0 (+http://search.msn.com/msnbot.htm)
  Mozilla/5.0 (compatible; Mail.RU_Bot/2.0; +http://go.mail.ru/help/robots)
  Mozilla/5.0 (Linux; Android 5.0; Nexus 5 Build/LRX21O) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/46.0.2490.76 Mobile Safari/537.36 PTST/281
  TurnitinBot (https://turnitin.com/robot/crawlerinfo.html)
  Scrapy/1.0.5 (+http://scrapy.org)
  yacybot (/global; amd64 Linux 4.4.0-31-generic; java 1.8.0_91; Europe/en) http://yacy.net/bot.html
  XmlSitemapGenerator - http://xmlsitemapgenerator.org
  Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.0) Match by Siteimprove.com
  Mozilla/5.0 (compatible; SeznamBot/3.2; +http://napoveda.seznam.cz/en/seznambot-intro/)
  WatchMouse/18990 (http://watchmouse.com/ ; gab)
  WatchMouse/8.4.0.3 (http://watchmouse.com/ ; hkhkg02.watchmouse.net)
  Mozilla/5.0 (compatible; LinkpadBot/1.06; +http://www.linkpad.ru)
  Mozilla/5.0 (compatible; heritrix/1.14.2 +http://rjpower.org)
  yacybot (webportal/global; amd64 Linux 2.6.32-5-amd64; java 1.6.0_18; Europe/en) http://yacy.net/bot.html
  PercolateCrawler/4 (ops@percolate.com)
  msnbot-UDiscovery/2.0b (+http://search.msn.com/msnbot.htm)
  Mozilla/5.0 (compatible; MSIE or Firefox mutant; not on Windows server;) Daum 4.1
  Mozilla/5.0 (compatible; spbot/4.0.3; +http://www.seoprofiler.com/bot )
  Mozilla/5.0 (compatible; LoadTimeBot/0.9; +http://www.loadtime.net/bot.html)
  UnwindFetchor/1.0 (+http://www.gnip.com/)
  nrsbot/5.0(loopip.com/robot.html)
  SemrushBot/0.9
  Mozilla/5.0 (compatible; UASlinkChecker/2.1; +https://udger.com/support/UASlinkChecker)
  yacybot (/global; amd64 Linux 3.11.10-21-desktop; java 1.7.0_51; America/en) http://yacy.net/bot.html
  Netvibes (http://www.netvibes.com)
  Acoon v4.1.0 (www.acoon.de)
  msnbot/2.0b (+http://search.msn.com/msnbot.htm).
  Mozilla/5.0 (compatible; ltbot/3.2.0.10 +http://www.kdsl.tu-darmstadt.de/de/kdsl/research-program/crawling-and-semantic-structuring/)
  smart.apnoti.com Robot/v1.34 (http://smart.apnoti.com/en/aboutApnotiWebCrawler.html)
  HubSpot Links Crawler 2.0 http://www.hubspot.com/
  yacybot (/global; x86 Windows XP 5.1; java 1.7.0_55; Europe/de) http://yacy.net/bot.html
  Mozilla/5.0 (compatible; DCPbot/1.0; +http://domains.checkparams.com/)
  Mozilla/5.0 (compatible; Exabot/3.0 (BiggerBetter); +http://www.exabot.com/go/robot)
  WebDoc (abuse-webdoc at service.moquadv.com)
  coccoc
  Mozilla/5.0 (compatible; spbot/4.0b; +http://www.seoprofiler.com/bot )
  Mozilla/5.0 (compatible; ExchangleBot/3.0; +https://www.exchangle.com/exchangling)
  Mozilla/5.0 (compatible; Qwantify/2.0n; +https://www.qwant.com/)
  OOZBOT/0.20 ( Setooz výrazný ako say-th-uuz, znamená mosty. ; http://www.setooz.com/oozbot.html ; agentname at setooz dot_com )
  SpeedySpider - http://www.entireweb.com
  Mozilla/5.0 (compatible; Heurekabot/3.1; +http://sluzby.heureka.cz/)
  crawler for netopian (http://www.netopian.co.uk/)
  L.webis/0.51 (http://webalgo.iit.cnr.it/index.php?pg=lwebis)
  Influencebot/0.9; (Automatic classification of websites; http://www.influencebox.com/; info@influencebox.com)
  Baiduspider+(+http://www.baidu.com/search/spider.htm)
  Mozilla/5.0 (X11; compatible; semantic-visions.com crawler; HTTPClient 3.1)
  Mozilla/5.0 (Windows; U; Windows NT 6.0; en-US; rv:1.9.2.28) Gecko/20120306 Firefox/99.0 YottaaMonitor
  Scrapy/1.1.1 (+http://scrapy.org)
  Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/27.0.1453.116 Safari/537.36 HubSpot Marketing Grader
  RyzeCrawler/1.1.1 ( http://www.domain2day.nl/crawler/)
  eBot / v.1.0a (http://alfa.elchron.cz)
  Sogou News Spider/4.0(+http://www.sogou.com/docs/help/webmasters.htm#07)
  Crowsnest/0.5 (+http://www.crowsnest.tv/)
  Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.4 (KHTML, like Gecko; Google Page Speed Insights) Chrome/22.0.1229 Safari/537.4
  DoCoMo/2.0 N902iS(c100;TB;W24H12)(compatible; moba-crawler; http://crawler.dena.jp/)
  Yeti/1.1 (NHN Corp.; http://help.naver.com/robots/)
  Experibot_v1
  Nekstbot - http://www.ipipan.waw.pl/nekst/nekstbot/
  Mozilla/5.0 (compatible; MJ12bot/v1.4.5; http://www.majestic12.co.uk/bot.php?+)
  Mozilla/5.0 (X11; Linux i686) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/30.0.1599.101 Safari/537.36; SSL-Crawler: http://crawler.dcsec.uni-hannover.de
  CorpusCrawler 2.0.19 (http://corpora.fi.muni.cz/crawler/);Project:CzCorpus
  Mozilla/5.0 (compatible; Semager/1.4; http://www.semager.de/blog/semager-bots/)
  Mozilla/5.0 (compatible; dlcbot/0.1; +http://www.drlinkcheck.com/)
  yacybot (freeworld/global; amd64 Linux 3.11.10-21-desktop; java 1.7.0_51; Europe/de) http://yacy.net/bot.html
  A6-Indexer/1.0 (http://www.a6corp.com/a6-web-scraping-policy/)
  CopperEgg/RevealUptime/DallasTX(linode)
  Mozilla/5.0 (compatible; Qwantify/2.1dw; +https://www.qwant.com/)/*
  Curious George - www.analyticsseo.com
  GozaikBot (www.gozaik.com;webmaster@gozaik.com;www.gozaik.com/gozaikbot.html)
  Pu_iN Crawler (+http://semanticjuice.com/)
  Mozilla/5.0 (compatible; OpenindexDeepSpider/Nutch-1.5-dev; +http://www.openindex.io/en/webmasters/spider.html; systemsATopenindexDOTio)
  yacybot (freeworld/global; i386 Linux 3.0.0-17-generic-pae; java 1.6.0_23; Europe/en) http://yacy.net/bot.html
  yacybot (-global; amd64 FreeBSD 9.2-RELEASE-p10; java 1.7.0_65; Europe/en) http://yacy.net/bot.html
  FAST Enteprise Crawler/6 (www dot fastsearch dot com)
  Mozilla/5.0 (compatible; parsijoo-update-crawler; +http://www.parsijoo.ir/; ehsan.mousakazemi@gmail.com)
  yacybot (freeworld/global; amd64 Windows Server 2008 6.0; java 1.7.0_25; Europe/en) http://yacy.net/bot.html
  Inspingbot/1.0 (+https://www.insping.com/)
  OrgProbe/0.9.4 (+http://www.blocked.org.uk)
  Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; Trident/5.0); 360Spider
  Mozilla/5.0 (Windows NT 6.1; WOW64; Trident/7.0; rv:11.0) like Gecko PTST/276
  Mozilla/5.0 (Windows NT 6.3; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/42.0.2311.90 Safari/537.36 PTST/276
  Mozilla/5.0 (compatible; SemrushBot/0.97~bl; +http://www.semrush.com/bot.html)
  Feedspot http://www.feedspot.com
  Zookabot/2.0;++http://zookabot.com
  drupact/0.7; http://www.arocom.de/drupact
  RobotsChecker/0.6 (+http://www.blocked.org.uk)
  yacybot (/global; i386 Linux 3.13.0-37-generic; java 1.7.0_65; Europe/en) http://yacy.net/bot.html
  Scrapy/1.0.3 (+http://scrapy.org)
  Testomatobot/1.0 (Linux x86_64; +http://www.testomato.com/testomatobot) minicrawler/4.0.0~beta12
  Slack-ImgProxy 0.66 (+https://api.slack.com/robots)
  Mozilla/5.0 (compatible; WebCookies/1.0; +https://webcookies.org/faq/#agent)
  NalezenCzBot/1.0 (http://www.nalezen.cz)
  yacybot (freeworld/global; amd64 Windows Server 2008 6.0; java 1.7.0_03; Europe/en) http://yacy.net/bot.html
  Mozilla/5.0 (compatible; AhrefsBot/5.1; +http://ahrefs.com/robot/)
  yacybot (freeworld/global; amd64 Windows 8 6.2; java 1.7.0_51; Europe/de) http://yacy.net/bot.html
  yacybot (/global; amd64 Windows 7 6.1; java 1.8.0_65; Europe/de) http://yacy.net/bot.html
  Mozilla/4.0 compatible ZyBorg/1.0 (wn-16.zyborg@looksmart.net; http://www.WISEnutbot.com)
  Mozilla/5.0 (en-us) AppleWebKit/537.36(KHTML, like Gecko; Google-Adwords-DisplayAds-WebRender;) Chrome/27.0.1453Safari/537.36
  Crawler powered by contentDetection (www.mindup.de)
  PagePeeker.com (info: http://pagepeeker.com/robots)
  Mozilla/5.0 (compatible; Linux x86_64; Mail.RU_Bot/Fast/2.0; +http://go.mail.ru/help/robots)
  coccoc/1.0 (http://help.coccoc.com/)
  Favicon downloader (+https://favico.be/bot.html)
  Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0) (larbin2.6.3@unspecified.mail)
  Mozilla/5.0 (compatible; DIY-SEOBot/0.1a; +http://www.upcity.com/bot.html)
  Mozilla/5.0 (Windows NT 6.3; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/49.0.2623.112 Safari/537.36 PTST/276
  Mozilla/5.0 (compatible; theoldreader.com)
  Mozilla/5.0+(compatible; UptimeRobot/2.0; http://www.uptimerobot.com/)
  hawkReader/1.8 (Link Parser; http://www.hawkreader.com/; Allow like Gecko) Build/f2b2566
  Mozilla/5.0 (compatible; heritrix/1.12.1 +http://www.webarchiv.cz)
  Mozilla/5.0 (compatible; YandexImages/3.0; +http://yandex.com/bots)
  yacybot (/global; amd64 Linux 3.14.32-xxxx-grs-ipv6-64; java 1.7.0_75; Europe/en) http://yacy.net/bot.html
  Sogou web spider/4.0(+http://www.sogou.com/docs/help/webmasters.htm#07)
  yacybot (/global; amd64 Windows 7 6.1; java 1.8.0_45; Europe/de) http://yacy.net/bot.html
  larbin_2.6.3 gqnmgsp@ruc.edu.cn
  yacybot (freeworld/global; amd64 Linux 3.10.17-gentoo; java 1.7.0_45; UTC/en) http://yacy.net/bot.html
  MeMoNewsBot/2.0 (http://www.memonews.com/en/crawler)
  Mozilla/5.0 (compatible; MSIE 10.0; Windows NT 6.1; WOW64; Trident/6.0) PTST/281
  Mozilla/5.0 (compatible; DCPbot/1.5; +http://domains.checkparams.com/)
  Mozilla/5.0 (compatible; SemrushBot/0.97; +http://www.semrush.com/bot.html)
  BlackBerry9000/4.6.0.167 Profile/MIDP-2.0 Configuration/CLDC-1.1 VendorID/102 ips-agent
  Mozilla/5.0 (compatible; Baiduspider-cpro; +http://www.baidu.com/search/spider.html)
  yacybot (/global; amd64 Linux 3.2.0-4-amd64; java 1.7.0_65; Europe/en) http://yacy.net/bot.html
  Mozilla/5.0 (compatible; electricmonk/3.1.1 +https://www.duedil.com/our-crawler/)
  Hatena Antenna/0.5 (http://a.hatena.ne.jp/help)
  Backlink-Ceck.de (+http://www.backlink-check.de/bot.html)
  Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; WOW64; Trident/5.0) commoncrawl.org/research//Nutch-1.7-SNAPSHOT
  Mozilla/5.0 (compatible; Steeler/3.5; http://www.tkl.iis.u-tokyo.ac.jp/~crawler/)
  LapozzBot/1.4 (+http://robot.lapozz.com)
  Mozilla/5.0 (WhatsMyIP.org HTTP_Compression_Test) http://whatsmyip.org/ua
  Mozilla/5.0 (compatible; MSIE or Firefox mutant; not on Windows server;) Daumoa/4.0 (Following Mediapartners-Google)
  Mozilla/5.0 (compatible; ExaleadCloudview/6;)
  ADmantX Platform Semantic Analyzer - ADmantX Inc. - www.admantx.com - support@admantx.com
  msnbot/1.0 (+http://search.msn.com/msnbot.htm)
  Mozilla/5.0 (Windows NT 5.1) AppleWebKit/537.4 (KHTML, like Gecko) Chrome/22.0.1229.79 Safari/537.4 LinkTiger 2.0
  Pixray-Seeker/1.1 (Pixray-Seeker; crawler@pixray.com)
  mbot v.1.16
  MXT/Nutch-1.12-SNAPSHOT (http://t.co/GSRLLKex24; informatique at mixdata dot com)
  Testomatobot/1.0 (Linux x86_64; +http://www.testomato.com/testomatobot) minicrawler/3.0.0
  polybot 1.0 (http://cis.poly.edu/polybot/)
  Mozilla/5.0 (compatible; YandexMetrika/2.0; +http://yandex.com/bots mtweb01t.yandex.ru)
  Mozilla/5.0 (compatible; memoryBot/1.21.46 +http://internetmemory.org/en/)
  Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_1) AppleWebKit/600.2.5 (KHTML, like Gecko) Version/8.0.2 Safari/600.2.5 (Applebot/0.1)
  yacybot (amd64 Linux 2.6.26-2-amd64; java 1.6.0_0; Europe/en) http://yacy.net/bot.html
  yacybot (/global; x86 Windows 8.1 6.3; java 1.8.0_45; America/en) http://yacy.net/bot.html
  Mozilla/5.0 (compatible; startmebot/1.0; +http://www.start.me/bot)
  yacybot (amd64 Windows 7 6.1; java 1.6.0_21; Europe/fr) http://yacy.net/bot.html
  yacybot (/global; amd64 Windows 7 6.1; java 1.8.0_101; Asia/ru) http://yacy.net/bot.html
  yacybot (freeworld/global; amd64 Linux 3.0.0-17-generic; java 1.6.0_23; Europe/de) http://yacy.net/bot.html
  alexa v0.1.4 (http://www.openwebspider.org/)
  http://arachnode.net 1.4
  Photon/1.0
  NetpeakCheckerBot
  GIDBot/3.0 (+http://www.gidnetwork.com/tools/gzip-test.php)
  Yandex/1.01.001 (compatible; Win16; P)
  w3dt.net httphr/2.0
  yacybot (/global; amd64 Windows 7 6.1; java 1.8.0_51; Europe/de) http://yacy.net/bot.html
  iqdb/0.1 (+http://iqdb.org/)
  Mozilla/5.0 (compatible; GimmeUSAbot/1.0; +https://gimmeusa.com/pages/crawler)
  Motoricerca-Robots.txt-Checker/1.0 (http://tool.motoricerca.info/robots-checker.phtml)
  Mozilla/5.0 (compatible; Tagoobot/3.0; +http://www.tagoo.ru)
  Mozilla/5.0 (Windows NT 6.1; WOW64; rv:26.0) Gecko/20100101 Firefox/26.0 Evidon (lab@evidon.com)
  Mozilla/5.0 (compatible; Nmap Scripting Engine; http://nmap.org/book/nse.html)
  yacybot (i386 Linux 2.6.32-22-generic; java 1.6.0_20; Europe/de) http://yacy.net/bot.html
  Mozilla/5.0 (Linux; Android 4.4.3; HTC One Build/KTU84L) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/40.0.2125.111 Mobile Safari/537.36 DareBoost
  mindUpBot (datenbutler.de)
  Mozilla/5.0 (compatible; monitis - premium monitoring service; http://www.monitis.com)
  Mozilla/5.0 (compatible; RankSonicSiteAuditor/1.0; +https://ranksonic.com/ranksonic_sab.html)
  Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/40.0.2125.111 Safari/537.36 DareBoost
  Zookabot/2.1;++http://zookabot.com
  yacybot (freeworld/global; amd64 Windows 7 6.1; java 1.6.0_24; Europe/de) http://yacy.net/bot.html
  Mozilla/5.0 (compatible; Google-Structured-Data-Testing-Tool +http://developers.google.com/structured-data/testing-tool/)
  Mozilla/5.0 (compatible; Plukkie/1.2; http://www.botje.com/plukkie.htm)
  focusbot/1.1
  Mozilla/5.0 (compatible; idmarch Automatic.beta/1.2; +http://www.idmarch.org/bot.html)
  CorpusCrawler 2.0.14 (http://corpora.fi.muni.cz/crawler/)
  Mozilla/5.0 (compatible; XML Sitemaps Generator; https://www.xml-sitemaps.com) Gecko XML-Sitemaps/1.0
  ImplisenseBot 1.1
  Promotion_Tools_www.searchenginepromotionhelp.com
  Aboundex/0.2 (http://www.aboundex.com/crawler/)
  CorpusCrawler 2.0.22 (http://corpora.fi.muni.cz/crawler/);Project:CzCorpus
  yacybot (/global; amd64 no-os-name no-os-version; java no-java-version; Europe/en) http://yacy.net/bot.html
  LumpImageSearch/0.1 (+http://lump.co/about/bot)
  Mozilla/4.0 (compatible; NaverBot/1.0; http://help.naver.com/customer_webtxt_02.jsp)
  Mozilla/5.0 (compatible; UASlinkChecker/2.0; +http://udger.com/support/UASlinkChecker)
  GAChecker (+http://www.gachecker.com)
  yacybot (freeworld/global; x86 Windows 7 6.1; java 1.7.0_25; Europe/de) http://yacy.net/bot.html
  Influencebot/0.9; (Automatic classification of websites; http://www.influencebox.com/; info@influencebox.com)User-Agent: Mozilla/5.0 (X11; Linux i686; rv:9.0) Gecko/20100101 Firefox/9.0
  Experibot_v1 (https://dl.dropboxusercontent.com/u/8024465/site/Info.html)
  DNSPod-reporting(http://www.dnspod.cn/reporting)
  yacybot (freeworld/global; i386 Linux 3.2.0-23-generic; java 1.6.0_27; Europe/en) http://yacy.net/bot.html
  Mozilla/5.0 (Windows NT 6.3;compatible; Leikibot/1.0; +http://www.leiki.com)
  PostPost/1.0 (+http://postpo.st/crawlers)
  envolk/1.7 (+http://www.envolk.com/envolkspiderinfo.html)
  Snapbot/1.0 (Snap Shots, +http://www.snap.com)
  Mozilla/5.0 (Linux; Android 4.0.4; Galaxy Nexus Build/IMM76B) AppleWebKit/537.36 (KHTML, like Gecko; Google-Publisher-Plugin) Chrome/27.0.1453 Mobile Safari/537.36
  Mozilla/5.0 (compatible; Uptimebot/0.2.18; +http://www.uptime.com/uptimebot)
  MIA DEV/search:robot/0.0.1 (This is the MIA Bot - crawling for mia research project. If you feel unhappy and do not want to be visited by our crawler send an email to spider@neofonie.de; http://spider.neofonie.de; spider@neofonie.de)
  Mozilla/5.0 (Windows NT 6.1; Trident/7.0; rv:11.0; PTST 2.386) like Gecko
  Jyxobot/1
  WebAlta Crawler/2.0 (http://www.webalta.net/ru/about_webmaster.html) (Windows; U; Windows NT 5.1; ru-RU)
  GC3pro+dir SEO Tools - Vers. 3.00b - For more informations: http://chkme.com/
  Mozilla/5.0 (X11; Linux x86_64; rv:45.0; GTmetrix https://gtmetrix.com/) Gecko/20100101 Firefox/45.0
  Mozilla/5.0 (Windows NT 6.1; WOW64; rv:18.0) Gecko/20100101 Firefox/18.0 AppEngine-Google; (+http://code.google.com/appengine; appid: s~aeshortener)
  Priceonomics Analysis Engine - Fetch/1.0
  Ruky-Roboter (Version: 1.06, powered by www.ruky.de +http://www.ruky.de/bot.html)
  Baiduspider+(+http://help.baidu.jp/system/05.html)
  Openstat/0.1
  Yandex/1.01.001 (compatible; Win16; m)
  Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_1) AppleWebKit/600.2.5 (KHTML, like Gecko) Version/8.0.2 Safari/600.2.5 (Applebot/0.1; +http://www.apple.com/go/applebot)
  Mozilla/5.0 (compatible; OpenindexSpider/Nutch-1.5-dev; +http://www.openindex.io/en/webmasters/spider.html)
  Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; Multiviewbot
  CERT.at-Statistics-Survey/1.0 (http://www.cert.at/about/consec/content.html)
  Mozilla/5.0 (compatible; pmoz.info ODP link checker; +http://pmoz.info/doc/botinfo.htm)
  yacybot (freeworld/global; x86_64 Mac OS X 10.6.8; java 1.6.0_29; Asia/ru) http://yacy.net/bot.html
  gonzo/1[P] (+http://www.suchen.de/faq.html)
  MixBot (+http://t.co/GSRLLKex24)
  yacybot (/global; amd64 Linux 3.2.0-4-amd64; java 1.7.0_60; Europe/en) http://yacy.net/bot.html
  Mozilla/5.0 (compatible; imbot/0.1 +http://internetmemory.org/en/
  Mozilla/5.0 (en-us) AppleWebKit/525.13 (KHTML, like Gecko; Google Web Preview) Version/3.1 Safari/525.13
  Mozilla/5.0 (compatible; Pi-Monster; https://pricepi.com/)
  ThumbSniper (http://thumbsniper.com)
  Shelob (shelob@gmx.net)
  Mozilla/5.0 (Macintosh; Intel Mac OS X 10_7_3) AppleWebKit/537.36 (KHTML, like Gecko, Google-Publisher-Plugin) Chrome/27.0.1453 Safari/537.36
  Mozilla/5.0 (compatible; KaloogaBot; http://www.kalooga.com/info.html?page=crawler)
  yacybot (/global; arm Linux 4.1.13+; java 1.8.0_40-internal; Etc/de) http://yacy.net/bot.html
  CorporateNewsSearchEngine/Nutch-1.7 (http://pibs.co/news-search-engine)
  Mozilla/5.0 (compatible; MSIE 10.0; Windows NT 6.1; WOW64; Trident/6.0) PTST/276
  Yandex/1.01.001 (compatible; Win16; I)
  FlightDeckReportsBot/2.0 (http://www.flightdeckreports.com/pages/bot)
  Scrapy/0.24.4 (+http://scrapy.org)
  ADmantX Platform Semantic Analyzer US - Turn - ADmantX Inc. - www.admantx.com - support@admantx.com
  Kyoto-Tohoku-Crawler/v1 (Mozilla-compatible; kyoto-crawler-contact@nlp.ist.i.kyoto-u.ac.jp; http://nlp.ist.i.kyoto-u.ac.jp/?crawling-kt)
  Mozilla/5.0 (compatible; Scarlett/ 1.0; +http://www.ellerdale.com/crawler.html)
  Mozilla/5.0 (compatible; NetcraftSurveyAgent/1.0; +info@netcraft.com)
  www.deadlinkchecker.com Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/46.0.2490.86 Safari/537.36
  http://arachnode.net 1.2
  Mozilla/5.0 (compatible; Plukkie/1.5; http://www.botje.com/plukkie.htm)
  yacybot (freeworld/global; arm Linux 4.4.11-v7+; java 1.7.0_101; Etc/en) http://yacy.net/bot.html
  Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.0.16) Gecko/2009121601 Ubuntu/9.04 (jaunty) Firefox/3.0.16 Specificfeeds- http://www.specificfeeds.com
  Mozilla/5.0 (compatible; evc-batch/2.0)
  Orbiter/1.2 (http://dailyorbit.com/)
  crawler4j (https://github.com/yasserg/crawler4j/)
  Mozilla/5.0 (compatible; SEOdiver/1.0; +http://www.seodiver.com/bot)
  Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/51.0.2704.79 Safari/537.36 (https://shrinktheweb.com)
  Scopia crawler 1.0 (+http://www.scopia.co)
  yacybot (i386 Linux 2.6.23; java 1.6.0_06; Europe/en) http://yacy.net/bot.html
  Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.1.2) Gecko/20090729 Firefox/3.5.2 (.NET CLR 3.5.30729; Diffbot/0.1; +http://www.diffbot.com)
  Mozilla/5.0 (compatible; LXRbot/1.0;http://www.lxrmarketplace.com/,support@lxrmarketplace.com)
  yacybot (freeworld/global; amd64 Linux 3.8.13-gentoo; java 1.7.0_21; Europe/de) http://yacy.net/bot.html
  Mozilla/5.0 (compatible; Sitemap Generator/1.3; +http://www.check-domains.com/sitemap/index.php)
  DuckDuckBot/1.1; (+http://duckduckgo.com/duckduckbot.html)
  Mozilla/5.0 (compatible; BLEXBot/1.0; +http://webmeup-crawler.com/)
  Slack-ImgProxy 0.59 (+https://api.slack.com/robots)
  Mozilla/5.0 (compatible; Ezooms/1.0; ezooms.bot@gmail.com)
  Mozilla/5.0 (compatible; YandexPagechecker/2.0; +http://yandex.com/bots)
  CoinCornerBot/1.1 ( https://www.coincorner.com/BitcoinBot)
  yacybot (freeworld/global; amd64 Linux 3.8.0-21-generic; java 1.6.0_27; Pacific/en) http://yacy.net/bot.html
  ScreenerBot Crawler Beta 2.0 (+http://www.ScreenerBot.com)
  gonzo1[P] +http://www.suchen.de/faq.html
  Mozilla/5.0 (compatible; MSIE or Firefox mutant;) Daum 4.1
  Mozilla/5.0 (compatible; 008/0.83; http://www.80legs.com/spider.html;) Gecko/2008032620
  Sogou web spider/4.02525A
  Visbot/2.0 (+http://www.visvo.com/en/webmasters.jsp;bot@visvo.com)
  Mozilla/5.0 (compatible; AcoonBot/4.10.8; +http://www.acoon.de/robot.asp)
  WatchMouse/18990 (http://watchmouse.com/ ; d3.watchmouse.com)
  Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/534.34 (KHTML, like Gecko) Qt/4.8.1 Safari/534.34 ShoppimonAgent/1.0 (feedback+agent@shoppimon.com)
  webinatorbot 1.1; +http://www.webinator.de
  findlinks/2.0.5 (+http://wortschatz.uni-leipzig.de/findlinks/)
  WikiDo/1.1 (http://wikido.com; crawler@wikido.com)
  yacybot (freeworld/global; amd64 Linux 3.5.0-27-generic; java 1.7.0_03; Europe/de) http://yacy.net/bot.html
  Mozilla/5.0 (compatible; Uptimebot/0.1.73; +http://www.uptime.com/uptimebot)
  Semantifire1/0.20 ( -- ; http://www.setooz.com/oozbot.html ; agentname at setooz dot_com )
  Speedy Spider (http://www.entireweb.com/about/search_tech/speedy_spider/)
  Mozilla/5.0 compatible; yelpspider/yelpspider-1.0 (Crawlerbot run by Yelp Inc; yelpbot at yelp dot com)
  Mozilla/5.0 (compatible; OpenHoseBot/2.1; +http://www.openhose.org/bot.html)
  Mozilla/5.0 (compatible; emefgebot/beta; +http://emefge.de/bot.html)
  Mozilla/5.0 (compatible; OpenindexShallowSpider/Nutch-1.5-dev; +http://www.openindex.io/en/webmasters/spider.html; systemsATopenindexDOTio)
  Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/49.0.2623.75 Safari/537.36 Google (+https://developers.google.com/+/web/snippet/)
  Mozilla/5.0 (compatible; MetaJobBot; http://www.metajob.de/crawler)
  CCBot/1.0 (+http://www.commoncrawl.org/bot.html)
  Lijit Crawler (+http://www.lijit.com/robot/crawler)
  baypup/1.1 (Baypup; http://www.baypup.com/; jason@baypup.com)
  Mozilla/5.0 (FauBot/0.1; +http://buzzvolume.com/fau/)
  Mozilla/5.0 (compatible; NLNZ_IAHarvester2016/3.3.0 +https://natlib.govt.nz/publishers-and-authors/web-harvesting/domain-harvest)
  yacybot (/global; amd64 Linux 3.12.1; java 1.7.0_65; Europe/en) http://yacy.net/bot.html
  yacybot (amd64 Windows 7 6.1; java 1.6.0_18; Europe/de) http://yacy.net/bot.html
  SEOENGWorldBot/1.0 (+http://www.seoengine.com/seoengbot.htm)
  Mozilla/5.0 (compatible; Finderbots finder bot; +http://wiki.github.com/bixo/bixo/bixocrawler; bixo-dev@yahoogroups.com)
  Pompos/1.3 http://dir.com/pompos.html
  Mozilla/5.0 (compatible; Gimme60bot/1.0; +http://gimme60.com) Firefox/16.0
  Mozilla/5.0 (compatible; MJ12bot/v1.4.6; http://mj12bot.com/)
  Mozilla/5.0 (compatible; Sysomos/1.0; +http://www.sysomos.com/; Sysomos)
  urlfan-bot/1.0; +http://www.urlfan.com/site/bot/350.html
  al_viewer (larbin2.6.3@unspecified.mail)
  LoadImpactRload/3.1.1 (Load Impact; http://loadimpact.com);
  findlinks/2.0.2 (+http://wortschatz.uni-leipzig.de/findlinks/)
  Mozilla/5.0 (compatible; spbot/3.1; +http://www.seoprofiler.com/bot )
  Mozilla/5.0 (compatible; AcoonBot/4.11.0; +http://www.acoon.de/robot.asp)
  WillyBot/1.1 (http://www.willyfogg.com/info/willybot)
  Norton-Safeweb
  WMCAI-robot (http://www.topicmaster.jp/wmcai/crawler.html)
  rogerbot/1.1 (http://moz.com/help/guides/search-overview/crawl-diagnostics#more-help, rogerbot-crawler+pr4-crawler-15@moz.com)
  Szukacz/1.5 (robot; www.szukacz.pl/jakdzialarobot.html; info@szukacz.pl)
  rogerbot/1.0 (http://www.seomoz.org/dp/rogerbot, rogerbot-wherecat@moz.com)
  ROR Sitemap Generator (http://www.rorweb.com)
  http://domino.research.ibm.com/comm/research_projects.nsf/pages/sai-crawler.callingcard.html
  MozillaTest/5.0 (compatible; YodaoBot/1.0; http://www.yodao.com/help/webmaster/spider/; )
  Mozilla/5.0 (compatible; Peew/1.0; http://www.peew.de/crawler/)
  Mozilla/5.0 (compatible; Website Analyzer/1.1; +http://www.check-domains.com/website-analysis/website-analyzer.php)
  Mozilla/5.0 (compatible; Gluten Free Crawler/1.0; +http://glutenfreepleasure.com/)
  PagePeeker.com
  CorpusCrawler 2.0.10 (http://corpora.fi.muni.cz/crawler/)
  yacybot (/global; amd64 Linux 3.2.0-4-amd64; java 1.7.0_101; Europe/cs) http://yacy.net/bot.html
  yacybot (/global; amd64 Linux 4.0.7-1-ck; java 1.8.0_45; Europe/de) http://yacy.net/bot.html
  Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/51.0.2704.103 Safari/537.36 PTST/284
  yacybot (freeworld/global; i386 Linux 3.12-1-686-pae; java 1.7.0_21; Europe/fr) http://yacy.net/bot.html
  Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.1; Trident/4.0; PTST 2.295)
  CovarioIDS/1.1 (http://www.covario.com/ids; support at covario dot com)
  Mozilla/5.0 (compatible; heritrix/1.14.4 +http://parsijoo.ir)
  www.adressendeutschland.de
  DialogSearch.com Bot 1.0;http://dialogsearch.com/webmasters
  librabot/2.0 (+http://search.msn.com/msnbot.htm)
  Mozilla/5.0 (compatible; heritrix/3.1.1-SNAPSHOT-20120116.200628 +http://www.archive.org/details/archive.org_bot)
  wsAnalyzer/1.0; ++http://www.wsanalyzer.com/bot.html
  Mozilla/5.0 (compatible; websays; +http://wiki.github.com/bixo/bixo/bixocrawler; bixo-dev@yahoogroups.com)
  facebookexternalhit/1.0 (+http://www.facebook.com/externalhit_uatext.php)
  yacybot (/global; amd64 Linux 4.4.5-1-ARCH; java 1.8.0_77; America/en) http://yacy.net/bot.html
  websitepulse checker/1.1 (compatible; MSIE 5.5; Netscape 4.75; Linux)
  Mozilla/5.0 (compatible; memoryBot/1.21.14 +http://mignify.com/bot.html)
  SalesIntelligent/v1.0
  larbin_2.6.2 pierre@micro-fun.ch
  Mozilla/5.0 (compatible; SiteCondor; http://www.sitecondor.com)
  yacybot (freeworld/global; amd64 Windows Server 2012 6.2; java 1.7.0_51; Europe/de) http://yacy.net/bot.html
  Mozilla/5.0 (compatible; Crawlera/1.10.2; UID 43063)
  yacybot (-global; amd64 Linux 2.6.32-042stab111.11; java 1.7.0_79; Europe/en) http://yacy.net/bot.html
  Mozilla/5.0 (compatible; SemrushBot/0.96.2; +http://www.semrush.com/bot.html)
  Linguee Bot (http://www.linguee.com/bot)
  yacybot (/global; x86 Windows XP 5.1; java 1.7.0_51; Europe/de) http://yacy.net/bot.html
  ICC-Crawler(Mozilla-compatible; ; http://kc.nict.go.jp/project1/crawl.html)
  Mozilla/5.0 (X11; U; Linux i686 (x86_64); en-US; rv:1.8.1.11) Gecko/20080109 (Charlotte/0.9t; http://www.searchme.com/support/) (Charlotte/0.9t; http://www.searchme.com/support/)
  Mozilla/5.0 (compatible; Goodzer/2.0; crawler@goodzer.com)
  Acoon v4.10.5 (www.acoon.de)
  CorpusCrawler 2.0.20 (http://corpora.fi.muni.cz/crawler/);Project:CzCorpus
  Mozilla/5.0 (compatible; AMZNKAssocBot/4.0 +http://affiliate-program.amazon.com)
  Mozilla/5.0 (compatible; MojeekBot/0.5; http://www.mojeek.com/bot.html)
  yacybot (/global; amd64 Linux 4.2.0-22-generic; java 1.7.0_91; Europe/en) http://yacy.net/bot.html
  Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/534.34 (KHTML, like Gecko) Qt/4.8.3 Safari/534.34 https://linkpeek.com
  MetaTagRobot/0.2 (http://www.seocentro.com/tools/search-engines/metatag-analyzer.html)
  Mozilla/5.0 (compatible; MJ12bot/v1.3.1; http://www.majestic12.co.uk/bot.php?+)
  Mozilla/5.0 (compatible; Uptimebot/0.2.29; +http://www.uptime.com/uptimebot)
  Mozilla/5.0 (compatible; Pandeo Bot; +http://pandeo.de/bot.php)
  Pixray-Seeker/1.1 (Pixray-Seeker; http://www.pixray.com/pixraybot; crawler@pixray.com)
  EasyBib AutoCite (http://autocite-info.citation-api.com/)
  Mozilla/5.0 (compatible; OptimizationCrawler/0.2; +http://www.domainoptima.com/robot)
  AboutUsBot/Harpy (Website Analysis; http://www.aboutus.org/Aboutus:Bot; help@aboutus.org)
  Gigabot/2.0
  DoCoMo/2.0 N905i(c100;TB;W24H16) (compatible; Googlebot-Mobile/2.1; +http://www.google.com/bot.html)
  Morning Paper 1.0 (robots.txt compliant!)
  Mozilla/5.0 (compatible; SurdotlyBot/1.0; +http://sur.ly/bot.html; Linux; Android 4; iPhone; CPU iPhone OS 6_0_1 like Mac OS X) AppleWebKit/536.26 (KHTML, like Gecko) Version/6.0 Mobile/10A523 Safari/8536.25
  Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.1; Trident/4.0; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; Media Center PC 6.0; .NET4.0C; .NET4.0E; GWX:RED; PTST 2.386)
  SuperPagesUrlVerifyBot/1.0
  CopperEgg/RevealUptime/LondonUK(linode)
  Mozilla/5.0 (compatible; www.monitor.us - free monitoring service; http://www.monitor.us)
  Mozilla/5.0 (compatible; LinkMarketbot/1.2; +http://www.linkmarket.com/)
  Mozilla/5.0 (Windows; U; Windows NT 5.1; zh-CN; ) Firefox/1.5.0.11; 360Spider
  wangling
  Mozilla/5.0 (compatible; linkdexbot/2.0; +http://www.linkdex.com/about/bots/)
  RSSMicro.com RSS/Atom Feed Robot
  GarlikCrawler/1.1 (http://garlik.com/, crawler@garik.com)
  al_org_viewer (larbin2.6.3@unspecified.mail)
  Mozilla/5.0 (compatible; JadynAveBot; +http://www.jadynave.com/robot)
  dj-research/Nutch-1.11 (analytics@@demandjump.com)
  Karneval-Bot (Version: 1.06, powered by www.karnevalsuchmaschine.de +http://www.karnevalsuchmaschine.de/bot.html)
  Baiduspider-image+(+http://www.baidu.com/search/spider.htm)\nReferer: http://image.baidu.com/i?ct=503316480&z=0&tn=baiduimagedetail
  SauceNAO/1.0 (+http://saucenao.com/)
  Mozilla/5.0 (compatible; JobdiggerSpider +http://www.jobdigger.nl/spider)
  Yepi/1.0 (NHN Corp.; http://help.naver.com/robots/)
  Mozilla/5.0 (compatible; coccoc/1.0; +http://help.coccoc.com/)
  SSL Labs (https://www.ssllabs.com/about/assessment.html)
  Mozilla/5.0 (compatible; http://alyze.info)
  GigablastOpenSource/1
  Mozilla/5.0 (Windows; U; Windows NT 5.1;fr;rv:1.8.1) VoilaBotCollector BETA 0.1 (http://www.voila.com/)
  Vorboss Web Crawler [crawl@vorboss.net]/Nutch-2.3
  Mozilla/5.0 (compatible; SecretSerachEngineLabs.com-SBSearch/0.9; http://www.secretsearchenginelabs.com/secret-web-crawler.php)
  Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko; Google Search Console) Chrome/27.0.1453 Safari/537.36
  YahooCacheSystem
  search.KumKie.com
  yacybot (webportal-global; x86 Windows Vista 6.0; java 1.7.0_25; Europe/en) http://yacy.net/bot.html
  Embedly +support@embed.ly
  stq_bot (+http://www.searchteq.de)
  MSRBOT
  Mozilla/5.0 (compatible; WBSearchBot/1.1; +http://www.warebay.com/bot.html)
  CorpusCrawler 2.0.17 (http://corpora.fi.muni.cz/crawler/);Project:CzCorpus
  Mozilla/5.0 (compatible; heritrix/3.1.0-RC1 +http://boston.lti.cs.cmu.edu/crawler_12/)
  Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; Trident/5.0) PTST/276
  Mozilla/5.0 (compatible; WebCookies/1.0; +http://webcookies.org/faq/#agent)
  L.webis/0.44 (http://webalgo.iit.cnr.it/index.php?pg=lwebis)
  yacybot (freeworld/global; amd64 Linux 3.0.0-17-generic; java 1.6.0_23; America/en) http://yacy.net/bot.html
  Mozilla/5.0 (Windows NT 6.1; WOW64; rv:46.0) Gecko/20100101 Firefox/46.0 PTST/279
  Mozilla/5.0 (compatible; MSIE or Firefox mutant; not on Windows server;) Daumoa/4.0
  Mozilla/5.0 (compatible; YandexMetrika/2.0; +http://yandex.com/bots mtmon01i.yandex.ru)
  Y!J-BSC/1.0 crawler (http://help.yahoo.co.jp/help/jp/blog-search/)
  Mozilla/5.0 (compatible; UptimeRobot/1.0; http://www.uptimerobot.com/)
  NG-Search/0.86 (+http://www.ng-search.com)
  ichiro/3.0 (http://help.goo.ne.jp/help/article/1142)
  TwengaBot/1.1 (+http://www.twenga.com/bot.html)
  WebImages 0.3 ( http://herbert.groot.jebbink.nl/?app=WebImages )
  Mozilla/5.0 (compatible; aiHitBot/2.8; +http://endb-consolidated.aihit.com/)
  RyzeCrawler/1.1.1 (+http://www.domain2day.nl/crawler/)
  Mozilla/5.0 (Nekstbot; http://www.ipipan.waw.pl/nekst/nekstbot/)
  Mozilla/5.0 (compatible; adidxbot/2.0; http://www.bing.com/bingbot.htm)
  Mozilla/5.0 (compatible; SWEBot/1.0; +http://swebot-crawler.net)
  WatchMouse/8.4.0.3 (http://watchmouse.com/ ; gblon01.watchmouse.net)
  yacybot (/global; amd64 Windows 8.1 6.3; java 1.7.0_55; Europe/de) http://yacy.net/bot.html
  Page Analyzer v4.0 ( http://www.ranks.nl/ )
  web_bh (larbin2.6.3@unspecified.mail)
  findlinks/1.1.6-beta3 (+http://wortschatz.uni-leipzig.de/findlinks/)
  Mozilla/5.0 (compatible; Findxbot/1.0; +http://www.findxbot.com)
  findlinks/2.0 (+http://wortschatz.uni-leipzig.de/findlinks/)
  Mozilla/5.0 (compatible; imagecoccoc/1.0; +http://help.coccoc.com/)
  PagesInventory (robot http://www.pagesinventory.com)
  Mozilla/5.0 (compatible; aiHitBot/1.0-DS; +http://www.aihit.com/)
  tagSeoBot/1.0 (http://www.tagseoblog.de/tools)
  Mozilla/5.0 (en-us) AppleWebKit/537.36 (KHTML, like Gecko; Google PP Default) Chrome/27.0.1453 Safari/537.36
  404 Checker [http://www.404checker.com/user-agent]
  CopperEgg/RevealUptime/
  Mozilla/5.0 (compatible; adidxbot/2.0; +http://www.bing.com/bingbot.htm)
  Mozilla/5.0 (compatible; ZumBot/1.0; http://help.zum.com/inquiry)
  CopperEgg/RevealUptime/TokyoJapan
  Mozilla/5.0 (compatible; MojeekBot/0.2; http://www.mojeek.com/bot.html)
  GoSquared-Status-Checker/0.2
  Mozilla/5.0 (Windows NT 6.0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/99.0 Safari/537.36 YottaaMonitor
  WatchMouse/18990 (http://watchmouse.com/ ; bc.watchmouse.com)
  yacybot (freeworld/global; amd64 Linux 3.2.1-gentoo-r2; java 1.6.0_24; Europe/de) http://yacy.net/bot.html
  Mozilla/5.0 (compatible; adbeat_bot; +support@adbeat.com; support@adbeat.com)
  linkdexbot/Nutch-1.0-dev (http://www.linkdex.com/; crawl at linkdex dot com)
  Heurekabot-Feed/1.0 (+http://sluzby.heureka.cz/napoveda/heurekabot/)
  Mozilla/5.0 (compatible; Charlotte/1.1; http://www.searchme.com/support/)
  yacybot (/global; amd64 Linux 3.10.0-229.7.2.el7.x86_64; java 1.8.0_45; Europe/en) http://yacy.net/bot.html
  LSSRocketCrawler/1.0 LightspeedSystems
  Mozilla/5.0 (X11; U; Linux Core i7-4980HQ; de; rv:32.0; compatible; Jobboerse.com; http://www.xn--jobbrse-d1a.com) Gecko/20100401 Firefox/24.0
  findlinks/2.2 (+http://wortschatz.uni-leipzig.de/findlinks/)
  Mozilla/5.0 (compatible; kulturarw3 +http://www.kb.se/om/projekt/Svenska-webbsidor---Kulturarw3/)
  Mozilla/5.0 (compatible; CloudFlare-AlwaysOnline/1.0; +http://www.cloudflare.com/always-online) AppleWebKit/534.34
  Mozilla/5.0 (compatible; SemrushBot/0.96.4; +http://www.semrush.com/bot.html)
  woobot/2.0
  GarlikCrawler/1.2 (http://garlik.com/, crawler@garlik.com)
  yacybot (webportal-global; amd64 Linux 3.2.0-4-amd64; java 1.7.0_67; Europe/en) http://yacy.net/bot.html
  Mozilla/5.0 (compatible; AboutUsBot Johnny5/2.0; +http://www.AboutUs.org/)
  yacybot (/global; amd64 Linux 3.10.0-327.22.2.el7.x86_64; java 1.7.0_101; Etc/en) http://yacy.net/bot.html
  Mozilla/5.0 (Linux; Android 4.1.2; Galaxy Nexus Build/JZO54K; GTmetrix http://gtmetrix.com/) AppleWebKit/537.22 (KHTML, like Gecko) Chrome/26.0.1410.58 Mobile Safari/537.22
  Nuhk/2.4 ( http://www.neti.ee/cgi-bin/abi/Otsing/Nuhk/)
  Grahambot/0.1 (+http://www.sunaga-lab.com/graham-bot)
  yacybot (/global; amd64 Linux 3.16.0-4-amd64; java 1.7.0_91; Europe/de) http://yacy.net/bot.html
  SEO Consulting; Redirect Checker Tool V.02; IP:
  Mozilla/5.0 (compatible; Pro Sitemaps Generator; https://pro-sitemaps.com) Gecko Pro-Sitemaps/1.0
  yacybot (/global; amd64 Linux 4.4.10-antix.1-amd64-smp; java 1.8.0_101; Europe/en) http://yacy.net/bot.html
  Mozilla/5.0 (Windows NT 6.2; WOW64) Runet-Research-Crawler (itrack.ru/research/cmsrate; rating@itrack.ru)
  Mozilla/5.0 (compatible; Crawler/0.9; http://linkfluence.net/)
  ADmantX Platform Semantic Analyzer US Async - ADmantX Inc. - www.admantx.com - support@admantx.com
  GetProxi.es-bot/1.1 (http://getproxi.es/spiderinfo/)
  Mozilla/5.0 (Windows NT 6.3; WOW64; rv:46.0) Gecko/20100101 Firefox/46.0 PTST/277
  Pinterest/0.2 (+http://www.pinterest.com/)
  CopperEgg/RevealUptime/AtlantaGA(linode)
  OdklBot/1.0 (klass@odnoklassniki.ru)
  Mozilla/5.0 (compatible; Exabot-Images/3.0; +http://www.exabot.com/go/robot)
  yacybot (freeworld/global; amd64 Linux 3.2.1-gentoo-r2; java 1.6.0_22; Europe/de) http://yacy.net/bot.html
  MojeekBot/0.2 (archi; http://www.mojeek.com/bot.html)
  Sogou web spider/4.05252A
  Mozilla/5.0 (Windows NT 6.3; WOW64; rv:45.0) Gecko/20100101 Firefox/45.0 PTST/276
  Mozilla/5.0 (compatible; MotoMinerBot/1.0; +https://motominer.com/Bot)
  NG/2.0
  Mozilla/5.0 (compatible; heritrix/1.14.2 +http://www.webarchiv.cz)
  StackRambler/2.0 (MSIE incompatible)
  Baiduspider+(+http://www.baidu.jp/spider/)
  yacybot (freeworld/global; amd64 Windows 7 6.1; java 1.6.0_25; Europe/de) http://yacy.net/bot.html
  Mozilla/5.0 (compatible; houzzbot; +http://www.houzz.com/)
  Woko robot 3.0
  Mozilla/5.0 (compatible; Qwantify/2.0; +https://www.qwant.com/)
  yacybot (/global; amd64 Linux 4.2.0-27-generic; java 1.8.0_66-internal; America/en) http://yacy.net/bot.html
  ADmantX Platform Semantic Analyzer - APAC - ADmantX Inc. - www.admantx.com - support@admantx.com
  Mozilla/5.0 (compatible; Uptimebot/0.2.40; +http://www.uptime.com/uptimebot)
  Mozilla/5.0 (compatible; ExpertSearchSpider +http://www.expertsearch.nl/spider)
  Mozilla/5.0 (compatible; coccocbot-web/1.0; +http://help.coccoc.com/searchengine)
  Acoon v4.10.4 (www.acoon.de)
  Mozilla/5.0 (compatible; memoryBot/1.20.210 +http://internetmemory.org/en/)
  Readability/740ec9 - http://readability.com/about/
  Mozilla/5.0 (compatible; Apercite; +http://www.apercite.fr/robot/index.html)
  yacybot (i386 Linux 2.6.28-gentoo-r5; java 1.5.0_18; Europe/en) http://yacy.net/bot.html
  Mozilla/5.0 (compatible; suggybot v0.01a, http://blog.suggy.com/was-ist-suggy/suggy-webcrawler/)
  yacybot (amd64 Windows 7 6.1; java 1.6.0_14; Europe/de) http://yacy.net/bot.html
  yacybot (freeworld/global; amd64 Linux 3.3.4-1-ARCH; java 1.6.0_24; Europe/en) http://yacy.net/bot.html
  Mozilla/5.0 (compatible; AportWorm/3.2; +http://www.aport.ru/help)
  Mozilla/5.0 (compatible; memoryBot/1.20.235 +http://internetmemory.org/en/)
  findlinks/2.6 (+http://wortschatz.uni-leipzig.de/findlinks/)
  Mozilla/5.0 (compatible; Hailoobot/1.2; +http://www.hailoo.com/spider.html)
  eCommerceBot (http://www.ehandel.se/botinfo.html)
  Mozilla/5.0(compatible;Sosospider/2.0;+http://help.soso.com/webspider.htm)
  yacybot (/global; amd64 Linux 4.3.0-gentoo-ARCH; java 1.7.0_85; Europe/en) http://yacy.net/bot.html
  Nymesis/2.0 (http://nymesis.com)
  CopperEgg/RevealUptime/OregonUSA
  uclassify.com/1.0
  Mozilla/5.0 (compatible; Butterfly/1.0; +http://labs.topsy.com/butterfly.html) Gecko/2009032608 Firefox/3.0.8
  Mozilla/5.0 (compatible; Prlog/1.0; +http://prlog.ru/)
  Slack-ImgProxy 1.106 (+https://api.slack.com/robots)
  AdnormCrawler www.adnorm.com/crawler
  Mozilla/5.0 (compatible; YandexZakladki/3.0; +http://yandex.com/bots)
  Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/49.0.2623.75 Safari/537.36 Google Favicon
  Mozilla/5.0 (compatible; Sonic/1.0; http://www.yama.info.waseda.ac.jp/~crawler/info.html)
  Mozilla/5.0 (iPhone; U; CPU iPhone OS 4_1 like Mac OS X; en-us) AppleWebKit/532.9 (KHTML, like Gecko) Version/4.0.5 Mobile/8B117 Safari/6531.22.7 (compatible; Googlebot-Mobile/2.1; +http://www.google.com/bot.html)
  ICC-Crawler/2.0 (Mozilla-compatible; ; http://kc.nict.go.jp/project1/crawl.html)
  Mozilla/4.0 (xcm@huaweisymantec.com)
  bot-pge.chlooe.com/1.0.0 (+http://www.chlooe.com/)
  Mozilla/5.0 (compatible; GroupHigh/1.0; +http://www.grouphigh.com/)
  Mozilla/5.0 (compatible; Webmaster tools +http://sitexy.com/)
  yacybot (/global; amd64 Windows 8.1 6.3; java 1.8.0_40; Europe/de) http://yacy.net/bot.html
  Mozilla/5.0 (compatible; MJ12bot/v1.4.1; http://www.majestic12.co.uk/bot.php?+)
  Mozilla/5.0 (compatible; spbot/4.0.6; +http://www.seoprofiler.com/bot )
  Mozilla/5.0 (compatible; EuripBot/2.0; +http://www.eurip.com)
  findlinks/2.1 (+http://wortschatz.uni-leipzig.de/findlinks/)
  Sogou web spider/4.025251
  SETOOZBOT/5.0 ( http://www.setooz.com/bot.html )
  Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0) (larbin@unspecified.mail)
  Mozilla/5.0 (compatible; spbot/2.0.4; +http://www.seoprofiler.com/bot )
  Mozilla/5.0 (TweetmemeBot/4.0; +http://datasift.com/bot.html) Gecko/20100101 Firefox/31.0
  mozilla/5.0 (larbin2.6.3@unspecified.mail)
  Mozilla/5.0 (compatible; spbot/2.1; +http://www.seoprofiler.com/bot )
  Mozilla/5.0 (Windows NT 6.2; WOW64) AppleWebKit/537.4 (KHTML, like Gecko) Chrome/98 Safari/537.4 (StatusCake SSL Monitor)
  Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; Trident/5.0; PTST 2.385)
  Mozilla/5.0 (compatible; evc-batch/2.0.20160608212921)
  Mozilla/5.0 (compatible; Mail.RU_Bot/2.0)
  seebot/2.0 (+http://www.seegnify.com/bot)
  bl.uk_lddc_bot/3.3.0-LBS-2016-02 (+http://www.bl.uk/aboutus/legaldeposit/websites/websites/faqswebmaster/index.html)
  CommaFeed/2.3.0-SNAPSHOT (https://www.commafeed.com)
  OmniExplorer_Bot/5.91c (+http://www.omni-explorer.com) WorldIndexer
  hledejLevne.cz/2.0
  page_verifier (http://www.securecomputing.com/goto/pv)
  url_test (larbin2.6.3@unspecified.mail)
  Mozilla/5.0 (X11; Linux x86_64; rv:10.0.12) Gecko/20100101 Firefox/21.0 WordPress.com mShots
  Mozilla/5.0 (compatible; Linux x86_64; Mail.RU_Bot/2.0; +http://go.mail.ru/help/robots)
  Is is up? (+http://isitup.org)
  Metaspinner/0.01 (Metaspinner; http://www.meta-spinner.de/; support@meta-spinner.de/)
  TwengaBot-2.0 Champigny (+http://www.twenga.com/bot.html)
  LivelapBot/0.2 (http://site.livelap.com/crawler)
  HubSpot Crawler 1.0 http://www.hubspot.com/
  Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/45.0.2454.101 Safari/537.36 TinEye/1.0 (via http://www.tineye.com/)
  yacybot (/global; x86 Windows 10 10.0; java 1.8.0_73; Europe/de) http://yacy.net/bot.html
  Scrapy/0.24.6 (+http://scrapy.org)
  FAST-WebCrawler/3.6/FirstPage (atw-crawler at fast dot no;http://fast.no/support/crawler.asp)
  Baiduspider+(+http://www.baidu.com/search/spider_jp.html)
  Mozilla/5.0 (compatible; seplinkbot/1.0 )
  Mozilla/5.0 (compatible; Falconsbot; +http://ws.nju.edu.cn/falcons/)
  Mozilla/5.0 (Windows NT 6.1; WOW64; rv:46.0) Gecko/20100101 Firefox/46.0 PTST/277
  Mozilla/4.0 (Toread-Crawler/1.1; +http://news.toread.cc/crawler.php)
  TinEye-bot/0.02 (see http://www.tineye.com/crawler.html)
  yacybot (freeworld/global; i386 Linux 2.6.32-39-generic-pae; java 1.6.0_20; Europe/en) http://yacy.net/bot.html
  Mozilla/5.0 (compatible; parsijoo; +http://www.parsijoo.ir/; ehsan.mousakazemi@gmail.com)
  Mozilla/5.0 (compatible; spbot/4.4.0; +http://OpenLinkProfiler.org/bot )
  MaxPoint Bot (+http://www.maxpoint.com)
  Mozilla/5.0 (compatible; Infohelfer/1.2.0; +http://www.infohelfer.de/crawler.php)
  ExB Language Crawler 2.1.1 (+http://www.exb.de/crawler)
  cg-eye interactive
  ZumBot/1.0 (ZUM Search; http://help.zum.com/inquiry)
  rogerbot/1.1 (http://moz.com/help/guides/search-overview/crawl-diagnostics#more-help, rogerbot-crawler+pr2-crawler-05@moz.com)
  Mozilla/5.0 (compatible; Crawlera/1.10.2; UID 47129)
  Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:14.0) Gecko/20100101 Firefox/14.0.1 AppEngine-Google; (+http://code.google.com/appengine; appid: s~metacdn-hr)
  Mozilla/5.0 (compatible; WoTBoT; +https://www.wslta.com/WoTBoT.html)
  HolmesBot (http://holmes.ge)
  Baiduspider-image+(+http://www.baidu.com/search/spider.htm)
  Mozilla/5.0 (compatible; alexa site audit/1.0; +http://www.alexa.com/help/webmasters; no-reply@alexa.com)
  Mediapartners-Google
  Mozilla/5.0 (compatible; MFGPagesBot/2.1; http://www.mfgpages.com)
  larbin_2.6.2 kalou@kalou.net
  Mozilla/5.0 (Windows; U; Windows NT 6.0; en-GB; rv:1.0; trendictionbot0.5.0; trendiction search; http://www.trendiction.de/bot; please let us know of any problems; web at trendiction.com) Gecko/20071127 Firefox/3.0.0.11
  BlogPulseLive (support@blogpulse.com)
  WeSEE:Search/0.1 (Alpha, http://www.wesee.com/en/support/bot/)
  yacybot (freeworld/global; i386 Linux 3.0.0-17-generic; java 1.6.0_23; America/en) http://yacy.net/bot.html
  ImplisenseBot 1.0
  Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; WOW64; Trident/5.0; PTST 2.386)
  gonzo2[P] +http://www.suchen.de/faq.html
  Mozilla/5.0 (compatible; LXRbot/1.0; http://lxrseo.com/, support@lxrseo.com)
  Mozilla/5.0 (compatible; Arachnophilia/1.0; +http://arachnys.com/)
  Mozilla/5.0 (compatible; CloudServerMarketSpider/1.0; +http://cloudservermarket.com/spider.html)
  kalooga/KaloogaBot (Kalooga; http://www.kalooga.com/info.html?page=crawler)
  yacybot (webportal/global; x86_64 Mac OS X 10.9.2; java 1.6.0_65; Europe/de) http://yacy.net/bot.html
  Mozilla/5.0 (compatible; DotBot/1.1; http://www.dotnetdotcom.org/, crawler@dotnetdotcom.org)
  yacybot (/global; amd64 Linux 4.1.19-gentoo; java 1.7.0_95; Europe/pl) http://yacy.net/bot.html
  Mozilla/5.0 (Windows NT 6.3; WOW64; Trident/7.0; rv:11.0) like Gecko PTST/276
  Mozilla/5.0 (compatible; RankActiveLinkBot; +https://rankactive.com/resources/rankactive-linkbot)
  audisto.com full crawler 3.26.431 (refer to in robots.txt as audisto, see https://audisto.com/bot)
  Mozilla/5.0 (compatible; spbot/4.0.1; +http://www.seoprofiler.com/bot )
  Mozilla/5.0 (compatible; OpenindexShallowSpider/Nutch-1.5-dev; +http://www.openindex.io/en/webmasters/spider.html)
  mozilla/5.0 (compatible; discobot/1.1; +http://discoveryengine.com/discobot.html)
  gonzo2[p] (+http://www.suchen.de/faq.html)
  Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; Trident/5.0); 360Spider(compatible; HaosouSpider; http://www.haosou.com/help/help_3_2.html)
  Mozilla/5.0 (compatible; spbot/2.0.1; +http://www.seoprofiler.com/bot/ )
  WatchMouse/18990 (http://watchmouse.com/ ; uk)
  yacybot (freeworld/global; amd64 Linux 3.1.10-hardened; java 1.7.0_03-icedtea; Europe/en) http://yacy.net/bot.html
  VeBot (+http://www.veinteractive.com/vebot)
  Mozilla/5.0 (compatible; NLNZ_IAHarvester2013 +http://natlib.govt.nz/about-us/current-initiatives/web-harvest-2012)
  findlinks/1.1.3-beta8 (+http://wortschatz.uni-leipzig.de/findlinks/)
  Mozilla/5.0 (Linux; U; Android 2.3.4; generic) AppleWebKit/537.36 (KHTML, like Gecko; Google Web Preview) Version/4.0 Mobile Safari/537.36
  DialogSearch.com Bot 1.4;http://dialogsearch.com/webmasters
  Mozilla/5.0 (compatible; GurujiBot/1.0; +http://www.guruji.com/en/WebmasterFAQ.html)
  Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9) AppleWebKit/537.71 (KHTML, like Gecko) Version/7.0 Safari/537.71 (Rival IQ, rivaliq.com)
  wscheck.com/1.0.0 (+http://wscheck.com/)
  Mozilla/5.0 (Windows NT 6.3; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/30.0.1599.69 Safari/537.36 Webthumb/2.0
  Mozilla/5.0 (compatible; FlipboardProxy/1.1; +http://flipboard.com/browserproxy)
  WeSEE:Ads/PictureBot (http://www.wesee.com/bot/)
  Mozilla/5.0 (compatible; Dataprovider/6.101; +https://www.dataprovider.com/)
  Kyoto-Crawler/n1.0 (Mozilla-compatible; kyoto-crawler-contact@nlp.ist.i.kyoto-u.ac.jp; http://nlp.ist.i.kyoto-u.ac.jp/?crawling)
  Mozilla/5.0 (compatible; SEOkicks-Robot; +http://www.seokicks.de/robot.html)
  Mozilla/5.0 (compatible; spbot/4.0.9; +http://OpenLinkProfiler.org/bot )
  Mozilla/5.0 (compatible; spbot/4.2.0; +http://OpenLinkProfiler.org/bot )
  seo-nastroj.cz
  LoadImpactPageAnalyzer/1.3.0 (Load Impact; http://loadimpact.com/)
  CSS Certificate Spider (http://www.css-security.com/certificatespider/)
  MetaGeneratorCrawler/1.3.2 (www.metagenerator.info)
  Testomatobot/1.0 (Linux x86_64; +http://www.testomato.com/testomatobot) minicrawler/4.0.0~beta8
  BLEXBot
  Mozilla/2.0 (compatible; Ask Jeeves/Teoma)
  Testomatobot/1.0 (Linux x86_64; +http://www.testomato.com/testomatobot) minicrawler/4.0.0~beta7
  Mozilla/5.0 (compatible; XoviBot/2.0; +http://www.xovibot.net/)
  Mozilla/5.0 (compatible; spbot/4.4.1; +http://OpenLinkProfiler.org/bot )
  ia_archiver (+http://www.alexa.com/site/help/webmasters; crawler@alexa.com)
  HeartRails Robot/0.1 (http://www.heartrails.com)
  Mozilla/5.0 (compatible; Faveeo/1.0; +http://www.faveeo.com)
  yacybot (/global; amd64 Linux 3.16.0-49-generic; java 1.7.0_79; Europe/en) http://yacy.net/bot.html
  Castabot/0.1 (+http://topixtream.com/)
  Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.9.0.5) Gecko/2010033101 Gentoo Firefox/3.0.5 (Dot TK - spider 3.0)
  Mozilla/5.0 (compatible; MSIE or Firefox mutant; not on Windows server; +http://ws.daum.net/aboutWebSearch.html) Daumoa/2.0
  istellabot-nutch/Nutch-1.10
  Mail.RU/2.0
  ichiro/2.0 (http://help.goo.ne.jp/door/crawler.html)
  LexxeBot/1.0 (lexxebot@lexxe.com)
  Mozilla/5.0 (iPhone; CPU iPhone OS 8_1 like Mac OS X) AppleWebKit/600.1.4 (KHTML, like Gecko) Version/8.0 Mobile/12B410 Safari/600.1.4 (Applebot/0.1; +http://www.apple.com/go/applebot)
  Mozilla/5.0 (compatible; FlipboardRSS/1.1; +http://flipboard.com/browserproxy)
  Mozilla/5.0 (compatible; Linux x86_64; Mail.RU_Bot/Robots/2.0; +http://go.mail.ru/help/robots)
  yacybot (-global; amd64 Linux 3.10.0-229.4.2.el7.x86_64; java 1.7.0_79; Europe/en) http://yacy.net/bot.html
  it2media-domain-crawler/1.0 on crawler-prod.it2media.de
  yacybot (/global; amd64 Windows 8.1 6.3; java 1.8.0_25; Europe/de) http://yacy.net/bot.html
  yacybot (freeworld-global; amd64 Linux 3.16.0-4-amd64; java 1.7.0_79; Europe/de) http://yacy.net/bot.html
  Mozilla/5.0 (WhatsMyIP.org HTTP_Headers) http://whatsmyip.org/ua
  checkgzipcompression.com robot
  Mozilla/5.0 eCairn-Grabber/1.0 (+http://ecairn.com/grabber)
  RankurBot/3.3 (+http://rankur.com)
  L.webis/0.50 (http://webalgo.iit.cnr.it/index.php?pg=lwebis)
  Speedy Spider (Submit your site at http://www.entireweb.com/free_submission/)
  oBot
  Snappy/2.0 ( http://www.urltrends.com/ )
  Mozilla/5.0 (compatible; MSIE 10.0; Windows NT 6.2; WOW64; Trident/6.0) CrawlerProcess (http://www.PowerMapper.com) /5.23.770.0
  Mozilla/5.0 (compatible; alexa site audit/1.0; +http://www.alexa.com/help/webmasters; )
  flatlandbot/baypup (Flatland Industries Web Spider; http://www.flatlandindustries.com/flatlandbot; jason@flatlandindustries.com)
  istellabot/Nutch-1.11
  GetintentCrawler getintent.com
  Covario-IDS/1.0 (Covario; http://www.covario.com/ids; support at covario dot com)
  Mozilla/5.0 (compatible; FatBot 2.0; http://www.thefind.com/crawler)
  Mozilla/5.0 (compatible; MegaIndex.com/2.0; +http://megaindex.com/crawler)
  Microsearch.ru/1.0; http://microsearch.ru/webmasters
  TurnitinBot/3.0 (http://www.turnitin.com/robot/crawlerinfo.html)
  BacklinkCrawler V (http://www.backlinktest.com/crawler.html)
  FeedlyBot/1.0 (http://feedly.com)
  Clickagy Intelligence Bot v2
  GetURLInfo/1.0
  DoCoMo/2.0 P900i(c100;TB;W24H11)(compatible; ichiro/mobile goo;+http://help.goo.ne.jp/door/crawler.html)
  Scrapy/1.1.0 (+http://scrapy.org)
  yacybot (/global; amd64 Linux 3.16.0-4-amd64; java 1.7.0_75; America/en) http://yacy.net/bot.html
  Mozilla/5.0 (compatible; VSAgent/1.2)
  Experibot_v1 [goo.gl/n6zrAf]
  Mozilla/5.0 (compatible; DNS-Digger-Explorer/1.0; +http://www.dnsdigger.com)
  boitho.com-robot/1.0
  yacybot (/global; amd64 Windows Server 2012 6.2; java 1.7.0_51; Europe/de) http://yacy.net/bot.html
  RelateIQ Crawler www.relateiq.com
  Mozilla/4.0 (compatible; Netcraft Web Server Survey)
  Mozilla/5.0 (X11; U; Linux Core i7-4980HQ; de; rv:32.0; compatible; JobboerseBot; https://www.jobboerse.com/bot.htm) Gecko/20100101 Firefox/38.0
  yacybot (i386 Linux 2.6.9-023stab046.2-smp; java 1.6.0_05; Europe/en) http://yacy.net/bot.html
  Scrubby/2.2 (http://www.scrubtheweb.com/)
  Mozilla/5.0 (Yahoo-MMCrawler/4.0; mailto:vertical-crawl-support@yahoo-inc.com)
  Fetch/2.0a (CMS Detection/Web/SEO analysis tool, see http://guess.scritch.org)
  bitlybot/3.0 (+http://bit.ly/)
  Mozilla/5.0 (compatible; heritrix/1.14.4 +http://www.exif-search.com)
  yacybot (freeworld/global; amd64 Windows Server 2012 6.2; java 1.7.0_25; Europe/de) http://yacy.net/bot.html
  agentslug.com - website monitoring tool
  SafeDNSBot (https://www.safedns.com/searchbot)
  Feedbin
  Mozilla/5.0 (compatible; proximic; +http://www.proximic.com)
  Mozilla/5.0 (compatible; AcoonBot/4.11.1; +http://www.acoon.de/robot.asp)
  Mozilla/5.0 (compatible; MagiBot/3.4.3; +http://magi.peak-labs.com/robots.txt)
  Mozilla/5.0 (compatible; XML Sitemaps Generator; http://www.xml-sitemaps.com) Gecko XML-Sitemaps/1.0
  baypup/colbert (Baypup; http://sf.baypup.com/webmasters; jason@baypup.com)
  MergeFlow-PageReader/0.91;+(+http://mergeflow.net/info/pagereader) Mozilla/5.0 (Windows) compatible
  Mozilla/5.0 (Mobile; rv:18.0) Gecko/18.0 Firefox/18.0 commoncrawl.org/research//Nutch-1.7-SNAPSHOT
  Mozilla/5.0 (compatible; spbot/4.1.0; +http://OpenLinkProfiler.org/bot )
  Mozilla/5.0 (compatible; OsO; http://oso.octopodus.com/abot.html)
  Mozilla/5.0 (compatible; gofind; +http://govid.mobi/bot.php)
  CatchBot/2.0; +http://www.catchbot.com
  Mozilla/5.0 (compatible; BusinessSeek.biz_Spider; http://www.businessseek.biz/)
  Quora Link Preview/1.0 (http://www.quora.com)
  radian6_default_(www.radian6.com/crawler)
  DWDS-Crawler +http://odo.dwds.de/dwds-crawler.html
  Mozilla/5.0 (compatible; DuckDuckGo-Favicons-Bot/1.0; +http://duckduckgo.com)
  sogou spider
  Mozilla/5.0 (compatible; GigablastOpenSource/1.0)
  Mozilla/5.0 (compatible; HomeTags/1.0; +http://www.hometags.nl/bot)
  CorpusCrawler 2.0.24 (http://corpora.fi.muni.cz/crawler/);Project:CzCorpus
  Scooter/3.3
  Zookabot/2.5;++http://zookabot.com
  Mozilla/5.0 (iPhone; CPU iPhone OS 8_1 like Mac OS X) AppleWebKit/600.1.4 (KHTML, like Gecko) Version/8.0 Mobile/12B411 Safari/600.1.4 (compatible; YandexMobileBot/3.0; +http://yandex.com/bots)
  Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/534.34 (KHTML, like Gecko) Safari/534.34; +http://sniptracker.com
  Mozilla/5.0 (compatible; archive.org_bot; Wayback Machine Live Record; +http://archive.org/details/archive.org_bot)
  es_com_viewer (larbin2.6.3@unspecified.mail)
  Mozilla/5.0 (compatible; Uptimebot/0.2.14; +http://www.uptime.com/uptimebot)
  yacybot (freeworld/global; amd64 Linux 3.0.0-14-generic; java 1.6.0_23; Europe/en) http://yacy.net/bot.html
  Mozilla/5.0 (X11; Linux x86_64; rv:35.0) Gecko/20100101 Firefox/35.0 DareBoost
  yacybot (freeworld/global; amd64 Linux 2.6.32-41-server; java 1.6.0_26; Europe/de) http://yacy.net/bot.html
  yacybot (amd64 Linux 2.6.28-18-generic; java 1.6.0_16; GMT/en) http://yacy.net/bot.html
  Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.1; Trident/4.0; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; Media Center PC 6.0; .NET4.0C; .NET4.0E; PTST 2.386)
  KD Bot
  Mozilla/5.0 (X11; U; Linux i686 (x86_64); en-US; rv:1.9.2.19) Gecko WebThumb/1.0
  Mozilla/5.0 (compatible; Yahoo! Slurp/3.0; http://help.yahoo.com/help/us/ysearch/slurp)
  rogerbot/1.0 (http://moz.com/help/pro/what-is-rogerbot-, rogerbot-crawler+partager@moz.com)
  MJ12bot/v1.2.0 (http://majestic12.co.uk/bot.php?+)
  Superarama.com-Tarama-Botu-v.01
  Mozilla/5.0 (compatible; Urlfilebot/2.2; +http://urlfile.com/bot.html)
  Vegi bot (we follow your robots.txt settings before crawling, you can slow down the bot by change the Crawl-Delay parameter in the settings.if you have an enquiry, please email to: abuse-report@terrykyleseoagency.com)
  MnoGoSearch/3.3.9
  Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/534+ (KHTML, like Gecko) BingPreview/1.0b
  JyxobotRSS/0.06
  Mozilla/5.0 (compatible; BigBozz/2.2.1; +http://www.bigbozz.com/)
  Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/534.57.2 (KHTML, like Gecko) Version/5.1.7 Safari/534.57.2 PTST/277
  KiNShooboT (compatible; KiNShooboT/1.0.C; +http://www.kinshoo.com/bot.html)
  SentiBot www.sentibot.eu (compatible with Googlebot)
  Topicbot/1.0 (Mozilla;I;+http://92.42.190.57/)
  IDG/IT (http://spaziodati.eu/)
  LinkAider (http://linkaider.com/crawler/)
  Mozilla/5.0 (compatible; coccoc/1.0; +http://help.coccoc.com/searchengine)
  yacybot (/global; amd64 Linux 2.6.32-573.3.1.el6.x86_64; java 1.7.0_85; Europe/en) http://yacy.net/bot.html
  bitlybot
  Mozilla/5.0 (compatible; Mp3Bot/0.7; +http://mp3realm.org/mp3bot/)
  WWW::LayeredExtractor::Handler::Feed/0.01
  Mozilla/5.0 (compatible; SWEBot/1.0; +http://swebot.net)
  Mozzila/5.0 (compatible; Sonic/1.0; http://www.yama.info.waseda.ac.jp/~crawler/info.html)
  MetaTagRobot/2.1 (http://www.seocentro.com/tools/search-engines/metatag-analyzer.html)
  Mozilla/5.0 (compatible; Goodzer/1.0)
  yacybot (/global; amd64 Linux 3.16.0-4-amd64; java 1.8.0_40; Europe/en) http://yacy.net/bot.html
  Mozilla/5.0 (Windows; U; Windows NT 6.1; en-GB; rv:1.9.2.3) Gecko/20100401 Firefox/3.6.3 (NetShelter ContentScan)
  Facebot/1.0
  SafeAds.xyz bot
  Mozilla/5.0 (compatible; Yeti/1.1; +http://help.naver.com/robots/)
  yacybot (/global; amd64 Windows 7 6.1; java 1.7.0_55; Europe/en) http://yacy.net/bot.html
  scrapyproject (+http://www.profound.net)
  Mozilla/5.0 (compatible; Online Domain Tools - Online Website Link Checker/1.2; +http://website-link-checker.online-domain-tools.com)
  Mozilla/5.0 (compatible; heritrix/2.0.2 +http://aihit.com)
  Mozilla/5.0 (compatible; Scrubby/3.1; +http://www.scrubtheweb.com/help/technology.html)
  Semantifire1/0.20 ( http://www.setooz.com/oozbot.html ; agentname at setooz dot_com )
  Mozilla/5.0 (X11; Ubuntu; Linux i686; rv:14.0; ips-agent) Gecko/20100101 Firefox/14.0.1
  UnisterBot (Mozilla/5.0 compatible; crawler@unister-gmbh.de)
  Mozilla/5.0 (compatible; Ezooms/1.0; help@moz.com)
  Mozilla/5.0 (compatible; YandexVideo/3.0; +http://yandex.com/bots)
  Mozilla/5.0 (compatible; spbot/4.4.2; +http://OpenLinkProfiler.org/bot )
  findlinks/2.1.3 (+http://wortschatz.uni-leipzig.de/findlinks/)
  Mozilla/5.0 (compatible; MJ12bot/v1.2.5; http://www.majestic12.co.uk/bot.php?+)
  Speedy Spider (Entireweb; Beta/1.2; http://www.entireweb.com/about/search_tech/speedyspider/)
  Mozilla/5.0 (compatible; kazbtbot/0.1; +http://kazbt.com/)
  Orgbybot/OrgbyBot v1.3 (Spider; http://orgby.com/bot/ ; Orgby.com Search Engine)
  Mozilla/5.0 (compatible; aiHitBot-DM/2.0.2 +http://www.aihit.com)
  YahooSeeker-Testing/v3.9 (compatible; Mozilla 4.0; MSIE 5.5; http://search.yahoo.com/)
  Mozilla/5.0 (compatible; Crawlera/1.10.2; UID 40409)
  Mozilla/5.0 (compatible; image.coccoc/1.0; +http://help.coccoc.com/)
  yacybot (/global; amd64 Linux 3.19.2-1-ARCH; java 1.8.0_40; Europe/de) http://yacy.net/bot.html
  Mozilla/5.0 (compatible; evc-batch/2.0.20161017175820)
  datagnionbot (+http://www.datagnion.com/bot.html)
  CopperEgg/RevealUptime/TokyoJP(linode)
  MB-SiteCrawler
  WatchMouse/18990 (http://watchmouse.com/ ; liz)
  HybridBot (hybrid.ru/about. If our bot caused problems please contact us. Contact email: m.lyashkov@targetix.net)
  Mozilla/5.0 (compatible; Crawlera/1.10.2; UID 24522)
  yacybot (/global; amd64 Linux 3.13.0-74-generic; java 1.7.0_91; Europe/en) http://yacy.net/bot.html
  findlinks/1.1.6-beta5 (+http://wortschatz.uni-leipzig.de/findlinks/)
  yacybot (freeworld/global; amd64 Linux 3.13.0-24-generic; java 1.7.0_55; Europe/de) http://yacy.net/bot.html
  Mozilla/5.0 (compatible; OpenfosBot/2.4; +http://www.openfos.com)
  Nusearch Spider (www.nusearch.com)
  WatchMouse/8.4.0.3 (http://watchmouse.com/ ; sesto02.watchmouse.net)
  ICC-Crawler/2.0 (Mozilla-compatible; ; http://www.nict.go.jp/en/univ-com/plan/crawl.html)
  Mozilla/5.0 (compatible; spbot/1.2; +http://www.seoprofiler.com/bot/ )
  Orbiter/1.3 (http://dailyorbit.com/)
  CCBot/2.0
  Mozilla/5.0 (compatible; linkdexbot/2.1; +http://www.linkdex.com/about/bots/)
  Mozilla/5.0 (compatible; DNS-Digger/1.0; +http://www.dnsdigger.com)
  AppEngine-Google; (+http://code.google.com/appengine; appid: s~feedly-social)
  yacybot (/global; x86 Windows 7 6.1; java 1.8.0_71; Europe/de) http://yacy.net/bot.html
  Mozilla/5.0 (Windows Phone 8.1; ARM; Trident/7.0; Touch; rv:11.0; IEMobile/11.0; NOKIA; Lumia 530) like Gecko (compatible; adidxbot/2.0; +http://www.bing.com/bingbot.htm)
  Dex Social Bot
  yacybot (/global; amd64 Linux 4.7.6-200.fc24.x86_64; java 1.8.0_102; Etc/en) http://yacy.net/bot.html
  Mozilla/5.0 (compatible; heritrix/1.14.3 +http://archive.org)
  CRAZYWEBCRAWLER 0.9.10, http://www.crazywebcrawler.com
  Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/50.0.2661.102 Safari/537.36 PTST/277
  Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.1.3; ips-agent) Gecko/20090824 Fedora/1.0.7-1.1.fc4 Firefox/3.5.3
  SafeSearch microdata crawler (https://safesearch.avira.com, safesearch-abuse@avira.com)
  Zemanta Aggregator/0.9 +http://www.zemanta.com
  Mozilla/5.0 (compatible; special_archiver/3.2.0 +http://www.loc.gov/webarchiving/notice_to_webmasters.html)
  Mozilla/5.0 (compatible; GimmeUSAbot/1.0; +https://gimmeusa.com/crawler.html)
  yacybot (freeworld/global; amd64 Windows 7 6.1; java 1.7.0_45; Europe/de) http://yacy.net/bot.html
  Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.1 (KHTML, like Gecko) Chrome/21.0.1180.89 Safari/537.1; 360Spider
  FyberSpider/1.3 (http://www.fybersearch.com/fyberspider.php)
  yacybot (/global; amd64 FreeBSD 10.3-RELEASE; java 1.8.0_77; GMT/en) http://yacy.net/bot.html
  Searchie/1.0 (a Storm-based crawler; https://www.searchie.org; admin@searchie.org)
  yacybot (freeworld/global; amd64 Linux 3.1.10-1-desktop; java 1.6.0_22; Europe/de) http://yacy.net/bot.html
  holmes/3.12.4 (http://morfeo.centrum.cz/bot)
  WatchMouse/18990 (http://watchmouse.com/ ; se.watchmouse.com)
  MXT/Nutch-1.10 (http://t.co/GSRLLKex24; informatique at mixdata dot com)
  Mozilla/5.0 (compatible; Shareaholicbot/1.0; +http://www.shareaholic.com/bot)
  ZoomInformation Bot
  Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 5.1; www.alertra.com)
  Mozilla/5.0 (compatible; HomeTags/1.0; http://www.hometags.nl/bot)
  Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.2.17) Gecko/20110515 HeartRails_Capture/1.0.4 (+http://capture.heartrails.com/) Namoroka/3.6.17
  OmniExplorer_Bot/5.20 (+http://www.omni-explorer.com) WorldIndexer
  WatchMouse/18990 (http://watchmouse.com/ ; it)
  Mozilla/5.0 (compatible; imrbot/1.10.8 +http://www.mignify.com)
  Mozilla/5.0 (compatible; spbot/4.0.2; +http://www.seoprofiler.com/bot )
  Readability/1900e6 - http://readability.com/about/
  Mozilla/5.0 (compatible; Dataprovider/6.92; +https://www.dataprovider.com/)
  Mozilla/5.0 (iPhone; CPU iPhone OS 6_0_1 like Mac OS X) AppleWebKit/537.36 (KHTML, like Gecko; Google Page Speed Insights) Version/6.0 Mobile/10A525 Safari/8536.25
  VegeBot (we follow your robots.txt settings before crawling, you can slow down the bot by change the Crawl-Delay parameter in the settings.if you have an enquiry, please email to: abuse-report@terrykyleseoagency.com)
  yacybot (webportal-global; amd64 Linux 3.2.0-4-amd64; java 1.7.0_65; Europe/en) http://yacy.net/bot.html
  FeedBucket/1.0 (+http://www.feedbucket.com)
  topster.de HTTP-Header 1.0
  MaxPointCrawler/Nutch-1.10 (maxpoint.crawler at maxpointinteractive dot com)
  Mozilla/5.0 (Windows NT 6.0; rv:45.0) Gecko/20100101 Firefox/45.0 PTST/276
  yacybot (/global; amd64 Windows 7 6.1; java 1.8.0_05; Europe/es) http://yacy.net/bot.html
  NETCRAFT
  Mozilla/5.0 (compatible; WbSrch/1.0; +https://wbsrch.com)
  WebCookies/1.0 (+http://webcookies.info/faq/#agent)
  netEstate Impressumscrawler (+http://www.netestate.de/De/Loesungen/Impressumscrawler)
  msnbot-media/2.0b (+http://search.msn.com/msnbot.htm)
  CopperEgg/RevealUptime/DublinIEUSA
  StormCrawler/1.0 (a Storm-based crawler; https://github.com/DigitalPebble/storm-crawler; stormcrawler@digitalpebble.com)
  WatchMouse/8.4.0.3 (http://watchmouse.com/ ; uschi02.watchmouse.net)
  Mozilla/5.0 (Linux; Android 5.0.2; SM-G920T Build/LRX22G) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/40.0.2125.111 Mobile Safari/537.36 DareBoost
  Mozilla/5.0 (compatible; DeuSu/0.1.0; +https://deusu.org)
  Mozilla/5.0 (compatible; MJ12bot/v1.4.4 (domain ownership verifier); http://www.majestic12.co.uk/bot.php?+)
  Mozilla/5.0 (compatible; SemrushBot/0.99~bl; +http://www.semrush.com/bot.html)
  WebWatch/Robot_txtChecker
  Mozilla/5.0 (compatible; linkdexbot/2.2; +http://www.linkdex.com/bots/)
  Feedly/1.0 (+http://www.feedly.com/fetcher.html; like FeedFetcher-Google)
  Toweyabot: toweya.com
  Mozilla/5.0 (compatible; Infohelfer/1.4.3; +http://www.infohelfer.de/crawler.php)
  Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.2.1; aggregator:Spinn3r (Spinn3r 3.1); http://spinn3r.com/robot) Gecko/20021130
  FeedCatBot/3.0 (+http://www.feedcat.net/)
  LinkedInBot/1.0 (compatible; Mozilla/5.0; Apache-HttpClient +http://www.linkedin.com), libot/Nutch-1.9 (http://www.linkedin.com; libot@linkedin.com)
  R6_CommentReader(www.radian6.com/crawler)
  Mozilla/5.0 (compatible; Crawlera/1.10.2; UID 70350)
  Mozilla/5.0 (compatible; MJ12bot/v1.2.3; http://www.majestic12.co.uk/bot.php?+)
  Domain Re-Animator Bot (http://domainreanimator.com) - support@domainreanimator.com
  Riddler (http://riddler.io/about.html)
  Mozilla/5.0 (compatible; Esribot/1.0; http://www.esrihu.hu/)
  Favicon downloader (http://favicon.netk6.com/)
  findlinks/1.1.5-beta7 (+http://wortschatz.uni-leipzig.de/findlinks/)
  ConveraCrawler/0.9d (+http://www.authoritativeweb.com/crawl)
  Mozilla/5.0 (Windows NT 6.1; rv:6.0) Gecko/20110814 Firefox/6.0 Google favicon
  GetFoundBot (+http://www.getfound.cz/getfoundbot/)
  WatchMouse/8.4.0.3 (http://watchmouse.com/ ; usdal02.watchmouse.net)
  Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/48.0.2564.97 Safari/537.36 Yandex.Translate
  Mozilla/5.0 (compatible; NLNZ_IAHarvester2013 +http://natlib.govt.nz/about-us/current-initiatives/web-harvest-2013)
  COMODOSpider/Nutch-1.2
  Mozilla/5.0 (compatible; DomainMacroCrawler/0.1; +http://domainmacro.com)
  Mozilla/5.0 (compatible; heritrix/3.2.0 +http://www.exif-search.com)
  Mozilla/5.0 (compatible; MagiBot/3.6.2; +http://magi.peak-labs.com/robots.txt)
  Mozilla/5.0 (compatible; Alexabot/1.0; +http://www.alexa.com/help/certifyscan; certifyscan@alexa.com)
  psbot-image (+http://www.picsearch.com/bot.html)
  Mozilla/5.0 (Macintosh; Intel Mac OS X 10.9; rv:28.0) Gecko/20100101 Firefox/28.0 (FlipboardProxy/1.6; +http://flipboard.com/browserproxy)
  larbin_2.6.4 (atyzos@yahoo.com)
  yacybot (/global; amd64 Windows 7 6.1; java 1.7.0_55; Europe/ru) http://yacy.net/bot.html
  BacklinkCrawler (http://www.backlinktest.com/crawler.html)
  Mozilla/5.0 (Windows NT 6.1; WOW64) adbeat.com/policy AppleWebKit/537.36 (KHTML, like Gecko) Chrome/48.0.2564.116 Safari/537.36
  Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/534.57.2 (KHTML, like Gecko) Version/5.1.7 Safari/534.57.2 PTST/276
  Tools4noobs.com/1.0 Spider
  yacybot (freeworld/global; amd64 Linux 2.6.32-40-generic; java 1.6.0_20; Europe/de) http://yacy.net/bot.html
  Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US) Speedy Spider (http://www.entireweb.com/about/search_tech/speedy_spider/)
  istellabot/t.1
  Mozilla/5.0 (compatible; Lipperhey-Kaus-Australis/5.0; +https://www.lipperhey.com/en/about/)
  Mozilla/5.0 (compatible; Veoozbot/1.0; +http://www.veooz.com/veoozbot.html)
  SemrushBot/Nutch-1.5-SNAPSHOT
  Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)
  Mozilla/5.0 (X11; Linux x86_64; rv:47.0; GTmetrix https://gtmetrix.com/) Gecko/20100101 Firefox/47.0
  CopperEgg/RevealUptime/Oregon(aws)
  yacybot (/global; amd64 Linux 3.16.0-4-amd64; java 1.8.0_102; Europe/en) http://yacy.net/bot.html
  rogerbot/1.0 (http://www.seomoz.org/dp/rogerbot, rogerbot-crawler@seomoz.org)
  yacybot (amd64 Linux 2.6.26-2-amd64; java 1.6.0_20; Europe/en) http://yacy.net/bot.html
  Mozilla/5.0 (Windows NT 6.3; WOW64; Trident/7.0; rv:11.0; BingPreview/1.0b) like Gecko
  Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 5.1; Trident/4.0; .NET CLR 1.1.4322; .NET CLR 2.0.50727; .NET CLR 3.0.04506.30; .NET CLR 3.0.4506.2152; .NET CLR 3.5.30729; MDDS; PTST 2.386)
  Mozilla/5.0 (compatible; RavenCrawler/2.0; +https://raventools.com/seo-website-auditor/)
  Mozilla/5.0 (compatible; YandexMetrika/2.0; +http://yandex.com/bots)
  Mozilla/5.0 (compatible; Scrubby/3.2; +http://seotools.scrubtheweb.com/webpage-analyzer.html)
  dubaiindex (adressendeutschland.de)
  Mozilla/5.0 (compatible; spbot/5.0.2; +http://OpenLinkProfiler.org/bot )
  Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; Netcraft SSL Server Survey - contact info@netcraft.com)
  Mozilla/5.0 (compatible; CompSpyBot/1.0; +http://www.compspy.com/spider.html)
  Mozilla/5.0 (compatible; LA2; +http://www.zeerch.com/zeerch2/bot.php)
  yacybot (/global; amd64 Linux 2.6.32-042stab108.8; java 1.7.0_91; America/en) http://yacy.net/bot.html
  updated/0.1-alpha (updated crawler; http://www.updated.com; crawler@updated.com)
  CopperEgg/RevealUptime/FrankfurtGermany
  Scrapy/1.1.2 (+http://scrapy.org)
  yrspider (Mozilla/5.0 (compatible; YRSpider; +http://www.yunrang.com/yrspider.html))
  Mozilla/5.0 (compatible; YandexAntivirus/2.0; +http://yandex.com/bots)
  yacybot (freeworld/global; amd64 Windows 7 6.1; java 1.6.0_31; Europe/de) http://yacy.net/bot.html
  Surphace Scout&v4.0 - scout at surphace dot com
  yacybot (freeworld/global; amd64 Linux 2.6.32-5-xen-amd64; java 1.6.0_18; Europe/fr) http://yacy.net/bot.html
  MetaGeneratorCrawler/1.1 (www.metagenerator.info)
  Porkbun/Mustache (Website Analysis; http://porkbun.com; tech@porkbun.com)
  Mozilla/5.0 (compatible; ScoutJet; +http://www.scoutjet.com/)
  Ruby, link_thumbnailer
  yacybot (freeworld/global; amd64 Windows Server 2008 R2 6.1; java 1.7.0_25; Europe/de) http://yacy.net/bot.html
  yacybot (/global; amd64 Linux 3.13.0-042stab093.4; java 1.7.0_79; Europe/de) http://yacy.net/bot.html
  Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.11 (KHTML, like Gecko) Chrome/23.0.1271.64 Safari/537.11 GotSiteMonitor.com
  Mozilla/5.0 (Linux; Android 4.4; Nexus 5 Build/KRT16M) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/40.0.2125.111 Mobile Safari/537.36 DareBoost
  Mozilla/5.0 (KeepRight OpenStreetMap Checker; http://keepright.at) Gecko/20100101 Firefox/22.0
  Mozilla/5.0 (Windows NT 6.1; WOW64; Trident/7.0; rv:11.0; topster.de Linkchecker 5.0) like Gecko (194.228.205.74)
  Mozilla/5.0 AppleWebkit/537.36 (KHTML, like Gecko) Trident/7.0; rv:11.0, Chrome/42.0.2311.135, Edge/12.10136, http://www.shrinktheweb.com/, Webshot/0.9
  topicbot/1.0 (Mozilla;I;+http://www.topic.bot/contact_page.html)
  DataparkSearch/4.53 (+http://dataparksearch.org/bot)
  Mozilla/5.0 (compatible; Linux; InfegyAtlas/1.0; en-US; collection@infegy.com)
  Mozilla/5.0 (X11; Linux x86_64; rv:41.0; GTmetrix https://gtmetrix.com/) Gecko/20100101 Firefox/41.0
  Mozilla/5.0 (compatible; EasouSpider; +http://www.easou.com/search/spider.html)
  Mozilla/5.0 (compatible; parsijoo-update-crawler; +http://www.parsijoo.ir/; ehsanmousa@parsijoo.ir)
  Abrave Spider v4 Robot 1 (http://robot.abrave.co.uk)
  Mozilla/5.0 (compatible; memoryBot/1.21.24 +http://internetmemory.org/en/)
  Mozilla/5.0 (compatible; STINGbot/1.0; +http://136.186.231.16)
  domainsbot (+http://www.domainsbot.com)
  WEPA/3.1 (http://www.wepa.com/; webmaster@wepa.com)
  Mozilla/5.0 (compatible; linkdexbot/2.1; +http://www.linkdex.com/bots/)
  Mozilla/5.0 (compatible; forensiq; +http://www.forensiq.com)
  192.comAgent
  JUST-CRAWLER(+http://www.justsystems.com/jp/tech/crawler/)
  Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm) SitemapProbe
  HuaweiSymantecSpider/1.0+DSE-support@huaweisymantec.com+(compatible; MSIE 7.0; Windows NT 5.1; Trident/4.0; .NET CLR 2.0.50727; .NET CLR 3.0.4506.2152; .NET CLR ; http://www.huaweisymantec.com/en/IRL/spider)
  Mozilla/5.0 ( Macintosh; Intel Mac OS X 10_10_1 ) AppleWebKit/600.2.5 ( KHTML, like Gecko ) Version/8.0.2 Safari/600.2.5 ( compatible; CloudServerMarketSpider/1.0; +http://cloudservermarket.com/spider.html )
  Mozilla/5.0 (compatible; MJ12bot/v1.3.0; http://www.majestic12.co.uk/bot.php?+)
  Mozilla/5.0 (compatible; AcoonBot/4.10.6; +http://www.acoon.de/robot.asp)
  Scrapy/1.0.1 (+http://scrapy.org)
  seegnifybot/1.0.0 (http://www.seegnify.com/bot)
  omgili/0.5 +http://omgili.com
  Slack-ImgProxy 149 (+https://api.slack.com/robots)
  yacybot (/global; amd64 Linux 3.16.0-4-amd64; java 1.7.0_79; Europe/de) http://yacy.net/bot.html
  www.integromedb.org/Crawler
  Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm
  Mozilla/5.0 (compatible) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/47.0.2526.73 Safari/537.36 collection@infegy.com
  Mozilla/5.0 (compatible; IstellaBot/1.18.81 +http://www.tiscali.it/)
  Gigabot/1.0
  page_test (larbin2.6.3@unspecified.mail)
  istellabot/Nutch-1.10
  Mozilla/5.0 (iPhone; U; CPU iPhone OS 4_1 like Mac OS X; en-us) AppleWebKit/532.9 (KHTML, like Gecko) Version/4.0.5 Mobile/8B117 Safari/6531.22.7 (compatible; Mediapartners-Google/2.1; +http://www.google.com/bot.html)
  ldspider (http://code.google.com/p/ldspider/wiki/Robots)
  yacybot (/global; amd64 Windows 8.1 6.3; java 1.8.0_45; Europe/ru) http://yacy.net/bot.html
  facebookplatform/1.0 (+http://developers.facebook.com)
  Mozilla/5.0 (compatible; RTGI; http://linkfluence.net/)
  yacybot (freeworld/global; amd64 Linux 3.12.43-52.6-default; java 1.8.0_40; Europe/en) http://yacy.net/bot.html
  Mozilla/5.0 (compatible; YandexBot/3.0; MirrorDetector; +http://yandex.com/bots)
  Mozilla/5.0 (compatible; DCPbot/1.2; +http://domains.checkparams.com/)
  WatchMouse/18990 (http://watchmouse.com/ ; d2.watchmouse.com)
  MnoGoSearch/3.3.15
  Mozilla/5.0 (compatible; AboutUsBot/0.9; +http://www.aboutus.org/AboutUsBot)
  Web-sniffer/1.1.0 (+http://web-sniffer.net/)
  Mozilla/4.0 (compatible; Fooooo_Web_Video_Crawl http://fooooo.com/bot.html)
  50.nu/0.01 ( +http://50.nu/bot.html )
  Mozilla/4.0 (compatible; MSIE 7.0; Windows; Windows NT 5.1) BrokenLinkCheck.com/1.1
  Comodo-Certificates-Spider
  Iframely/0.9.8 (+https://iframely.com/;)
  MnoGoSearch/3.3.12
  CopperEgg/RevealUptime/FrankfurtEU
  Python-urllib/2.7 (+http://ella.juls.savba.sk/aranea_about)
  Mozilla/5.0 (compatible; JobKereso; +http://www.kozvetlen-allasok.hu/robot.jsp info@kozvetlen-allasok.hu)
  Mozilla/5.0 (Windows NT 5.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/49.0.2623.112 Safari/537.36 PTST/201
  Mozilla/5.0 (compatible; Google Keyword Tool; +http://adwords.google.com/select/KeywordToolExternal)
  WinWebBot/1.0; (Balaena Ltd, UK); http://www.balaena.com/winwebbot.html; winwebbot@balaena.com;)
  AppleNewsBot
  classbot (+http://allclasses.com)
  yacybot (freeworld/global; amd64 Windows Server 2008 R2 6.1; java 1.6.0_31; America/pt) http://yacy.net/bot.html
  Mozilla/5.0 (iPhone; CPU iPhone OS 7_0 like Mac OS X) AppleWebKit/537.51.1 (KHTML, like Gecko) Version/7.0 Mobile/11A465 Safari/9537.53 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)
  yacybot (/global; arm Linux 3.12.28+; java 1.7.0_71; Europe/en) http://yacy.net/bot.html
  Mozilla/5.0 (compatible; YandexMetrika/2.0; +http://yandex.com/bots mtmon01e.yandex.ru)
  CopperEgg/RevealUptime/TokyoJP
  Mozilla/5.0 (compatible; AhrefsBot/3.1; +http://ahrefs.com/robot/)
  CJNetworkQuality; http://www.cj.com/networkquality
  psbot-page (+http://www.picsearch.com/bot.html)
  DealGates Bot/1.1 by Luc Michalski (http://spider.dealgates.com/bot.html)
  Pingdom GIGRIB (http://www.pingdom.com)
  Mozilla/5.0 (compatible; SEOlyticsCrawler/3.0; +http://crawler.seolytics.net/)
  Mozilla/5.0 (compatible; EventGuruBot/1.0; +http://www.eventguru.com/spider.html)
  Python-urllib/2.7 (+http://aranea.juls.savba.sk/aranea_about.html)
  yacybot (/global; amd64 Windows Vista 6.0; java 1.7.0_55; America/en) http://yacy.net/bot.html
  The Lemur Web Crawler/Nutch-1.3 (Lemur Web Crawler; http://boston.lti.cs.cmu.edu/crawler_12/; admin@lemurproject.org)
  SAMSUNG-SGH-E250/1.0 Profile/MIDP-2.0 Configuration/CLDC-1.1 UP.Browser/6.2.3.3.c.1.101 (GUI) MMP/2.0 (compatible; Googlebot-Mobile/2.1; +http://www.google.com/bot.html)
  Feedfetcher-Nuesbyte; (+http://www.nuesbyte.com)
  copyright sheriff (+http://www.copyrightsheriff.com/)
  yacybot (/global; amd64 Linux 3.2.0-4-amd64; java 1.7.0_75; Europe/en) http://yacy.net/bot.html
  Touche (+http://www.touche.com.ve)
  Panopta v1.1
  Mozilla/5.0 (compatible; parsijoo-bot; +http://www.parsijoo.ir/; ehsan.mousakazemi@gmail.com)
  Zookabot/2.2;++http://zookabot.com
  yacybot (/global; amd64 Linux 2.6.32-042stab093.4; java 1.7.0_65; Etc/en) http://yacy.net/bot.html
  it2media-domain-crawler/2.0
  yacybot (/global; amd64 Linux 4.2.0-27-generic; java 1.7.0_95; Europe/en) http://yacy.net/bot.html
  Mozilla/5.0 (Android; Mobile; rv:21.0) Gecko/21.0 Firefox/21.0 commoncrawl.org/research//Nutch-1.7-SNAPSHOT
  Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 5.2; Trident/4.0; Data Center; .NET CLR 1.1.4322; .NET CLR 2.0.50727; .NET CLR 3.0.04506.648; .NET CLR 3.5.21022; .NET CLR 3.0.4506.2152; .NET CLR 3.5.30729; PTST 2.386)
  Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/99.0 Safari/537.36 YottaaMonitor
  CopperEgg/RevealUptime/AtlantaGAUSA
  Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
  yacybot (/global; amd64 Windows 7 6.1; java 1.7.0_55; Europe/fr) http://yacy.net/bot.html
  Mozilla/5.0 (compatible; MJ12bot/v1.2.4; http://www.majestic12.co.uk/bot.php?+)
  Mozilla/5.0 (compatible; Applebot/0.3; +http://www.apple.com/go/applebot)
  ICC-Crawler/2.0 (Mozilla-compatible; ; http://www.nict.go.jp/en/ucri/plan/crawl.html)
  Mozilla/5.0 (compatible; YYSpider; +http://www.yunyun.com/spider.html)
  Mozilla/5.0 (iPhone; CPU iPhone OS 5_0_1 like Mac OS X) (compatible; Yeti-Mobile/0.1; +http://help.naver.com/robots/)
  Mozilla/5.0 (WhatsMyIP.org Text_to_Code_Ratio_Tool) http://whatsmyip.org/ua
  Thumbnail.CZ robot 1.1 (http://thumbnail.cz/why-no-robots-txt.html)
  Google-Calendar-Importer
  Mozilla/5.0 (Windows NT 6.3; WOW64) AppleWebKit/534.34 (KHTML, like Gecko) Websnapr/3.0 Safari/534.34
  Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/50.0.2661.75 Safari/537.36 PTST/276
  WeSEE
  librabot/2.0 (+http://academic.research.microsoft.com/)
  Testomatobot/1.0 (Linux x86_64; +http://www.testomato.com/testomatobot) minicrawler/3.0.1
  Mozilla/5.0 (compatible; MSIE 10.0; Windows NT 6.1; Trident/6.0) PTST/276
  Cliqzbot/0.1 (+http://cliqz.com +cliqzbot@cliqz.com)
  Scrubby/3.0 (+http://www.scrubtheweb.com/help/technology.html)
  Pinterest/0.1 +http://pinterest.com/
  Mozilla/5.0 (compatible; Crawlera/1.10.2; UID 52327)
  Comodo-Webinspector-Crawler 2.2.0, http://www.comodorobot.com
  Iframely/0.6.0 (+http://iframely.com/;)
  Mozilla/5.0 (compatible; PaperLiBot/2.1; http://support.paper.li/entries/20023257-what-is-paper-li)
  Mozilla/5.0 (compatible; SiteExplorer/1.0b; +http://siteexplorer.info/)
  Y!J-BSC/1.0 (http://help.yahoo.co.jp/help/jp/blog-search/)
  Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/49.0.2623.112 Safari/537.36 PTST/276
  Mozilla/5.0 (compatible; U; Hootsuite-WebFeed/1.0; mailto:abuse@hootsuite.com)
  RankFlex.com Webspider
  TipTop http://feeltiptop.com
  vebidoobot
  Mozilla/5.0 (compatible; Dow Jones Searchbot)
  Mozilla/5.0 (compatible; OrangeBot-Collector/2.0; support.orangebot@orange.com)
  Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/534+ (KHTML, like Gecko) MsnBot-Media /1.0b
  Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1.6) Gecko/20070725 Firefox/2.0.0.6 Ara.com.tr AraBot 1.0
  Mozilla/5.0 (iPhone; CPU iPhone OS 9_1 like Mac OS X) AppleWebKit/601.1.46 (KHTML, like Gecko) Version/9.0 Mobile/13B143 Safari/601.1 (compatible; AdsBot-Google-Mobile; +http://www.google.com/mobile/adsbot.html)
  Mozilla/5.0 (compatible; YandexMetrika/2.0; +http://yandex.com/bots mtmon01g.yandex.ru)
  BUbiNG (+http://law.di.unimi.it/BUbiNG.html)
  downnotifier.com monitoring
  A6-Indexer
  Mozilla/5.0 (compatible; htrix/1.14.4 ++https://webarchiv.onb.ac.at/robot.html)
  MetaURI API/2.0 +metauri.com
  Mozilla/5.0 (compatible; MixrankBot; crawler@mixrank.com)
  Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/44.0.2403.155 Safari/537.36 TinEye/1.0 (via http://www.tineye.com/)
  Mozilla/5.0 Project 25499 (project25499.com)
  COMODOspider/Nutch-1.0
  Mozilla/5.0 (compatible; CareerBot/1.1; +http://www.career-x.de/bot.html)
  Uptime.com_(http://uptime.com/)
  AcoonBot/4.10.5 (+http://www.acoon.de)
  BDFetch
  20090717_aa_page (larbin2.6.3@unspecified.mail)
  Metaspinner/1.0 (Metaspinner Search Engine; http://www.meta-spinner.de/; support@meta-spinner.de)
  Mozilla/5.0 (compatible; DBLBot/1.0; +http://www.dontbuylists.com/)
  Mozilla/5.0 (compatible; spbot/2.0; +http://www.seoprofiler.com/bot/ )
  LYT.SR v1.5 +http://lyt.sr/ (running on http://lyt.sr)
  Putin (+http://semanticjuice.com/)
  Mozilla/5.0 (compatible; LA1; +http://www.zeerch.com/bot.php)
  WatchMouse/18990 (http://watchmouse.com/ ; ny)
  bitlybot/2.0
  yacybot (/global; amd64 Linux 3.13.0-45-generic; java 1.7.0_75; Europe/en) http://yacy.net/bot.html
  Webshot/0.9 ( http://www.shrinktheweb.com/ )
  MetaURI API +metauri.com
  Mozilla/5.0 (compatible; ProductoDownloadUrlBot/1.0; +http://www.producto.de/)
  Google-Adwords-Instant (+http://www.google.com/adsbot.html)
  yacybot (freeworld/global; amd64 Linux 3.8.0-19-generic; java 1.7.0_25; Europe/en) http://yacy.net/bot.html
  yacybot (freeworld/global; i386 Linux 2.6.32-5-686; java 1.6.0_18; Europe/en) http://yacy.net/bot.html
  Mozilla/5.0 (Windows NT 6.3; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/42.0.2311.90 Safari/537.36 PTST/277
  Mozilla/5.0 (compatible; DomainSigmaCrawler/0.1; +http://domainsigma.com/robot)
  HyperZbozi.cz Feeder/3.1
  Mozilla/5.0 (Windows; U; Windows NT 5.1; en; rv:1.9.0.13) Gecko/2009073022 Firefox/3.5.2 (.NET CLR 3.5.30729) SurveyBot/2.3 (DomainTools)
  yacybot (/global; amd64 Windows Server 2012 R2 6.3; java 1.7.0_55; Europe/de) http://yacy.net/bot.html
  Google-AdWords-Express
  Mozilla/5.0 (compatible; MJ12bot/v1.4.0; http://www.majestic12.co.uk/bot.php?+)
  SMTBot (similartech.com/smtbot)
  Mozilla/5.0 (compatible; Lipperhey SEO Service; http://www.lipperhey.com/)
  MetaInspector/4.7.2 (+https://github.com/jaimeiniesta/metainspector)
  Mozilla/5.0 (Compatible; Vedma/0.91Beta; +http://www.vedma.ru/bot.htm)
  Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.21 (KHTML, like Gecko) web-capture.net/0.9 Safari/537.21
  Nokia6680/1.0 (4.04.07) SymbianOS/8.0 Series60/2.6 Profile/MIDP-2.0 Configuration/CLDC-1.1 (botmobi find.mobi/bot.html find@mtld.mobi)
  Mozilla/5.0 (compatible; Exploratodo/1.0; +http://www.exploratodo.com
  iZSearch.com
  Mozilla/5.0 (compatible; YandexFavicons/1.0; +http://yandex.com/bots)
  bl.uk_lddc_bot/3.3.0-SNAPSHOT-2014-10-07T09:33:31Z (+http://www.bl.uk/aboutus/legaldeposit/websites/websites/faqswebmaster/index.html)
  yacybot (/global; amd64 Linux 3.14-0.bpo.1-amd64; java 1.7.0_55; Europe/de) http://yacy.net/bot.html
  HuaweiSymantecSpider/1.0+DSE-support@huaweisymantec.com+(compatible; MSIE 7.0; Windows NT 5.1; Trident/4.0; .NET CLR 2.0.50727; .NET CLR 3.0.4506.2152; .NET CLR ; http://www.huaweisymantec.com/cn/IRL/spider)
  FAST-WebCrawler/3.7/FirstPage (atw-crawler at fast dot no;http://fast.no/support/crawler.asp)
  Mozilla/5.0 (compatible; DripfeedBot/2.0; +http://dripfeedbookmark.com/bot.html
  Mozilla/5.0 (compatible; heritrix/2.0.2 +http://seekda.com)
  yacybot (freeworld/global; i386 Linux 3.2.0-23-generic-pae; java 1.7.0_03; Europe/en) http://yacy.net/bot.html
  hawkReader/0.9b (Feed Parser; http://www.hawkreader.com; Allow like Gecko)
  ruijie (larbin@unspecified.mail)
  CorpusCrawler 2.0.0 (http://corpora.fi.muni.cz/crawler/)
  ExactSeek Crawler (http://www.exactseek.com/)/Nutch-1.4
  WordPress.com; https://botd.wordpress.com
  Mozilla/5.0 (compatible; Qwantify/2.0n; +https://www.qwant.com/)/*
  Mozilla/5.0 (compatible; memoryBot/1.20.268 +http://internetmemory.org/en/)
  ShowyouBot (http://showyou.com/crawler)
  Yoleo Consumer v0.2
  Mozilla/5.0 (compatible; IstellaBot/1.10.2 +http://www.tiscali.it/)
  Mozilla/5.0 (compatible; MSIE or Firefox mutant; not on Windows server; +http://ws.daum.net/aboutWebSearch.html) Daumoa/3.0
  Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/46.0.2490.86 YaBrowser/15.12.0.6151 Safari/537.36 Yandex.Translate
  Mozilla/5.0 (compatible; TeeRaidBot; +https://teeraid.com/bot/)
  Mozilla/5.0 (compatible; Scopia Crawler 1.1; +http://www.scopia.co)
  CopperEgg/RevealUptime/DublinIE(aws)
  yacybot (freeworld/global; amd64 Windows 7 6.1; java 1.7.0_04; Asia/ja) http://yacy.net/bot.html
  RankurBot/Rankur2.1 (http://rankur.com; info at rankur dot com)
  Gaisbot/3.0+(robot06@gais.cs.ccu.edu.tw;+http://gais.cs.ccu.edu.tw/robot.php)
  Mozilla/5.0 (compatible; zitebot support [at] zite [dot] com +http://zite.com)
  MetaTagRobot/2.0 (http://www.check-domains.com/website-analysis/website-analyzer.php)
  Virusdie crawler/2.1
  findlinks/2.0.4 (+http://wortschatz.uni-leipzig.de/findlinks/)
  Mozilla/5.0 (compatible; AcoonBot/4.10.7; +http://www.acoon.de/robot.asp)
  Mozilla/5.0 (compatible; DeuSu/5.0.2; +https://deusu.de/robot.html)
  yacybot (-global; amd64 Linux 3.19.0-15-generic; java 1.8.0_45-internal; Europe/de) http://yacy.net/bot.html
  Google_Analytics_Snippet_Validator
  SurcentroBot
  Mozilla/5.0 (compatible; Diffbot/0.1; +http://www.diffbot.com)
  findlinks/1.1.3-beta9 (+http://wortschatz.uni-leipzig.de/findlinks/)
  Mozilla/5.0 (compatible; memoryBot/1.20.199 +http://internetmemory.org/en/)
  Y!J-ASR/0.1 crawler (http://www.yahoo-help.jp/app/answers/detail/p/595/a_id/42716/)
  TwengaBot-Discover (http://www.twenga.fr/bot-discover.html)
  Mozilla/5.0 (bl.uk_lddc_bot; Linux x86_64) PhantomJS/1.9.7 (+http://www.bl.uk/aboutus/legaldeposit/websites/websites/faqswebmaster/index.html)
  panscient.com
  Mozilla/5.0 (compatible; TestCrawler; +http://wiki.github.com/bixo/bixo/bixocrawler; bixo-dev@yahoogroups.com)
  DataparkSearch/4.37-23012006 ( http://www.dataparksearch.org/)
  GOFORITBOT ( http://www.goforit.com/about/ )
  CopperEgg/RevealUptime/SydneyAustralia
  msnbot-media/1.0 (+http://search.msn.com/msnbot.htm)
  Mozilla/5.0 (compatible; MegaIndex.ru/2.0; +https://www.megaindex.ru/?tab=linkAnalyze)
  MetaCommentBot; http://metacomment.io/about
  Mozilla/5.0 (compatible; YesupBot/1.0; +http://www.yesup.net/bot.html)
  Twikle/1.0 , http://twikle.com , contact@twikle.com
  yacybot (/global; amd64 Windows Server 2008 R2 6.1; java 1.8.0_60; Europe/de) http://yacy.net/bot.html
  Mozilla/4.0 (compatible; Vagabondo/4.0Beta; webcrawler at wise-guys dot nl; http://webagent.wise-guys.nl/; http://www.wise-guys.nl/)
  Comodo-Webinspector-Crawler 2.1
  Mozilla/5.0 (Macintosh; Intel Mac OS X 10_7_5) AppleWebKit/537.11 (KHTML, like Gecko) Chrome/23.0.1271.64 Safari/537.11 KimonoLabs/0.2
  Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1 + FairShare-http://fairshare.cc)
  Scrapy/0.24.5 (+http://scrapy.org)
  Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/534.34 (KHTML, like Gecko) Site-Shot/2.1 (http://www.site-shot.com/) Safari/534.34
  Port Monitor check service 1.0 | http://www.port-monitor.com
  Mozilla/5.0 (compatible; Uptimebot/0.1.43; +http://www.uptime.com/uptimebot)
  yacybot (freeworld/global; amd64 Linux 2.6.32-5-amd64; java 1.6.0_18; Europe/en) http://yacy.net/bot.html
  SolomonoBot/1.04 (http://www.solomono.ru)
  Easy-Thumb (https://www.easy-thumb.net/)
  Mozilla/5.0 (Windows NT 6.0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/49.0.2623.112 Safari/537.36 PTST/276
  nic.it crawler v.1.221115 (7.35.0)
  Mozilla/5.0 (compatible; memorybot/1.24.12 +https://internetmemory.net/en/bots)
  Mozilla/5.0 (compatible; inoreader.com-like FeedFetcher-Google)
  yacybot (/global; amd64 Windows 7 6.1; java 1.7.0_72; Europe/de) http://yacy.net/bot.html
  WatchMouse/8.4.0.3 (http://watchmouse.com/ ; demuc02.watchmouse.net)
  Mozilla/5.0 (X11; Linux x86_64; rv:43.0; GTmetrix https://gtmetrix.com/) Gecko/20100101 Firefox/43.0
  ShopWiki/1.0 ( +http://www.shopwiki.com/wiki/Help:Bot)
  Mozilla/5.0 (compatible; UnisterBot; http://www.bluekiwi.de/misc/imprint)
  Web-sniffer/1.0.31 (+http://web-sniffer.net/)
  woobot
  Mozilla/5.0 (X11; U; Linux Core i7-4980HQ; de; rv:32.0; compatible; JobboerseBot; https://www.jobboerse.com/bot.htm) Gecko/20100401 Firefox/24.0
  AntBot/Ant-Nutch-1.1 (Ant Nutch Crawler; http://www.ant.com; crawler@ant.com)
  Mozilla/5.0 (compatible; YandexMedia/3.0; +http://yandex.com/bots)
  Mozilla/5.0 (compatible; Evrinid Iudex 1.0.0; +http://www.evri.com/evrinid)
  adidxbot/1.1 (+http://search.msn.com/msnbot.htm)
  Mozilla/5.0 (compatible; WebThumbnail/2.2; Website Thumbnail Generator; +http://webthumbnail.org)
  Mozilla/5.0 (compatible; Embedly/0.2; +http://support.embed.ly/)
  EasyBib AutoCite (http://content.easybib.com/autocite/)
  yacybot (-global; amd64 Linux 3.2.0-4-amd64; java 1.7.0_65; Europe/en) http://yacy.net/bot.html
  Mozilla/5.0 (compatible; woriobot +http://worio.com)
  Mozilla/4.0 (compatible; Vagabondo/4.0; http://webagent.wise-guys.nl/)
  SniffRSS/0.5beta (+http://www.blogator.com/)
  woobot/1.1
  Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) WordPress.com mShots Safari/537.36
  ADmantX Platform Semantic Analyzer - Turn - ADmantX Inc. - www.admantx.com - support@admantx.com
  Mozilla/5.0 (compatible; UnisterBot; crawler@unister.de)
  Mozilla/5.0 (compatible; electricmonk/3.2.0 +https://www.duedil.com/our-crawler/)
  UptimeDog Robot (www.uptimedog.com)
  Googlebot/2.1 (+http://www.google.com/bot.html)
  Mozilla/5.0 (Windows NT 6.3; WOW64; rv:36.0) Gecko/20100101 Firefox/36.0 (NetShelter ContentScan, contact abuse@inpwrd.com for information)
  Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.0; Trident/4.0; SLCC1; .NET CLR 2.0.50727; Media Center PC 5.0; .NET CLR 3.5.30729; .NET4.0C; .NET CLR 3.0.30729; .NET4.0E) PTST/276
  ServiceUptime.robot
  Mozilla/4.0 (compatible;HostTracker/2.0;+http://www.host-tracker.com/)
  SEO Browser
  Leikibot/1.0 (+http://www.leiki.com)
  yacybot (webportal-global; amd64 Windows 7 6.1; java 1.7.0_04; America/en) http://yacy.net/bot.html
  AppEngine-Google; (+http://code.google.com/appengine; appid: downforeveryoneorjustme)
  Mozilla/5.0 (Windows NT 5.1) BrokenLinkCheck.com/1.1
  Seobility (SEO-Check; http://bit.ly/1dJuuzs)
  Mozilla/5.0 (seoanalyzer; compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)
  Mozilla/5.0 (compatible; Digincore bot; https://www.digincore.com/crawler.html for rules and instructions.
  Site24x7
  websitepulse checker/3.0 (compatible; MSIE 5.5; Netscape 4.75; Linux)
  Mozilla/5.0 (compatible; SemrushBot-SA/0.97; +http://www.semrush.com/bot.html)
  Mozilla/5.0 (compatible; NerdByNature.Bot; http://www.nerdbynature.net/bot)
  Abrave Spider v4 Robot 2 (http://robot.abrave.co.uk)
  HubPages V0.2.2 (http://hubpages.com/help/crawlingpolicy)
  lmspider (lmspider@scansoft.com)
  magpie-crawler/1.1 (U; Linux amd64; en-GB; +http://www.brandwatch.net)
  Mozilla/5.0 (compatible; WbSrch/1.1 +http://wbsrch.com
  Mozilla/5.0 (Windows; U; Windows NT 5.1; de; rv:1.9.0.7; Google-SearchByImage) Gecko/2009021910 Firefox/3.0.7
  Mozilla/5.0 (compatible; Najdi.si/3.1)
  larbin_2.6.3 (larbin2.6.3@unspecified.mail)
  Mozilla/5.0 (compatible; aiHitBot/1.1; +http://www.aihit.com/)
  Mozilla/5.0 (Windows NT 6.3; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/50.0.2661.102 Safari/537.36 PTST/277
  Mozilla/5.0 (YahooYSMcm/3.0.0; http://help.yahoo.com)
  yacybot (/global; amd64 Linux 3.16.0-4-amd64; java 1.7.0_79; Europe/en) http://yacy.net/bot.html
  OOZBOT/0.20 ( http://www.setooz.com/oozbot.html ; agentname at setooz dot_com )
  findlinks/2.0.9 (+http://wortschatz.uni-leipzig.de/findlinks/)
  Mozilla/5.0 (compatible; Lipperhey Link Explorer; http://www.lipperhey.com/)
  Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.6; en-US; rv:1.9.2) Gecko/20100115 Firefox/3.6 (FlipboardProxy/1.1; +http://flipboard.com/browserproxy)
  Mozilla/5.0 (Twiceler-0.9 http://www.cuil.com/twiceler/robot.html)
  Mozilla/5.0 (compatible; memoryBot/1.24.11 +http://internetmemory.org/en/)
  RED/1 (https://redbot.org/)
  Twitterbot/1.0
  Mozilla/5.0 (compatible; ptd-crawler; +http://bixolabs.com/crawler/ptd/; crawler@bixolabs.com)
  Open Web Analytics Bot 1.5.4
  JikeSpider Mozilla/5.0 (compatible; JikeSpider; +http://shoulu.jike.com/spider.html)
  Mozilla/5.0 (compatible; vkShare; +http://vk.com/dev/Share)
  Mozilla/5.0 (compatible; oBot/2.3.1; +http://www-935.ibm.com/services/us/index.wss/detail/iss/a1029077?cntxt=a1027244)
  elefent/Elefent 1.2 (A friendly web elefent.; http://elefent.eu/; webmaster@elefent.eu)
  Genderanalyzer/1.0
  elefent/1.2 (Web Crawler; https://crawler.elefent.net; webmaster at elefent dot net)
  Yandex/1.01.001 (compatible; Win16; H)
  crawler4j for XQuery
  Mozilla/5.0 (compatible; Plukkie/1.4; http://www.botje.com/plukkie.htm)
  LinguaBot/v0.001-dev (MultiLinual Sarch Engine v0.001; LinguaSeek; admin at linguaseek dot com)
  MrakyPraceCzBot/1.0 (http://www.mrakyprace.cz/about-crawler)
  Y!J-BRJ/YATS crawler (http://help.yahoo.co.jp/help/jp/search/indexing/indexing-15.html)
  WatchMouse/8.4.0.3 (http://watchmouse.com/ ; plgdn01.watchmouse.net)
  Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2272.96 Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
  AskQuickly v2 (http://askquickly.org/)
  Mozilla/5.0 (compatible; SISTRIX Crawler; http://crawler.sistrix.net/)
  Miniflux (http://miniflux.net)
  Mozilla/5.0 (compatible; spbot/5.0.1; +http://OpenLinkProfiler.org/bot )
  Mozilla/5.0 (compatible; wmtips.com/1.0; +http://www.wmtips.com/tools/)
  Mozilla/5.0 (compatible; linkdexbot/2.0; +http://www.linkdex.com/bots/)
  Mozilla/2.0 (compatible; Ask Jeeves/Teoma; +http://sp.ask.com/docs/about/tech_crawling.html)
  Google favicon
  Mozilla/5.0(compatible; Sosospider/2.0; +http://help.soso.com/webspider.htm)
  MiaDev/0.0.1 (MIA Bot for research project MIA (www.MIA-marktplatz.de); http://www.mia-marktplatz.de/spider; spider@mia-marktplatz.de)
  Mozilla/5.0 (compatible; proximic; +http://www.proximic.com/info/spider.php)
  Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/33.0.1750.146 Safari/537.36 Specificfeeds- http://www.specificfeeds.com
  Mozilla/5.0 (compatible; socialbm_bot/1.0; +http://spider.socialbm.net)
  wp.com feedbot/1.0 (+https://wp.com)
  Mozilla/5.0 (compatible; CligooRobot/2.0; +http://www.cligoo.de/wk/technik.php)
  Slack-ImgProxy (+https://api.slack.com/robots)
  Mozilla/5.0 (compatible; Online Domain Tools - HTTP Headers Online/1.0; +http://http-headers.online-domain-tools.com)
  Mozilla/5.0 (compatible; SurdotlyBot/1.0; +http://sur.ly/bot.html)
  WGETbot/1.0 (+http://wget.alanreed.org)
  yacybot (/global; amd64 Linux 4.0.9-2-ck; java 1.8.0_51; Europe/de) http://yacy.net/bot.html
  Y!J-BRI/0.0.1 crawler ( http://help.yahoo.co.jp/help/jp/search/indexing/indexing-15.html )
  dlvr.it/1.0 (+http://dlvr.it/) Mozilla/5.0
  yacybot (/global; arm Linux 4.1.13-v7+; java 1.8.0; Etc/de) http://yacy.net/bot.html
  Mozilla/5.0 (compatible; GimmeUSAbot/1.0; +http://gimmeusa.com/crawler.html)
  yacybot (freeworld/global; amd64 Windows 7 6.1; java 1.6.0_29; Europe/fr) http://yacy.net/bot.html
  Link Valet Online 1.2
  Mozilla/5.0 (compatible; METASpider; +http://meta.ua/spider)
  NeutrinoAPI/2.0.1
  thumbshots-de-bot (+http://www.thumbshots.de/)
  Mozilla/5.0 (iPhone; CPU iPhone OS 6_0 like Mac OS X) AppleWebKit/536.26 (KHTML, like Gecko) Version/6.0 Mobile/10A5376e Safari/8536.25 (compatible; Applebot/0.3; +http://www.apple.com/go/applebot)
  argenfybot
  MergadoBot
  Ant/Ant-Nutch-1.1 (Ant Nutch Crawler; http://www.ant.com; crawler@ant.com)
  CopperEgg/RevealUptime/SaoPauloBrazil
  Mozilla/4.0 (compatible; Vagabondo/4.0/EU; http://webagent.wise-guys.nl/)
  Mozilla/4.0 (compatible; DAWINCI ANTIPLAG SPIDER)
  Mozilla/5.0 (Windows NT 6.1; WOW64; rv:45.0) Gecko/20100101 Firefox/45.0 PTST/276
  HetrixTools.COM Uptime Monitoring Bot. https://hetrixtools.com/uptime-monitoring-bot.html
  yacybot (/global; amd64 Linux 3.2.0-4-amd64; java 1.7.0_79; Europe/en) http://yacy.net/bot.html
  Mozilla/5.0 (Windows NT 6.1) (compatible; SMTBot/1.0; +http://www.similartech.com/smtbot)
  Mozilla/5.0 (compatible; InbyBot/2.1; +http://inbyapp.com)
  all_web1 (larbin2.6.3@unspecified.mail)
  Mozilla/5.0 (compatible; BusinessSeek.biz_Directory; http://www.businessseek.biz/)
  Wotbox/2.0 (bot@wotbox.com; http://www.wotbox.com)
  Mediatoolkitbot (info@mediatoolkit.com)
  OpenWebSpider v0.1.2.B (http://www.openwebspider.org/)
  Mozilla/5.0 (compatible; WebCorp/5.0; +http://www.webcorp.org.uk)
  Testomatobot/1.0 (Linux x86_64; +http://www.testomato.com/testomatobot) minicrawler/4.0.0
  fastbot crawler beta 2.0 (+http://www.fastbot.de)
  Sogou Web Spider
  DoCoMo/2.0 P900i(c100;TB;W24H11) (compatible; ichiro/mobile goo;+http://search.goo.ne.jp/option/use/sub4/sub4-1/)
  Mozilla/5.0 (compatible; imagecoccoc/1.0; +http://help.coccoc.com/searchengine)
  XoviBot/1.0
  Mozilla/5.0 (compatible; Online Domain Tools - Online Website Link Checker/1.1; +http://website-link-checker.online-domain-tools.com)
  WatchMouse/8.4.0.3 (http://watchmouse.com/ ; decgn02.watchmouse.net)
  yacybot (i386 Linux 2.6.31-21-generic; java 1.6.0_0; Europe/en) http://yacy.net/bot.html
  ZnajdzFoto/ImageBot 2.0b
  Mozilla/5.0 (compatible; heritrix/3.0.0-SNAPSHOT-20091120.021634 +http://crawler.archive.org)
  Mozilla/5.0 (compatible; ProCogSEOBot/1.0; +http://www.procog.com/ )
  Mozilla/5.0 (compatible; Plukkie/1.6; http://www.botje.com/plukkie.htm)
  yacybot (freeworld/global; amd64 Linux 3.2.0-21-generic; java 1.7.0_03-icedtea; America/en) http://yacy.net/bot.html
  Mozilla/5.0 (iPhone; CPU iPhone OS 8_3 like Mac OS X) AppleWebKit/600.1.4 (KHTML, like Gecko) Version/8.0 Mobile/12F70 Safari/600.1.4 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
  Mozilla/5.0 (compatible; MSIE or Firefox mutant; not on Windows server; + http://tab.search.daum.net/aboutWebSearch.html) Daumoa/3.0
  Mozilla/5.0 (compatible; NLNZ_IAHarvester2014 +http://natlib.govt.nz/about-us/current-initiatives/web-harvest-2014)
  Mozilla/5.0 (compatible; archive.org_bot +http://archive.org/details/archive.org_bot)
  Mozilla/5.0 (compatible; special_archiver/3.3.0 +http://www.loc.gov/webarchiving/notice_to_webmasters.html)
  yacybot (freeworld/global; i386 Linux 3.4.2-linode44; java 1.6.0_27; America/en) http://yacy.net/bot.html
  Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/47.0.2526.111 Safari/537.36 Yandex.Translate
  yacybot (webportal-global; amd64 Linux 3.2.0-4-amd64; java 1.7.0_25; Europe/en) http://yacy.net/bot.html
  FreeWebMonitoring SiteChecker/0.1 (+http://www.freewebmonitoring.com)
  Mozilla/5.0 (compatible; Infohelfer/1.3.0; +http://www.infohelfer.de/crawler.php)
  Kemvibot/1.0 (http://kemvi.com)
  Mozilla/4.0 (compatible; HostTracker.com/1.0;+http://host-tracker.com/)
  AdsBot-Google-Mobile (+http://www.google.com/mobile/adsbot.html) Mozilla (iPhone; U; CPU iPhone OS 3 0 like Mac OS X) AppleWebKit (KHTML, like Gecko) Mobile Safari
  Mozilla/5.0 (compatible; GeliyooBot/1.0; +http://www.geliyoo.com/)
  yacybot (x86 Windows XP 5.1; java 1.6.0_21; Europe/de) http://yacy.net/bot.html
  Mozilla/5.0 (compatible; archive.org_bot/3.1.2 +http://natlib.govt.nz/about-us/current-initiatives/web-harvest-2012)
  Mozilla/5.0 (compatible; StorygizeBot; http://www.storygize.com)
  infohelfer/1.1.0 (http://www.infohelfer.de/)
  yacybot (/global; amd64 Linux 3.16.0-41-generic; java 1.7.0_80; Europe/de) http://yacy.net/bot.html
  SeoCheckBot (FischerNetzDesign Seo Checker, info@fischernetzdesign.de)
  netEstate NE Crawler (+http://www.website-datenbank.de/)
  Mail.Ru/1.0
  binlar_2.6.3 (larbin2.6.3@unspecified.mail)
  LinkedInBot/1.0 (compatible; Mozilla/5.0; Jakarta Commons-HttpClient/3.1 +http://www.linkedin.com)
  bot.wsowner.com/1.0.0 (+http://wsowner.com/)
  Ronzoobot/1.5 (http://www.ronzoo.com/about/)
  Mozilla/5.0 (compatible; heritrix/3.2.0 +http://www.mixdata.com)
  Mozilla/5.0 (compatible; Yahoo! Slurp China; http://misc.yahoo.com.cn/help.html)
  findlinks/1.1.6-beta6 (+http://wortschatz.uni-leipzig.de/findlinks/)
  Mozilla/5.0 (compatible; LoadTimeBot/0.7; +http://www.load-time.com/bot.html)
  Mozilla/5.0 (compatible; MJ12bot/v1.4.2; http://www.majestic12.co.uk/bot.php?+)
  Mozilla/4.0 (compatible; MSIE 5.01; GomezAgent 2.0; Windows NT)
  Mozilla/5.0 (compatible; MJ12bot/v1.3.2; http://www.majestic12.co.uk/bot.php?+)
  SEOENGBot/1.2 (+http://learn.seoeng.com/seoengbot.htm)
  yacybot (i386 Linux 2.6.31-18-generic; java 1.6.0_0; Europe/en) http://yacy.net/bot.html
  Online Virus Scanner: http://tools.geek-tools.org
  DwnldBot (+http://dwnld.me)
  Mozilla/5.0 (compatible; OpenfosBot/2.1; http://www.openfos.com)
  taptubot *** please read http://www.taptu.com/corp/taptubot ***
  yacybot (freeworld/global; amd64 Linux 3.9.3-1-ARCH; java 1.7.0_21; America/en) http://yacy.net/bot.html
  Testomatobot/1.0 (Linux x86_64; +http://www.testomato.com/testomatobot) minicrawler/4.0.0~beta10
  rogerbot/1.0 (http://www.moz.com/dp/rogerbot, rogerbot-crawler@moz.com)
  Mozilla/5.0 (Windows NT 6.1; Win64; x64) KomodiaBot/1.0
  Mozilla/5.0 (compatible; spbot/3.0; +http://www.seoprofiler.com/bot )
  Mozilla/5.0 (compatible; DomainTunoCrawler/0.1; +https://www.domaintuno.com/robot)
  Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Googlebot/2.1; +http://www.google.com/bot.html) Safari/537.36
  yacybot (amd64 Linux 2.6.32-gentoo; java 1.6.0_17; Europe/en) http://yacy.net/bot.html
  Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko; Google Page Speed Insights) Chrome/27.0.1453 Mobile Safari/537.36
  Mozilla/5.0 (compatible; GimmeUSAbot/1.0; +http://gimmeusa-update.com/crawler)
  Cloud mapping experiment. Contact research@pdrlabs.net
  emefgebot/beta (+http://emefge.de/bot.html)
  DoCoMo/2.0 D903i(c100;TB;W28H20) (compatible; Hatena-Mobile-Gateway/1.2; +http://mgw.hatena.ne.jp/help)
  ichiro/3.0 (http://help.goo.ne.jp/door/crawler.html)
  Mozilla/5.0 (compatible; SecurityResearch.bot; +http://besome1.info/securityresearchbot.html)
  WPScan v2.9 (http://wpscan.org)
  Ronzoobot/1.6 (http://www.ronzoo.com/about/)
  livedoor ScreenShot/0.10
  Mozilla/5.0 (compatible; MSIE 8.0; Windows NT 5.1) KomodiaBot/1.0
  Mozilla/5.0 (compatible; memoryBot/1.20.289 +http://internetmemory.org/en/)
  Mozilla/5.0 (compatible; DomainTunoCrawler/0.1; +http://www.domaintuno.com/robot)
  yacybot (freeworld/global; amd64 Linux 2.6.26-2-amd64; java 1.6.0_18; Europe/de) http://yacy.net/bot.html
  Mozilla/5.0 (compatible; GimmeUSAbot/1.0; +http://gimmeusa.com/crawler)
  Mozilla/5.0 (compatible; spbot/5.0; +http://OpenLinkProfiler.org/bot )
  Mozilla/5.0 (compatible; Webauskunft.org; http://www.webauskunft.org)
  CCResearchBot/1.0 commoncrawl.org/research//Nutch-1.7-SNAPSHOT
  Mozilla/4.0 (compatible; MSIE 5.5; Windows NT 4.0; obot)
  linkdex.com/v2.0
  Mozilla/5.0 (compatible; Crawlera/1.10.2; UID 73393)
  Mozilla/5.0 (compatible; SEODat/0.1 http://crawler.seodat.com)
  yacybot (freeworld/global; amd64 Linux 3.2.0-4-amd64; java 1.6.0_27; Europe/en) http://yacy.net/bot.html
  CorpusCrawler 2.0.12 (http://corpora.fi.muni.cz/crawler/)
  Mozilla/5.0 (compatible; spbot/4.3.0; +http://OpenLinkProfiler.org/bot )
  Mozilla/5.0 (compatible; aiHitBot/2.7; +http://www.aihit.com/)
  Mozilla/5.0 (compatible; SiteAnalyzerBot/4.0; +https://www.site-analyzer.com/)
  DataparkSearch/4.40.1 (+http://www.dataparksearch.org/)
  yacybot (freeworld-global; amd64 Linux 3.0.0-21-generic; java 1.6.0_26; America/en) http://yacy.net/bot.html
  findlinks/1.1.6-beta1 (+http://wortschatz.uni-leipzig.de/findlinks/; YaCy 0.1; yacy.net)
  CopperEgg/RevealUptime/SydneyAU(aws)
  OpenAcoon v4.1.0 (www.openacoon.de)
  semanticbot (info@semanticaudience.com)
  Mozilla/5.0 (compatible; Crawlera/1.10.2; UID 75425)
  Mozilla/5.0 (compatible; Crawlera/1.10.2; UID 11912)
  Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp)
  Mozilla/5.0 (Windows NT 5.1; U; Win64; fr; rv:1.8.1) VoilaBot BETA 1.2 (support.voilabot@orange-ftgroup.com)
  yacybot (-global; amd64 Linux 3.13.0-61-generic; java 1.7.0_79; Europe/en) http://yacy.net/bot.html
  CorpusCrawler 2.0.25 (http://corpora.fi.muni.cz/crawler/);Project:CzCorpus
  ichiro/3.0 (http://search.goo.ne.jp/option/use/sub4/sub4-1/)
  R6_FeedFetcher_(www.radian6.com/crawler)
  sosoimagespider+(+http://help.soso.com/soso-image-spider.htm)
  Mozilla/5.0 (X11; Linux x86_64; rv:39.0; GTmetrix http://gtmetrix.com/) Gecko/20100101 Firefox/39.0
  yacybot (freeworld/global; amd64 Linux 3.2.0-24-generic; java 1.6.0_24; Europe/en) http://yacy.net/bot.html
  Mozilla/4.0 (compatible; Vagabondo/4.0; webcrawler at wise-guys dot nl; http://webagent.wise-guys.nl/)
  CommaFeed/2.4.0-SNAPSHOT (https://github.com/Athou/commafeed)
  Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; Google Web Preview Analytics) Chrome/27.0.1453 Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
  yacybot (/global; x86 Windows 7 6.1; java 1.8.0_31; Europe/de) http://yacy.net/bot.html
  Mozilla/5.0 (compatible; woriobot support [at] zite [dot] com +http://zite.com)
  Mozilla/5.0 (compatible; MetaJobBot; http://www.metajob.at/crawler)
  bl.uk_lddc_bot/3.3.0-SNAPSHOT-20140519-0844 (+http://www.bl.uk/aboutus/legaldeposit/websites/websites/faqswebmaster/index.html)
  FFbot/1.0; +http://feedfury.com
  Mozzila/5.0 (compatible; YamanaLab-Robot/1.0; http://www.yama.info.waseda.ac.jp/~crawler/info.html)
  LinqiaRSSBot/1.0 (eng@linqia.com)
  FreeWebMonitoring SiteChecker/0.2 (+http://www.freewebmonitoring.com/bot.html)
  Mozilla/5.0 (compatible; Web-Monitoring/1.0; +http://monoid.nic.ru/)
  Mozilla/5.0 (compatible; heritrix/1.12.1b +http://netarkivet.dk/website/info.html)
  Mozilla/5.0 (compatible; SpiderLing (a SPIDER for LINGustic research); http://nlp.fi.muni.cz/projects/biwec/)
  UrlTrends Ranking Updater/2.0 ( http://www.urltrends.com/ )
  yacybot (/global; amd64 Linux 3.16.1-gentoo; java 1.7.0_55; Europe/en) http://yacy.net/bot.html
  Mozilla/5.0 (compatible; gimme60bot/1.0 ; +http://gimme60.com)
  CatchBot/5.0; +http://www.catchbot.com
  elefent/Elefent 1.1 (Friendly web walker.; http://elefent.eu/; jason@elefent.eu)
  findlinks/2.0.1 (+http://wortschatz.uni-leipzig.de/findlinks/)
  Googlebot-Video/1.0
  Mozilla/5.0 (compatible; XmarksFetch/1.0; +http://www.xmarks.com/about/crawler; info@xmarks.com)
  XRL/3.00 (Linux; i686; en-us) (+http://metamark.net/about)
  Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/50.0.2661.87 Safari/537.36 PTST/276
  CorpusCrawler 2.0.8 (http://corpora.fi.muni.cz/crawler/)
  Mozilla/5.0 (compatible; archive.org_bot +http://www.archive.org/details/archive.org_bot)
  Mozilla/5.0 (compatible; OpenindexDeepSpider/Nutch-1.5-dev; +http://openindex.io/spider.html; systemsATopenindexDOTio)
  Ronzoobot/1.3 (http://www.ronzoo.com/about.php)
  MetaInspector/2.2.1 (+https://github.com/jaimeiniesta/metainspector)
  Mozilla/5.0 (compatible; WASALive-Bot ; http://blog.wasalive.com/wasalive-bots/)
  Mozilla/5.0 (compatible; YioopBot; +http://www.yioop.com/bot.php)
  Mozilla/5.0 (compatible; Mozilla/5.0; +http://wiki.github.com/bixo/bixo/bixocrawler; bixo-dev@yahoogroups.com)
  Mozilla/5.0 (compatible; Kraken/0.1; http://linkfluence.net/; bot@linkfluence.net)
  Mozilla/5.0 (compatible; Qseero; +http://www.q0.com)
  yacybot (-global; amd64 Linux 3.10.0-229.4.2.el7.x86_64; java 1.8.0_45; Europe/en) http://yacy.net/bot.html
  Mozilla/5.0 (compatible; ProCogBot/1.0; +http://www.procog.com/spider.html)
  Mozilla/5.0 (compatible; MJ12bot/v1.4.3; http://www.majestic12.co.uk/bot.php?+)
  Mozilla/5.0 (Anturis Agent)
  Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/534.34 (KHTML, like Gecko) Chromium/31.0.1650.63 Site-Shot/2.1 (http://www.site-shot.com/) Safari/534.34
  webnumbrFetcher/1.0 (http://webnumbr.com/)
  Mozilla/5.0 (compatible; meanpathbot/1.0; +http://www.meanpath.com/meanpathbot.html)
  NIF/1.1 (http://www.newsisfree.com/robot.php users:)
  Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/51.0.2704.103 Safari/537.36 PTST/284
  DoCoMo/2.0 P901i(c100;TB;W24H11) (compatible; ichiro/mobile goo; +http://help.goo.ne.jp/door/crawler.html)
  Iframely/0.8.8 (+https://iframely.com/;)
  yacybot (ppc Mac OS X 10.5.2; java 1.5.0_13; Europe/de) http://yacy.net/bot.html
  Mozilla/5.0 (compatible; UptimeRobot/2.0; http://www.uptimerobot.com/)
  yacybot (/global; amd64 Linux 3.13.0-85-generic; java 1.7.0_95; Europe/en) http://yacy.net/bot.html
  Mozilla/5.0 (compatible; FeedBooster; +http://feeds.qsensei.com)
  yacybot (freeworld/global; amd64 Linux 2.6.32-5-amd64; java 1.6.0_18; Europe/de) http://yacy.net/bot.html
  Mozilla/5.0 (compatible; SputnikFaviconBot/1.2; +http://corp.sputnik.ru/webmaster)
  Mozilla/5.0 (compatible; heritrix/3.1.1; UniLeipzigASV +http://corpora.informatik.uni-leipzig.de/crawler_faq.html)
  Mozilla/5.0 (compatible; OrangeBot/2.0; support.voilabot@orange.com)
  Yeti-FeedItemCrawler/1.0 (NHN Corp.; http://help.naver.com/robots/)
  Mozilla/5.0 (compatible; SeznamBot/3.2-test1-1; +http://napoveda.seznam.cz/en/seznambot-intro/)
  Mozilla/5.0 (compatible; heritrix/3.2.0 +http://www.crim.ca)
  yacybot (/global; amd64 Linux 4.0.5-gentoo; java 1.8.0_51; Europe/en) http://yacy.net/bot.html
  pr-cy.ru Screenshot Bot
  GentleSource Short URL Checker ( http://www.gentlesource.com/ )
  crazywebcrawler 0.9.0, http://www.crazywebcrawler.com
  a.pr-cy.ru
  Mozilla/5.0 (iPhone; CPU iPhone OS 7_0 like Mac OS X) AppleWebKit/537.51.1 (KHTML, like Gecko) Version/7.0 Mobile/11A465 Safari/9537.53 (compatible; bingbot/2.0; http://www.bing.com/bingbot.htm)
  findlinks/1.1.6-beta1 (+http://wortschatz.uni-leipzig.de/findlinks/)
  ichiro/5.0 (http://help.goo.ne.jp/door/crawler.html)
  yacybot (/global; amd64 Windows 7 6.1; java 1.8.0_25; Europe/en) http://yacy.net/bot.html
  Yeti/1.1 (Naver Corp.; http://help.naver.com/robots/)
  Technoratibot/8.0
  Setooz/Nutch-1.0 (http://www.setooz.com)
  Mozilla/5.0 (Windows NT 5.2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/49.0.2623.112 Safari/537.36 PTST/201
  semanticbot
  Mozilla/5.0 (compatible; JikeSpider; +http://shoulu.jike.com/spider.html)
  Mozilla/5.0 (compatible; MetamojiCrawler/1.0; +http://www.metamoji.com/jp/crawler.html
  Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1.6) Gecko/20070725 Firefox/2.0.0.6 - James BOT - WebCrawler http://cognitiveseo.com/bot.html
  Mozilla/5.0 (Windows NT 6.1; compatible; BDCbot/1.0; +http://ecommerce.bigdatacorp.com.br/faq.aspx) ppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2272.118 Safari/537.36
  Mozilla/5.0 (compatible; CloudServerMarketSpider/1.0; +http://www.cloudservermarket.com/spider.html)
  Mozilla/5.0 (Windows NT 6.1; WOW64; rv:38.0; Neustar WPM) Gecko/20100101 Firefox/38.0
  Mozilla/5.0 (compatible;+ParchBot/1.0;++http://www.parchmenthill.com/search.htm)
  Sogou-Test-Spider/4.0 (compatible; MSIE 5.5; Windows 98)
  MetaHeadersBot (+http://www.metaheaders.com/bot.html)
  Mozilla/5.0 (Windows; U; Windows NT 6.0; en-GB; rv:1.0; trendictionbot0.4.2; trendiction media ssppiiddeerr; http://www.trendiction.com/bot/; please let us know of any problems; ssppiiddeerr at trendiction.com) Gecko/20071127 Firefox/2.0.0.11
  voyager/2.0 (http://www.kosmix.com/crawler.html)
  Mozilla/5.0 (compatible; YodaoBot/1.0; http://www.yodao.com/help/webmaster/spider/; )
  hivaBot/hivaBot-1.0 (Iranian Search Engine bot; http://yooz.ir; info@yooz.ir)
  yacybot (freeworld/global; amd64 Linux 3.2.0-4-amd64; java 1.7.0_55; Europe/de) http://yacy.net/bot.html
  AppEngine-Google; (+http://code.google.com/appengine; appid: s~closure-compiler-hrd)
  larbin_2.5.0 (larbin2.5.0@unspecified.mail)
  Scrapy/0.22.2 (+http://scrapy.org)
  Ronzoobot/1.2 (http://www.ronzoo.com/about.php)
  gooblog/2.0 (http://help.goo.ne.jp/contact/)
  Pingdom.com_bot_version_1.4_(http://www.pingdom.com/)
  Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/535.21 (KHTML, like Gecko) Chrome/19.0.1042.0 Safari/535.21 PagePeeker/2.1; +http://pagepeeker.com/robots/
  WeSEE:Ads/PageBot (http://www.wesee.com/bot/)
  yacybot (amd64 Linux 2.6.18-164.el5; java 1.6.0; Europe/de) http://yacy.net/bot.html
  SemanticScholarBot/1.0 (+http://s2.allenai.org/bot.html)
  Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/535. Safari/535.22+; (+http://immediatenet.com/thumbnail_api.html; free thumbnails)
  Mozilla/5.0 (compatible; Crawlera/1.10.2; UID 85747)
  SBIder/Nutch-1.0-dev (http://www.sitesell.com/sbider.html)
  Scrapy/1.0.4 (+http://scrapy.org)
  SanszBot/1.7(http://www.sansz.org/sanszbot, spider@sansz.org) (spider@sansz.org)
  Mozilla/5.0 (compatible; KHTML, like Gecko) Chrome/43.0.2357.81 Safari/537.36 collection@infegy.com
  Mozilla/5.0 (iPhone; U; CPU iPhone OS 4_3 like Mac OS X; en-GB) adbeat.com/policy AppleWebKit/533.17.9 (KHTML, like Gecko) Version/5.0.2 Mobile/8F190 Safari/6533.18.5
  Curious George - www.analyticsseo.com/crawler
  Grammarly/1.0 (http://www.grammarly.com)
  OpenCalaisSemanticProxy
  yacybot (freeworld/global; amd64 Linux 2.6.32-49-server; java 1.6.0_27; Europe/en) http://yacy.net/bot.html
  netEstate RSS crawler (+http://www.rss-directory.info/)
  ipv6-test.com validator
  Mozilla/5.0 (compatible; Qualidator.com Bot 1.0;)
  ichiro/2.0 (ichiro@nttr.co.jp)
  yacybot (freeworld/global; amd64 Linux 2.6.38-14-generic; java 1.6.0_22; Europe/en) http://yacy.net/bot.html
  cg-eye live
  Mozilla/5.0 (compatible; OpenindexSpider; +http://www.openindex.io/en/webmasters/spider.html)
  Mozilla/5.0 (Windows; U; MSIE 9.0; Windows NT 9.0; en-US) AppEngine-Google; (+http://code.google.com/appengine; appid: s~virustotalcloud)
  Mozilla/5.0 (compatible; heritrix/1.14.3.r6601 +http://www.buddybuzz.net/yptrino)
  MetaGeneratorCrawler/1.3.4 (www.metagenerator.info)
  Mozilla/5.0 (Linux; Android 5.0; LG-D855 Build/LRX21R.A1422018487) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/40.0.2214.109 Mobile Safari/537.36 DareBoost
  Mozilla/5.0 (compatible; Nigma.ru/3.0; crawler@nigma.ru)
  Mozilla/5.0 (compatible; Wappalyzer; +https://github.com/AliasIO/Wappalyzer)
  yacybot (/global; x86 Windows XP 5.1; java 1.7.0_71; Europe/de) http://yacy.net/bot.html
  Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/50.0.2661.102 Safari/537.36 PTST/277
  OpenWebSpider v0.1.4 (http://www.openwebspider.org/)
  HubSpot Connect 1.0 (http://dev.hubspot.com/)
  Page Valet/4.1pre5
  msnbot/2.0b (+http://search.msn.com/msnbot.htm)
  amibot - http://www.amidalla.de - tech@amidalla.com libwww-perl/5.831
  SETOOZBOT/5.0 ( compatible; SETOOZBOT/0.30 ; http://www.setooz.com/bot.html )
  Mozilla/5.0 (compatible; YoudaoBot/1.0; http://www.youdao.com/help/webmaster/spider/; )
  Mozilla/5.0 (compatible; MSIE or Firefox mutant; not on Windows server;) Daumoa 4.0
  Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.0) LinkCheck by Siteimprove.com
  EdisterBot (http://www.edister.com/bot.html)
  Mozilla/5.0 (iPhone; CPU iPhone OS 6_0 like Mac OS X) AppleWebKit/536.26 (KHTML, like Gecko) Version/6.0 Mobile/10A5376e Safari/8536.25 (compatible; SMTBot/1.0; +http://www.similartech.com/smtbot)
  Tarantula/BETA-0.5 (Tarantula de la Enlle, el mejor buscador en la red; http://www.enlle.com; tarantula@enlle.com)
  Mozilla/5.0 (Windows; U; Windows NT 5.1; fr; rv:1.8.1) VoilaBot BETA 1.2 (support.voilabot@orange-ftgroup.com)
  yacybot (freeworld/global; amd64 Linux 2.6.38-8-generic; java 1.6.0_22; Europe/en) http://yacy.net/bot.html
  Mozilla/5.0 (compatible; SemrushBot/0.95; +http://www.semrush.com/bot.html)
  CatchBot/3.0; +http://www.catchbot.com
  Mozilla/5.0 (compatible; uMBot-LN/1.0; mailto: crawling@ubermetrics-technologies.com)
  Mozilla/5.0 (compatible; RukiCrawler/1.0; +http://ruki.rezko.net)
  adbeat_bot
  Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/27.0.1453.116 Safari/537.36 HubSpot Webcrawler
  Mozilla/5.0 (compatible; YandexCatalog/3.0; +http://yandex.com/bots)
  NutchCVS/0.7.1 (Nutch; http://lucene.apache.org/nutch/bot.html; nutch-agent@lucene.apache.org)
  Statastico/4.0
  facebookexternalhit/1.1 (+http://www.facebook.com/externalhit_uatext.php)
  Mozilla/5.0 (X11; U; Linux x86_64; en-US) AppleWebKit/534.16 (KHTML, like Gecko, Google Keyword Suggestion) Chrome/10.0.648.127 Safari/534.16
  Mozilla/5.0 (iPhone; CPU iPhone OS 7_0 like Mac OS X) AppleWebKit/537.51.1 (KHTML, like Gecko) Version/7.0 Mobile/11A465 Safari/9537.53 BingPreview/1.0b
  EuripBot/2.0 (+http://www.eurip.com) GetRobots
  Mozilla/5.0 (compatible; MJ12bot/v1.2.1; http://www.majestic12.co.uk/bot.php?+)
  Braintree-Webhooks
  Nutch Checksem/Nutch-1.10
  Mozilla/5.0 (compatible; HyperCrawl/0.2; +http://www.seograph.net/bot.html)
  quickobot/quickobot-1 (Quicko Labs; http://quicko.co; robot at quicko dot co)
  Whibse
  Yahoo! Site Explorer Feed Validator http://help.yahoo.com/l/us/yahoo/search/siteexplorer/manage/
  Mozilla/5.0 (compatible; AhrefsBot/5.0; +http://ahrefs.com/robot/)
  Siteshooter B0t 2.1
  GingerCrawler/1.0 (Language Assistant for Dyslexics; www.gingersoftware.com/crawler_agent.htm; support at ginger software dot com)
  Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko; Google Page Speed Insights) Chrome/27.0.1453 Safari/537.36
  eBot / v.0.1e (http://alfa.elchron.cz)
  Mozilla/5.0 (compatible; memoryBot/1.20.247 +http://internetmemory.org/en/)
  yacybot (x86 Windows 2003 5.2; java 1.6.0_20; America/de) http://yacy.net/bot.html
  YandexSomething/1.0
  Scoop
  Mozilla/5.0 (compatible; MegaIndex.ru/2.0; +http://megaindex.com/crawler)
  Mozilla/5.0 (compatible; SputnikBot/2.3; +http://corp.sputnik.ru/webmaster)
  TurnitinBot/2.1 (http://www.turnitin.com/robot/crawlerinfo.html)
  Mozilla/5.0 (compatible; Ask Jeeves/Teoma; +http://about.ask.com/en/docs/about/webmasters.shtml)
  Mozilla/5.0 (compatible; CrawlForMe/3.0.2; +info@crawlforme.com)
  Seologies/0.1 Bot
  www.deadlinkchecker.com XMLHTTP/1.0 Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/46.0.2490.86 Safari/537.36
  dubaiindex (addressendeutschland.de)
  yacybot (/global; x86 Windows 7 6.1; java 1.8.0_40; America/es) http://yacy.net/bot.html
  Mozilla/5.0+(compatible;+PiplBot;++http://www.pipl.com/bot/)
  LinkStats Bot
  Mozilla/5.0 (compatible; adbeat-publisher-description-fetcher; +crawler@scaleunlimited.com; crawler@scaleunlimited.com)
  CISPA Vulnerability Notification (https://notify.mmci.uni-saarland.de)
  librabot/1.0 (+http://search.msn.com/msnbot.htm)
  SeoCheckBot (Seomira Seo Check, http://www.kfsw.de/bot.html)
  yacybot (x86 Windows 2003 5.2; java 1.6.0_16; Europe/de) http://yacy.net/bot.html
  WatchMouse/8.4.0.3 (http://watchmouse.com/ ; itpda01.watchmouse.net)
  yacybot (/global; amd64 Linux 3.13.0-83-generic; java 1.7.0_95; Europe/en) http://yacy.net/bot.html
  BrainbruBot/1.0 (+http://www.brainbru.com/)
  Docoloc
  Mozilla/5.0 (Unknown; Linux x86_64) AppleWebKit/538.1 (KHTML, like Gecko) PhantomJS/2.1.1 Safari/538.1 Prerender (+https://github.com/prerender/prerender)
  dlvr.it/1.0 (+http://dlvr.it/)
  Mozilla/5.0 (compatible; parsijoo-bot; +http://www.parsijoo.ir/; ehsanmousa@parsijoo.ir)
  DataparkSearch/4.35-02122005 ( http://www.dataparksearch.org/)
  Pixray-Seeker/2.0 (Pixray-Seeker; +http://www.pixray.com/pixraybot; +crawler@pixray.com)
  Abrave v5.5 (http://robot.abrave.co.uk)
  Mozilla/5.0 (X11; U; Linux i686 (x86_64); en-US; rv:1.8.1.20) Gecko/20090429 HeartRails_Capture/0.6 (+http://capture.heartrails.com/) BonEcho/2.0.0.20
  yacybot (/global; amd64 Linux 3.19.0-26-generic; java 1.7.0_95; Europe/en) http://yacy.net/bot.html
  MLBot (www.metadatalabs.com/mlbot)
  yacybot (freeworld/global; amd64 Linux 2.6.32-34-server; java 1.6.0_26; Europe/de) http://yacy.net/bot.html
  Mozilla/5.0 (compatible; LinkpadBot/1.08; +http://www.linkpad.ru)
  Mozilla/5.0 (compatible; SearchmetricsBot; http://www.searchmetrics.com/en/searchmetrics-bot/)
  msnbot-NewsBlogs/2.0b (+http://search.msn.com/msnbot.htm)
  ZnajdzFoto/Image 2.0
  Mozilla/5.0 (compatible; Magibot/4.0.0; +http://magi.peak-labs.com/robots.txt)
  Mozilla/5.0 (compatible; spbot/4.0.8; +http://OpenLinkProfiler.org/bot )
  Eurobot/1.2 (http://eurobot.ayell.eu)
  Mozilla/5.0 (compatible; Embedly/0.2; snap; +http://support.embed.ly/)
  Yaanb/1.5.001 (compatible; Win64;+http://www.yaanb.com/company/bot.hmtl)
  yacybot (i386 Linux 2.6.31-16-generic; java 1.6.0_15; Europe/en) http://yacy.net/bot.html
  Mozilla/5.0 (compatible; YioopBot; +http://173.13.143.74/bot.php)
  CopperEgg/RevealUptime/Singapore(aws)
  Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/534.24 (KHTML, like Gecko; Google Web Preview) Chrome/11.0.696 Safari/534.24
  yacybot (freeworld/global; amd64 Windows 8 6.2; java 1.7.0_25; Europe/de) http://yacy.net/bot.html
  Server Density Service Monitoring v2
  Testomatobot/1.0 (Linux x86_64; +http://www.testomato.com/testomatobot) minicrawler/3.0~beta
  TinEye/1.0; +http://www.tineye.com/
  Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.9.0.19; aggregator:Spinn3r (Spinn3r 3.1); http://spinn3r.com/robot) Gecko/2010040121 Firefox/3.0.19
  GigablastOpenSource/1.0
  Googlebot (gocrawl v0.4)
  CSmartCrawler (email: abuse[at]adsadv[dot]com)
  Testomatobot/1.0 (Linux x86_64; +http://www.testomato.com/testomatobot) minicrawler/4.0.0~beta9
  KOCMOHABT (+http://kozmonavt.tk/) AppEngine-Google; (+http://code.google.com/appengine; appid: s~web-explorer)
  Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.2.18) Gecko/20110622 HeartRails_Capture/1.0.5 (+http://capture.heartrails.com/) Namoroka/3.6.18
  PingSpot/1.0 (http://www.pingspot.org/)
  AppEngine-Google; (+http://code.google.com/appengine; appid: s~buzzstream-tools-hrd)
  AddThis.com (http://support.addthis.com/)
  Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko; Google Web Preview) Chrome/27.0.1453 Safari/537.36
  ExactSeekCrawler/1.0
  Yeti/1.0 (NHN Corp.; http://help.naver.com/robots/)
  Mozilla/5.0 (compatible; jpg-newsbot/2.0; +http://vipnytt.no/bot.html)
  g2reader-bot/1.0 (+http://www.g2reader.com/)
  MetaGeneratorCrawler/1.3.12 (www.metagenerator.info)
  QuerySeekerSpider ( http://queryseeker.com/bot.html )
  KDDI-CA31 UP.Browser/6.2.0.7.3.129 (GUI) MMP/2.0 (compatible; ichiro/mobile goo; +http://help.goo.ne.jp/help/article/1142/)
  Mozilla/5.0 (compatible; GrapeshotCrawler/2.0; +http://www.grapeshot.co.uk/crawler.php)
  Mozilla/5.0 (compatible; MSIE 10.0; Windows NT 6.2; Trident/6.0; WOW64; Trident/6.0; BingPreview/1.0b)
  Mozilla/5.0 (compatible; firmilybot/0.3; +http://www.firmily.com/bot.php
  Mozilla/5.0 (X11; U; Linux Core i7-4980HQ; de; rv:32.0; compatible; JobboerseBot; http://www.jobboerse.com/bot.htm) Gecko/20100101 Firefox/38.0
  Mozilla/5.0 (compatible; metager2-verification-bot; +http://metager2.de/technology.php)
  crawler4j (http://code.google.com/p/crawler4j/)
  CirrusExplorer/1.1 (http://www.cireu.com/explorer.php)
  Mozilla/5.0 (compatible; RankvalBot/1.0; +https://rankval.com)
  Mozilla/5.0 (compatible; heritrix/1.14.3 +http://www.accelobot.com)
  Mozilla/5.0 (compatible; BLEXBotTest/1.0; +http://webmeup.com/crawler.html)
  nWormFeedFinder (http://www.nworm.com)
  netEstate NE Crawler (+http://www.sengine.info/)
  Mozila/5.0 (compatible; StudioFACA Search; +http://www.studiofaca.com/)
  Mozilla/5.0 (compatible; Crawlera/1.10.2; UID 92751)
  wikiwix-bot-3.0
  SEOCentro Page Keyword Analyzer v1.2
  yacybot (amd64 Linux 2.6.18-164.15.1.el5xen; java 1.6.0_0; Europe/en) http://yacy.net/bot.html
  Mozilla/5.0 (compatible; bingbot/2.0; http://www.bing.com/bingbot.htm)
  Automattic Analytics Crawler/0.1; http://wordpress.com/crawler/
  SemrushBot/0.92
  http://www.uni-koblenz.de/~flocke/robot-info.txt
  Mozilla/5.0 (compatible; idmarch Automatic.beta/1.3; +http://www.idmarch.org/bot.html)
  Mozilla/5.0 (X11; Linux i686 on x86_64; rv:25.0; GTmetrix http://gtmetrix.com/) Gecko/20100101 Firefox/25.0
  msnbot/2.0b (+http://search.msn.com/msnbot.htm)._
  Stroke.cz (http://stroke.cz)
  Mozilla (+http://web.nli.org.il/sites/NLI/English/collections/israel-collection/internet_archive/Pages/default.aspx)
  audisto.com full crawler 4.33.470 (refer to in robots.txt as audisto, see https://audisto.com/bot)
  http://arachnode.net 2.5
  SurveyBot/2.3 (Whois Source)
  ant.com/Nutch-1.7 (http://ant.com)
  CorpusCrawler 2.0.13 (http://corpora.fi.muni.cz/crawler/)
  Seo Servis - Analyza zdrojoveho kodu
  SEOENGBot/1.2 (+http://www.seoengine.com/seoengbot.htm)
  Mozilla/5.0 (compatible; Genieo/1.0 http://www.genieo.com/webfilter.html)
  Mozilla/5.0 (compatible; evc-batch/2.0.20160708184126)
  HubSpot Marketing Grader
  Mozilla/5.0 (compatible; MSIE 10.0; Windows NT 6.1; Trident/6.0) LinkCheck by Siteimprove.com
  nrsbot/6.0(loopip.com/robot.html)
  Mozilla/5.0 (compatible; BLEXBot/1.0; +http://webmeup.com/crawler.html)
  yacybot (freeworld-global; amd64 Windows 7 6.1; java 1.7.0_02-ea; America/en) http://yacy.net/bot.html
  Mozilla/5.0 (compatible; Blekkobot; ScoutJet; +http://blekko.com/about/blekkobot)
  Mozilla/5.0 (compatible; Topicbot/12.0rc-2; +http://topicbot.awardspace.us/)
  Mozilla/5.0 (compatible; Gimme60bot/1.0; +http://gimme60.com)
  yacybot (i386 Linux 2.6.28-11-generic; java 1.6.0_13; Europe/de) http://yacy.net/bot.html
  Mozilla/5.0 (compatible; DotBot/1.1; http://www.opensiteexplorer.org/dotbot, help@moz.com)
  Kemvibot/1.0 (http://kemvi.com, marco@kemvi.com)
  CorpusCrawler 2.0.15 (http://corpora.fi.muni.cz/crawler/)
  dotbot
  Mozilla/5.0 (iPhone; CPU iPhone OS 7_0 like Mac OS X) AppleWebKit/537.51.1 (KHTML, like Gecko) Version/7.0 Mobile/11A465 Safari/9537.53 (compatible; adidxbot/2.0; +http://www.bing.com/bingbot.htm)
  page_viewer (larbin2.6.3@unspecified.mail)
  HootSuite Image proxy (hosting@hootsuite.com)
  Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US) Speedy Spider for SpeedyAds (http://www.entireweb.com/about/search_tech/speedy_spider/)
  PINGOMETER_BOT_(HTTPS://PINGOMETER.COM)
  Mozilla/5.0 (compatible; Vagabondo/2.1; webcrawler at wise-guys dot nl; http://webagent.wise-guys.nl/)
  VSE/1.0 (rabraham@multiview.com)
  Scrapy/0.24.0 (+http://scrapy.org)
  Speedy Spider (http://www.entireweb.com)
  Mozilla/5.0 (X11; Linux x86_64; GTmetrix https://gtmetrix.com/) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/51.0.2704.106 Safari/537.36
  WatchMouse/18990 (http://watchmouse.com/ ; hk)
  Mozilla/5.0 (compatible; yoozBot-2.2; http://yooz.ir; info@yooz.ir)
  Mozilla/5.0 (compatible; Crawlera/1.10.2; UID 87214)
  Mozilla/5.0 (compatible; discobot/2.0; +http://discoveryengine.com/discobot.html)
  yacybot (freeworld/global; amd64 Windows 7 6.1; java 1.8.0_11; Europe/de) http://yacy.net/bot.html
  Testomatobot/1.0 (Linux x86_64; +http://www.testomato.com/testomatobot) minicrawler/4.0.0~beta6
  yacybot (freeworld/global; amd64 Linux 3.0.0-15-server; java 1.6.0_23; Europe/en) http://yacy.net/bot.html
  Mozilla/5.0 (Macintosh; Intel Mac OS X 10.9; rv:28.0) Gecko/20100101 Firefox/28.0 (FlipboardProxy/1.1; +http://flipboard.com/browserproxy)
  YisouSpider
  Slack-ImgProxy 1.138 (+https://api.slack.com/robots)
  mozilla/5.0 (compatible; webmastercoffee/0.7; +http://webmastercoffee.com/about)
  Mozilla/5.0 (compatible; Uptimebot/0.2.22; +http://www.uptime.com/uptimebot)
  Mozilla/5.0 (compatible; AhrefsBot/1.0; +http://ahrefs.com/robot/)
  Mozilla/5.0 (compatible; Feedspotbot/1.0; +http://www.feedspot.com/fs/bot)
  sg-Orbiter/1.0 (+http://searchgears.de/uber-uns/crawling-faq.html)
  Site24x7 Tools
  NextGenSearchBot 1 (for information visit http://www.zoominfo.com/About/misc/NextGenSearchBot.aspx)
  Aboundex/0.3 (http://www.aboundex.com/crawler/)
  yacybot (freeworld/global; i386 Linux 2.6.37.6-0.5-desktop; java 1.6.0_20; Europe/en) http://yacy.net/bot.html
  Mozilla/5.0 (Windows NT 6.2; WOW64) AppleWebKit/537.4 (KHTML, like Gecko) Chrome/98 Safari/537.4 (StatusCake)
  Mozilla/5.0 (compatible; DomainSONOCrawler/0.1; +http://domainsono.com)
  Mozilla/5.0 (compatible; Online Domain Tools - Server Monitor/1.0; +http://server-monitoring.online-domain-tools.com)
  Mozilla/5.0 (compatible; Konqueror/3.5; Linux) KHTML/3.5.5 (like Gecko) (Exabot-Thumbnails)
  Mozilla/5.0 (compatible; spbot/4.0.4; +http://www.seoprofiler.com/bot )
  Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/27.0.1453 Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
  Factbot 1.09 (see http://www.factbites.com/webmasters.php)
  CopperEgg/RevealUptime/SaoPauloBR(aws)
  CRIM Crawler/Nutch-2.3 (Crawler du Centre de Recherche Informatique de Montr\xc3\xa9al (CRIM))
  Mozilla/5.0 (compatible; heritrix/1.14.3 +http://www.webarchiv.cz)
  Mozilla/5.0 (compatible; WormlyBot; +http://wormly.com)
  yacybot (i386 Linux 2.6.23; java 1.6.0_17; Europe/en) http://yacy.net/bot.html
  Mozilla/5.0 (compatible; Cliqzbot/0.1 +http://cliqz.com/company/cliqzbot)
  URLitor.com
  RavenCrawler
  Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.2.18) Gecko/20130331 HeartRails_Capture/1.0.5 (+http://capture.heartrails.com/) Namoroka/3.6.18
  Mozilla/5.0 (compatible; WebThumbnail/3.x; Website Thumbnail Generator; +http://webthumbnail.org)
  Kyoto-Crawler/2.0 (Mozilla-compatible; kyoto-crawler-contact(at)nlp(dot)kuee(dot)kyoto-u(dot)ac(dot)jp; http://nlp.ist.i.kyoto-u.ac.jp/)
  Mozilla/5.0 (compatible; SputnikImageBot/2.3; +http://corp.sputnik.ru/webmaster)
  WatchMouse/8.4.0.3 (http://watchmouse.com/ ; lvps91-250-96-109.dedicated.hosteurope.de)
  MetaGeneratorCrawler/1.3.3 (www.metagenerator.info)
  LongURL API
  Mozilla/5.0 ( compatible; SETOOZBOT/0.30 ; http://www.setooz.com/bot.html ; agentname at setooz dot_com )
  Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.6; en-US; rv:1.9.2) Gecko/20100115 Firefox/3.6 (FlipboardProxy/2.0; +http://flipboard.com/browserproxy)
  http://tools.geek-tools.org/link-counter/
  SCFCrawler/Nutch-1.8 (Image Crawler for StolenCameraFinder.com; http://www.stolencamerafinder.com/; crawler@stolencamerafinder.com)
  Mozilla/5.0 (compatible; Uptimebot/0.2.41; +http://www.uptime.com/uptimebot)
  Mozilla/5.0 (compatible; DomainAppender /1.0; +http://www.profound.net/domainappender)
  Mozilla/5.0 (compatible; hypestat/1.0; +http://www.hypestat.com/bot)
  yacybot (/global; amd64 Linux 3.13.0-042stab093.4; java 1.7.0_79; Europe/en) http://yacy.net/bot.html
  PagesInventory (robot +http://www.pagesinventory.com)
  Sosospider+(+http://help.soso.com/webspider.htm)
  Mozilla/5.0 (compatible; AntBot/1.0; +http://www.ant.com/)
  PagesInventory (robot http://www.pagesinvenotry.com)
  Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; http://www.changedetection.com/bot.html )
  Wotbox/2.01 (+http://www.wotbox.com/bot/)
  Mozilla/5.0 (compatible; bixolabs/1.0; +http://bixolabs.com/crawler/general; crawler@bixolabs.com)
  Mozilla/5.0 (compatible; SpiderLing (a SPIDER for LINGustic research); +http://nlp.fi.muni.cz/projects/biwec/)
  CopperEgg/RevealUptime/DallasTXUSA
  yacybot (freeworld/global; amd64 Linux 2.6.32-custom; java 1.6.0_26; Europe/en) http://yacy.net/bot.html
  Bad-Neighborhood Link Analyzer (http://www.bad-neighborhood.com/)
  Mozilla/5.0 (compatible; oBot/2.3.1; +http://filterdb.iss.net/crawler/)
  Pixray-Seeker/2.0 (Pixray-Seeker; http://www.pixray.com/pixraybot; crawler@pixray.com)
  Cityreview Robot (+http://www.cityreview.org/crawler/)
  yacybot (/global; amd64 Linux 3.16.0-4-amd64; java 1.7.0_95; Europe/en) http://yacy.net/bot.html
  Dlvr.it/1.0 (http://dlvr.it/)
  Zookabot/2.4;++http://zookabot.com
  AddThis.com robot tech.support@clearspring.com
  Mozilla/5.0 (compatible; discoverybot/2.0; +http://discoveryengine.com/discoverybot.html)
  TwengaBot
  Mozilla/5.0 (compatible; 4SeoHuntBot; +http://4seohunt.biz/about.html)
  Web-sniffer.me/1.0.0 (+http://web-sniffer.me/)
  KDDI-CA31 UP.Browser/6.2.0.7.3.129 (GUI) MMP/2.0 (compatible; ichiro/mobile goo; +http://search.goo.ne.jp/option/use/sub4/sub4-1/)
  gosquared-thumbnailer/1.0
  yacybot (/global; amd64 Windows 7 6.1; java 1.7.0_55; Asia/zh) http://yacy.net/bot.html
  BPImageWalker/2.0 (www.bdbrandprotect.com)
  Acoon v4.10.3 (www.acoon.de)
  Mozilla/4.0 (CMS Crawler: http://www.cmscrawler.com)
  WP Engine Install Performance API
  Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.2.24; ips-agent) Gecko/20111107 Ubuntu/10.04 (lucid) Firefox/3.6.24
  Mozilla/5.0 (Windows NT 6.1; WOW64; Trident/7.0; rv:11.0) like Gecko PTST/277
  CamontSpider/1.0 +http://epweb2.ph.bham.ac.uk/user/slater/camont/info.html
  SEOENGBot/1.2 (+http://learn.seoengine.com/seoengbot.htm)
  Y!J-BRJ/YATS crawler (http://listing.yahoo.co.jp/support/faq/int/other/other_001.html)
  CopperEgg/RevealUptime/N.Virginia(aws)
  Mozilla/5.0, Google-AdWords-Express
  Mozilla/5.0 (compatible;YodaoBot-Image/1.0;http://www.youdao.com/help/webmaster/spider/;)
  yacybot (/global; amd64 Linux 3.16.0-4-amd64; java 1.8.0_91; Europe/en) http://yacy.net/bot.html
  yacybot (freeworld/global; amd64 Windows 7 6.1; java 1.7.0_09; Europe/de) http://yacy.net/bot.html
  Acoon v4.10.1 (www.acoon.de)
  Mediatoolkitbot (complaints@mediatoolkit.com)
  Mozilla/5.0 (compatible; Speedy Spider; http://www.entireweb.com/about/search_tech/speedy_spider/)
  Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.2.3) SPEng
  2Bone_LinkChkr/1.0 libwww-perl/6.03
  Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.6; en-US; rv:1.9.2) Gecko/20100115 Firefox/3.6 (FlipboardBrowserProxy/0.0.5; +http://flipboard.com/browserproxy)
  Mozilla/5.0 (compatible; SeznamBot/3.2-test1; +http://napoveda.seznam.cz/en/seznambot-intro/)
  Comodo SSL Checker
  Comodo Spider 1.2
  Who.is Bot
  Mozilla/5.0 (compatible; online-webceo-bot/1.0; +http://online.webceo.com)
  yacybot (freeworld/global; amd64 Windows Server 2008 R2 6.1; java 1.6.0_29; Europe/en) http://yacy.net/bot.html
  psbot/0.1 (+http://www.picsearch.com/bot.html)
  IlTrovatore-Setaccio/1.2 (http://www.iltrovatore.it/aiuto/faq.html)
  Mozilla/5.0 (compatible; Twingly Recon; twingly.com)
  Mozilla/5.0 Moreover/5.1 (+http://www.moreover.com; webmaster@moreover.com)
  Orgbybot/OrgbyBot v1.2 (Spidering the net for Orgby; http://www.orgby.com/ ; Orgby.com Search Engine)
  Browsershots
  AdsBot-Google (+http://www.google.com/adsbot.html)
  Qseero v1.0.0
  Mozilla/5.0 (compatible; ltbot/0.3.6 +http://www.kdsl.tu-darmstadt.de/de/kdsl/research-program/crawling-and-semantic-structuring/)
  yacybot (x86 Windows XP 5.1; java 1.6.0_18; Europe/de) http://yacy.net/bot.html
  visionutils/0.2
  Mozilla/5.0 (compatible; Googlebot/2.1; https://www.deepcrawl.com/bot)
  Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.4 (KHTML, like Gecko; Google Web Preview) Chrome/22.0.1229 Safari/537.4
  Mozilla/5.0 (compatible; Crawlera/1.10.2; UID 8942)
  Mozilla/4.0 (compatible; LinkMarket-Bot v1.2)
  Buzzbot/1.0 (Buzzbot; http://www.buzzstream.com; buzzbot@buzzstream.com)
  WordPress.com; https://public-api.wordpress.com
  Mozilla/5.0 ( compatible; SETOOZBOT/0.30 ; http://www.setooz.com/bot.html )
  CopperEgg/RevealUptime/FremontCAUSA
  Mozilla/5.0 (Windows; U; Windows NT 6.0; en-GB; rv:1.0; trendictionbot0.4.5; trendiction search; http://www.trendiction.de/bot; please let us know of any problems; web at trendiction.com) Gecko/20071127 Firefox/3.0.0.11
  yacybot (/global; amd64 Linux 2.6.32-042stab094.8; java 1.7.0_79; America/en) http://yacy.net/bot.html
  Mozilla/5.0 (compatible; Windows NT 6.1?; ZumBot/1.0; http://help.zum.com/inquiry)
  Mozilla/5.0 (Windows NT 6.1; WOW64; Trident/7.0; rv:11.0; topster.de Linkchecker 5.0) like Gecko (90.177.52.133)
  Mozilla/5.0+(compatible; Monitority/1.0; http://www.monitority.com/)
  yacybot (freeworld/global; x86 Windows 7 6.1; java 1.8.0_25; Europe/de) http://yacy.net/bot.html
  Mozilla/5.0 (compatible; abby/1.0; +http://www.ellerdale.com/crawler.html)
  Mozilla/5.0 (compatible; SemrushBot/1.1~bl; +http://www.semrush.com/bot.html)
  YowedoBot/Yowedo 1.0 (Search Engine crawler for yowedo.com; http://yowedo.com/en/partners.html; crawler@yowedo.com)
  Mozilla/5.0 (compatible; oBot/2.3.1; http://filterdb.iss.net/crawler/)
  Nokia6820/2.0 (4.83) Profile/MIDP-1.0 Configuration/CLDC-1.0 (compatible; Googlebot-Mobile/2.1; +http://www.google.com/bot.html)
  Mozilla/5.0 (compatible; IstellaBot/1.23.15 +http://www.tiscali.it/)
  GetIntent Crawler (http://getintent.com/bot.html)
  Mozilla/5.0 (compatible; Uptimebot/0.1.89; +http://www.uptime.com/uptimebot)
  Cliqzbot
  AboutUsBot
  yacybot (-global; amd64 Linux 3.2.0-4-amd64; java 1.7.0_67; Europe/en) http://yacy.net/bot.html
  Sufog/Nutch-2.2.1 (www.sufog.com; www.sufog.com)
  Mozilla/5.0 (compatible; imbot/0.1 +http://internetmemory.org/en/)
  KOCMOHABT (+http://kozmonavt.tk/) Mozilla/5.0 (compatible; Web Explorer 1) AppEngine-Google; (+http://code.google.com/appengine; appid: s~web-explorer)
  yacybot (/global; amd64 Linux 3.16.0-53-generic; java 1.7.0_85; Europe/en) http://yacy.net/bot.html
  L.webis/0.53 (http://webalgo.iit.cnr.it/index.php?pg=lwebis)
  Mozilla/5.0 (compatible; memoryBot/1.21.23 +http://internetmemory.org/en/)
  ExB Language Crawler 2.1.5 (+http://www.exb.de/crawler)
  Peeplo Screenshot Bot/0.20 ( abuse at peeplo dot_com )
  Mozilla/5.0 (compatible; Qwantify/2.1w; +https://www.qwant.com/)/*
  Experibot_v1 [bit.ly/2bLMJdw]
  WikioFeedBot 1.0 (http://www.wikio.com)
  SalesIntelligent/v1.1 (https://www.salesintelligent.com)
  Mozilla/5.0 (compatible; ExaleadCloudView/5;)
  Mozilla/5.0 (compatible; rogerBot/1.0; UrlCrawler; http://www.seomoz.org/dp/rogerbot)
  Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/44.0.2403.157 Safari/537.36 TinEye/1.0 (via http://www.tineye.com/)
  Mozilla/5.0 (compatible; Search17Bot/1.1; http://www.search17.com/bot.php)
  FollowSite Bot ( http://www.followsite.com/bot.html )
  Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/27.0.1453.116 Safari/537.36 HubSpot Webcrawler - web-crawlers@hubspot.com
  Mozilla/5.0 (compatible; BlinkaCrawler/1.0; +http://www.blinka.jp/crawler/)
  Mozilla/5.0 (compatible; Qwantify/2.2w; +https://www.qwant.com/)/*
  findlinks/1.1.6-beta2 (+http://wortschatz.uni-leipzig.de/findlinks/)
  yacybot (/global; amd64 Linux 4.4.0-28-generic; java 1.8.0_91; Europe/pl) http://yacy.net/bot.html
  yacybot (/global; amd64 Windows Server 2008 R2 6.1; java 1.8.0_40; Europe/de) http://yacy.net/bot.html
  Mozilla/5.0 (compatible; 200PleaseBot/1.0; +http://www.200please.com/bot)
  WatchMouse/8.4.0.3 (http://watchmouse.com/ ; chzrh01.watchmouse.net)
  Iframely/0.8.8 (+http://iframely.com/;)
  Mozilla/5.0 (compatible; URLAppendBot/1.0; +http://www.profound.net/urlappendbot.html)
  LoadImpactRload/3.2.0 (Load Impact; http://loadimpact.com);
  RSSingBot (http://www.rssing.com)
  RADaR-Bot/Nutch-1.3 (http://radar-bot.com/)
  Ocelli/1.4 (http://www.globalspec.com/Ocelli)
  Mozilla/5.0 (compatible; BuzzSumo; +http://www.buzzsumo.com/bot.html)
  Mozilla/4.0 (compatible; IDwhois/2.0alpha-10102619; Linux; en-US; +http://idwhois.info/about)
  omgilibot/0.4 +http://omgili.com
  yacybot (freeworld/global; amd64 Windows 7 6.1; java 1.6.0_23; Europe/de) http://yacy.net/bot.html
  Y!J-BRW/1.0 crawler (http://help.yahoo.co.jp/help/jp/search/indexing/indexing-15.html)
  Mozilla/5.0 (compatible; AhrefsBot/2.0; +http://ahrefs.com/robot/)
  Mozilla/5.0 (compatible; VideoSurf_bot +http://www.videosurf.com/bot.html)
  Symfony Spider (http://symfony.com/spider)
  Mozilla/5.0 (compatible; Sosospider/2.0; +http://help.soso.com/webspider.htm)
  Mozilla/5.0 (compatible; OpenindexDeepSpider/Nutch-1.5-dev; +http://www.openindex.io/en/webmasters/spider.html)
  ADmantX Platform Semantic Analyzer - ADform - ADmantX Inc. - www.admantx.com - support@admantx.com
  Mozilla/5.0 (compatible; MJ12bot/v1.3.3; http://www.majestic12.co.uk/bot.php?+)
  cuwhois/1.0 (+http://www.cuwhois.com/)
  Mozilla/5.0 (compatible; memoryBot/1.20.311 +http://internetmemory.org/en/)
  Mozilla/4.0 (compatible; Vagabondo/4.0; webcrawler at wise-guys dot nl; http://webagent.wise-guys.nl/; http://www.wise-guys.nl/)
  yacybot (-global; amd64 Linux 3.16.0-4-amd64; java 1.7.0_75; Europe/en) http://yacy.net/bot.html
  Mozilla/5.0 (compatible; Dazoobot/1.0; http://dazoo.fr)
  Mozilla/5.0 (compatible; Semager/1.4c; +http://www.semager.de/blog/semager-bots/)
  EuripBot/1.1 (+http://www.eurip.com) GetRobots
  yacybot (freeworld/global; amd64 Linux 3.8.0-23-generic; java 1.6.0_27; Pacific/en) http://yacy.net/bot.html
  yacybot (amd64 Linux 2.6.24-23-generic; java 1.6.0_07; Europe/de) http://yacy.net/bot.html
  yacybot (freeworld/global; x86 Windows 2003 5.2; java 1.6.0_32; Europe/de) http://yacy.net/bot.html
  yacybot (freeworld/global; amd64 Linux 3.2.0-4-amd64; java 1.7.0_25; Europe/de) http://yacy.net/bot.html
  socialbm_bot http://spider.socialbm.net
  Mozilla/5.0 (compatible; 007ac9 Crawler; http://crawler.007ac9.net/)
  webinatorbot 1.0; +http://www.webinator.de
  WorldBrewBot/2.1 (+http://www.marketbrew.com/)
  BotOnParade, http://www.bots-on-para.de/bot.html
  ltx71 - (http://ltx71.com/)
  VSB-TUO/1.0 (+http://www.vsb.cz)
  yacybot (/global; amd64 Linux 3.13.0-85-generic; java 1.7.0_101; Europe/en) http://yacy.net/bot.html
  CopperEgg/RevealUptime/TokyoJP(aws)
  Mozilla/5.0 (compatible; etoolsbot/1.0; +https://etools.io/bots/)
  Comodo-Webinspector-Crawler 2.2.2, http://www.comodorobot.com
  Mozilla/5.0 (compatible; TWMBot/0.1; +http://thewebminer.com)
  Mozilla/5.0 (compatible; Plukkie/1.3; http://www.botje.com/plukkie.htm)
  Autonomy
  yacybot (i386 Linux 2.6.30-2-686; java 1.6.0_0; SystemV/en) http://yacy.net/bot.html
  Testomatobot/1.0 (Linux x86_64; +http://www.testomato.com/testomatobot) minicrawler/4.1.0
  CatchBot/1.0; +http://www.catchbot.com
  Sitedomain-Bot(Sitedomain-Bot 1.0, http://www.sitedomain.de/sitedomain-bot/)
  Mozilla/5.0 (compatible; Uptimebot/0.2.42; +http://www.uptime.com/uptimebot)
  yacybot (/global; amd64 Linux 3.2.0-4-amd64; java 1.7.0_03; Europe/en) http://yacy.net/bot.html
  SiteTruth.com site rating system
  SiteUptime.com
  yacybot (i386 Linux 2.6.24-23-generic; java 1.6.0_16; Europe/en) http://yacy.net/bot.html
  UrlTrends Ranking Updater/2.0
  yacybot (/global; amd64 Windows 7 6.1; java 1.7.0_55; Europe/de) http://yacy.net/bot.html
  yacybot (freeworld/global; amd64 Linux 3.5.0-27-generic; java 1.7.0_25; Europe/en) http://yacy.net/bot.html
  GarlikCrawler/1.1 (http://garlik.com/, crawler@garlik.com)
  Mozilla/5.0 (compatible; ParsijooBot; +http://www.parsijoo.ir/; crawler@parsijoo.ir)
  Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.1 (KHTML, like Gecko) Chrome/21.0.1180.89 Safari/537.1; 360Spider(compatible; HaosouSpider; http://www.haosou.com/help/help_3_2.html)
  Willow Internet Crawler by Twotrees V2.1
  KrOWLer/0.1.0, matentzn at cs dot man dot ac dot uk
  SimplyFast.info Headers
  yacybot (freeworld/global; amd64 Linux 2.6.32-431.29.2.el6.x86_64; java 1.7.0_65; Europe/en) http://yacy.net/bot.html
  Mozilla/5.0 (compatible; YandexBlogs/0.99; robot; +http://yandex.com/bots)
  msnbot-media/1.1 (+http://search.msn.com/msnbot.htm)
  Mozilla/5.0 (Windows NT 6.1; WOW64; rv:46.0) Gecko/20100101 Firefox/46.0 PTST/278
  Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/534.57.2 (KHTML, like Gecko) Version/5.1.7 Safari/534.57.2 PTST/278
  gipo-crawler/Nutch-1.10 (Global Internet Policy Observatory crawler)
  Mozilla/5.0 (compatible; sukibot_heritrix/3.1.1 +http://suki.ling.helsinki.fi/eng/webmasters.html)
  PayPal IPN ( https://www.paypal.com/ipn )
  Mozilla/5.0 (compatible; KaloogaBot; http://kalooga.com/crawler)
  eCairn-Grabber/1.0 (+http://ecairn.com/grabber) curl/7.15
  Mozilla/5.0 (Windows NT 6.2; WOW64) AppleWebKit/537.15 (KHTML like Gecko; +http://www.diffbot.com) Chrome/24.0.1295.0 Safari/537.15
  ImageEngine/1.0
  DER-bot (+https://goo.gl/9ZibHG)
  yacybot (freeworld/global; amd64 Windows 7 6.1; java 1.7.0_60; Europe/de) http://yacy.net/bot.html
  Mozilla/5.0 (compatible; SemrushBot/0.96.3; +http://www.semrush.com/bot.html)
  Mozilla/5.0 (compatible; Exabot/3.0; +http://www.exabot.com/go/robot)
  Mozilla/5.0 (compatible; YandexDirect/3.0; +http://yandex.com/bots)
  Mozilla/5.0 (compatible; aiHitBot/2.9; +https://www.aihitdata.com/about)
  Linguee Bot (http://www.linguee.com/bot; bot@linguee.com)
  MetaGeneratorCrawler/1.3.9 (www.metagenerator.info)
  TelegramBot (like TwitterBot)
  Mozilla/5.0 (compatible; Siteliner/1.0; +http://www.siteliner.com/bot)
  calculon spider/0.1.0 mailto:abuse@yieldbot.com
  yacybot (/global; amd64 Linux 3.13.0-83-generic; java 1.7.0_95; Europe/de) http://yacy.net/bot.html
  PocketParser/2.0 (+https://getpocket.com/pocketparser_ua)
  montastic-monitor http://www.montastic.com
  Mozilla/5.0 (compatible; trovitBot 1.0; +http://www.trovit.com/bot.html)
  Mozilla/5.0 (Windows NT 6.1)+(compatible; SMTBot/1.0; +http://www.similartech.com/smtbot)
  Yaanb/1.5.001 (compatible; Win64;)
  Mozilla/5.0 (compatible; Crawlera/1.10.2; UID 70054)
  Mozilla/5.0 (compatible; spbot/2.0.2; +http://www.seoprofiler.com/bot/ )
  yacybot (freeworld-global; amd64 Linux 3.2.0-4-amd64; java 1.6.0_24; Europe/en) http://yacy.net/bot.html
  Mozilla/5.0 (compatible; ImageFetcher/5.6; +http://images.weserv.nl/)
  CopperEgg/RevealUptime/N.VirginiaUSA
  Huaweisymantecspider (compatible; MSIE 8.0; DSE-support@huaweisymantec.com)
  OrangeCrawler/Nutch-1.0 (ldorange.crawler@orange-ftgroup.com)
  CopperEgg/RevealUptime/AmsterdamHolland
  Mozilla/5.0 (compatible; SemrushBot-SI/0.97; +http://www.semrush.com/bot.html)
  Mozilla/5.0 (X11; Linux x86_64; GTmetrix https://gtmetrix.com/) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/50.0.2661.75 Safari/537.36
  Mozilla/5.0 (compatible; Genieo/1.0 http://www.genieo.com/webfilter.html) AppEngine-Google; (+http://code.google.com/appengine; appid: s~natmacdevice)
  yacybot (/global; amd64 Linux 3.2.41-042stab094.7; java 1.7.0_75; Europe/en) http://yacy.net/bot.html
  yacybot (x86 Windows Vista 6.1; java 1.6.0_13; Europe/de) http://yacy.net/bot.html
  SmabblerBot/1.0 (http://www.smabbler.com/)
  Mozilla/5.0 (compatible; grapeFX/0.9; crawler@grapeshot.co.uk
  Mozilla/5.0 (compatible; SemrushBot/1~bl; +http://www.semrush.com/bot.html)
  MetaGeneratorCrawler/1.3.14 (www.metagenerator.info)
  oku-taka-lab-bot (Mozilla-compatible; htakimoto@lr.pi.titech.ac.jp; http://lr-www.pi.titech.ac.jp/wp/)
  Mozilla/5.0 (Linux; Android 4.1.2; Galaxy Nexus Build/JZO54K; GTmetrix https://gtmetrix.com/) AppleWebKit/537.22 (KHTML, like Gecko) Chrome/26.0.1410.58 Mobile Safari/537.22
  Mozilla/5.0 (compatible; Feedage/2.0; +http://www.feedage.com/bot.php)
  yacybot (amd64 Linux 2.6.26-2-openvz-amd64; java 1.6.0_12; UTC/en) http://yacy.net/bot.html
  yacybot (-global; amd64 Linux 2.6.32-042stab116.1; java 1.7.0_79; Europe/en) http://yacy.net/bot.html
  Mozilla/5.0 (compatible; DKIMRepBot/1.0; +http://www.dkim-reputation.org)
  Mozilla/5.0 (Windows; U; Windows NT 5.1; zh-CN; rv:1.8.0.11) Firefox/1.5.0.11; 360Spider
  Mozilla/5.0 (compatible; pub-crawler; +http://wiki.github.com/bixo/bixo/bixocrawler; bixo-dev@yahoogroups.com)
  mfibot/1.1 (http://www.mfisoft.ru/analyst/; <admin@mfisoft.ru>; en-RU)
  Visited by http://tools.geek-tools.org
  peerindex
  Mozilla/5.0 (compatible; janforman.com/2.4; +http://janforman.com/)
  http://www.amagit.com/
  yacybot (freeworld/global; amd64 Linux 2.6.32-40-server; java 1.6.0_20; Europe/en) http://yacy.net/bot.html
  Mozilla/5.0 (iPhone; CPU iPhone OS 8_3 like Mac OS X) AppleWebKit/537.36 (KHTML, like Gecko) Version/8.0 Mobile/12F70 Safari/600.1.4 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
  Mozilla/5.0 (compatible; NLNZ_IAHarvester2014 +https://natlib.govt.nz/publishers-and-authors/web-harvesting/domain-harvest)
  Mozilla/5.0 (compatible; YandexImageResizer/2.0; +http://yandex.com/bots)
  mahonie, neofonie search:robot/search:robot/0.0.1 (This is the MIA Bot - crawling for mia research project. If you feel unhappy and do not want to be visited by our crawler send an email to spider@neofonie.de; http://spider.neofonie.de; spider@neofonie.de)
  Mozilla/5.0 (Linux; Android 5.0; Nexus 5 Build/LRX21O) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/46.0.2490.76 Mobile Safari/537.36 PTST/276
  Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; Trident/5.0; WOW64; Trident/5.0; BingPreview/1.0b)
  Mozilla/5.0 (compatible; OptimizationCrawler/0.2; +http://www.domainoptima.com/bot.html)
  Mozilla/5.0 (compatible; Cloudinary/1.0)
  rogerbot/1.0 (http://moz.com/help/pro/what-is-rogerbot-, rogerbot-wherecat@moz.com)
  Iframely/0.9.8 (+http://iframely.com/;)
  yacybot (webportal-global; amd64 Linux 2.6.32-5-amd64; java 1.6.0_18; Europe/en) http://yacy.net/bot.html
  Interdose AntiSpamBot/2.0 (+http://www.piep.net)
  CheckMarkNetwork/1.0 (+http://www.checkmarknetwork.com/spider.html)
  Mozilla/5.0 (iPhone; CPU iPhone OS 8_3 like Mac OS X) AppleWebKit/537.36 (KHTML, like Gecko) Version/8.0 Mobile/12F70 Safari/600.1.4 (compatible; Google Search Console)
  rogerbot/1.0 (http://www.seomoz.org/dp/rogerbot, rogerbot-crawler+shiny@seomoz.org)
  findlinks/2.5 (+http://wortschatz.uni-leipzig.de/findlinks/)
  Mozilla/5.0 (compatible; MSIE 10.0; Windows NT 6.1; WOW64; Trident/6.0) PTST/277
  yacybot (/global; x86 Windows 2003 5.2; java 1.7.0_67; Europe/de) http://yacy.net/bot.html
  NeutrinoAPI/2.0.3
  Mozilla/5.0 (compatible; spbot/5.0.3; +http://OpenLinkProfiler.org/bot )
  Mozilla/5.0 (compatible; MJ12bot/v1.4.4; http://www.majestic12.co.uk/bot.php?+)
  Site24x7 Agent
  Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.2; WOW64; Trident/6.0; .NET4.0E; .NET4.0C; .NET CLR 3.5.30729; .NET CLR 2.0.50727; .NET CLR 3.0.30729) CrawlerProcess (http://www.PowerMapper.com) /5.7.704.0
  Mozilla/5.0 (compatible; Falconsbot; +http://iws.seu.edu.cn/services/falcons/contact_us.jsp)
  Super Monitoring
  Seznam-Zbozi-robot/3.0
  LinkedInBot/1.0 (compatible; Mozilla/5.0; Jakarta Commons-HttpClient/4.3 +http://www.linkedin.com)
  Mozilla/5.0 (compatible; EveryoneSocialBot/1.0; support@everyonesocial.com http://everyonesocial.com/)
  Mozilla/5.0 (compatible; IstellaBot/1.01.18 +http://www.tiscali.it/)
  Mozilla/5.0 (compatible; ToutiaoSpider/1.0; http://web.toutiao.com/media_cooperation/;)
  Mozilla/5.0 (compatible; BegunAdvertising/3.0; +http://begun.ru/begun/technology/indexer/)
  Netcraft SSL Server Survey - contact info@netcraft.com
  Mozilla/5.0 (compatible; Qualidator.com SiteAnalyzer 1.0;)
  Mozilla/5.0 (compatible; OrangeBot/2.0; support.orangebot@orange.com)
  Mozilla/5.0 (compatible; CloudFlare-AlwaysOnline/1.0; +http://www.cloudflare.com/always-online)
  seebot/1.0.0 (http://www.seegnify.com/bot)
  Slackbot-LinkExpanding 1.0 (+https://api.slack.com/robots)
  Eurobot/1.1 (http://eurobot.ayell.eu)
  Mozilla/5.0 (compatible; IXEbot; +http://medialab.di.unipi.it/IXEbot.html)
  Mozilla/5.0 (GetLinkInfo.com - http://www.getlinkinfo.com)
  Mozilla/5.0 (compatible; spbot/4.0.5; +http://www.seoprofiler.com/bot )
  yacybot (freeworld/global; amd64 Windows 7 6.1; java 1.6.0_31; America/en) http://yacy.net/bot.html
  Mozilla/5.0 (Windows NT 6.1; WOW64; rv:48.0) Gecko/20100101 Firefox/48.0 PTST/314
  Mozilla/5.0 (compatible; AhrefsBot/4.0; +http://ahrefs.com/robot/)
  UXCrawlerBot
  Pixray-Seeker/2.0 (http://www.pixray.com/pixraybot; crawler@pixray.com)
  Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/50.0.2661.94 Safari/537.36 PTST/276
  FAST-WebCrawler/3.7 (atw-crawler at fast dot no; http://fast.no/support/crawler.asp)
  yacybot (freeworld/global; amd64 Windows 10 10.0; java 1.8.0_60; Europe/de) http://yacy.net/bot.html
  Technoratibot/7.0
  Mozilla/4.0 (compatible; MSIE 5.01; Windows NT 5.0; DomainDB-1.1; http://domaindb.com/crawler/)
  Abrave v6.0 (http://robot.abrave.com)
  Testomatobot/1.0 (Linux x86_64; +http://www.testomato.com/testomatobot) minicrawler/4.0.0~beta11
  DowntimeDetector/1.0 (+http://downforeveryoneorjustme.com)
  Comodo Spider 1.1
  Link Valet Online 1.1
  yacybot (freeworld/global; amd64 Windows XP 5.2; java 1.7.0_04; America/en) http://yacy.net/bot.html
  li_viewer (larbin2.6.3@unspecified.mail)
  Mozilla/5.0 (compatible; Mail.RU/2.0c)
  Mozilla/5.0 (compatible; aiHitBot/1.0; +http://www.aihit.com/)
  Mozilla/5.0 (compatible; Uptimebot/0.2.19; +http://www.uptime.com/uptimebot)
  Mozilla/5.0 (compatible; YandexBot/3.0; +http://yandex.com/bots)
  Mozilla/5.0 (iPhone; CPU iPhone OS 6_0 like Mac OS X) AppleWebKit/536.26 (KHTML, like Gecko) Version/6.0 Mobile/10A5376e Safari/8536.25 ( compatible; CloudServerMarketSpider/1.0; +http://cloudservermarket.com/spider.html )
  yacybot (i386 Linux 2.6.28-13-generic; java 1.6.0_13; Europe/en) http://yacy.net/bot.html
  linguatools-bot/Nutch-1.6 (searching for translated pages; http://www.linguatools.de/linguatoolsbot.html; peter dot kolb at linguatools dot org)
  Mozilla/5.0 (compatible; Lippershey/5.2; +https://www.lipperhey.com/en/about/)
  NextGenSearchBot 69 (for information visit http://www.zoominfo.com/business/nextgensearchbot.html)
  Mozilla/5.0 (compatible; YandexWebmaster/2.0; +http://yandex.com/bots)
  Mozilla/5.0 (compatible; Baiduspider/2.0; +http://www.baidu.com/search/spider.html)
  yacybot (freeworld-global; amd64 Linux 3.2.0-35-generic; java 1.7.0_09; Europe/en) http://yacy.net/bot.html
  peerindex/0.1 (http://www.peerindex.com/; crawler AT peerindex DOT com)
  iqdb/0.2 (+http://iqdb.harry.lu/)
  Mozilla/5.0 (compatible; LuminateBot/1.0; +http://www.luminate.com/bot/)
  Mozilla/5.0 (X11; Linux x86_64) adbeat.com/policy AppleWebKit/537.21 (KHTML, like Gecko) Firefox/34.0 Safari/537.21
  AntBot/1.0 (http://www.ant.com)
  Mozilla/5.0 (compatible; Thumbshots.ru; +http://thumbshots.ru/bot) Firefox/3
  Mozilla/5.0 (compatible; value/1.1; +http://www.aihit.com/)
  Digg Deeper/v1 (http://digg.com/about)
  Elmer, the Thinglink ImageBot (http://www.thinglink.com/help/ThinglinkImageBot)
  rogerbot/1.0 (http://moz.com/help/pro/what-is-rogerbot-, rogerbot-crawler+shiny@moz.com)
  Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.1 (KHTML, like Gecko) Chrome/21.0.1180.89 Safari/537.1; http://spyonweb.com spyonweb@gmail.com
  BabalooSpider/1.3 (BabalooSpider; http://www.babaloo.si; spider@babaloo.si)
  Mozilla/5.0 (compatible; dotSemantic/1.0; +http://www.dotsemantic.org)
  TinEye/1.1 (http://tineye.com/crawler.html)
  Mozilla/5.0 (compatible; SemrushBot/0.98~bl; +http://www.semrush.com/bot.html)
  CorpusCrawler 2.0.9 (http://corpora.fi.muni.cz/crawler/)
  aboutthedomain
  Cliqzbot/0.1 (+http://cliqz.com/company/cliqzbot)
  Mozilla/5.0 (compatible; Uptimebot/0.2.43; +http://www.uptime.com/uptimebot)
  LinqiaMetadataDownloaderBot/1.0 (eng@linqia.com)
  yacybot (freeworld/global; amd64 Linux 3.2.0-32-generic; java 1.6.0_24; Europe/en) http://yacy.net/bot.html
  NeumobBot/0.2.16 (+http://www.neumob.com/bot)
  Mozilla/5.0 (compatible; Mp3Bot/0.4; +http://mp3realm.org/mp3bot/)
  Miniflux (https://miniflux.net)
  Mozilla/5.0 (iPhone; CPU iPhone OS 8_3 like Mac OS X) AppleWebKit/537.36 (KHTML, like Gecko; Google Page Speed Insights) Version/8.0 Mobile/12F70 Safari/600.1.4
  Digincore crawler bot. See https://www.digincore.com/crawler.html for rules and instructions.
  Mozilla/5.0 (iPhone; CPU iPhone OS 7_0 like Mac OS X) AppleWebKit/537.51.1 (KHTML, like Gecko) Version/7.0 Mobile/11A465 Safari/9537.53 (compatible; adidxbot/2.0; http://www.bing.com/bingbot.htm)
  NerdyBot
  Nutch/2.2.1 (page scorer; http://integralads.com/site-indexing-policy/)
  Mozilla/5.0 (compatible; 008/0.83; http://www.80legs.com/webcrawler.html;) Gecko/2008032620
  Mozilla/4.0 (compatible; LinkMarket-Bot)
  SeoCheckBot (Seo-Check, http://www.kfsw.de/bot.html)
  Mozilla/5.0 (compatible; Whynder Magnet; http://whynder.com/magnet)
  IdeelaborPlagiaat/1
  x28-job-bot; +http://x28.ch/bot.html
  Mozilla/5.0 (compatible; spbot/4.0.7; +https://www.seoprofiler.com/bot )
  Mozilla/5.0 (compatible; RelateIQ Crawler/0.4; +www.relateiq.com; info@relateiq.com)
  WebRankSpider/1.37 (+http://ulm191.server4you.de/crawler/)
  Mozilla/5.0 (iPhone; U; CPU iPhone OS 10_0 like Mac OS X; en-us) AppleWebKit/602.1.38 (KHTML, like Gecko) Version/10.0 Mobile/14A5297c Safari/602.1 (compatible; Mediapartners-Google/2.1; +http://www.google.com/bot.html)
  LinkWalker/3.0 (http://www.brandprotect.com)
  Mozilla/5.0 (compatible; discobot/1.0; +http://discoveryengine.com/discobot.html)
  ia_archiver-web.archive.org
  Mozilla/5.0 (compatible; Online Domain Tools - SEO Checker/1.0; +http://online-domain-tools.com)
  Mozilla/5.0 (compatible; AhrefsBot/3.0; +http://ahrefs.com/robot/)
  Mozilla/5.0 (Macintosh; Intel Mac OS X 10_6_8) AppleWebKit/537.13+ (KHTML, like Gecko) Version/5.1.7 Safari/534.57.2; Shoppimon Analyzer (http://www.shoppimon.com/)
  Mozilla/5.0 (compatible; NewShareCounts.com/1.0; +http://newsharecounts.com/crawler)
  Mozilla/5.0 (compatible; Mozilla; +http://wiki.github.com/bixo/bixo/bixocrawler; bixo-dev@yahoogroups.com)
  Mozilla/5.0 (compatible; CukBot; Not a spammer; http://www.companiesintheuk.co.uk/bot.html)
  yacybot (freeworld/global; amd64 Linux 3.1.10-1.9-default; java 1.6.0_24; Europe/de) http://yacy.net/bot.html
  Nuhk/2.4 (+http://www.neti.ee/cgi-bin/abi/otsing.html)
  elefent/Elefent 1.1 (Friendly web walker.; http://www.elefent.eu/; jason@elefent.eu)
  HubSpot Links Crawler 1.0 http://www.hubspot.com/
  Mozilla/5.0 (compatible; Siteluxbot/1.0; +https://sitelux.co.uk/)
  CopperEgg/RevealUptime/AmsterdamNL(softlayer)
  Mozilla/5.0 (compatible; spbot/4.0.7; +http://OpenLinkProfiler.org/bot )
  Mediumbot-MetaTagFetcher/0.1 (+https://medium.com/)
  Mozilla/2.0 (compatible; Ask Jeeves/Teoma; +http://about.ask.com/en/docs/about/webmasters.shtml)
  Mozilla/5.0 (compatible; 008/0.85; http://www.80legs.com/webcrawler.html) Gecko/2008032620
  yacybot (/global; amd64 Linux 4.4.0ct; java 1.7.0_91; Europe/de) http://yacy.net/bot.html
  Mozilla/5.0 (compatible; Scrubby/2.2; http://www.scrubtheweb.com/)
  yacybot (/global; amd64 Linux 3.2.0-4-amd64; java 1.8.0_05; Europe/en) http://yacy.net/bot.html
  Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.2.18) Gecko/20110622 HeartRails_Capture/1.0.4 (+http://capture.heartrails.com/) Namoroka/3.6.18
  Mozilla/5.0 (X11; U; Linux i686; de; rv:1.9.0.1; compatible; iCjobs Stellenangebote Jobs; http://www.icjobs.de) Gecko/20100401 iCjobs/3.2.3
  KOCMOHABT (+http://www.KOCMOHABT.tk/) AppEngine-Google; (+http://code.google.com/appengine; appid: s~web-explorer)
  yacybot (freeworld/global; x86 Windows 7 6.1; java 1.6.0_31; Europe/de) http://yacy.net/bot.html
  Mozilla/5.0 (compatible; MSIE 10.0; Windows NT 6.2; WOW64; Trident/6.0) CrawlerProcess (http://www.PowerMapper.com) /5.21.754.0
  Mozilla/5.0 (compatible; Linux x86_64; Mail.RU_Bot/Robots; +http://go.mail.ru/help/robots)
  Mozilla/5.0 (compatible; Googlebot/2.1; +http://import.io)
  ownCloud Server Crawler
  Mozilla/5.0 (compatible; Qwantify/2.1n; +https://www.qwant.com/)/*
  LoadImpactRload/3.1.5 (Load Impact; http://loadimpact.com);
  Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 4.0; Girafabot; girafabot at girafa dot com; http://www.girafa.com)
  OmniExplorer_Bot/6.65a (+http://www.omni-explorer.com) WorldIndexer
  Mozilla/5.0 (compatible; SMTBot/1.0; +http://www.similartech.com/smtbot)
  anonymous coward/1.0 (a Storm-based crawler; https://github.com/DigitalPebble/storm-crawler; someone@company.com)
  Mozilla/5.0 (compatible; Uptimebot/1.0; +http://www.uptime.com/uptimebot)
  Mozilla/5.0 (compatible; FoundSeoTool/1.0; +http://www.found.co.uk/seo-tool/)
  findlinks/2.1.5 (+http://wortschatz.uni-leipzig.de/findlinks/)
  yacybot (freeworld/global; amd64 Windows NT (unknown) 6.2; java 1.7.0_05; Africa/en) http://yacy.net/bot.html
  Mozilla/5.0 (compatible; memorybot/1.20.71 +http://archivethe.net/en/index.php/about/internet_memory1 on behalf of DNB)
  ownCloud News/5.3.6 (+https://owncloud.org/; 1 subscriber;)
  YottaBot Spider/Nutch-2.3
  Mozilla/5.0 (compatible; WeViKaBot/1.0; +http://www.wevika.de/)
  ThumbShots-Bot (+http://thumbshots.in/bot.html)
  Mnogosearch-3.1.21
  Favicon downloader (+https://favico.be/)
  Mozilla/5.0 (compatible; Page2RSS/0.7; +http://page2rss.com/)
  ExB Language Crawler 2.1.2 (+http://www.exb.de/crawler)
  PrivateSearch/0.1.0 (Polite Nutch Crawler; grierforensics.com)
  nodestackbot/0.1 (bot@nodestack.com http://nodestack.com/bot.html)
  iskanie (+http://www.iskanie.com)
  yacybot (/global; amd64 Windows 7 6.1; java 1.7.0_67; Europe/de) http://yacy.net/bot.html
  Mozilla/5.0 (compatible; Dataprovider Site Explorer; http://www.dataprovider.com/)
  Mozilla/5.0 (compatible; akula/12.0rc-2; +http://k311.fd.cvut.cz/)
  Mozilla/5.0 (compatible; bnf.fr_bot; +http://www.bnf.fr/fr/outils/a.dl_web_capture_robot.html)
  Mozilla/5.0 (iPhone; CPU iPhone OS 6_0 like Mac OS X) AppleWebKit/536.26 (KHTML, like Gecko) Version/6.0 Mobile/10A5376e Safari/8536.25 (compatible; Googlebot-Mobile/2.1; +http://www.google.com/bot.html)
  Mozilla/5.0 (compatible; MSIE 10.0; Windows NT 6.1; WOW64; Trident/6.0) PTST/284
  Mozilla/5.0 (compatible; Infohelfer/1.3.3; +http://www.infohelfer.de/crawler.php)
  www.deadlinkchecker.com XMLHTTP/1.0
  Mozilla/5.0 (compatible; TweetmemeBot/3.0; +http://tweetmeme.com/)
  Chrome/49.0.2623.87 (compatible; Wappalyzer; +https://github.com/AliasIO/Wappalyzer)
  Mozilla/5.0 (compatible; MSIE 10.0; Windows NT 6.2; WOW64; Trident/6.0) CrawlerProcess (http://www.PowerMapper.com) /5.11.740.0
  XING-contenttabreceiver/2.0
  Mozilla/5.0 (iPhone; CPU iPhone OS 6_0 like Mac OS X) AppleWebKit/536.26 (KHTML, like Gecko) Version/6.0 Mobile/10A5376e Safari/8536.25 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
  Mozilla/5.0 (compatible; MojeekBot/0.2; http://www.mojeek.com/bot.html#relaunch)
  Pingoscope
  Mozilla/5.0 (compatible; um-LN/1.0; mailto: techinfo@ubermetrics-technologies.com)
  Mozilla/5.0 (compatible; Pagespeed/1.1 Fetcher; +http://www.pagespeed.de)
  HTTP-Header-Abfrage/1.0 (http://www.internalscripts.de/werkzeuge/http-header-abfrage.php)
  Mozilla/5.0 (compatible; 80bot/0.71; http://www.80legs.com/spider.html;) Gecko/2008032620
  LoadImpactRload/3.0.3 (Load Impact; http://loadimpact.com);
  Mozilla/5.0 (compatible; Peepowbot/1.0; +http://www.peepow.com/bot.php)
  yacybot (freeworld/global; amd64 Linux 2.6.32-5-amd64; java 1.6.0_26; Atlantic/en) http://yacy.net/bot.html
  WatchMouse/8.4.0.3 (http://watchmouse.com/ ; grath01.watchmouse.net)
  Mozilla/5.0 (compatible; +http://servernfo.com/)
  Mozilla/5.0 (compatible; special_archiver/3.1.1 +http://www.archive.org/details/archive.org_bot)
  CopperEgg/RevealUptime/NewarkNJUSA
  Mozilla/5.0 (compatible; archive.org_bot/3.3.0 +http://www.archive.org/details/archive.org_bot)
  Mozilla/5.0 (compatible; Plukkie/1.1; http://www.botje.com/plukkie.htm)
  Mozilla/5.0 (compatible; musobot/1.0; info@muso.com; +http://www.muso.com)
  yacybot (freeworld/global; amd64 Linux 3.10.17-gentooxen; java 1.6.0_31; Europe/nb) http://yacy.net/bot.html
  ResponseCodeTest/1.1
  Mozilla/5.0 (compatible; MSIE 10.0; Windows NT 6.1; Trident/6.0) SiteCheck-sitecrawl by Siteimprove.com
  Acoon v4.9.5 (www.acoon.de)
  Mozilla/5.0 (Windows NT 6.1; WOW64) SkypeUriPreview Preview/0.5
  SiteGuardian/2.0 (Internet Monitoring)
  Mozilla/5.0 (compatible; DCPbot/1.4; +http://domains.checkparams.com/)
  BoardReader Favicon Fetcher /1.0 info@boardreader.com
  baypup/colbert (Baypup; http://www.baypup.com/webmasters; jason@baypup.com)
  Mozilla/5.0 (compatible; MonTools.com)
  yacybot (freeworld/global; amd64 Linux 3.2.0-4-amd64; java 1.7.0_03; Etc/de) http://yacy.net/bot.html
  Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/534.51 (KHTML, like Gecko; Google Web Preview) Chrome/12.0.742 Safari/534.51
  linkapediabot (+http://www.linkapedia.com)
  Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.7.12; ips-agent) Gecko/20050922 Fedora/1.0.7-1.1.fc4 Firefox/1.0.7
  BCKLINKS 1.0
  CorpusCrawler 2.0.21 (http://corpora.fi.muni.cz/crawler/);Project:CzCorpus
  Mozilla/5.0 (compatible; Yeti/1.1; +http://help.naver.com/support/robots.html)
  Mozilla/5.0 (compatible; pingbot/2.0; +http://www.pingdom.com/)
  Mozilla/5.0 (compatible; memoryBot/1.20.275 +http://internetmemory.org/en/)
  Mozilla/5.0 (compatible; spbot/1.1; +http://www.seoprofiler.com/bot/ )
  yacybot (/global; amd64 Linux 3.13.0-88-generic; java 1.7.0_101; Europe/en) http://yacy.net/bot.html
  BlogSearch/2 +http://www.icerocket.com/
  Mozilla/5.0 (compatible; LoadTimeBot/0.81; +http://www.load-time.com/bot.html)
  Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9) Gecko/2008052906 Firefox/3.0 (BrowserSpyBot/1.0) (http://browserspy.dk/)
  Mozilla/5.0 (compatible; MojeekBot/0.6; +https://www.mojeek.com/bot.html)
  CopperEgg/RevealUptime/NewarkNJ(linode)
  Automattic Analytics Crawler/0.2; http://wordpress.com/crawler/
  larbin_2.6.3 capveg@cs.umd.edu
  HatenaScreenshot
  CopperEgg/RevealUptime/LondonUK
  Mozilla/5.0 (compatible; MojeekBot/0.6; http://www.mojeek.com/bot.html)
  Mozilla/5.0 (compatible; evc-batch/2.0.20160902205649)
  yacybot (freeworld/global; amd64 Linux 3.2.0-23-generic; java 1.6.0_24; Europe/en) http://yacy.net/bot.html
  WordPress.com;
  Mozilla/5.0 (Windows NT 6.1; rv:6.0) Gecko/20110814 Firefox/6.0 Google (+https://developers.google.com/+/web/snippet/)
  Mozilla/5.0 (compatible; ScoutJet; http://www.scoutjet.com/)
  Mozilla/5.0 (compatible; Uptimebot/0.1.97; +http://www.uptime.com/uptimebot)
  WebTarantula.com Crawler
  WatchMouse/8.4.0.3 (http://watchmouse.com/ ; itmil01.watchmouse.net)
  LinqiaScrapeBot/1.0 (eng@linqia.com)
  Orbiter (+http://www.dailyorbit.com/bot.htm)
  Mozilla/5.0 (compatible; janforman.com/2.5; +http://janforman.com/)
  InternetSeer.com
  Mozilla/5.0 (compatible; aiHitBot/2.9; +http://www.aihitdata.com/about)
  Mozilla/5.0 (compatible; WebNL; +http://www.web.nl/webmasters/spider.html; helpdesk@web.nl)
  coccoc/1.0 (http://help.coccoc.vn/)
  Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/53.0.2785.116 Safari/537.36 PTST/314
  Mozilla/5.0 (compatible; Googlebot/2.1; startmebot/1.0; +https://start.me/bot)
  Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 5.2; Trident/4.0; .NET CLR 1.1.4322; .NET CLR 2.0.50727; .NET CLR 3.0.4506.2152; .NET CLR 3.5.30729; .NET4.0C; .NET4.0E; PTST 2.386)
  Mozilla/5.0 (compatible; PrivacyAwareBot/1.1; +http://www.privacyaware.org)
  Mozilla/5.0 (compatible; idmarch Automatic.beta/1.4; +http://www.idmarch.org/bot.html)
  Mozilla/5.0 (compatible; tldstat.com/Bot; +http://tldstat.com/bot.html)
  yacybot (/global; amd64 Linux 3.16.0-4-amd64; java 1.7.0_111; Europe/de) http://yacy.net/bot.html
  Mozilla/5.0 (compatible; Crawlera/1.10.2; UID 84846)
  Mozilla/5.0 (compatible; spbot/4.0a; +http://www.seoprofiler.com/bot )
  Mozilla/5.0 (Windows NT 6.1; Win64; x64; +http://www.komodia.com/newwiki/index.php/URL_server_crawler) KomodiaBot/1.0
  WebarooBot (Webaroo Bot; http://64.124.122.252/feedback.html)
  OOZBOT/0.20 ( -- ; http://www.setooz.com/oozbot.html ; agentname at setooz dot_com )
  Mozilla/5.0 (compatible; memoryBot/1.21.16 +http://internetmemory.org/en/)
  FAST-WebCrawler/3.8
  Googlebot-Image/1.0
  enlle punto com/Nutch-1.9
  OpenAcoon v4.10.5 (www.openacoon.de)
  yacybot (freeworld/global; amd64 Linux 3.0.0-21-generic; java 1.6.0_26; America/en) http://yacy.net/bot.html
  Microsearch.ru Bot 1.3;http://microsearch.ru/webmasters
  Mozilla/5.0 (compatible; Google-Site-Verification/1.0)
  Mozilla/5.0 (compatible; Qwantify/2.3w; +https://www.qwant.com/)/2.3w
  rogerbot/1.1 (http://moz.com/help/guides/search-overview/crawl-diagnostics#more-help, rogerbot-crawler+pr4-crawler-11@moz.com)
  HubSpot Connect 2.0 (http://dev.hubspot.com/)
  Robozilla/1.0
  Mozilla/5.0 (compatible; spbot/2.0.3; +http://www.seoprofiler.com/bot/ )
  Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/53.0.2785.116 Safari/537.36 PTST/314
  facebookexternalhit/1.1
  PercolateCrawler/3.1.30 (ops@percolate.com)
  Mozilla/5.0 (compatible; Nmap Scripting Engine; https://nmap.org/book/nse.html)
  Mozilla/5.0 (compatible; pdffillerbot/1.0; +http://www.pdffiller.com/en/support.htm)
  fastbot crawler beta 4.0 (+http://www.fastbot.de)
  SafeDNS search bot/Nutch-1.9 (https://www.safedns.com/searchbot; support [at] safedns [dot] com)
  Mozilla/5.0 (Nekstbot; http://nekst.ipipan.waw.pl/nekstbot/)
  yacybot (freeworld/global; amd64 Linux 2.6.32-5-amd64; java 1.6.0_18; US/en) http://yacy.net/bot.html
  yacybot (freeworld/global; amd64 Linux 3.0.0-12-generic; java 1.6.0_26; Europe/en) http://yacy.net/bot.html
  coccoc/1.0 ()
  larbin_2.6.3 (atyzos@yahoo.com)
  yacybot (i386 Linux 2.6.26-2-686; java 1.6.0_0; Europe/de) http://yacy.net/bot.html
  Mozilla/5.0 (Windows NT 6.1; compatible; BDCbot/1.0; +http://ecommerce.bigdatacorp.com.br/faq.aspx) AppleWebKit/535.1 (KHTML, like Gecko) Chrome/13.0.782.107 Safari/535.1
  yacybot (amd64 Linux 2.6.31-22-server; java 1.6.0_18; Asia/en) http://yacy.net/bot.html
  yacybot (/global; amd64 Windows Server 2012 6.2; java 1.8.0_91; Europe/de) http://yacy.net/bot.html
  Mozilla/5.0 (compatible; Netseer crawler/2.0; +http://www.netseer.com/crawler.html; crawler@netseer.com)
  Mozilla/5.0 (compatible; heritrix/3.1.1 +http://www.mixdata.com)
  Scrapy/0.25.1 (+http://scrapy.org)
  Mozilla/5.0 (compatible; Scopia Crawler 1.2; +http://www.scopia.co)
  cn_ab_viewer (larbin2.6.3@unspecified.mail)
  Mozilla/5.0 (Windows NT 6.3; Trident/7.0; rv 11.0) like Gecko (compatible; Zombiebot/2.1; +http://www.zombiedomain.net/robot/)
  Mozilla/5.0 (compatible; Uptimebot/0.1.38; +http://www.uptime.com/uptimebot)
  yacybot (freeworld/global; amd64 Windows 7 6.1; java 1.7.0_25; Europe/de) http://yacy.net/bot.html
  Python-urllib/2.7 (+http://sketch.juls.savba.sk/aranea_about)
  Mozilla/5.0 (compatible; spbot/4.0; +http://www.seoprofiler.com/bot )
  yacybot (-global; amd64 Windows 8 6.2; java 1.7.0_55; Europe/de) http://yacy.net/bot.html
  yacybot (/global; amd64 Linux 3.2.41-042stab094.7; java 1.7.0_79; Europe/en) http://yacy.net/bot.html
  Mozilla/5.0 (compatible; UASlinkChecker/1.0; +http://user-agent-string.info/UASlinkChecker)
  Y!J-BRO/YFSJ crawler (compatible; Mozilla 4.0; MSIE 5.5; http://help.yahoo.co.jp/help/jp/search/indexing/indexing-15.html; YahooFeedSeekerJp/2.0)
  yacybot (freeworld/global; amd64 Linux 3.10.15-1-MANJARO; java 1.7.0_40; Asia/en) http://yacy.net/bot.html
  Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.2; WOW64; Trident/6.0; .NET4.0E; .NET4.0C; .NET CLR 3.5.30729; .NET CLR 2.0.50727; .NET CLR 3.0.30729) CrawlerProcess (http://www.PowerMapper.com) /5.9.726.0
  Scrubby/2.1 (http://www.scrubtheweb.com/)
  Mozilla/5.0 (compatible; JobKeresoBot; +http://www.kozvetlen-allasok.hu/help.jsp; info@kozvetlen-allasok.hu; 5.0
  yacybot (/global; amd64 Windows 7 6.1; java 1.8.0_66; Europe/de) http://yacy.net/bot.html
  Mozilla/5.0 (X11; Linux x86_64 UXCrawlerBot http://zium.co)
  Slackbot-LinkExpanding (+https://api.slack.com/robots)
  yacybot (freeworld/global; amd64 Linux 3.12.15-gentoo; java 1.7.0_55; Europe/en) http://yacy.net/bot.html
  Mozilla/5.0 (compatible; LinkpadBot/1.07; +http://www.linkpad.ru)
  Mozilla/5.0 (compatible; ZB-1; +http://www.zeerch.com/bot.php)
  Mozilla/5.0 (compatible; WbSrch/1.1 +http://wbsrch.com)
  WeSEE:Search
  Mozilla/5.0 (compatible; DCPbot/1.1; +http://domains.checkparams.com/)
  yacybot (webportal-global; amd64 Linux 2.6.32-5-amd64; java 1.6.0_45; Europe/en) http://yacy.net/bot.html
  WordPress.com mShots; http://support.wordpress.com/contact/
  Mozilla/4.0 (compatible; MSIE 5.01; Windows NT 5.0; FeedFinder-2.0; http://bloggz.se/crawler)
  Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.6; en-US; rv:1.9.2) Gecko/20100115 Firefox/3.6 (FlipboardProxy/0.0.5; +http://flipboard.com/browserproxy)
  Mozilla/5.0 (compatible; coccocbot-image/1.0; +http://help.coccoc.com/searchengine)
  Mozilla/5.0 (X11; Linux x86_64; GTmetrix http://gtmetrix.com/) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/31.0.1650.63 Safari/537.36
  https_page_test (larbin2.6.3@unspecified.mail)
  Mozilla/5.0 (Windows; U; Windows NT 5.1; zh-CN; rv:1.8.0.11) Firefox/1.5.0.11 360Spider;
  yacybot (freeworld/global; amd64 Linux 3.2.0-2-amd64; java 1.6.0_24; Europe/en) http://yacy.net/bot.html
  Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_2) AppleWebKit/600.1.4 (KHTML, like Gecko) Safari/600.1.4 (compatible; Princetonbot/1.0; +http://http://tigress-web.princeton.edu/~fy/bot.html)
  Mozilla/5.0 (Windows; U; Windows NT 5.1; zh-CN; rv:1.8.0.11) Gecko/20070312 Firefox/1.5.0.11; 360Spider
  KeywordDensityRobot/0.8 (http://www.seocentro.com/tools/search-engines/keyword-density.html)
  Mozilla/5.0 (compatible; aiHitBot-BP/1.1; +http://www.aihit.com/)
  findlinks/1.1.6-beta4 (+http://wortschatz.uni-leipzig.de/findlinks/)
  Mozilla/5.0 (compatible; Butterfly/1.0; +http://labs.topsy.com/butterfly/) Gecko/2009032608 Firefox/3.0.8
  CopperEgg/RevealUptime/FremontCA(linode)
  Mozilla/5.0 (compatible; NTENTbot; +http://www.ntent.com/ntentbot)
  link_thumbnailer
  KeywordDensityRobot/0.9 (http://www.seocentro.com/tools/search-engines/keyword-density.html)
  Mozilla/5.0 (compatible;WI Job Roboter Spider Version 3;+http://www.webintegration.at)
  Mozilla/5.0 (compatible; Mail.RU/2.0)
  CopperEgg/RevealUptime/N.California(aws)
  Searchie Bot/1.0 (a Storm-based crawler; https://www.searchie.org; admin@searchie.org)
  ^Nail (http://CaretNail.com)
  MarketBrewBot/2.1 (+http://www.marketbrew.com/)
  Updownerbot (+http://www.updowner.com/bot)
  yacybot (-global; amd64 Windows 8.1 6.3; java 1.7.0_55; Europe/de) http://yacy.net/bot.html
  Yanga WorldSearch Bot v1.1/beta (http://www.yanga.co.uk/)
  Mozilla/5.0 (compatible; bixolabs/1.0; +http://bixolabs.com/crawler/general; crawler@mail.bixolabs.com)
  Mozilla/5.0 (compatible; Scopia Crawler 2.0; +http://www.scopia.co)
  asafaweb.com
  KrOWLer/0.0.1, matentzn at cs dot man dot ac dot uk
  Mozilla/5.0 (compatible; alexa site audit/1.0; +http://www.alexa.com/help/webmasters; siteaudit@alexa.com)
  findlinks/1.0 (+http://wortschatz.uni-leipzig.de/findlinks/)
  SygolBot http://www.sygol.com
  voltron
  Mozilla/5.0 (compatible; uMBot-FC/1.0; mailto: crawling@ubermetrics-technologies.com)
  Comodo-Webinspector-Crawler 2.2.1, http://www.comodorobot.com
  Superarama.com-Tarama-Botu-v.0.1
  yacybot (freeworld/global; amd64 Linux 3.2.0-3-amd64; java 1.6.0_24; Europe/en) http://yacy.net/bot.html
  Mozilla/5.0 (compatible; SSLBot/1.0; http://www.sslstats.com/sslbot)
  PagesInventory.com (robot http://www.pagesinvenotry.com)
  Mozilla/5.0 (iPhone; U; CPU iPhone OS 4_3 like Mac OS X; en-US) adbeat.com/policy AppleWebKit/533.17.9 (KHTML, like Gecko) Version/5.0.2 Mobile/8F190 Safari/6533.18.5
  WordPress.com Reader
  Mozilla/5.0 (compatible; SEOkicks-Robot +http://www.seokicks.de/robot.html)
  Woko 3.0
  Gigabot/3.0 (http://www.gigablast.com/spider.html)
  ichiro/4.0 (http://help.goo.ne.jp/door/crawler.html)
  bl.uk_lddc_bot/3.1.1 (+http://www.bl.uk/aboutus/legaldeposit/websites/websites/faqswebmaster/index.html)
  Urlstat (http://bit.ly/1dJuuzs)
  yacybot (/global; amd64 Windows 8.1 6.3; java 1.7.0_55; America/ru) http://yacy.net/bot.html
  Bad Neighborhood Header Detector (http://www.bad-neighborhood.com/header_detector.php)
  Mozilla/5.0 (compatible; GeliyooBot/1.0beta; +http://www.geliyoo.com/)
  NalezenCzBot/1.0 (http://www.nalezen.cz/about-crawler)
  WeSEE_Bot:we_help_monitize_your_site (http://www.wesee.com/bot/)
  yacybot (freeworld/global; x86 Windows XP 5.1; java 1.7.0_55; Asia/en) http://yacy.net/bot.html
  studylib download bot, see http://studylib.es/public/bot for additional info
  SemrushBot/0.91
  Jabse.com Crawler v.1.0 www.jabse.com/crawler.php//imagecrawler
  Mozilla/4.0 (compatible; MSIE 5.0; Windows NT; Girafabot; girafabot at girafa dot com; http://www.girafa.com)
  BizwikiBot/0.3 (http://www.bizwiki.com/bizwikibot.htm)
  Mozilla/5.0 (compatible; BIXOCRAWLER; +http://wiki.github.com/bixo/bixo/bixocrawler; bixo-dev@yahoogroups.com)
  PostPost/1.0 (+http://postpost.com/crawlers)
  Mozilla/5.0 (nomore404.com robot/1.1; +https://nomore404.com/)
  Riddler (http://riddler.io/about)
  Mozilla/5.0 (compatible; socketcrawler; http://nlp.fi.muni.cz/projects/biwec/)
  internetVista monitor (Mozilla compatible)
  2Bone_LinkChecker/1.0 libwww-perl/6.03
  MaxPointCrawler/Nutch-1.1 (maxpoint.crawler at maxpointinteractive dot com)
  DomainStatsBot/1.0 (http://domainstats.io/our-bot)
  KDDI-CA31 UP.Browser/6.2.0.7.3.129 (GUI) MMP/2.0 (compatible; ichiro/mobile goo;+http://search.goo.ne.jp/option/use/sub4/sub4-1/)
  Mergadobot/3.0.2 (+http://mergado.cz)
  DoCoMo/2.0 P900i(c100;TB;W24H11) (compatible; ichiro/mobile goo; +http://search.goo.ne.jp/option/use/sub4/sub4-1/)
  SalesIntelligent/v1.1
  SiteAnalyzerBot
  DoCoMo/2.0 P900i(c100;TB;W24H11) (compatible; ichiro/mobile goo; +http://help.goo.ne.jp/help/article/1142/)
  yacybot (/global; i386 Linux 3.16.3-gentoo; java 1.7.0_55; Australia/en) http://yacy.net/bot.html
  Mozilla/5.0 (compatible; Linux x86_64; Mail.RU_Bot/Img/2.0; +http://go.mail.ru/help/robots)
  Cliqz Bot (+http://www.cliqz.com)
  SalesIntelligent (+https://www.salesintelligent.com)
  Mozilla/5.0 (compatible; memoryBot/1.20.243 +http://internetmemory.org/en/)
  gimme60 (Gimme60 Store ID Bot; gimme60.com)
  Orbiter/1.0 (http://dailyorbit.com/)
  msnbot/1.1 (+http://search.msn.com/msnbot.htm)
  Mozilla/5.0 (compatible; memoryBot/1.20.241 +http://internetmemory.org/en/)
  Mozilla/5.0 (compatible; akula/k311; +http://k311.fd.cvut.cz/)
  Irokez.cz monitoring v1.2 - (http://www.irokez.cz, Irokez.cz, crawl)
  CCBot/2.0 (http://commoncrawl.org/faq/)
  Mozilla/5.0 (compatible; openstat.ru/Bot)
  Mozilla/5.0 (compatible; GigaBot/1.0; +http://www.gigablast.com/ )
  Readability/6a54d4 - http://readability.com/about/
  percbotspider <ops@percolate.com>
  Mozilla/5.0 (compatible; spbot/1.0; +http://www.seoprofiler.com/bot/ )
  HatenaScreenshot/1.0 (checker)
  Mozilla/5.0 (compatible; AcoonBot/4.12.1; +http://www.acoon.de/robot.asp)
  CRAZYWEBCRAWLER 0.9.2, http://www.crazywebcrawler.com
  Mozilla/5.0 (compatible; Online Domain Tools - Online Sitemap Generator/1.1; +http://sitemap-generator.online-domain-tools.com)
  Mozilla/5.0 (compatible; Google-Structured-Data-Testing-Tool +https://search.google.com/structured-data/testing-tool)
  OnPageBot (compatible; Googlebot 2.1; +https://bot.onpage.org/)
  Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/48.0.2564.103 Safari/537.36 Yandex.Translate
  Sogou web spider/4.0
  Mozilla/5.0 (compatible; YandexMetrika/2.0; +http://yandex.com/bots DEV)
  LinkWalker/2.0
  SeoCheck (FischerNetzDesign Seo Checker, info@fischernetzdesign.de)
  yacybot (freeworld/global; amd64 Linux 3.2.13-1-ARCH; java 1.7.0_03-icedtea; Europe/fr) http://yacy.net/bot.html
  SiteCheck - http://downforeveryoneorjustme.com
  Robots_Tester_http_www.searchenginepromotionhelp.com
  CRAZYWEBCRAWLER 0.9.4, http://www.crazywebcrawler.com
  Mozilla/5.0 (compatible; heritrix/3.1.1 +http://places.tomtom.com/crawlerinfo)
  Mozilla/5.0 (Windows; U; Windows NT 5.0; en-US; rv:1.7.10) Gecko/20050716 Thunderbird/1.0.6 - WebCrawler http://cognitiveseo.com/bot.html
  Mozilla/5.0 (compatible; Cliqzbot/1.0 +http://cliqz.com/company/cliqzbot)
  yacybot (/global; amd64 Linux 4.2.0-gentoo-r1; java 1.8.0_45; Australia/en) http://yacy.net/bot.html
  DomainsDB.net MetaCrawler v.0.9.7c (http://domainsdb.net/)
  Mozilla/4.0 (compatible; MSIE 5.5; Windows NT 5.0; mowser; http://www.mowser.com)
  www.probethenet.com scanner
  Mozilla/5.0 (compatible; Crawlera/1.10.2; UID 6573)
  LCC (+http://corpora.informatik.uni-leipzig.de/crawler_faq.html)
  Imagga Downloader/v0.14.3
  SeopultContentAnalyzer/1.0
  Mozilla/5.0 (compatible; +http://www.hosterstats.com/newgtld-websurvey.php)
  Mozilla/5.0 (compatible; GoogleDocs; apps-spreadsheets; +http://docs.google.com)
  Mozilla/5.0 (compatible;acapbot/0.1)
  Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1;Alibaba.Security.Heimdall)
  HTTPClient/1.0 (2.8.0, ruby 2.0.0 (2014-05-08))
  Jetty/9.3.z-SNAPSHOT
  Jetty/9.3.z-SNAPSHOT, SMUrlExpander
  Jetty/9.3.z-SNAPSHOT, SMViewer
  Liferea/0.x.x (Linux; en_US.UTF-8; http://liferea.sf.net/)
  SynHttpClient-Built/5825
  asynchttp
  SMUrlExpander
  KickFire
  HAAARTLAND http client
  PEAR HTTPRequest class ( http://pear.php.net/ )
  Mozilla/4.0 (vBSEO; http://www.vbseo.com)
  HAARTLAND http client
  JS-Kit URL Resolver http://js-kit.com/
  http://www.checkprivacy.or.kr:6600/RS/PRIVACY_ENFAQ.jsp
  Kml-Google; (+http://code.google.com/apis/kml) gzip
  gvfs/1.20.3
  request.js
  Microsoft URL Control - 6.01.9782
  git/2.1.4
  IPS/1.0
  OxfordCloudService/1.0
  Mozilla/5.0 (compatible; informatique at mixdata dot com/0.6; +http://t.co/GSRLLKex24; informatique at mixdata dot com)
  Mozilla/5.0 (compatible; Feedwind/2.0; +http://feed.mikle.com/support/description/)
  Mozilla/5.0 (compatible; Feedwind/3.0; +http://feed.mikle.com/support/description/)
  Mozilla/5.0 (Macintosh; Intel Mac OS X 1068) AppleWebKit/534.50 (KHTML like Gecko) Version/5.1 Instapaper/4.0 (+http://www.instapaper.com/)
  Mozilla/5.0 (compatible; Keywords Research; http://www.check-domains.com/keyword-research/keywords-adwords.php)
  CAAM/0.1 (CAAM; caam dot crwlr at gmail dot com)
  i2kconnect/"1.7.0"
  resolver/0.1.12 (+http://github.com/andris9/resolver)
  DangDang
  Mozilla/5.0 (compatible; Tetrahedron/1.0; +https://compute.info/tet)
  PritTorrent/0.1
  FeedBunch/1.0 (+http://www.feedbunch.com)
  Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.21 (KHTML, like Gecko) www.HomepageScreenshot.de - Free ScreenShotService/1.0 Safari/537.21
  Mozilla/5.0 (compatible; Licorne Image Snapshot 1.0; +http://www.jednorozec.cz)
  Pinterest/0.2 (+https://www.pinterest.com/bot.html)
  Pinterest/0.1 (+https://www.pinterest.com/bot.html)
  Mozilla/5.0 (iplabel; Windows NT 6.3; WOW64; rv:33.0) Gecko/20100101 Firefox/33.0
  Mozilla/5.0 zgrab/0.x
  Zend_Http_Client
  Zend\Http\Client
  Zao/0.1 (http://www.kototoi.org/zao/)
  YOURLS v1.7.1 +http://yourls.org/ (running on http://flnx.co)
  yoogliFetchAgent/0.1
  Yo-yo
  Mozilla/5.0 (compatible; Yahoo Link Preview; https://help.yahoo.com/kb/mail/yahoo-link-preview-SLN23615.html)
  Mozilla/5.0 (compatible; Yahoo Ad monitoring; https://help.yahoo.com/kb/yahoo-ad-monitoring-SLN24857.html)
  Mozilla/5.0 (iPhone; CPU iPhone OS 7_1 like Mac OS X) AppleWebKit (KHTML, like Gecko) Mobile (compatible; Yahoo Ad monitoring; https://help.yahoo.com/kb/yahoo-ad-monitoring-SLN24857.html)
  xpymep.exe
  xpymep2.exe
  xpymep3.exe
  Xenu Link Sleuth 1.2b
  Xenu Link Sleuth 1.2c
  Xenu Link Sleuth 1.2d
  Xenu Link Sleuth 1.2e
  Xenu Link Sleuth 1.2f
  Xenu Link Sleuth 1.2g
  Xenu Link Sleuth 1.2h
  Xenu Link Sleuth 1.2i
  Xenu Link Sleuth/1.3.7
  Xenu Link Sleuth/1.3.8
  Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/40.0.2214.93 Safari/537.36 XaxisSemanticsClassifier/1.0 http://crystalsemantics.com
  WWW-Mechanize/1.73
  WordPress/4.0.1; http://bestcomp.hol.es
  WordPress/4.3.3; http://homebusinessreviews.online
  WordPress/4.3.3; http://legithomejobs.club
  WordPress/4.4.2; http://cheapautoinsuranceinva.com
  WordPress/4.4.2; http://citycollegeinmiami.com
  WordPress/4.4.2; http://cookingideabook.com
  WordPress/4.4.2; http://oldmanhealth.com
  WordPress/4.4.2; http://sapcrmarchitecture.com
  WordPress/4.4.2; http://www.spar3d.com
  WordPress/4.5.2; http://1.we-careu.xyz
  WordPress/x.x.x.x PHP/4.x.xxMozilla/5.0 (Unknown; Linux x86_64) AppleWebKit/534.34 (KHTML, like Gecko) PhantomJS/1.6.1 Safari/534.34
  Mozilla/5.0 (Macintosh; Intel Mac OS X) Word/14.0.0
  Mozilla/5.0 (Macintosh; Intel Mac OS X) Word/14.25.0
  Mozilla/5.0 (Macintosh; Intel Mac OS X) Word/14.47.0
  Mozilla/5.0 (Macintosh; Intel Mac OS X) Word/14.48.0
  Mozilla/5.0 (Macintosh; Intel Mac OS X) Word/14.50.0
  Mozilla/5.0 (Macintosh; Intel Mac OS X) Word/14.56.0
  Mozilla/5.0 (Macintosh; Intel Mac OS X) Word/14.61.0
  Mozilla/5.0 (Macintosh; Intel Mac OS X) Word/14.62.0
  Mozilla/5.0 (Macintosh; Intel Mac OS X) Word/14.63.0
  Mozilla/5.0 (Macintosh; Intel Mac OS X) Word/14.64.0
  WomlpeFactory/0.1 (+http://www.Womple.com/bot.html)
  Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/534.34 (KHTML, like Gecko) wkhtmltoimage Safari/534.34
  Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/534.34 (KHTML, like Gecko) wkhtmltopdf Safari/534.34
  Mozilla/4.0 (compatible; Win32; WinHttp.WinHttpRequest.5)
  Windows-RSS-Platform/2.0 (IE 11.0; Windows NT 10.0)
  WhatsApp
  WhatsApp/2.12.15/i
  WhatsApp/2.12.16/i
  WhatsApp/2.12.17/i
  WhatsApp/2.12.449 A
  WhatsApp/2.12.453 A
  WhatsApp/2.12.510 A
  WhatsApp/2.12.540 A
  WhatsApp/2.12.548 A
  WhatsApp/2.12.555 A
  WhatsApp/2.12.556 A
  WhatsApp/2.16.1/i
  WhatsApp/2.16.13 A
  WhatsApp/2.16.2/i
  WhatsApp/2.16.42 A
  WhatsApp/2.16.57 A
  WhatWeb/0.4.8-dev
  http://www.almaden.ibm.com/cs/crawler [wf84]
  WeLikeLinks - WeAreNotEvil
  Websquash.com (Add url robot)
  WeCrawlForThePeace - We are not Evil
  Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.21 (KHTML, like Gecko) webscreenie Safari/537.21
  Mozilla/3.0 (compatible; WebMon 1.0.11; Windows XP)
  Mozilla/5.0 (X11; BSD Four) AppleWebKit/534.34 (KHTML, like Gecko) webkit2png Safari/534.34
  WebFetch
  WebIndex
  webcollage/1.114
  webcollage/1.117
  webcollage/1.125
  webcollage/1.129
  webcollage/1.93
  Mozilla/3.0 (compatible; WebCapture 2.0; Auto; Windows)
  W3C_Unicorn/1.0
  W3C_I18n-Checker/1.0
  W3C-mobileOK/DDC-1.0
  W3C-checklink
  W3C-checklink/2.90 libwww-perl/5.64
  W3C-checklink/3.6.2.3 libwww-perl/5.64
  W3C-checklink/4.2 [4.20] libwww-perl/5.803
  W3C-checklink/4.2.1 [4.21] libwww-perl/5.803
  W3C-checklink/4.3 [4.42] libwww-perl/5.805
  W3C-checklink/4.3 [4.42] libwww-perl/5.808
  W3C-checklink/4.3 [4.42] libwww-perl/5.820
  W3C-checklink/4.5 [4.154] libwww-perl/5.823
  W3C-checklink/4.5 [4.160] libwww-perl/5.823
  w3af.org
  VYU2 (GNU; OpenRISC)
  VORTEX/1.2 ( http://marty.anstey.ca/robots/vortex/)
  Vortex/2.2 ( http://marty.anstey.ca/robots/vortex/)
  Vortex/2.2 (+http://marty.anstey.ca/robots/vortex/)
  Mozilla/5.0 (Windows NT 5.1; rv:11.0) Gecko Firefox/11.0 (via ggpht.com GoogleImageProxy)
  URLChecker
  urlresolver
  Upflow/1.0
  Typhoeus - https://github.com/typhoeus/typhoeus
  UdmSearch/3.1.x
  Mozilla/5.0 Tweetminster
  Mozilla/5.0 (compatible; +http://tweetedtimes.com)
  truwoGPS/1.0 (GNU/Linux; U; i686; en-US; +http://www.lan4lano.net/browser.html )
  Traackr.com
  More-Testing (+http://www.theinternetrules.com)
  The Expert HTML Source Viewer (http://www.expert-html.com)
  htmlayout 3.3; above-Windows-7; www.terrainformatica.com )
  WebFuck V2.1 T0PHackTeam www.t0p.xyz
  Symfony2 BrowserKit
  Mozilla/5.0 (compatible; SwiteScraper)
  Summify (Summify/1.0.1; +http://summify.com)
  Stratagems Kumo
  Sqworm/2.9.85-BETA (beta_release; 20011115-775; i686-pc-linux-gnu)
  Sprinklr 2.0
  spray-can/1.3.3
  SortSite/4.4.500.0
  Snoopy v2.0.0
  SMRF URL Expander
  silk/1.0 (+http://www.slider.com/silk.htm)/3.7
  SiteBar/3.3.5 (Bookmark Server; http://sitebar.org/)
  SiteBar/3.3.8 (Bookmark Server; http://sitebar.org/)
  SimplePie/1.3.1 (Feed Parser; http://simplepie.org; Allow like Gecko) Build/20160226060242
  ShortLinkTranslate
  Mozilla/5.0 (compatible; Seznam screenshot-generator 2.1; +http://fulltext.sblog.cz/screenshot/)
  SEOstats 2.0.9 https://github.com/eyecatchup/SEOstats
  semanticdiscovery/0.1
  SearchSight/2.0 (http://SearchSight.com/)
  ScoutURLMonitor/5.9.8
  Robosourcer/1.0
  Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; RetrevoPageAnalyzer; +http://www.retrevo.com/content/about-us)
  RestSharp/105.2.3.0
  Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_5) AppleWebKit/537.78.2 (KHTML, like Gecko) Version/7.0.1(2.5.3) Safari(ReederForMac)
  redback/v0-570-g26f8c96
  RebelMouse/0.1 Mozilla/5.0 (compatible; http://rebelmouse.com) Gecko/20100101 Firefox/7.0.1
  RealPlayer%20Downloader/1569 CFNetwork/720.3.13 Darwin/14.3.0 (x86_64)
  RealPlayer%20Downloader/1701 CFNetwork/760.5.1 Darwin/15.5.0 (x86_64)
  RealPlayer%20Downloader/1725 CFNetwork/720.3.13 Darwin/14.3.0 (x86_64)
  RealPlayer%20Downloader/1750 CFNetwork/720.5.7 Darwin/14.5.0 (x86_64)
  RealPlayer%20Downloader/1750 CFNetwork/760.0.5 Darwin/15.0.0 (x86_64)
  RealPlayer%20Downloader/1750 CFNetwork/760.4.2 Darwin/15.4.0 (x86_64)
  Qirina Hurdler v. 1.05 10.11.01 (+http://www.qirina.com/hurdler.html)
  python-requests/1.1.0 CPython/2.7.6 Linux/3.13.0-58-generic
  python-requests/1.2.0 CPython/2.7.3 Linux/3.2.0-29-virtual
  python-requests/2.1.0 CPython/2.7.3 Linux/3.2.0-4-amd64
  python-requests/2.10.0
  python-requests/2.2.1 CPython/2.7.6 Linux/3.13.0-48-generic
  python-requests/2.2.1 CPython/2.7.6 Linux/3.13.0-83-generic
  python-requests/2.3.0 CPython/2.7.6 Linux/3.13.0-57-generic
  python-requests/2.5.0 CPython/2.7.9 Linux/3.14.20-20.44.amzn1.x86_64
  python-requests/2.6.0 CPython/2.7.3 Linux/3.2.0-4-amd64
  python-requests/2.6.0 CPython/2.7.5 Linux/3.10.0-327.3.1.el7.x86_64
  python-requests/2.7.0 CPython/2.7.6 Linux/3.13.0-85-generic
  python-requests/2.8.0
  python-requests/2.8.1
  python-requests/2.9.1
  Python-httplib2/0.7.4 (gzip)
  Pulsepoint XT3 web scraper
  Mozilla/5.0 (compatible; ProspectB2B/2.0; http://prospectb2b.com)
  Mozilla/5.0 (Macintosh; Intel Mac OS X) PowerPoint/14.39.0
  PostRank/2.0 (postrank.com; 1 subscribers)
  PostRank/2.0 (postrank.com)
  postano
  Ploetz + Zeller (http://www.ploetz-zeller.de) Link Validator v1.0 (support@p-und-z.de) for ARIS Business Architect
  Pizilla++ ver 2.45
  phpcrawl
  Pearltrees
  Pcore-HTTP/v0.23.20
  Mozilla/5.0 (Windows; U; Windows NT 6.0; en-US; rv:1.9.0.4) Gecko/2008102920 http://ow.ly web crawler (.NET CLR 3.5.30729)
  Mozilla/5.0 (compatible; Optimizer)
  Mozilla/5.0 [en] (X11, U; OpenVAS 7.0.10)
  JetBrains Omea Reader 1.0.x (http://www.jetbrains.com/omea_reader/)
  JetBrains Omea Reader 2.0 Release Candidate 1 (http://www.jetbrains.com/omea_reader/)
  okhttp/2.5.0
  oegp v. 1.3.0
  Nuzzel
  Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/43.0.2357.134 Safari/537.36 http://notifyninja.com/monitoring
  Notifixious/LinkChecker (http://notifixio.us)
  http://www.nominet.org.uk/privacypolicy
  node-superagent/1.8.3
  node.io
  Kimengi/nineconnections.com
  newspaper/0.0.9.8
  newspaper/0.1.7
  newsme/1.0; feedback@news.me
  NewsGator/2.0 Bot (http://www.newsgator.com)
  NewsGator/2.5 (http://www.newsgator.com; Microsoft Windows NT 5.1.2600.0; .NET CLR 1.1.4322.2032)
  NetTrack Anonymous Web Statistics https://nettrack.info/support.php
  NetTrack.info Crawler
  NetResearchServer(http://www.look.com)
  NetResearchServer/2.5(loopimprovements.com/robot.html)
  NetResearchServer/2.7(loopimprovements.com/robot.html)
  NetResearchServer/2.8(loopimprovements.com/robot.html)
  NetResearchServer/3.5(loopimprovements.com/robot.html)
  NetResearchServer/4.0(loopimprovements.com/robot.html)
  NetLyzer FastProbe (See http://netlyzer.com/report/www.domain.com for info)
  check_http/v2.0.3 (nagios-plugins 2.0.3)
  MVAClient
  Link Validity Check From: http://www.w3dir.com/cgi-bin (Using: Hot Links SQL by Mrcgiguy.com)
  Mojolicious (Perl)
  mogimogi/1.0
  Mindjet
  Mindjet MindManager
  Microsoft Windows Network Diagnostics
  Microsoft Office Excel 2013
  Microsoft Office Excel 2014
  Microsoft Office Excel 2014 (16.0.6528) Windows NT 10.0
  Microsoft Office Existence Discovery
  Microsoft Office Mobile/15.0
  Microsoft Office OneNote 2013
  Microsoft Office PowerPoint 2013
  Microsoft Office PowerPoint 2013 (15.0.4420) Windows NT 6.2
  Microsoft Office PowerPoint 2013 (15.0.4641) Windows NT 6.2
  Microsoft Office PowerPoint 2013 (15.0.4797) Windows NT 10.0
  Microsoft Office PowerPoint 2013 (15.0.4797) Windows NT 6.2
  Microsoft Office PowerPoint 2013 (15.0.4805) Windows NT 6.1
  Microsoft Office PowerPoint 2013 (15.0.4805) Windows NT 6.2
  Microsoft Office PowerPoint 2013 (15.0.4815) Windows NT 10.0
  Microsoft Office PowerPoint 2014
  Microsoft Office Word 2013
  Microsoft Office Word 2013 (15.0.4623) Windows NT 6.2
  Microsoft Office Word 2013 (15.0.4701) Windows NT 6.1
  Microsoft Office Word 2013 (15.0.4797) Windows NT 10.0
  Microsoft Office Word 2013 (15.0.4805) Windows NT 10.0
  Microsoft Office Word 2013 (15.0.4805) Windows NT 6.1
  Microsoft Office Word 2013 (15.0.4823) Windows NT 10.0
  Microsoft Office Word 2014
  Melvil Rawi/1.0
  marketinggrader.com
  Mandrill-Webhook/1.0
  MailChimp.com
  MagpieRSS/0.72 (+http://magpierss.sf.net)
  http://www.mabontland.com
  lwp-trivial/1.33
  lwp-trivial/1.35
  lwp-trivial/1.36
  lwp-trivial/1.38
  lwp-trivial/1.41
  looksystems.net
  LinkExaminer/1.01 (Windows)
  VLC/2.2.4 LibVLC/2.2.4
  hello@knows.is
  Mozilla/5.0 (compatible; Jobrapido/1.1; +http://www.jobrapido.com)
  JoBo/1.x (http://www.matuschek.net/jobo.html)
  Jigsaw/2.3.0 W3C_CSS_Validator_JFouffa/2.0
  Mozilla/5.0 (compatible; IODC-Internet Engineering Odysseus 001; +http://iodc.co.uk)
  Mozilla/5.0 (compatible; IODC-Internet Engineering Odysseus 005; +http://iodc.co.uk)
  Mozilla/5.0 (compatible; IODC-Internet Engineering Odysseus 006; +http://iodc.co.uk)
  Mozilla/5.0 (compatible; IODC-Internet Engineering Odysseus 007; +http://iodc.co.uk)
  Mozilla/5.0 (compatible; IODC-Internet Engineering Odysseus 011; +http://iodc.co.uk)
  Mozilla/5.0 (compatible; IODC-Internet Engineering Odysseus 013; +http://iodc.co.uk)
  Mozilla/5.0 (compatible; IODC-Odysseus Survey 29691-100-070315021100-72; +http://iodc.co.uk)
  Mozilla/5.0 (compatible; IODC-Odysseus Survey 29692-100-060415034231-192; +http://iodc.co.uk)
  Mozilla/5.0 (compatible; IODC-Odysseus Survey 31193-100-080615161452-372; +http://iodc.co.uk)
  Mozilla/5.0 (compatible; IODC-Odysseus Survey 37044-100-110516191742-025; +https://iodc.co.uk)
  Mozilla/5.0 (compatible; IODC-Odysseus Survey 41437-100-090715214314-122; +http://iodc.co.uk)
  Integrity/6
  InfoWizards Reciprocal Link System PRO - (http://www.infowizards.com)
  InDesign%20CC/11300 CFNetwork/720.4.4 Darwin/14.4.0 (x86_64)
  inbound.li parser
  InAGist URL Resolver (http://inagist.com)
  igdeSpyder (compatible; igde.ru; +http://igde.ru/doc/tech.html)
  IDG Twitter Links Resolver
  Mozilla/4.5 (compatible; HTTrack 3.0x; Windows 98)
  Jersey/2.19 (HttpUrlConnection 1.7.0_85)
  httpunit/1.x
  httpssites_power/1 CFNetwork/758.3.15 Darwin/15.4.0
  httpscheck (unknown version) CFNetwork/760.5.1 Darwin/15.5.0 (x86_64)
  HttpComponents/1.1
  http_requester/0.1
  HTTP_Request2/@package_version@ (http://pear.php.net/package/http_request2) PHP/5.5.9-1ubuntu4.14
  HTTP_Request2/2.2.0 (http://pear.php.net/package/http_request2) PHP/5.3.29
  HTTP_Request2/2.2.1 (http://pear.php.net/package/http_request2) PHP/5.3.29
  HTTP_Request2/2.2.1 (http://pear.php.net/package/http_request2) PHP/5.5.33
  HTTP_Request2/2.2.1 (http://pear.php.net/package/http_request2) PHP/5.5.35
  HTTP_Request2/2.3.0 (http://pear.php.net/package/http_request2) PHP/5.3.29
  HTTP-Tiny/0.054
  http-kit/2.0
  HTMLParser/2.0
  htdig/3.1.5
  htdig/3.1.5 (infosys@storm.rmi.org)
  htdig/3.1.5 (root@localhost)
  htdig/3.1.5 (webmaster@online-medien.de)
  htdig/3.1.6 (mathieu.peltier@inrialpes.fr)
  htdig/3.1.6 (unconfigured@htdig.searchengine.maintainer)
  ht://check/2.0.0-rc1 (Linux 2.6.32-573.18.1.el6.x86_64 x86_64)
  HEADMasterSEO
  Mozilla/4.0 (compatible; grub-client-0.3.0; Crawl your own stuff with http://grub.org)
  Mozilla/4.0 (compatible; grub-client-1.0.4; Crawl your own stuff with http://grub.org)
  Mozilla/4.0 (compatible; grub-client-1.0.5; Crawl your own stuff with http://grub.org)
  Mozilla/4.0 (compatible; grub-client-1.0.6; Crawl your own stuff with http://grub.org)
  Mozilla/4.0 (compatible; grub-client-1.0.7; Crawl your own stuff with http://grub.org)
  Mozilla/4.0 (compatible; grub-client-1.1.1; Crawl your own stuff with http://grub.org)
  Mozilla/4.0 (compatible; grub-client-1.2.1; Crawl your own stuff with http://grub.org)
  Mozilla/4.0 (compatible; grub-client-1.3.1; Crawl your own stuff with http://grub.org)
  Mozilla/4.0 (compatible; grub-client-1.3.7; Crawl your own stuff with http://grub.org)
  Mozilla/4.0 (compatible; grub-client-1.4.3; Crawl your own stuff with http://grub.org)
  Mozilla/4.0 (compatible; grub-client-1.5.3; Crawl your own stuff with http://grub.org)
  Mozilla/4.0 (compatible; grub-client-2.3)
  Mozilla/4.0 (compatible; grub-client-2.6.0)
  Mozilla/4.0 (compatible; grub-client-2.6.1)
  GoSpotCheck 1.0.6 (iPhone; iPhone OS 5.1.1; en_US)
  GoScraper
  GoogleProducer; (+http://goo.gl/7y4SX)
  GoogleHC/
  Google-HTTP-Java-Client/1.17.0-rc (gzip)
  Google-HTTP-Java-Client/1.9.0-alpha
  Mozilla/5.0 (compatible; Google-Apps-Script)
  Go-http-client/1.1
  Go 1.1 package http
  Mozilla/5.0 (Macintosh; Intel Mac OS X 10_6_6) AppleWebKit/534.24 (KHTML, like Gecko) (Contact: backend@getprismatic.com)
  Mozilla/5.0 (compatible; Funnelback) RPT-HTTPClient/0.3-3E
  Mozilla/5.0 (compatible; Snacktory; +https://flynxapp.com)
  Mozilla/3.0 (compatible; Fluffy the spider; http://www.searchhippo.com/; info@searchhippo.com)
  Flamingo_SearchEngine (+http://www.flamingosearch.com/bot)
  Mozilla/4.0 (compatible; MSIE 5.5; Windows NT 5.0) Fetch API Request
  FeedChecker-Zocle/1.0 (+https://zocle.com/zoclechecker)
  FeedBurner/1.0 (http://www.FeedBurner.com)
  FavOrg
  Faraday v0.9.1
  Mozilla/5.0 (Macintosh; Intel Mac OS X) Excel/14.10.0
  Mozilla/5.0 (Macintosh; Intel Mac OS X) Excel/14.62.0
  ExactSearch
  EventMachine HttpClient
  Mozilla/5.0 (compatible; MSIE 7.0 +http://www.europarchive.org)
  Embed PHP Library
  ECCP/1.2.1+(productlists@eniro.com)
  ec2linkfinder
  Mozilla/5.0 (compatible; EARTHCOM/2.2; +http://enter4u.eu)
  Drupal (+http://drupal.org/)
  Dragonfly File Reader
  DNS-Tools Header-Analyzer
  Dispatch/0.11.1-SNAPSHOT
  luis@cybo.com
  CyberPatrol SiteCat Webbot (http://www.cyberpatrol.com/cyberpatrolcrawler.asp)
  curb
  Mozilla/4.0 (compatible; cron-job.org; http://cron-job.org/abuse/)
  cosmos/0.9_(robot@xyleme.com)
  Mozilla/5.0/Firefox/42.0 (contactbigdatafr at gmail.com)
  Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; .NET CLR 2.0.50727; MAXTHON 2.0); Connect Us: webauth@cmcm.com
  Pattern/2.6 +http://www.clips.ua.ac.be/pattern
  Mozilla/4.0 (compatible; Cerberian Drtrs Version-3.2-Build-0)
  Mozilla/4.0 (compatible; Cerberian Drtrs Version-3.2-Build-1)
  CC Metadata Scaper http://wiki.creativecommons.org/Metadata_Scraper
  cb crawl (+http://www.companybooknetworking.com)
  CapsuleChecker (http://www.capsulink.com/)
  CakePHP
  Branch-Passthrough
  Branch Metrics API
  boitho.com-dc/0.79 ( http://www.boitho.com/dcbot.html )
  boitho.com-dc/0.81 ( http://www.boitho.com/dcbot.html )
  boitho.com-dc/0.82 ( http://www.boitho.com/dcbot.html )
  boitho.com-dc/0.83 ( http://www.boitho.com/dcbot.html )
  boitho.com-dc/0.85 ( http://www.boitho.com/dcbot.html )
  boitho.com-robot/1.1
  Blackboard Safeassign
  Mozilla/5.0 (Windows NT 6.3; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/42.0.2311.152 Safari/537.36 (BingLocalSearch)
  BIGLOTRON (Beta 2;GNU/Linux)
  Mozilla/5.0 (compatible; bnf.fr_bot; +http://bibnum.bnf.fr/robot/bnf.html)
  Mozilla/4.0 (compatible; B-l-i-t-z-B-O-T)
  Astute SRM
  aria2/1.18.8
  Mozilla/5.0 (compatible; archive-com.com/1.1; +http://archive-com.com/bot)
  Mozilla/4.0 (compatible; Arachmo)
  Apache-HttpClient/4.2.3 (java 1.5)
  Apache-HttpClient/4.2.5 (java 1.5)
  Apache-HttpClient/4.3.1 (java 1.5)
  Apache-HttpClient/4.3.3 (java 1.5)
  Apache-HttpClient/4.3.5 (java 1.5)
  Apache-HttpClient/4.5 (Java/1.8.0_51)
  Apache-HttpClient/4.5 (Java/1.8.0_72)
  Apache-HttpClient/4.5 (Java/1.8.0_74)
  Apache-HttpClient/4.5.1 (Java/1.7.0_79)
  Apache-HttpClient/4.5.1 (Java/1.8.0_40)
  Apache-HttpClient/4.5.2 (Java/1.8.0_72)
  Apache-HttpClient/UNAVAILABLE (java 1.4)
  Mozilla/5.0 (compatible; U; AnyEvent-HTTP/2.22; +http://software.schmorp.de/pkg/AnyEvent)
  Mozilla/5.0 (compatible; U; AnyEvent-HTTP/2.23; +http://software.schmorp.de/pkg/AnyEvent)
  Anemone/0.7.2
  AndroidDownloadManager/6.0.1 (Linux; U; Android 6.0.1; SM-G900F Build/MMB29M)
  AndroidDownloadManager/4.4.2 (Linux; U; Android 4.4.2; SM-T310 Build/KOT49H)
  AndroidDownloadManager/4.4.2 (Linux; U; Android 4.4.2; SM-T210 Build/KOT49H)
  Airmail 1.3.3 rv:237 (Macintosh; Mac OS X 10.9.3; sk_SK)
  Python/3.5 aiohttp/0.21.6
  AHC/1.0
  Mozilla/4.0 (compatible; Advanced Email Extractor v2.61)
  Pulp/1.5.2 (iPad; http://www.acrylicapps.com/pulp/)
  Accoona-AI-Agent/1.1.1 (crawler at accoona dot com)
  Accoona-AI-Agent/1.1.2
  Accoona-AI-Agent/1.1.2 (aicrawler at accoonabot dot com)
  Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1) shipra.goel@a3logics.in
  scrutiny/6
  RMA/1.0 (compatible; RealMedia)
  PHP/5.5
  NING/1.0
  Mget/0.1.9
  Java/1.4.1_04
  Java/1.6.0
  Java/1.6.0_04
  Java/1.6.0_22
  Java/1.6.0_24
  Java/1.6.0_26
  Java/1.6.0_31
  Java/1.6.0_32
  Java/1.6.0_34
  Java/1.6.0_37
  Java/1.7.0_101
  Java/1.7.0_45
  Java/1.7.0_51
  Java/1.7.0_65
  Java/1.7.0_80
  Java/1.8.0_05
  Java/1.8.0_20
  Java/1.8.0_31
  Java/1.8.0_40
  Java/1.8.0_45
  Java/1.8.0_51
  Java/1.8.0_60
  Java/1.8.0_65
  Java/1.8.0_66
  Java/1.8.0_73
  Java/1.8.0_77
  Java/1.8.0_91
  Java/1.8.0_92
  Goose/1.0.25
  FDM 3.x
  bluefish 0.6 HTML editor
  2ip.ru CMS Detector (http://2ip.ru/cms/)
  masscan/1.0 (https://github.com/robertdavidgraham/masscan)
  VidibleScraper/1.0
  WhereGoes? Redirect Tracer/1.0
  unfurlist (https://github.com/Doist/unfurlist)
  pshtt, https scanning
  gofetch
  COMODO DCV
  CATExplorador/1.0beta (sistemes at domini dot cat; http://domini.cat/catexplorador.html)
  ZmEu
  RSSOwl/2.2.1.201312301314 (Windows; U; en)
  WebClient/1.0
  Microsoft-WebDAV-MiniRedir/10.0.10586
  Microsoft-WebDAV-MiniRedir/10.0.14393
  Microsoft-WebDAV-MiniRedir/5.1.2600
  Microsoft-WebDAV-MiniRedir/6.0.6001
  Microsoft-WebDAV-MiniRedir/6.1.7601
  Microsoft-WebDAV-MiniRedir/6.3.9600
  Microsoft Data Access Internet Publishing Provider Protocol Discovery
  Test Certificate Info
  Needle/1.4.2 (Node.js v6.9.2; linux x64)
  DavClnt
  Mozilla/5.0 (iPhone; CPU iPhone OS 6_0 like Mac OS X) AppleWebKit/536.26 (KHTML, like Gecko) Version/6.0 Mobile/10A5376e Safari/8536.25 mixed-content-scan
  Mozilla/5.0 (compatible; Semiocast HTTP client; http://semiocast.com/)
  Mozilla/5.0 (compatible; The Drop Reaper; +http://tag1consulting.com/dropreaper.html)
  /n software IPWorks HTTP/S Component - www.nsoftware.com
  safe-agent-scanner
  Mozilla/5.0 (compatible; ACHE/Unknown Version; +https://github.com/ViDA-NYU/ache; )
  ANAM (github.com/dutchcoders/anam)
  Dolphin http client/11.2.8(385) (Android)
  Dolphin http client/11.3.4(405) (Android)
  Calypso v/0.01
  -Mozilla/4.0 (compatible; http://search.thunderstone.com/texis/websearch/about.html)
  Mozilla/5.0 (compatible; Tweezler/2.0; http://tweezler.com)
  Jeode/1.x.x
  KeyCDN Perf Test
  Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) YLT Chrome/27.0.1453.110 Safari/537.36
  Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/51.0.2704.84 Safari/537.36 DMBrowser-UV
  Taringa UGC www.taringa.net/shared-link.php
  Mozilla/5.0 (Unknown; Linux x86_64) AppleWebKit/534.34 (KHTML, like Gecko) PingdomTMS/0.8.5 Safari/534.34
  ganarvisitas/1.0 (+http://www.ganarvisitas.com/)
  FeedFetcher-Google; (+http://www.google.com/feedfetcher.html)
  Mozilla/5.0 (compatible) Feedfetcher-Google; (+http://www.google.com/feedfetcher.html)
  Mozilla/5.0 (compatible; Exabot PyExalead/3.0; +http://www.exabot.com/go/robot)
  Mozilla/5.0 PhantomJS (compatible; Seznam screenshot-generator 2.1; +http://fulltext.sblog.cz/screenshot/)
  Mozilla/5.0 (compatible; SeznamBot/3.2-test4; +http://napoveda.seznam.cz/en/seznambot-intro/)
  Mozilla/5.0 (compatible; SeznamBot/3.2-test2; +http://napoveda.seznam.cz/en/seznambot-intro/)
  Mozilla/5.0 (compatible; SeznamBot/3.2-test1; +http://fulltext.sblog.cz/)
  Mozilla/5.0 (compatible; SeznamBot/3.2-test4; +http://fulltext.sblog.cz/)
  Mozilla/5.0 (compatible; SeznamBot/3.2; +http://fulltext.sblog.cz/)
  Seznam-Zbozi-robot/3.2.1
  SeznamBot/2.0 (+http://fulltext.sblog.cz/robot/)
  Seznam-Zbozi-robot/3.2.2
  Mozilla/5.0 (compatible; SeznamBot/3.2-test2; +http://fulltext.sblog.cz/)
  Mozilla/5.0 (compatible; SeznamBot/3.2-test3; +http://fulltext.sblog.cz/)
  Seznam-Zbozi-robot/3.3
  SklikBot/2.0 (sklik@firma.seznam.cz;+http://napoveda.sklik.cz/)
  SeznamBot/3.0 (+http://fulltext.sblog.cz/)
  Mozilla/5.0 (compatible; SeznamBot/3.1-test1; +http://fulltext.sblog.cz/)
  Mozilla/5.0 (Linux; U; Android 4.1.2; cs-cz; Seznam screenshot-generator Build/Q3) AppleWebKit/534.30 (KHTML, like Gecko) Version/4.0 Mobile Safari/534.30
  SeznamBot/3.0-test (+http://fulltext.sblog.cz/), I
  SeznamBot/3.0 (HaF+http://fulltext.sblog.cz/)
  SeznamBot/3.0-test (+http://fulltext.sblog.cz/)
  SeznamBot/3.0-beta (+http://fulltext.sblog.cz/), I
  SeznamBot/3.0-beta (+http://fulltext.sblog.cz/)
  SeznamBot/3.0-alpha (+http://fulltext.sblog.cz/)
  SeznamBot/2.0 (+http://fulltext.seznam.cz/)
  SeznamBot/2.0-Test (+http://fulltext.sblog.cz/robot/)
  Mozilla/5.0 (compatible; Seznam screenshot-generator 2.0; +http://fulltext.sblog.cz/screenshot/)
  Mozilla/5.0 (compatible; FlipboardRSS/1.2; +http://flipboard.com/browserproxy)
  Mozilla/5.0 (Macintosh; Intel Mac OS X 10.11; rv:49.0) Gecko/20100101 Firefox/49.0 (FlipboardProxy/1.2; +http://flipboard.com/browserproxy)
  Mozilla/5.0 (compatible; MJ12bot/v1.4.7; http://mj12bot.com/)
  Mozilla/5.0 (Macintosh; Intel Mac OS X 10_7_2) AppleWebKit/535.7 (KHTML, like Gecko) Chrome/16.0.912.77 Safari/535.7 AppEngine-Google; (+http://code.google.com/appengine; appid: s~feedly-social)
  Mozilla/5.0 (Macintosh; Intel Mac OS X 10_7_2) AppleWebKit/535.7 (KHTML, like Gecko) Chrome/23.0.912.77 Safari/535.7 AppEngine-Google; (+http://code.google.com/appengine; appid: s~feedly-nikon3)
  Mozilla/5.0 (compatible; SemrushBot/1.2~bl; +http://www.semrush.com/bot.html)
  Mozilla/5.0 (compatible; YandexMetrika/2.0; +http://yandex.com/bots mtmon01it.mtrs.yandex.ru)
  Mozilla/5.0 (iPhone; CPU iPhone OS 8_1 like Mac OS X) AppleWebKit/600.1.4 (KHTML, like Gecko) Version/8.0 Mobile/12B411 Safari/600.1.4 (compatible; YandexBot/3.0; +http://yandex.com/bots)
  Mozilla/5.0 (compatible; memoryBot/1.24.61 +http://internetmemory.org/en/)
  Mozilla/5.0 (compatible; memoryBot/1.24.54 +http://internetmemory.org/en/)
  BUbiNG - Research at Brno University of Technology - KNOT group - http://knot.fit.vutbr.cz/crawling/ - Stop? http://law.di.unimi.it/BUbiNG.html
  Mozilla/5.0 (compatible; MSIE 9.0; Firefox/50.1) Daum/4.1
  Mozilla/5.0 (Unknown; Linux x86_64) AppleWebKit/534.34 (KHTML, like Gecko) PhantomJS/1.9.7 Daum Raam Edition b20140820 Safari/534.34
  Sogou Pic Spider/3.0(+http://www.sogou.com/docs/help/webmasters.htm#07)
  yacybot (/global; amd64 Linux 4.4.0-59-generic; java 1.8.0_121; Europe/de) http://yacy.net/bot.html
  yacybot (-global; amd64 Linux 4.4.0-57-generic; java 9-internal; Europe/en) http://yacy.net/bot.html
  yacybot (/global; amd64 Windows 8.1 6.3; java 1.8.0_111; Europe/de) http://yacy.net/bot.html
  yacybot (/global; amd64 Linux 3.16.0-4-amd64; java 1.8.0_111; Europe/en) http://yacy.net/bot.html
  yacybot (/global; amd64 FreeBSD 10.3-RELEASE-p7; java 1.7.0_95; GMT/en) http://yacy.net/bot.html
  yacybot (webportal-global; amd64 Linux 4.4.0-46-generic; java 1.8.0_91; Europe/en) http://yacy.net/bot.html
  yacybot (-global; amd64 Linux 3.14.32-xxxx-grs-ipv6-64; java 1.8.0_111; Europe/de) http://yacy.net/bot.html
  Mozilla/5.0 (compatible; special_archiver/3.3.0 bot@archive.org +https://archive.org/details/archive.org_bot)
  Mozilla/5.0 (compatible; heritrix/3.3.0-SNAPSHOT-20140702-2247 +http://archive.org/details/archive.org_bot)
  Mozilla/5.0 (compatible; archive.org_bot/heritrix-1.15.4 +http://www.archive.org)
  SeobilityBot (SEO-Check; http://bit.ly/1dJuuzs)
  Testomatobot/1.0 (Linux x86_64; +http://www.testomato.com/testomatobot) minicrawler/5.0.4
  Testomatobot/1.0 (Linux x86_64; +http://www.testomato.com/testomatobot) minicrawler/5.0.3
  Testomatobot/1.0 (Linux x86_64; +http://www.testomato.com/testomatobot) minicrawler/5.0.2
  Testomatobot/1.0 (Linux x86_64; +http://www.testomato.com/testomatobot) minicrawler/5.0.1
  Mozilla/5.0 (compatible; AhrefsBot/5.2; +http://ahrefs.com/robot/)
  Heurekabot-Feed/1.0 (+https://sluzby.heureka.cz/napoveda/heurekabot/)
  Heurekabot-ImageFullText/1.0 (+https://sluzby.heureka.cz/napoveda/heurekabot/)
  Heurekabot-ImageFullText/1.0 (+http://sluzby.heureka.cz/napoveda/heurekabot/)
  Mozilla/5.0 (Windows NT 6.2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/31.0.1650.63 Safari/537.36 QIHU 360SE; 360Spider
  ZoomBot (Linkbot 1.0 http://suite.seozoom.it/bot.html)
  Mozilla/5.0 (compatible; SOLOFIELD/1.0 +http://solofield.net/bot.html)
  Qwantify/1.0
  Mozilla/5.0 (compatible; Yeti/1.1; +http://naver.me/bot)
  Scrapy/1.3.0 (+http://scrapy.org)
  Scrapy/1.0.5.post4+g4b324a8 (+http://scrapy.org)
  Scrapy/1.2.1 (+http://scrapy.org)
  Scrapy/0.16.5 (+http://scrapy.org)
  Mozilla/5.0 (compatible; Dataprovider.com;)
  Mozilla/5.0 (compatible; Dataprovider; https://www.dataprovider.com/spider/)
  Mozilla/5.0 (compatible; SiteExplorer/1.1b; +http://siteexplorer.info/Backlink-Checker-Spider/)
  ICC-Crawler/2.0 (Mozilla-compatible; ; http://ucri.nict.go.jp/en/icccrawler.html)
  DuckDuckBot/1.0; (+http://duckduckgo.com/duckduckbot.html)
  Mozilla/5.0 (compatible; SMTBot/1.0; http://www.similartech.com/smtbot)
  Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/538.1 (KHTML, like Gecko) Chromium/31.0.1650.63 Site-Shot/2.1 (http://www.site-shot.com/) Safari/538.1
  Mozilla/5.0 (Windows NT 6.1; WOW64; Neustar WPM) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/48.0.2564.82 Safari/537.36
  Mozilla/5.0 (iPad; CPU OS 6_1_2 like Mac OS X; Neustar WPM) AppleWebKit/536.26 (KHTML, like Gecko) Version/6.0 Mobile/10A5376e Safari/8536.25
  Mozilla/5.0 (Windows NT 6.0; Neustar WPM) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/48.0.2564.82 Safari/537.36
  Mozilla/5.0 (Windows NT 6.1; Neustar WPM) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/48.0.2564.82 Safari/537.36
  Mozilla/5.0 (Windows NT 6.0; rv:38.0; Neustar WPM) Gecko/20100101 Firefox/38.0
  Mozilla/5.0 (Windows NT 6.1; rv:38.0; Neustar WPM) Gecko/20100101 Firefox/38.0
  Mozilla/5.0 (Windows NT 6.1; Neustar WPM) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/52.0.2743.82 Safari/537.36
  Mozilla/5.0 (Linux; Android 4.1.1; HTC6435LVW Build/JRO03C; Neustar WPM) AppleWebKit/537.22 (KHTML, like Gecko) Chrome/25.0.1364.169 Mobile Safari/537.22
  Mozilla/5.0 (compatible; Whoiswebsitebot/0.1; +http://www.whoiswebsite.net)
  Mozilla/5.0 (Linux; Android 5.0.2; SM-G920T Build/LRX22G) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/54.0.2840 Mobile Safari/537.36 DareBoost
  Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/54.0.2840 Safari/537.36 DareBoost
  Mozilla/5.0 (Linux; Android 5.0; LG-D855 Build/LRX21R.A1422018487) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/54.0.2840 Mobile Safari/537.36 DareBoost
  Mozilla/5.0 (iPhone; CPU iPhone OS 10_0_1 like Mac OS X) AppleWebKit/602.1.50 (KHTML, like Gecko) Version/10.0 Mobile/14A403 Safari/602.1 DareBoost
  Mozilla/5.0 (Linux; Android 4.4.3; HTC One Build/KTU84L) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/54.0.2840 Mobile Safari/537.36 DareBoost
  Mozilla/5.0 (Linux; Android 4.4; Nexus 5 Build/KRT16M) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/54.0.2840 Mobile Safari/537.36 DareBoost
  HubSpot Website Grader (web-crawlers@hubspot.com)
  HubSpot Marketing Grader, HubSpot Marketing Grader
  com.hubspot.spooks.core.UserAgent@753535c0, HubSpot Marketing Grader
  Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/51.0.2704.79 Safari/537.36 (http://www.shrinktheweb.com)
  scrapy-redis (+https://github.com/rolando/scrapy-redis)
  Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/55.0.2883.87 Safari/537.36 PTST/348
  Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/54.0.2840.99 Safari/537.36 PTST/345
  TeamSpeak3-ImageFetcher-1.0
  wf_crawler (http://www.websitefigures.com)
  wf_crawler (http://websitefigures.com)
  Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/55.0.2883.87 Safari/537.36 TinEye/1.0 (via http://www.tineye.com/)
  TinEye-bot/0.61 (see http://www.tineye.com/crawler.html)
  Mozilla/5.0 (compatible; Googlebot/2.1; https://deepcrawl.com/bot)
  Googlebot deepcrawl
  Mozilla/5.0 (compatible; evc-batch/2.0.20170110125844)
  Mozilla/5.0 (compatible; evc-batch/2.0.20161220184139)
  Mozilla/5.0 (compatible; evc-batch/2.0.20161122223033)
  Mozilla/5.0 (compatible; evc-batch/2.0.20161026010401)
  Mozilla/5.0 (X11; Linux x86_64; rv:49.0; GTmetrix https://gtmetrix.com/) Gecko/20100101 Firefox/49.0
  Mozilla/5.0 (Linux; Android 4.3; Galaxy Nexus Build/JWR66Y; GTmetrix https://gtmetrix.com/) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/54.0.2840.68 Mobile Safari/537.36
  Mozilla/5.0 (X11; Linux x86_64; GTmetrix https://gtmetrix.com/) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/54.0.2840.71 Safari/537.36
  Mediumbot-MetaTagFetcher/0.3 (+https://medium.com/)
  Mediumbot-ProxyFetcher/0.1 (+https://medium.com/)
  YOOBI.de - WebBot 1.0
  YOOBI - WebBot 1.0
  Mozilla/5.0 (compatible; heritrix/3.3.0-SNAPSHOT-20160309-0050; UniLeipzigASV +http://corpora.informatik.uni-leipzig.de/crawler_faq.html)
  IAS crawler (page scorer; http://integralads.com/site-indexing-policy/)
  Mozilla/5.0 (compatible; JobKeresoBot; +https://www.kozvetlen-allasok.hu/help.jsp; info@kozvetlen-allasok.hu; 5.0
  Mozilla/5.0 (compatible; JobKereso; +https://www.kozvetlen-allasok.hu/robot.jsp info@kozvetlen-allasok.hu)
  pyspider/0.3.9 (+http://pyspider.org/)
  Mozilla/5.0 (X11; U; Linux x86_64; de-DE; rv:1.9.0.19) Gecko/2010120923 ThumbShotsBot (KFSW 3.0.6-3)
  Mozilla/5.0 (compatible; Discordbot/1.0; +https://discordapp.com)
  Mozilla/5.0 (compatible; Dead Link Checker; http://www.dead-link-checker.com/)
  Mozilla/5.0 (compatible; MSIE 10.0; Windows NT 6.2; WOW64; Trident/6.0) CrawlerProcess (http://www.PowerMapper.com) /5.24.776.0
  Mozilla/4.0 (compatible; BOTW Spider; +http://botw.org)
  TwengaBot-2.0
  Iframely/1.0.0 (+https://iframely.com/;)
  Iframely/0.8.5 (+http://iframely.com/;)
  rogerbot/1.2 (http://moz.com/help/pro/what-is-rogerbot-, rogerbot-crawler+phaser-testing-crawler-01@moz.com)
  rogerbot/1.0 (http://moz.com/help/pro/what-is-rogerbot-, rogerbot-wherecat@moz.com
  Mozilla/5.0 (compatible; Sitemap Generator/1.3; http://www.check-domains.com/sitemap/index.php) Gecko Check-domains)
  mindUpBot
  Mozilla/5.0 (Windows NT 6.1; compatible; BDCbot/1.0; +http://bigweb.bigdatacorp.com.br/faq.aspx) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2272.118 Safari/537.36
  Contacts Crawler (+http://www.scrapinghub.com)
  TA SEO Crawler v0.1 URLData.pm (smcquaker@tripadvisor.com)
  Mozilla/5.0 (compatible; scanbot/1.0; +http://dazzlepod.com/ip/)
  ArielisBot/1.1 (+http://arielis.com/
  RedesScrapy/0.24.1 (+http://g2pi.tsc.uc3m.es/es)
  Blackboard Safeassign/0.1 (a Storm-based Blackboard Safeassign web-crawler; https://github.com/DigitalPebble/storm-crawler; stormcrawler@digitalpebble.com)
  Mozilla/5.0 (Macintosh; Intel Mac OS X 10.11; rv:48.0) Gecko/20100101 Firefox/48.0 glindahl-cocrawler/0.01 (+http://www.pbm.com/~lindahl/glindahl-cocrawler.html)
  botrobin/Nutch-1.13-SNAPSHOT (http://smarter.codes/bot-robin/; botrobin@smarter.codes)
  Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/51.0.2704.63 Safari/537.36 (compatible; aylienbot/0.2; +http://www.aylien.com/bot.html)
  GloomarBot/1.1 (https://www.gloomar.com/bot)
  Hatena::Bookmark/2.00 (Hatena::Bookmark; master;)
  Opendi Screenshot Bot bot@opendi.com
  Mozilla/5.0 (compatible; YaK/1.0; http://linkfluence.com/; bot@linkfluence.com)
  Mozilla/5.0 (compatible; KAZ.KZ_Bot/3.0)
  BuckyOHare/1.3 (Googlebot/2.1; +https://hypefactors.com/webcrawler)
  uipbot/1.0 (uipbot@semasio.net)
  gMozilla/5.0 (compatible; Infohelfer/1.4.3; +http://www.infohelfer.de/crawler.php)
  Mozilla/5.0 (compatible; lincobot/0.3; http://www.linksco.com/about/robot.html) AppEngine-Google; (+http://code.google.com/appengine)
  Mozilla/5.0 (compatible; MSIE 10.0; Windows NT 6.2; WOW64; Trident/6.0) MapperCmd/5.24.780.0
  Seeker v.1 http://LookSeek.com/seeker/
  AppManager RPT-HTTPClient/0.3-3E
  LinkAlarm/3.1
  Mozilla/5.0 (X11; U; Linux i686) Firefox/2.0.0.11 wprecon.com survey
  Jeffrey's Exif Viewer (http://regex.info/exif)
  Untiny
  Mozilla/5.0 (Grabify.link - https://grabify.link/expander)
  biNu image cacher (info@binu-inc.com)
  Mozilla/5.0 (compatible; Google-Youtube-Links)
  ping.blo.gs/2.0
  Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; QQDownload 1.7; .NET CLR 2.0.50727)
  Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.1; Trident/4.0; QQDownload 663; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; .NET4.0C; .NET CLR 1.1.4322)
  Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 5.1; Trident/4.0; QQDownload 760; .NET CLR 2.0.50727; .NET CLR 3.0.4506.2152; .NET CLR 3.5.30729; InfoPath.2; .NET4.0E; .NET4.0C; .NET CLR 1.1.4322)
  Mozilla/5.0 (X11; Linux x86_64; rv:52.0.2) Gecko/20100101 Firefox/52.0.2 Ghost Inspector
  Mozilla/5.0 (X11; Linux x86_64; rv:52.0) Gecko/20100101 Firefox/52.0 Ghost Inspector