Юзер агенты ботов

[dexion]

Есть ли у кого-нибудь более содержательный список юзер агентов ботов, чем у меня? Просьба поделиться или дополнить мой.) 

Будет полезно, например, для модуля источников заказов или различных обработчиков логов.

Мой список под спойлером ниже:

Спойлер

YandexBot
YandexImages
YandexMetrika
YaDirectFetcher
YandexMobileBot
YandexMarket
YandexImageResizer
Yandex Market picture robot
YandexAccessibilityBot
YandexCatalog
Googlebot
AdsBot-Google
Mail.RU_Bot
bingbot
msnbot-media
Baiduspider-
Baiduspider
Yahoo! Slurp
curl
AhrefsBot
MJ12bot
SeznamBot
SemrushBot
price.ru-image-robot
ltx71.com
Riddler
Exabot
SurdotlyBot
ActivizmBot
glindahl-cocrawler
Twitterbot
facebookexternalhit
Virusdie crawler
Cliqzbot
statdom.ru/Bot
openstat.ru/Bot
linkdexbot
Linkbot
coccocbot-web
SMTBot
top100.rambler.ru crawler
WebMoney MegaStock robot
CommonCrawler Node
archive.org_bot
SEOkicks-Robot
oBot
TorgMailRuBot
Python-urllib/2.7
Yandex.Commerce.Pinger
BLEXBot
linux-gnu
Googlebot-Image
AdsBot-Google-Mobile
360Spider
Trident-не бот
YisouSpider
spider
PHP-Curl-Class
slurp
crawler
pinger
robot
WGet tools
FavIconizer
FaceBook bot
CFNetwork
NG 1.x (Exalead)
Phantom
RoadHouse Crawling System
SurveyBot
Perl tool
Sogou Spider
StackRambler
TITAN
Feedfetcher-Google
Netcraft
Fish search
RBSE Spider
Sift
W3C Validator
Konqueror
Yahoo Slurp
Mail.RU Bot
archive.org bot
Apache-HttpClient/
Flamingo_SearchEngine

 

[Shann]

Ну гугл-ботов больше

 

Googlebot
Googlebot-Nes
Googlebot-Images
Googlebot-Video
Google-Mobile
Google-Mobile AdSense
Google-AdSense
Google-AdsBot

и вот еще парочка которых нет у тебя

Teoma
ia_archiver
Lycos
Aport
WebAlta

 

Змінено 28 вересня 2017 користувачем L0ky

[agefremov]

Внесу свои 5 копеек

Боты:

Спойлер

TinEye-bot/0.51 (see http://www.tineye.com/crawler.html)
	Whoismindbot/1.0 (+http://www.whoismind.com/bot.html)
	Mozilla/5.0 (compatible; MSIE 8.0; Windows NT 6.3; www.alertra.com)
	R6_FeedFetcher(www.radian6.com/crawler)
	CheckHost (http://check-host.net/)
	YandeG 1.03
	yacybot (/global; amd64 Linux 3.16-0.bpo.2-amd64; java 1.7.0_65; Europe/en) http://yacy.net/bot.html
	SEMrushBot
	Protopage/3.0 (http://www.protopage.com)
	yacybot (/global; amd64 Linux 3.16.0-4-amd64; java 1.7.0_75; Europe/en) http://yacy.net/bot.html
	StatoolsBot (+http://www.statools.com/bot.html)
	adidxbot/2.0 (+http://search.msn.com/msnbot.htm)
	Mozilla/5.0 (compatible; Mail.RU_Bot/2.0; +http://go.mail.ru/help/robots)
	Mozilla/5.0 (Linux; Android 5.0; Nexus 5 Build/LRX21O) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/46.0.2490.76 Mobile Safari/537.36 PTST/281
	TurnitinBot (https://turnitin.com/robot/crawlerinfo.html)
	Scrapy/1.0.5 (+http://scrapy.org)
	yacybot (/global; amd64 Linux 4.4.0-31-generic; java 1.8.0_91; Europe/en) http://yacy.net/bot.html
	XmlSitemapGenerator - http://xmlsitemapgenerator.org
	Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.0) Match by Siteimprove.com
	Mozilla/5.0 (compatible; SeznamBot/3.2; +http://napoveda.seznam.cz/en/seznambot-intro/)
	WatchMouse/18990 (http://watchmouse.com/ ; gab)
	WatchMouse/8.4.0.3 (http://watchmouse.com/ ; hkhkg02.watchmouse.net)
	Mozilla/5.0 (compatible; LinkpadBot/1.06; +http://www.linkpad.ru)
	Mozilla/5.0 (compatible; heritrix/1.14.2 +http://rjpower.org)
	yacybot (webportal/global; amd64 Linux 2.6.32-5-amd64; java 1.6.0_18; Europe/en) http://yacy.net/bot.html
	PercolateCrawler/4 ([email protected])
	msnbot-UDiscovery/2.0b (+http://search.msn.com/msnbot.htm)
	Mozilla/5.0 (compatible; MSIE or Firefox mutant; not on Windows server;) Daum 4.1
	Mozilla/5.0 (compatible; spbot/4.0.3; +http://www.seoprofiler.com/bot )
	Mozilla/5.0 (compatible; LoadTimeBot/0.9; +http://www.loadtime.net/bot.html)
	UnwindFetchor/1.0 (+http://www.gnip.com/)
	nrsbot/5.0(loopip.com/robot.html)
	SemrushBot/0.9
	Mozilla/5.0 (compatible; UASlinkChecker/2.1; +https://udger.com/support/UASlinkChecker)
	yacybot (/global; amd64 Linux 3.11.10-21-desktop; java 1.7.0_51; America/en) http://yacy.net/bot.html
	Netvibes (http://www.netvibes.com)
	Acoon v4.1.0 (www.acoon.de)
	msnbot/2.0b (+http://search.msn.com/msnbot.htm).
	Mozilla/5.0 (compatible; ltbot/3.2.0.10 +http://www.kdsl.tu-darmstadt.de/de/kdsl/research-program/crawling-and-semantic-structuring/)
	smart.apnoti.com Robot/v1.34 (http://smart.apnoti.com/en/aboutApnotiWebCrawler.html)
	HubSpot Links Crawler 2.0 http://www.hubspot.com/
	yacybot (/global; x86 Windows XP 5.1; java 1.7.0_55; Europe/de) http://yacy.net/bot.html
	Mozilla/5.0 (compatible; DCPbot/1.0; +http://domains.checkparams.com/)
	Mozilla/5.0 (compatible; Exabot/3.0 (BiggerBetter); +http://www.exabot.com/go/robot)
	WebDoc (abuse-webdoc at service.moquadv.com)
	coccoc
	Mozilla/5.0 (compatible; spbot/4.0b; +http://www.seoprofiler.com/bot )
	Mozilla/5.0 (compatible; ExchangleBot/3.0; +https://www.exchangle.com/exchangling)
	Mozilla/5.0 (compatible; Qwantify/2.0n; +https://www.qwant.com/)
	OOZBOT/0.20 ( Setooz výrazný ako say-th-uuz, znamená mosty. ; http://www.setooz.com/oozbot.html ; agentname at setooz dot_com )
	SpeedySpider - http://www.entireweb.com
	Mozilla/5.0 (compatible; Heurekabot/3.1; +http://sluzby.heureka.cz/)
	crawler for netopian (http://www.netopian.co.uk/)
	L.webis/0.51 (http://webalgo.iit.cnr.it/index.php?pg=lwebis)
	Influencebot/0.9; (Automatic classification of websites; http://www.influencebox.com/; [email protected])
	Baiduspider+(+http://www.baidu.com/search/spider.htm)
	Mozilla/5.0 (X11; compatible; semantic-visions.com crawler; HTTPClient 3.1)
	Mozilla/5.0 (Windows; U; Windows NT 6.0; en-US; rv:1.9.2.28) Gecko/20120306 Firefox/99.0 YottaaMonitor
	Scrapy/1.1.1 (+http://scrapy.org)
	Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/27.0.1453.116 Safari/537.36 HubSpot Marketing Grader
	RyzeCrawler/1.1.1 ( http://www.domain2day.nl/crawler/)
	eBot / v.1.0a (http://alfa.elchron.cz)
	Sogou News Spider/4.0(+http://www.sogou.com/docs/help/webmasters.htm#07)
	Crowsnest/0.5 (+http://www.crowsnest.tv/)
	Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.4 (KHTML, like Gecko; Google Page Speed Insights) Chrome/22.0.1229 Safari/537.4
	DoCoMo/2.0 N902iS(c100;TB;W24H12)(compatible; moba-crawler; http://crawler.dena.jp/)
	Yeti/1.1 (NHN Corp.; http://help.naver.com/robots/)
	Experibot_v1
	Nekstbot - http://www.ipipan.waw.pl/nekst/nekstbot/
	Mozilla/5.0 (compatible; MJ12bot/v1.4.5; http://www.majestic12.co.uk/bot.php?+)
	Mozilla/5.0 (X11; Linux i686) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/30.0.1599.101 Safari/537.36; SSL-Crawler: http://crawler.dcsec.uni-hannover.de
	CorpusCrawler 2.0.19 (http://corpora.fi.muni.cz/crawler/);Project:CzCorpus
	Mozilla/5.0 (compatible; Semager/1.4; http://www.semager.de/blog/semager-bots/)
	Mozilla/5.0 (compatible; dlcbot/0.1; +http://www.drlinkcheck.com/)
	yacybot (freeworld/global; amd64 Linux 3.11.10-21-desktop; java 1.7.0_51; Europe/de) http://yacy.net/bot.html
	A6-Indexer/1.0 (http://www.a6corp.com/a6-web-scraping-policy/)
	CopperEgg/RevealUptime/DallasTX(linode)
	Mozilla/5.0 (compatible; Qwantify/2.1dw; +https://www.qwant.com/)/*
	Curious George - www.analyticsseo.com
	GozaikBot (www.gozaik.com;[email protected];www.gozaik.com/gozaikbot.html)
	Pu_iN Crawler (+http://semanticjuice.com/)
	Mozilla/5.0 (compatible; OpenindexDeepSpider/Nutch-1.5-dev; +http://www.openindex.io/en/webmasters/spider.html; systemsATopenindexDOTio)
	yacybot (freeworld/global; i386 Linux 3.0.0-17-generic-pae; java 1.6.0_23; Europe/en) http://yacy.net/bot.html
	yacybot (-global; amd64 FreeBSD 9.2-RELEASE-p10; java 1.7.0_65; Europe/en) http://yacy.net/bot.html
	FAST Enteprise Crawler/6 (www dot fastsearch dot com)
	Mozilla/5.0 (compatible; parsijoo-update-crawler; +http://www.parsijoo.ir/; [email protected])
	yacybot (freeworld/global; amd64 Windows Server 2008 6.0; java 1.7.0_25; Europe/en) http://yacy.net/bot.html
	Inspingbot/1.0 (+https://www.insping.com/)
	OrgProbe/0.9.4 (+http://www.blocked.org.uk)
	Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; Trident/5.0); 360Spider
	Mozilla/5.0 (Windows NT 6.1; WOW64; Trident/7.0; rv:11.0) like Gecko PTST/276
	Mozilla/5.0 (Windows NT 6.3; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/42.0.2311.90 Safari/537.36 PTST/276
	Mozilla/5.0 (compatible; SemrushBot/0.97~bl; +http://www.semrush.com/bot.html)
	Feedspot http://www.feedspot.com
	Zookabot/2.0;++http://zookabot.com
	drupact/0.7; http://www.arocom.de/drupact
	RobotsChecker/0.6 (+http://www.blocked.org.uk)
	yacybot (/global; i386 Linux 3.13.0-37-generic; java 1.7.0_65; Europe/en) http://yacy.net/bot.html
	Scrapy/1.0.3 (+http://scrapy.org)
	Testomatobot/1.0 (Linux x86_64; +http://www.testomato.com/testomatobot) minicrawler/4.0.0~beta12
	Slack-ImgProxy 0.66 (+https://api.slack.com/robots)
	Mozilla/5.0 (compatible; WebCookies/1.0; +https://webcookies.org/faq/#agent)
	NalezenCzBot/1.0 (http://www.nalezen.cz)
	yacybot (freeworld/global; amd64 Windows Server 2008 6.0; java 1.7.0_03; Europe/en) http://yacy.net/bot.html
	Mozilla/5.0 (compatible; AhrefsBot/5.1; +http://ahrefs.com/robot/)
	yacybot (freeworld/global; amd64 Windows 8 6.2; java 1.7.0_51; Europe/de) http://yacy.net/bot.html
	yacybot (/global; amd64 Windows 7 6.1; java 1.8.0_65; Europe/de) http://yacy.net/bot.html
	Mozilla/4.0 compatible ZyBorg/1.0 ([email protected]; http://www.WISEnutbot.com)
	Mozilla/5.0 (en-us) AppleWebKit/537.36(KHTML, like Gecko; Google-Adwords-DisplayAds-WebRender;) Chrome/27.0.1453Safari/537.36
	Crawler powered by contentDetection (www.mindup.de)
	PagePeeker.com (info: http://pagepeeker.com/robots)
	Mozilla/5.0 (compatible; Linux x86_64; Mail.RU_Bot/Fast/2.0; +http://go.mail.ru/help/robots)
	coccoc/1.0 (http://help.coccoc.com/)
	Favicon downloader (+https://favico.be/bot.html)
	Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0) ([email protected])
	Mozilla/5.0 (compatible; DIY-SEOBot/0.1a; +http://www.upcity.com/bot.html)
	Mozilla/5.0 (Windows NT 6.3; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/49.0.2623.112 Safari/537.36 PTST/276
	Mozilla/5.0 (compatible; theoldreader.com)
	Mozilla/5.0+(compatible; UptimeRobot/2.0; http://www.uptimerobot.com/)
	hawkReader/1.8 (Link Parser; http://www.hawkreader.com/; Allow like Gecko) Build/f2b2566
	Mozilla/5.0 (compatible; heritrix/1.12.1 +http://www.webarchiv.cz)
	Mozilla/5.0 (compatible; YandexImages/3.0; +http://yandex.com/bots)
	yacybot (/global; amd64 Linux 3.14.32-xxxx-grs-ipv6-64; java 1.7.0_75; Europe/en) http://yacy.net/bot.html
	Sogou web spider/4.0(+http://www.sogou.com/docs/help/webmasters.htm#07)
	yacybot (/global; amd64 Windows 7 6.1; java 1.8.0_45; Europe/de) http://yacy.net/bot.html
	larbin_2.6.3 [email protected]
	yacybot (freeworld/global; amd64 Linux 3.10.17-gentoo; java 1.7.0_45; UTC/en) http://yacy.net/bot.html
	MeMoNewsBot/2.0 (http://www.memonews.com/en/crawler)
	Mozilla/5.0 (compatible; MSIE 10.0; Windows NT 6.1; WOW64; Trident/6.0) PTST/281
	Mozilla/5.0 (compatible; DCPbot/1.5; +http://domains.checkparams.com/)
	Mozilla/5.0 (compatible; SemrushBot/0.97; +http://www.semrush.com/bot.html)
	BlackBerry9000/4.6.0.167 Profile/MIDP-2.0 Configuration/CLDC-1.1 VendorID/102 ips-agent
	Mozilla/5.0 (compatible; Baiduspider-cpro; +http://www.baidu.com/search/spider.html)
	yacybot (/global; amd64 Linux 3.2.0-4-amd64; java 1.7.0_65; Europe/en) http://yacy.net/bot.html
	Mozilla/5.0 (compatible; electricmonk/3.1.1 +https://www.duedil.com/our-crawler/)
	Hatena Antenna/0.5 (http://a.hatena.ne.jp/help)
	Backlink-Ceck.de (+http://www.backlink-check.de/bot.html)
	Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; WOW64; Trident/5.0) commoncrawl.org/research//Nutch-1.7-SNAPSHOT
	Mozilla/5.0 (compatible; Steeler/3.5; http://www.tkl.iis.u-tokyo.ac.jp/~crawler/)
	LapozzBot/1.4 (+http://robot.lapozz.com)
	Mozilla/5.0 (WhatsMyIP.org HTTP_Compression_Test) http://whatsmyip.org/ua
	Mozilla/5.0 (compatible; MSIE or Firefox mutant; not on Windows server;) Daumoa/4.0 (Following Mediapartners-Google)
	Mozilla/5.0 (compatible; ExaleadCloudview/6;)
	ADmantX Platform Semantic Analyzer - ADmantX Inc. - www.admantx.com - [email protected]
	msnbot/1.0 (+http://search.msn.com/msnbot.htm)
	Mozilla/5.0 (Windows NT 5.1) AppleWebKit/537.4 (KHTML, like Gecko) Chrome/22.0.1229.79 Safari/537.4 LinkTiger 2.0
	Pixray-Seeker/1.1 (Pixray-Seeker; [email protected])
	mbot v.1.16
	MXT/Nutch-1.12-SNAPSHOT (http://t.co/GSRLLKex24; informatique at mixdata dot com)
	Testomatobot/1.0 (Linux x86_64; +http://www.testomato.com/testomatobot) minicrawler/3.0.0
	polybot 1.0 (http://cis.poly.edu/polybot/)
	Mozilla/5.0 (compatible; YandexMetrika/2.0; +http://yandex.com/bots mtweb01t.yandex.ru)
	Mozilla/5.0 (compatible; memoryBot/1.21.46 +http://internetmemory.org/en/)
	Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_1) AppleWebKit/600.2.5 (KHTML, like Gecko) Version/8.0.2 Safari/600.2.5 (Applebot/0.1)
	yacybot (amd64 Linux 2.6.26-2-amd64; java 1.6.0_0; Europe/en) http://yacy.net/bot.html
	yacybot (/global; x86 Windows 8.1 6.3; java 1.8.0_45; America/en) http://yacy.net/bot.html
	Mozilla/5.0 (compatible; startmebot/1.0; +http://www.start.me/bot)
	yacybot (amd64 Windows 7 6.1; java 1.6.0_21; Europe/fr) http://yacy.net/bot.html
	yacybot (/global; amd64 Windows 7 6.1; java 1.8.0_101; Asia/ru) http://yacy.net/bot.html
	yacybot (freeworld/global; amd64 Linux 3.0.0-17-generic; java 1.6.0_23; Europe/de) http://yacy.net/bot.html
	alexa v0.1.4 (http://www.openwebspider.org/)
	http://arachnode.net 1.4
	Photon/1.0
	NetpeakCheckerBot
	GIDBot/3.0 (+http://www.gidnetwork.com/tools/gzip-test.php)
	Yandex/1.01.001 (compatible; Win16; P)
	w3dt.net httphr/2.0
	yacybot (/global; amd64 Windows 7 6.1; java 1.8.0_51; Europe/de) http://yacy.net/bot.html
	iqdb/0.1 (+http://iqdb.org/)
	Mozilla/5.0 (compatible; GimmeUSAbot/1.0; +https://gimmeusa.com/pages/crawler)
	Motoricerca-Robots.txt-Checker/1.0 (http://tool.motoricerca.info/robots-checker.phtml)
	Mozilla/5.0 (compatible; Tagoobot/3.0; +http://www.tagoo.ru)
	Mozilla/5.0 (Windows NT 6.1; WOW64; rv:26.0) Gecko/20100101 Firefox/26.0 Evidon ([email protected])
	Mozilla/5.0 (compatible; Nmap Scripting Engine; http://nmap.org/book/nse.html)
	yacybot (i386 Linux 2.6.32-22-generic; java 1.6.0_20; Europe/de) http://yacy.net/bot.html
	Mozilla/5.0 (Linux; Android 4.4.3; HTC One Build/KTU84L) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/40.0.2125.111 Mobile Safari/537.36 DareBoost
	mindUpBot (datenbutler.de)
	Mozilla/5.0 (compatible; monitis - premium monitoring service; http://www.monitis.com)
	Mozilla/5.0 (compatible; RankSonicSiteAuditor/1.0; +https://ranksonic.com/ranksonic_sab.html)
	Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/40.0.2125.111 Safari/537.36 DareBoost
	Zookabot/2.1;++http://zookabot.com
	yacybot (freeworld/global; amd64 Windows 7 6.1; java 1.6.0_24; Europe/de) http://yacy.net/bot.html
	Mozilla/5.0 (compatible; Google-Structured-Data-Testing-Tool +http://developers.google.com/structured-data/testing-tool/)
	Mozilla/5.0 (compatible; Plukkie/1.2; http://www.botje.com/plukkie.htm)
	focusbot/1.1
	Mozilla/5.0 (compatible; idmarch Automatic.beta/1.2; +http://www.idmarch.org/bot.html)
	CorpusCrawler 2.0.14 (http://corpora.fi.muni.cz/crawler/)
	Mozilla/5.0 (compatible; XML Sitemaps Generator; https://www.xml-sitemaps.com) Gecko XML-Sitemaps/1.0
	ImplisenseBot 1.1
	Promotion_Tools_www.searchenginepromotionhelp.com
	Aboundex/0.2 (http://www.aboundex.com/crawler/)
	CorpusCrawler 2.0.22 (http://corpora.fi.muni.cz/crawler/);Project:CzCorpus
	yacybot (/global; amd64 no-os-name no-os-version; java no-java-version; Europe/en) http://yacy.net/bot.html
	LumpImageSearch/0.1 (+http://lump.co/about/bot)
	Mozilla/4.0 (compatible; NaverBot/1.0; http://help.naver.com/customer_webtxt_02.jsp)
	Mozilla/5.0 (compatible; UASlinkChecker/2.0; +http://udger.com/support/UASlinkChecker)
	GAChecker (+http://www.gachecker.com)
	yacybot (freeworld/global; x86 Windows 7 6.1; java 1.7.0_25; Europe/de) http://yacy.net/bot.html
	Influencebot/0.9; (Automatic classification of websites; http://www.influencebox.com/; [email protected])User-Agent: Mozilla/5.0 (X11; Linux i686; rv:9.0) Gecko/20100101 Firefox/9.0
	Experibot_v1 (https://dl.dropboxusercontent.com/u/8024465/site/Info.html)
	DNSPod-reporting(http://www.dnspod.cn/reporting)
	yacybot (freeworld/global; i386 Linux 3.2.0-23-generic; java 1.6.0_27; Europe/en) http://yacy.net/bot.html
	Mozilla/5.0 (Windows NT 6.3;compatible; Leikibot/1.0; +http://www.leiki.com)
	PostPost/1.0 (+http://postpo.st/crawlers)
	envolk/1.7 (+http://www.envolk.com/envolkspiderinfo.html)
	Snapbot/1.0 (Snap Shots, +http://www.snap.com)
	Mozilla/5.0 (Linux; Android 4.0.4; Galaxy Nexus Build/IMM76B) AppleWebKit/537.36 (KHTML, like Gecko; Google-Publisher-Plugin) Chrome/27.0.1453 Mobile Safari/537.36
	Mozilla/5.0 (compatible; Uptimebot/0.2.18; +http://www.uptime.com/uptimebot)
	MIA DEV/search:robot/0.0.1 (This is the MIA Bot - crawling for mia research project. If you feel unhappy and do not want to be visited by our crawler send an email to [email protected]; http://spider.neofonie.de; [email protected])
	Mozilla/5.0 (Windows NT 6.1; Trident/7.0; rv:11.0; PTST 2.386) like Gecko
	Jyxobot/1
	WebAlta Crawler/2.0 (http://www.webalta.net/ru/about_webmaster.html) (Windows; U; Windows NT 5.1; ru-RU)
	GC3pro+dir SEO Tools - Vers. 3.00b - For more informations: http://chkme.com/
	Mozilla/5.0 (X11; Linux x86_64; rv:45.0; GTmetrix https://gtmetrix.com/) Gecko/20100101 Firefox/45.0
	Mozilla/5.0 (Windows NT 6.1; WOW64; rv:18.0) Gecko/20100101 Firefox/18.0 AppEngine-Google; (+http://code.google.com/appengine; appid: s~aeshortener)
	Priceonomics Analysis Engine - Fetch/1.0
	Ruky-Roboter (Version: 1.06, powered by www.ruky.de +http://www.ruky.de/bot.html)
	Baiduspider+(+http://help.baidu.jp/system/05.html)
	Openstat/0.1
	Yandex/1.01.001 (compatible; Win16; m)
	Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_1) AppleWebKit/600.2.5 (KHTML, like Gecko) Version/8.0.2 Safari/600.2.5 (Applebot/0.1; +http://www.apple.com/go/applebot)
	Mozilla/5.0 (compatible; OpenindexSpider/Nutch-1.5-dev; +http://www.openindex.io/en/webmasters/spider.html)
	Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; Multiviewbot
	CERT.at-Statistics-Survey/1.0 (http://www.cert.at/about/consec/content.html)
	Mozilla/5.0 (compatible; pmoz.info ODP link checker; +http://pmoz.info/doc/botinfo.htm)
	yacybot (freeworld/global; x86_64 Mac OS X 10.6.8; java 1.6.0_29; Asia/ru) http://yacy.net/bot.html
	gonzo/1[P] (+http://www.suchen.de/faq.html)
	MixBot (+http://t.co/GSRLLKex24)
	yacybot (/global; amd64 Linux 3.2.0-4-amd64; java 1.7.0_60; Europe/en) http://yacy.net/bot.html
	Mozilla/5.0 (compatible; imbot/0.1 +http://internetmemory.org/en/
	Mozilla/5.0 (en-us) AppleWebKit/525.13 (KHTML, like Gecko; Google Web Preview) Version/3.1 Safari/525.13
	Mozilla/5.0 (compatible; Pi-Monster; https://pricepi.com/)
	ThumbSniper (http://thumbsniper.com)
	Shelob ([email protected])
	Mozilla/5.0 (Macintosh; Intel Mac OS X 10_7_3) AppleWebKit/537.36 (KHTML, like Gecko, Google-Publisher-Plugin) Chrome/27.0.1453 Safari/537.36
	Mozilla/5.0 (compatible; KaloogaBot; http://www.kalooga.com/info.html?page=crawler)
	yacybot (/global; arm Linux 4.1.13+; java 1.8.0_40-internal; Etc/de) http://yacy.net/bot.html
	CorporateNewsSearchEngine/Nutch-1.7 (http://pibs.co/news-search-engine)
	Mozilla/5.0 (compatible; MSIE 10.0; Windows NT 6.1; WOW64; Trident/6.0) PTST/276
	Yandex/1.01.001 (compatible; Win16; I)
	FlightDeckReportsBot/2.0 (http://www.flightdeckreports.com/pages/bot)
	Scrapy/0.24.4 (+http://scrapy.org)
	ADmantX Platform Semantic Analyzer US - Turn - ADmantX Inc. - www.admantx.com - [email protected]
	Kyoto-Tohoku-Crawler/v1 (Mozilla-compatible; [email protected]; http://nlp.ist.i.kyoto-u.ac.jp/?crawling-kt)
	Mozilla/5.0 (compatible; Scarlett/ 1.0; +http://www.ellerdale.com/crawler.html)
	Mozilla/5.0 (compatible; NetcraftSurveyAgent/1.0; [email protected])
	www.deadlinkchecker.com Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/46.0.2490.86 Safari/537.36
	http://arachnode.net 1.2
	Mozilla/5.0 (compatible; Plukkie/1.5; http://www.botje.com/plukkie.htm)
	yacybot (freeworld/global; arm Linux 4.4.11-v7+; java 1.7.0_101; Etc/en) http://yacy.net/bot.html
	Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.0.16) Gecko/2009121601 Ubuntu/9.04 (jaunty) Firefox/3.0.16 Specificfeeds- http://www.specificfeeds.com
	Mozilla/5.0 (compatible; evc-batch/2.0)
	Orbiter/1.2 (http://dailyorbit.com/)
	crawler4j (https://github.com/yasserg/crawler4j/)
	Mozilla/5.0 (compatible; SEOdiver/1.0; +http://www.seodiver.com/bot)
	Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/51.0.2704.79 Safari/537.36 (https://shrinktheweb.com)
	Scopia crawler 1.0 (+http://www.scopia.co)
	yacybot (i386 Linux 2.6.23; java 1.6.0_06; Europe/en) http://yacy.net/bot.html
	Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.1.2) Gecko/20090729 Firefox/3.5.2 (.NET CLR 3.5.30729; Diffbot/0.1; +http://www.diffbot.com)
	Mozilla/5.0 (compatible; LXRbot/1.0;http://www.lxrmarketplace.com/,[email protected])
	yacybot (freeworld/global; amd64 Linux 3.8.13-gentoo; java 1.7.0_21; Europe/de) http://yacy.net/bot.html
	Mozilla/5.0 (compatible; Sitemap Generator/1.3; +http://www.check-domains.com/sitemap/index.php)
	DuckDuckBot/1.1; (+http://duckduckgo.com/duckduckbot.html)
	Mozilla/5.0 (compatible; BLEXBot/1.0; +http://webmeup-crawler.com/)
	Slack-ImgProxy 0.59 (+https://api.slack.com/robots)
	Mozilla/5.0 (compatible; Ezooms/1.0; [email protected])
	Mozilla/5.0 (compatible; YandexPagechecker/2.0; +http://yandex.com/bots)
	CoinCornerBot/1.1 ( https://www.coincorner.com/BitcoinBot)
	yacybot (freeworld/global; amd64 Linux 3.8.0-21-generic; java 1.6.0_27; Pacific/en) http://yacy.net/bot.html
	ScreenerBot Crawler Beta 2.0 (+http://www.ScreenerBot.com)
	gonzo1[P] +http://www.suchen.de/faq.html
	Mozilla/5.0 (compatible; MSIE or Firefox mutant;) Daum 4.1
	Mozilla/5.0 (compatible; 008/0.83; http://www.80legs.com/spider.html;) Gecko/2008032620
	Sogou web spider/4.02525A
	Visbot/2.0 (+http://www.visvo.com/en/webmasters.jsp;[email protected])
	Mozilla/5.0 (compatible; AcoonBot/4.10.8; +http://www.acoon.de/robot.asp)
	WatchMouse/18990 (http://watchmouse.com/ ; d3.watchmouse.com)
	Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/534.34 (KHTML, like Gecko) Qt/4.8.1 Safari/534.34 ShoppimonAgent/1.0 ([email protected])
	webinatorbot 1.1; +http://www.webinator.de
	findlinks/2.0.5 (+http://wortschatz.uni-leipzig.de/findlinks/)
	WikiDo/1.1 (http://wikido.com; [email protected])
	yacybot (freeworld/global; amd64 Linux 3.5.0-27-generic; java 1.7.0_03; Europe/de) http://yacy.net/bot.html
	Mozilla/5.0 (compatible; Uptimebot/0.1.73; +http://www.uptime.com/uptimebot)
	Semantifire1/0.20 ( -- ; http://www.setooz.com/oozbot.html ; agentname at setooz dot_com )
	Speedy Spider (http://www.entireweb.com/about/search_tech/speedy_spider/)
	Mozilla/5.0 compatible; yelpspider/yelpspider-1.0 (Crawlerbot run by Yelp Inc; yelpbot at yelp dot com)
	Mozilla/5.0 (compatible; OpenHoseBot/2.1; +http://www.openhose.org/bot.html)
	Mozilla/5.0 (compatible; emefgebot/beta; +http://emefge.de/bot.html)
	Mozilla/5.0 (compatible; OpenindexShallowSpider/Nutch-1.5-dev; +http://www.openindex.io/en/webmasters/spider.html; systemsATopenindexDOTio)
	Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/49.0.2623.75 Safari/537.36 Google (+https://developers.google.com/+/web/snippet/)
	Mozilla/5.0 (compatible; MetaJobBot; http://www.metajob.de/crawler)
	CCBot/1.0 (+http://www.commoncrawl.org/bot.html)
	Lijit Crawler (+http://www.lijit.com/robot/crawler)
	baypup/1.1 (Baypup; http://www.baypup.com/; [email protected])
	Mozilla/5.0 (FauBot/0.1; +http://buzzvolume.com/fau/)
	Mozilla/5.0 (compatible; NLNZ_IAHarvester2016/3.3.0 +https://natlib.govt.nz/publishers-and-authors/web-harvesting/domain-harvest)
	yacybot (/global; amd64 Linux 3.12.1; java 1.7.0_65; Europe/en) http://yacy.net/bot.html
	yacybot (amd64 Windows 7 6.1; java 1.6.0_18; Europe/de) http://yacy.net/bot.html
	SEOENGWorldBot/1.0 (+http://www.seoengine.com/seoengbot.htm)
	Mozilla/5.0 (compatible; Finderbots finder bot; +http://wiki.github.com/bixo/bixo/bixocrawler; [email protected])
	Pompos/1.3 http://dir.com/pompos.html
	Mozilla/5.0 (compatible; Gimme60bot/1.0; +http://gimme60.com) Firefox/16.0
	Mozilla/5.0 (compatible; MJ12bot/v1.4.6; http://mj12bot.com/)
	Mozilla/5.0 (compatible; Sysomos/1.0; +http://www.sysomos.com/; Sysomos)
	urlfan-bot/1.0; +http://www.urlfan.com/site/bot/350.html
	al_viewer ([email protected])
	LoadImpactRload/3.1.1 (Load Impact; http://loadimpact.com);
	findlinks/2.0.2 (+http://wortschatz.uni-leipzig.de/findlinks/)
	Mozilla/5.0 (compatible; spbot/3.1; +http://www.seoprofiler.com/bot )
	Mozilla/5.0 (compatible; AcoonBot/4.11.0; +http://www.acoon.de/robot.asp)
	WillyBot/1.1 (http://www.willyfogg.com/info/willybot)
	Norton-Safeweb
	WMCAI-robot (http://www.topicmaster.jp/wmcai/crawler.html)
	rogerbot/1.1 (http://moz.com/help/guides/search-overview/crawl-diagnostics#more-help, [email protected])
	Szukacz/1.5 (robot; www.szukacz.pl/jakdzialarobot.html; [email protected])
	rogerbot/1.0 (http://www.seomoz.org/dp/rogerbot, [email protected])
	ROR Sitemap Generator (http://www.rorweb.com)
	http://domino.research.ibm.com/comm/research_projects.nsf/pages/sai-crawler.callingcard.html
	MozillaTest/5.0 (compatible; YodaoBot/1.0; http://www.yodao.com/help/webmaster/spider/; )
	Mozilla/5.0 (compatible; Peew/1.0; http://www.peew.de/crawler/)
	Mozilla/5.0 (compatible; Website Analyzer/1.1; +http://www.check-domains.com/website-analysis/website-analyzer.php)
	Mozilla/5.0 (compatible; Gluten Free Crawler/1.0; +http://glutenfreepleasure.com/)
	PagePeeker.com
	CorpusCrawler 2.0.10 (http://corpora.fi.muni.cz/crawler/)
	yacybot (/global; amd64 Linux 3.2.0-4-amd64; java 1.7.0_101; Europe/cs) http://yacy.net/bot.html
	yacybot (/global; amd64 Linux 4.0.7-1-ck; java 1.8.0_45; Europe/de) http://yacy.net/bot.html
	Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/51.0.2704.103 Safari/537.36 PTST/284
	yacybot (freeworld/global; i386 Linux 3.12-1-686-pae; java 1.7.0_21; Europe/fr) http://yacy.net/bot.html
	Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.1; Trident/4.0; PTST 2.295)
	CovarioIDS/1.1 (http://www.covario.com/ids; support at covario dot com)
	Mozilla/5.0 (compatible; heritrix/1.14.4 +http://parsijoo.ir)
	www.adressendeutschland.de
	DialogSearch.com Bot 1.0;http://dialogsearch.com/webmasters
	librabot/2.0 (+http://search.msn.com/msnbot.htm)
	Mozilla/5.0 (compatible; heritrix/3.1.1-SNAPSHOT-20120116.200628 +http://www.archive.org/details/archive.org_bot)
	wsAnalyzer/1.0; ++http://www.wsanalyzer.com/bot.html
	Mozilla/5.0 (compatible; websays; +http://wiki.github.com/bixo/bixo/bixocrawler; [email protected])
	facebookexternalhit/1.0 (+http://www.facebook.com/externalhit_uatext.php)
	yacybot (/global; amd64 Linux 4.4.5-1-ARCH; java 1.8.0_77; America/en) http://yacy.net/bot.html
	websitepulse checker/1.1 (compatible; MSIE 5.5; Netscape 4.75; Linux)
	Mozilla/5.0 (compatible; memoryBot/1.21.14 +http://mignify.com/bot.html)
	SalesIntelligent/v1.0
	larbin_2.6.2 [email protected]
	Mozilla/5.0 (compatible; SiteCondor; http://www.sitecondor.com)
	yacybot (freeworld/global; amd64 Windows Server 2012 6.2; java 1.7.0_51; Europe/de) http://yacy.net/bot.html
	Mozilla/5.0 (compatible; Crawlera/1.10.2; UID 43063)
	yacybot (-global; amd64 Linux 2.6.32-042stab111.11; java 1.7.0_79; Europe/en) http://yacy.net/bot.html
	Mozilla/5.0 (compatible; SemrushBot/0.96.2; +http://www.semrush.com/bot.html)
	Linguee Bot (http://www.linguee.com/bot)
	yacybot (/global; x86 Windows XP 5.1; java 1.7.0_51; Europe/de) http://yacy.net/bot.html
	ICC-Crawler(Mozilla-compatible; ; http://kc.nict.go.jp/project1/crawl.html)
	Mozilla/5.0 (X11; U; Linux i686 (x86_64); en-US; rv:1.8.1.11) Gecko/20080109 (Charlotte/0.9t; http://www.searchme.com/support/) (Charlotte/0.9t; http://www.searchme.com/support/)
	Mozilla/5.0 (compatible; Goodzer/2.0; [email protected])
	Acoon v4.10.5 (www.acoon.de)
	CorpusCrawler 2.0.20 (http://corpora.fi.muni.cz/crawler/);Project:CzCorpus
	Mozilla/5.0 (compatible; AMZNKAssocBot/4.0 +http://affiliate-program.amazon.com)
	Mozilla/5.0 (compatible; MojeekBot/0.5; http://www.mojeek.com/bot.html)
	yacybot (/global; amd64 Linux 4.2.0-22-generic; java 1.7.0_91; Europe/en) http://yacy.net/bot.html
	Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/534.34 (KHTML, like Gecko) Qt/4.8.3 Safari/534.34 https://linkpeek.com
	MetaTagRobot/0.2 (http://www.seocentro.com/tools/search-engines/metatag-analyzer.html)
	Mozilla/5.0 (compatible; MJ12bot/v1.3.1; http://www.majestic12.co.uk/bot.php?+)
	Mozilla/5.0 (compatible; Uptimebot/0.2.29; +http://www.uptime.com/uptimebot)
	Mozilla/5.0 (compatible; Pandeo Bot; +http://pandeo.de/bot.php)
	Pixray-Seeker/1.1 (Pixray-Seeker; http://www.pixray.com/pixraybot; [email protected])
	EasyBib AutoCite (http://autocite-info.citation-api.com/)
	Mozilla/5.0 (compatible; OptimizationCrawler/0.2; +http://www.domainoptima.com/robot)
	AboutUsBot/Harpy (Website Analysis; http://www.aboutus.org/Aboutus:Bot; [email protected])
	Gigabot/2.0
	DoCoMo/2.0 N905i(c100;TB;W24H16) (compatible; Googlebot-Mobile/2.1; +http://www.google.com/bot.html)
	Morning Paper 1.0 (robots.txt compliant!)
	Mozilla/5.0 (compatible; SurdotlyBot/1.0; +http://sur.ly/bot.html; Linux; Android 4; iPhone; CPU iPhone OS 6_0_1 like Mac OS X) AppleWebKit/536.26 (KHTML, like Gecko) Version/6.0 Mobile/10A523 Safari/8536.25
	Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.1; Trident/4.0; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; Media Center PC 6.0; .NET4.0C; .NET4.0E; GWX:RED; PTST 2.386)
	SuperPagesUrlVerifyBot/1.0
	CopperEgg/RevealUptime/LondonUK(linode)
	Mozilla/5.0 (compatible; www.monitor.us - free monitoring service; http://www.monitor.us)
	Mozilla/5.0 (compatible; LinkMarketbot/1.2; +http://www.linkmarket.com/)
	Mozilla/5.0 (Windows; U; Windows NT 5.1; zh-CN; ) Firefox/1.5.0.11; 360Spider
	wangling
	Mozilla/5.0 (compatible; linkdexbot/2.0; +http://www.linkdex.com/about/bots/)
	RSSMicro.com RSS/Atom Feed Robot
	GarlikCrawler/1.1 (http://garlik.com/, [email protected])
	al_org_viewer ([email protected])
	Mozilla/5.0 (compatible; JadynAveBot; +http://www.jadynave.com/robot)
	dj-research/Nutch-1.11 (analytics@@demandjump.com)
	Karneval-Bot (Version: 1.06, powered by www.karnevalsuchmaschine.de +http://www.karnevalsuchmaschine.de/bot.html)
	Baiduspider-image+(+http://www.baidu.com/search/spider.htm)\nReferer: http://image.baidu.com/i?ct=503316480&z=0&tn=baiduimagedetail
	SauceNAO/1.0 (+http://saucenao.com/)
	Mozilla/5.0 (compatible; JobdiggerSpider +http://www.jobdigger.nl/spider)
	Yepi/1.0 (NHN Corp.; http://help.naver.com/robots/)
	Mozilla/5.0 (compatible; coccoc/1.0; +http://help.coccoc.com/)
	SSL Labs (https://www.ssllabs.com/about/assessment.html)
	Mozilla/5.0 (compatible; http://alyze.info)
	GigablastOpenSource/1
	Mozilla/5.0 (Windows; U; Windows NT 5.1;fr;rv:1.8.1) VoilaBotCollector BETA 0.1 (http://www.voila.com/)
	Vorboss Web Crawler [[email protected]]/Nutch-2.3
	Mozilla/5.0 (compatible; SecretSerachEngineLabs.com-SBSearch/0.9; http://www.secretsearchenginelabs.com/secret-web-crawler.php)
	Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko; Google Search Console) Chrome/27.0.1453 Safari/537.36
	YahooCacheSystem
	search.KumKie.com
	yacybot (webportal-global; x86 Windows Vista 6.0; java 1.7.0_25; Europe/en) http://yacy.net/bot.html
	Embedly [email protected]
	stq_bot (+http://www.searchteq.de)
	MSRBOT
	Mozilla/5.0 (compatible; WBSearchBot/1.1; +http://www.warebay.com/bot.html)
	CorpusCrawler 2.0.17 (http://corpora.fi.muni.cz/crawler/);Project:CzCorpus
	Mozilla/5.0 (compatible; heritrix/3.1.0-RC1 +http://boston.lti.cs.cmu.edu/crawler_12/)
	Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; Trident/5.0) PTST/276
	Mozilla/5.0 (compatible; WebCookies/1.0; +http://webcookies.org/faq/#agent)
	L.webis/0.44 (http://webalgo.iit.cnr.it/index.php?pg=lwebis)
	yacybot (freeworld/global; amd64 Linux 3.0.0-17-generic; java 1.6.0_23; America/en) http://yacy.net/bot.html
	Mozilla/5.0 (Windows NT 6.1; WOW64; rv:46.0) Gecko/20100101 Firefox/46.0 PTST/279
	Mozilla/5.0 (compatible; MSIE or Firefox mutant; not on Windows server;) Daumoa/4.0
	Mozilla/5.0 (compatible; YandexMetrika/2.0; +http://yandex.com/bots mtmon01i.yandex.ru)
	Y!J-BSC/1.0 crawler (http://help.yahoo.co.jp/help/jp/blog-search/)
	Mozilla/5.0 (compatible; UptimeRobot/1.0; http://www.uptimerobot.com/)
	NG-Search/0.86 (+http://www.ng-search.com)
	ichiro/3.0 (http://help.goo.ne.jp/help/article/1142)
	TwengaBot/1.1 (+http://www.twenga.com/bot.html)
	WebImages 0.3 ( http://herbert.groot.jebbink.nl/?app=WebImages )
	Mozilla/5.0 (compatible; aiHitBot/2.8; +http://endb-consolidated.aihit.com/)
	RyzeCrawler/1.1.1 (+http://www.domain2day.nl/crawler/)
	Mozilla/5.0 (Nekstbot; http://www.ipipan.waw.pl/nekst/nekstbot/)
	Mozilla/5.0 (compatible; adidxbot/2.0; http://www.bing.com/bingbot.htm)
	Mozilla/5.0 (compatible; SWEBot/1.0; +http://swebot-crawler.net)
	WatchMouse/8.4.0.3 (http://watchmouse.com/ ; gblon01.watchmouse.net)
	yacybot (/global; amd64 Windows 8.1 6.3; java 1.7.0_55; Europe/de) http://yacy.net/bot.html
	Page Analyzer v4.0 ( http://www.ranks.nl/ )
	web_bh ([email protected])
	findlinks/1.1.6-beta3 (+http://wortschatz.uni-leipzig.de/findlinks/)
	Mozilla/5.0 (compatible; Findxbot/1.0; +http://www.findxbot.com)
	findlinks/2.0 (+http://wortschatz.uni-leipzig.de/findlinks/)
	Mozilla/5.0 (compatible; imagecoccoc/1.0; +http://help.coccoc.com/)
	PagesInventory (robot http://www.pagesinventory.com)
	Mozilla/5.0 (compatible; aiHitBot/1.0-DS; +http://www.aihit.com/)
	tagSeoBot/1.0 (http://www.tagseoblog.de/tools)
	Mozilla/5.0 (en-us) AppleWebKit/537.36 (KHTML, like Gecko; Google PP Default) Chrome/27.0.1453 Safari/537.36
	404 Checker [http://www.404checker.com/user-agent]
	CopperEgg/RevealUptime/
	Mozilla/5.0 (compatible; adidxbot/2.0; +http://www.bing.com/bingbot.htm)
	Mozilla/5.0 (compatible; ZumBot/1.0; http://help.zum.com/inquiry)
	CopperEgg/RevealUptime/TokyoJapan
	Mozilla/5.0 (compatible; MojeekBot/0.2; http://www.mojeek.com/bot.html)
	GoSquared-Status-Checker/0.2
	Mozilla/5.0 (Windows NT 6.0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/99.0 Safari/537.36 YottaaMonitor
	WatchMouse/18990 (http://watchmouse.com/ ; bc.watchmouse.com)
	yacybot (freeworld/global; amd64 Linux 3.2.1-gentoo-r2; java 1.6.0_24; Europe/de) http://yacy.net/bot.html
	Mozilla/5.0 (compatible; adbeat_bot; [email protected]; [email protected])
	linkdexbot/Nutch-1.0-dev (http://www.linkdex.com/; crawl at linkdex dot com)
	Heurekabot-Feed/1.0 (+http://sluzby.heureka.cz/napoveda/heurekabot/)
	Mozilla/5.0 (compatible; Charlotte/1.1; http://www.searchme.com/support/)
	yacybot (/global; amd64 Linux 3.10.0-229.7.2.el7.x86_64; java 1.8.0_45; Europe/en) http://yacy.net/bot.html
	LSSRocketCrawler/1.0 LightspeedSystems
	Mozilla/5.0 (X11; U; Linux Core i7-4980HQ; de; rv:32.0; compatible; Jobboerse.com; http://www.xn--jobbrse-d1a.com) Gecko/20100401 Firefox/24.0
	findlinks/2.2 (+http://wortschatz.uni-leipzig.de/findlinks/)
	Mozilla/5.0 (compatible; kulturarw3 +http://www.kb.se/om/projekt/Svenska-webbsidor---Kulturarw3/)
	Mozilla/5.0 (compatible; CloudFlare-AlwaysOnline/1.0; +http://www.cloudflare.com/always-online) AppleWebKit/534.34
	Mozilla/5.0 (compatible; SemrushBot/0.96.4; +http://www.semrush.com/bot.html)
	woobot/2.0
	GarlikCrawler/1.2 (http://garlik.com/, [email protected])
	yacybot (webportal-global; amd64 Linux 3.2.0-4-amd64; java 1.7.0_67; Europe/en) http://yacy.net/bot.html
	Mozilla/5.0 (compatible; AboutUsBot Johnny5/2.0; +http://www.AboutUs.org/)
	yacybot (/global; amd64 Linux 3.10.0-327.22.2.el7.x86_64; java 1.7.0_101; Etc/en) http://yacy.net/bot.html
	Mozilla/5.0 (Linux; Android 4.1.2; Galaxy Nexus Build/JZO54K; GTmetrix http://gtmetrix.com/) AppleWebKit/537.22 (KHTML, like Gecko) Chrome/26.0.1410.58 Mobile Safari/537.22
	Nuhk/2.4 ( http://www.neti.ee/cgi-bin/abi/Otsing/Nuhk/)
	Grahambot/0.1 (+http://www.sunaga-lab.com/graham-bot)
	yacybot (/global; amd64 Linux 3.16.0-4-amd64; java 1.7.0_91; Europe/de) http://yacy.net/bot.html
	SEO Consulting; Redirect Checker Tool V.02; IP:
	Mozilla/5.0 (compatible; Pro Sitemaps Generator; https://pro-sitemaps.com) Gecko Pro-Sitemaps/1.0
	yacybot (/global; amd64 Linux 4.4.10-antix.1-amd64-smp; java 1.8.0_101; Europe/en) http://yacy.net/bot.html
	Mozilla/5.0 (Windows NT 6.2; WOW64) Runet-Research-Crawler (itrack.ru/research/cmsrate; [email protected])
	Mozilla/5.0 (compatible; Crawler/0.9; http://linkfluence.net/)
	ADmantX Platform Semantic Analyzer US Async - ADmantX Inc. - www.admantx.com - [email protected]
	GetProxi.es-bot/1.1 (http://getproxi.es/spiderinfo/)
	Mozilla/5.0 (Windows NT 6.3; WOW64; rv:46.0) Gecko/20100101 Firefox/46.0 PTST/277
	Pinterest/0.2 (+http://www.pinterest.com/)
	CopperEgg/RevealUptime/AtlantaGA(linode)
	OdklBot/1.0 ([email protected])
	Mozilla/5.0 (compatible; Exabot-Images/3.0; +http://www.exabot.com/go/robot)
	yacybot (freeworld/global; amd64 Linux 3.2.1-gentoo-r2; java 1.6.0_22; Europe/de) http://yacy.net/bot.html
	MojeekBot/0.2 (archi; http://www.mojeek.com/bot.html)
	Sogou web spider/4.05252A
	Mozilla/5.0 (Windows NT 6.3; WOW64; rv:45.0) Gecko/20100101 Firefox/45.0 PTST/276
	Mozilla/5.0 (compatible; MotoMinerBot/1.0; +https://motominer.com/Bot)
	NG/2.0
	Mozilla/5.0 (compatible; heritrix/1.14.2 +http://www.webarchiv.cz)
	StackRambler/2.0 (MSIE incompatible)
	Baiduspider+(+http://www.baidu.jp/spider/)
	yacybot (freeworld/global; amd64 Windows 7 6.1; java 1.6.0_25; Europe/de) http://yacy.net/bot.html
	Mozilla/5.0 (compatible; houzzbot; +http://www.houzz.com/)
	Woko robot 3.0
	Mozilla/5.0 (compatible; Qwantify/2.0; +https://www.qwant.com/)
	yacybot (/global; amd64 Linux 4.2.0-27-generic; java 1.8.0_66-internal; America/en) http://yacy.net/bot.html
	ADmantX Platform Semantic Analyzer - APAC - ADmantX Inc. - www.admantx.com - [email protected]
	Mozilla/5.0 (compatible; Uptimebot/0.2.40; +http://www.uptime.com/uptimebot)
	Mozilla/5.0 (compatible; ExpertSearchSpider +http://www.expertsearch.nl/spider)
	Mozilla/5.0 (compatible; coccocbot-web/1.0; +http://help.coccoc.com/searchengine)
	Acoon v4.10.4 (www.acoon.de)
	Mozilla/5.0 (compatible; memoryBot/1.20.210 +http://internetmemory.org/en/)
	Readability/740ec9 - http://readability.com/about/
	Mozilla/5.0 (compatible; Apercite; +http://www.apercite.fr/robot/index.html)
	yacybot (i386 Linux 2.6.28-gentoo-r5; java 1.5.0_18; Europe/en) http://yacy.net/bot.html
	Mozilla/5.0 (compatible; suggybot v0.01a, http://blog.suggy.com/was-ist-suggy/suggy-webcrawler/)
	yacybot (amd64 Windows 7 6.1; java 1.6.0_14; Europe/de) http://yacy.net/bot.html
	yacybot (freeworld/global; amd64 Linux 3.3.4-1-ARCH; java 1.6.0_24; Europe/en) http://yacy.net/bot.html
	Mozilla/5.0 (compatible; AportWorm/3.2; +http://www.aport.ru/help)
	Mozilla/5.0 (compatible; memoryBot/1.20.235 +http://internetmemory.org/en/)
	findlinks/2.6 (+http://wortschatz.uni-leipzig.de/findlinks/)
	Mozilla/5.0 (compatible; Hailoobot/1.2; +http://www.hailoo.com/spider.html)
	eCommerceBot (http://www.ehandel.se/botinfo.html)
	Mozilla/5.0(compatible;Sosospider/2.0;+http://help.soso.com/webspider.htm)
	yacybot (/global; amd64 Linux 4.3.0-gentoo-ARCH; java 1.7.0_85; Europe/en) http://yacy.net/bot.html
	Nymesis/2.0 (http://nymesis.com)
	CopperEgg/RevealUptime/OregonUSA
	uclassify.com/1.0
	Mozilla/5.0 (compatible; Butterfly/1.0; +http://labs.topsy.com/butterfly.html) Gecko/2009032608 Firefox/3.0.8
	Mozilla/5.0 (compatible; Prlog/1.0; +http://prlog.ru/)
	Slack-ImgProxy 1.106 (+https://api.slack.com/robots)
	AdnormCrawler www.adnorm.com/crawler
	Mozilla/5.0 (compatible; YandexZakladki/3.0; +http://yandex.com/bots)
	Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/49.0.2623.75 Safari/537.36 Google Favicon
	Mozilla/5.0 (compatible; Sonic/1.0; http://www.yama.info.waseda.ac.jp/~crawler/info.html)
	Mozilla/5.0 (iPhone; U; CPU iPhone OS 4_1 like Mac OS X; en-us) AppleWebKit/532.9 (KHTML, like Gecko) Version/4.0.5 Mobile/8B117 Safari/6531.22.7 (compatible; Googlebot-Mobile/2.1; +http://www.google.com/bot.html)
	ICC-Crawler/2.0 (Mozilla-compatible; ; http://kc.nict.go.jp/project1/crawl.html)
	Mozilla/4.0 ([email protected])
	bot-pge.chlooe.com/1.0.0 (+http://www.chlooe.com/)
	Mozilla/5.0 (compatible; GroupHigh/1.0; +http://www.grouphigh.com/)
	Mozilla/5.0 (compatible; Webmaster tools +http://sitexy.com/)
	yacybot (/global; amd64 Windows 8.1 6.3; java 1.8.0_40; Europe/de) http://yacy.net/bot.html
	Mozilla/5.0 (compatible; MJ12bot/v1.4.1; http://www.majestic12.co.uk/bot.php?+)
	Mozilla/5.0 (compatible; spbot/4.0.6; +http://www.seoprofiler.com/bot )
	Mozilla/5.0 (compatible; EuripBot/2.0; +http://www.eurip.com)
	findlinks/2.1 (+http://wortschatz.uni-leipzig.de/findlinks/)
	Sogou web spider/4.025251
	SETOOZBOT/5.0 ( http://www.setooz.com/bot.html )
	Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0) ([email protected])
	Mozilla/5.0 (compatible; spbot/2.0.4; +http://www.seoprofiler.com/bot )
	Mozilla/5.0 (TweetmemeBot/4.0; +http://datasift.com/bot.html) Gecko/20100101 Firefox/31.0
	mozilla/5.0 ([email protected])
	Mozilla/5.0 (compatible; spbot/2.1; +http://www.seoprofiler.com/bot )
	Mozilla/5.0 (Windows NT 6.2; WOW64) AppleWebKit/537.4 (KHTML, like Gecko) Chrome/98 Safari/537.4 (StatusCake SSL Monitor)
	Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; Trident/5.0; PTST 2.385)
	Mozilla/5.0 (compatible; evc-batch/2.0.20160608212921)
	Mozilla/5.0 (compatible; Mail.RU_Bot/2.0)
	seebot/2.0 (+http://www.seegnify.com/bot)
	bl.uk_lddc_bot/3.3.0-LBS-2016-02 (+http://www.bl.uk/aboutus/legaldeposit/websites/websites/faqswebmaster/index.html)
	CommaFeed/2.3.0-SNAPSHOT (https://www.commafeed.com)
	OmniExplorer_Bot/5.91c (+http://www.omni-explorer.com) WorldIndexer
	hledejLevne.cz/2.0
	page_verifier (http://www.securecomputing.com/goto/pv)
	url_test ([email protected])
	Mozilla/5.0 (X11; Linux x86_64; rv:10.0.12) Gecko/20100101 Firefox/21.0 WordPress.com mShots
	Mozilla/5.0 (compatible; Linux x86_64; Mail.RU_Bot/2.0; +http://go.mail.ru/help/robots)
	Is is up? (+http://isitup.org)
	Metaspinner/0.01 (Metaspinner; http://www.meta-spinner.de/; [email protected]/)
	TwengaBot-2.0 Champigny (+http://www.twenga.com/bot.html)
	LivelapBot/0.2 (http://site.livelap.com/crawler)
	HubSpot Crawler 1.0 http://www.hubspot.com/
	Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/45.0.2454.101 Safari/537.36 TinEye/1.0 (via http://www.tineye.com/)
	yacybot (/global; x86 Windows 10 10.0; java 1.8.0_73; Europe/de) http://yacy.net/bot.html
	Scrapy/0.24.6 (+http://scrapy.org)
	FAST-WebCrawler/3.6/FirstPage (atw-crawler at fast dot no;http://fast.no/support/crawler.asp)
	Baiduspider+(+http://www.baidu.com/search/spider_jp.html)
	Mozilla/5.0 (compatible; seplinkbot/1.0 )
	Mozilla/5.0 (compatible; Falconsbot; +http://ws.nju.edu.cn/falcons/)
	Mozilla/5.0 (Windows NT 6.1; WOW64; rv:46.0) Gecko/20100101 Firefox/46.0 PTST/277
	Mozilla/4.0 (Toread-Crawler/1.1; +http://news.toread.cc/crawler.php)
	TinEye-bot/0.02 (see http://www.tineye.com/crawler.html)
	yacybot (freeworld/global; i386 Linux 2.6.32-39-generic-pae; java 1.6.0_20; Europe/en) http://yacy.net/bot.html
	Mozilla/5.0 (compatible; parsijoo; +http://www.parsijoo.ir/; [email protected])
	Mozilla/5.0 (compatible; spbot/4.4.0; +http://OpenLinkProfiler.org/bot )
	MaxPoint Bot (+http://www.maxpoint.com)
	Mozilla/5.0 (compatible; Infohelfer/1.2.0; +http://www.infohelfer.de/crawler.php)
	ExB Language Crawler 2.1.1 (+http://www.exb.de/crawler)
	cg-eye interactive
	ZumBot/1.0 (ZUM Search; http://help.zum.com/inquiry)
	rogerbot/1.1 (http://moz.com/help/guides/search-overview/crawl-diagnostics#more-help, [email protected])
	Mozilla/5.0 (compatible; Crawlera/1.10.2; UID 47129)
	Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:14.0) Gecko/20100101 Firefox/14.0.1 AppEngine-Google; (+http://code.google.com/appengine; appid: s~metacdn-hr)
	Mozilla/5.0 (compatible; WoTBoT; +https://www.wslta.com/WoTBoT.html)
	HolmesBot (http://holmes.ge)
	Baiduspider-image+(+http://www.baidu.com/search/spider.htm)
	Mozilla/5.0 (compatible; alexa site audit/1.0; +http://www.alexa.com/help/webmasters; [email protected])
	Mediapartners-Google
	Mozilla/5.0 (compatible; MFGPagesBot/2.1; http://www.mfgpages.com)
	larbin_2.6.2 [email protected]
	Mozilla/5.0 (Windows; U; Windows NT 6.0; en-GB; rv:1.0; trendictionbot0.5.0; trendiction search; http://www.trendiction.de/bot; please let us know of any problems; web at trendiction.com) Gecko/20071127 Firefox/3.0.0.11
	BlogPulseLive ([email protected])
	WeSEE:Search/0.1 (Alpha, http://www.wesee.com/en/support/bot/)
	yacybot (freeworld/global; i386 Linux 3.0.0-17-generic; java 1.6.0_23; America/en) http://yacy.net/bot.html
	ImplisenseBot 1.0
	Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; WOW64; Trident/5.0; PTST 2.386)
	gonzo2[P] +http://www.suchen.de/faq.html
	Mozilla/5.0 (compatible; LXRbot/1.0; http://lxrseo.com/, [email protected])
	Mozilla/5.0 (compatible; Arachnophilia/1.0; +http://arachnys.com/)
	Mozilla/5.0 (compatible; CloudServerMarketSpider/1.0; +http://cloudservermarket.com/spider.html)
	kalooga/KaloogaBot (Kalooga; http://www.kalooga.com/info.html?page=crawler)
	yacybot (webportal/global; x86_64 Mac OS X 10.9.2; java 1.6.0_65; Europe/de) http://yacy.net/bot.html
	Mozilla/5.0 (compatible; DotBot/1.1; http://www.dotnetdotcom.org/, [email protected])
	yacybot (/global; amd64 Linux 4.1.19-gentoo; java 1.7.0_95; Europe/pl) http://yacy.net/bot.html
	Mozilla/5.0 (Windows NT 6.3; WOW64; Trident/7.0; rv:11.0) like Gecko PTST/276
	Mozilla/5.0 (compatible; RankActiveLinkBot; +https://rankactive.com/resources/rankactive-linkbot)
	audisto.com full crawler 3.26.431 (refer to in robots.txt as audisto, see https://audisto.com/bot)
	Mozilla/5.0 (compatible; spbot/4.0.1; +http://www.seoprofiler.com/bot )
	Mozilla/5.0 (compatible; OpenindexShallowSpider/Nutch-1.5-dev; +http://www.openindex.io/en/webmasters/spider.html)
	mozilla/5.0 (compatible; discobot/1.1; +http://discoveryengine.com/discobot.html)
	gonzo2[p] (+http://www.suchen.de/faq.html)
	Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; Trident/5.0); 360Spider(compatible; HaosouSpider; http://www.haosou.com/help/help_3_2.html)
	Mozilla/5.0 (compatible; spbot/2.0.1; +http://www.seoprofiler.com/bot/ )
	WatchMouse/18990 (http://watchmouse.com/ ; uk)
	yacybot (freeworld/global; amd64 Linux 3.1.10-hardened; java 1.7.0_03-icedtea; Europe/en) http://yacy.net/bot.html
	VeBot (+http://www.veinteractive.com/vebot)
	Mozilla/5.0 (compatible; NLNZ_IAHarvester2013 +http://natlib.govt.nz/about-us/current-initiatives/web-harvest-2012)
	findlinks/1.1.3-beta8 (+http://wortschatz.uni-leipzig.de/findlinks/)
	Mozilla/5.0 (Linux; U; Android 2.3.4; generic) AppleWebKit/537.36 (KHTML, like Gecko; Google Web Preview) Version/4.0 Mobile Safari/537.36
	DialogSearch.com Bot 1.4;http://dialogsearch.com/webmasters
	Mozilla/5.0 (compatible; GurujiBot/1.0; +http://www.guruji.com/en/WebmasterFAQ.html)
	Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9) AppleWebKit/537.71 (KHTML, like Gecko) Version/7.0 Safari/537.71 (Rival IQ, rivaliq.com)
	wscheck.com/1.0.0 (+http://wscheck.com/)
	Mozilla/5.0 (Windows NT 6.3; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/30.0.1599.69 Safari/537.36 Webthumb/2.0
	Mozilla/5.0 (compatible; FlipboardProxy/1.1; +http://flipboard.com/browserproxy)
	WeSEE:Ads/PictureBot (http://www.wesee.com/bot/)
	Mozilla/5.0 (compatible; Dataprovider/6.101; +https://www.dataprovider.com/)
	Kyoto-Crawler/n1.0 (Mozilla-compatible; [email protected]; http://nlp.ist.i.kyoto-u.ac.jp/?crawling)
	Mozilla/5.0 (compatible; SEOkicks-Robot; +http://www.seokicks.de/robot.html)
	Mozilla/5.0 (compatible; spbot/4.0.9; +http://OpenLinkProfiler.org/bot )
	Mozilla/5.0 (compatible; spbot/4.2.0; +http://OpenLinkProfiler.org/bot )
	seo-nastroj.cz
	LoadImpactPageAnalyzer/1.3.0 (Load Impact; http://loadimpact.com/)
	CSS Certificate Spider (http://www.css-security.com/certificatespider/)
	MetaGeneratorCrawler/1.3.2 (www.metagenerator.info)
	Testomatobot/1.0 (Linux x86_64; +http://www.testomato.com/testomatobot) minicrawler/4.0.0~beta8
	BLEXBot
	Mozilla/2.0 (compatible; Ask Jeeves/Teoma)
	Testomatobot/1.0 (Linux x86_64; +http://www.testomato.com/testomatobot) minicrawler/4.0.0~beta7
	Mozilla/5.0 (compatible; XoviBot/2.0; +http://www.xovibot.net/)
	Mozilla/5.0 (compatible; spbot/4.4.1; +http://OpenLinkProfiler.org/bot )
	ia_archiver (+http://www.alexa.com/site/help/webmasters; [email protected])
	HeartRails Robot/0.1 (http://www.heartrails.com)
	Mozilla/5.0 (compatible; Faveeo/1.0; +http://www.faveeo.com)
	yacybot (/global; amd64 Linux 3.16.0-49-generic; java 1.7.0_79; Europe/en) http://yacy.net/bot.html
	Castabot/0.1 (+http://topixtream.com/)
	Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.9.0.5) Gecko/2010033101 Gentoo Firefox/3.0.5 (Dot TK - spider 3.0)
	Mozilla/5.0 (compatible; MSIE or Firefox mutant; not on Windows server; +http://ws.daum.net/aboutWebSearch.html) Daumoa/2.0
	istellabot-nutch/Nutch-1.10
	Mail.RU/2.0
	ichiro/2.0 (http://help.goo.ne.jp/door/crawler.html)
	LexxeBot/1.0 ([email protected])
	Mozilla/5.0 (iPhone; CPU iPhone OS 8_1 like Mac OS X) AppleWebKit/600.1.4 (KHTML, like Gecko) Version/8.0 Mobile/12B410 Safari/600.1.4 (Applebot/0.1; +http://www.apple.com/go/applebot)
	Mozilla/5.0 (compatible; FlipboardRSS/1.1; +http://flipboard.com/browserproxy)
	Mozilla/5.0 (compatible; Linux x86_64; Mail.RU_Bot/Robots/2.0; +http://go.mail.ru/help/robots)
	yacybot (-global; amd64 Linux 3.10.0-229.4.2.el7.x86_64; java 1.7.0_79; Europe/en) http://yacy.net/bot.html
	it2media-domain-crawler/1.0 on crawler-prod.it2media.de
	yacybot (/global; amd64 Windows 8.1 6.3; java 1.8.0_25; Europe/de) http://yacy.net/bot.html
	yacybot (freeworld-global; amd64 Linux 3.16.0-4-amd64; java 1.7.0_79; Europe/de) http://yacy.net/bot.html
	Mozilla/5.0 (WhatsMyIP.org HTTP_Headers) http://whatsmyip.org/ua
	checkgzipcompression.com robot
	Mozilla/5.0 eCairn-Grabber/1.0 (+http://ecairn.com/grabber)
	RankurBot/3.3 (+http://rankur.com)
	L.webis/0.50 (http://webalgo.iit.cnr.it/index.php?pg=lwebis)
	Speedy Spider (Submit your site at http://www.entireweb.com/free_submission/)
	oBot
	Snappy/2.0 ( http://www.urltrends.com/ )
	Mozilla/5.0 (compatible; MSIE 10.0; Windows NT 6.2; WOW64; Trident/6.0) CrawlerProcess (http://www.PowerMapper.com) /5.23.770.0
	Mozilla/5.0 (compatible; alexa site audit/1.0; +http://www.alexa.com/help/webmasters; )
	flatlandbot/baypup (Flatland Industries Web Spider; http://www.flatlandindustries.com/flatlandbot; [email protected])
	istellabot/Nutch-1.11
	GetintentCrawler getintent.com
	Covario-IDS/1.0 (Covario; http://www.covario.com/ids; support at covario dot com)
	Mozilla/5.0 (compatible; FatBot 2.0; http://www.thefind.com/crawler)
	Mozilla/5.0 (compatible; MegaIndex.com/2.0; +http://megaindex.com/crawler)
	Microsearch.ru/1.0; http://microsearch.ru/webmasters
	TurnitinBot/3.0 (http://www.turnitin.com/robot/crawlerinfo.html)
	BacklinkCrawler V (http://www.backlinktest.com/crawler.html)
	FeedlyBot/1.0 (http://feedly.com)
	Clickagy Intelligence Bot v2
	GetURLInfo/1.0
	DoCoMo/2.0 P900i(c100;TB;W24H11)(compatible; ichiro/mobile goo;+http://help.goo.ne.jp/door/crawler.html)
	Scrapy/1.1.0 (+http://scrapy.org)
	yacybot (/global; amd64 Linux 3.16.0-4-amd64; java 1.7.0_75; America/en) http://yacy.net/bot.html
	Mozilla/5.0 (compatible; VSAgent/1.2)
	Experibot_v1 [goo.gl/n6zrAf]
	Mozilla/5.0 (compatible; DNS-Digger-Explorer/1.0; +http://www.dnsdigger.com)
	boitho.com-robot/1.0
	yacybot (/global; amd64 Windows Server 2012 6.2; java 1.7.0_51; Europe/de) http://yacy.net/bot.html
	RelateIQ Crawler www.relateiq.com
	Mozilla/4.0 (compatible; Netcraft Web Server Survey)
	Mozilla/5.0 (X11; U; Linux Core i7-4980HQ; de; rv:32.0; compatible; JobboerseBot; https://www.jobboerse.com/bot.htm) Gecko/20100101 Firefox/38.0
	yacybot (i386 Linux 2.6.9-023stab046.2-smp; java 1.6.0_05; Europe/en) http://yacy.net/bot.html
	Scrubby/2.2 (http://www.scrubtheweb.com/)
	Mozilla/5.0 (Yahoo-MMCrawler/4.0; mailto:[email protected])
	Fetch/2.0a (CMS Detection/Web/SEO analysis tool, see http://guess.scritch.org)
	bitlybot/3.0 (+http://bit.ly/)
	Mozilla/5.0 (compatible; heritrix/1.14.4 +http://www.exif-search.com)
	yacybot (freeworld/global; amd64 Windows Server 2012 6.2; java 1.7.0_25; Europe/de) http://yacy.net/bot.html
	agentslug.com - website monitoring tool
	SafeDNSBot (https://www.safedns.com/searchbot)
	Feedbin
	Mozilla/5.0 (compatible; proximic; +http://www.proximic.com)
	Mozilla/5.0 (compatible; AcoonBot/4.11.1; +http://www.acoon.de/robot.asp)
	Mozilla/5.0 (compatible; MagiBot/3.4.3; +http://magi.peak-labs.com/robots.txt)
	Mozilla/5.0 (compatible; XML Sitemaps Generator; http://www.xml-sitemaps.com) Gecko XML-Sitemaps/1.0
	baypup/colbert (Baypup; http://sf.baypup.com/webmasters; [email protected])
	MergeFlow-PageReader/0.91;+(+http://mergeflow.net/info/pagereader) Mozilla/5.0 (Windows) compatible
	Mozilla/5.0 (Mobile; rv:18.0) Gecko/18.0 Firefox/18.0 commoncrawl.org/research//Nutch-1.7-SNAPSHOT
	Mozilla/5.0 (compatible; spbot/4.1.0; +http://OpenLinkProfiler.org/bot )
	Mozilla/5.0 (compatible; OsO; http://oso.octopodus.com/abot.html)
	Mozilla/5.0 (compatible; gofind; +http://govid.mobi/bot.php)
	CatchBot/2.0; +http://www.catchbot.com
	Mozilla/5.0 (compatible; BusinessSeek.biz_Spider; http://www.businessseek.biz/)
	Quora Link Preview/1.0 (http://www.quora.com)
	radian6_default_(www.radian6.com/crawler)
	DWDS-Crawler +http://odo.dwds.de/dwds-crawler.html
	Mozilla/5.0 (compatible; DuckDuckGo-Favicons-Bot/1.0; +http://duckduckgo.com)
	sogou spider
	Mozilla/5.0 (compatible; GigablastOpenSource/1.0)
	Mozilla/5.0 (compatible; HomeTags/1.0; +http://www.hometags.nl/bot)
	CorpusCrawler 2.0.24 (http://corpora.fi.muni.cz/crawler/);Project:CzCorpus
	Scooter/3.3
	Zookabot/2.5;++http://zookabot.com
	Mozilla/5.0 (iPhone; CPU iPhone OS 8_1 like Mac OS X) AppleWebKit/600.1.4 (KHTML, like Gecko) Version/8.0 Mobile/12B411 Safari/600.1.4 (compatible; YandexMobileBot/3.0; +http://yandex.com/bots)
	Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/534.34 (KHTML, like Gecko) Safari/534.34; +http://sniptracker.com
	Mozilla/5.0 (compatible; archive.org_bot; Wayback Machine Live Record; +http://archive.org/details/archive.org_bot)
	es_com_viewer ([email protected])
	Mozilla/5.0 (compatible; Uptimebot/0.2.14; +http://www.uptime.com/uptimebot)
	yacybot (freeworld/global; amd64 Linux 3.0.0-14-generic; java 1.6.0_23; Europe/en) http://yacy.net/bot.html
	Mozilla/5.0 (X11; Linux x86_64; rv:35.0) Gecko/20100101 Firefox/35.0 DareBoost
	yacybot (freeworld/global; amd64 Linux 2.6.32-41-server; java 1.6.0_26; Europe/de) http://yacy.net/bot.html
	yacybot (amd64 Linux 2.6.28-18-generic; java 1.6.0_16; GMT/en) http://yacy.net/bot.html
	Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.1; Trident/4.0; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; Media Center PC 6.0; .NET4.0C; .NET4.0E; PTST 2.386)
	KD Bot
	Mozilla/5.0 (X11; U; Linux i686 (x86_64); en-US; rv:1.9.2.19) Gecko WebThumb/1.0
	Mozilla/5.0 (compatible; Yahoo! Slurp/3.0; http://help.yahoo.com/help/us/ysearch/slurp)
	rogerbot/1.0 (http://moz.com/help/pro/what-is-rogerbot-, [email protected])
	MJ12bot/v1.2.0 (http://majestic12.co.uk/bot.php?+)
	Superarama.com-Tarama-Botu-v.01
	Mozilla/5.0 (compatible; Urlfilebot/2.2; +http://urlfile.com/bot.html)
	Vegi bot (we follow your robots.txt settings before crawling, you can slow down the bot by change the Crawl-Delay parameter in the settings.if you have an enquiry, please email to: [email protected])
	MnoGoSearch/3.3.9
	Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/534+ (KHTML, like Gecko) BingPreview/1.0b
	JyxobotRSS/0.06
	Mozilla/5.0 (compatible; BigBozz/2.2.1; +http://www.bigbozz.com/)
	Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/534.57.2 (KHTML, like Gecko) Version/5.1.7 Safari/534.57.2 PTST/277
	KiNShooboT (compatible; KiNShooboT/1.0.C; +http://www.kinshoo.com/bot.html)
	SentiBot www.sentibot.eu (compatible with Googlebot)
	Topicbot/1.0 (Mozilla;I;+http://92.42.190.57/)
	IDG/IT (http://spaziodati.eu/)
	LinkAider (http://linkaider.com/crawler/)
	Mozilla/5.0 (compatible; coccoc/1.0; +http://help.coccoc.com/searchengine)
	yacybot (/global; amd64 Linux 2.6.32-573.3.1.el6.x86_64; java 1.7.0_85; Europe/en) http://yacy.net/bot.html
	bitlybot
	Mozilla/5.0 (compatible; Mp3Bot/0.7; +http://mp3realm.org/mp3bot/)
	WWW::LayeredExtractor::Handler::Feed/0.01
	Mozilla/5.0 (compatible; SWEBot/1.0; +http://swebot.net)
	Mozzila/5.0 (compatible; Sonic/1.0; http://www.yama.info.waseda.ac.jp/~crawler/info.html)
	MetaTagRobot/2.1 (http://www.seocentro.com/tools/search-engines/metatag-analyzer.html)
	Mozilla/5.0 (compatible; Goodzer/1.0)
	yacybot (/global; amd64 Linux 3.16.0-4-amd64; java 1.8.0_40; Europe/en) http://yacy.net/bot.html
	Mozilla/5.0 (Windows; U; Windows NT 6.1; en-GB; rv:1.9.2.3) Gecko/20100401 Firefox/3.6.3 (NetShelter ContentScan)
	Facebot/1.0
	SafeAds.xyz bot
	Mozilla/5.0 (compatible; Yeti/1.1; +http://help.naver.com/robots/)
	yacybot (/global; amd64 Windows 7 6.1; java 1.7.0_55; Europe/en) http://yacy.net/bot.html
	scrapyproject (+http://www.profound.net)
	Mozilla/5.0 (compatible; Online Domain Tools - Online Website Link Checker/1.2; +http://website-link-checker.online-domain-tools.com)
	Mozilla/5.0 (compatible; heritrix/2.0.2 +http://aihit.com)
	Mozilla/5.0 (compatible; Scrubby/3.1; +http://www.scrubtheweb.com/help/technology.html)
	Semantifire1/0.20 ( http://www.setooz.com/oozbot.html ; agentname at setooz dot_com )
	Mozilla/5.0 (X11; Ubuntu; Linux i686; rv:14.0; ips-agent) Gecko/20100101 Firefox/14.0.1
	UnisterBot (Mozilla/5.0 compatible; [email protected])
	Mozilla/5.0 (compatible; Ezooms/1.0; [email protected])
	Mozilla/5.0 (compatible; YandexVideo/3.0; +http://yandex.com/bots)
	Mozilla/5.0 (compatible; spbot/4.4.2; +http://OpenLinkProfiler.org/bot )
	findlinks/2.1.3 (+http://wortschatz.uni-leipzig.de/findlinks/)
	Mozilla/5.0 (compatible; MJ12bot/v1.2.5; http://www.majestic12.co.uk/bot.php?+)
	Speedy Spider (Entireweb; Beta/1.2; http://www.entireweb.com/about/search_tech/speedyspider/)
	Mozilla/5.0 (compatible; kazbtbot/0.1; +http://kazbt.com/)
	Orgbybot/OrgbyBot v1.3 (Spider; http://orgby.com/bot/ ; Orgby.com Search Engine)
	Mozilla/5.0 (compatible; aiHitBot-DM/2.0.2 +http://www.aihit.com)
	YahooSeeker-Testing/v3.9 (compatible; Mozilla 4.0; MSIE 5.5; http://search.yahoo.com/)
	Mozilla/5.0 (compatible; Crawlera/1.10.2; UID 40409)
	Mozilla/5.0 (compatible; image.coccoc/1.0; +http://help.coccoc.com/)
	yacybot (/global; amd64 Linux 3.19.2-1-ARCH; java 1.8.0_40; Europe/de) http://yacy.net/bot.html
	Mozilla/5.0 (compatible; evc-batch/2.0.20161017175820)
	datagnionbot (+http://www.datagnion.com/bot.html)
	CopperEgg/RevealUptime/TokyoJP(linode)
	MB-SiteCrawler
	WatchMouse/18990 (http://watchmouse.com/ ; liz)
	HybridBot (hybrid.ru/about. If our bot caused problems please contact us. Contact email: [email protected])
	Mozilla/5.0 (compatible; Crawlera/1.10.2; UID 24522)
	yacybot (/global; amd64 Linux 3.13.0-74-generic; java 1.7.0_91; Europe/en) http://yacy.net/bot.html
	findlinks/1.1.6-beta5 (+http://wortschatz.uni-leipzig.de/findlinks/)
	yacybot (freeworld/global; amd64 Linux 3.13.0-24-generic; java 1.7.0_55; Europe/de) http://yacy.net/bot.html
	Mozilla/5.0 (compatible; OpenfosBot/2.4; +http://www.openfos.com)
	Nusearch Spider (www.nusearch.com)
	WatchMouse/8.4.0.3 (http://watchmouse.com/ ; sesto02.watchmouse.net)
	ICC-Crawler/2.0 (Mozilla-compatible; ; http://www.nict.go.jp/en/univ-com/plan/crawl.html)
	Mozilla/5.0 (compatible; spbot/1.2; +http://www.seoprofiler.com/bot/ )
	Orbiter/1.3 (http://dailyorbit.com/)
	CCBot/2.0
	Mozilla/5.0 (compatible; linkdexbot/2.1; +http://www.linkdex.com/about/bots/)
	Mozilla/5.0 (compatible; DNS-Digger/1.0; +http://www.dnsdigger.com)
	AppEngine-Google; (+http://code.google.com/appengine; appid: s~feedly-social)
	yacybot (/global; x86 Windows 7 6.1; java 1.8.0_71; Europe/de) http://yacy.net/bot.html
	Mozilla/5.0 (Windows Phone 8.1; ARM; Trident/7.0; Touch; rv:11.0; IEMobile/11.0; NOKIA; Lumia 530) like Gecko (compatible; adidxbot/2.0; +http://www.bing.com/bingbot.htm)
	Dex Social Bot
	yacybot (/global; amd64 Linux 4.7.6-200.fc24.x86_64; java 1.8.0_102; Etc/en) http://yacy.net/bot.html
	Mozilla/5.0 (compatible; heritrix/1.14.3 +http://archive.org)
	CRAZYWEBCRAWLER 0.9.10, http://www.crazywebcrawler.com
	Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/50.0.2661.102 Safari/537.36 PTST/277
	Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.1.3; ips-agent) Gecko/20090824 Fedora/1.0.7-1.1.fc4 Firefox/3.5.3
	SafeSearch microdata crawler (https://safesearch.avira.com, [email protected])
	Zemanta Aggregator/0.9 +http://www.zemanta.com
	Mozilla/5.0 (compatible; special_archiver/3.2.0 +http://www.loc.gov/webarchiving/notice_to_webmasters.html)
	Mozilla/5.0 (compatible; GimmeUSAbot/1.0; +https://gimmeusa.com/crawler.html)
	yacybot (freeworld/global; amd64 Windows 7 6.1; java 1.7.0_45; Europe/de) http://yacy.net/bot.html
	Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.1 (KHTML, like Gecko) Chrome/21.0.1180.89 Safari/537.1; 360Spider
	FyberSpider/1.3 (http://www.fybersearch.com/fyberspider.php)
	yacybot (/global; amd64 FreeBSD 10.3-RELEASE; java 1.8.0_77; GMT/en) http://yacy.net/bot.html
	Searchie/1.0 (a Storm-based crawler; https://www.searchie.org; [email protected])
	yacybot (freeworld/global; amd64 Linux 3.1.10-1-desktop; java 1.6.0_22; Europe/de) http://yacy.net/bot.html
	holmes/3.12.4 (http://morfeo.centrum.cz/bot)
	WatchMouse/18990 (http://watchmouse.com/ ; se.watchmouse.com)
	MXT/Nutch-1.10 (http://t.co/GSRLLKex24; informatique at mixdata dot com)
	Mozilla/5.0 (compatible; Shareaholicbot/1.0; +http://www.shareaholic.com/bot)
	ZoomInformation Bot
	Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 5.1; www.alertra.com)
	Mozilla/5.0 (compatible; HomeTags/1.0; http://www.hometags.nl/bot)
	Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.2.17) Gecko/20110515 HeartRails_Capture/1.0.4 (+http://capture.heartrails.com/) Namoroka/3.6.17
	OmniExplorer_Bot/5.20 (+http://www.omni-explorer.com) WorldIndexer
	WatchMouse/18990 (http://watchmouse.com/ ; it)
	Mozilla/5.0 (compatible; imrbot/1.10.8 +http://www.mignify.com)
	Mozilla/5.0 (compatible; spbot/4.0.2; +http://www.seoprofiler.com/bot )
	Readability/1900e6 - http://readability.com/about/
	Mozilla/5.0 (compatible; Dataprovider/6.92; +https://www.dataprovider.com/)
	Mozilla/5.0 (iPhone; CPU iPhone OS 6_0_1 like Mac OS X) AppleWebKit/537.36 (KHTML, like Gecko; Google Page Speed Insights) Version/6.0 Mobile/10A525 Safari/8536.25
	VegeBot (we follow your robots.txt settings before crawling, you can slow down the bot by change the Crawl-Delay parameter in the settings.if you have an enquiry, please email to: [email protected])
	yacybot (webportal-global; amd64 Linux 3.2.0-4-amd64; java 1.7.0_65; Europe/en) http://yacy.net/bot.html
	FeedBucket/1.0 (+http://www.feedbucket.com)
	topster.de HTTP-Header 1.0
	MaxPointCrawler/Nutch-1.10 (maxpoint.crawler at maxpointinteractive dot com)
	Mozilla/5.0 (Windows NT 6.0; rv:45.0) Gecko/20100101 Firefox/45.0 PTST/276
	yacybot (/global; amd64 Windows 7 6.1; java 1.8.0_05; Europe/es) http://yacy.net/bot.html
	NETCRAFT
	Mozilla/5.0 (compatible; WbSrch/1.0; +https://wbsrch.com)
	WebCookies/1.0 (+http://webcookies.info/faq/#agent)
	netEstate Impressumscrawler (+http://www.netestate.de/De/Loesungen/Impressumscrawler)
	msnbot-media/2.0b (+http://search.msn.com/msnbot.htm)
	CopperEgg/RevealUptime/DublinIEUSA
	StormCrawler/1.0 (a Storm-based crawler; https://github.com/DigitalPebble/storm-crawler; [email protected])
	WatchMouse/8.4.0.3 (http://watchmouse.com/ ; uschi02.watchmouse.net)
	Mozilla/5.0 (Linux; Android 5.0.2; SM-G920T Build/LRX22G) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/40.0.2125.111 Mobile Safari/537.36 DareBoost
	Mozilla/5.0 (compatible; DeuSu/0.1.0; +https://deusu.org)
	Mozilla/5.0 (compatible; MJ12bot/v1.4.4 (domain ownership verifier); http://www.majestic12.co.uk/bot.php?+)
	Mozilla/5.0 (compatible; SemrushBot/0.99~bl; +http://www.semrush.com/bot.html)
	WebWatch/Robot_txtChecker
	Mozilla/5.0 (compatible; linkdexbot/2.2; +http://www.linkdex.com/bots/)
	Feedly/1.0 (+http://www.feedly.com/fetcher.html; like FeedFetcher-Google)
	Toweyabot: toweya.com
	Mozilla/5.0 (compatible; Infohelfer/1.4.3; +http://www.infohelfer.de/crawler.php)
	Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.2.1; aggregator:Spinn3r (Spinn3r 3.1); http://spinn3r.com/robot) Gecko/20021130
	FeedCatBot/3.0 (+http://www.feedcat.net/)
	LinkedInBot/1.0 (compatible; Mozilla/5.0; Apache-HttpClient +http://www.linkedin.com), libot/Nutch-1.9 (http://www.linkedin.com; [email protected])
	R6_CommentReader(www.radian6.com/crawler)
	Mozilla/5.0 (compatible; Crawlera/1.10.2; UID 70350)
	Mozilla/5.0 (compatible; MJ12bot/v1.2.3; http://www.majestic12.co.uk/bot.php?+)
	Domain Re-Animator Bot (http://domainreanimator.com) - [email protected]
	Riddler (http://riddler.io/about.html)
	Mozilla/5.0 (compatible; Esribot/1.0; http://www.esrihu.hu/)
	Favicon downloader (http://favicon.netk6.com/)
	findlinks/1.1.5-beta7 (+http://wortschatz.uni-leipzig.de/findlinks/)
	ConveraCrawler/0.9d (+http://www.authoritativeweb.com/crawl)
	Mozilla/5.0 (Windows NT 6.1; rv:6.0) Gecko/20110814 Firefox/6.0 Google favicon
	GetFoundBot (+http://www.getfound.cz/getfoundbot/)
	WatchMouse/8.4.0.3 (http://watchmouse.com/ ; usdal02.watchmouse.net)
	Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/48.0.2564.97 Safari/537.36 Yandex.Translate
	Mozilla/5.0 (compatible; NLNZ_IAHarvester2013 +http://natlib.govt.nz/about-us/current-initiatives/web-harvest-2013)
	COMODOSpider/Nutch-1.2
	Mozilla/5.0 (compatible; DomainMacroCrawler/0.1; +http://domainmacro.com)
	Mozilla/5.0 (compatible; heritrix/3.2.0 +http://www.exif-search.com)
	Mozilla/5.0 (compatible; MagiBot/3.6.2; +http://magi.peak-labs.com/robots.txt)
	Mozilla/5.0 (compatible; Alexabot/1.0; +http://www.alexa.com/help/certifyscan; [email protected])
	psbot-image (+http://www.picsearch.com/bot.html)
	Mozilla/5.0 (Macintosh; Intel Mac OS X 10.9; rv:28.0) Gecko/20100101 Firefox/28.0 (FlipboardProxy/1.6; +http://flipboard.com/browserproxy)
	larbin_2.6.4 ([email protected])
	yacybot (/global; amd64 Windows 7 6.1; java 1.7.0_55; Europe/ru) http://yacy.net/bot.html
	BacklinkCrawler (http://www.backlinktest.com/crawler.html)
	Mozilla/5.0 (Windows NT 6.1; WOW64) adbeat.com/policy AppleWebKit/537.36 (KHTML, like Gecko) Chrome/48.0.2564.116 Safari/537.36
	Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/534.57.2 (KHTML, like Gecko) Version/5.1.7 Safari/534.57.2 PTST/276
	Tools4noobs.com/1.0 Spider
	yacybot (freeworld/global; amd64 Linux 2.6.32-40-generic; java 1.6.0_20; Europe/de) http://yacy.net/bot.html
	Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US) Speedy Spider (http://www.entireweb.com/about/search_tech/speedy_spider/)
	istellabot/t.1
	Mozilla/5.0 (compatible; Lipperhey-Kaus-Australis/5.0; +https://www.lipperhey.com/en/about/)
	Mozilla/5.0 (compatible; Veoozbot/1.0; +http://www.veooz.com/veoozbot.html)
	SemrushBot/Nutch-1.5-SNAPSHOT
	Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)
	Mozilla/5.0 (X11; Linux x86_64; rv:47.0; GTmetrix https://gtmetrix.com/) Gecko/20100101 Firefox/47.0
	CopperEgg/RevealUptime/Oregon(aws)
	yacybot (/global; amd64 Linux 3.16.0-4-amd64; java 1.8.0_102; Europe/en) http://yacy.net/bot.html
	rogerbot/1.0 (http://www.seomoz.org/dp/rogerbot, [email protected])
	yacybot (amd64 Linux 2.6.26-2-amd64; java 1.6.0_20; Europe/en) http://yacy.net/bot.html
	Mozilla/5.0 (Windows NT 6.3; WOW64; Trident/7.0; rv:11.0; BingPreview/1.0b) like Gecko
	Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 5.1; Trident/4.0; .NET CLR 1.1.4322; .NET CLR 2.0.50727; .NET CLR 3.0.04506.30; .NET CLR 3.0.4506.2152; .NET CLR 3.5.30729; MDDS; PTST 2.386)
	Mozilla/5.0 (compatible; RavenCrawler/2.0; +https://raventools.com/seo-website-auditor/)
	Mozilla/5.0 (compatible; YandexMetrika/2.0; +http://yandex.com/bots)
	Mozilla/5.0 (compatible; Scrubby/3.2; +http://seotools.scrubtheweb.com/webpage-analyzer.html)
	dubaiindex (adressendeutschland.de)
	Mozilla/5.0 (compatible; spbot/5.0.2; +http://OpenLinkProfiler.org/bot )
	Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; Netcraft SSL Server Survey - contact [email protected])
	Mozilla/5.0 (compatible; CompSpyBot/1.0; +http://www.compspy.com/spider.html)
	Mozilla/5.0 (compatible; LA2; +http://www.zeerch.com/zeerch2/bot.php)
	yacybot (/global; amd64 Linux 2.6.32-042stab108.8; java 1.7.0_91; America/en) http://yacy.net/bot.html
	updated/0.1-alpha (updated crawler; http://www.updated.com; [email protected])
	CopperEgg/RevealUptime/FrankfurtGermany
	Scrapy/1.1.2 (+http://scrapy.org)
	yrspider (Mozilla/5.0 (compatible; YRSpider; +http://www.yunrang.com/yrspider.html))
	Mozilla/5.0 (compatible; YandexAntivirus/2.0; +http://yandex.com/bots)
	yacybot (freeworld/global; amd64 Windows 7 6.1; java 1.6.0_31; Europe/de) http://yacy.net/bot.html
	Surphace Scout&v4.0 - scout at surphace dot com
	yacybot (freeworld/global; amd64 Linux 2.6.32-5-xen-amd64; java 1.6.0_18; Europe/fr) http://yacy.net/bot.html
	MetaGeneratorCrawler/1.1 (www.metagenerator.info)
	Porkbun/Mustache (Website Analysis; http://porkbun.com; [email protected])
	Mozilla/5.0 (compatible; ScoutJet; +http://www.scoutjet.com/)
	Ruby, link_thumbnailer
	yacybot (freeworld/global; amd64 Windows Server 2008 R2 6.1; java 1.7.0_25; Europe/de) http://yacy.net/bot.html
	yacybot (/global; amd64 Linux 3.13.0-042stab093.4; java 1.7.0_79; Europe/de) http://yacy.net/bot.html
	Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.11 (KHTML, like Gecko) Chrome/23.0.1271.64 Safari/537.11 GotSiteMonitor.com
	Mozilla/5.0 (Linux; Android 4.4; Nexus 5 Build/KRT16M) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/40.0.2125.111 Mobile Safari/537.36 DareBoost
	Mozilla/5.0 (KeepRight OpenStreetMap Checker; http://keepright.at) Gecko/20100101 Firefox/22.0
	Mozilla/5.0 (Windows NT 6.1; WOW64; Trident/7.0; rv:11.0; topster.de Linkchecker 5.0) like Gecko (194.228.205.74)
	Mozilla/5.0 AppleWebkit/537.36 (KHTML, like Gecko) Trident/7.0; rv:11.0, Chrome/42.0.2311.135, Edge/12.10136, http://www.shrinktheweb.com/, Webshot/0.9
	topicbot/1.0 (Mozilla;I;+http://www.topic.bot/contact_page.html)
	DataparkSearch/4.53 (+http://dataparksearch.org/bot)
	Mozilla/5.0 (compatible; Linux; InfegyAtlas/1.0; en-US; [email protected])
	Mozilla/5.0 (X11; Linux x86_64; rv:41.0; GTmetrix https://gtmetrix.com/) Gecko/20100101 Firefox/41.0
	Mozilla/5.0 (compatible; EasouSpider; +http://www.easou.com/search/spider.html)
	Mozilla/5.0 (compatible; parsijoo-update-crawler; +http://www.parsijoo.ir/; [email protected])
	Abrave Spider v4 Robot 1 (http://robot.abrave.co.uk)
	Mozilla/5.0 (compatible; memoryBot/1.21.24 +http://internetmemory.org/en/)
	Mozilla/5.0 (compatible; STINGbot/1.0; +http://136.186.231.16)
	domainsbot (+http://www.domainsbot.com)
	WEPA/3.1 (http://www.wepa.com/; [email protected])
	Mozilla/5.0 (compatible; linkdexbot/2.1; +http://www.linkdex.com/bots/)
	Mozilla/5.0 (compatible; forensiq; +http://www.forensiq.com)
	192.comAgent
	JUST-CRAWLER(+http://www.justsystems.com/jp/tech/crawler/)
	Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm) SitemapProbe
	HuaweiSymantecSpider/[email protected]+(compatible; MSIE 7.0; Windows NT 5.1; Trident/4.0; .NET CLR 2.0.50727; .NET CLR 3.0.4506.2152; .NET CLR ; http://www.huaweisymantec.com/en/IRL/spider)
	Mozilla/5.0 ( Macintosh; Intel Mac OS X 10_10_1 ) AppleWebKit/600.2.5 ( KHTML, like Gecko ) Version/8.0.2 Safari/600.2.5 ( compatible; CloudServerMarketSpider/1.0; +http://cloudservermarket.com/spider.html )
	Mozilla/5.0 (compatible; MJ12bot/v1.3.0; http://www.majestic12.co.uk/bot.php?+)
	Mozilla/5.0 (compatible; AcoonBot/4.10.6; +http://www.acoon.de/robot.asp)
	Scrapy/1.0.1 (+http://scrapy.org)
	seegnifybot/1.0.0 (http://www.seegnify.com/bot)
	omgili/0.5 +http://omgili.com
	Slack-ImgProxy 149 (+https://api.slack.com/robots)
	yacybot (/global; amd64 Linux 3.16.0-4-amd64; java 1.7.0_79; Europe/de) http://yacy.net/bot.html
	www.integromedb.org/Crawler
	Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm
	Mozilla/5.0 (compatible) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/47.0.2526.73 Safari/537.36 [email protected]
	Mozilla/5.0 (compatible; IstellaBot/1.18.81 +http://www.tiscali.it/)
	Gigabot/1.0
	page_test ([email protected])
	istellabot/Nutch-1.10
	Mozilla/5.0 (iPhone; U; CPU iPhone OS 4_1 like Mac OS X; en-us) AppleWebKit/532.9 (KHTML, like Gecko) Version/4.0.5 Mobile/8B117 Safari/6531.22.7 (compatible; Mediapartners-Google/2.1; +http://www.google.com/bot.html)
	ldspider (http://code.google.com/p/ldspider/wiki/Robots)
	yacybot (/global; amd64 Windows 8.1 6.3; java 1.8.0_45; Europe/ru) http://yacy.net/bot.html
	facebookplatform/1.0 (+http://developers.facebook.com)
	Mozilla/5.0 (compatible; RTGI; http://linkfluence.net/)
	yacybot (freeworld/global; amd64 Linux 3.12.43-52.6-default; java 1.8.0_40; Europe/en) http://yacy.net/bot.html
	Mozilla/5.0 (compatible; YandexBot/3.0; MirrorDetector; +http://yandex.com/bots)
	Mozilla/5.0 (compatible; DCPbot/1.2; +http://domains.checkparams.com/)
	WatchMouse/18990 (http://watchmouse.com/ ; d2.watchmouse.com)
	MnoGoSearch/3.3.15
	Mozilla/5.0 (compatible; AboutUsBot/0.9; +http://www.aboutus.org/AboutUsBot)
	Web-sniffer/1.1.0 (+http://web-sniffer.net/)
	Mozilla/4.0 (compatible; Fooooo_Web_Video_Crawl http://fooooo.com/bot.html)
	50.nu/0.01 ( +http://50.nu/bot.html )
	Mozilla/4.0 (compatible; MSIE 7.0; Windows; Windows NT 5.1) BrokenLinkCheck.com/1.1
	Comodo-Certificates-Spider
	Iframely/0.9.8 (+https://iframely.com/;)
	MnoGoSearch/3.3.12
	CopperEgg/RevealUptime/FrankfurtEU
	Python-urllib/2.7 (+http://ella.juls.savba.sk/aranea_about)
	Mozilla/5.0 (compatible; JobKereso; +http://www.kozvetlen-allasok.hu/robot.jsp [email protected])
	Mozilla/5.0 (Windows NT 5.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/49.0.2623.112 Safari/537.36 PTST/201
	Mozilla/5.0 (compatible; Google Keyword Tool; +http://adwords.google.com/select/KeywordToolExternal)
	WinWebBot/1.0; (Balaena Ltd, UK); http://www.balaena.com/winwebbot.html; [email protected];)
	AppleNewsBot
	classbot (+http://allclasses.com)
	yacybot (freeworld/global; amd64 Windows Server 2008 R2 6.1; java 1.6.0_31; America/pt) http://yacy.net/bot.html
	Mozilla/5.0 (iPhone; CPU iPhone OS 7_0 like Mac OS X) AppleWebKit/537.51.1 (KHTML, like Gecko) Version/7.0 Mobile/11A465 Safari/9537.53 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)
	yacybot (/global; arm Linux 3.12.28+; java 1.7.0_71; Europe/en) http://yacy.net/bot.html
	Mozilla/5.0 (compatible; YandexMetrika/2.0; +http://yandex.com/bots mtmon01e.yandex.ru)
	CopperEgg/RevealUptime/TokyoJP
	Mozilla/5.0 (compatible; AhrefsBot/3.1; +http://ahrefs.com/robot/)
	CJNetworkQuality; http://www.cj.com/networkquality
	psbot-page (+http://www.picsearch.com/bot.html)
	DealGates Bot/1.1 by Luc Michalski (http://spider.dealgates.com/bot.html)
	Pingdom GIGRIB (http://www.pingdom.com)
	Mozilla/5.0 (compatible; SEOlyticsCrawler/3.0; +http://crawler.seolytics.net/)
	Mozilla/5.0 (compatible; EventGuruBot/1.0; +http://www.eventguru.com/spider.html)
	Python-urllib/2.7 (+http://aranea.juls.savba.sk/aranea_about.html)
	yacybot (/global; amd64 Windows Vista 6.0; java 1.7.0_55; America/en) http://yacy.net/bot.html
	The Lemur Web Crawler/Nutch-1.3 (Lemur Web Crawler; http://boston.lti.cs.cmu.edu/crawler_12/; [email protected])
	SAMSUNG-SGH-E250/1.0 Profile/MIDP-2.0 Configuration/CLDC-1.1 UP.Browser/6.2.3.3.c.1.101 (GUI) MMP/2.0 (compatible; Googlebot-Mobile/2.1; +http://www.google.com/bot.html)
	Feedfetcher-Nuesbyte; (+http://www.nuesbyte.com)
	copyright sheriff (+http://www.copyrightsheriff.com/)
	yacybot (/global; amd64 Linux 3.2.0-4-amd64; java 1.7.0_75; Europe/en) http://yacy.net/bot.html
	Touche (+http://www.touche.com.ve)
	Panopta v1.1
	Mozilla/5.0 (compatible; parsijoo-bot; +http://www.parsijoo.ir/; [email protected])
	Zookabot/2.2;++http://zookabot.com
	yacybot (/global; amd64 Linux 2.6.32-042stab093.4; java 1.7.0_65; Etc/en) http://yacy.net/bot.html
	it2media-domain-crawler/2.0
	yacybot (/global; amd64 Linux 4.2.0-27-generic; java 1.7.0_95; Europe/en) http://yacy.net/bot.html
	Mozilla/5.0 (Android; Mobile; rv:21.0) Gecko/21.0 Firefox/21.0 commoncrawl.org/research//Nutch-1.7-SNAPSHOT
	Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 5.2; Trident/4.0; Data Center; .NET CLR 1.1.4322; .NET CLR 2.0.50727; .NET CLR 3.0.04506.648; .NET CLR 3.5.21022; .NET CLR 3.0.4506.2152; .NET CLR 3.5.30729; PTST 2.386)
	Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/99.0 Safari/537.36 YottaaMonitor
	CopperEgg/RevealUptime/AtlantaGAUSA
	Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
	yacybot (/global; amd64 Windows 7 6.1; java 1.7.0_55; Europe/fr) http://yacy.net/bot.html
	Mozilla/5.0 (compatible; MJ12bot/v1.2.4; http://www.majestic12.co.uk/bot.php?+)
	Mozilla/5.0 (compatible; Applebot/0.3; +http://www.apple.com/go/applebot)
	ICC-Crawler/2.0 (Mozilla-compatible; ; http://www.nict.go.jp/en/ucri/plan/crawl.html)
	Mozilla/5.0 (compatible; YYSpider; +http://www.yunyun.com/spider.html)
	Mozilla/5.0 (iPhone; CPU iPhone OS 5_0_1 like Mac OS X) (compatible; Yeti-Mobile/0.1; +http://help.naver.com/robots/)
	Mozilla/5.0 (WhatsMyIP.org Text_to_Code_Ratio_Tool) http://whatsmyip.org/ua
	Thumbnail.CZ robot 1.1 (http://thumbnail.cz/why-no-robots-txt.html)
	Google-Calendar-Importer
	Mozilla/5.0 (Windows NT 6.3; WOW64) AppleWebKit/534.34 (KHTML, like Gecko) Websnapr/3.0 Safari/534.34
	Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/50.0.2661.75 Safari/537.36 PTST/276
	WeSEE
	librabot/2.0 (+http://academic.research.microsoft.com/)
	Testomatobot/1.0 (Linux x86_64; +http://www.testomato.com/testomatobot) minicrawler/3.0.1
	Mozilla/5.0 (compatible; MSIE 10.0; Windows NT 6.1; Trident/6.0) PTST/276
	Cliqzbot/0.1 (+http://cliqz.com [email protected])
	Scrubby/3.0 (+http://www.scrubtheweb.com/help/technology.html)
	Pinterest/0.1 +http://pinterest.com/
	Mozilla/5.0 (compatible; Crawlera/1.10.2; UID 52327)
	Comodo-Webinspector-Crawler 2.2.0, http://www.comodorobot.com
	Iframely/0.6.0 (+http://iframely.com/;)
	Mozilla/5.0 (compatible; PaperLiBot/2.1; http://support.paper.li/entries/20023257-what-is-paper-li)
	Mozilla/5.0 (compatible; SiteExplorer/1.0b; +http://siteexplorer.info/)
	Y!J-BSC/1.0 (http://help.yahoo.co.jp/help/jp/blog-search/)
	Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/49.0.2623.112 Safari/537.36 PTST/276
	Mozilla/5.0 (compatible; U; Hootsuite-WebFeed/1.0; mailto:[email protected])
	RankFlex.com Webspider
	TipTop http://feeltiptop.com
	vebidoobot
	Mozilla/5.0 (compatible; Dow Jones Searchbot)
	Mozilla/5.0 (compatible; OrangeBot-Collector/2.0; [email protected])
	Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/534+ (KHTML, like Gecko) MsnBot-Media /1.0b
	Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1.6) Gecko/20070725 Firefox/2.0.0.6 Ara.com.tr AraBot 1.0
	Mozilla/5.0 (iPhone; CPU iPhone OS 9_1 like Mac OS X) AppleWebKit/601.1.46 (KHTML, like Gecko) Version/9.0 Mobile/13B143 Safari/601.1 (compatible; AdsBot-Google-Mobile; +http://www.google.com/mobile/adsbot.html)
	Mozilla/5.0 (compatible; YandexMetrika/2.0; +http://yandex.com/bots mtmon01g.yandex.ru)
	BUbiNG (+http://law.di.unimi.it/BUbiNG.html)
	downnotifier.com monitoring
	A6-Indexer
	Mozilla/5.0 (compatible; htrix/1.14.4 ++https://webarchiv.onb.ac.at/robot.html)
	MetaURI API/2.0 +metauri.com
	Mozilla/5.0 (compatible; MixrankBot; [email protected])
	Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/44.0.2403.155 Safari/537.36 TinEye/1.0 (via http://www.tineye.com/)
	Mozilla/5.0 Project 25499 (project25499.com)
	COMODOspider/Nutch-1.0
	Mozilla/5.0 (compatible; CareerBot/1.1; +http://www.career-x.de/bot.html)
	Uptime.com_(http://uptime.com/)
	AcoonBot/4.10.5 (+http://www.acoon.de)
	BDFetch
	20090717_aa_page ([email protected])
	Metaspinner/1.0 (Metaspinner Search Engine; http://www.meta-spinner.de/; [email protected])
	Mozilla/5.0 (compatible; DBLBot/1.0; +http://www.dontbuylists.com/)
	Mozilla/5.0 (compatible; spbot/2.0; +http://www.seoprofiler.com/bot/ )
	LYT.SR v1.5 +http://lyt.sr/ (running on http://lyt.sr)
	Putin (+http://semanticjuice.com/)
	Mozilla/5.0 (compatible; LA1; +http://www.zeerch.com/bot.php)
	WatchMouse/18990 (http://watchmouse.com/ ; ny)
	bitlybot/2.0
	yacybot (/global; amd64 Linux 3.13.0-45-generic; java 1.7.0_75; Europe/en) http://yacy.net/bot.html
	Webshot/0.9 ( http://www.shrinktheweb.com/ )
	MetaURI API +metauri.com
	Mozilla/5.0 (compatible; ProductoDownloadUrlBot/1.0; +http://www.producto.de/)
	Google-Adwords-Instant (+http://www.google.com/adsbot.html)
	yacybot (freeworld/global; amd64 Linux 3.8.0-19-generic; java 1.7.0_25; Europe/en) http://yacy.net/bot.html
	yacybot (freeworld/global; i386 Linux 2.6.32-5-686; java 1.6.0_18; Europe/en) http://yacy.net/bot.html
	Mozilla/5.0 (Windows NT 6.3; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/42.0.2311.90 Safari/537.36 PTST/277
	Mozilla/5.0 (compatible; DomainSigmaCrawler/0.1; +http://domainsigma.com/robot)
	HyperZbozi.cz Feeder/3.1
	Mozilla/5.0 (Windows; U; Windows NT 5.1; en; rv:1.9.0.13) Gecko/2009073022 Firefox/3.5.2 (.NET CLR 3.5.30729) SurveyBot/2.3 (DomainTools)
	yacybot (/global; amd64 Windows Server 2012 R2 6.3; java 1.7.0_55; Europe/de) http://yacy.net/bot.html
	Google-AdWords-Express
	Mozilla/5.0 (compatible; MJ12bot/v1.4.0; http://www.majestic12.co.uk/bot.php?+)
	SMTBot (similartech.com/smtbot)
	Mozilla/5.0 (compatible; Lipperhey SEO Service; http://www.lipperhey.com/)
	MetaInspector/4.7.2 (+https://github.com/jaimeiniesta/metainspector)
	Mozilla/5.0 (Compatible; Vedma/0.91Beta; +http://www.vedma.ru/bot.htm)
	Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.21 (KHTML, like Gecko) web-capture.net/0.9 Safari/537.21
	Nokia6680/1.0 (4.04.07) SymbianOS/8.0 Series60/2.6 Profile/MIDP-2.0 Configuration/CLDC-1.1 (botmobi find.mobi/bot.html [email protected])
	Mozilla/5.0 (compatible; Exploratodo/1.0; +http://www.exploratodo.com
	iZSearch.com
	Mozilla/5.0 (compatible; YandexFavicons/1.0; +http://yandex.com/bots)
	bl.uk_lddc_bot/3.3.0-SNAPSHOT-2014-10-07T09:33:31Z (+http://www.bl.uk/aboutus/legaldeposit/websites/websites/faqswebmaster/index.html)
	yacybot (/global; amd64 Linux 3.14-0.bpo.1-amd64; java 1.7.0_55; Europe/de) http://yacy.net/bot.html
	HuaweiSymantecSpider/[email protected]+(compatible; MSIE 7.0; Windows NT 5.1; Trident/4.0; .NET CLR 2.0.50727; .NET CLR 3.0.4506.2152; .NET CLR ; http://www.huaweisymantec.com/cn/IRL/spider)
	FAST-WebCrawler/3.7/FirstPage (atw-crawler at fast dot no;http://fast.no/support/crawler.asp)
	Mozilla/5.0 (compatible; DripfeedBot/2.0; +http://dripfeedbookmark.com/bot.html
	Mozilla/5.0 (compatible; heritrix/2.0.2 +http://seekda.com)
	yacybot (freeworld/global; i386 Linux 3.2.0-23-generic-pae; java 1.7.0_03; Europe/en) http://yacy.net/bot.html
	hawkReader/0.9b (Feed Parser; http://www.hawkreader.com; Allow like Gecko)
	ruijie ([email protected])
	CorpusCrawler 2.0.0 (http://corpora.fi.muni.cz/crawler/)
	ExactSeek Crawler (http://www.exactseek.com/)/Nutch-1.4
	WordPress.com; https://botd.wordpress.com
	Mozilla/5.0 (compatible; Qwantify/2.0n; +https://www.qwant.com/)/*
	Mozilla/5.0 (compatible; memoryBot/1.20.268 +http://internetmemory.org/en/)
	ShowyouBot (http://showyou.com/crawler)
	Yoleo Consumer v0.2
	Mozilla/5.0 (compatible; IstellaBot/1.10.2 +http://www.tiscali.it/)
	Mozilla/5.0 (compatible; MSIE or Firefox mutant; not on Windows server; +http://ws.daum.net/aboutWebSearch.html) Daumoa/3.0
	Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/46.0.2490.86 YaBrowser/15.12.0.6151 Safari/537.36 Yandex.Translate
	Mozilla/5.0 (compatible; TeeRaidBot; +https://teeraid.com/bot/)
	Mozilla/5.0 (compatible; Scopia Crawler 1.1; +http://www.scopia.co)
	CopperEgg/RevealUptime/DublinIE(aws)
	yacybot (freeworld/global; amd64 Windows 7 6.1; java 1.7.0_04; Asia/ja) http://yacy.net/bot.html
	RankurBot/Rankur2.1 (http://rankur.com; info at rankur dot com)
	Gaisbot/3.0+([email protected];+http://gais.cs.ccu.edu.tw/robot.php)
	Mozilla/5.0 (compatible; zitebot support [at] zite [dot] com +http://zite.com)
	MetaTagRobot/2.0 (http://www.check-domains.com/website-analysis/website-analyzer.php)
	Virusdie crawler/2.1
	findlinks/2.0.4 (+http://wortschatz.uni-leipzig.de/findlinks/)
	Mozilla/5.0 (compatible; AcoonBot/4.10.7; +http://www.acoon.de/robot.asp)
	Mozilla/5.0 (compatible; DeuSu/5.0.2; +https://deusu.de/robot.html)
	yacybot (-global; amd64 Linux 3.19.0-15-generic; java 1.8.0_45-internal; Europe/de) http://yacy.net/bot.html
	Google_Analytics_Snippet_Validator
	SurcentroBot
	Mozilla/5.0 (compatible; Diffbot/0.1; +http://www.diffbot.com)
	findlinks/1.1.3-beta9 (+http://wortschatz.uni-leipzig.de/findlinks/)
	Mozilla/5.0 (compatible; memoryBot/1.20.199 +http://internetmemory.org/en/)
	Y!J-ASR/0.1 crawler (http://www.yahoo-help.jp/app/answers/detail/p/595/a_id/42716/)
	TwengaBot-Discover (http://www.twenga.fr/bot-discover.html)
	Mozilla/5.0 (bl.uk_lddc_bot; Linux x86_64) PhantomJS/1.9.7 (+http://www.bl.uk/aboutus/legaldeposit/websites/websites/faqswebmaster/index.html)
	panscient.com
	Mozilla/5.0 (compatible; TestCrawler; +http://wiki.github.com/bixo/bixo/bixocrawler; [email protected])
	DataparkSearch/4.37-23012006 ( http://www.dataparksearch.org/)
	GOFORITBOT ( http://www.goforit.com/about/ )
	CopperEgg/RevealUptime/SydneyAustralia
	msnbot-media/1.0 (+http://search.msn.com/msnbot.htm)
	Mozilla/5.0 (compatible; MegaIndex.ru/2.0; +https://www.megaindex.ru/?tab=linkAnalyze)
	MetaCommentBot; http://metacomment.io/about
	Mozilla/5.0 (compatible; YesupBot/1.0; +http://www.yesup.net/bot.html)
	Twikle/1.0 , http://twikle.com , [email protected]
	yacybot (/global; amd64 Windows Server 2008 R2 6.1; java 1.8.0_60; Europe/de) http://yacy.net/bot.html
	Mozilla/4.0 (compatible; Vagabondo/4.0Beta; webcrawler at wise-guys dot nl; http://webagent.wise-guys.nl/; http://www.wise-guys.nl/)
	Comodo-Webinspector-Crawler 2.1
	Mozilla/5.0 (Macintosh; Intel Mac OS X 10_7_5) AppleWebKit/537.11 (KHTML, like Gecko) Chrome/23.0.1271.64 Safari/537.11 KimonoLabs/0.2
	Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1 + FairShare-http://fairshare.cc)
	Scrapy/0.24.5 (+http://scrapy.org)
	Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/534.34 (KHTML, like Gecko) Site-Shot/2.1 (http://www.site-shot.com/) Safari/534.34
	Port Monitor check service 1.0 | http://www.port-monitor.com
	Mozilla/5.0 (compatible; Uptimebot/0.1.43; +http://www.uptime.com/uptimebot)
	yacybot (freeworld/global; amd64 Linux 2.6.32-5-amd64; java 1.6.0_18; Europe/en) http://yacy.net/bot.html
	SolomonoBot/1.04 (http://www.solomono.ru)
	Easy-Thumb (https://www.easy-thumb.net/)
	Mozilla/5.0 (Windows NT 6.0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/49.0.2623.112 Safari/537.36 PTST/276
	nic.it crawler v.1.221115 (7.35.0)
	Mozilla/5.0 (compatible; memorybot/1.24.12 +https://internetmemory.net/en/bots)
	Mozilla/5.0 (compatible; inoreader.com-like FeedFetcher-Google)
	yacybot (/global; amd64 Windows 7 6.1; java 1.7.0_72; Europe/de) http://yacy.net/bot.html
	WatchMouse/8.4.0.3 (http://watchmouse.com/ ; demuc02.watchmouse.net)
	Mozilla/5.0 (X11; Linux x86_64; rv:43.0; GTmetrix https://gtmetrix.com/) Gecko/20100101 Firefox/43.0
	ShopWiki/1.0 ( +http://www.shopwiki.com/wiki/Help:Bot)
	Mozilla/5.0 (compatible; UnisterBot; http://www.bluekiwi.de/misc/imprint)
	Web-sniffer/1.0.31 (+http://web-sniffer.net/)
	woobot
	Mozilla/5.0 (X11; U; Linux Core i7-4980HQ; de; rv:32.0; compatible; JobboerseBot; https://www.jobboerse.com/bot.htm) Gecko/20100401 Firefox/24.0
	AntBot/Ant-Nutch-1.1 (Ant Nutch Crawler; http://www.ant.com; [email protected])
	Mozilla/5.0 (compatible; YandexMedia/3.0; +http://yandex.com/bots)
	Mozilla/5.0 (compatible; Evrinid Iudex 1.0.0; +http://www.evri.com/evrinid)
	adidxbot/1.1 (+http://search.msn.com/msnbot.htm)
	Mozilla/5.0 (compatible; WebThumbnail/2.2; Website Thumbnail Generator; +http://webthumbnail.org)
	Mozilla/5.0 (compatible; Embedly/0.2; +http://support.embed.ly/)
	EasyBib AutoCite (http://content.easybib.com/autocite/)
	yacybot (-global; amd64 Linux 3.2.0-4-amd64; java 1.7.0_65; Europe/en) http://yacy.net/bot.html
	Mozilla/5.0 (compatible; woriobot +http://worio.com)
	Mozilla/4.0 (compatible; Vagabondo/4.0; http://webagent.wise-guys.nl/)
	SniffRSS/0.5beta (+http://www.blogator.com/)
	woobot/1.1
	Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) WordPress.com mShots Safari/537.36
	ADmantX Platform Semantic Analyzer - Turn - ADmantX Inc. - www.admantx.com - [email protected]
	Mozilla/5.0 (compatible; UnisterBot; [email protected])
	Mozilla/5.0 (compatible; electricmonk/3.2.0 +https://www.duedil.com/our-crawler/)
	UptimeDog Robot (www.uptimedog.com)
	Googlebot/2.1 (+http://www.google.com/bot.html)
	Mozilla/5.0 (Windows NT 6.3; WOW64; rv:36.0) Gecko/20100101 Firefox/36.0 (NetShelter ContentScan, contact [email protected] for information)
	Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.0; Trident/4.0; SLCC1; .NET CLR 2.0.50727; Media Center PC 5.0; .NET CLR 3.5.30729; .NET4.0C; .NET CLR 3.0.30729; .NET4.0E) PTST/276
	ServiceUptime.robot
	Mozilla/4.0 (compatible;HostTracker/2.0;+http://www.host-tracker.com/)
	SEO Browser
	Leikibot/1.0 (+http://www.leiki.com)
	yacybot (webportal-global; amd64 Windows 7 6.1; java 1.7.0_04; America/en) http://yacy.net/bot.html
	AppEngine-Google; (+http://code.google.com/appengine; appid: downforeveryoneorjustme)
	Mozilla/5.0 (Windows NT 5.1) BrokenLinkCheck.com/1.1
	Seobility (SEO-Check; http://bit.ly/1dJuuzs)
	Mozilla/5.0 (seoanalyzer; compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)
	Mozilla/5.0 (compatible; Digincore bot; https://www.digincore.com/crawler.html for rules and instructions.
	Site24x7
	websitepulse checker/3.0 (compatible; MSIE 5.5; Netscape 4.75; Linux)
	Mozilla/5.0 (compatible; SemrushBot-SA/0.97; +http://www.semrush.com/bot.html)
	Mozilla/5.0 (compatible; NerdByNature.Bot; http://www.nerdbynature.net/bot)
	Abrave Spider v4 Robot 2 (http://robot.abrave.co.uk)
	HubPages V0.2.2 (http://hubpages.com/help/crawlingpolicy)
	lmspider ([email protected])
	magpie-crawler/1.1 (U; Linux amd64; en-GB; +http://www.brandwatch.net)
	Mozilla/5.0 (compatible; WbSrch/1.1 +http://wbsrch.com
	Mozilla/5.0 (Windows; U; Windows NT 5.1; de; rv:1.9.0.7; Google-SearchByImage) Gecko/2009021910 Firefox/3.0.7
	Mozilla/5.0 (compatible; Najdi.si/3.1)
	larbin_2.6.3 ([email protected])
	Mozilla/5.0 (compatible; aiHitBot/1.1; +http://www.aihit.com/)
	Mozilla/5.0 (Windows NT 6.3; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/50.0.2661.102 Safari/537.36 PTST/277
	Mozilla/5.0 (YahooYSMcm/3.0.0; http://help.yahoo.com)
	yacybot (/global; amd64 Linux 3.16.0-4-amd64; java 1.7.0_79; Europe/en) http://yacy.net/bot.html
	OOZBOT/0.20 ( http://www.setooz.com/oozbot.html ; agentname at setooz dot_com )
	findlinks/2.0.9 (+http://wortschatz.uni-leipzig.de/findlinks/)
	Mozilla/5.0 (compatible; Lipperhey Link Explorer; http://www.lipperhey.com/)
	Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.6; en-US; rv:1.9.2) Gecko/20100115 Firefox/3.6 (FlipboardProxy/1.1; +http://flipboard.com/browserproxy)
	Mozilla/5.0 (Twiceler-0.9 http://www.cuil.com/twiceler/robot.html)
	Mozilla/5.0 (compatible; memoryBot/1.24.11 +http://internetmemory.org/en/)
	RED/1 (https://redbot.org/)
	Twitterbot/1.0
	Mozilla/5.0 (compatible; ptd-crawler; +http://bixolabs.com/crawler/ptd/; [email protected])
	Open Web Analytics Bot 1.5.4
	JikeSpider Mozilla/5.0 (compatible; JikeSpider; +http://shoulu.jike.com/spider.html)
	Mozilla/5.0 (compatible; vkShare; +http://vk.com/dev/Share)
	Mozilla/5.0 (compatible; oBot/2.3.1; +http://www-935.ibm.com/services/us/index.wss/detail/iss/a1029077?cntxt=a1027244)
	elefent/Elefent 1.2 (A friendly web elefent.; http://elefent.eu/; [email protected])
	Genderanalyzer/1.0
	elefent/1.2 (Web Crawler; https://crawler.elefent.net; webmaster at elefent dot net)
	Yandex/1.01.001 (compatible; Win16; H)
	crawler4j for XQuery
	Mozilla/5.0 (compatible; Plukkie/1.4; http://www.botje.com/plukkie.htm)
	LinguaBot/v0.001-dev (MultiLinual Sarch Engine v0.001; LinguaSeek; admin at linguaseek dot com)
	MrakyPraceCzBot/1.0 (http://www.mrakyprace.cz/about-crawler)
	Y!J-BRJ/YATS crawler (http://help.yahoo.co.jp/help/jp/search/indexing/indexing-15.html)
	WatchMouse/8.4.0.3 (http://watchmouse.com/ ; plgdn01.watchmouse.net)
	Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2272.96 Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
	AskQuickly v2 (http://askquickly.org/)
	Mozilla/5.0 (compatible; SISTRIX Crawler; http://crawler.sistrix.net/)
	Miniflux (http://miniflux.net)
	Mozilla/5.0 (compatible; spbot/5.0.1; +http://OpenLinkProfiler.org/bot )
	Mozilla/5.0 (compatible; wmtips.com/1.0; +http://www.wmtips.com/tools/)
	Mozilla/5.0 (compatible; linkdexbot/2.0; +http://www.linkdex.com/bots/)
	Mozilla/2.0 (compatible; Ask Jeeves/Teoma; +http://sp.ask.com/docs/about/tech_crawling.html)
	Google favicon
	Mozilla/5.0(compatible; Sosospider/2.0; +http://help.soso.com/webspider.htm)
	MiaDev/0.0.1 (MIA Bot for research project MIA (www.MIA-marktplatz.de); http://www.mia-marktplatz.de/spider; [email protected])
	Mozilla/5.0 (compatible; proximic; +http://www.proximic.com/info/spider.php)
	Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/33.0.1750.146 Safari/537.36 Specificfeeds- http://www.specificfeeds.com
	Mozilla/5.0 (compatible; socialbm_bot/1.0; +http://spider.socialbm.net)
	wp.com feedbot/1.0 (+https://wp.com)
	Mozilla/5.0 (compatible; CligooRobot/2.0; +http://www.cligoo.de/wk/technik.php)
	Slack-ImgProxy (+https://api.slack.com/robots)
	Mozilla/5.0 (compatible; Online Domain Tools - HTTP Headers Online/1.0; +http://http-headers.online-domain-tools.com)
	Mozilla/5.0 (compatible; SurdotlyBot/1.0; +http://sur.ly/bot.html)
	WGETbot/1.0 (+http://wget.alanreed.org)
	yacybot (/global; amd64 Linux 4.0.9-2-ck; java 1.8.0_51; Europe/de) http://yacy.net/bot.html
	Y!J-BRI/0.0.1 crawler ( http://help.yahoo.co.jp/help/jp/search/indexing/indexing-15.html )
	dlvr.it/1.0 (+http://dlvr.it/) Mozilla/5.0
	yacybot (/global; arm Linux 4.1.13-v7+; java 1.8.0; Etc/de) http://yacy.net/bot.html
	Mozilla/5.0 (compatible; GimmeUSAbot/1.0; +http://gimmeusa.com/crawler.html)
	yacybot (freeworld/global; amd64 Windows 7 6.1; java 1.6.0_29; Europe/fr) http://yacy.net/bot.html
	Link Valet Online 1.2
	Mozilla/5.0 (compatible; METASpider; +http://meta.ua/spider)
	NeutrinoAPI/2.0.1
	thumbshots-de-bot (+http://www.thumbshots.de/)
	Mozilla/5.0 (iPhone; CPU iPhone OS 6_0 like Mac OS X) AppleWebKit/536.26 (KHTML, like Gecko) Version/6.0 Mobile/10A5376e Safari/8536.25 (compatible; Applebot/0.3; +http://www.apple.com/go/applebot)
	argenfybot
	MergadoBot
	Ant/Ant-Nutch-1.1 (Ant Nutch Crawler; http://www.ant.com; [email protected])
	CopperEgg/RevealUptime/SaoPauloBrazil
	Mozilla/4.0 (compatible; Vagabondo/4.0/EU; http://webagent.wise-guys.nl/)
	Mozilla/4.0 (compatible; DAWINCI ANTIPLAG SPIDER)
	Mozilla/5.0 (Windows NT 6.1; WOW64; rv:45.0) Gecko/20100101 Firefox/45.0 PTST/276
	HetrixTools.COM Uptime Monitoring Bot. https://hetrixtools.com/uptime-monitoring-bot.html
	yacybot (/global; amd64 Linux 3.2.0-4-amd64; java 1.7.0_79; Europe/en) http://yacy.net/bot.html
	Mozilla/5.0 (Windows NT 6.1) (compatible; SMTBot/1.0; +http://www.similartech.com/smtbot)
	Mozilla/5.0 (compatible; InbyBot/2.1; +http://inbyapp.com)
	all_web1 ([email protected])
	Mozilla/5.0 (compatible; BusinessSeek.biz_Directory; http://www.businessseek.biz/)
	Wotbox/2.0 ([email protected]; http://www.wotbox.com)
	Mediatoolkitbot ([email protected])
	OpenWebSpider v0.1.2.B (http://www.openwebspider.org/)
	Mozilla/5.0 (compatible; WebCorp/5.0; +http://www.webcorp.org.uk)
	Testomatobot/1.0 (Linux x86_64; +http://www.testomato.com/testomatobot) minicrawler/4.0.0
	fastbot crawler beta 2.0 (+http://www.fastbot.de)
	Sogou Web Spider
	DoCoMo/2.0 P900i(c100;TB;W24H11) (compatible; ichiro/mobile goo;+http://search.goo.ne.jp/option/use/sub4/sub4-1/)
	Mozilla/5.0 (compatible; imagecoccoc/1.0; +http://help.coccoc.com/searchengine)
	XoviBot/1.0
	Mozilla/5.0 (compatible; Online Domain Tools - Online Website Link Checker/1.1; +http://website-link-checker.online-domain-tools.com)
	WatchMouse/8.4.0.3 (http://watchmouse.com/ ; decgn02.watchmouse.net)
	yacybot (i386 Linux 2.6.31-21-generic; java 1.6.0_0; Europe/en) http://yacy.net/bot.html
	ZnajdzFoto/ImageBot 2.0b
	Mozilla/5.0 (compatible; heritrix/3.0.0-SNAPSHOT-20091120.021634 +http://crawler.archive.org)
	Mozilla/5.0 (compatible; ProCogSEOBot/1.0; +http://www.procog.com/ )
	Mozilla/5.0 (compatible; Plukkie/1.6; http://www.botje.com/plukkie.htm)
	yacybot (freeworld/global; amd64 Linux 3.2.0-21-generic; java 1.7.0_03-icedtea; America/en) http://yacy.net/bot.html
	Mozilla/5.0 (iPhone; CPU iPhone OS 8_3 like Mac OS X) AppleWebKit/600.1.4 (KHTML, like Gecko) Version/8.0 Mobile/12F70 Safari/600.1.4 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
	Mozilla/5.0 (compatible; MSIE or Firefox mutant; not on Windows server; + http://tab.search.daum.net/aboutWebSearch.html) Daumoa/3.0
	Mozilla/5.0 (compatible; NLNZ_IAHarvester2014 +http://natlib.govt.nz/about-us/current-initiatives/web-harvest-2014)
	Mozilla/5.0 (compatible; archive.org_bot +http://archive.org/details/archive.org_bot)
	Mozilla/5.0 (compatible; special_archiver/3.3.0 +http://www.loc.gov/webarchiving/notice_to_webmasters.html)
	yacybot (freeworld/global; i386 Linux 3.4.2-linode44; java 1.6.0_27; America/en) http://yacy.net/bot.html
	Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/47.0.2526.111 Safari/537.36 Yandex.Translate
	yacybot (webportal-global; amd64 Linux 3.2.0-4-amd64; java 1.7.0_25; Europe/en) http://yacy.net/bot.html
	FreeWebMonitoring SiteChecker/0.1 (+http://www.freewebmonitoring.com)
	Mozilla/5.0 (compatible; Infohelfer/1.3.0; +http://www.infohelfer.de/crawler.php)
	Kemvibot/1.0 (http://kemvi.com)
	Mozilla/4.0 (compatible; HostTracker.com/1.0;+http://host-tracker.com/)
	AdsBot-Google-Mobile (+http://www.google.com/mobile/adsbot.html) Mozilla (iPhone; U; CPU iPhone OS 3 0 like Mac OS X) AppleWebKit (KHTML, like Gecko) Mobile Safari
	Mozilla/5.0 (compatible; GeliyooBot/1.0; +http://www.geliyoo.com/)
	yacybot (x86 Windows XP 5.1; java 1.6.0_21; Europe/de) http://yacy.net/bot.html
	Mozilla/5.0 (compatible; archive.org_bot/3.1.2 +http://natlib.govt.nz/about-us/current-initiatives/web-harvest-2012)
	Mozilla/5.0 (compatible; StorygizeBot; http://www.storygize.com)
	infohelfer/1.1.0 (http://www.infohelfer.de/)
	yacybot (/global; amd64 Linux 3.16.0-41-generic; java 1.7.0_80; Europe/de) http://yacy.net/bot.html
	SeoCheckBot (FischerNetzDesign Seo Checker, [email protected])
	netEstate NE Crawler (+http://www.website-datenbank.de/)
	Mail.Ru/1.0
	binlar_2.6.3 ([email protected])
	LinkedInBot/1.0 (compatible; Mozilla/5.0; Jakarta Commons-HttpClient/3.1 +http://www.linkedin.com)
	bot.wsowner.com/1.0.0 (+http://wsowner.com/)
	Ronzoobot/1.5 (http://www.ronzoo.com/about/)
	Mozilla/5.0 (compatible; heritrix/3.2.0 +http://www.mixdata.com)
	Mozilla/5.0 (compatible; Yahoo! Slurp China; http://misc.yahoo.com.cn/help.html)
	findlinks/1.1.6-beta6 (+http://wortschatz.uni-leipzig.de/findlinks/)
	Mozilla/5.0 (compatible; LoadTimeBot/0.7; +http://www.load-time.com/bot.html)
	Mozilla/5.0 (compatible; MJ12bot/v1.4.2; http://www.majestic12.co.uk/bot.php?+)
	Mozilla/4.0 (compatible; MSIE 5.01; GomezAgent 2.0; Windows NT)
	Mozilla/5.0 (compatible; MJ12bot/v1.3.2; http://www.majestic12.co.uk/bot.php?+)
	SEOENGBot/1.2 (+http://learn.seoeng.com/seoengbot.htm)
	yacybot (i386 Linux 2.6.31-18-generic; java 1.6.0_0; Europe/en) http://yacy.net/bot.html
	Online Virus Scanner: http://tools.geek-tools.org
	DwnldBot (+http://dwnld.me)
	Mozilla/5.0 (compatible; OpenfosBot/2.1; http://www.openfos.com)
	taptubot *** please read http://www.taptu.com/corp/taptubot ***
	yacybot (freeworld/global; amd64 Linux 3.9.3-1-ARCH; java 1.7.0_21; America/en) http://yacy.net/bot.html
	Testomatobot/1.0 (Linux x86_64; +http://www.testomato.com/testomatobot) minicrawler/4.0.0~beta10
	rogerbot/1.0 (http://www.moz.com/dp/rogerbot, [email protected])
	Mozilla/5.0 (Windows NT 6.1; Win64; x64) KomodiaBot/1.0
	Mozilla/5.0 (compatible; spbot/3.0; +http://www.seoprofiler.com/bot )
	Mozilla/5.0 (compatible; DomainTunoCrawler/0.1; +https://www.domaintuno.com/robot)
	Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Googlebot/2.1; +http://www.google.com/bot.html) Safari/537.36
	yacybot (amd64 Linux 2.6.32-gentoo; java 1.6.0_17; Europe/en) http://yacy.net/bot.html
	Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko; Google Page Speed Insights) Chrome/27.0.1453 Mobile Safari/537.36
	Mozilla/5.0 (compatible; GimmeUSAbot/1.0; +http://gimmeusa-update.com/crawler)
	Cloud mapping experiment. Contact [email protected]
	emefgebot/beta (+http://emefge.de/bot.html)
	DoCoMo/2.0 D903i(c100;TB;W28H20) (compatible; Hatena-Mobile-Gateway/1.2; +http://mgw.hatena.ne.jp/help)
	ichiro/3.0 (http://help.goo.ne.jp/door/crawler.html)
	Mozilla/5.0 (compatible; SecurityResearch.bot; +http://besome1.info/securityresearchbot.html)
	WPScan v2.9 (http://wpscan.org)
	Ronzoobot/1.6 (http://www.ronzoo.com/about/)
	livedoor ScreenShot/0.10
	Mozilla/5.0 (compatible; MSIE 8.0; Windows NT 5.1) KomodiaBot/1.0
	Mozilla/5.0 (compatible; memoryBot/1.20.289 +http://internetmemory.org/en/)
	Mozilla/5.0 (compatible; DomainTunoCrawler/0.1; +http://www.domaintuno.com/robot)
	yacybot (freeworld/global; amd64 Linux 2.6.26-2-amd64; java 1.6.0_18; Europe/de) http://yacy.net/bot.html
	Mozilla/5.0 (compatible; GimmeUSAbot/1.0; +http://gimmeusa.com/crawler)
	Mozilla/5.0 (compatible; spbot/5.0; +http://OpenLinkProfiler.org/bot )
	Mozilla/5.0 (compatible; Webauskunft.org; http://www.webauskunft.org)
	CCResearchBot/1.0 commoncrawl.org/research//Nutch-1.7-SNAPSHOT
	Mozilla/4.0 (compatible; MSIE 5.5; Windows NT 4.0; obot)
	linkdex.com/v2.0
	Mozilla/5.0 (compatible; Crawlera/1.10.2; UID 73393)
	Mozilla/5.0 (compatible; SEODat/0.1 http://crawler.seodat.com)
	yacybot (freeworld/global; amd64 Linux 3.2.0-4-amd64; java 1.6.0_27; Europe/en) http://yacy.net/bot.html
	CorpusCrawler 2.0.12 (http://corpora.fi.muni.cz/crawler/)
	Mozilla/5.0 (compatible; spbot/4.3.0; +http://OpenLinkProfiler.org/bot )
	Mozilla/5.0 (compatible; aiHitBot/2.7; +http://www.aihit.com/)
	Mozilla/5.0 (compatible; SiteAnalyzerBot/4.0; +https://www.site-analyzer.com/)
	DataparkSearch/4.40.1 (+http://www.dataparksearch.org/)
	yacybot (freeworld-global; amd64 Linux 3.0.0-21-generic; java 1.6.0_26; America/en) http://yacy.net/bot.html
	findlinks/1.1.6-beta1 (+http://wortschatz.uni-leipzig.de/findlinks/; YaCy 0.1; yacy.net)
	CopperEgg/RevealUptime/SydneyAU(aws)
	OpenAcoon v4.1.0 (www.openacoon.de)
	semanticbot ([email protected])
	Mozilla/5.0 (compatible; Crawlera/1.10.2; UID 75425)
	Mozilla/5.0 (compatible; Crawlera/1.10.2; UID 11912)
	Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp)
	Mozilla/5.0 (Windows NT 5.1; U; Win64; fr; rv:1.8.1) VoilaBot BETA 1.2 ([email protected])
	yacybot (-global; amd64 Linux 3.13.0-61-generic; java 1.7.0_79; Europe/en) http://yacy.net/bot.html
	CorpusCrawler 2.0.25 (http://corpora.fi.muni.cz/crawler/);Project:CzCorpus
	ichiro/3.0 (http://search.goo.ne.jp/option/use/sub4/sub4-1/)
	R6_FeedFetcher_(www.radian6.com/crawler)
	sosoimagespider+(+http://help.soso.com/soso-image-spider.htm)
	Mozilla/5.0 (X11; Linux x86_64; rv:39.0; GTmetrix http://gtmetrix.com/) Gecko/20100101 Firefox/39.0
	yacybot (freeworld/global; amd64 Linux 3.2.0-24-generic; java 1.6.0_24; Europe/en) http://yacy.net/bot.html
	Mozilla/4.0 (compatible; Vagabondo/4.0; webcrawler at wise-guys dot nl; http://webagent.wise-guys.nl/)
	CommaFeed/2.4.0-SNAPSHOT (https://github.com/Athou/commafeed)
	Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; Google Web Preview Analytics) Chrome/27.0.1453 Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
	yacybot (/global; x86 Windows 7 6.1; java 1.8.0_31; Europe/de) http://yacy.net/bot.html
	Mozilla/5.0 (compatible; woriobot support [at] zite [dot] com +http://zite.com)
	Mozilla/5.0 (compatible; MetaJobBot; http://www.metajob.at/crawler)
	bl.uk_lddc_bot/3.3.0-SNAPSHOT-20140519-0844 (+http://www.bl.uk/aboutus/legaldeposit/websites/websites/faqswebmaster/index.html)
	FFbot/1.0; +http://feedfury.com
	Mozzila/5.0 (compatible; YamanaLab-Robot/1.0; http://www.yama.info.waseda.ac.jp/~crawler/info.html)
	LinqiaRSSBot/1.0 ([email protected])
	FreeWebMonitoring SiteChecker/0.2 (+http://www.freewebmonitoring.com/bot.html)
	Mozilla/5.0 (compatible; Web-Monitoring/1.0; +http://monoid.nic.ru/)
	Mozilla/5.0 (compatible; heritrix/1.12.1b +http://netarkivet.dk/website/info.html)
	Mozilla/5.0 (compatible; SpiderLing (a SPIDER for LINGustic research); http://nlp.fi.muni.cz/projects/biwec/)
	UrlTrends Ranking Updater/2.0 ( http://www.urltrends.com/ )
	yacybot (/global; amd64 Linux 3.16.1-gentoo; java 1.7.0_55; Europe/en) http://yacy.net/bot.html
	Mozilla/5.0 (compatible; gimme60bot/1.0 ; +http://gimme60.com)
	CatchBot/5.0; +http://www.catchbot.com
	elefent/Elefent 1.1 (Friendly web walker.; http://elefent.eu/; [email protected])
	findlinks/2.0.1 (+http://wortschatz.uni-leipzig.de/findlinks/)
	Googlebot-Video/1.0
	Mozilla/5.0 (compatible; XmarksFetch/1.0; +http://www.xmarks.com/about/crawler; [email protected])
	XRL/3.00 (Linux; i686; en-us) (+http://metamark.net/about)
	Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/50.0.2661.87 Safari/537.36 PTST/276
	CorpusCrawler 2.0.8 (http://corpora.fi.muni.cz/crawler/)
	Mozilla/5.0 (compatible; archive.org_bot +http://www.archive.org/details/archive.org_bot)
	Mozilla/5.0 (compatible; OpenindexDeepSpider/Nutch-1.5-dev; +http://openindex.io/spider.html; systemsATopenindexDOTio)
	Ronzoobot/1.3 (http://www.ronzoo.com/about.php)
	MetaInspector/2.2.1 (+https://github.com/jaimeiniesta/metainspector)
	Mozilla/5.0 (compatible; WASALive-Bot ; http://blog.wasalive.com/wasalive-bots/)
	Mozilla/5.0 (compatible; YioopBot; +http://www.yioop.com/bot.php)
	Mozilla/5.0 (compatible; Mozilla/5.0; +http://wiki.github.com/bixo/bixo/bixocrawler; [email protected])
	Mozilla/5.0 (compatible; Kraken/0.1; http://linkfluence.net/; [email protected])
	Mozilla/5.0 (compatible; Qseero; +http://www.q0.com)
	yacybot (-global; amd64 Linux 3.10.0-229.4.2.el7.x86_64; java 1.8.0_45; Europe/en) http://yacy.net/bot.html
	Mozilla/5.0 (compatible; ProCogBot/1.0; +http://www.procog.com/spider.html)
	Mozilla/5.0 (compatible; MJ12bot/v1.4.3; http://www.majestic12.co.uk/bot.php?+)
	Mozilla/5.0 (Anturis Agent)
	Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/534.34 (KHTML, like Gecko) Chromium/31.0.1650.63 Site-Shot/2.1 (http://www.site-shot.com/) Safari/534.34
	webnumbrFetcher/1.0 (http://webnumbr.com/)
	Mozilla/5.0 (compatible; meanpathbot/1.0; +http://www.meanpath.com/meanpathbot.html)
	NIF/1.1 (http://www.newsisfree.com/robot.php users:)
	Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/51.0.2704.103 Safari/537.36 PTST/284
	DoCoMo/2.0 P901i(c100;TB;W24H11) (compatible; ichiro/mobile goo; +http://help.goo.ne.jp/door/crawler.html)
	Iframely/0.8.8 (+https://iframely.com/;)
	yacybot (ppc Mac OS X 10.5.2; java 1.5.0_13; Europe/de) http://yacy.net/bot.html
	Mozilla/5.0 (compatible; UptimeRobot/2.0; http://www.uptimerobot.com/)
	yacybot (/global; amd64 Linux 3.13.0-85-generic; java 1.7.0_95; Europe/en) http://yacy.net/bot.html
	Mozilla/5.0 (compatible; FeedBooster; +http://feeds.qsensei.com)
	yacybot (freeworld/global; amd64 Linux 2.6.32-5-amd64; java 1.6.0_18; Europe/de) http://yacy.net/bot.html
	Mozilla/5.0 (compatible; SputnikFaviconBot/1.2; +http://corp.sputnik.ru/webmaster)
	Mozilla/5.0 (compatible; heritrix/3.1.1; UniLeipzigASV +http://corpora.informatik.uni-leipzig.de/crawler_faq.html)
	Mozilla/5.0 (compatible; OrangeBot/2.0; [email protected])
	Yeti-FeedItemCrawler/1.0 (NHN Corp.; http://help.naver.com/robots/)
	Mozilla/5.0 (compatible; SeznamBot/3.2-test1-1; +http://napoveda.seznam.cz/en/seznambot-intro/)
	Mozilla/5.0 (compatible; heritrix/3.2.0 +http://www.crim.ca)
	yacybot (/global; amd64 Linux 4.0.5-gentoo; java 1.8.0_51; Europe/en) http://yacy.net/bot.html
	pr-cy.ru Screenshot Bot
	GentleSource Short URL Checker ( http://www.gentlesource.com/ )
	crazywebcrawler 0.9.0, http://www.crazywebcrawler.com
	a.pr-cy.ru
	Mozilla/5.0 (iPhone; CPU iPhone OS 7_0 like Mac OS X) AppleWebKit/537.51.1 (KHTML, like Gecko) Version/7.0 Mobile/11A465 Safari/9537.53 (compatible; bingbot/2.0; http://www.bing.com/bingbot.htm)
	findlinks/1.1.6-beta1 (+http://wortschatz.uni-leipzig.de/findlinks/)
	ichiro/5.0 (http://help.goo.ne.jp/door/crawler.html)
	yacybot (/global; amd64 Windows 7 6.1; java 1.8.0_25; Europe/en) http://yacy.net/bot.html
	Yeti/1.1 (Naver Corp.; http://help.naver.com/robots/)
	Technoratibot/8.0
	Setooz/Nutch-1.0 (http://www.setooz.com)
	Mozilla/5.0 (Windows NT 5.2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/49.0.2623.112 Safari/537.36 PTST/201
	semanticbot
	Mozilla/5.0 (compatible; JikeSpider; +http://shoulu.jike.com/spider.html)
	Mozilla/5.0 (compatible; MetamojiCrawler/1.0; +http://www.metamoji.com/jp/crawler.html
	Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1.6) Gecko/20070725 Firefox/2.0.0.6 - James BOT - WebCrawler http://cognitiveseo.com/bot.html
	Mozilla/5.0 (Windows NT 6.1; compatible; BDCbot/1.0; +http://ecommerce.bigdatacorp.com.br/faq.aspx) ppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2272.118 Safari/537.36
	Mozilla/5.0 (compatible; CloudServerMarketSpider/1.0; +http://www.cloudservermarket.com/spider.html)
	Mozilla/5.0 (Windows NT 6.1; WOW64; rv:38.0; Neustar WPM) Gecko/20100101 Firefox/38.0
	Mozilla/5.0 (compatible;+ParchBot/1.0;++http://www.parchmenthill.com/search.htm)
	Sogou-Test-Spider/4.0 (compatible; MSIE 5.5; Windows 98)
	MetaHeadersBot (+http://www.metaheaders.com/bot.html)
	Mozilla/5.0 (Windows; U; Windows NT 6.0; en-GB; rv:1.0; trendictionbot0.4.2; trendiction media ssppiiddeerr; http://www.trendiction.com/bot/; please let us know of any problems; ssppiiddeerr at trendiction.com) Gecko/20071127 Firefox/2.0.0.11
	voyager/2.0 (http://www.kosmix.com/crawler.html)
	Mozilla/5.0 (compatible; YodaoBot/1.0; http://www.yodao.com/help/webmaster/spider/; )
	hivaBot/hivaBot-1.0 (Iranian Search Engine bot; http://yooz.ir; [email protected])
	yacybot (freeworld/global; amd64 Linux 3.2.0-4-amd64; java 1.7.0_55; Europe/de) http://yacy.net/bot.html
	AppEngine-Google; (+http://code.google.com/appengine; appid: s~closure-compiler-hrd)
	larbin_2.5.0 ([email protected])
	Scrapy/0.22.2 (+http://scrapy.org)
	Ronzoobot/1.2 (http://www.ronzoo.com/about.php)
	gooblog/2.0 (http://help.goo.ne.jp/contact/)
	Pingdom.com_bot_version_1.4_(http://www.pingdom.com/)
	Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/535.21 (KHTML, like Gecko) Chrome/19.0.1042.0 Safari/535.21 PagePeeker/2.1; +http://pagepeeker.com/robots/
	WeSEE:Ads/PageBot (http://www.wesee.com/bot/)
	yacybot (amd64 Linux 2.6.18-164.el5; java 1.6.0; Europe/de) http://yacy.net/bot.html
	SemanticScholarBot/1.0 (+http://s2.allenai.org/bot.html)
	Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/535. Safari/535.22+; (+http://immediatenet.com/thumbnail_api.html; free thumbnails)
	Mozilla/5.0 (compatible; Crawlera/1.10.2; UID 85747)
	SBIder/Nutch-1.0-dev (http://www.sitesell.com/sbider.html)
	Scrapy/1.0.4 (+http://scrapy.org)
	SanszBot/1.7(http://www.sansz.org/sanszbot, [email protected]) ([email protected])
	Mozilla/5.0 (compatible; KHTML, like Gecko) Chrome/43.0.2357.81 Safari/537.36 [email protected]
	Mozilla/5.0 (iPhone; U; CPU iPhone OS 4_3 like Mac OS X; en-GB) adbeat.com/policy AppleWebKit/533.17.9 (KHTML, like Gecko) Version/5.0.2 Mobile/8F190 Safari/6533.18.5
	Curious George - www.analyticsseo.com/crawler
	Grammarly/1.0 (http://www.grammarly.com)
	OpenCalaisSemanticProxy
	yacybot (freeworld/global; amd64 Linux 2.6.32-49-server; java 1.6.0_27; Europe/en) http://yacy.net/bot.html
	netEstate RSS crawler (+http://www.rss-directory.info/)
	ipv6-test.com validator
	Mozilla/5.0 (compatible; Qualidator.com Bot 1.0;)
	ichiro/2.0 ([email protected])
	yacybot (freeworld/global; amd64 Linux 2.6.38-14-generic; java 1.6.0_22; Europe/en) http://yacy.net/bot.html
	cg-eye live
	Mozilla/5.0 (compatible; OpenindexSpider; +http://www.openindex.io/en/webmasters/spider.html)
	Mozilla/5.0 (Windows; U; MSIE 9.0; Windows NT 9.0; en-US) AppEngine-Google; (+http://code.google.com/appengine; appid: s~virustotalcloud)
	Mozilla/5.0 (compatible; heritrix/1.14.3.r6601 +http://www.buddybuzz.net/yptrino)
	MetaGeneratorCrawler/1.3.4 (www.metagenerator.info)
	Mozilla/5.0 (Linux; Android 5.0; LG-D855 Build/LRX21R.A1422018487) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/40.0.2214.109 Mobile Safari/537.36 DareBoost
	Mozilla/5.0 (compatible; Nigma.ru/3.0; [email protected])
	Mozilla/5.0 (compatible; Wappalyzer; +https://github.com/AliasIO/Wappalyzer)
	yacybot (/global; x86 Windows XP 5.1; java 1.7.0_71; Europe/de) http://yacy.net/bot.html
	Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/50.0.2661.102 Safari/537.36 PTST/277
	OpenWebSpider v0.1.4 (http://www.openwebspider.org/)
	HubSpot Connect 1.0 (http://dev.hubspot.com/)
	Page Valet/4.1pre5
	msnbot/2.0b (+http://search.msn.com/msnbot.htm)
	amibot - http://www.amidalla.de - [email protected] libwww-perl/5.831
	SETOOZBOT/5.0 ( compatible; SETOOZBOT/0.30 ; http://www.setooz.com/bot.html )
	Mozilla/5.0 (compatible; YoudaoBot/1.0; http://www.youdao.com/help/webmaster/spider/; )
	Mozilla/5.0 (compatible; MSIE or Firefox mutant; not on Windows server;) Daumoa 4.0
	Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.0) LinkCheck by Siteimprove.com
	EdisterBot (http://www.edister.com/bot.html)
	Mozilla/5.0 (iPhone; CPU iPhone OS 6_0 like Mac OS X) AppleWebKit/536.26 (KHTML, like Gecko) Version/6.0 Mobile/10A5376e Safari/8536.25 (compatible; SMTBot/1.0; +http://www.similartech.com/smtbot)
	Tarantula/BETA-0.5 (Tarantula de la Enlle, el mejor buscador en la red; http://www.enlle.com; [email protected])
	Mozilla/5.0 (Windows; U; Windows NT 5.1; fr; rv:1.8.1) VoilaBot BETA 1.2 ([email protected])
	yacybot (freeworld/global; amd64 Linux 2.6.38-8-generic; java 1.6.0_22; Europe/en) http://yacy.net/bot.html
	Mozilla/5.0 (compatible; SemrushBot/0.95; +http://www.semrush.com/bot.html)
	CatchBot/3.0; +http://www.catchbot.com
	Mozilla/5.0 (compatible; uMBot-LN/1.0; mailto: [email protected])
	Mozilla/5.0 (compatible; RukiCrawler/1.0; +http://ruki.rezko.net)
	adbeat_bot
	Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/27.0.1453.116 Safari/537.36 HubSpot Webcrawler
	Mozilla/5.0 (compatible; YandexCatalog/3.0; +http://yandex.com/bots)
	NutchCVS/0.7.1 (Nutch; http://lucene.apache.org/nutch/bot.html; [email protected])
	Statastico/4.0
	facebookexternalhit/1.1 (+http://www.facebook.com/externalhit_uatext.php)
	Mozilla/5.0 (X11; U; Linux x86_64; en-US) AppleWebKit/534.16 (KHTML, like Gecko, Google Keyword Suggestion) Chrome/10.0.648.127 Safari/534.16
	Mozilla/5.0 (iPhone; CPU iPhone OS 7_0 like Mac OS X) AppleWebKit/537.51.1 (KHTML, like Gecko) Version/7.0 Mobile/11A465 Safari/9537.53 BingPreview/1.0b
	EuripBot/2.0 (+http://www.eurip.com) GetRobots
	Mozilla/5.0 (compatible; MJ12bot/v1.2.1; http://www.majestic12.co.uk/bot.php?+)
	Braintree-Webhooks
	Nutch Checksem/Nutch-1.10
	Mozilla/5.0 (compatible; HyperCrawl/0.2; +http://www.seograph.net/bot.html)
	quickobot/quickobot-1 (Quicko Labs; http://quicko.co; robot at quicko dot co)
	Whibse
	Yahoo! Site Explorer Feed Validator http://help.yahoo.com/l/us/yahoo/search/siteexplorer/manage/
	Mozilla/5.0 (compatible; AhrefsBot/5.0; +http://ahrefs.com/robot/)
	Siteshooter B0t 2.1
	GingerCrawler/1.0 (Language Assistant for Dyslexics; www.gingersoftware.com/crawler_agent.htm; support at ginger software dot com)
	Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko; Google Page Speed Insights) Chrome/27.0.1453 Safari/537.36
	eBot / v.0.1e (http://alfa.elchron.cz)
	Mozilla/5.0 (compatible; memoryBot/1.20.247 +http://internetmemory.org/en/)
	yacybot (x86 Windows 2003 5.2; java 1.6.0_20; America/de) http://yacy.net/bot.html
	YandexSomething/1.0
	Scoop
	Mozilla/5.0 (compatible; MegaIndex.ru/2.0; +http://megaindex.com/crawler)
	Mozilla/5.0 (compatible; SputnikBot/2.3; +http://corp.sputnik.ru/webmaster)
	TurnitinBot/2.1 (http://www.turnitin.com/robot/crawlerinfo.html)
	Mozilla/5.0 (compatible; Ask Jeeves/Teoma; +http://about.ask.com/en/docs/about/webmasters.shtml)
	Mozilla/5.0 (compatible; CrawlForMe/3.0.2; [email protected])
	Seologies/0.1 Bot
	www.deadlinkchecker.com XMLHTTP/1.0 Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/46.0.2490.86 Safari/537.36
	dubaiindex (addressendeutschland.de)
	yacybot (/global; x86 Windows 7 6.1; java 1.8.0_40; America/es) http://yacy.net/bot.html
	Mozilla/5.0+(compatible;+PiplBot;++http://www.pipl.com/bot/)
	LinkStats Bot
	Mozilla/5.0 (compatible; adbeat-publisher-description-fetcher; [email protected]; [email protected])
	CISPA Vulnerability Notification (https://notify.mmci.uni-saarland.de)
	librabot/1.0 (+http://search.msn.com/msnbot.htm)
	SeoCheckBot (Seomira Seo Check, http://www.kfsw.de/bot.html)
	yacybot (x86 Windows 2003 5.2; java 1.6.0_16; Europe/de) http://yacy.net/bot.html
	WatchMouse/8.4.0.3 (http://watchmouse.com/ ; itpda01.watchmouse.net)
	yacybot (/global; amd64 Linux 3.13.0-83-generic; java 1.7.0_95; Europe/en) http://yacy.net/bot.html
	BrainbruBot/1.0 (+http://www.brainbru.com/)
	Docoloc
	Mozilla/5.0 (Unknown; Linux x86_64) AppleWebKit/538.1 (KHTML, like Gecko) PhantomJS/2.1.1 Safari/538.1 Prerender (+https://github.com/prerender/prerender)
	dlvr.it/1.0 (+http://dlvr.it/)
	Mozilla/5.0 (compatible; parsijoo-bot; +http://www.parsijoo.ir/; [email protected])
	DataparkSearch/4.35-02122005 ( http://www.dataparksearch.org/)
	Pixray-Seeker/2.0 (Pixray-Seeker; +http://www.pixray.com/pixraybot; [email protected])
	Abrave v5.5 (http://robot.abrave.co.uk)
	Mozilla/5.0 (X11; U; Linux i686 (x86_64); en-US; rv:1.8.1.20) Gecko/20090429 HeartRails_Capture/0.6 (+http://capture.heartrails.com/) BonEcho/2.0.0.20
	yacybot (/global; amd64 Linux 3.19.0-26-generic; java 1.7.0_95; Europe/en) http://yacy.net/bot.html
	MLBot (www.metadatalabs.com/mlbot)
	yacybot (freeworld/global; amd64 Linux 2.6.32-34-server; java 1.6.0_26; Europe/de) http://yacy.net/bot.html
	Mozilla/5.0 (compatible; LinkpadBot/1.08; +http://www.linkpad.ru)
	Mozilla/5.0 (compatible; SearchmetricsBot; http://www.searchmetrics.com/en/searchmetrics-bot/)
	msnbot-NewsBlogs/2.0b (+http://search.msn.com/msnbot.htm)
	ZnajdzFoto/Image 2.0
	Mozilla/5.0 (compatible; Magibot/4.0.0; +http://magi.peak-labs.com/robots.txt)
	Mozilla/5.0 (compatible; spbot/4.0.8; +http://OpenLinkProfiler.org/bot )
	Eurobot/1.2 (http://eurobot.ayell.eu)
	Mozilla/5.0 (compatible; Embedly/0.2; snap; +http://support.embed.ly/)
	Yaanb/1.5.001 (compatible; Win64;+http://www.yaanb.com/company/bot.hmtl)
	yacybot (i386 Linux 2.6.31-16-generic; java 1.6.0_15; Europe/en) http://yacy.net/bot.html
	Mozilla/5.0 (compatible; YioopBot; +http://173.13.143.74/bot.php)
	CopperEgg/RevealUptime/Singapore(aws)
	Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/534.24 (KHTML, like Gecko; Google Web Preview) Chrome/11.0.696 Safari/534.24
	yacybot (freeworld/global; amd64 Windows 8 6.2; java 1.7.0_25; Europe/de) http://yacy.net/bot.html
	Server Density Service Monitoring v2
	Testomatobot/1.0 (Linux x86_64; +http://www.testomato.com/testomatobot) minicrawler/3.0~beta
	TinEye/1.0; +http://www.tineye.com/
	Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.9.0.19; aggregator:Spinn3r (Spinn3r 3.1); http://spinn3r.com/robot) Gecko/2010040121 Firefox/3.0.19
	GigablastOpenSource/1.0
	Googlebot (gocrawl v0.4)
	CSmartCrawler (email: abuse[at]adsadv[dot]com)
	Testomatobot/1.0 (Linux x86_64; +http://www.testomato.com/testomatobot) minicrawler/4.0.0~beta9
	KOCMOHABT (+http://kozmonavt.tk/) AppEngine-Google; (+http://code.google.com/appengine; appid: s~web-explorer)
	Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.2.18) Gecko/20110622 HeartRails_Capture/1.0.5 (+http://capture.heartrails.com/) Namoroka/3.6.18
	PingSpot/1.0 (http://www.pingspot.org/)
	AppEngine-Google; (+http://code.google.com/appengine; appid: s~buzzstream-tools-hrd)
	AddThis.com (http://support.addthis.com/)
	Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko; Google Web Preview) Chrome/27.0.1453 Safari/537.36
	ExactSeekCrawler/1.0
	Yeti/1.0 (NHN Corp.; http://help.naver.com/robots/)
	Mozilla/5.0 (compatible; jpg-newsbot/2.0; +http://vipnytt.no/bot.html)
	g2reader-bot/1.0 (+http://www.g2reader.com/)
	MetaGeneratorCrawler/1.3.12 (www.metagenerator.info)
	QuerySeekerSpider ( http://queryseeker.com/bot.html )
	KDDI-CA31 UP.Browser/6.2.0.7.3.129 (GUI) MMP/2.0 (compatible; ichiro/mobile goo; +http://help.goo.ne.jp/help/article/1142/)
	Mozilla/5.0 (compatible; GrapeshotCrawler/2.0; +http://www.grapeshot.co.uk/crawler.php)
	Mozilla/5.0 (compatible; MSIE 10.0; Windows NT 6.2; Trident/6.0; WOW64; Trident/6.0; BingPreview/1.0b)
	Mozilla/5.0 (compatible; firmilybot/0.3; +http://www.firmily.com/bot.php
	Mozilla/5.0 (X11; U; Linux Core i7-4980HQ; de; rv:32.0; compatible; JobboerseBot; http://www.jobboerse.com/bot.htm) Gecko/20100101 Firefox/38.0
	Mozilla/5.0 (compatible; metager2-verification-bot; +http://metager2.de/technology.php)
	crawler4j (http://code.google.com/p/crawler4j/)
	CirrusExplorer/1.1 (http://www.cireu.com/explorer.php)
	Mozilla/5.0 (compatible; RankvalBot/1.0; +https://rankval.com)
	Mozilla/5.0 (compatible; heritrix/1.14.3 +http://www.accelobot.com)
	Mozilla/5.0 (compatible; BLEXBotTest/1.0; +http://webmeup.com/crawler.html)
	nWormFeedFinder (http://www.nworm.com)
	netEstate NE Crawler (+http://www.sengine.info/)
	Mozila/5.0 (compatible; StudioFACA Search; +http://www.studiofaca.com/)
	Mozilla/5.0 (compatible; Crawlera/1.10.2; UID 92751)
	wikiwix-bot-3.0
	SEOCentro Page Keyword Analyzer v1.2
	yacybot (amd64 Linux 2.6.18-164.15.1.el5xen; java 1.6.0_0; Europe/en) http://yacy.net/bot.html
	Mozilla/5.0 (compatible; bingbot/2.0; http://www.bing.com/bingbot.htm)
	Automattic Analytics Crawler/0.1; http://wordpress.com/crawler/
	SemrushBot/0.92
	http://www.uni-koblenz.de/~flocke/robot-info.txt
	Mozilla/5.0 (compatible; idmarch Automatic.beta/1.3; +http://www.idmarch.org/bot.html)
	Mozilla/5.0 (X11; Linux i686 on x86_64; rv:25.0; GTmetrix http://gtmetrix.com/) Gecko/20100101 Firefox/25.0
	msnbot/2.0b (+http://search.msn.com/msnbot.htm)._
	Stroke.cz (http://stroke.cz)
	Mozilla (+http://web.nli.org.il/sites/NLI/English/collections/israel-collection/internet_archive/Pages/default.aspx)
	audisto.com full crawler 4.33.470 (refer to in robots.txt as audisto, see https://audisto.com/bot)
	http://arachnode.net 2.5
	SurveyBot/2.3 (Whois Source)
	ant.com/Nutch-1.7 (http://ant.com)
	CorpusCrawler 2.0.13 (http://corpora.fi.muni.cz/crawler/)
	Seo Servis - Analyza zdrojoveho kodu
	SEOENGBot/1.2 (+http://www.seoengine.com/seoengbot.htm)
	Mozilla/5.0 (compatible; Genieo/1.0 http://www.genieo.com/webfilter.html)
	Mozilla/5.0 (compatible; evc-batch/2.0.20160708184126)
	HubSpot Marketing Grader
	Mozilla/5.0 (compatible; MSIE 10.0; Windows NT 6.1; Trident/6.0) LinkCheck by Siteimprove.com
	nrsbot/6.0(loopip.com/robot.html)
	Mozilla/5.0 (compatible; BLEXBot/1.0; +http://webmeup.com/crawler.html)
	yacybot (freeworld-global; amd64 Windows 7 6.1; java 1.7.0_02-ea; America/en) http://yacy.net/bot.html
	Mozilla/5.0 (compatible; Blekkobot; ScoutJet; +http://blekko.com/about/blekkobot)
	Mozilla/5.0 (compatible; Topicbot/12.0rc-2; +http://topicbot.awardspace.us/)
	Mozilla/5.0 (compatible; Gimme60bot/1.0; +http://gimme60.com)
	yacybot (i386 Linux 2.6.28-11-generic; java 1.6.0_13; Europe/de) http://yacy.net/bot.html
	Mozilla/5.0 (compatible; DotBot/1.1; http://www.opensiteexplorer.org/dotbot, [email protected])
	Kemvibot/1.0 (http://kemvi.com, [email protected])
	CorpusCrawler 2.0.15 (http://corpora.fi.muni.cz/crawler/)
	dotbot
	Mozilla/5.0 (iPhone; CPU iPhone OS 7_0 like Mac OS X) AppleWebKit/537.51.1 (KHTML, like Gecko) Version/7.0 Mobile/11A465 Safari/9537.53 (compatible; adidxbot/2.0; +http://www.bing.com/bingbot.htm)
	page_viewer ([email protected])
	HootSuite Image proxy ([email protected])
	Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US) Speedy Spider for SpeedyAds (http://www.entireweb.com/about/search_tech/speedy_spider/)
	PINGOMETER_BOT_(HTTPS://PINGOMETER.COM)
	Mozilla/5.0 (compatible; Vagabondo/2.1; webcrawler at wise-guys dot nl; http://webagent.wise-guys.nl/)
	VSE/1.0 ([email protected])
	Scrapy/0.24.0 (+http://scrapy.org)
	Speedy Spider (http://www.entireweb.com)
	Mozilla/5.0 (X11; Linux x86_64; GTmetrix https://gtmetrix.com/) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/51.0.2704.106 Safari/537.36
	WatchMouse/18990 (http://watchmouse.com/ ; hk)
	Mozilla/5.0 (compatible; yoozBot-2.2; http://yooz.ir; [email protected])
	Mozilla/5.0 (compatible; Crawlera/1.10.2; UID 87214)
	Mozilla/5.0 (compatible; discobot/2.0; +http://discoveryengine.com/discobot.html)
	yacybot (freeworld/global; amd64 Windows 7 6.1; java 1.8.0_11; Europe/de) http://yacy.net/bot.html
	Testomatobot/1.0 (Linux x86_64; +http://www.testomato.com/testomatobot) minicrawler/4.0.0~beta6
	yacybot (freeworld/global; amd64 Linux 3.0.0-15-server; java 1.6.0_23; Europe/en) http://yacy.net/bot.html
	Mozilla/5.0 (Macintosh; Intel Mac OS X 10.9; rv:28.0) Gecko/20100101 Firefox/28.0 (FlipboardProxy/1.1; +http://flipboard.com/browserproxy)
	YisouSpider
	Slack-ImgProxy 1.138 (+https://api.slack.com/robots)
	mozilla/5.0 (compatible; webmastercoffee/0.7; +http://webmastercoffee.com/about)
	Mozilla/5.0 (compatible; Uptimebot/0.2.22; +http://www.uptime.com/uptimebot)
	Mozilla/5.0 (compatible; AhrefsBot/1.0; +http://ahrefs.com/robot/)
	Mozilla/5.0 (compatible; Feedspotbot/1.0; +http://www.feedspot.com/fs/bot)
	sg-Orbiter/1.0 (+http://searchgears.de/uber-uns/crawling-faq.html)
	Site24x7 Tools
	NextGenSearchBot 1 (for information visit http://www.zoominfo.com/About/misc/NextGenSearchBot.aspx)
	Aboundex/0.3 (http://www.aboundex.com/crawler/)
	yacybot (freeworld/global; i386 Linux 2.6.37.6-0.5-desktop; java 1.6.0_20; Europe/en) http://yacy.net/bot.html
	Mozilla/5.0 (Windows NT 6.2; WOW64) AppleWebKit/537.4 (KHTML, like Gecko) Chrome/98 Safari/537.4 (StatusCake)
	Mozilla/5.0 (compatible; DomainSONOCrawler/0.1; +http://domainsono.com)
	Mozilla/5.0 (compatible; Online Domain Tools - Server Monitor/1.0; +http://server-monitoring.online-domain-tools.com)
	Mozilla/5.0 (compatible; Konqueror/3.5; Linux) KHTML/3.5.5 (like Gecko) (Exabot-Thumbnails)
	Mozilla/5.0 (compatible; spbot/4.0.4; +http://www.seoprofiler.com/bot )
	Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/27.0.1453 Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
	Factbot 1.09 (see http://www.factbites.com/webmasters.php)
	CopperEgg/RevealUptime/SaoPauloBR(aws)
	CRIM Crawler/Nutch-2.3 (Crawler du Centre de Recherche Informatique de Montr\xc3\xa9al (CRIM))
	Mozilla/5.0 (compatible; heritrix/1.14.3 +http://www.webarchiv.cz)
	Mozilla/5.0 (compatible; WormlyBot; +http://wormly.com)
	yacybot (i386 Linux 2.6.23; java 1.6.0_17; Europe/en) http://yacy.net/bot.html
	Mozilla/5.0 (compatible; Cliqzbot/0.1 +http://cliqz.com/company/cliqzbot)
	URLitor.com
	RavenCrawler
	Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.2.18) Gecko/20130331 HeartRails_Capture/1.0.5 (+http://capture.heartrails.com/) Namoroka/3.6.18
	Mozilla/5.0 (compatible; WebThumbnail/3.x; Website Thumbnail Generator; +http://webthumbnail.org)
	Kyoto-Crawler/2.0 (Mozilla-compatible; kyoto-crawler-contact(at)nlp(dot)kuee(dot)kyoto-u(dot)ac(dot)jp; http://nlp.ist.i.kyoto-u.ac.jp/)
	Mozilla/5.0 (compatible; SputnikImageBot/2.3; +http://corp.sputnik.ru/webmaster)
	WatchMouse/8.4.0.3 (http://watchmouse.com/ ; lvps91-250-96-109.dedicated.hosteurope.de)
	MetaGeneratorCrawler/1.3.3 (www.metagenerator.info)
	LongURL API
	Mozilla/5.0 ( compatible; SETOOZBOT/0.30 ; http://www.setooz.com/bot.html ; agentname at setooz dot_com )
	Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.6; en-US; rv:1.9.2) Gecko/20100115 Firefox/3.6 (FlipboardProxy/2.0; +http://flipboard.com/browserproxy)
	http://tools.geek-tools.org/link-counter/
	SCFCrawler/Nutch-1.8 (Image Crawler for StolenCameraFinder.com; http://www.stolencamerafinder.com/; [email protected])
	Mozilla/5.0 (compatible; Uptimebot/0.2.41; +http://www.uptime.com/uptimebot)
	Mozilla/5.0 (compatible; DomainAppender /1.0; +http://www.profound.net/domainappender)
	Mozilla/5.0 (compatible; hypestat/1.0; +http://www.hypestat.com/bot)
	yacybot (/global; amd64 Linux 3.13.0-042stab093.4; java 1.7.0_79; Europe/en) http://yacy.net/bot.html
	PagesInventory (robot +http://www.pagesinventory.com)
	Sosospider+(+http://help.soso.com/webspider.htm)
	Mozilla/5.0 (compatible; AntBot/1.0; +http://www.ant.com/)
	PagesInventory (robot http://www.pagesinvenotry.com)
	Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; http://www.changedetection.com/bot.html )
	Wotbox/2.01 (+http://www.wotbox.com/bot/)
	Mozilla/5.0 (compatible; bixolabs/1.0; +http://bixolabs.com/crawler/general; [email protected])
	Mozilla/5.0 (compatible; SpiderLing (a SPIDER for LINGustic research); +http://nlp.fi.muni.cz/projects/biwec/)
	CopperEgg/RevealUptime/DallasTXUSA
	yacybot (freeworld/global; amd64 Linux 2.6.32-custom; java 1.6.0_26; Europe/en) http://yacy.net/bot.html
	Bad-Neighborhood Link Analyzer (http://www.bad-neighborhood.com/)
	Mozilla/5.0 (compatible; oBot/2.3.1; +http://filterdb.iss.net/crawler/)
	Pixray-Seeker/2.0 (Pixray-Seeker; http://www.pixray.com/pixraybot; [email protected])
	Cityreview Robot (+http://www.cityreview.org/crawler/)
	yacybot (/global; amd64 Linux 3.16.0-4-amd64; java 1.7.0_95; Europe/en) http://yacy.net/bot.html
	Dlvr.it/1.0 (http://dlvr.it/)
	Zookabot/2.4;++http://zookabot.com
	AddThis.com robot [email protected]
	Mozilla/5.0 (compatible; discoverybot/2.0; +http://discoveryengine.com/discoverybot.html)
	TwengaBot
	Mozilla/5.0 (compatible; 4SeoHuntBot; +http://4seohunt.biz/about.html)
	Web-sniffer.me/1.0.0 (+http://web-sniffer.me/)
	KDDI-CA31 UP.Browser/6.2.0.7.3.129 (GUI) MMP/2.0 (compatible; ichiro/mobile goo; +http://search.goo.ne.jp/option/use/sub4/sub4-1/)
	gosquared-thumbnailer/1.0
	yacybot (/global; amd64 Windows 7 6.1; java 1.7.0_55; Asia/zh) http://yacy.net/bot.html
	BPImageWalker/2.0 (www.bdbrandprotect.com)
	Acoon v4.10.3 (www.acoon.de)
	Mozilla/4.0 (CMS Crawler: http://www.cmscrawler.com)
	WP Engine Install Performance API
	Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.2.24; ips-agent) Gecko/20111107 Ubuntu/10.04 (lucid) Firefox/3.6.24
	Mozilla/5.0 (Windows NT 6.1; WOW64; Trident/7.0; rv:11.0) like Gecko PTST/277
	CamontSpider/1.0 +http://epweb2.ph.bham.ac.uk/user/slater/camont/info.html
	SEOENGBot/1.2 (+http://learn.seoengine.com/seoengbot.htm)
	Y!J-BRJ/YATS crawler (http://listing.yahoo.co.jp/support/faq/int/other/other_001.html)
	CopperEgg/RevealUptime/N.Virginia(aws)
	Mozilla/5.0, Google-AdWords-Express
	Mozilla/5.0 (compatible;YodaoBot-Image/1.0;http://www.youdao.com/help/webmaster/spider/;)
	yacybot (/global; amd64 Linux 3.16.0-4-amd64; java 1.8.0_91; Europe/en) http://yacy.net/bot.html
	yacybot (freeworld/global; amd64 Windows 7 6.1; java 1.7.0_09; Europe/de) http://yacy.net/bot.html
	Acoon v4.10.1 (www.acoon.de)
	Mediatoolkitbot ([email protected])
	Mozilla/5.0 (compatible; Speedy Spider; http://www.entireweb.com/about/search_tech/speedy_spider/)
	Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.2.3) SPEng
	2Bone_LinkChkr/1.0 libwww-perl/6.03
	Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.6; en-US; rv:1.9.2) Gecko/20100115 Firefox/3.6 (FlipboardBrowserProxy/0.0.5; +http://flipboard.com/browserproxy)
	Mozilla/5.0 (compatible; SeznamBot/3.2-test1; +http://napoveda.seznam.cz/en/seznambot-intro/)
	Comodo SSL Checker
	Comodo Spider 1.2
	Who.is Bot
	Mozilla/5.0 (compatible; online-webceo-bot/1.0; +http://online.webceo.com)
	yacybot (freeworld/global; amd64 Windows Server 2008 R2 6.1; java 1.6.0_29; Europe/en) http://yacy.net/bot.html
	psbot/0.1 (+http://www.picsearch.com/bot.html)
	IlTrovatore-Setaccio/1.2 (http://www.iltrovatore.it/aiuto/faq.html)
	Mozilla/5.0 (compatible; Twingly Recon; twingly.com)
	Mozilla/5.0 Moreover/5.1 (+http://www.moreover.com; [email protected])
	Orgbybot/OrgbyBot v1.2 (Spidering the net for Orgby; http://www.orgby.com/ ; Orgby.com Search Engine)
	Browsershots
	AdsBot-Google (+http://www.google.com/adsbot.html)
	Qseero v1.0.0
	Mozilla/5.0 (compatible; ltbot/0.3.6 +http://www.kdsl.tu-darmstadt.de/de/kdsl/research-program/crawling-and-semantic-structuring/)
	yacybot (x86 Windows XP 5.1; java 1.6.0_18; Europe/de) http://yacy.net/bot.html
	visionutils/0.2
	Mozilla/5.0 (compatible; Googlebot/2.1; https://www.deepcrawl.com/bot)
	Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.4 (KHTML, like Gecko; Google Web Preview) Chrome/22.0.1229 Safari/537.4
	Mozilla/5.0 (compatible; Crawlera/1.10.2; UID 8942)
	Mozilla/4.0 (compatible; LinkMarket-Bot v1.2)
	Buzzbot/1.0 (Buzzbot; http://www.buzzstream.com; [email protected])
	WordPress.com; https://public-api.wordpress.com
	Mozilla/5.0 ( compatible; SETOOZBOT/0.30 ; http://www.setooz.com/bot.html )
	CopperEgg/RevealUptime/FremontCAUSA
	Mozilla/5.0 (Windows; U; Windows NT 6.0; en-GB; rv:1.0; trendictionbot0.4.5; trendiction search; http://www.trendiction.de/bot; please let us know of any problems; web at trendiction.com) Gecko/20071127 Firefox/3.0.0.11
	yacybot (/global; amd64 Linux 2.6.32-042stab094.8; java 1.7.0_79; America/en) http://yacy.net/bot.html
	Mozilla/5.0 (compatible; Windows NT 6.1?; ZumBot/1.0; http://help.zum.com/inquiry)
	Mozilla/5.0 (Windows NT 6.1; WOW64; Trident/7.0; rv:11.0; topster.de Linkchecker 5.0) like Gecko (90.177.52.133)
	Mozilla/5.0+(compatible; Monitority/1.0; http://www.monitority.com/)
	yacybot (freeworld/global; x86 Windows 7 6.1; java 1.8.0_25; Europe/de) http://yacy.net/bot.html
	Mozilla/5.0 (compatible; abby/1.0; +http://www.ellerdale.com/crawler.html)
	Mozilla/5.0 (compatible; SemrushBot/1.1~bl; +http://www.semrush.com/bot.html)
	YowedoBot/Yowedo 1.0 (Search Engine crawler for yowedo.com; http://yowedo.com/en/partners.html; [email protected])
	Mozilla/5.0 (compatible; oBot/2.3.1; http://filterdb.iss.net/crawler/)
	Nokia6820/2.0 (4.83) Profile/MIDP-1.0 Configuration/CLDC-1.0 (compatible; Googlebot-Mobile/2.1; +http://www.google.com/bot.html)
	Mozilla/5.0 (compatible; IstellaBot/1.23.15 +http://www.tiscali.it/)
	GetIntent Crawler (http://getintent.com/bot.html)
	Mozilla/5.0 (compatible; Uptimebot/0.1.89; +http://www.uptime.com/uptimebot)
	Cliqzbot
	AboutUsBot
	yacybot (-global; amd64 Linux 3.2.0-4-amd64; java 1.7.0_67; Europe/en) http://yacy.net/bot.html
	Sufog/Nutch-2.2.1 (www.sufog.com; www.sufog.com)
	Mozilla/5.0 (compatible; imbot/0.1 +http://internetmemory.org/en/)
	KOCMOHABT (+http://kozmonavt.tk/) Mozilla/5.0 (compatible; Web Explorer 1) AppEngine-Google; (+http://code.google.com/appengine; appid: s~web-explorer)
	yacybot (/global; amd64 Linux 3.16.0-53-generic; java 1.7.0_85; Europe/en) http://yacy.net/bot.html
	L.webis/0.53 (http://webalgo.iit.cnr.it/index.php?pg=lwebis)
	Mozilla/5.0 (compatible; memoryBot/1.21.23 +http://internetmemory.org/en/)
	ExB Language Crawler 2.1.5 (+http://www.exb.de/crawler)
	Peeplo Screenshot Bot/0.20 ( abuse at peeplo dot_com )
	Mozilla/5.0 (compatible; Qwantify/2.1w; +https://www.qwant.com/)/*
	Experibot_v1 [bit.ly/2bLMJdw]
	WikioFeedBot 1.0 (http://www.wikio.com)
	SalesIntelligent/v1.1 (https://www.salesintelligent.com)
	Mozilla/5.0 (compatible; ExaleadCloudView/5;)
	Mozilla/5.0 (compatible; rogerBot/1.0; UrlCrawler; http://www.seomoz.org/dp/rogerbot)
	Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/44.0.2403.157 Safari/537.36 TinEye/1.0 (via http://www.tineye.com/)
	Mozilla/5.0 (compatible; Search17Bot/1.1; http://www.search17.com/bot.php)
	FollowSite Bot ( http://www.followsite.com/bot.html )
	Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/27.0.1453.116 Safari/537.36 HubSpot Webcrawler - [email protected]
	Mozilla/5.0 (compatible; BlinkaCrawler/1.0; +http://www.blinka.jp/crawler/)
	Mozilla/5.0 (compatible; Qwantify/2.2w; +https://www.qwant.com/)/*
	findlinks/1.1.6-beta2 (+http://wortschatz.uni-leipzig.de/findlinks/)
	yacybot (/global; amd64 Linux 4.4.0-28-generic; java 1.8.0_91; Europe/pl) http://yacy.net/bot.html
	yacybot (/global; amd64 Windows Server 2008 R2 6.1; java 1.8.0_40; Europe/de) http://yacy.net/bot.html
	Mozilla/5.0 (compatible; 200PleaseBot/1.0; +http://www.200please.com/bot)
	WatchMouse/8.4.0.3 (http://watchmouse.com/ ; chzrh01.watchmouse.net)
	Iframely/0.8.8 (+http://iframely.com/;)
	Mozilla/5.0 (compatible; URLAppendBot/1.0; +http://www.profound.net/urlappendbot.html)
	LoadImpactRload/3.2.0 (Load Impact; http://loadimpact.com);
	RSSingBot (http://www.rssing.com)
	RADaR-Bot/Nutch-1.3 (http://radar-bot.com/)
	Ocelli/1.4 (http://www.globalspec.com/Ocelli)
	Mozilla/5.0 (compatible; BuzzSumo; +http://www.buzzsumo.com/bot.html)
	Mozilla/4.0 (compatible; IDwhois/2.0alpha-10102619; Linux; en-US; +http://idwhois.info/about)
	omgilibot/0.4 +http://omgili.com
	yacybot (freeworld/global; amd64 Windows 7 6.1; java 1.6.0_23; Europe/de) http://yacy.net/bot.html
	Y!J-BRW/1.0 crawler (http://help.yahoo.co.jp/help/jp/search/indexing/indexing-15.html)
	Mozilla/5.0 (compatible; AhrefsBot/2.0; +http://ahrefs.com/robot/)
	Mozilla/5.0 (compatible; VideoSurf_bot +http://www.videosurf.com/bot.html)
	Symfony Spider (http://symfony.com/spider)
	Mozilla/5.0 (compatible; Sosospider/2.0; +http://help.soso.com/webspider.htm)
	Mozilla/5.0 (compatible; OpenindexDeepSpider/Nutch-1.5-dev; +http://www.openindex.io/en/webmasters/spider.html)
	ADmantX Platform Semantic Analyzer - ADform - ADmantX Inc. - www.admantx.com - [email protected]
	Mozilla/5.0 (compatible; MJ12bot/v1.3.3; http://www.majestic12.co.uk/bot.php?+)
	cuwhois/1.0 (+http://www.cuwhois.com/)
	Mozilla/5.0 (compatible; memoryBot/1.20.311 +http://internetmemory.org/en/)
	Mozilla/4.0 (compatible; Vagabondo/4.0; webcrawler at wise-guys dot nl; http://webagent.wise-guys.nl/; http://www.wise-guys.nl/)
	yacybot (-global; amd64 Linux 3.16.0-4-amd64; java 1.7.0_75; Europe/en) http://yacy.net/bot.html
	Mozilla/5.0 (compatible; Dazoobot/1.0; http://dazoo.fr)
	Mozilla/5.0 (compatible; Semager/1.4c; +http://www.semager.de/blog/semager-bots/)
	EuripBot/1.1 (+http://www.eurip.com) GetRobots
	yacybot (freeworld/global; amd64 Linux 3.8.0-23-generic; java 1.6.0_27; Pacific/en) http://yacy.net/bot.html
	yacybot (amd64 Linux 2.6.24-23-generic; java 1.6.0_07; Europe/de) http://yacy.net/bot.html
	yacybot (freeworld/global; x86 Windows 2003 5.2; java 1.6.0_32; Europe/de) http://yacy.net/bot.html
	yacybot (freeworld/global; amd64 Linux 3.2.0-4-amd64; java 1.7.0_25; Europe/de) http://yacy.net/bot.html
	socialbm_bot http://spider.socialbm.net
	Mozilla/5.0 (compatible; 007ac9 Crawler; http://crawler.007ac9.net/)
	webinatorbot 1.0; +http://www.webinator.de
	WorldBrewBot/2.1 (+http://www.marketbrew.com/)
	BotOnParade, http://www.bots-on-para.de/bot.html
	ltx71 - (http://ltx71.com/)
	VSB-TUO/1.0 (+http://www.vsb.cz)
	yacybot (/global; amd64 Linux 3.13.0-85-generic; java 1.7.0_101; Europe/en) http://yacy.net/bot.html
	CopperEgg/RevealUptime/TokyoJP(aws)
	Mozilla/5.0 (compatible; etoolsbot/1.0; +https://etools.io/bots/)
	Comodo-Webinspector-Crawler 2.2.2, http://www.comodorobot.com
	Mozilla/5.0 (compatible; TWMBot/0.1; +http://thewebminer.com)
	Mozilla/5.0 (compatible; Plukkie/1.3; http://www.botje.com/plukkie.htm)
	Autonomy
	yacybot (i386 Linux 2.6.30-2-686; java 1.6.0_0; SystemV/en) http://yacy.net/bot.html
	Testomatobot/1.0 (Linux x86_64; +http://www.testomato.com/testomatobot) minicrawler/4.1.0
	CatchBot/1.0; +http://www.catchbot.com
	Sitedomain-Bot(Sitedomain-Bot 1.0, http://www.sitedomain.de/sitedomain-bot/)
	Mozilla/5.0 (compatible; Uptimebot/0.2.42; +http://www.uptime.com/uptimebot)
	yacybot (/global; amd64 Linux 3.2.0-4-amd64; java 1.7.0_03; Europe/en) http://yacy.net/bot.html
	SiteTruth.com site rating system
	SiteUptime.com
	yacybot (i386 Linux 2.6.24-23-generic; java 1.6.0_16; Europe/en) http://yacy.net/bot.html
	UrlTrends Ranking Updater/2.0
	yacybot (/global; amd64 Windows 7 6.1; java 1.7.0_55; Europe/de) http://yacy.net/bot.html
	yacybot (freeworld/global; amd64 Linux 3.5.0-27-generic; java 1.7.0_25; Europe/en) http://yacy.net/bot.html
	GarlikCrawler/1.1 (http://garlik.com/, [email protected])
	Mozilla/5.0 (compatible; ParsijooBot; +http://www.parsijoo.ir/; [email protected])
	Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.1 (KHTML, like Gecko) Chrome/21.0.1180.89 Safari/537.1; 360Spider(compatible; HaosouSpider; http://www.haosou.com/help/help_3_2.html)
	Willow Internet Crawler by Twotrees V2.1
	KrOWLer/0.1.0, matentzn at cs dot man dot ac dot uk
	SimplyFast.info Headers
	yacybot (freeworld/global; amd64 Linux 2.6.32-431.29.2.el6.x86_64; java 1.7.0_65; Europe/en) http://yacy.net/bot.html
	Mozilla/5.0 (compatible; YandexBlogs/0.99; robot; +http://yandex.com/bots)
	msnbot-media/1.1 (+http://search.msn.com/msnbot.htm)
	Mozilla/5.0 (Windows NT 6.1; WOW64; rv:46.0) Gecko/20100101 Firefox/46.0 PTST/278
	Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/534.57.2 (KHTML, like Gecko) Version/5.1.7 Safari/534.57.2 PTST/278
	gipo-crawler/Nutch-1.10 (Global Internet Policy Observatory crawler)
	Mozilla/5.0 (compatible; sukibot_heritrix/3.1.1 +http://suki.ling.helsinki.fi/eng/webmasters.html)
	PayPal IPN ( https://www.paypal.com/ipn )
	Mozilla/5.0 (compatible; KaloogaBot; http://kalooga.com/crawler)
	eCairn-Grabber/1.0 (+http://ecairn.com/grabber) curl/7.15
	Mozilla/5.0 (Windows NT 6.2; WOW64) AppleWebKit/537.15 (KHTML like Gecko; +http://www.diffbot.com) Chrome/24.0.1295.0 Safari/537.15
	ImageEngine/1.0
	DER-bot (+https://goo.gl/9ZibHG)
	yacybot (freeworld/global; amd64 Windows 7 6.1; java 1.7.0_60; Europe/de) http://yacy.net/bot.html
	Mozilla/5.0 (compatible; SemrushBot/0.96.3; +http://www.semrush.com/bot.html)
	Mozilla/5.0 (compatible; Exabot/3.0; +http://www.exabot.com/go/robot)
	Mozilla/5.0 (compatible; YandexDirect/3.0; +http://yandex.com/bots)
	Mozilla/5.0 (compatible; aiHitBot/2.9; +https://www.aihitdata.com/about)
	Linguee Bot (http://www.linguee.com/bot; [email protected])
	MetaGeneratorCrawler/1.3.9 (www.metagenerator.info)
	TelegramBot (like TwitterBot)
	Mozilla/5.0 (compatible; Siteliner/1.0; +http://www.siteliner.com/bot)
	calculon spider/0.1.0 mailto:[email protected]
	yacybot (/global; amd64 Linux 3.13.0-83-generic; java 1.7.0_95; Europe/de) http://yacy.net/bot.html
	PocketParser/2.0 (+https://getpocket.com/pocketparser_ua)
	montastic-monitor http://www.montastic.com
	Mozilla/5.0 (compatible; trovitBot 1.0; +http://www.trovit.com/bot.html)
	Mozilla/5.0 (Windows NT 6.1)+(compatible; SMTBot/1.0; +http://www.similartech.com/smtbot)
	Yaanb/1.5.001 (compatible; Win64;)
	Mozilla/5.0 (compatible; Crawlera/1.10.2; UID 70054)
	Mozilla/5.0 (compatible; spbot/2.0.2; +http://www.seoprofiler.com/bot/ )
	yacybot (freeworld-global; amd64 Linux 3.2.0-4-amd64; java 1.6.0_24; Europe/en) http://yacy.net/bot.html
	Mozilla/5.0 (compatible; ImageFetcher/5.6; +http://images.weserv.nl/)
	CopperEgg/RevealUptime/N.VirginiaUSA
	Huaweisymantecspider (compatible; MSIE 8.0; [email protected])
	OrangeCrawler/Nutch-1.0 ([email protected])
	CopperEgg/RevealUptime/AmsterdamHolland
	Mozilla/5.0 (compatible; SemrushBot-SI/0.97; +http://www.semrush.com/bot.html)
	Mozilla/5.0 (X11; Linux x86_64; GTmetrix https://gtmetrix.com/) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/50.0.2661.75 Safari/537.36
	Mozilla/5.0 (compatible; Genieo/1.0 http://www.genieo.com/webfilter.html) AppEngine-Google; (+http://code.google.com/appengine; appid: s~natmacdevice)
	yacybot (/global; amd64 Linux 3.2.41-042stab094.7; java 1.7.0_75; Europe/en) http://yacy.net/bot.html
	yacybot (x86 Windows Vista 6.1; java 1.6.0_13; Europe/de) http://yacy.net/bot.html
	SmabblerBot/1.0 (http://www.smabbler.com/)
	Mozilla/5.0 (compatible; grapeFX/0.9; [email protected]
	Mozilla/5.0 (compatible; SemrushBot/1~bl; +http://www.semrush.com/bot.html)
	MetaGeneratorCrawler/1.3.14 (www.metagenerator.info)
	oku-taka-lab-bot (Mozilla-compatible; [email protected]; http://lr-www.pi.titech.ac.jp/wp/)
	Mozilla/5.0 (Linux; Android 4.1.2; Galaxy Nexus Build/JZO54K; GTmetrix https://gtmetrix.com/) AppleWebKit/537.22 (KHTML, like Gecko) Chrome/26.0.1410.58 Mobile Safari/537.22
	Mozilla/5.0 (compatible; Feedage/2.0; +http://www.feedage.com/bot.php)
	yacybot (amd64 Linux 2.6.26-2-openvz-amd64; java 1.6.0_12; UTC/en) http://yacy.net/bot.html
	yacybot (-global; amd64 Linux 2.6.32-042stab116.1; java 1.7.0_79; Europe/en) http://yacy.net/bot.html
	Mozilla/5.0 (compatible; DKIMRepBot/1.0; +http://www.dkim-reputation.org)
	Mozilla/5.0 (Windows; U; Windows NT 5.1; zh-CN; rv:1.8.0.11) Firefox/1.5.0.11; 360Spider
	Mozilla/5.0 (compatible; pub-crawler; +http://wiki.github.com/bixo/bixo/bixocrawler; [email protected])
	mfibot/1.1 (http://www.mfisoft.ru/analyst/; <[email protected]>; en-RU)
	Visited by http://tools.geek-tools.org
	peerindex
	Mozilla/5.0 (compatible; janforman.com/2.4; +http://janforman.com/)
	http://www.amagit.com/
	yacybot (freeworld/global; amd64 Linux 2.6.32-40-server; java 1.6.0_20; Europe/en) http://yacy.net/bot.html
	Mozilla/5.0 (iPhone; CPU iPhone OS 8_3 like Mac OS X) AppleWebKit/537.36 (KHTML, like Gecko) Version/8.0 Mobile/12F70 Safari/600.1.4 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
	Mozilla/5.0 (compatible; NLNZ_IAHarvester2014 +https://natlib.govt.nz/publishers-and-authors/web-harvesting/domain-harvest)
	Mozilla/5.0 (compatible; YandexImageResizer/2.0; +http://yandex.com/bots)
	mahonie, neofonie search:robot/search:robot/0.0.1 (This is the MIA Bot - crawling for mia research project. If you feel unhappy and do not want to be visited by our crawler send an email to [email protected]; http://spider.neofonie.de; [email protected])
	Mozilla/5.0 (Linux; Android 5.0; Nexus 5 Build/LRX21O) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/46.0.2490.76 Mobile Safari/537.36 PTST/276
	Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; Trident/5.0; WOW64; Trident/5.0; BingPreview/1.0b)
	Mozilla/5.0 (compatible; OptimizationCrawler/0.2; +http://www.domainoptima.com/bot.html)
	Mozilla/5.0 (compatible; Cloudinary/1.0)
	rogerbot/1.0 (http://moz.com/help/pro/what-is-rogerbot-, [email protected])
	Iframely/0.9.8 (+http://iframely.com/;)
	yacybot (webportal-global; amd64 Linux 2.6.32-5-amd64; java 1.6.0_18; Europe/en) http://yacy.net/bot.html
	Interdose AntiSpamBot/2.0 (+http://www.piep.net)
	CheckMarkNetwork/1.0 (+http://www.checkmarknetwork.com/spider.html)
	Mozilla/5.0 (iPhone; CPU iPhone OS 8_3 like Mac OS X) AppleWebKit/537.36 (KHTML, like Gecko) Version/8.0 Mobile/12F70 Safari/600.1.4 (compatible; Google Search Console)
	rogerbot/1.0 (http://www.seomoz.org/dp/rogerbot, [email protected])
	findlinks/2.5 (+http://wortschatz.uni-leipzig.de/findlinks/)
	Mozilla/5.0 (compatible; MSIE 10.0; Windows NT 6.1; WOW64; Trident/6.0) PTST/277
	yacybot (/global; x86 Windows 2003 5.2; java 1.7.0_67; Europe/de) http://yacy.net/bot.html
	NeutrinoAPI/2.0.3
	Mozilla/5.0 (compatible; spbot/5.0.3; +http://OpenLinkProfiler.org/bot )
	Mozilla/5.0 (compatible; MJ12bot/v1.4.4; http://www.majestic12.co.uk/bot.php?+)
	Site24x7 Agent
	Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.2; WOW64; Trident/6.0; .NET4.0E; .NET4.0C; .NET CLR 3.5.30729; .NET CLR 2.0.50727; .NET CLR 3.0.30729) CrawlerProcess (http://www.PowerMapper.com) /5.7.704.0
	Mozilla/5.0 (compatible; Falconsbot; +http://iws.seu.edu.cn/services/falcons/contact_us.jsp)
	Super Monitoring
	Seznam-Zbozi-robot/3.0
	LinkedInBot/1.0 (compatible; Mozilla/5.0; Jakarta Commons-HttpClient/4.3 +http://www.linkedin.com)
	Mozilla/5.0 (compatible; EveryoneSocialBot/1.0; [email protected] http://everyonesocial.com/)
	Mozilla/5.0 (compatible; IstellaBot/1.01.18 +http://www.tiscali.it/)
	Mozilla/5.0 (compatible; ToutiaoSpider/1.0; http://web.toutiao.com/media_cooperation/;)
	Mozilla/5.0 (compatible; BegunAdvertising/3.0; +http://begun.ru/begun/technology/indexer/)
	Netcraft SSL Server Survey - contact [email protected]
	Mozilla/5.0 (compatible; Qualidator.com SiteAnalyzer 1.0;)
	Mozilla/5.0 (compatible; OrangeBot/2.0; [email protected])
	Mozilla/5.0 (compatible; CloudFlare-AlwaysOnline/1.0; +http://www.cloudflare.com/always-online)
	seebot/1.0.0 (http://www.seegnify.com/bot)
	Slackbot-LinkExpanding 1.0 (+https://api.slack.com/robots)
	Eurobot/1.1 (http://eurobot.ayell.eu)
	Mozilla/5.0 (compatible; IXEbot; +http://medialab.di.unipi.it/IXEbot.html)
	Mozilla/5.0 (GetLinkInfo.com - http://www.getlinkinfo.com)
	Mozilla/5.0 (compatible; spbot/4.0.5; +http://www.seoprofiler.com/bot )
	yacybot (freeworld/global; amd64 Windows 7 6.1; java 1.6.0_31; America/en) http://yacy.net/bot.html
	Mozilla/5.0 (Windows NT 6.1; WOW64; rv:48.0) Gecko/20100101 Firefox/48.0 PTST/314
	Mozilla/5.0 (compatible; AhrefsBot/4.0; +http://ahrefs.com/robot/)
	UXCrawlerBot
	Pixray-Seeker/2.0 (http://www.pixray.com/pixraybot; [email protected])
	Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/50.0.2661.94 Safari/537.36 PTST/276
	FAST-WebCrawler/3.7 (atw-crawler at fast dot no; http://fast.no/support/crawler.asp)
	yacybot (freeworld/global; amd64 Windows 10 10.0; java 1.8.0_60; Europe/de) http://yacy.net/bot.html
	Technoratibot/7.0
	Mozilla/4.0 (compatible; MSIE 5.01; Windows NT 5.0; DomainDB-1.1; http://domaindb.com/crawler/)
	Abrave v6.0 (http://robot.abrave.com)
	Testomatobot/1.0 (Linux x86_64; +http://www.testomato.com/testomatobot) minicrawler/4.0.0~beta11
	DowntimeDetector/1.0 (+http://downforeveryoneorjustme.com)
	Comodo Spider 1.1
	Link Valet Online 1.1
	yacybot (freeworld/global; amd64 Windows XP 5.2; java 1.7.0_04; America/en) http://yacy.net/bot.html
	li_viewer ([email protected])
	Mozilla/5.0 (compatible; Mail.RU/2.0c)
	Mozilla/5.0 (compatible; aiHitBot/1.0; +http://www.aihit.com/)
	Mozilla/5.0 (compatible; Uptimebot/0.2.19; +http://www.uptime.com/uptimebot)
	Mozilla/5.0 (compatible; YandexBot/3.0; +http://yandex.com/bots)
	Mozilla/5.0 (iPhone; CPU iPhone OS 6_0 like Mac OS X) AppleWebKit/536.26 (KHTML, like Gecko) Version/6.0 Mobile/10A5376e Safari/8536.25 ( compatible; CloudServerMarketSpider/1.0; +http://cloudservermarket.com/spider.html )
	yacybot (i386 Linux 2.6.28-13-generic; java 1.6.0_13; Europe/en) http://yacy.net/bot.html
	linguatools-bot/Nutch-1.6 (searching for translated pages; http://www.linguatools.de/linguatoolsbot.html; peter dot kolb at linguatools dot org)
	Mozilla/5.0 (compatible; Lippershey/5.2; +https://www.lipperhey.com/en/about/)
	NextGenSearchBot 69 (for information visit http://www.zoominfo.com/business/nextgensearchbot.html)
	Mozilla/5.0 (compatible; YandexWebmaster/2.0; +http://yandex.com/bots)
	Mozilla/5.0 (compatible; Baiduspider/2.0; +http://www.baidu.com/search/spider.html)
	yacybot (freeworld-global; amd64 Linux 3.2.0-35-generic; java 1.7.0_09; Europe/en) http://yacy.net/bot.html
	peerindex/0.1 (http://www.peerindex.com/; crawler AT peerindex DOT com)
	iqdb/0.2 (+http://iqdb.harry.lu/)
	Mozilla/5.0 (compatible; LuminateBot/1.0; +http://www.luminate.com/bot/)
	Mozilla/5.0 (X11; Linux x86_64) adbeat.com/policy AppleWebKit/537.21 (KHTML, like Gecko) Firefox/34.0 Safari/537.21
	AntBot/1.0 (http://www.ant.com)
	Mozilla/5.0 (compatible; Thumbshots.ru; +http://thumbshots.ru/bot) Firefox/3
	Mozilla/5.0 (compatible; value/1.1; +http://www.aihit.com/)
	Digg Deeper/v1 (http://digg.com/about)
	Elmer, the Thinglink ImageBot (http://www.thinglink.com/help/ThinglinkImageBot)
	rogerbot/1.0 (http://moz.com/help/pro/what-is-rogerbot-, [email protected])
	Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.1 (KHTML, like Gecko) Chrome/21.0.1180.89 Safari/537.1; http://spyonweb.com [email protected]
	BabalooSpider/1.3 (BabalooSpider; http://www.babaloo.si; [email protected])
	Mozilla/5.0 (compatible; dotSemantic/1.0; +http://www.dotsemantic.org)
	TinEye/1.1 (http://tineye.com/crawler.html)
	Mozilla/5.0 (compatible; SemrushBot/0.98~bl; +http://www.semrush.com/bot.html)
	CorpusCrawler 2.0.9 (http://corpora.fi.muni.cz/crawler/)
	aboutthedomain
	Cliqzbot/0.1 (+http://cliqz.com/company/cliqzbot)
	Mozilla/5.0 (compatible; Uptimebot/0.2.43; +http://www.uptime.com/uptimebot)
	LinqiaMetadataDownloaderBot/1.0 ([email protected])
	yacybot (freeworld/global; amd64 Linux 3.2.0-32-generic; java 1.6.0_24; Europe/en) http://yacy.net/bot.html
	NeumobBot/0.2.16 (+http://www.neumob.com/bot)
	Mozilla/5.0 (compatible; Mp3Bot/0.4; +http://mp3realm.org/mp3bot/)
	Miniflux (https://miniflux.net)
	Mozilla/5.0 (iPhone; CPU iPhone OS 8_3 like Mac OS X) AppleWebKit/537.36 (KHTML, like Gecko; Google Page Speed Insights) Version/8.0 Mobile/12F70 Safari/600.1.4
	Digincore crawler bot. See https://www.digincore.com/crawler.html for rules and instructions.
	Mozilla/5.0 (iPhone; CPU iPhone OS 7_0 like Mac OS X) AppleWebKit/537.51.1 (KHTML, like Gecko) Version/7.0 Mobile/11A465 Safari/9537.53 (compatible; adidxbot/2.0; http://www.bing.com/bingbot.htm)
	NerdyBot
	Nutch/2.2.1 (page scorer; http://integralads.com/site-indexing-policy/)
	Mozilla/5.0 (compatible; 008/0.83; http://www.80legs.com/webcrawler.html;) Gecko/2008032620
	Mozilla/4.0 (compatible; LinkMarket-Bot)
	SeoCheckBot (Seo-Check, http://www.kfsw.de/bot.html)
	Mozilla/5.0 (compatible; Whynder Magnet; http://whynder.com/magnet)
	IdeelaborPlagiaat/1
	x28-job-bot; +http://x28.ch/bot.html
	Mozilla/5.0 (compatible; spbot/4.0.7; +https://www.seoprofiler.com/bot )
	Mozilla/5.0 (compatible; RelateIQ Crawler/0.4; +www.relateiq.com; [email protected])
	WebRankSpider/1.37 (+http://ulm191.server4you.de/crawler/)
	Mozilla/5.0 (iPhone; U; CPU iPhone OS 10_0 like Mac OS X; en-us) AppleWebKit/602.1.38 (KHTML, like Gecko) Version/10.0 Mobile/14A5297c Safari/602.1 (compatible; Mediapartners-Google/2.1; +http://www.google.com/bot.html)
	LinkWalker/3.0 (http://www.brandprotect.com)
	Mozilla/5.0 (compatible; discobot/1.0; +http://discoveryengine.com/discobot.html)
	ia_archiver-web.archive.org
	Mozilla/5.0 (compatible; Online Domain Tools - SEO Checker/1.0; +http://online-domain-tools.com)
	Mozilla/5.0 (compatible; AhrefsBot/3.0; +http://ahrefs.com/robot/)
	Mozilla/5.0 (Macintosh; Intel Mac OS X 10_6_8) AppleWebKit/537.13+ (KHTML, like Gecko) Version/5.1.7 Safari/534.57.2; Shoppimon Analyzer (http://www.shoppimon.com/)
	Mozilla/5.0 (compatible; NewShareCounts.com/1.0; +http://newsharecounts.com/crawler)
	Mozilla/5.0 (compatible; Mozilla; +http://wiki.github.com/bixo/bixo/bixocrawler; [email protected])
	Mozilla/5.0 (compatible; CukBot; Not a spammer; http://www.companiesintheuk.co.uk/bot.html)
	yacybot (freeworld/global; amd64 Linux 3.1.10-1.9-default; java 1.6.0_24; Europe/de) http://yacy.net/bot.html
	Nuhk/2.4 (+http://www.neti.ee/cgi-bin/abi/otsing.html)
	elefent/Elefent 1.1 (Friendly web walker.; http://www.elefent.eu/; [email protected])
	HubSpot Links Crawler 1.0 http://www.hubspot.com/
	Mozilla/5.0 (compatible; Siteluxbot/1.0; +https://sitelux.co.uk/)
	CopperEgg/RevealUptime/AmsterdamNL(softlayer)
	Mozilla/5.0 (compatible; spbot/4.0.7; +http://OpenLinkProfiler.org/bot )
	Mediumbot-MetaTagFetcher/0.1 (+https://medium.com/)
	Mozilla/2.0 (compatible; Ask Jeeves/Teoma; +http://about.ask.com/en/docs/about/webmasters.shtml)
	Mozilla/5.0 (compatible; 008/0.85; http://www.80legs.com/webcrawler.html) Gecko/2008032620
	yacybot (/global; amd64 Linux 4.4.0ct; java 1.7.0_91; Europe/de) http://yacy.net/bot.html
	Mozilla/5.0 (compatible; Scrubby/2.2; http://www.scrubtheweb.com/)
	yacybot (/global; amd64 Linux 3.2.0-4-amd64; java 1.8.0_05; Europe/en) http://yacy.net/bot.html
	Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.2.18) Gecko/20110622 HeartRails_Capture/1.0.4 (+http://capture.heartrails.com/) Namoroka/3.6.18
	Mozilla/5.0 (X11; U; Linux i686; de; rv:1.9.0.1; compatible; iCjobs Stellenangebote Jobs; http://www.icjobs.de) Gecko/20100401 iCjobs/3.2.3
	KOCMOHABT (+http://www.KOCMOHABT.tk/) AppEngine-Google; (+http://code.google.com/appengine; appid: s~web-explorer)
	yacybot (freeworld/global; x86 Windows 7 6.1; java 1.6.0_31; Europe/de) http://yacy.net/bot.html
	Mozilla/5.0 (compatible; MSIE 10.0; Windows NT 6.2; WOW64; Trident/6.0) CrawlerProcess (http://www.PowerMapper.com) /5.21.754.0
	Mozilla/5.0 (compatible; Linux x86_64; Mail.RU_Bot/Robots; +http://go.mail.ru/help/robots)
	Mozilla/5.0 (compatible; Googlebot/2.1; +http://import.io)
	ownCloud Server Crawler
	Mozilla/5.0 (compatible; Qwantify/2.1n; +https://www.qwant.com/)/*
	LoadImpactRload/3.1.5 (Load Impact; http://loadimpact.com);
	Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 4.0; Girafabot; girafabot at girafa dot com; http://www.girafa.com)
	OmniExplorer_Bot/6.65a (+http://www.omni-explorer.com) WorldIndexer
	Mozilla/5.0 (compatible; SMTBot/1.0; +http://www.similartech.com/smtbot)
	anonymous coward/1.0 (a Storm-based crawler; https://github.com/DigitalPebble/storm-crawler; [email protected])
	Mozilla/5.0 (compatible; Uptimebot/1.0; +http://www.uptime.com/uptimebot)
	Mozilla/5.0 (compatible; FoundSeoTool/1.0; +http://www.found.co.uk/seo-tool/)
	findlinks/2.1.5 (+http://wortschatz.uni-leipzig.de/findlinks/)
	yacybot (freeworld/global; amd64 Windows NT (unknown) 6.2; java 1.7.0_05; Africa/en) http://yacy.net/bot.html
	Mozilla/5.0 (compatible; memorybot/1.20.71 +http://archivethe.net/en/index.php/about/internet_memory1 on behalf of DNB)
	ownCloud News/5.3.6 (+https://owncloud.org/; 1 subscriber;)
	YottaBot Spider/Nutch-2.3
	Mozilla/5.0 (compatible; WeViKaBot/1.0; +http://www.wevika.de/)
	ThumbShots-Bot (+http://thumbshots.in/bot.html)
	Mnogosearch-3.1.21
	Favicon downloader (+https://favico.be/)
	Mozilla/5.0 (compatible; Page2RSS/0.7; +http://page2rss.com/)
	ExB Language Crawler 2.1.2 (+http://www.exb.de/crawler)
	PrivateSearch/0.1.0 (Polite Nutch Crawler; grierforensics.com)
	nodestackbot/0.1 ([email protected] http://nodestack.com/bot.html)
	iskanie (+http://www.iskanie.com)
	yacybot (/global; amd64 Windows 7 6.1; java 1.7.0_67; Europe/de) http://yacy.net/bot.html
	Mozilla/5.0 (compatible; Dataprovider Site Explorer; http://www.dataprovider.com/)
	Mozilla/5.0 (compatible; akula/12.0rc-2; +http://k311.fd.cvut.cz/)
	Mozilla/5.0 (compatible; bnf.fr_bot; +http://www.bnf.fr/fr/outils/a.dl_web_capture_robot.html)
	Mozilla/5.0 (iPhone; CPU iPhone OS 6_0 like Mac OS X) AppleWebKit/536.26 (KHTML, like Gecko) Version/6.0 Mobile/10A5376e Safari/8536.25 (compatible; Googlebot-Mobile/2.1; +http://www.google.com/bot.html)
	Mozilla/5.0 (compatible; MSIE 10.0; Windows NT 6.1; WOW64; Trident/6.0) PTST/284
	Mozilla/5.0 (compatible; Infohelfer/1.3.3; +http://www.infohelfer.de/crawler.php)
	www.deadlinkchecker.com XMLHTTP/1.0
	Mozilla/5.0 (compatible; TweetmemeBot/3.0; +http://tweetmeme.com/)
	Chrome/49.0.2623.87 (compatible; Wappalyzer; +https://github.com/AliasIO/Wappalyzer)
	Mozilla/5.0 (compatible; MSIE 10.0; Windows NT 6.2; WOW64; Trident/6.0) CrawlerProcess (http://www.PowerMapper.com) /5.11.740.0
	XING-contenttabreceiver/2.0
	Mozilla/5.0 (iPhone; CPU iPhone OS 6_0 like Mac OS X) AppleWebKit/536.26 (KHTML, like Gecko) Version/6.0 Mobile/10A5376e Safari/8536.25 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
	Mozilla/5.0 (compatible; MojeekBot/0.2; http://www.mojeek.com/bot.html#relaunch)
	Pingoscope
	Mozilla/5.0 (compatible; um-LN/1.0; mailto: [email protected])
	Mozilla/5.0 (compatible; Pagespeed/1.1 Fetcher; +http://www.pagespeed.de)
	HTTP-Header-Abfrage/1.0 (http://www.internalscripts.de/werkzeuge/http-header-abfrage.php)
	Mozilla/5.0 (compatible; 80bot/0.71; http://www.80legs.com/spider.html;) Gecko/2008032620
	LoadImpactRload/3.0.3 (Load Impact; http://loadimpact.com);
	Mozilla/5.0 (compatible; Peepowbot/1.0; +http://www.peepow.com/bot.php)
	yacybot (freeworld/global; amd64 Linux 2.6.32-5-amd64; java 1.6.0_26; Atlantic/en) http://yacy.net/bot.html
	WatchMouse/8.4.0.3 (http://watchmouse.com/ ; grath01.watchmouse.net)
	Mozilla/5.0 (compatible; +http://servernfo.com/)
	Mozilla/5.0 (compatible; special_archiver/3.1.1 +http://www.archive.org/details/archive.org_bot)
	CopperEgg/RevealUptime/NewarkNJUSA
	Mozilla/5.0 (compatible; archive.org_bot/3.3.0 +http://www.archive.org/details/archive.org_bot)
	Mozilla/5.0 (compatible; Plukkie/1.1; http://www.botje.com/plukkie.htm)
	Mozilla/5.0 (compatible; musobot/1.0; [email protected]; +http://www.muso.com)
	yacybot (freeworld/global; amd64 Linux 3.10.17-gentooxen; java 1.6.0_31; Europe/nb) http://yacy.net/bot.html
	ResponseCodeTest/1.1
	Mozilla/5.0 (compatible; MSIE 10.0; Windows NT 6.1; Trident/6.0) SiteCheck-sitecrawl by Siteimprove.com
	Acoon v4.9.5 (www.acoon.de)
	Mozilla/5.0 (Windows NT 6.1; WOW64) SkypeUriPreview Preview/0.5
	SiteGuardian/2.0 (Internet Monitoring)
	Mozilla/5.0 (compatible; DCPbot/1.4; +http://domains.checkparams.com/)
	BoardReader Favicon Fetcher /1.0 [email protected]
	baypup/colbert (Baypup; http://www.baypup.com/webmasters; [email protected])
	Mozilla/5.0 (compatible; MonTools.com)
	yacybot (freeworld/global; amd64 Linux 3.2.0-4-amd64; java 1.7.0_03; Etc/de) http://yacy.net/bot.html
	Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/534.51 (KHTML, like Gecko; Google Web Preview) Chrome/12.0.742 Safari/534.51
	linkapediabot (+http://www.linkapedia.com)
	Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.7.12; ips-agent) Gecko/20050922 Fedora/1.0.7-1.1.fc4 Firefox/1.0.7
	BCKLINKS 1.0
	CorpusCrawler 2.0.21 (http://corpora.fi.muni.cz/crawler/);Project:CzCorpus
	Mozilla/5.0 (compatible; Yeti/1.1; +http://help.naver.com/support/robots.html)
	Mozilla/5.0 (compatible; pingbot/2.0; +http://www.pingdom.com/)
	Mozilla/5.0 (compatible; memoryBot/1.20.275 +http://internetmemory.org/en/)
	Mozilla/5.0 (compatible; spbot/1.1; +http://www.seoprofiler.com/bot/ )
	yacybot (/global; amd64 Linux 3.13.0-88-generic; java 1.7.0_101; Europe/en) http://yacy.net/bot.html
	BlogSearch/2 +http://www.icerocket.com/
	Mozilla/5.0 (compatible; LoadTimeBot/0.81; +http://www.load-time.com/bot.html)
	Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9) Gecko/2008052906 Firefox/3.0 (BrowserSpyBot/1.0) (http://browserspy.dk/)
	Mozilla/5.0 (compatible; MojeekBot/0.6; +https://www.mojeek.com/bot.html)
	CopperEgg/RevealUptime/NewarkNJ(linode)
	Automattic Analytics Crawler/0.2; http://wordpress.com/crawler/
	larbin_2.6.3 [email protected]
	HatenaScreenshot
	CopperEgg/RevealUptime/LondonUK
	Mozilla/5.0 (compatible; MojeekBot/0.6; http://www.mojeek.com/bot.html)
	Mozilla/5.0 (compatible; evc-batch/2.0.20160902205649)
	yacybot (freeworld/global; amd64 Linux 3.2.0-23-generic; java 1.6.0_24; Europe/en) http://yacy.net/bot.html
	WordPress.com;
	Mozilla/5.0 (Windows NT 6.1; rv:6.0) Gecko/20110814 Firefox/6.0 Google (+https://developers.google.com/+/web/snippet/)
	Mozilla/5.0 (compatible; ScoutJet; http://www.scoutjet.com/)
	Mozilla/5.0 (compatible; Uptimebot/0.1.97; +http://www.uptime.com/uptimebot)
	WebTarantula.com Crawler
	WatchMouse/8.4.0.3 (http://watchmouse.com/ ; itmil01.watchmouse.net)
	LinqiaScrapeBot/1.0 ([email protected])
	Orbiter (+http://www.dailyorbit.com/bot.htm)
	Mozilla/5.0 (compatible; janforman.com/2.5; +http://janforman.com/)
	InternetSeer.com
	Mozilla/5.0 (compatible; aiHitBot/2.9; +http://www.aihitdata.com/about)
	Mozilla/5.0 (compatible; WebNL; +http://www.web.nl/webmasters/spider.html; [email protected])
	coccoc/1.0 (http://help.coccoc.vn/)
	Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/53.0.2785.116 Safari/537.36 PTST/314
	Mozilla/5.0 (compatible; Googlebot/2.1; startmebot/1.0; +https://start.me/bot)
	Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 5.2; Trident/4.0; .NET CLR 1.1.4322; .NET CLR 2.0.50727; .NET CLR 3.0.4506.2152; .NET CLR 3.5.30729; .NET4.0C; .NET4.0E; PTST 2.386)
	Mozilla/5.0 (compatible; PrivacyAwareBot/1.1; +http://www.privacyaware.org)
	Mozilla/5.0 (compatible; idmarch Automatic.beta/1.4; +http://www.idmarch.org/bot.html)
	Mozilla/5.0 (compatible; tldstat.com/Bot; +http://tldstat.com/bot.html)
	yacybot (/global; amd64 Linux 3.16.0-4-amd64; java 1.7.0_111; Europe/de) http://yacy.net/bot.html
	Mozilla/5.0 (compatible; Crawlera/1.10.2; UID 84846)
	Mozilla/5.0 (compatible; spbot/4.0a; +http://www.seoprofiler.com/bot )
	Mozilla/5.0 (Windows NT 6.1; Win64; x64; +http://www.komodia.com/newwiki/index.php/URL_server_crawler) KomodiaBot/1.0
	WebarooBot (Webaroo Bot; http://64.124.122.252/feedback.html)
	OOZBOT/0.20 ( -- ; http://www.setooz.com/oozbot.html ; agentname at setooz dot_com )
	Mozilla/5.0 (compatible; memoryBot/1.21.16 +http://internetmemory.org/en/)
	FAST-WebCrawler/3.8
	Googlebot-Image/1.0
	enlle punto com/Nutch-1.9
	OpenAcoon v4.10.5 (www.openacoon.de)
	yacybot (freeworld/global; amd64 Linux 3.0.0-21-generic; java 1.6.0_26; America/en) http://yacy.net/bot.html
	Microsearch.ru Bot 1.3;http://microsearch.ru/webmasters
	Mozilla/5.0 (compatible; Google-Site-Verification/1.0)
	Mozilla/5.0 (compatible; Qwantify/2.3w; +https://www.qwant.com/)/2.3w
	rogerbot/1.1 (http://moz.com/help/guides/search-overview/crawl-diagnostics#more-help, [email protected])
	HubSpot Connect 2.0 (http://dev.hubspot.com/)
	Robozilla/1.0
	Mozilla/5.0 (compatible; spbot/2.0.3; +http://www.seoprofiler.com/bot/ )
	Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/53.0.2785.116 Safari/537.36 PTST/314
	facebookexternalhit/1.1
	PercolateCrawler/3.1.30 ([email protected])
	Mozilla/5.0 (compatible; Nmap Scripting Engine; https://nmap.org/book/nse.html)
	Mozilla/5.0 (compatible; pdffillerbot/1.0; +http://www.pdffiller.com/en/support.htm)
	fastbot crawler beta 4.0 (+http://www.fastbot.de)
	SafeDNS search bot/Nutch-1.9 (https://www.safedns.com/searchbot; support [at] safedns [dot] com)
	Mozilla/5.0 (Nekstbot; http://nekst.ipipan.waw.pl/nekstbot/)
	yacybot (freeworld/global; amd64 Linux 2.6.32-5-amd64; java 1.6.0_18; US/en) http://yacy.net/bot.html
	yacybot (freeworld/global; amd64 Linux 3.0.0-12-generic; java 1.6.0_26; Europe/en) http://yacy.net/bot.html
	coccoc/1.0 ()
	larbin_2.6.3 ([email protected])
	yacybot (i386 Linux 2.6.26-2-686; java 1.6.0_0; Europe/de) http://yacy.net/bot.html
	Mozilla/5.0 (Windows NT 6.1; compatible; BDCbot/1.0; +http://ecommerce.bigdatacorp.com.br/faq.aspx) AppleWebKit/535.1 (KHTML, like Gecko) Chrome/13.0.782.107 Safari/535.1
	yacybot (amd64 Linux 2.6.31-22-server; java 1.6.0_18; Asia/en) http://yacy.net/bot.html
	yacybot (/global; amd64 Windows Server 2012 6.2; java 1.8.0_91; Europe/de) http://yacy.net/bot.html
	Mozilla/5.0 (compatible; Netseer crawler/2.0; +http://www.netseer.com/crawler.html; [email protected])
	Mozilla/5.0 (compatible; heritrix/3.1.1 +http://www.mixdata.com)
	Scrapy/0.25.1 (+http://scrapy.org)
	Mozilla/5.0 (compatible; Scopia Crawler 1.2; +http://www.scopia.co)
	cn_ab_viewer ([email protected])
	Mozilla/5.0 (Windows NT 6.3; Trident/7.0; rv 11.0) like Gecko (compatible; Zombiebot/2.1; +http://www.zombiedomain.net/robot/)
	Mozilla/5.0 (compatible; Uptimebot/0.1.38; +http://www.uptime.com/uptimebot)
	yacybot (freeworld/global; amd64 Windows 7 6.1; java 1.7.0_25; Europe/de) http://yacy.net/bot.html
	Python-urllib/2.7 (+http://sketch.juls.savba.sk/aranea_about)
	Mozilla/5.0 (compatible; spbot/4.0; +http://www.seoprofiler.com/bot )
	yacybot (-global; amd64 Windows 8 6.2; java 1.7.0_55; Europe/de) http://yacy.net/bot.html
	yacybot (/global; amd64 Linux 3.2.41-042stab094.7; java 1.7.0_79; Europe/en) http://yacy.net/bot.html
	Mozilla/5.0 (compatible; UASlinkChecker/1.0; +http://user-agent-string.info/UASlinkChecker)
	Y!J-BRO/YFSJ crawler (compatible; Mozilla 4.0; MSIE 5.5; http://help.yahoo.co.jp/help/jp/search/indexing/indexing-15.html; YahooFeedSeekerJp/2.0)
	yacybot (freeworld/global; amd64 Linux 3.10.15-1-MANJARO; java 1.7.0_40; Asia/en) http://yacy.net/bot.html
	Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.2; WOW64; Trident/6.0; .NET4.0E; .NET4.0C; .NET CLR 3.5.30729; .NET CLR 2.0.50727; .NET CLR 3.0.30729) CrawlerProcess (http://www.PowerMapper.com) /5.9.726.0
	Scrubby/2.1 (http://www.scrubtheweb.com/)
	Mozilla/5.0 (compatible; JobKeresoBot; +http://www.kozvetlen-allasok.hu/help.jsp; [email protected]; 5.0
	yacybot (/global; amd64 Windows 7 6.1; java 1.8.0_66; Europe/de) http://yacy.net/bot.html
	Mozilla/5.0 (X11; Linux x86_64 UXCrawlerBot http://zium.co)
	Slackbot-LinkExpanding (+https://api.slack.com/robots)
	yacybot (freeworld/global; amd64 Linux 3.12.15-gentoo; java 1.7.0_55; Europe/en) http://yacy.net/bot.html
	Mozilla/5.0 (compatible; LinkpadBot/1.07; +http://www.linkpad.ru)
	Mozilla/5.0 (compatible; ZB-1; +http://www.zeerch.com/bot.php)
	Mozilla/5.0 (compatible; WbSrch/1.1 +http://wbsrch.com)
	WeSEE:Search
	Mozilla/5.0 (compatible; DCPbot/1.1; +http://domains.checkparams.com/)
	yacybot (webportal-global; amd64 Linux 2.6.32-5-amd64; java 1.6.0_45; Europe/en) http://yacy.net/bot.html
	WordPress.com mShots; http://support.wordpress.com/contact/
	Mozilla/4.0 (compatible; MSIE 5.01; Windows NT 5.0; FeedFinder-2.0; http://bloggz.se/crawler)
	Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.6; en-US; rv:1.9.2) Gecko/20100115 Firefox/3.6 (FlipboardProxy/0.0.5; +http://flipboard.com/browserproxy)
	Mozilla/5.0 (compatible; coccocbot-image/1.0; +http://help.coccoc.com/searchengine)
	Mozilla/5.0 (X11; Linux x86_64; GTmetrix http://gtmetrix.com/) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/31.0.1650.63 Safari/537.36
	https_page_test ([email protected])
	Mozilla/5.0 (Windows; U; Windows NT 5.1; zh-CN; rv:1.8.0.11) Firefox/1.5.0.11 360Spider;
	yacybot (freeworld/global; amd64 Linux 3.2.0-2-amd64; java 1.6.0_24; Europe/en) http://yacy.net/bot.html
	Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_2) AppleWebKit/600.1.4 (KHTML, like Gecko) Safari/600.1.4 (compatible; Princetonbot/1.0; +http://http://tigress-web.princeton.edu/~fy/bot.html)
	Mozilla/5.0 (Windows; U; Windows NT 5.1; zh-CN; rv:1.8.0.11) Gecko/20070312 Firefox/1.5.0.11; 360Spider
	KeywordDensityRobot/0.8 (http://www.seocentro.com/tools/search-engines/keyword-density.html)
	Mozilla/5.0 (compatible; aiHitBot-BP/1.1; +http://www.aihit.com/)
	findlinks/1.1.6-beta4 (+http://wortschatz.uni-leipzig.de/findlinks/)
	Mozilla/5.0 (compatible; Butterfly/1.0; +http://labs.topsy.com/butterfly/) Gecko/2009032608 Firefox/3.0.8
	CopperEgg/RevealUptime/FremontCA(linode)
	Mozilla/5.0 (compatible; NTENTbot; +http://www.ntent.com/ntentbot)
	link_thumbnailer
	KeywordDensityRobot/0.9 (http://www.seocentro.com/tools/search-engines/keyword-density.html)
	Mozilla/5.0 (compatible;WI Job Roboter Spider Version 3;+http://www.webintegration.at)
	Mozilla/5.0 (compatible; Mail.RU/2.0)
	CopperEgg/RevealUptime/N.California(aws)
	Searchie Bot/1.0 (a Storm-based crawler; https://www.searchie.org; [email protected])
	^Nail (http://CaretNail.com)
	MarketBrewBot/2.1 (+http://www.marketbrew.com/)
	Updownerbot (+http://www.updowner.com/bot)
	yacybot (-global; amd64 Windows 8.1 6.3; java 1.7.0_55; Europe/de) http://yacy.net/bot.html
	Yanga WorldSearch Bot v1.1/beta (http://www.yanga.co.uk/)
	Mozilla/5.0 (compatible; bixolabs/1.0; +http://bixolabs.com/crawler/general; [email protected])
	Mozilla/5.0 (compatible; Scopia Crawler 2.0; +http://www.scopia.co)
	asafaweb.com
	KrOWLer/0.0.1, matentzn at cs dot man dot ac dot uk
	Mozilla/5.0 (compatible; alexa site audit/1.0; +http://www.alexa.com/help/webmasters; [email protected])
	findlinks/1.0 (+http://wortschatz.uni-leipzig.de/findlinks/)
	SygolBot http://www.sygol.com
	voltron
	Mozilla/5.0 (compatible; uMBot-FC/1.0; mailto: [email protected])
	Comodo-Webinspector-Crawler 2.2.1, http://www.comodorobot.com
	Superarama.com-Tarama-Botu-v.0.1
	yacybot (freeworld/global; amd64 Linux 3.2.0-3-amd64; java 1.6.0_24; Europe/en) http://yacy.net/bot.html
	Mozilla/5.0 (compatible; SSLBot/1.0; http://www.sslstats.com/sslbot)
	PagesInventory.com (robot http://www.pagesinvenotry.com)
	Mozilla/5.0 (iPhone; U; CPU iPhone OS 4_3 like Mac OS X; en-US) adbeat.com/policy AppleWebKit/533.17.9 (KHTML, like Gecko) Version/5.0.2 Mobile/8F190 Safari/6533.18.5
	WordPress.com Reader
	Mozilla/5.0 (compatible; SEOkicks-Robot +http://www.seokicks.de/robot.html)
	Woko 3.0
	Gigabot/3.0 (http://www.gigablast.com/spider.html)
	ichiro/4.0 (http://help.goo.ne.jp/door/crawler.html)
	bl.uk_lddc_bot/3.1.1 (+http://www.bl.uk/aboutus/legaldeposit/websites/websites/faqswebmaster/index.html)
	Urlstat (http://bit.ly/1dJuuzs)
	yacybot (/global; amd64 Windows 8.1 6.3; java 1.7.0_55; America/ru) http://yacy.net/bot.html
	Bad Neighborhood Header Detector (http://www.bad-neighborhood.com/header_detector.php)
	Mozilla/5.0 (compatible; GeliyooBot/1.0beta; +http://www.geliyoo.com/)
	NalezenCzBot/1.0 (http://www.nalezen.cz/about-crawler)
	WeSEE_Bot:we_help_monitize_your_site (http://www.wesee.com/bot/)
	yacybot (freeworld/global; x86 Windows XP 5.1; java 1.7.0_55; Asia/en) http://yacy.net/bot.html
	studylib download bot, see http://studylib.es/public/bot for additional info
	SemrushBot/0.91
	Jabse.com Crawler v.1.0 www.jabse.com/crawler.php//imagecrawler
	Mozilla/4.0 (compatible; MSIE 5.0; Windows NT; Girafabot; girafabot at girafa dot com; http://www.girafa.com)
	BizwikiBot/0.3 (http://www.bizwiki.com/bizwikibot.htm)
	Mozilla/5.0 (compatible; BIXOCRAWLER; +http://wiki.github.com/bixo/bixo/bixocrawler; [email protected])
	PostPost/1.0 (+http://postpost.com/crawlers)
	Mozilla/5.0 (nomore404.com robot/1.1; +https://nomore404.com/)
	Riddler (http://riddler.io/about)
	Mozilla/5.0 (compatible; socketcrawler; http://nlp.fi.muni.cz/projects/biwec/)
	internetVista monitor (Mozilla compatible)
	2Bone_LinkChecker/1.0 libwww-perl/6.03
	MaxPointCrawler/Nutch-1.1 (maxpoint.crawler at maxpointinteractive dot com)
	DomainStatsBot/1.0 (http://domainstats.io/our-bot)
	KDDI-CA31 UP.Browser/6.2.0.7.3.129 (GUI) MMP/2.0 (compatible; ichiro/mobile goo;+http://search.goo.ne.jp/option/use/sub4/sub4-1/)
	Mergadobot/3.0.2 (+http://mergado.cz)
	DoCoMo/2.0 P900i(c100;TB;W24H11) (compatible; ichiro/mobile goo; +http://search.goo.ne.jp/option/use/sub4/sub4-1/)
	SalesIntelligent/v1.1
	SiteAnalyzerBot
	DoCoMo/2.0 P900i(c100;TB;W24H11) (compatible; ichiro/mobile goo; +http://help.goo.ne.jp/help/article/1142/)
	yacybot (/global; i386 Linux 3.16.3-gentoo; java 1.7.0_55; Australia/en) http://yacy.net/bot.html
	Mozilla/5.0 (compatible; Linux x86_64; Mail.RU_Bot/Img/2.0; +http://go.mail.ru/help/robots)
	Cliqz Bot (+http://www.cliqz.com)
	SalesIntelligent (+https://www.salesintelligent.com)
	Mozilla/5.0 (compatible; memoryBot/1.20.243 +http://internetmemory.org/en/)
	gimme60 (Gimme60 Store ID Bot; gimme60.com)
	Orbiter/1.0 (http://dailyorbit.com/)
	msnbot/1.1 (+http://search.msn.com/msnbot.htm)
	Mozilla/5.0 (compatible; memoryBot/1.20.241 +http://internetmemory.org/en/)
	Mozilla/5.0 (compatible; akula/k311; +http://k311.fd.cvut.cz/)
	Irokez.cz monitoring v1.2 - (http://www.irokez.cz, Irokez.cz, crawl)
	CCBot/2.0 (http://commoncrawl.org/faq/)
	Mozilla/5.0 (compatible; openstat.ru/Bot)
	Mozilla/5.0 (compatible; GigaBot/1.0; +http://www.gigablast.com/ )
	Readability/6a54d4 - http://readability.com/about/
	percbotspider <[email protected]>
	Mozilla/5.0 (compatible; spbot/1.0; +http://www.seoprofiler.com/bot/ )
	HatenaScreenshot/1.0 (checker)
	Mozilla/5.0 (compatible; AcoonBot/4.12.1; +http://www.acoon.de/robot.asp)
	CRAZYWEBCRAWLER 0.9.2, http://www.crazywebcrawler.com
	Mozilla/5.0 (compatible; Online Domain Tools - Online Sitemap Generator/1.1; +http://sitemap-generator.online-domain-tools.com)
	Mozilla/5.0 (compatible; Google-Structured-Data-Testing-Tool +https://search.google.com/structured-data/testing-tool)
	OnPageBot (compatible; Googlebot 2.1; +https://bot.onpage.org/)
	Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/48.0.2564.103 Safari/537.36 Yandex.Translate
	Sogou web spider/4.0
	Mozilla/5.0 (compatible; YandexMetrika/2.0; +http://yandex.com/bots DEV)
	LinkWalker/2.0
	SeoCheck (FischerNetzDesign Seo Checker, [email protected])
	yacybot (freeworld/global; amd64 Linux 3.2.13-1-ARCH; java 1.7.0_03-icedtea; Europe/fr) http://yacy.net/bot.html
	SiteCheck - http://downforeveryoneorjustme.com
	Robots_Tester_http_www.searchenginepromotionhelp.com
	CRAZYWEBCRAWLER 0.9.4, http://www.crazywebcrawler.com
	Mozilla/5.0 (compatible; heritrix/3.1.1 +http://places.tomtom.com/crawlerinfo)
	Mozilla/5.0 (Windows; U; Windows NT 5.0; en-US; rv:1.7.10) Gecko/20050716 Thunderbird/1.0.6 - WebCrawler http://cognitiveseo.com/bot.html
	Mozilla/5.0 (compatible; Cliqzbot/1.0 +http://cliqz.com/company/cliqzbot)
	yacybot (/global; amd64 Linux 4.2.0-gentoo-r1; java 1.8.0_45; Australia/en) http://yacy.net/bot.html
	DomainsDB.net MetaCrawler v.0.9.7c (http://domainsdb.net/)
	Mozilla/4.0 (compatible; MSIE 5.5; Windows NT 5.0; mowser; http://www.mowser.com)
	www.probethenet.com scanner
	Mozilla/5.0 (compatible; Crawlera/1.10.2; UID 6573)
	LCC (+http://corpora.informatik.uni-leipzig.de/crawler_faq.html)
	Imagga Downloader/v0.14.3
	SeopultContentAnalyzer/1.0
	Mozilla/5.0 (compatible; +http://www.hosterstats.com/newgtld-websurvey.php)
	Mozilla/5.0 (compatible; GoogleDocs; apps-spreadsheets; +http://docs.google.com)
	Mozilla/5.0 (compatible;acapbot/0.1)
	Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1;Alibaba.Security.Heimdall)
	HTTPClient/1.0 (2.8.0, ruby 2.0.0 (2014-05-08))
	Jetty/9.3.z-SNAPSHOT
	Jetty/9.3.z-SNAPSHOT, SMUrlExpander
	Jetty/9.3.z-SNAPSHOT, SMViewer
	Liferea/0.x.x (Linux; en_US.UTF-8; http://liferea.sf.net/)
	SynHttpClient-Built/5825
	asynchttp
	SMUrlExpander
	KickFire
	HAAARTLAND http client
	PEAR HTTPRequest class ( http://pear.php.net/ )
	Mozilla/4.0 (vBSEO; http://www.vbseo.com)
	HAARTLAND http client
	JS-Kit URL Resolver http://js-kit.com/
	http://www.checkprivacy.or.kr:6600/RS/PRIVACY_ENFAQ.jsp
	Kml-Google; (+http://code.google.com/apis/kml) gzip
	gvfs/1.20.3
	request.js
	Microsoft URL Control - 6.01.9782
	git/2.1.4
	IPS/1.0
	OxfordCloudService/1.0
	Mozilla/5.0 (compatible; informatique at mixdata dot com/0.6; +http://t.co/GSRLLKex24; informatique at mixdata dot com)
	Mozilla/5.0 (compatible; Feedwind/2.0; +http://feed.mikle.com/support/description/)
	Mozilla/5.0 (compatible; Feedwind/3.0; +http://feed.mikle.com/support/description/)
	Mozilla/5.0 (Macintosh; Intel Mac OS X 1068) AppleWebKit/534.50 (KHTML like Gecko) Version/5.1 Instapaper/4.0 (+http://www.instapaper.com/)
	Mozilla/5.0 (compatible; Keywords Research; http://www.check-domains.com/keyword-research/keywords-adwords.php)
	CAAM/0.1 (CAAM; caam dot crwlr at gmail dot com)
	i2kconnect/"1.7.0"
	resolver/0.1.12 (+http://github.com/andris9/resolver)
	DangDang
	Mozilla/5.0 (compatible; Tetrahedron/1.0; +https://compute.info/tet)
	PritTorrent/0.1
	FeedBunch/1.0 (+http://www.feedbunch.com)
	Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.21 (KHTML, like Gecko) www.HomepageScreenshot.de - Free ScreenShotService/1.0 Safari/537.21
	Mozilla/5.0 (compatible; Licorne Image Snapshot 1.0; +http://www.jednorozec.cz)
	Pinterest/0.2 (+https://www.pinterest.com/bot.html)
	Pinterest/0.1 (+https://www.pinterest.com/bot.html)
	Mozilla/5.0 (iplabel; Windows NT 6.3; WOW64; rv:33.0) Gecko/20100101 Firefox/33.0
	Mozilla/5.0 zgrab/0.x
	Zend_Http_Client
	Zend\Http\Client
	Zao/0.1 (http://www.kototoi.org/zao/)
	YOURLS v1.7.1 +http://yourls.org/ (running on http://flnx.co)
	yoogliFetchAgent/0.1
	Yo-yo
	Mozilla/5.0 (compatible; Yahoo Link Preview; https://help.yahoo.com/kb/mail/yahoo-link-preview-SLN23615.html)
	Mozilla/5.0 (compatible; Yahoo Ad monitoring; https://help.yahoo.com/kb/yahoo-ad-monitoring-SLN24857.html)
	Mozilla/5.0 (iPhone; CPU iPhone OS 7_1 like Mac OS X) AppleWebKit (KHTML, like Gecko) Mobile (compatible; Yahoo Ad monitoring; https://help.yahoo.com/kb/yahoo-ad-monitoring-SLN24857.html)
	xpymep.exe
	xpymep2.exe
	xpymep3.exe
	Xenu Link Sleuth 1.2b
	Xenu Link Sleuth 1.2c
	Xenu Link Sleuth 1.2d
	Xenu Link Sleuth 1.2e
	Xenu Link Sleuth 1.2f
	Xenu Link Sleuth 1.2g
	Xenu Link Sleuth 1.2h
	Xenu Link Sleuth 1.2i
	Xenu Link Sleuth/1.3.7
	Xenu Link Sleuth/1.3.8
	Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/40.0.2214.93 Safari/537.36 XaxisSemanticsClassifier/1.0 http://crystalsemantics.com
	WWW-Mechanize/1.73
	WordPress/4.0.1; http://bestcomp.hol.es
	WordPress/4.3.3; http://homebusinessreviews.online
	WordPress/4.3.3; http://legithomejobs.club
	WordPress/4.4.2; http://cheapautoinsuranceinva.com
	WordPress/4.4.2; http://citycollegeinmiami.com
	WordPress/4.4.2; http://cookingideabook.com
	WordPress/4.4.2; http://oldmanhealth.com
	WordPress/4.4.2; http://sapcrmarchitecture.com
	WordPress/4.4.2; http://www.spar3d.com
	WordPress/4.5.2; http://1.we-careu.xyz
	WordPress/x.x.x.x PHP/4.x.xxMozilla/5.0 (Unknown; Linux x86_64) AppleWebKit/534.34 (KHTML, like Gecko) PhantomJS/1.6.1 Safari/534.34
	Mozilla/5.0 (Macintosh; Intel Mac OS X) Word/14.0.0
	Mozilla/5.0 (Macintosh; Intel Mac OS X) Word/14.25.0
	Mozilla/5.0 (Macintosh; Intel Mac OS X) Word/14.47.0
	Mozilla/5.0 (Macintosh; Intel Mac OS X) Word/14.48.0
	Mozilla/5.0 (Macintosh; Intel Mac OS X) Word/14.50.0
	Mozilla/5.0 (Macintosh; Intel Mac OS X) Word/14.56.0
	Mozilla/5.0 (Macintosh; Intel Mac OS X) Word/14.61.0
	Mozilla/5.0 (Macintosh; Intel Mac OS X) Word/14.62.0
	Mozilla/5.0 (Macintosh; Intel Mac OS X) Word/14.63.0
	Mozilla/5.0 (Macintosh; Intel Mac OS X) Word/14.64.0
	WomlpeFactory/0.1 (+http://www.Womple.com/bot.html)
	Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/534.34 (KHTML, like Gecko) wkhtmltoimage Safari/534.34
	Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/534.34 (KHTML, like Gecko) wkhtmltopdf Safari/534.34
	Mozilla/4.0 (compatible; Win32; WinHttp.WinHttpRequest.5)
	Windows-RSS-Platform/2.0 (IE 11.0; Windows NT 10.0)
	WhatsApp
	WhatsApp/2.12.15/i
	WhatsApp/2.12.16/i
	WhatsApp/2.12.17/i
	WhatsApp/2.12.449 A
	WhatsApp/2.12.453 A
	WhatsApp/2.12.510 A
	WhatsApp/2.12.540 A
	WhatsApp/2.12.548 A
	WhatsApp/2.12.555 A
	WhatsApp/2.12.556 A
	WhatsApp/2.16.1/i
	WhatsApp/2.16.13 A
	WhatsApp/2.16.2/i
	WhatsApp/2.16.42 A
	WhatsApp/2.16.57 A
	WhatWeb/0.4.8-dev
	http://www.almaden.ibm.com/cs/crawler [wf84]
	WeLikeLinks - WeAreNotEvil
	Websquash.com (Add url robot)
	WeCrawlForThePeace - We are not Evil
	Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.21 (KHTML, like Gecko) webscreenie Safari/537.21
	Mozilla/3.0 (compatible; WebMon 1.0.11; Windows XP)
	Mozilla/5.0 (X11; BSD Four) AppleWebKit/534.34 (KHTML, like Gecko) webkit2png Safari/534.34
	WebFetch
	WebIndex
	webcollage/1.114
	webcollage/1.117
	webcollage/1.125
	webcollage/1.129
	webcollage/1.93
	Mozilla/3.0 (compatible; WebCapture 2.0; Auto; Windows)
	W3C_Unicorn/1.0
	W3C_I18n-Checker/1.0
	W3C-mobileOK/DDC-1.0
	W3C-checklink
	W3C-checklink/2.90 libwww-perl/5.64
	W3C-checklink/3.6.2.3 libwww-perl/5.64
	W3C-checklink/4.2 [4.20] libwww-perl/5.803
	W3C-checklink/4.2.1 [4.21] libwww-perl/5.803
	W3C-checklink/4.3 [4.42] libwww-perl/5.805
	W3C-checklink/4.3 [4.42] libwww-perl/5.808
	W3C-checklink/4.3 [4.42] libwww-perl/5.820
	W3C-checklink/4.5 [4.154] libwww-perl/5.823
	W3C-checklink/4.5 [4.160] libwww-perl/5.823
	w3af.org
	VYU2 (GNU; OpenRISC)
	VORTEX/1.2 ( http://marty.anstey.ca/robots/vortex/)
	Vortex/2.2 ( http://marty.anstey.ca/robots/vortex/)
	Vortex/2.2 (+http://marty.anstey.ca/robots/vortex/)
	Mozilla/5.0 (Windows NT 5.1; rv:11.0) Gecko Firefox/11.0 (via ggpht.com GoogleImageProxy)
	URLChecker
	urlresolver
	Upflow/1.0
	Typhoeus - https://github.com/typhoeus/typhoeus
	UdmSearch/3.1.x
	Mozilla/5.0 Tweetminster
	Mozilla/5.0 (compatible; +http://tweetedtimes.com)
	truwoGPS/1.0 (GNU/Linux; U; i686; en-US; +http://www.lan4lano.net/browser.html )
	Traackr.com
	More-Testing (+http://www.theinternetrules.com)
	The Expert HTML Source Viewer (http://www.expert-html.com)
	htmlayout 3.3; above-Windows-7; www.terrainformatica.com )
	WebFuck V2.1 T0PHackTeam www.t0p.xyz
	Symfony2 BrowserKit
	Mozilla/5.0 (compatible; SwiteScraper)
	Summify (Summify/1.0.1; +http://summify.com)
	Stratagems Kumo
	Sqworm/2.9.85-BETA (beta_release; 20011115-775; i686-pc-linux-gnu)
	Sprinklr 2.0
	spray-can/1.3.3
	SortSite/4.4.500.0
	Snoopy v2.0.0
	SMRF URL Expander
	silk/1.0 (+http://www.slider.com/silk.htm)/3.7
	SiteBar/3.3.5 (Bookmark Server; http://sitebar.org/)
	SiteBar/3.3.8 (Bookmark Server; http://sitebar.org/)
	SimplePie/1.3.1 (Feed Parser; http://simplepie.org; Allow like Gecko) Build/20160226060242
	ShortLinkTranslate
	Mozilla/5.0 (compatible; Seznam screenshot-generator 2.1; +http://fulltext.sblog.cz/screenshot/)
	SEOstats 2.0.9 https://github.com/eyecatchup/SEOstats
	semanticdiscovery/0.1
	SearchSight/2.0 (http://SearchSight.com/)
	ScoutURLMonitor/5.9.8
	Robosourcer/1.0
	Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; RetrevoPageAnalyzer; +http://www.retrevo.com/content/about-us)
	RestSharp/105.2.3.0
	Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_5) AppleWebKit/537.78.2 (KHTML, like Gecko) Version/7.0.1(2.5.3) Safari(ReederForMac)
	redback/v0-570-g26f8c96
	RebelMouse/0.1 Mozilla/5.0 (compatible; http://rebelmouse.com) Gecko/20100101 Firefox/7.0.1
	RealPlayer%20Downloader/1569 CFNetwork/720.3.13 Darwin/14.3.0 (x86_64)
	RealPlayer%20Downloader/1701 CFNetwork/760.5.1 Darwin/15.5.0 (x86_64)
	RealPlayer%20Downloader/1725 CFNetwork/720.3.13 Darwin/14.3.0 (x86_64)
	RealPlayer%20Downloader/1750 CFNetwork/720.5.7 Darwin/14.5.0 (x86_64)
	RealPlayer%20Downloader/1750 CFNetwork/760.0.5 Darwin/15.0.0 (x86_64)
	RealPlayer%20Downloader/1750 CFNetwork/760.4.2 Darwin/15.4.0 (x86_64)
	Qirina Hurdler v. 1.05 10.11.01 (+http://www.qirina.com/hurdler.html)
	python-requests/1.1.0 CPython/2.7.6 Linux/3.13.0-58-generic
	python-requests/1.2.0 CPython/2.7.3 Linux/3.2.0-29-virtual
	python-requests/2.1.0 CPython/2.7.3 Linux/3.2.0-4-amd64
	python-requests/2.10.0
	python-requests/2.2.1 CPython/2.7.6 Linux/3.13.0-48-generic
	python-requests/2.2.1 CPython/2.7.6 Linux/3.13.0-83-generic
	python-requests/2.3.0 CPython/2.7.6 Linux/3.13.0-57-generic
	python-requests/2.5.0 CPython/2.7.9 Linux/3.14.20-20.44.amzn1.x86_64
	python-requests/2.6.0 CPython/2.7.3 Linux/3.2.0-4-amd64
	python-requests/2.6.0 CPython/2.7.5 Linux/3.10.0-327.3.1.el7.x86_64
	python-requests/2.7.0 CPython/2.7.6 Linux/3.13.0-85-generic
	python-requests/2.8.0
	python-requests/2.8.1
	python-requests/2.9.1
	Python-httplib2/0.7.4 (gzip)
	Pulsepoint XT3 web scraper
	Mozilla/5.0 (compatible; ProspectB2B/2.0; http://prospectb2b.com)
	Mozilla/5.0 (Macintosh; Intel Mac OS X) PowerPoint/14.39.0
	PostRank/2.0 (postrank.com; 1 subscribers)
	PostRank/2.0 (postrank.com)
	postano
	Ploetz + Zeller (http://www.ploetz-zeller.de) Link Validator v1.0 ([email protected]) for ARIS Business Architect
	Pizilla++ ver 2.45
	phpcrawl
	Pearltrees
	Pcore-HTTP/v0.23.20
	Mozilla/5.0 (Windows; U; Windows NT 6.0; en-US; rv:1.9.0.4) Gecko/2008102920 http://ow.ly web crawler (.NET CLR 3.5.30729)
	Mozilla/5.0 (compatible; Optimizer)
	Mozilla/5.0 [en] (X11, U; OpenVAS 7.0.10)
	JetBrains Omea Reader 1.0.x (http://www.jetbrains.com/omea_reader/)
	JetBrains Omea Reader 2.0 Release Candidate 1 (http://www.jetbrains.com/omea_reader/)
	okhttp/2.5.0
	oegp v. 1.3.0
	Nuzzel
	Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/43.0.2357.134 Safari/537.36 http://notifyninja.com/monitoring
	Notifixious/LinkChecker (http://notifixio.us)
	http://www.nominet.org.uk/privacypolicy
	node-superagent/1.8.3
	node.io
	Kimengi/nineconnections.com
	newspaper/0.0.9.8
	newspaper/0.1.7
	newsme/1.0; [email protected]
	NewsGator/2.0 Bot (http://www.newsgator.com)
	NewsGator/2.5 (http://www.newsgator.com; Microsoft Windows NT 5.1.2600.0; .NET CLR 1.1.4322.2032)
	NetTrack Anonymous Web Statistics https://nettrack.info/support.php
	NetTrack.info Crawler
	NetResearchServer(http://www.look.com)
	NetResearchServer/2.5(loopimprovements.com/robot.html)
	NetResearchServer/2.7(loopimprovements.com/robot.html)
	NetResearchServer/2.8(loopimprovements.com/robot.html)
	NetResearchServer/3.5(loopimprovements.com/robot.html)
	NetResearchServer/4.0(loopimprovements.com/robot.html)
	NetLyzer FastProbe (See http://netlyzer.com/report/www.domain.com for info)
	check_http/v2.0.3 (nagios-plugins 2.0.3)
	MVAClient
	Link Validity Check From: http://www.w3dir.com/cgi-bin (Using: Hot Links SQL by Mrcgiguy.com)
	Mojolicious (Perl)
	mogimogi/1.0
	Mindjet
	Mindjet MindManager
	Microsoft Windows Network Diagnostics
	Microsoft Office Excel 2013
	Microsoft Office Excel 2014
	Microsoft Office Excel 2014 (16.0.6528) Windows NT 10.0
	Microsoft Office Existence Discovery
	Microsoft Office Mobile/15.0
	Microsoft Office OneNote 2013
	Microsoft Office PowerPoint 2013
	Microsoft Office PowerPoint 2013 (15.0.4420) Windows NT 6.2
	Microsoft Office PowerPoint 2013 (15.0.4641) Windows NT 6.2
	Microsoft Office PowerPoint 2013 (15.0.4797) Windows NT 10.0
	Microsoft Office PowerPoint 2013 (15.0.4797) Windows NT 6.2
	Microsoft Office PowerPoint 2013 (15.0.4805) Windows NT 6.1
	Microsoft Office PowerPoint 2013 (15.0.4805) Windows NT 6.2
	Microsoft Office PowerPoint 2013 (15.0.4815) Windows NT 10.0
	Microsoft Office PowerPoint 2014
	Microsoft Office Word 2013
	Microsoft Office Word 2013 (15.0.4623) Windows NT 6.2
	Microsoft Office Word 2013 (15.0.4701) Windows NT 6.1
	Microsoft Office Word 2013 (15.0.4797) Windows NT 10.0
	Microsoft Office Word 2013 (15.0.4805) Windows NT 10.0
	Microsoft Office Word 2013 (15.0.4805) Windows NT 6.1
	Microsoft Office Word 2013 (15.0.4823) Windows NT 10.0
	Microsoft Office Word 2014
	Melvil Rawi/1.0
	marketinggrader.com
	Mandrill-Webhook/1.0
	MailChimp.com
	MagpieRSS/0.72 (+http://magpierss.sf.net)
	http://www.mabontland.com
	lwp-trivial/1.33
	lwp-trivial/1.35
	lwp-trivial/1.36
	lwp-trivial/1.38
	lwp-trivial/1.41
	looksystems.net
	LinkExaminer/1.01 (Windows)
	VLC/2.2.4 LibVLC/2.2.4
	[email protected]
	Mozilla/5.0 (compatible; Jobrapido/1.1; +http://www.jobrapido.com)
	JoBo/1.x (http://www.matuschek.net/jobo.html)
	Jigsaw/2.3.0 W3C_CSS_Validator_JFouffa/2.0
	Mozilla/5.0 (compatible; IODC-Internet Engineering Odysseus 001; +http://iodc.co.uk)
	Mozilla/5.0 (compatible; IODC-Internet Engineering Odysseus 005; +http://iodc.co.uk)
	Mozilla/5.0 (compatible; IODC-Internet Engineering Odysseus 006; +http://iodc.co.uk)
	Mozilla/5.0 (compatible; IODC-Internet Engineering Odysseus 007; +http://iodc.co.uk)
	Mozilla/5.0 (compatible; IODC-Internet Engineering Odysseus 011; +http://iodc.co.uk)
	Mozilla/5.0 (compatible; IODC-Internet Engineering Odysseus 013; +http://iodc.co.uk)
	Mozilla/5.0 (compatible; IODC-Odysseus Survey 29691-100-070315021100-72; +http://iodc.co.uk)
	Mozilla/5.0 (compatible; IODC-Odysseus Survey 29692-100-060415034231-192; +http://iodc.co.uk)
	Mozilla/5.0 (compatible; IODC-Odysseus Survey 31193-100-080615161452-372; +http://iodc.co.uk)
	Mozilla/5.0 (compatible; IODC-Odysseus Survey 37044-100-110516191742-025; +https://iodc.co.uk)
	Mozilla/5.0 (compatible; IODC-Odysseus Survey 41437-100-090715214314-122; +http://iodc.co.uk)
	Integrity/6
	InfoWizards Reciprocal Link System PRO - (http://www.infowizards.com)
	InDesign%20CC/11300 CFNetwork/720.4.4 Darwin/14.4.0 (x86_64)
	inbound.li parser
	InAGist URL Resolver (http://inagist.com)
	igdeSpyder (compatible; igde.ru; +http://igde.ru/doc/tech.html)
	IDG Twitter Links Resolver
	Mozilla/4.5 (compatible; HTTrack 3.0x; Windows 98)
	Jersey/2.19 (HttpUrlConnection 1.7.0_85)
	httpunit/1.x
	httpssites_power/1 CFNetwork/758.3.15 Darwin/15.4.0
	httpscheck (unknown version) CFNetwork/760.5.1 Darwin/15.5.0 (x86_64)
	HttpComponents/1.1
	http_requester/0.1
	HTTP_Request2/@package_version@ (http://pear.php.net/package/http_request2) PHP/5.5.9-1ubuntu4.14
	HTTP_Request2/2.2.0 (http://pear.php.net/package/http_request2) PHP/5.3.29
	HTTP_Request2/2.2.1 (http://pear.php.net/package/http_request2) PHP/5.3.29
	HTTP_Request2/2.2.1 (http://pear.php.net/package/http_request2) PHP/5.5.33
	HTTP_Request2/2.2.1 (http://pear.php.net/package/http_request2) PHP/5.5.35
	HTTP_Request2/2.3.0 (http://pear.php.net/package/http_request2) PHP/5.3.29
	HTTP-Tiny/0.054
	http-kit/2.0
	HTMLParser/2.0
	htdig/3.1.5
	htdig/3.1.5 ([email protected])
	htdig/3.1.5 (root@localhost)
	htdig/3.1.5 ([email protected])
	htdig/3.1.6 ([email protected])
	htdig/3.1.6 ([email protected])
	ht://check/2.0.0-rc1 (Linux 2.6.32-573.18.1.el6.x86_64 x86_64)
	HEADMasterSEO
	Mozilla/4.0 (compatible; grub-client-0.3.0; Crawl your own stuff with http://grub.org)
	Mozilla/4.0 (compatible; grub-client-1.0.4; Crawl your own stuff with http://grub.org)
	Mozilla/4.0 (compatible; grub-client-1.0.5; Crawl your own stuff with http://grub.org)
	Mozilla/4.0 (compatible; grub-client-1.0.6; Crawl your own stuff with http://grub.org)
	Mozilla/4.0 (compatible; grub-client-1.0.7; Crawl your own stuff with http://grub.org)
	Mozilla/4.0 (compatible; grub-client-1.1.1; Crawl your own stuff with http://grub.org)
	Mozilla/4.0 (compatible; grub-client-1.2.1; Crawl your own stuff with http://grub.org)
	Mozilla/4.0 (compatible; grub-client-1.3.1; Crawl your own stuff with http://grub.org)
	Mozilla/4.0 (compatible; grub-client-1.3.7; Crawl your own stuff with http://grub.org)
	Mozilla/4.0 (compatible; grub-client-1.4.3; Crawl your own stuff with http://grub.org)
	Mozilla/4.0 (compatible; grub-client-1.5.3; Crawl your own stuff with http://grub.org)
	Mozilla/4.0 (compatible; grub-client-2.3)
	Mozilla/4.0 (compatible; grub-client-2.6.0)
	Mozilla/4.0 (compatible; grub-client-2.6.1)
	GoSpotCheck 1.0.6 (iPhone; iPhone OS 5.1.1; en_US)
	GoScraper
	GoogleProducer; (+http://goo.gl/7y4SX)
	GoogleHC/
	Google-HTTP-Java-Client/1.17.0-rc (gzip)
	Google-HTTP-Java-Client/1.9.0-alpha
	Mozilla/5.0 (compatible; Google-Apps-Script)
	Go-http-client/1.1
	Go 1.1 package http
	Mozilla/5.0 (Macintosh; Intel Mac OS X 10_6_6) AppleWebKit/534.24 (KHTML, like Gecko) (Contact: [email protected])
	Mozilla/5.0 (compatible; Funnelback) RPT-HTTPClient/0.3-3E
	Mozilla/5.0 (compatible; Snacktory; +https://flynxapp.com)
	Mozilla/3.0 (compatible; Fluffy the spider; http://www.searchhippo.com/; [email protected])
	Flamingo_SearchEngine (+http://www.flamingosearch.com/bot)
	Mozilla/4.0 (compatible; MSIE 5.5; Windows NT 5.0) Fetch API Request
	FeedChecker-Zocle/1.0 (+https://zocle.com/zoclechecker)
	FeedBurner/1.0 (http://www.FeedBurner.com)
	FavOrg
	Faraday v0.9.1
	Mozilla/5.0 (Macintosh; Intel Mac OS X) Excel/14.10.0
	Mozilla/5.0 (Macintosh; Intel Mac OS X) Excel/14.62.0
	ExactSearch
	EventMachine HttpClient
	Mozilla/5.0 (compatible; MSIE 7.0 +http://www.europarchive.org)
	Embed PHP Library
	ECCP/1.2.1+([email protected])
	ec2linkfinder
	Mozilla/5.0 (compatible; EARTHCOM/2.2; +http://enter4u.eu)
	Drupal (+http://drupal.org/)
	Dragonfly File Reader
	DNS-Tools Header-Analyzer
	Dispatch/0.11.1-SNAPSHOT
	[email protected]
	CyberPatrol SiteCat Webbot (http://www.cyberpatrol.com/cyberpatrolcrawler.asp)
	curb
	Mozilla/4.0 (compatible; cron-job.org; http://cron-job.org/abuse/)
	cosmos/0.9_([email protected])
	Mozilla/5.0/Firefox/42.0 (contactbigdatafr at gmail.com)
	Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; .NET CLR 2.0.50727; MAXTHON 2.0); Connect Us: [email protected]
	Pattern/2.6 +http://www.clips.ua.ac.be/pattern
	Mozilla/4.0 (compatible; Cerberian Drtrs Version-3.2-Build-0)
	Mozilla/4.0 (compatible; Cerberian Drtrs Version-3.2-Build-1)
	CC Metadata Scaper http://wiki.creativecommons.org/Metadata_Scraper
	cb crawl (+http://www.companybooknetworking.com)
	CapsuleChecker (http://www.capsulink.com/)
	CakePHP
	Branch-Passthrough
	Branch Metrics API
	boitho.com-dc/0.79 ( http://www.boitho.com/dcbot.html )
	boitho.com-dc/0.81 ( http://www.boitho.com/dcbot.html )
	boitho.com-dc/0.82 ( http://www.boitho.com/dcbot.html )
	boitho.com-dc/0.83 ( http://www.boitho.com/dcbot.html )
	boitho.com-dc/0.85 ( http://www.boitho.com/dcbot.html )
	boitho.com-robot/1.1
	Blackboard Safeassign
	Mozilla/5.0 (Windows NT 6.3; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/42.0.2311.152 Safari/537.36 (BingLocalSearch)
	BIGLOTRON (Beta 2;GNU/Linux)
	Mozilla/5.0 (compatible; bnf.fr_bot; +http://bibnum.bnf.fr/robot/bnf.html)
	Mozilla/4.0 (compatible; B-l-i-t-z-B-O-T)
	Astute SRM
	aria2/1.18.8
	Mozilla/5.0 (compatible; archive-com.com/1.1; +http://archive-com.com/bot)
	Mozilla/4.0 (compatible; Arachmo)
	Apache-HttpClient/4.2.3 (java 1.5)
	Apache-HttpClient/4.2.5 (java 1.5)
	Apache-HttpClient/4.3.1 (java 1.5)
	Apache-HttpClient/4.3.3 (java 1.5)
	Apache-HttpClient/4.3.5 (java 1.5)
	Apache-HttpClient/4.5 (Java/1.8.0_51)
	Apache-HttpClient/4.5 (Java/1.8.0_72)
	Apache-HttpClient/4.5 (Java/1.8.0_74)
	Apache-HttpClient/4.5.1 (Java/1.7.0_79)
	Apache-HttpClient/4.5.1 (Java/1.8.0_40)
	Apache-HttpClient/4.5.2 (Java/1.8.0_72)
	Apache-HttpClient/UNAVAILABLE (java 1.4)
	Mozilla/5.0 (compatible; U; AnyEvent-HTTP/2.22; +http://software.schmorp.de/pkg/AnyEvent)
	Mozilla/5.0 (compatible; U; AnyEvent-HTTP/2.23; +http://software.schmorp.de/pkg/AnyEvent)
	Anemone/0.7.2
	AndroidDownloadManager/6.0.1 (Linux; U; Android 6.0.1; SM-G900F Build/MMB29M)
	AndroidDownloadManager/4.4.2 (Linux; U; Android 4.4.2; SM-T310 Build/KOT49H)
	AndroidDownloadManager/4.4.2 (Linux; U; Android 4.4.2; SM-T210 Build/KOT49H)
	Airmail 1.3.3 rv:237 (Macintosh; Mac OS X 10.9.3; sk_SK)
	Python/3.5 aiohttp/0.21.6
	AHC/1.0
	Mozilla/4.0 (compatible; Advanced Email Extractor v2.61)
	Pulp/1.5.2 (iPad; http://www.acrylicapps.com/pulp/)
	Accoona-AI-Agent/1.1.1 (crawler at accoona dot com)
	Accoona-AI-Agent/1.1.2
	Accoona-AI-Agent/1.1.2 (aicrawler at accoonabot dot com)
	Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1) [email protected]
	scrutiny/6
	RMA/1.0 (compatible; RealMedia)
	PHP/5.5
	NING/1.0
	Mget/0.1.9
	Java/1.4.1_04
	Java/1.6.0
	Java/1.6.0_04
	Java/1.6.0_22
	Java/1.6.0_24
	Java/1.6.0_26
	Java/1.6.0_31
	Java/1.6.0_32
	Java/1.6.0_34
	Java/1.6.0_37
	Java/1.7.0_101
	Java/1.7.0_45
	Java/1.7.0_51
	Java/1.7.0_65
	Java/1.7.0_80
	Java/1.8.0_05
	Java/1.8.0_20
	Java/1.8.0_31
	Java/1.8.0_40
	Java/1.8.0_45
	Java/1.8.0_51
	Java/1.8.0_60
	Java/1.8.0_65
	Java/1.8.0_66
	Java/1.8.0_73
	Java/1.8.0_77
	Java/1.8.0_91
	Java/1.8.0_92
	Goose/1.0.25
	FDM 3.x
	bluefish 0.6 HTML editor
	2ip.ru CMS Detector (http://2ip.ru/cms/)
	masscan/1.0 (https://github.com/robertdavidgraham/masscan)
	VidibleScraper/1.0
	WhereGoes? Redirect Tracer/1.0
	unfurlist (https://github.com/Doist/unfurlist)
	pshtt, https scanning
	gofetch
	COMODO DCV
	CATExplorador/1.0beta (sistemes at domini dot cat; http://domini.cat/catexplorador.html)
	ZmEu
	RSSOwl/2.2.1.201312301314 (Windows; U; en)
	WebClient/1.0
	Microsoft-WebDAV-MiniRedir/10.0.10586
	Microsoft-WebDAV-MiniRedir/10.0.14393
	Microsoft-WebDAV-MiniRedir/5.1.2600
	Microsoft-WebDAV-MiniRedir/6.0.6001
	Microsoft-WebDAV-MiniRedir/6.1.7601
	Microsoft-WebDAV-MiniRedir/6.3.9600
	Microsoft Data Access Internet Publishing Provider Protocol Discovery
	Test Certificate Info
	Needle/1.4.2 (Node.js v6.9.2; linux x64)
	DavClnt
	Mozilla/5.0 (iPhone; CPU iPhone OS 6_0 like Mac OS X) AppleWebKit/536.26 (KHTML, like Gecko) Version/6.0 Mobile/10A5376e Safari/8536.25 mixed-content-scan
	Mozilla/5.0 (compatible; Semiocast HTTP client; http://semiocast.com/)
	Mozilla/5.0 (compatible; The Drop Reaper; +http://tag1consulting.com/dropreaper.html)
	/n software IPWorks HTTP/S Component - www.nsoftware.com
	safe-agent-scanner
	Mozilla/5.0 (compatible; ACHE/Unknown Version; +https://github.com/ViDA-NYU/ache; )
	ANAM (github.com/dutchcoders/anam)
	Dolphin http client/11.2.8(385) (Android)
	Dolphin http client/11.3.4(405) (Android)
	Calypso v/0.01
	-Mozilla/4.0 (compatible; http://search.thunderstone.com/texis/websearch/about.html)
	Mozilla/5.0 (compatible; Tweezler/2.0; http://tweezler.com)
	Jeode/1.x.x
	KeyCDN Perf Test
	Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) YLT Chrome/27.0.1453.110 Safari/537.36
	Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/51.0.2704.84 Safari/537.36 DMBrowser-UV
	Taringa UGC www.taringa.net/shared-link.php
	Mozilla/5.0 (Unknown; Linux x86_64) AppleWebKit/534.34 (KHTML, like Gecko) PingdomTMS/0.8.5 Safari/534.34
	ganarvisitas/1.0 (+http://www.ganarvisitas.com/)
	FeedFetcher-Google; (+http://www.google.com/feedfetcher.html)
	Mozilla/5.0 (compatible) Feedfetcher-Google; (+http://www.google.com/feedfetcher.html)
	Mozilla/5.0 (compatible; Exabot PyExalead/3.0; +http://www.exabot.com/go/robot)
	Mozilla/5.0 PhantomJS (compatible; Seznam screenshot-generator 2.1; +http://fulltext.sblog.cz/screenshot/)
	Mozilla/5.0 (compatible; SeznamBot/3.2-test4; +http://napoveda.seznam.cz/en/seznambot-intro/)
	Mozilla/5.0 (compatible; SeznamBot/3.2-test2; +http://napoveda.seznam.cz/en/seznambot-intro/)
	Mozilla/5.0 (compatible; SeznamBot/3.2-test1; +http://fulltext.sblog.cz/)
	Mozilla/5.0 (compatible; SeznamBot/3.2-test4; +http://fulltext.sblog.cz/)
	Mozilla/5.0 (compatible; SeznamBot/3.2; +http://fulltext.sblog.cz/)
	Seznam-Zbozi-robot/3.2.1
	SeznamBot/2.0 (+http://fulltext.sblog.cz/robot/)
	Seznam-Zbozi-robot/3.2.2
	Mozilla/5.0 (compatible; SeznamBot/3.2-test2; +http://fulltext.sblog.cz/)
	Mozilla/5.0 (compatible; SeznamBot/3.2-test3; +http://fulltext.sblog.cz/)
	Seznam-Zbozi-robot/3.3
	SklikBot/2.0 ([email protected];+http://napoveda.sklik.cz/)
	SeznamBot/3.0 (+http://fulltext.sblog.cz/)
	Mozilla/5.0 (compatible; SeznamBot/3.1-test1; +http://fulltext.sblog.cz/)
	Mozilla/5.0 (Linux; U; Android 4.1.2; cs-cz; Seznam screenshot-generator Build/Q3) AppleWebKit/534.30 (KHTML, like Gecko) Version/4.0 Mobile Safari/534.30
	SeznamBot/3.0-test (+http://fulltext.sblog.cz/), I
	SeznamBot/3.0 (HaF+http://fulltext.sblog.cz/)
	SeznamBot/3.0-test (+http://fulltext.sblog.cz/)
	SeznamBot/3.0-beta (+http://fulltext.sblog.cz/), I
	SeznamBot/3.0-beta (+http://fulltext.sblog.cz/)
	SeznamBot/3.0-alpha (+http://fulltext.sblog.cz/)
	SeznamBot/2.0 (+http://fulltext.seznam.cz/)
	SeznamBot/2.0-Test (+http://fulltext.sblog.cz/robot/)
	Mozilla/5.0 (compatible; Seznam screenshot-generator 2.0; +http://fulltext.sblog.cz/screenshot/)
	Mozilla/5.0 (compatible; FlipboardRSS/1.2; +http://flipboard.com/browserproxy)
	Mozilla/5.0 (Macintosh; Intel Mac OS X 10.11; rv:49.0) Gecko/20100101 Firefox/49.0 (FlipboardProxy/1.2; +http://flipboard.com/browserproxy)
	Mozilla/5.0 (compatible; MJ12bot/v1.4.7; http://mj12bot.com/)
	Mozilla/5.0 (Macintosh; Intel Mac OS X 10_7_2) AppleWebKit/535.7 (KHTML, like Gecko) Chrome/16.0.912.77 Safari/535.7 AppEngine-Google; (+http://code.google.com/appengine; appid: s~feedly-social)
	Mozilla/5.0 (Macintosh; Intel Mac OS X 10_7_2) AppleWebKit/535.7 (KHTML, like Gecko) Chrome/23.0.912.77 Safari/535.7 AppEngine-Google; (+http://code.google.com/appengine; appid: s~feedly-nikon3)
	Mozilla/5.0 (compatible; SemrushBot/1.2~bl; +http://www.semrush.com/bot.html)
	Mozilla/5.0 (compatible; YandexMetrika/2.0; +http://yandex.com/bots mtmon01it.mtrs.yandex.ru)
	Mozilla/5.0 (iPhone; CPU iPhone OS 8_1 like Mac OS X) AppleWebKit/600.1.4 (KHTML, like Gecko) Version/8.0 Mobile/12B411 Safari/600.1.4 (compatible; YandexBot/3.0; +http://yandex.com/bots)
	Mozilla/5.0 (compatible; memoryBot/1.24.61 +http://internetmemory.org/en/)
	Mozilla/5.0 (compatible; memoryBot/1.24.54 +http://internetmemory.org/en/)
	BUbiNG - Research at Brno University of Technology - KNOT group - http://knot.fit.vutbr.cz/crawling/ - Stop? http://law.di.unimi.it/BUbiNG.html
	Mozilla/5.0 (compatible; MSIE 9.0; Firefox/50.1) Daum/4.1
	Mozilla/5.0 (Unknown; Linux x86_64) AppleWebKit/534.34 (KHTML, like Gecko) PhantomJS/1.9.7 Daum Raam Edition b20140820 Safari/534.34
	Sogou Pic Spider/3.0(+http://www.sogou.com/docs/help/webmasters.htm#07)
	yacybot (/global; amd64 Linux 4.4.0-59-generic; java 1.8.0_121; Europe/de) http://yacy.net/bot.html
	yacybot (-global; amd64 Linux 4.4.0-57-generic; java 9-internal; Europe/en) http://yacy.net/bot.html
	yacybot (/global; amd64 Windows 8.1 6.3; java 1.8.0_111; Europe/de) http://yacy.net/bot.html
	yacybot (/global; amd64 Linux 3.16.0-4-amd64; java 1.8.0_111; Europe/en) http://yacy.net/bot.html
	yacybot (/global; amd64 FreeBSD 10.3-RELEASE-p7; java 1.7.0_95; GMT/en) http://yacy.net/bot.html
	yacybot (webportal-global; amd64 Linux 4.4.0-46-generic; java 1.8.0_91; Europe/en) http://yacy.net/bot.html
	yacybot (-global; amd64 Linux 3.14.32-xxxx-grs-ipv6-64; java 1.8.0_111; Europe/de) http://yacy.net/bot.html
	Mozilla/5.0 (compatible; special_archiver/3.3.0 [email protected] +https://archive.org/details/archive.org_bot)
	Mozilla/5.0 (compatible; heritrix/3.3.0-SNAPSHOT-20140702-2247 +http://archive.org/details/archive.org_bot)
	Mozilla/5.0 (compatible; archive.org_bot/heritrix-1.15.4 +http://www.archive.org)
	SeobilityBot (SEO-Check; http://bit.ly/1dJuuzs)
	Testomatobot/1.0 (Linux x86_64; +http://www.testomato.com/testomatobot) minicrawler/5.0.4
	Testomatobot/1.0 (Linux x86_64; +http://www.testomato.com/testomatobot) minicrawler/5.0.3
	Testomatobot/1.0 (Linux x86_64; +http://www.testomato.com/testomatobot) minicrawler/5.0.2
	Testomatobot/1.0 (Linux x86_64; +http://www.testomato.com/testomatobot) minicrawler/5.0.1
	Mozilla/5.0 (compatible; AhrefsBot/5.2; +http://ahrefs.com/robot/)
	Heurekabot-Feed/1.0 (+https://sluzby.heureka.cz/napoveda/heurekabot/)
	Heurekabot-ImageFullText/1.0 (+https://sluzby.heureka.cz/napoveda/heurekabot/)
	Heurekabot-ImageFullText/1.0 (+http://sluzby.heureka.cz/napoveda/heurekabot/)
	Mozilla/5.0 (Windows NT 6.2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/31.0.1650.63 Safari/537.36 QIHU 360SE; 360Spider
	ZoomBot (Linkbot 1.0 http://suite.seozoom.it/bot.html)
	Mozilla/5.0 (compatible; SOLOFIELD/1.0 +http://solofield.net/bot.html)
	Qwantify/1.0
	Mozilla/5.0 (compatible; Yeti/1.1; +http://naver.me/bot)
	Scrapy/1.3.0 (+http://scrapy.org)
	Scrapy/1.0.5.post4+g4b324a8 (+http://scrapy.org)
	Scrapy/1.2.1 (+http://scrapy.org)
	Scrapy/0.16.5 (+http://scrapy.org)
	Mozilla/5.0 (compatible; Dataprovider.com;)
	Mozilla/5.0 (compatible; Dataprovider; https://www.dataprovider.com/spider/)
	Mozilla/5.0 (compatible; SiteExplorer/1.1b; +http://siteexplorer.info/Backlink-Checker-Spider/)
	ICC-Crawler/2.0 (Mozilla-compatible; ; http://ucri.nict.go.jp/en/icccrawler.html)
	DuckDuckBot/1.0; (+http://duckduckgo.com/duckduckbot.html)
	Mozilla/5.0 (compatible; SMTBot/1.0; http://www.similartech.com/smtbot)
	Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/538.1 (KHTML, like Gecko) Chromium/31.0.1650.63 Site-Shot/2.1 (http://www.site-shot.com/) Safari/538.1
	Mozilla/5.0 (Windows NT 6.1; WOW64; Neustar WPM) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/48.0.2564.82 Safari/537.36
	Mozilla/5.0 (iPad; CPU OS 6_1_2 like Mac OS X; Neustar WPM) AppleWebKit/536.26 (KHTML, like Gecko) Version/6.0 Mobile/10A5376e Safari/8536.25
	Mozilla/5.0 (Windows NT 6.0; Neustar WPM) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/48.0.2564.82 Safari/537.36
	Mozilla/5.0 (Windows NT 6.1; Neustar WPM) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/48.0.2564.82 Safari/537.36
	Mozilla/5.0 (Windows NT 6.0; rv:38.0; Neustar WPM) Gecko/20100101 Firefox/38.0
	Mozilla/5.0 (Windows NT 6.1; rv:38.0; Neustar WPM) Gecko/20100101 Firefox/38.0
	Mozilla/5.0 (Windows NT 6.1; Neustar WPM) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/52.0.2743.82 Safari/537.36
	Mozilla/5.0 (Linux; Android 4.1.1; HTC6435LVW Build/JRO03C; Neustar WPM) AppleWebKit/537.22 (KHTML, like Gecko) Chrome/25.0.1364.169 Mobile Safari/537.22
	Mozilla/5.0 (compatible; Whoiswebsitebot/0.1; +http://www.whoiswebsite.net)
	Mozilla/5.0 (Linux; Android 5.0.2; SM-G920T Build/LRX22G) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/54.0.2840 Mobile Safari/537.36 DareBoost
	Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/54.0.2840 Safari/537.36 DareBoost
	Mozilla/5.0 (Linux; Android 5.0; LG-D855 Build/LRX21R.A1422018487) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/54.0.2840 Mobile Safari/537.36 DareBoost
	Mozilla/5.0 (iPhone; CPU iPhone OS 10_0_1 like Mac OS X) AppleWebKit/602.1.50 (KHTML, like Gecko) Version/10.0 Mobile/14A403 Safari/602.1 DareBoost
	Mozilla/5.0 (Linux; Android 4.4.3; HTC One Build/KTU84L) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/54.0.2840 Mobile Safari/537.36 DareBoost
	Mozilla/5.0 (Linux; Android 4.4; Nexus 5 Build/KRT16M) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/54.0.2840 Mobile Safari/537.36 DareBoost
	HubSpot Website Grader ([email protected])
	HubSpot Marketing Grader, HubSpot Marketing Grader
	com.hubspot.spooks.core.UserAgent@753535c0, HubSpot Marketing Grader
	Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/51.0.2704.79 Safari/537.36 (http://www.shrinktheweb.com)
	scrapy-redis (+https://github.com/rolando/scrapy-redis)
	Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/55.0.2883.87 Safari/537.36 PTST/348
	Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/54.0.2840.99 Safari/537.36 PTST/345
	TeamSpeak3-ImageFetcher-1.0
	wf_crawler (http://www.websitefigures.com)
	wf_crawler (http://websitefigures.com)
	Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/55.0.2883.87 Safari/537.36 TinEye/1.0 (via http://www.tineye.com/)
	TinEye-bot/0.61 (see http://www.tineye.com/crawler.html)
	Mozilla/5.0 (compatible; Googlebot/2.1; https://deepcrawl.com/bot)
	Googlebot deepcrawl
	Mozilla/5.0 (compatible; evc-batch/2.0.20170110125844)
	Mozilla/5.0 (compatible; evc-batch/2.0.20161220184139)
	Mozilla/5.0 (compatible; evc-batch/2.0.20161122223033)
	Mozilla/5.0 (compatible; evc-batch/2.0.20161026010401)
	Mozilla/5.0 (X11; Linux x86_64; rv:49.0; GTmetrix https://gtmetrix.com/) Gecko/20100101 Firefox/49.0
	Mozilla/5.0 (Linux; Android 4.3; Galaxy Nexus Build/JWR66Y; GTmetrix https://gtmetrix.com/) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/54.0.2840.68 Mobile Safari/537.36
	Mozilla/5.0 (X11; Linux x86_64; GTmetrix https://gtmetrix.com/) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/54.0.2840.71 Safari/537.36
	Mediumbot-MetaTagFetcher/0.3 (+https://medium.com/)
	Mediumbot-ProxyFetcher/0.1 (+https://medium.com/)
	YOOBI.de - WebBot 1.0
	YOOBI - WebBot 1.0
	Mozilla/5.0 (compatible; heritrix/3.3.0-SNAPSHOT-20160309-0050; UniLeipzigASV +http://corpora.informatik.uni-leipzig.de/crawler_faq.html)
	IAS crawler (page scorer; http://integralads.com/site-indexing-policy/)
	Mozilla/5.0 (compatible; JobKeresoBot; +https://www.kozvetlen-allasok.hu/help.jsp; [email protected]; 5.0
	Mozilla/5.0 (compatible; JobKereso; +https://www.kozvetlen-allasok.hu/robot.jsp [email protected])
	pyspider/0.3.9 (+http://pyspider.org/)
	Mozilla/5.0 (X11; U; Linux x86_64; de-DE; rv:1.9.0.19) Gecko/2010120923 ThumbShotsBot (KFSW 3.0.6-3)
	Mozilla/5.0 (compatible; Discordbot/1.0; +https://discordapp.com)
	Mozilla/5.0 (compatible; Dead Link Checker; http://www.dead-link-checker.com/)
	Mozilla/5.0 (compatible; MSIE 10.0; Windows NT 6.2; WOW64; Trident/6.0) CrawlerProcess (http://www.PowerMapper.com) /5.24.776.0
	Mozilla/4.0 (compatible; BOTW Spider; +http://botw.org)
	TwengaBot-2.0
	Iframely/1.0.0 (+https://iframely.com/;)
	Iframely/0.8.5 (+http://iframely.com/;)
	rogerbot/1.2 (http://moz.com/help/pro/what-is-rogerbot-, [email protected])
	rogerbot/1.0 (http://moz.com/help/pro/what-is-rogerbot-, [email protected]
	Mozilla/5.0 (compatible; Sitemap Generator/1.3; http://www.check-domains.com/sitemap/index.php) Gecko Check-domains)
	mindUpBot
	Mozilla/5.0 (Windows NT 6.1; compatible; BDCbot/1.0; +http://bigweb.bigdatacorp.com.br/faq.aspx) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2272.118 Safari/537.36
	Contacts Crawler (+http://www.scrapinghub.com)
	TA SEO Crawler v0.1 URLData.pm ([email protected])
	Mozilla/5.0 (compatible; scanbot/1.0; +http://dazzlepod.com/ip/)
	ArielisBot/1.1 (+http://arielis.com/
	RedesScrapy/0.24.1 (+http://g2pi.tsc.uc3m.es/es)
	Blackboard Safeassign/0.1 (a Storm-based Blackboard Safeassign web-crawler; https://github.com/DigitalPebble/storm-crawler; [email protected])
	Mozilla/5.0 (Macintosh; Intel Mac OS X 10.11; rv:48.0) Gecko/20100101 Firefox/48.0 glindahl-cocrawler/0.01 (+http://www.pbm.com/~lindahl/glindahl-cocrawler.html)
	botrobin/Nutch-1.13-SNAPSHOT (http://smarter.codes/bot-robin/; [email protected])
	Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/51.0.2704.63 Safari/537.36 (compatible; aylienbot/0.2; +http://www.aylien.com/bot.html)
	GloomarBot/1.1 (https://www.gloomar.com/bot)
	Hatena::Bookmark/2.00 (Hatena::Bookmark; master;)
	Opendi Screenshot Bot [email protected]
	Mozilla/5.0 (compatible; YaK/1.0; http://linkfluence.com/; [email protected])
	Mozilla/5.0 (compatible; KAZ.KZ_Bot/3.0)
	BuckyOHare/1.3 (Googlebot/2.1; +https://hypefactors.com/webcrawler)
	uipbot/1.0 ([email protected])
	gMozilla/5.0 (compatible; Infohelfer/1.4.3; +http://www.infohelfer.de/crawler.php)
	Mozilla/5.0 (compatible; lincobot/0.3; http://www.linksco.com/about/robot.html) AppEngine-Google; (+http://code.google.com/appengine)
	Mozilla/5.0 (compatible; MSIE 10.0; Windows NT 6.2; WOW64; Trident/6.0) MapperCmd/5.24.780.0
	Seeker v.1 http://LookSeek.com/seeker/
	AppManager RPT-HTTPClient/0.3-3E
	LinkAlarm/3.1
	Mozilla/5.0 (X11; U; Linux i686) Firefox/2.0.0.11 wprecon.com survey
	Jeffrey's Exif Viewer (http://regex.info/exif)
	Untiny
	Mozilla/5.0 (Grabify.link - https://grabify.link/expander)
	biNu image cacher ([email protected])
	Mozilla/5.0 (compatible; Google-Youtube-Links)
	ping.blo.gs/2.0
	Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; QQDownload 1.7; .NET CLR 2.0.50727)
	Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.1; Trident/4.0; QQDownload 663; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; .NET4.0C; .NET CLR 1.1.4322)
	Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 5.1; Trident/4.0; QQDownload 760; .NET CLR 2.0.50727; .NET CLR 3.0.4506.2152; .NET CLR 3.5.30729; InfoPath.2; .NET4.0E; .NET4.0C; .NET CLR 1.1.4322)
	Mozilla/5.0 (X11; Linux x86_64; rv:52.0.2) Gecko/20100101 Firefox/52.0.2 Ghost Inspector
	Mozilla/5.0 (X11; Linux x86_64; rv:52.0) Gecko/20100101 Firefox/52.0 Ghost Inspector
	Mozilla/5.0 (X11; Linux x86_64; rv:50.1.0) Gecko/20100101 Firefox/50.1.0 Ghost Inspector
	Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/538.1 (KHTML, like Gecko) QtWebEngine/5.3.0 Safari/538.1 Ghost Inspector
	Mozilla/5.0 (X11; Linux x86_64; rv:45.0.1) Gecko/20100101 Firefox/45.0.1 Ghost Inspector
	Mozilla/5.0 (X11; Linux x86_64; rv:46.0.1) Gecko/20100101 Firefox/46.0.1 Ghost Inspector
	Mozilla/5.0 (X11; Linux x86_64; rv:47.0.2) Gecko/20100101 Firefox/47.0.2 Ghost Inspector
	Mozilla/5.0 (X11; Linux x86_64; rv:48.0.2) Gecko/20100101 Firefox/48.0.2 Ghost Inspector
	Mozilla/5.0 (X11; Linux x86_64; rv:49.0.2) Gecko/20100101 Firefox/49.0.2 Ghost Inspector
	Netpursual/1.0
	Kaspersky Lab CFR link resolver [email protected]
	Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/27.0.1453 Safari/537.36 (compatible; Google-HotelAdsVerifier/2.0)
	b0t
	"echocrawl 2.0"
	masscan/1.0
	Mozilla/5.0 (iPhone; CPU iPhone OS 8_1 like Mac OS X) AppleWebKit/600.1.4 (KHTML, like Gecko) Version/8.0 Mobile/12B411 Safari/600.1.4 (compatible; YandexBot/3.0; +http://yandex.com/bots)
	Mozilla/5.0 (compatible; YandexAccessibilityBot/3.0; +http://yandex.com/bots)
	Mozilla/5.0 (iPhone; CPU iPhone OS 8_1 like Mac OS X) AppleWebKit/600.1.4 (KHTML, like Gecko) Version/8.0 Mobile/12B411 Safari/600.1.4 (compatible; YandexMobileBot/3.0; +http://yandex.com/bots)
	Mozilla/5.0 (compatible; YandexDirectDyn/1.0; +http://yandex.com/bots
	Mozilla/5.0 (compatible; YandexImages/3.0; +http://yandex.com/bots)
	Mozilla/5.0 (compatible; YandexVideo/3.0; +http://yandex.com/bots)
	Mozilla/5.0 (compatible; YandexMedia/3.0; +http://yandex.com/bots)
	Mozilla/5.0 (compatible; YandexBlogs/0.99; robot; +http://yandex.com/bots)
	Mozilla/5.0 (compatible; YandexFavicons/1.0; +http://yandex.com/bots)
	Mozilla/5.0 (compatible; YandexWebmaster/2.0; +http://yandex.com/bots)
	Mozilla/5.0 (compatible; YandexPagechecker/1.0; +http://yandex.com/bots)
	Mozilla/5.0 (compatible; YandexImageResizer/2.0; +http://yandex.com/bots)
	Mozilla/5.0 (compatible; YaDirectFetcher/1.0; Dyatel; +http://yandex.com/bots)
	Mozilla/5.0 (compatible; YandexCalendar/1.0; +http://yandex.com/bots)
	Mozilla/5.0 (compatible; YandexSitelinks; Dyatel; +http://yandex.com/bots)
	Mozilla/5.0 (compatible; YandexMetrika/3.0; +http://yandex.com/bots)
	Mozilla/5.0 (compatible; YandexAntivirus/2.0; +http://yandex.com/bots)
	Mozilla/5.0 (compatible; YandexVertis/3.0; +http://yandex.com/bots)
	Mozilla/5.0 (compatible; YandexBot/3.0; MirrorDetector; +http://yandex.com/bots)
	jetmon/1.0 (Jetpack Site Uptime Monitor by WordPress.com)
	Mozilla/5.0 (compatible; Abonti/0.8 - http://www.abonti.com)
	CloudEndure Scanner ([email protected])
	AFB/3.0 (+http://allloadin.com)
	TLSProbe/1.0 (+https://scan.trustnet.venafi.com/)
	Sens.ai http://sovereign.ai/contact
	eContext/1.0 (eContext Classification Engine)
	Owler
	Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Slack/1.2.6 Chrome/45.0.2454.85 AtomShell/0.34.3 Safari/537.36 Slack_SSB/1.2.6
	Mozilla/5.0 (compatible; ChangesMeter/1.9.1; http://intuiware.com/apps/changes-meter)
	Mozilla/5.0 (compatible; Gookey.co/1.0; +http://gookey.co/)
	TrapitAgent/0.1 (feed processor; +http://trapit.com/about)
	sqlmap/1.0-dev-nongit-201612050a8c (http://sqlmap.org)
	UniversalFeedParser/3.3 +http://feedparser.org/
	TulipChain/5.xx (http://ostermiller.org/tulipchain/) Java/1.x.1_0x (http://apple.com/) Mac_OS_X/10.2.8
	scalaj-http/1.0
	HggH PhantomJS Screenshoter
	fasthttp
	fasthttp, Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/53.0.2785.101 Safari/537.36
	lua-resty-http/0.10 (Lua) ngx_lua/10000
	GIS-LABS:CertMon
	GuzzleHttp/6.2.1 PHP/7.0.15-0ubuntu0.16.04.4
	http-request/v0.7.0 (http://git.io/tl_S2w) node.js/v0.10.29
	intraVnews/1.x
	MailChimp
	G-i-g-a-b-o-t
	Symfony BrowserKit
	VCPP71 http-get 1.0a
	Jaunt/1.2
	Mozilla/5.0 (compatible; Owler/0.4; +; )
	MovableType/x.x
	OMSC
	websitepulse agent/0.1 (compatible; MSIE 5.5; Netscape 4.75; Linux)
	Mozilla/5.0 (compatible; Uptimia; www.uptimia.com)
	ssl-tools.net
	Mozilla/5.0 (compatible; MSIE 10.0; Windows NT 6.1; Trident/6.0; .NET CLR 1.1.4322; .NET CLR 1.0.3705; .NET CLR 2.0.50727; .NET CLR 3.0.04506.648; .NET CLR 3.5.21022; .NET CLR 3.0.4506.2152; .NET CLR 3.5.30729) DMBrowser/2.1
	keycdn-tools/perf
	keycdn-tools/br
	keycdn-tools/h2
	keycdn-tools/perf
	keycdn-tools/br
	keycdn-tools/h2
	KOCMOHABT (+https://kozmonavt.tk/) Mozilla/5.0 (compatible; Web Explorer 2)
	Donuts Content Explorer (www.donuts.domains)
	Netsparker
	Mozilla/5.0+(compatible; MxToolbox/Beta7; http://www.mxtoolbox.com/)
	Apache-HttpAsyncClient/4.1.3 (Java/1.8.0_121)
	Tenon.io Automated Web Accessibility Tester
	ApacheBench/2.3
	Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; Clarsentia)
	Mozilla/5.0 (compatible; historious/2.0; +http://historio.us/)
	Mozilla/5.0 (compatible; uCrawlr/1.0 ; +https://blog.ucoz.ru/upolicy)
	Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)
	Brodie/1.0 (+http://www.15miles.com/)
	Jobg8 URL Monitor
	Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.21 (KHTML, like Gecko) wpif Safari/537.21
	Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.21 (KHTML, like Gecko) imgsizer Safari/537.21
	Mozilla/5.0 (compatible; AppBeat/1.0; +http://www.appbeat.io/)
	Mozilla/5.0 (compatible; Chirp/1.0; +http://www.binarycanary.com)
	Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; WOW64; Trident/5.0; SiteMonitor)
	Mixnode/3.0.1 (+https://www.mixnode.com/; Speed analysis; no data is stored)
	Pingability.com Monitoring Service

Тоже, но с учетом регулярных выражений:

Спойлер

.*Java.*outbrain

008\/

192\.comAgent

2ip\.ru

404checker

^bluefish

^Calypso v\/

^COMODO DCV

^DangDang

^DavClnt

^FDM

^git\/

^Goose\/

^HTTPClient\/

^Java\/

^Jeode\/

^Jetty\/

^Mget

^Microsoft URL Control

^NG\/[0-9\.]

^NING\/

^PHP\/[0-9]

^RMA\/

^Ruby|Ruby\/[0-9]

^scrutiny\/

^VSE\/[0-9]

^WordPress\.com

^XRL\/[0-9]

^ZmEu

a3logics\.in

A6-Indexer

a\.pr-cy\.ru

Abonti\/

Aboundex

aboutthedomain

Accoona-AI-Agent

acoon

acrylicapps\.com\/pulp

adbeat

AddThis

ADmantX

adressendeutschland

Advanced Email Extractor v

agentslug

AHC

aihit

aiohttp\/

Airmail

akula\/

alertra

alexa site audit

Alibaba\.Security\.Heimdall

allloadin\.com

alyze\.info

amagit

AndroidDownloadManager

Anemone

Ant\.com

Anturis Agent

AnyEvent-HTTP\/

ApacheBench\/

Apache-HttpAsyncClient\/

Apache-HttpClient\/

AportWorm\/[0-9]

AppBeat\/[0-9]

AppEngine-Google

Arachmo

arachnode

Arachnophilia

aria2

asafaweb.com

AskQuickly

Astute

asynchttp

autocite

Autonomy

B-l-i-t-z-B-O-T

^b0t$

Backlink-Ceck\.de

Bad-Neighborhood

baidu\.com

baypup\/[0-9]

baypup\/colbert

BazQux

BCKLINKS

BDFetch

BegunAdvertising\/

BigBozz

biglotron

BingLocalSearch

BingPreview

binlar

biNu image cacher

biz_Directory

Blackboard Safeassign

Bloglovin

BlogPulseLive

BlogSearch

Blogtrottr

boitho\.com-dc

BPImageWalker

Braintree-Webhooks

Branch Metrics API

Branch-Passthrough

Brodie\/

Browsershots

BUbiNG

Butterfly\/

BuzzSumo

CAAM\/[0-9]

CakePHP

CapsuleChecker

CaretNail

catexplorador

cb crawl

CC Metadata Scaper

Cerberian Drtrs

CERT\.at-Statistics-Survey

cg-eye

changedetection

ChangesMeter\/

Charlotte

CheckHost

checkprivacy

chkme\.com

Chirp\/[0-9]

CirrusExplorer\/

CISPA Vulnerability Notification

CJNetworkQuality

Clarsentia

clips\.ua\.ac\.be

Cloud mapping experiment

CloudEndure

CloudFlare-AlwaysOnline

Cloudinary\/[0-9]

cmcm\.com

coccoc

CommaFeed

Commons-HttpClient

Comodo SSL Checker

contactbigdatafr

convera

copyright sheriff

Covario-IDS

CrawlForMe\/[0-9]

cron-job\.org

Crowsnest

curb

Curious George

curl

cuwhois\/[0-9]

cybo\.com

DareBoost

DataparkSearch

dataprovider

Daum(oa)?[ \/][0-9]

DeuSu

developers\.google\.com\/\+\/web\/snippet\/

Digg

Dispatch\/

dlvr

DMBrowser

DNS-Tools Header-Analyzer

DNSPod-reporting

docoloc

Dolphin http client\/

DomainAppender

Donuts Content Explorer

dotSemantic

downforeveryoneorjustme

downnotifier\.com

DowntimeDetector

Dragonfly File Reader

drupact

Drupal \(\+http:\/\/drupal\.org\/\)

dubaiindex

EARTHCOM

Easy-Thumb

ec2linkfinder

eCairn-Grabber

ECCP

echocrawl

eContext\/

ElectricMonk

elefent

EMail Exractor

EmailWolf

Embed PHP Library

Embedly

europarchive\.org

evc-batch\/[0-9]

EventMachine HttpClient

Evidon

Evrinid

ExactSearch

ExaleadCloudview

Excel\/

Exif Viewer

Exploratodo

ezooms

facebookexternalhit

facebookplatform

fairshare

Faraday v

fasthttp

Faveeo

Favicon downloader

FavOrg

Feed Wrangler

Feedbin

FeedBooster

FeedBucket

FeedBunch\/[0-9]

FeedBurner

FeedChecker

Feedly

Feedspot

Feedwind\/[0-9]

feeltiptop

Fetch API

Fetch\/[0-9]

Fever\/[0-9]

findlink

findthatfile

FlipboardBrowserProxy

FlipboardProxy

FlipboardRSS

fluffy

flynxapp

forensiq

FoundSeoTool\/[0-9]

free thumbnails

FreeWebMonitoring SiteChecker

Funnelback

G-i-g-a-b-o-t

g00g1e\.net

GAChecker

ganarvisitas\/[0-9]

geek-tools

Genderanalyzer

Genieo

GentleSource

GetLinkInfo

getprismatic\.com

GetURLInfo\/[0-9]

Ghost Inspector

GigablastOpenSource

GIS-LABS

github\.com\/

Go [\d\.]* package http

Go-http-client

gofetch

GomezAgent

gooblog

Goodzer\/[0-9]

Google favicon

Google Keyword Suggestion

Google Keyword Tool

Google Page Speed Insights

Google PP Default

Google Search Console

Google Web Preview

Google-Adwords

Google-Apps-Script

Google-Calendar-Importer

Google-HotelAdsVerifier

Google-HTTP-Java-Client

Google-Publisher-Plugin

Google-SearchByImage

Google-Site-Verification

Google-Structured-Data-Testing-Tool

Google-Youtube-Links

google_partner_monitoring

GoogleDocs

GoogleHC\/

GoogleProducer

Gookey

GoScraper

GoSpotCheck

GoSquared-Status-Checker

gosquared-thumbnailer

grabify

Grammarly

grouphigh

grub-client

GTmetrix

GuzzleHttp

gvfs\/

HAA(A)?RTLAND http client

Hatena

hawkReader

HEADMasterSEO

HeartRails_Capture

heritrix

historious\/

hledejLevne\.cz\/[0-9]

Holmes

HootSuite Image proxy

Hootsuite-WebFeed\/[0-9]

HostTracker

ht:\/\/check

htdig

HTMLParser\/

http-get

HTTP-Header-Abfrage

http-kit

http-request\/

HTTP-Tiny

HTTP_Compression_Test

http_request2

http_requester

HttpComponents

httphr

HTTPMon

PEAR HTTPRequest

httpscheck

httpssites_power

httpunit

HttpUrlConnection

httrack

hosterstats

huaweisymantec

HubPages.*crawlingpolicy

HubSpot Connect

HubSpot Marketing Grader

HyperZbozi.cz Feeder

i2kconnect\/

ichiro

IdeelaborPlagiaat

IDG Twitter Links Resolver

IDwhois\/[0-9]

Iframely

igdeSpyder

IlTrovatore

ImageEngine\/

Imagga

imgsizer

InAGist

inbound\.li parser

InDesign%20CC

infegy

infohelfer

InfoWizards Reciprocal Link System PRO

Instapaper

inpwrd\.com

Integrity

integromedb

internet_archive

InternetSeer

internetVista monitor

intraVnews

IODC

IOI

iplabel

IPS\/[0-9]

ips-agent

IPWorks HTTP\/S Component

iqdb\/

Irokez

isitup\.org

iskanie

iZSearch

janforman

Jaunt\/

Jigsaw

Jobboerse

Jobg8 URL Monitor

jobo

Jobrapido

JS-Kit

Kaspersky Lab CFR link resolver

KeepRight OpenStreetMap Checker

KeyCDN

Keywords Research

KickFire

KimonoLabs\/

Kml-Google

knows\.is

KOCMOHABT

kouio

kulturarw3

KumKie

L\.webis

Larbin

LayeredExtractor

letsencrypt

LibVLC

libwww

Licorne Image Snapshot

Liferea\/

link checker

Link Valet

link_thumbnailer

LinkAlarm\/

linkCheck

linkdex

LinkExaminer

linkfluence

linkpeek

LinkTiger

LinkWalker

Lipperhey

livedoor ScreenShot

LoadImpactPageAnalyzer

LoadImpactRload

LongURL API

looksystems\.net

ltx71

lua-resty-http

lwp-trivial

lycos

LYT\.SR

mabontland

MagpieRSS

Mail.Ru

MailChimp

Mandrill

MapperCmd

marketinggrader

masscan\/[0-9]

Mediapartners-Google

MegaIndex\.ru

Melvil Rawi\/

MergeFlow-PageReader

Metaspinner

MetaURI

Microsearch

Microsoft-WebDAV-MiniRedir

Microsoft Data Access Internet Publishing Provider Protocol

Microsoft Office

Microsoft Windows Network Diagnostics

Mindjet

Miniflux

mixdata dot com

mixed-content-scan

mixnode

Mnogosearch

mogimogi

Mojolicious \(Perl\)

monitis

Monitority\/[0-9]

montastic

MonTools

Moreover

Morning Paper

mowser

MovableType

Mrcgiguy

mShots

MxToolbox\/

MVAClient

nagios

Najdi\.si\/

Needle\/

NETCRAFT

NetLyzer FastProbe

Netpursual

netresearch

NetShelter ContentScan

Netsparker

NetTrack

Netvibes

Neustar WPM

NeutrinoAPI

NewsBlur .*Finder

NewsGator

newsme

newspaper\/

NG-Search

nineconnections\.com

NLNZ_IAHarvester

Nmap Scripting Engine

node-superagent

node\.io

nominet\.org\.uk

Norton-Safeweb

Notifixious

notifyninja

nuhk

nutch

Nuzzel

nWormFeedFinder

Nymesis

Ocelli\/[0-9]

oegp

okhttp

Omea Reader

omgili

OMSC

Online Domain Tools

OpenCalaisSemanticProxy

Openstat\/

OpenVAS

Optimizer

Orbiter

OrgProbe\/[0-9]

Owler

ow\.ly

ownCloud News

OxfordCloudService\/[0-9]

Page Analyzer

Page Valet

page2rss

page_verifier

PagePeeker

Pagespeed\/[0-9]

Panopta

panscient

parsijoo

PayPal IPN

Pcore-HTTP

Pearltrees

peerindex

Peew

PhantomJS\/

PhantomJS Screenshoter

Photon\/

phpcrawl

phpservermon

Pi-Monster

ping\.blo\.gs\/

Pingability

Pingdom

Pingoscope

PingSpot

pinterest\.com

Pizilla

Ploetz \+ Zeller

Plukkie

PocketParser

Pompos

Porkbun

Port Monitor

postano

PostPost

postrank

PowerPoint\/

Priceonomics Analysis Engine

PritTorrent\/[0-9]

Prlog

probethenet

Project 25499

Promotion_Tools_www.searchenginepromotionhelp.com

prospectb2b

Protopage

proximic

pshtt, https scanning

PTST

PTST\/[0-9]+

Pulsepoint XT3 web scraper

Python-httplib2

python-requests

Python-urllib

Qirina Hurdler

QQDownload

Qseero

Qualidator.com SiteAnalyzer

Quora Link Preview

Qwantify

Radian6

RankSonicSiteAuditor

Readability

RealPlayer%20Downloader

RebelMouse

redback\/

Redirect Checker Tool

ReederForMac

request\.js

ResponseCodeTest\/[0-9]

RestSharp

RetrevoPageAnalyzer

Riddler

Rival IQ

Robosourcer

Robozilla\/[0-9]

ROI Hunter

RPT-HTTPClient

RSSOwl

safe-agent-scanner

SalesIntelligent

SauceNAO

SBIder

scalaj-http

Scoop

scooter

ScoutJet

ScoutURLMonitor

Scrapy

ScreenShotService\/[0-9]

Scrubby

search\.thunderstone

SearchSight

Seeker

semanticdiscovery

semanticjuice

Semiocast HTTP client

SEO Browser

Seo Servis

seo-nastroj.cz

Seobility

SEOCentro

SeoCheck

SeopultContentAnalyzer

Server Density Service Monitoring

servernfo\.com

Seznam screenshot-generator

Shelob

Shoppimon Analyzer

ShoppimonAgent\/[0-9]

ShopWiki

ShortLinkTranslate

shrinktheweb

SilverReader

SimplePie

SimplyFast

Site-Shot\/

Site24x7

SiteBar

SiteCondor

siteexplorer\.info

SiteGuardian

Siteimprove\.com

Sitemap(s)? Generator

SiteMonitor

Siteshooter B0t

SiteTruth

sitexy\.com

SkypeUriPreview

Slack\/

slider\.com

slurp

SMRF URL Expander

SMUrlExpander

Snappy

SniffRSS

sniptracker

Snoopy

sogou web

SortSite

sovereign\.ai

spaziodati

Specificfeeds

speedy

SPEng

Spinn3r

spray-can

Sprinklr

sqlmap

spyonweb

Sqworm

SSL Labs

ssl-tools

StackRambler

Statastico\/

StatusCake

Stratagems Kumo

Stroke.cz

StudioFACA

suchen

summify

Super Monitoring

Surphace Scout

SwiteScraper

Symfony2 BrowserKit

SynHttpClient-Built

Sysomos

Symfony BrowserKit

T0PHackTeam

Tarantula\/

Taringa UGC

Tenon\.io

teoma

terrainformatica\.com

Test Certificate Info

Tetrahedron\/[0-9]

The Drop Reaper

The Expert HTML Source Viewer

theinternetrules

theoldreader\.com

Thumbshots

ThumbSniper

TinEye

Tiny Tiny RSS

TLSProbe\/

topster

touche.com

Traackr.com

TrapitAgent

truwoGPS

TulipChain

tweetedtimes\.com

Tweetminster

Tweezler\/

Twikle

Twingly

ubermetrics-technologies

uclassify

uCrawlr\/

UdmSearch

UniversalFeedParser

Untiny

UnwindFetchor

updated

Upflow

Uptimia

URLChecker

URLitor.com

urlresolver

Urlstat

UrlTrends Ranking Updater

Vagabondo

vBSEO

via ggpht\.com GoogleImageProxy

VidibleScraper\/

visionutils

vkShare

voltron

voyager\/

VSAgent\/[0-9]

VSB-TUO\/[0-9]

VYU2

w3af\.org

W3C-checklink

W3C-mobileOK

W3C_I18n-Checker

W3C_Unicorn

wangling

WatchMouse

WbSrch\/

web-capture\.net

Web-Monitoring

Web-sniffer

Webauskunft

WebCapture

WebClient\/

webcollage

WebCookies

WebCorp

WebDoc

WebFetch

WebImages

WebIndex

webkit2png

webmastercoffee

webmon

webscreenie

Webshot

Website Analyzer\/

websitepulse agent

websitepulse[+ ]checker

Websnapr\/

Webthumb\/[0-9]

WebThumbnail

WeCrawlForThePeace

WeLikeLinks

WEPA

WeSEE

wf84

wget

WhatsApp

WhatsMyIP

WhatWeb

WhereGoes\?

Whibse

Whynder Magnet

Windows-RSS-Platform

WinHttpRequest

wkhtmlto

wmtips

Woko

Word\/

WordPress\/

wotbox

WP Engine Install Performance API

wpif

wprecon\.com survey

WPScan

wscheck

WWW-Mechanize

www\.monitor\.us

XaxisSemanticsClassifier

Xenu Link Sleuth

XING-contenttabreceiver\/[0-9]

XmlSitemapGenerator

xpymep([0-9]?)\.exe

Y!J-(ASR|BSC)

Yaanb

yacy

Yahoo Ad monitoring

Yahoo Link Preview

YahooCacheSystem

YahooYSMcm

YandeG

Yandex(?!Search)

yanga

yeti

YLT

Yo-yo

Yoleo Consumer

yoogliFetchAgent

YottaaMonitor

yourls\.org

Zao

Zemanta Aggregator

Zend\\Http\\Client

Zend_Http_Client

zgrab

ZnajdzFoto

ZyBorg

[a-z0-9\-_]*((?<!cu)bot|crawler|archiver|transcoder|spider|uptime|validator|fetcher)

Источник здесь https://github.com/JayBizzle/Crawler-Detect .

[dexion]

6 часов назад, agefremov сказал:

Внесу свои 5 копеек

Боты:

  Показать контент

TinEye-bot/0.51 (see http://www.tineye.com/crawler.html)
	Whoismindbot/1.0 (+http://www.whoismind.com/bot.html)
	Mozilla/5.0 (compatible; MSIE 8.0; Windows NT 6.3; www.alertra.com)
	R6_FeedFetcher(www.radian6.com/crawler)
	CheckHost (http://check-host.net/)
	YandeG 1.03
	yacybot (/global; amd64 Linux 3.16-0.bpo.2-amd64; java 1.7.0_65; Europe/en) http://yacy.net/bot.html
	SEMrushBot
	Protopage/3.0 (http://www.protopage.com)
	yacybot (/global; amd64 Linux 3.16.0-4-amd64; java 1.7.0_75; Europe/en) http://yacy.net/bot.html
	StatoolsBot (+http://www.statools.com/bot.html)
	adidxbot/2.0 (+http://search.msn.com/msnbot.htm)
	Mozilla/5.0 (compatible; Mail.RU_Bot/2.0; +http://go.mail.ru/help/robots)
	Mozilla/5.0 (Linux; Android 5.0; Nexus 5 Build/LRX21O) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/46.0.2490.76 Mobile Safari/537.36 PTST/281
	TurnitinBot (https://turnitin.com/robot/crawlerinfo.html)
	Scrapy/1.0.5 (+http://scrapy.org)
	yacybot (/global; amd64 Linux 4.4.0-31-generic; java 1.8.0_91; Europe/en) http://yacy.net/bot.html
	XmlSitemapGenerator - http://xmlsitemapgenerator.org
	Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.0) Match by Siteimprove.com
	Mozilla/5.0 (compatible; SeznamBot/3.2; +http://napoveda.seznam.cz/en/seznambot-intro/)
	WatchMouse/18990 (http://watchmouse.com/ ; gab)
	WatchMouse/8.4.0.3 (http://watchmouse.com/ ; hkhkg02.watchmouse.net)
	Mozilla/5.0 (compatible; LinkpadBot/1.06; +http://www.linkpad.ru)
	Mozilla/5.0 (compatible; heritrix/1.14.2 +http://rjpower.org)
	yacybot (webportal/global; amd64 Linux 2.6.32-5-amd64; java 1.6.0_18; Europe/en) http://yacy.net/bot.html
	PercolateCrawler/4 ([email protected])
	msnbot-UDiscovery/2.0b (+http://search.msn.com/msnbot.htm)
	Mozilla/5.0 (compatible; MSIE or Firefox mutant; not on Windows server;) Daum 4.1
	Mozilla/5.0 (compatible; spbot/4.0.3; +http://www.seoprofiler.com/bot )
	Mozilla/5.0 (compatible; LoadTimeBot/0.9; +http://www.loadtime.net/bot.html)
	UnwindFetchor/1.0 (+http://www.gnip.com/)
	nrsbot/5.0(loopip.com/robot.html)
	SemrushBot/0.9
	Mozilla/5.0 (compatible; UASlinkChecker/2.1; +https://udger.com/support/UASlinkChecker)
	yacybot (/global; amd64 Linux 3.11.10-21-desktop; java 1.7.0_51; America/en) http://yacy.net/bot.html
	Netvibes (http://www.netvibes.com)
	Acoon v4.1.0 (www.acoon.de)
	msnbot/2.0b (+http://search.msn.com/msnbot.htm).
	Mozilla/5.0 (compatible; ltbot/3.2.0.10 +http://www.kdsl.tu-darmstadt.de/de/kdsl/research-program/crawling-and-semantic-structuring/)
	smart.apnoti.com Robot/v1.34 (http://smart.apnoti.com/en/aboutApnotiWebCrawler.html)
	HubSpot Links Crawler 2.0 http://www.hubspot.com/
	yacybot (/global; x86 Windows XP 5.1; java 1.7.0_55; Europe/de) http://yacy.net/bot.html
	Mozilla/5.0 (compatible; DCPbot/1.0; +http://domains.checkparams.com/)
	Mozilla/5.0 (compatible; Exabot/3.0 (BiggerBetter); +http://www.exabot.com/go/robot)
	WebDoc (abuse-webdoc at service.moquadv.com)
	coccoc
	Mozilla/5.0 (compatible; spbot/4.0b; +http://www.seoprofiler.com/bot )
	Mozilla/5.0 (compatible; ExchangleBot/3.0; +https://www.exchangle.com/exchangling)
	Mozilla/5.0 (compatible; Qwantify/2.0n; +https://www.qwant.com/)
	OOZBOT/0.20 ( Setooz výrazný ako say-th-uuz, znamená mosty. ; http://www.setooz.com/oozbot.html ; agentname at setooz dot_com )
	SpeedySpider - http://www.entireweb.com
	Mozilla/5.0 (compatible; Heurekabot/3.1; +http://sluzby.heureka.cz/)
	crawler for netopian (http://www.netopian.co.uk/)
	L.webis/0.51 (http://webalgo.iit.cnr.it/index.php?pg=lwebis)
	Influencebot/0.9; (Automatic classification of websites; http://www.influencebox.com/; [email protected])
	Baiduspider+(+http://www.baidu.com/search/spider.htm)
	Mozilla/5.0 (X11; compatible; semantic-visions.com crawler; HTTPClient 3.1)
	Mozilla/5.0 (Windows; U; Windows NT 6.0; en-US; rv:1.9.2.28) Gecko/20120306 Firefox/99.0 YottaaMonitor
	Scrapy/1.1.1 (+http://scrapy.org)
	Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/27.0.1453.116 Safari/537.36 HubSpot Marketing Grader
	RyzeCrawler/1.1.1 ( http://www.domain2day.nl/crawler/)
	eBot / v.1.0a (http://alfa.elchron.cz)
	Sogou News Spider/4.0(+http://www.sogou.com/docs/help/webmasters.htm#07)
	Crowsnest/0.5 (+http://www.crowsnest.tv/)
	Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.4 (KHTML, like Gecko; Google Page Speed Insights) Chrome/22.0.1229 Safari/537.4
	DoCoMo/2.0 N902iS(c100;TB;W24H12)(compatible; moba-crawler; http://crawler.dena.jp/)
	Yeti/1.1 (NHN Corp.; http://help.naver.com/robots/)
	Experibot_v1
	Nekstbot - http://www.ipipan.waw.pl/nekst/nekstbot/
	Mozilla/5.0 (compatible; MJ12bot/v1.4.5; http://www.majestic12.co.uk/bot.php?+)
	Mozilla/5.0 (X11; Linux i686) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/30.0.1599.101 Safari/537.36; SSL-Crawler: http://crawler.dcsec.uni-hannover.de
	CorpusCrawler 2.0.19 (http://corpora.fi.muni.cz/crawler/);Project:CzCorpus
	Mozilla/5.0 (compatible; Semager/1.4; http://www.semager.de/blog/semager-bots/)
	Mozilla/5.0 (compatible; dlcbot/0.1; +http://www.drlinkcheck.com/)
	yacybot (freeworld/global; amd64 Linux 3.11.10-21-desktop; java 1.7.0_51; Europe/de) http://yacy.net/bot.html
	A6-Indexer/1.0 (http://www.a6corp.com/a6-web-scraping-policy/)
	CopperEgg/RevealUptime/DallasTX(linode)
	Mozilla/5.0 (compatible; Qwantify/2.1dw; +https://www.qwant.com/)/*
	Curious George - www.analyticsseo.com
	GozaikBot (www.gozaik.com;[email protected];www.gozaik.com/gozaikbot.html)
	Pu_iN Crawler (+http://semanticjuice.com/)
	Mozilla/5.0 (compatible; OpenindexDeepSpider/Nutch-1.5-dev; +http://www.openindex.io/en/webmasters/spider.html; systemsATopenindexDOTio)
	yacybot (freeworld/global; i386 Linux 3.0.0-17-generic-pae; java 1.6.0_23; Europe/en) http://yacy.net/bot.html
	yacybot (-global; amd64 FreeBSD 9.2-RELEASE-p10; java 1.7.0_65; Europe/en) http://yacy.net/bot.html
	FAST Enteprise Crawler/6 (www dot fastsearch dot com)
	Mozilla/5.0 (compatible; parsijoo-update-crawler; +http://www.parsijoo.ir/; [email protected])
	yacybot (freeworld/global; amd64 Windows Server 2008 6.0; java 1.7.0_25; Europe/en) http://yacy.net/bot.html
	Inspingbot/1.0 (+https://www.insping.com/)
	OrgProbe/0.9.4 (+http://www.blocked.org.uk)
	Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; Trident/5.0); 360Spider
	Mozilla/5.0 (Windows NT 6.1; WOW64; Trident/7.0; rv:11.0) like Gecko PTST/276
	Mozilla/5.0 (Windows NT 6.3; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/42.0.2311.90 Safari/537.36 PTST/276
	Mozilla/5.0 (compatible; SemrushBot/0.97~bl; +http://www.semrush.com/bot.html)
	Feedspot http://www.feedspot.com
	Zookabot/2.0;++http://zookabot.com
	drupact/0.7; http://www.arocom.de/drupact
	RobotsChecker/0.6 (+http://www.blocked.org.uk)
	yacybot (/global; i386 Linux 3.13.0-37-generic; java 1.7.0_65; Europe/en) http://yacy.net/bot.html
	Scrapy/1.0.3 (+http://scrapy.org)
	Testomatobot/1.0 (Linux x86_64; +http://www.testomato.com/testomatobot) minicrawler/4.0.0~beta12
	Slack-ImgProxy 0.66 (+https://api.slack.com/robots)
	Mozilla/5.0 (compatible; WebCookies/1.0; +https://webcookies.org/faq/#agent)
	NalezenCzBot/1.0 (http://www.nalezen.cz)
	yacybot (freeworld/global; amd64 Windows Server 2008 6.0; java 1.7.0_03; Europe/en) http://yacy.net/bot.html
	Mozilla/5.0 (compatible; AhrefsBot/5.1; +http://ahrefs.com/robot/)
	yacybot (freeworld/global; amd64 Windows 8 6.2; java 1.7.0_51; Europe/de) http://yacy.net/bot.html
	yacybot (/global; amd64 Windows 7 6.1; java 1.8.0_65; Europe/de) http://yacy.net/bot.html
	Mozilla/4.0 compatible ZyBorg/1.0 ([email protected]; http://www.WISEnutbot.com)
	Mozilla/5.0 (en-us) AppleWebKit/537.36(KHTML, like Gecko; Google-Adwords-DisplayAds-WebRender;) Chrome/27.0.1453Safari/537.36
	Crawler powered by contentDetection (www.mindup.de)
	PagePeeker.com (info: http://pagepeeker.com/robots)
	Mozilla/5.0 (compatible; Linux x86_64; Mail.RU_Bot/Fast/2.0; +http://go.mail.ru/help/robots)
	coccoc/1.0 (http://help.coccoc.com/)
	Favicon downloader (+https://favico.be/bot.html)
	Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0) ([email protected])
	Mozilla/5.0 (compatible; DIY-SEOBot/0.1a; +http://www.upcity.com/bot.html)
	Mozilla/5.0 (Windows NT 6.3; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/49.0.2623.112 Safari/537.36 PTST/276
	Mozilla/5.0 (compatible; theoldreader.com)
	Mozilla/5.0+(compatible; UptimeRobot/2.0; http://www.uptimerobot.com/)
	hawkReader/1.8 (Link Parser; http://www.hawkreader.com/; Allow like Gecko) Build/f2b2566
	Mozilla/5.0 (compatible; heritrix/1.12.1 +http://www.webarchiv.cz)
	Mozilla/5.0 (compatible; YandexImages/3.0; +http://yandex.com/bots)
	yacybot (/global; amd64 Linux 3.14.32-xxxx-grs-ipv6-64; java 1.7.0_75; Europe/en) http://yacy.net/bot.html
	Sogou web spider/4.0(+http://www.sogou.com/docs/help/webmasters.htm#07)
	yacybot (/global; amd64 Windows 7 6.1; java 1.8.0_45; Europe/de) http://yacy.net/bot.html
	larbin_2.6.3 [email protected]
	yacybot (freeworld/global; amd64 Linux 3.10.17-gentoo; java 1.7.0_45; UTC/en) http://yacy.net/bot.html
	MeMoNewsBot/2.0 (http://www.memonews.com/en/crawler)
	Mozilla/5.0 (compatible; MSIE 10.0; Windows NT 6.1; WOW64; Trident/6.0) PTST/281
	Mozilla/5.0 (compatible; DCPbot/1.5; +http://domains.checkparams.com/)
	Mozilla/5.0 (compatible; SemrushBot/0.97; +http://www.semrush.com/bot.html)
	BlackBerry9000/4.6.0.167 Profile/MIDP-2.0 Configuration/CLDC-1.1 VendorID/102 ips-agent
	Mozilla/5.0 (compatible; Baiduspider-cpro; +http://www.baidu.com/search/spider.html)
	yacybot (/global; amd64 Linux 3.2.0-4-amd64; java 1.7.0_65; Europe/en) http://yacy.net/bot.html
	Mozilla/5.0 (compatible; electricmonk/3.1.1 +https://www.duedil.com/our-crawler/)
	Hatena Antenna/0.5 (http://a.hatena.ne.jp/help)
	Backlink-Ceck.de (+http://www.backlink-check.de/bot.html)
	Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; WOW64; Trident/5.0) commoncrawl.org/research//Nutch-1.7-SNAPSHOT
	Mozilla/5.0 (compatible; Steeler/3.5; http://www.tkl.iis.u-tokyo.ac.jp/~crawler/)
	LapozzBot/1.4 (+http://robot.lapozz.com)
	Mozilla/5.0 (WhatsMyIP.org HTTP_Compression_Test) http://whatsmyip.org/ua
	Mozilla/5.0 (compatible; MSIE or Firefox mutant; not on Windows server;) Daumoa/4.0 (Following Mediapartners-Google)
	Mozilla/5.0 (compatible; ExaleadCloudview/6;)
	ADmantX Platform Semantic Analyzer - ADmantX Inc. - www.admantx.com - [email protected]
	msnbot/1.0 (+http://search.msn.com/msnbot.htm)
	Mozilla/5.0 (Windows NT 5.1) AppleWebKit/537.4 (KHTML, like Gecko) Chrome/22.0.1229.79 Safari/537.4 LinkTiger 2.0
	Pixray-Seeker/1.1 (Pixray-Seeker; [email protected])
	mbot v.1.16
	MXT/Nutch-1.12-SNAPSHOT (http://t.co/GSRLLKex24; informatique at mixdata dot com)
	Testomatobot/1.0 (Linux x86_64; +http://www.testomato.com/testomatobot) minicrawler/3.0.0
	polybot 1.0 (http://cis.poly.edu/polybot/)
	Mozilla/5.0 (compatible; YandexMetrika/2.0; +http://yandex.com/bots mtweb01t.yandex.ru)
	Mozilla/5.0 (compatible; memoryBot/1.21.46 +http://internetmemory.org/en/)
	Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_1) AppleWebKit/600.2.5 (KHTML, like Gecko) Version/8.0.2 Safari/600.2.5 (Applebot/0.1)
	yacybot (amd64 Linux 2.6.26-2-amd64; java 1.6.0_0; Europe/en) http://yacy.net/bot.html
	yacybot (/global; x86 Windows 8.1 6.3; java 1.8.0_45; America/en) http://yacy.net/bot.html
	Mozilla/5.0 (compatible; startmebot/1.0; +http://www.start.me/bot)
	yacybot (amd64 Windows 7 6.1; java 1.6.0_21; Europe/fr) http://yacy.net/bot.html
	yacybot (/global; amd64 Windows 7 6.1; java 1.8.0_101; Asia/ru) http://yacy.net/bot.html
	yacybot (freeworld/global; amd64 Linux 3.0.0-17-generic; java 1.6.0_23; Europe/de) http://yacy.net/bot.html
	alexa v0.1.4 (http://www.openwebspider.org/)
	http://arachnode.net 1.4
	Photon/1.0
	NetpeakCheckerBot
	GIDBot/3.0 (+http://www.gidnetwork.com/tools/gzip-test.php)
	Yandex/1.01.001 (compatible; Win16; P)
	w3dt.net httphr/2.0
	yacybot (/global; amd64 Windows 7 6.1; java 1.8.0_51; Europe/de) http://yacy.net/bot.html
	iqdb/0.1 (+http://iqdb.org/)
	Mozilla/5.0 (compatible; GimmeUSAbot/1.0; +https://gimmeusa.com/pages/crawler)
	Motoricerca-Robots.txt-Checker/1.0 (http://tool.motoricerca.info/robots-checker.phtml)
	Mozilla/5.0 (compatible; Tagoobot/3.0; +http://www.tagoo.ru)
	Mozilla/5.0 (Windows NT 6.1; WOW64; rv:26.0) Gecko/20100101 Firefox/26.0 Evidon ([email protected])
	Mozilla/5.0 (compatible; Nmap Scripting Engine; http://nmap.org/book/nse.html)
	yacybot (i386 Linux 2.6.32-22-generic; java 1.6.0_20; Europe/de) http://yacy.net/bot.html
	Mozilla/5.0 (Linux; Android 4.4.3; HTC One Build/KTU84L) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/40.0.2125.111 Mobile Safari/537.36 DareBoost
	mindUpBot (datenbutler.de)
	Mozilla/5.0 (compatible; monitis - premium monitoring service; http://www.monitis.com)
	Mozilla/5.0 (compatible; RankSonicSiteAuditor/1.0; +https://ranksonic.com/ranksonic_sab.html)
	Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/40.0.2125.111 Safari/537.36 DareBoost
	Zookabot/2.1;++http://zookabot.com
	yacybot (freeworld/global; amd64 Windows 7 6.1; java 1.6.0_24; Europe/de) http://yacy.net/bot.html
	Mozilla/5.0 (compatible; Google-Structured-Data-Testing-Tool +http://developers.google.com/structured-data/testing-tool/)
	Mozilla/5.0 (compatible; Plukkie/1.2; http://www.botje.com/plukkie.htm)
	focusbot/1.1
	Mozilla/5.0 (compatible; idmarch Automatic.beta/1.2; +http://www.idmarch.org/bot.html)
	CorpusCrawler 2.0.14 (http://corpora.fi.muni.cz/crawler/)
	Mozilla/5.0 (compatible; XML Sitemaps Generator; https://www.xml-sitemaps.com) Gecko XML-Sitemaps/1.0
	ImplisenseBot 1.1
	Promotion_Tools_www.searchenginepromotionhelp.com
	Aboundex/0.2 (http://www.aboundex.com/crawler/)
	CorpusCrawler 2.0.22 (http://corpora.fi.muni.cz/crawler/);Project:CzCorpus
	yacybot (/global; amd64 no-os-name no-os-version; java no-java-version; Europe/en) http://yacy.net/bot.html
	LumpImageSearch/0.1 (+http://lump.co/about/bot)
	Mozilla/4.0 (compatible; NaverBot/1.0; http://help.naver.com/customer_webtxt_02.jsp)
	Mozilla/5.0 (compatible; UASlinkChecker/2.0; +http://udger.com/support/UASlinkChecker)
	GAChecker (+http://www.gachecker.com)
	yacybot (freeworld/global; x86 Windows 7 6.1; java 1.7.0_25; Europe/de) http://yacy.net/bot.html
	Influencebot/0.9; (Automatic classification of websites; http://www.influencebox.com/; [email protected])User-Agent: Mozilla/5.0 (X11; Linux i686; rv:9.0) Gecko/20100101 Firefox/9.0
	Experibot_v1 (https://dl.dropboxusercontent.com/u/8024465/site/Info.html)
	DNSPod-reporting(http://www.dnspod.cn/reporting)
	yacybot (freeworld/global; i386 Linux 3.2.0-23-generic; java 1.6.0_27; Europe/en) http://yacy.net/bot.html
	Mozilla/5.0 (Windows NT 6.3;compatible; Leikibot/1.0; +http://www.leiki.com)
	PostPost/1.0 (+http://postpo.st/crawlers)
	envolk/1.7 (+http://www.envolk.com/envolkspiderinfo.html)
	Snapbot/1.0 (Snap Shots, +http://www.snap.com)
	Mozilla/5.0 (Linux; Android 4.0.4; Galaxy Nexus Build/IMM76B) AppleWebKit/537.36 (KHTML, like Gecko; Google-Publisher-Plugin) Chrome/27.0.1453 Mobile Safari/537.36
	Mozilla/5.0 (compatible; Uptimebot/0.2.18; +http://www.uptime.com/uptimebot)
	MIA DEV/search:robot/0.0.1 (This is the MIA Bot - crawling for mia research project. If you feel unhappy and do not want to be visited by our crawler send an email to [email protected]; http://spider.neofonie.de; [email protected])
	Mozilla/5.0 (Windows NT 6.1; Trident/7.0; rv:11.0; PTST 2.386) like Gecko
	Jyxobot/1
	WebAlta Crawler/2.0 (http://www.webalta.net/ru/about_webmaster.html) (Windows; U; Windows NT 5.1; ru-RU)
	GC3pro+dir SEO Tools - Vers. 3.00b - For more informations: http://chkme.com/
	Mozilla/5.0 (X11; Linux x86_64; rv:45.0; GTmetrix https://gtmetrix.com/) Gecko/20100101 Firefox/45.0
	Mozilla/5.0 (Windows NT 6.1; WOW64; rv:18.0) Gecko/20100101 Firefox/18.0 AppEngine-Google; (+http://code.google.com/appengine; appid: s~aeshortener)
	Priceonomics Analysis Engine - Fetch/1.0
	Ruky-Roboter (Version: 1.06, powered by www.ruky.de +http://www.ruky.de/bot.html)
	Baiduspider+(+http://help.baidu.jp/system/05.html)
	Openstat/0.1
	Yandex/1.01.001 (compatible; Win16; m)
	Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_1) AppleWebKit/600.2.5 (KHTML, like Gecko) Version/8.0.2 Safari/600.2.5 (Applebot/0.1; +http://www.apple.com/go/applebot)
	Mozilla/5.0 (compatible; OpenindexSpider/Nutch-1.5-dev; +http://www.openindex.io/en/webmasters/spider.html)
	Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; Multiviewbot
	CERT.at-Statistics-Survey/1.0 (http://www.cert.at/about/consec/content.html)
	Mozilla/5.0 (compatible; pmoz.info ODP link checker; +http://pmoz.info/doc/botinfo.htm)
	yacybot (freeworld/global; x86_64 Mac OS X 10.6.8; java 1.6.0_29; Asia/ru) http://yacy.net/bot.html
	gonzo/1[P] (+http://www.suchen.de/faq.html)
	MixBot (+http://t.co/GSRLLKex24)
	yacybot (/global; amd64 Linux 3.2.0-4-amd64; java 1.7.0_60; Europe/en) http://yacy.net/bot.html
	Mozilla/5.0 (compatible; imbot/0.1 +http://internetmemory.org/en/
	Mozilla/5.0 (en-us) AppleWebKit/525.13 (KHTML, like Gecko; Google Web Preview) Version/3.1 Safari/525.13
	Mozilla/5.0 (compatible; Pi-Monster; https://pricepi.com/)
	ThumbSniper (http://thumbsniper.com)
	Shelob ([email protected])
	Mozilla/5.0 (Macintosh; Intel Mac OS X 10_7_3) AppleWebKit/537.36 (KHTML, like Gecko, Google-Publisher-Plugin) Chrome/27.0.1453 Safari/537.36
	Mozilla/5.0 (compatible; KaloogaBot; http://www.kalooga.com/info.html?page=crawler)
	yacybot (/global; arm Linux 4.1.13+; java 1.8.0_40-internal; Etc/de) http://yacy.net/bot.html
	CorporateNewsSearchEngine/Nutch-1.7 (http://pibs.co/news-search-engine)
	Mozilla/5.0 (compatible; MSIE 10.0; Windows NT 6.1; WOW64; Trident/6.0) PTST/276
	Yandex/1.01.001 (compatible; Win16; I)
	FlightDeckReportsBot/2.0 (http://www.flightdeckreports.com/pages/bot)
	Scrapy/0.24.4 (+http://scrapy.org)
	ADmantX Platform Semantic Analyzer US - Turn - ADmantX Inc. - www.admantx.com - [email protected]
	Kyoto-Tohoku-Crawler/v1 (Mozilla-compatible; [email protected]; http://nlp.ist.i.kyoto-u.ac.jp/?crawling-kt)
	Mozilla/5.0 (compatible; Scarlett/ 1.0; +http://www.ellerdale.com/crawler.html)
	Mozilla/5.0 (compatible; NetcraftSurveyAgent/1.0; [email protected])
	www.deadlinkchecker.com Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/46.0.2490.86 Safari/537.36
	http://arachnode.net 1.2
	Mozilla/5.0 (compatible; Plukkie/1.5; http://www.botje.com/plukkie.htm)
	yacybot (freeworld/global; arm Linux 4.4.11-v7+; java 1.7.0_101; Etc/en) http://yacy.net/bot.html
	Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.0.16) Gecko/2009121601 Ubuntu/9.04 (jaunty) Firefox/3.0.16 Specificfeeds- http://www.specificfeeds.com
	Mozilla/5.0 (compatible; evc-batch/2.0)
	Orbiter/1.2 (http://dailyorbit.com/)
	crawler4j (https://github.com/yasserg/crawler4j/)
	Mozilla/5.0 (compatible; SEOdiver/1.0; +http://www.seodiver.com/bot)
	Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/51.0.2704.79 Safari/537.36 (https://shrinktheweb.com)
	Scopia crawler 1.0 (+http://www.scopia.co)
	yacybot (i386 Linux 2.6.23; java 1.6.0_06; Europe/en) http://yacy.net/bot.html
	Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.1.2) Gecko/20090729 Firefox/3.5.2 (.NET CLR 3.5.30729; Diffbot/0.1; +http://www.diffbot.com)
	Mozilla/5.0 (compatible; LXRbot/1.0;http://www.lxrmarketplace.com/,[email protected])
	yacybot (freeworld/global; amd64 Linux 3.8.13-gentoo; java 1.7.0_21; Europe/de) http://yacy.net/bot.html
	Mozilla/5.0 (compatible; Sitemap Generator/1.3; +http://www.check-domains.com/sitemap/index.php)
	DuckDuckBot/1.1; (+http://duckduckgo.com/duckduckbot.html)
	Mozilla/5.0 (compatible; BLEXBot/1.0; +http://webmeup-crawler.com/)
	Slack-ImgProxy 0.59 (+https://api.slack.com/robots)
	Mozilla/5.0 (compatible; Ezooms/1.0; [email protected])
	Mozilla/5.0 (compatible; YandexPagechecker/2.0; +http://yandex.com/bots)
	CoinCornerBot/1.1 ( https://www.coincorner.com/BitcoinBot)
	yacybot (freeworld/global; amd64 Linux 3.8.0-21-generic; java 1.6.0_27; Pacific/en) http://yacy.net/bot.html
	ScreenerBot Crawler Beta 2.0 (+http://www.ScreenerBot.com)
	gonzo1[P] +http://www.suchen.de/faq.html
	Mozilla/5.0 (compatible; MSIE or Firefox mutant;) Daum 4.1
	Mozilla/5.0 (compatible; 008/0.83; http://www.80legs.com/spider.html;) Gecko/2008032620
	Sogou web spider/4.02525A
	Visbot/2.0 (+http://www.visvo.com/en/webmasters.jsp;[email protected])
	Mozilla/5.0 (compatible; AcoonBot/4.10.8; +http://www.acoon.de/robot.asp)
	WatchMouse/18990 (http://watchmouse.com/ ; d3.watchmouse.com)
	Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/534.34 (KHTML, like Gecko) Qt/4.8.1 Safari/534.34 ShoppimonAgent/1.0 ([email protected])
	webinatorbot 1.1; +http://www.webinator.de
	findlinks/2.0.5 (+http://wortschatz.uni-leipzig.de/findlinks/)
	WikiDo/1.1 (http://wikido.com; [email protected])
	yacybot (freeworld/global; amd64 Linux 3.5.0-27-generic; java 1.7.0_03; Europe/de) http://yacy.net/bot.html
	Mozilla/5.0 (compatible; Uptimebot/0.1.73; +http://www.uptime.com/uptimebot)
	Semantifire1/0.20 ( -- ; http://www.setooz.com/oozbot.html ; agentname at setooz dot_com )
	Speedy Spider (http://www.entireweb.com/about/search_tech/speedy_spider/)
	Mozilla/5.0 compatible; yelpspider/yelpspider-1.0 (Crawlerbot run by Yelp Inc; yelpbot at yelp dot com)
	Mozilla/5.0 (compatible; OpenHoseBot/2.1; +http://www.openhose.org/bot.html)
	Mozilla/5.0 (compatible; emefgebot/beta; +http://emefge.de/bot.html)
	Mozilla/5.0 (compatible; OpenindexShallowSpider/Nutch-1.5-dev; +http://www.openindex.io/en/webmasters/spider.html; systemsATopenindexDOTio)
	Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/49.0.2623.75 Safari/537.36 Google (+https://developers.google.com/+/web/snippet/)
	Mozilla/5.0 (compatible; MetaJobBot; http://www.metajob.de/crawler)
	CCBot/1.0 (+http://www.commoncrawl.org/bot.html)
	Lijit Crawler (+http://www.lijit.com/robot/crawler)
	baypup/1.1 (Baypup; http://www.baypup.com/; [email protected])
	Mozilla/5.0 (FauBot/0.1; +http://buzzvolume.com/fau/)
	Mozilla/5.0 (compatible; NLNZ_IAHarvester2016/3.3.0 +https://natlib.govt.nz/publishers-and-authors/web-harvesting/domain-harvest)
	yacybot (/global; amd64 Linux 3.12.1; java 1.7.0_65; Europe/en) http://yacy.net/bot.html
	yacybot (amd64 Windows 7 6.1; java 1.6.0_18; Europe/de) http://yacy.net/bot.html
	SEOENGWorldBot/1.0 (+http://www.seoengine.com/seoengbot.htm)
	Mozilla/5.0 (compatible; Finderbots finder bot; +http://wiki.github.com/bixo/bixo/bixocrawler; [email protected])
	Pompos/1.3 http://dir.com/pompos.html
	Mozilla/5.0 (compatible; Gimme60bot/1.0; +http://gimme60.com) Firefox/16.0
	Mozilla/5.0 (compatible; MJ12bot/v1.4.6; http://mj12bot.com/)
	Mozilla/5.0 (compatible; Sysomos/1.0; +http://www.sysomos.com/; Sysomos)
	urlfan-bot/1.0; +http://www.urlfan.com/site/bot/350.html
	al_viewer ([email protected])
	LoadImpactRload/3.1.1 (Load Impact; http://loadimpact.com);
	findlinks/2.0.2 (+http://wortschatz.uni-leipzig.de/findlinks/)
	Mozilla/5.0 (compatible; spbot/3.1; +http://www.seoprofiler.com/bot )
	Mozilla/5.0 (compatible; AcoonBot/4.11.0; +http://www.acoon.de/robot.asp)
	WillyBot/1.1 (http://www.willyfogg.com/info/willybot)
	Norton-Safeweb
	WMCAI-robot (http://www.topicmaster.jp/wmcai/crawler.html)
	rogerbot/1.1 (http://moz.com/help/guides/search-overview/crawl-diagnostics#more-help, [email protected])
	Szukacz/1.5 (robot; www.szukacz.pl/jakdzialarobot.html; [email protected])
	rogerbot/1.0 (http://www.seomoz.org/dp/rogerbot, [email protected])
	ROR Sitemap Generator (http://www.rorweb.com)
	http://domino.research.ibm.com/comm/research_projects.nsf/pages/sai-crawler.callingcard.html
	MozillaTest/5.0 (compatible; YodaoBot/1.0; http://www.yodao.com/help/webmaster/spider/; )
	Mozilla/5.0 (compatible; Peew/1.0; http://www.peew.de/crawler/)
	Mozilla/5.0 (compatible; Website Analyzer/1.1; +http://www.check-domains.com/website-analysis/website-analyzer.php)
	Mozilla/5.0 (compatible; Gluten Free Crawler/1.0; +http://glutenfreepleasure.com/)
	PagePeeker.com
	CorpusCrawler 2.0.10 (http://corpora.fi.muni.cz/crawler/)
	yacybot (/global; amd64 Linux 3.2.0-4-amd64; java 1.7.0_101; Europe/cs) http://yacy.net/bot.html
	yacybot (/global; amd64 Linux 4.0.7-1-ck; java 1.8.0_45; Europe/de) http://yacy.net/bot.html
	Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/51.0.2704.103 Safari/537.36 PTST/284
	yacybot (freeworld/global; i386 Linux 3.12-1-686-pae; java 1.7.0_21; Europe/fr) http://yacy.net/bot.html
	Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.1; Trident/4.0; PTST 2.295)
	CovarioIDS/1.1 (http://www.covario.com/ids; support at covario dot com)
	Mozilla/5.0 (compatible; heritrix/1.14.4 +http://parsijoo.ir)
	www.adressendeutschland.de
	DialogSearch.com Bot 1.0;http://dialogsearch.com/webmasters
	librabot/2.0 (+http://search.msn.com/msnbot.htm)
	Mozilla/5.0 (compatible; heritrix/3.1.1-SNAPSHOT-20120116.200628 +http://www.archive.org/details/archive.org_bot)
	wsAnalyzer/1.0; ++http://www.wsanalyzer.com/bot.html
	Mozilla/5.0 (compatible; websays; +http://wiki.github.com/bixo/bixo/bixocrawler; [email protected])
	facebookexternalhit/1.0 (+http://www.facebook.com/externalhit_uatext.php)
	yacybot (/global; amd64 Linux 4.4.5-1-ARCH; java 1.8.0_77; America/en) http://yacy.net/bot.html
	websitepulse checker/1.1 (compatible; MSIE 5.5; Netscape 4.75; Linux)
	Mozilla/5.0 (compatible; memoryBot/1.21.14 +http://mignify.com/bot.html)
	SalesIntelligent/v1.0
	larbin_2.6.2 [email protected]
	Mozilla/5.0 (compatible; SiteCondor; http://www.sitecondor.com)
	yacybot (freeworld/global; amd64 Windows Server 2012 6.2; java 1.7.0_51; Europe/de) http://yacy.net/bot.html
	Mozilla/5.0 (compatible; Crawlera/1.10.2; UID 43063)
	yacybot (-global; amd64 Linux 2.6.32-042stab111.11; java 1.7.0_79; Europe/en) http://yacy.net/bot.html
	Mozilla/5.0 (compatible; SemrushBot/0.96.2; +http://www.semrush.com/bot.html)
	Linguee Bot (http://www.linguee.com/bot)
	yacybot (/global; x86 Windows XP 5.1; java 1.7.0_51; Europe/de) http://yacy.net/bot.html
	ICC-Crawler(Mozilla-compatible; ; http://kc.nict.go.jp/project1/crawl.html)
	Mozilla/5.0 (X11; U; Linux i686 (x86_64); en-US; rv:1.8.1.11) Gecko/20080109 (Charlotte/0.9t; http://www.searchme.com/support/) (Charlotte/0.9t; http://www.searchme.com/support/)
	Mozilla/5.0 (compatible; Goodzer/2.0; [email protected])
	Acoon v4.10.5 (www.acoon.de)
	CorpusCrawler 2.0.20 (http://corpora.fi.muni.cz/crawler/);Project:CzCorpus
	Mozilla/5.0 (compatible; AMZNKAssocBot/4.0 +http://affiliate-program.amazon.com)
	Mozilla/5.0 (compatible; MojeekBot/0.5; http://www.mojeek.com/bot.html)
	yacybot (/global; amd64 Linux 4.2.0-22-generic; java 1.7.0_91; Europe/en) http://yacy.net/bot.html
	Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/534.34 (KHTML, like Gecko) Qt/4.8.3 Safari/534.34 https://linkpeek.com
	MetaTagRobot/0.2 (http://www.seocentro.com/tools/search-engines/metatag-analyzer.html)
	Mozilla/5.0 (compatible; MJ12bot/v1.3.1; http://www.majestic12.co.uk/bot.php?+)
	Mozilla/5.0 (compatible; Uptimebot/0.2.29; +http://www.uptime.com/uptimebot)
	Mozilla/5.0 (compatible; Pandeo Bot; +http://pandeo.de/bot.php)
	Pixray-Seeker/1.1 (Pixray-Seeker; http://www.pixray.com/pixraybot; [email protected])
	EasyBib AutoCite (http://autocite-info.citation-api.com/)
	Mozilla/5.0 (compatible; OptimizationCrawler/0.2; +http://www.domainoptima.com/robot)
	AboutUsBot/Harpy (Website Analysis; http://www.aboutus.org/Aboutus:Bot; [email protected])
	Gigabot/2.0
	DoCoMo/2.0 N905i(c100;TB;W24H16) (compatible; Googlebot-Mobile/2.1; +http://www.google.com/bot.html)
	Morning Paper 1.0 (robots.txt compliant!)
	Mozilla/5.0 (compatible; SurdotlyBot/1.0; +http://sur.ly/bot.html; Linux; Android 4; iPhone; CPU iPhone OS 6_0_1 like Mac OS X) AppleWebKit/536.26 (KHTML, like Gecko) Version/6.0 Mobile/10A523 Safari/8536.25
	Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.1; Trident/4.0; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; Media Center PC 6.0; .NET4.0C; .NET4.0E; GWX:RED; PTST 2.386)
	SuperPagesUrlVerifyBot/1.0
	CopperEgg/RevealUptime/LondonUK(linode)
	Mozilla/5.0 (compatible; www.monitor.us - free monitoring service; http://www.monitor.us)
	Mozilla/5.0 (compatible; LinkMarketbot/1.2; +http://www.linkmarket.com/)
	Mozilla/5.0 (Windows; U; Windows NT 5.1; zh-CN; ) Firefox/1.5.0.11; 360Spider
	wangling
	Mozilla/5.0 (compatible; linkdexbot/2.0; +http://www.linkdex.com/about/bots/)
	RSSMicro.com RSS/Atom Feed Robot
	GarlikCrawler/1.1 (http://garlik.com/, [email protected])
	al_org_viewer ([email protected])
	Mozilla/5.0 (compatible; JadynAveBot; +http://www.jadynave.com/robot)
	dj-research/Nutch-1.11 (analytics@@demandjump.com)
	Karneval-Bot (Version: 1.06, powered by www.karnevalsuchmaschine.de +http://www.karnevalsuchmaschine.de/bot.html)
	Baiduspider-image+(+http://www.baidu.com/search/spider.htm)\nReferer: http://image.baidu.com/i?ct=503316480&z=0&tn=baiduimagedetail
	SauceNAO/1.0 (+http://saucenao.com/)
	Mozilla/5.0 (compatible; JobdiggerSpider +http://www.jobdigger.nl/spider)
	Yepi/1.0 (NHN Corp.; http://help.naver.com/robots/)
	Mozilla/5.0 (compatible; coccoc/1.0; +http://help.coccoc.com/)
	SSL Labs (https://www.ssllabs.com/about/assessment.html)
	Mozilla/5.0 (compatible; http://alyze.info)
	GigablastOpenSource/1
	Mozilla/5.0 (Windows; U; Windows NT 5.1;fr;rv:1.8.1) VoilaBotCollector BETA 0.1 (http://www.voila.com/)
	Vorboss Web Crawler [[email protected]]/Nutch-2.3
	Mozilla/5.0 (compatible; SecretSerachEngineLabs.com-SBSearch/0.9; http://www.secretsearchenginelabs.com/secret-web-crawler.php)
	Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko; Google Search Console) Chrome/27.0.1453 Safari/537.36
	YahooCacheSystem
	search.KumKie.com
	yacybot (webportal-global; x86 Windows Vista 6.0; java 1.7.0_25; Europe/en) http://yacy.net/bot.html
	Embedly [email protected]
	stq_bot (+http://www.searchteq.de)
	MSRBOT
	Mozilla/5.0 (compatible; WBSearchBot/1.1; +http://www.warebay.com/bot.html)
	CorpusCrawler 2.0.17 (http://corpora.fi.muni.cz/crawler/);Project:CzCorpus
	Mozilla/5.0 (compatible; heritrix/3.1.0-RC1 +http://boston.lti.cs.cmu.edu/crawler_12/)
	Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; Trident/5.0) PTST/276
	Mozilla/5.0 (compatible; WebCookies/1.0; +http://webcookies.org/faq/#agent)
	L.webis/0.44 (http://webalgo.iit.cnr.it/index.php?pg=lwebis)
	yacybot (freeworld/global; amd64 Linux 3.0.0-17-generic; java 1.6.0_23; America/en) http://yacy.net/bot.html
	Mozilla/5.0 (Windows NT 6.1; WOW64; rv:46.0) Gecko/20100101 Firefox/46.0 PTST/279
	Mozilla/5.0 (compatible; MSIE or Firefox mutant; not on Windows server;) Daumoa/4.0
	Mozilla/5.0 (compatible; YandexMetrika/2.0; +http://yandex.com/bots mtmon01i.yandex.ru)
	Y!J-BSC/1.0 crawler (http://help.yahoo.co.jp/help/jp/blog-search/)
	Mozilla/5.0 (compatible; UptimeRobot/1.0; http://www.uptimerobot.com/)
	NG-Search/0.86 (+http://www.ng-search.com)
	ichiro/3.0 (http://help.goo.ne.jp/help/article/1142)
	TwengaBot/1.1 (+http://www.twenga.com/bot.html)
	WebImages 0.3 ( http://herbert.groot.jebbink.nl/?app=WebImages )
	Mozilla/5.0 (compatible; aiHitBot/2.8; +http://endb-consolidated.aihit.com/)
	RyzeCrawler/1.1.1 (+http://www.domain2day.nl/crawler/)
	Mozilla/5.0 (Nekstbot; http://www.ipipan.waw.pl/nekst/nekstbot/)
	Mozilla/5.0 (compatible; adidxbot/2.0; http://www.bing.com/bingbot.htm)
	Mozilla/5.0 (compatible; SWEBot/1.0; +http://swebot-crawler.net)
	WatchMouse/8.4.0.3 (http://watchmouse.com/ ; gblon01.watchmouse.net)
	yacybot (/global; amd64 Windows 8.1 6.3; java 1.7.0_55; Europe/de) http://yacy.net/bot.html
	Page Analyzer v4.0 ( http://www.ranks.nl/ )
	web_bh ([email protected])
	findlinks/1.1.6-beta3 (+http://wortschatz.uni-leipzig.de/findlinks/)
	Mozilla/5.0 (compatible; Findxbot/1.0; +http://www.findxbot.com)
	findlinks/2.0 (+http://wortschatz.uni-leipzig.de/findlinks/)
	Mozilla/5.0 (compatible; imagecoccoc/1.0; +http://help.coccoc.com/)
	PagesInventory (robot http://www.pagesinventory.com)
	Mozilla/5.0 (compatible; aiHitBot/1.0-DS; +http://www.aihit.com/)
	tagSeoBot/1.0 (http://www.tagseoblog.de/tools)
	Mozilla/5.0 (en-us) AppleWebKit/537.36 (KHTML, like Gecko; Google PP Default) Chrome/27.0.1453 Safari/537.36
	404 Checker [http://www.404checker.com/user-agent]
	CopperEgg/RevealUptime/
	Mozilla/5.0 (compatible; adidxbot/2.0; +http://www.bing.com/bingbot.htm)
	Mozilla/5.0 (compatible; ZumBot/1.0; http://help.zum.com/inquiry)
	CopperEgg/RevealUptime/TokyoJapan
	Mozilla/5.0 (compatible; MojeekBot/0.2; http://www.mojeek.com/bot.html)
	GoSquared-Status-Checker/0.2
	Mozilla/5.0 (Windows NT 6.0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/99.0 Safari/537.36 YottaaMonitor
	WatchMouse/18990 (http://watchmouse.com/ ; bc.watchmouse.com)
	yacybot (freeworld/global; amd64 Linux 3.2.1-gentoo-r2; java 1.6.0_24; Europe/de) http://yacy.net/bot.html
	Mozilla/5.0 (compatible; adbeat_bot; [email protected]; [email protected])
	linkdexbot/Nutch-1.0-dev (http://www.linkdex.com/; crawl at linkdex dot com)
	Heurekabot-Feed/1.0 (+http://sluzby.heureka.cz/napoveda/heurekabot/)
	Mozilla/5.0 (compatible; Charlotte/1.1; http://www.searchme.com/support/)
	yacybot (/global; amd64 Linux 3.10.0-229.7.2.el7.x86_64; java 1.8.0_45; Europe/en) http://yacy.net/bot.html
	LSSRocketCrawler/1.0 LightspeedSystems
	Mozilla/5.0 (X11; U; Linux Core i7-4980HQ; de; rv:32.0; compatible; Jobboerse.com; http://www.xn--jobbrse-d1a.com) Gecko/20100401 Firefox/24.0
	findlinks/2.2 (+http://wortschatz.uni-leipzig.de/findlinks/)
	Mozilla/5.0 (compatible; kulturarw3 +http://www.kb.se/om/projekt/Svenska-webbsidor---Kulturarw3/)
	Mozilla/5.0 (compatible; CloudFlare-AlwaysOnline/1.0; +http://www.cloudflare.com/always-online) AppleWebKit/534.34
	Mozilla/5.0 (compatible; SemrushBot/0.96.4; +http://www.semrush.com/bot.html)
	woobot/2.0
	GarlikCrawler/1.2 (http://garlik.com/, [email protected])
	yacybot (webportal-global; amd64 Linux 3.2.0-4-amd64; java 1.7.0_67; Europe/en) http://yacy.net/bot.html
	Mozilla/5.0 (compatible; AboutUsBot Johnny5/2.0; +http://www.AboutUs.org/)
	yacybot (/global; amd64 Linux 3.10.0-327.22.2.el7.x86_64; java 1.7.0_101; Etc/en) http://yacy.net/bot.html
	Mozilla/5.0 (Linux; Android 4.1.2; Galaxy Nexus Build/JZO54K; GTmetrix http://gtmetrix.com/) AppleWebKit/537.22 (KHTML, like Gecko) Chrome/26.0.1410.58 Mobile Safari/537.22
	Nuhk/2.4 ( http://www.neti.ee/cgi-bin/abi/Otsing/Nuhk/)
	Grahambot/0.1 (+http://www.sunaga-lab.com/graham-bot)
	yacybot (/global; amd64 Linux 3.16.0-4-amd64; java 1.7.0_91; Europe/de) http://yacy.net/bot.html
	SEO Consulting; Redirect Checker Tool V.02; IP:
	Mozilla/5.0 (compatible; Pro Sitemaps Generator; https://pro-sitemaps.com) Gecko Pro-Sitemaps/1.0
	yacybot (/global; amd64 Linux 4.4.10-antix.1-amd64-smp; java 1.8.0_101; Europe/en) http://yacy.net/bot.html
	Mozilla/5.0 (Windows NT 6.2; WOW64) Runet-Research-Crawler (itrack.ru/research/cmsrate; [email protected])
	Mozilla/5.0 (compatible; Crawler/0.9; http://linkfluence.net/)
	ADmantX Platform Semantic Analyzer US Async - ADmantX Inc. - www.admantx.com - [email protected]
	GetProxi.es-bot/1.1 (http://getproxi.es/spiderinfo/)
	Mozilla/5.0 (Windows NT 6.3; WOW64; rv:46.0) Gecko/20100101 Firefox/46.0 PTST/277
	Pinterest/0.2 (+http://www.pinterest.com/)
	CopperEgg/RevealUptime/AtlantaGA(linode)
	OdklBot/1.0 ([email protected])
	Mozilla/5.0 (compatible; Exabot-Images/3.0; +http://www.exabot.com/go/robot)
	yacybot (freeworld/global; amd64 Linux 3.2.1-gentoo-r2; java 1.6.0_22; Europe/de) http://yacy.net/bot.html
	MojeekBot/0.2 (archi; http://www.mojeek.com/bot.html)
	Sogou web spider/4.05252A
	Mozilla/5.0 (Windows NT 6.3; WOW64; rv:45.0) Gecko/20100101 Firefox/45.0 PTST/276
	Mozilla/5.0 (compatible; MotoMinerBot/1.0; +https://motominer.com/Bot)
	NG/2.0
	Mozilla/5.0 (compatible; heritrix/1.14.2 +http://www.webarchiv.cz)
	StackRambler/2.0 (MSIE incompatible)
	Baiduspider+(+http://www.baidu.jp/spider/)
	yacybot (freeworld/global; amd64 Windows 7 6.1; java 1.6.0_25; Europe/de) http://yacy.net/bot.html
	Mozilla/5.0 (compatible; houzzbot; +http://www.houzz.com/)
	Woko robot 3.0
	Mozilla/5.0 (compatible; Qwantify/2.0; +https://www.qwant.com/)
	yacybot (/global; amd64 Linux 4.2.0-27-generic; java 1.8.0_66-internal; America/en) http://yacy.net/bot.html
	ADmantX Platform Semantic Analyzer - APAC - ADmantX Inc. - www.admantx.com - [email protected]
	Mozilla/5.0 (compatible; Uptimebot/0.2.40; +http://www.uptime.com/uptimebot)
	Mozilla/5.0 (compatible; ExpertSearchSpider +http://www.expertsearch.nl/spider)
	Mozilla/5.0 (compatible; coccocbot-web/1.0; +http://help.coccoc.com/searchengine)
	Acoon v4.10.4 (www.acoon.de)
	Mozilla/5.0 (compatible; memoryBot/1.20.210 +http://internetmemory.org/en/)
	Readability/740ec9 - http://readability.com/about/
	Mozilla/5.0 (compatible; Apercite; +http://www.apercite.fr/robot/index.html)
	yacybot (i386 Linux 2.6.28-gentoo-r5; java 1.5.0_18; Europe/en) http://yacy.net/bot.html
	Mozilla/5.0 (compatible; suggybot v0.01a, http://blog.suggy.com/was-ist-suggy/suggy-webcrawler/)
	yacybot (amd64 Windows 7 6.1; java 1.6.0_14; Europe/de) http://yacy.net/bot.html
	yacybot (freeworld/global; amd64 Linux 3.3.4-1-ARCH; java 1.6.0_24; Europe/en) http://yacy.net/bot.html
	Mozilla/5.0 (compatible; AportWorm/3.2; +http://www.aport.ru/help)
	Mozilla/5.0 (compatible; memoryBot/1.20.235 +http://internetmemory.org/en/)
	findlinks/2.6 (+http://wortschatz.uni-leipzig.de/findlinks/)
	Mozilla/5.0 (compatible; Hailoobot/1.2; +http://www.hailoo.com/spider.html)
	eCommerceBot (http://www.ehandel.se/botinfo.html)
	Mozilla/5.0(compatible;Sosospider/2.0;+http://help.soso.com/webspider.htm)
	yacybot (/global; amd64 Linux 4.3.0-gentoo-ARCH; java 1.7.0_85; Europe/en) http://yacy.net/bot.html
	Nymesis/2.0 (http://nymesis.com)
	CopperEgg/RevealUptime/OregonUSA
	uclassify.com/1.0
	Mozilla/5.0 (compatible; Butterfly/1.0; +http://labs.topsy.com/butterfly.html) Gecko/2009032608 Firefox/3.0.8
	Mozilla/5.0 (compatible; Prlog/1.0; +http://prlog.ru/)
	Slack-ImgProxy 1.106 (+https://api.slack.com/robots)
	AdnormCrawler www.adnorm.com/crawler
	Mozilla/5.0 (compatible; YandexZakladki/3.0; +http://yandex.com/bots)
	Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/49.0.2623.75 Safari/537.36 Google Favicon
	Mozilla/5.0 (compatible; Sonic/1.0; http://www.yama.info.waseda.ac.jp/~crawler/info.html)
	Mozilla/5.0 (iPhone; U; CPU iPhone OS 4_1 like Mac OS X; en-us) AppleWebKit/532.9 (KHTML, like Gecko) Version/4.0.5 Mobile/8B117 Safari/6531.22.7 (compatible; Googlebot-Mobile/2.1; +http://www.google.com/bot.html)
	ICC-Crawler/2.0 (Mozilla-compatible; ; http://kc.nict.go.jp/project1/crawl.html)
	Mozilla/4.0 ([email protected])
	bot-pge.chlooe.com/1.0.0 (+http://www.chlooe.com/)
	Mozilla/5.0 (compatible; GroupHigh/1.0; +http://www.grouphigh.com/)
	Mozilla/5.0 (compatible; Webmaster tools +http://sitexy.com/)
	yacybot (/global; amd64 Windows 8.1 6.3; java 1.8.0_40; Europe/de) http://yacy.net/bot.html
	Mozilla/5.0 (compatible; MJ12bot/v1.4.1; http://www.majestic12.co.uk/bot.php?+)
	Mozilla/5.0 (compatible; spbot/4.0.6; +http://www.seoprofiler.com/bot )
	Mozilla/5.0 (compatible; EuripBot/2.0; +http://www.eurip.com)
	findlinks/2.1 (+http://wortschatz.uni-leipzig.de/findlinks/)
	Sogou web spider/4.025251
	SETOOZBOT/5.0 ( http://www.setooz.com/bot.html )
	Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0) ([email protected])
	Mozilla/5.0 (compatible; spbot/2.0.4; +http://www.seoprofiler.com/bot )
	Mozilla/5.0 (TweetmemeBot/4.0; +http://datasift.com/bot.html) Gecko/20100101 Firefox/31.0
	mozilla/5.0 ([email protected])
	Mozilla/5.0 (compatible; spbot/2.1; +http://www.seoprofiler.com/bot )
	Mozilla/5.0 (Windows NT 6.2; WOW64) AppleWebKit/537.4 (KHTML, like Gecko) Chrome/98 Safari/537.4 (StatusCake SSL Monitor)
	Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; Trident/5.0; PTST 2.385)
	Mozilla/5.0 (compatible; evc-batch/2.0.20160608212921)
	Mozilla/5.0 (compatible; Mail.RU_Bot/2.0)
	seebot/2.0 (+http://www.seegnify.com/bot)
	bl.uk_lddc_bot/3.3.0-LBS-2016-02 (+http://www.bl.uk/aboutus/legaldeposit/websites/websites/faqswebmaster/index.html)
	CommaFeed/2.3.0-SNAPSHOT (https://www.commafeed.com)
	OmniExplorer_Bot/5.91c (+http://www.omni-explorer.com) WorldIndexer
	hledejLevne.cz/2.0
	page_verifier (http://www.securecomputing.com/goto/pv)
	url_test ([email protected])
	Mozilla/5.0 (X11; Linux x86_64; rv:10.0.12) Gecko/20100101 Firefox/21.0 WordPress.com mShots
	Mozilla/5.0 (compatible; Linux x86_64; Mail.RU_Bot/2.0; +http://go.mail.ru/help/robots)
	Is is up? (+http://isitup.org)
	Metaspinner/0.01 (Metaspinner; http://www.meta-spinner.de/; [email protected]/)
	TwengaBot-2.0 Champigny (+http://www.twenga.com/bot.html)
	LivelapBot/0.2 (http://site.livelap.com/crawler)
	HubSpot Crawler 1.0 http://www.hubspot.com/
	Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/45.0.2454.101 Safari/537.36 TinEye/1.0 (via http://www.tineye.com/)
	yacybot (/global; x86 Windows 10 10.0; java 1.8.0_73; Europe/de) http://yacy.net/bot.html
	Scrapy/0.24.6 (+http://scrapy.org)
	FAST-WebCrawler/3.6/FirstPage (atw-crawler at fast dot no;http://fast.no/support/crawler.asp)
	Baiduspider+(+http://www.baidu.com/search/spider_jp.html)
	Mozilla/5.0 (compatible; seplinkbot/1.0 )
	Mozilla/5.0 (compatible; Falconsbot; +http://ws.nju.edu.cn/falcons/)
	Mozilla/5.0 (Windows NT 6.1; WOW64; rv:46.0) Gecko/20100101 Firefox/46.0 PTST/277
	Mozilla/4.0 (Toread-Crawler/1.1; +http://news.toread.cc/crawler.php)
	TinEye-bot/0.02 (see http://www.tineye.com/crawler.html)
	yacybot (freeworld/global; i386 Linux 2.6.32-39-generic-pae; java 1.6.0_20; Europe/en) http://yacy.net/bot.html
	Mozilla/5.0 (compatible; parsijoo; +http://www.parsijoo.ir/; [email protected])
	Mozilla/5.0 (compatible; spbot/4.4.0; +http://OpenLinkProfiler.org/bot )
	MaxPoint Bot (+http://www.maxpoint.com)
	Mozilla/5.0 (compatible; Infohelfer/1.2.0; +http://www.infohelfer.de/crawler.php)
	ExB Language Crawler 2.1.1 (+http://www.exb.de/crawler)
	cg-eye interactive
	ZumBot/1.0 (ZUM Search; http://help.zum.com/inquiry)
	rogerbot/1.1 (http://moz.com/help/guides/search-overview/crawl-diagnostics#more-help, [email protected])
	Mozilla/5.0 (compatible; Crawlera/1.10.2; UID 47129)
	Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:14.0) Gecko/20100101 Firefox/14.0.1 AppEngine-Google; (+http://code.google.com/appengine; appid: s~metacdn-hr)
	Mozilla/5.0 (compatible; WoTBoT; +https://www.wslta.com/WoTBoT.html)
	HolmesBot (http://holmes.ge)
	Baiduspider-image+(+http://www.baidu.com/search/spider.htm)
	Mozilla/5.0 (compatible; alexa site audit/1.0; +http://www.alexa.com/help/webmasters; [email protected])
	Mediapartners-Google
	Mozilla/5.0 (compatible; MFGPagesBot/2.1; http://www.mfgpages.com)
	larbin_2.6.2 [email protected]
	Mozilla/5.0 (Windows; U; Windows NT 6.0; en-GB; rv:1.0; trendictionbot0.5.0; trendiction search; http://www.trendiction.de/bot; please let us know of any problems; web at trendiction.com) Gecko/20071127 Firefox/3.0.0.11
	BlogPulseLive ([email protected])
	WeSEE:Search/0.1 (Alpha, http://www.wesee.com/en/support/bot/)
	yacybot (freeworld/global; i386 Linux 3.0.0-17-generic; java 1.6.0_23; America/en) http://yacy.net/bot.html
	ImplisenseBot 1.0
	Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; WOW64; Trident/5.0; PTST 2.386)
	gonzo2[P] +http://www.suchen.de/faq.html
	Mozilla/5.0 (compatible; LXRbot/1.0; http://lxrseo.com/, [email protected])
	Mozilla/5.0 (compatible; Arachnophilia/1.0; +http://arachnys.com/)
	Mozilla/5.0 (compatible; CloudServerMarketSpider/1.0; +http://cloudservermarket.com/spider.html)
	kalooga/KaloogaBot (Kalooga; http://www.kalooga.com/info.html?page=crawler)
	yacybot (webportal/global; x86_64 Mac OS X 10.9.2; java 1.6.0_65; Europe/de) http://yacy.net/bot.html
	Mozilla/5.0 (compatible; DotBot/1.1; http://www.dotnetdotcom.org/, [email protected])
	yacybot (/global; amd64 Linux 4.1.19-gentoo; java 1.7.0_95; Europe/pl) http://yacy.net/bot.html
	Mozilla/5.0 (Windows NT 6.3; WOW64; Trident/7.0; rv:11.0) like Gecko PTST/276
	Mozilla/5.0 (compatible; RankActiveLinkBot; +https://rankactive.com/resources/rankactive-linkbot)
	audisto.com full crawler 3.26.431 (refer to in robots.txt as audisto, see https://audisto.com/bot)
	Mozilla/5.0 (compatible; spbot/4.0.1; +http://www.seoprofiler.com/bot )
	Mozilla/5.0 (compatible; OpenindexShallowSpider/Nutch-1.5-dev; +http://www.openindex.io/en/webmasters/spider.html)
	mozilla/5.0 (compatible; discobot/1.1; +http://discoveryengine.com/discobot.html)
	gonzo2[p] (+http://www.suchen.de/faq.html)
	Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; Trident/5.0); 360Spider(compatible; HaosouSpider; http://www.haosou.com/help/help_3_2.html)
	Mozilla/5.0 (compatible; spbot/2.0.1; +http://www.seoprofiler.com/bot/ )
	WatchMouse/18990 (http://watchmouse.com/ ; uk)
	yacybot (freeworld/global; amd64 Linux 3.1.10-hardened; java 1.7.0_03-icedtea; Europe/en) http://yacy.net/bot.html
	VeBot (+http://www.veinteractive.com/vebot)
	Mozilla/5.0 (compatible; NLNZ_IAHarvester2013 +http://natlib.govt.nz/about-us/current-initiatives/web-harvest-2012)
	findlinks/1.1.3-beta8 (+http://wortschatz.uni-leipzig.de/findlinks/)
	Mozilla/5.0 (Linux; U; Android 2.3.4; generic) AppleWebKit/537.36 (KHTML, like Gecko; Google Web Preview) Version/4.0 Mobile Safari/537.36
	DialogSearch.com Bot 1.4;http://dialogsearch.com/webmasters
	Mozilla/5.0 (compatible; GurujiBot/1.0; +http://www.guruji.com/en/WebmasterFAQ.html)
	Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9) AppleWebKit/537.71 (KHTML, like Gecko) Version/7.0 Safari/537.71 (Rival IQ, rivaliq.com)
	wscheck.com/1.0.0 (+http://wscheck.com/)
	Mozilla/5.0 (Windows NT 6.3; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/30.0.1599.69 Safari/537.36 Webthumb/2.0
	Mozilla/5.0 (compatible; FlipboardProxy/1.1; +http://flipboard.com/browserproxy)
	WeSEE:Ads/PictureBot (http://www.wesee.com/bot/)
	Mozilla/5.0 (compatible; Dataprovider/6.101; +https://www.dataprovider.com/)
	Kyoto-Crawler/n1.0 (Mozilla-compatible; [email protected]; http://nlp.ist.i.kyoto-u.ac.jp/?crawling)
	Mozilla/5.0 (compatible; SEOkicks-Robot; +http://www.seokicks.de/robot.html)
	Mozilla/5.0 (compatible; spbot/4.0.9; +http://OpenLinkProfiler.org/bot )
	Mozilla/5.0 (compatible; spbot/4.2.0; +http://OpenLinkProfiler.org/bot )
	seo-nastroj.cz
	LoadImpactPageAnalyzer/1.3.0 (Load Impact; http://loadimpact.com/)
	CSS Certificate Spider (http://www.css-security.com/certificatespider/)
	MetaGeneratorCrawler/1.3.2 (www.metagenerator.info)
	Testomatobot/1.0 (Linux x86_64; +http://www.testomato.com/testomatobot) minicrawler/4.0.0~beta8
	BLEXBot
	Mozilla/2.0 (compatible; Ask Jeeves/Teoma)
	Testomatobot/1.0 (Linux x86_64; +http://www.testomato.com/testomatobot) minicrawler/4.0.0~beta7
	Mozilla/5.0 (compatible; XoviBot/2.0; +http://www.xovibot.net/)
	Mozilla/5.0 (compatible; spbot/4.4.1; +http://OpenLinkProfiler.org/bot )
	ia_archiver (+http://www.alexa.com/site/help/webmasters; [email protected])
	HeartRails Robot/0.1 (http://www.heartrails.com)
	Mozilla/5.0 (compatible; Faveeo/1.0; +http://www.faveeo.com)
	yacybot (/global; amd64 Linux 3.16.0-49-generic; java 1.7.0_79; Europe/en) http://yacy.net/bot.html
	Castabot/0.1 (+http://topixtream.com/)
	Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.9.0.5) Gecko/2010033101 Gentoo Firefox/3.0.5 (Dot TK - spider 3.0)
	Mozilla/5.0 (compatible; MSIE or Firefox mutant; not on Windows server; +http://ws.daum.net/aboutWebSearch.html) Daumoa/2.0
	istellabot-nutch/Nutch-1.10
	Mail.RU/2.0
	ichiro/2.0 (http://help.goo.ne.jp/door/crawler.html)
	LexxeBot/1.0 ([email protected])
	Mozilla/5.0 (iPhone; CPU iPhone OS 8_1 like Mac OS X) AppleWebKit/600.1.4 (KHTML, like Gecko) Version/8.0 Mobile/12B410 Safari/600.1.4 (Applebot/0.1; +http://www.apple.com/go/applebot)
	Mozilla/5.0 (compatible; FlipboardRSS/1.1; +http://flipboard.com/browserproxy)
	Mozilla/5.0 (compatible; Linux x86_64; Mail.RU_Bot/Robots/2.0; +http://go.mail.ru/help/robots)
	yacybot (-global; amd64 Linux 3.10.0-229.4.2.el7.x86_64; java 1.7.0_79; Europe/en) http://yacy.net/bot.html
	it2media-domain-crawler/1.0 on crawler-prod.it2media.de
	yacybot (/global; amd64 Windows 8.1 6.3; java 1.8.0_25; Europe/de) http://yacy.net/bot.html
	yacybot (freeworld-global; amd64 Linux 3.16.0-4-amd64; java 1.7.0_79; Europe/de) http://yacy.net/bot.html
	Mozilla/5.0 (WhatsMyIP.org HTTP_Headers) http://whatsmyip.org/ua
	checkgzipcompression.com robot
	Mozilla/5.0 eCairn-Grabber/1.0 (+http://ecairn.com/grabber)
	RankurBot/3.3 (+http://rankur.com)
	L.webis/0.50 (http://webalgo.iit.cnr.it/index.php?pg=lwebis)
	Speedy Spider (Submit your site at http://www.entireweb.com/free_submission/)
	oBot
	Snappy/2.0 ( http://www.urltrends.com/ )
	Mozilla/5.0 (compatible; MSIE 10.0; Windows NT 6.2; WOW64; Trident/6.0) CrawlerProcess (http://www.PowerMapper.com) /5.23.770.0
	Mozilla/5.0 (compatible; alexa site audit/1.0; +http://www.alexa.com/help/webmasters; )
	flatlandbot/baypup (Flatland Industries Web Spider; http://www.flatlandindustries.com/flatlandbot; [email protected])
	istellabot/Nutch-1.11
	GetintentCrawler getintent.com
	Covario-IDS/1.0 (Covario; http://www.covario.com/ids; support at covario dot com)
	Mozilla/5.0 (compatible; FatBot 2.0; http://www.thefind.com/crawler)
	Mozilla/5.0 (compatible; MegaIndex.com/2.0; +http://megaindex.com/crawler)
	Microsearch.ru/1.0; http://microsearch.ru/webmasters
	TurnitinBot/3.0 (http://www.turnitin.com/robot/crawlerinfo.html)
	BacklinkCrawler V (http://www.backlinktest.com/crawler.html)
	FeedlyBot/1.0 (http://feedly.com)
	Clickagy Intelligence Bot v2
	GetURLInfo/1.0
	DoCoMo/2.0 P900i(c100;TB;W24H11)(compatible; ichiro/mobile goo;+http://help.goo.ne.jp/door/crawler.html)
	Scrapy/1.1.0 (+http://scrapy.org)
	yacybot (/global; amd64 Linux 3.16.0-4-amd64; java 1.7.0_75; America/en) http://yacy.net/bot.html
	Mozilla/5.0 (compatible; VSAgent/1.2)
	Experibot_v1 [goo.gl/n6zrAf]
	Mozilla/5.0 (compatible; DNS-Digger-Explorer/1.0; +http://www.dnsdigger.com)
	boitho.com-robot/1.0
	yacybot (/global; amd64 Windows Server 2012 6.2; java 1.7.0_51; Europe/de) http://yacy.net/bot.html
	RelateIQ Crawler www.relateiq.com
	Mozilla/4.0 (compatible; Netcraft Web Server Survey)
	Mozilla/5.0 (X11; U; Linux Core i7-4980HQ; de; rv:32.0; compatible; JobboerseBot; https://www.jobboerse.com/bot.htm) Gecko/20100101 Firefox/38.0
	yacybot (i386 Linux 2.6.9-023stab046.2-smp; java 1.6.0_05; Europe/en) http://yacy.net/bot.html
	Scrubby/2.2 (http://www.scrubtheweb.com/)
	Mozilla/5.0 (Yahoo-MMCrawler/4.0; mailto:[email protected])
	Fetch/2.0a (CMS Detection/Web/SEO analysis tool, see http://guess.scritch.org)
	bitlybot/3.0 (+http://bit.ly/)
	Mozilla/5.0 (compatible; heritrix/1.14.4 +http://www.exif-search.com)
	yacybot (freeworld/global; amd64 Windows Server 2012 6.2; java 1.7.0_25; Europe/de) http://yacy.net/bot.html
	agentslug.com - website monitoring tool
	SafeDNSBot (https://www.safedns.com/searchbot)
	Feedbin
	Mozilla/5.0 (compatible; proximic; +http://www.proximic.com)
	Mozilla/5.0 (compatible; AcoonBot/4.11.1; +http://www.acoon.de/robot.asp)
	Mozilla/5.0 (compatible; MagiBot/3.4.3; +http://magi.peak-labs.com/robots.txt)
	Mozilla/5.0 (compatible; XML Sitemaps Generator; http://www.xml-sitemaps.com) Gecko XML-Sitemaps/1.0
	baypup/colbert (Baypup; http://sf.baypup.com/webmasters; [email protected])
	MergeFlow-PageReader/0.91;+(+http://mergeflow.net/info/pagereader) Mozilla/5.0 (Windows) compatible
	Mozilla/5.0 (Mobile; rv:18.0) Gecko/18.0 Firefox/18.0 commoncrawl.org/research//Nutch-1.7-SNAPSHOT
	Mozilla/5.0 (compatible; spbot/4.1.0; +http://OpenLinkProfiler.org/bot )
	Mozilla/5.0 (compatible; OsO; http://oso.octopodus.com/abot.html)
	Mozilla/5.0 (compatible; gofind; +http://govid.mobi/bot.php)
	CatchBot/2.0; +http://www.catchbot.com
	Mozilla/5.0 (compatible; BusinessSeek.biz_Spider; http://www.businessseek.biz/)
	Quora Link Preview/1.0 (http://www.quora.com)
	radian6_default_(www.radian6.com/crawler)
	DWDS-Crawler +http://odo.dwds.de/dwds-crawler.html
	Mozilla/5.0 (compatible; DuckDuckGo-Favicons-Bot/1.0; +http://duckduckgo.com)
	sogou spider
	Mozilla/5.0 (compatible; GigablastOpenSource/1.0)
	Mozilla/5.0 (compatible; HomeTags/1.0; +http://www.hometags.nl/bot)
	CorpusCrawler 2.0.24 (http://corpora.fi.muni.cz/crawler/);Project:CzCorpus
	Scooter/3.3
	Zookabot/2.5;++http://zookabot.com
	Mozilla/5.0 (iPhone; CPU iPhone OS 8_1 like Mac OS X) AppleWebKit/600.1.4 (KHTML, like Gecko) Version/8.0 Mobile/12B411 Safari/600.1.4 (compatible; YandexMobileBot/3.0; +http://yandex.com/bots)
	Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/534.34 (KHTML, like Gecko) Safari/534.34; +http://sniptracker.com
	Mozilla/5.0 (compatible; archive.org_bot; Wayback Machine Live Record; +http://archive.org/details/archive.org_bot)
	es_com_viewer ([email protected])
	Mozilla/5.0 (compatible; Uptimebot/0.2.14; +http://www.uptime.com/uptimebot)
	yacybot (freeworld/global; amd64 Linux 3.0.0-14-generic; java 1.6.0_23; Europe/en) http://yacy.net/bot.html
	Mozilla/5.0 (X11; Linux x86_64; rv:35.0) Gecko/20100101 Firefox/35.0 DareBoost
	yacybot (freeworld/global; amd64 Linux 2.6.32-41-server; java 1.6.0_26; Europe/de) http://yacy.net/bot.html
	yacybot (amd64 Linux 2.6.28-18-generic; java 1.6.0_16; GMT/en) http://yacy.net/bot.html
	Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.1; Trident/4.0; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; Media Center PC 6.0; .NET4.0C; .NET4.0E; PTST 2.386)
	KD Bot
	Mozilla/5.0 (X11; U; Linux i686 (x86_64); en-US; rv:1.9.2.19) Gecko WebThumb/1.0
	Mozilla/5.0 (compatible; Yahoo! Slurp/3.0; http://help.yahoo.com/help/us/ysearch/slurp)
	rogerbot/1.0 (http://moz.com/help/pro/what-is-rogerbot-, [email protected])
	MJ12bot/v1.2.0 (http://majestic12.co.uk/bot.php?+)
	Superarama.com-Tarama-Botu-v.01
	Mozilla/5.0 (compatible; Urlfilebot/2.2; +http://urlfile.com/bot.html)
	Vegi bot (we follow your robots.txt settings before crawling, you can slow down the bot by change the Crawl-Delay parameter in the settings.if you have an enquiry, please email to: [email protected])
	MnoGoSearch/3.3.9
	Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/534+ (KHTML, like Gecko) BingPreview/1.0b
	JyxobotRSS/0.06
	Mozilla/5.0 (compatible; BigBozz/2.2.1; +http://www.bigbozz.com/)
	Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/534.57.2 (KHTML, like Gecko) Version/5.1.7 Safari/534.57.2 PTST/277
	KiNShooboT (compatible; KiNShooboT/1.0.C; +http://www.kinshoo.com/bot.html)
	SentiBot www.sentibot.eu (compatible with Googlebot)
	Topicbot/1.0 (Mozilla;I;+http://92.42.190.57/)
	IDG/IT (http://spaziodati.eu/)
	LinkAider (http://linkaider.com/crawler/)
	Mozilla/5.0 (compatible; coccoc/1.0; +http://help.coccoc.com/searchengine)
	yacybot (/global; amd64 Linux 2.6.32-573.3.1.el6.x86_64; java 1.7.0_85; Europe/en) http://yacy.net/bot.html
	bitlybot
	Mozilla/5.0 (compatible; Mp3Bot/0.7; +http://mp3realm.org/mp3bot/)
	WWW::LayeredExtractor::Handler::Feed/0.01
	Mozilla/5.0 (compatible; SWEBot/1.0; +http://swebot.net)
	Mozzila/5.0 (compatible; Sonic/1.0; http://www.yama.info.waseda.ac.jp/~crawler/info.html)
	MetaTagRobot/2.1 (http://www.seocentro.com/tools/search-engines/metatag-analyzer.html)
	Mozilla/5.0 (compatible; Goodzer/1.0)
	yacybot (/global; amd64 Linux 3.16.0-4-amd64; java 1.8.0_40; Europe/en) http://yacy.net/bot.html
	Mozilla/5.0 (Windows; U; Windows NT 6.1; en-GB; rv:1.9.2.3) Gecko/20100401 Firefox/3.6.3 (NetShelter ContentScan)
	Facebot/1.0
	SafeAds.xyz bot
	Mozilla/5.0 (compatible; Yeti/1.1; +http://help.naver.com/robots/)
	yacybot (/global; amd64 Windows 7 6.1; java 1.7.0_55; Europe/en) http://yacy.net/bot.html
	scrapyproject (+http://www.profound.net)
	Mozilla/5.0 (compatible; Online Domain Tools - Online Website Link Checker/1.2; +http://website-link-checker.online-domain-tools.com)
	Mozilla/5.0 (compatible; heritrix/2.0.2 +http://aihit.com)
	Mozilla/5.0 (compatible; Scrubby/3.1; +http://www.scrubtheweb.com/help/technology.html)
	Semantifire1/0.20 ( http://www.setooz.com/oozbot.html ; agentname at setooz dot_com )
	Mozilla/5.0 (X11; Ubuntu; Linux i686; rv:14.0; ips-agent) Gecko/20100101 Firefox/14.0.1
	UnisterBot (Mozilla/5.0 compatible; [email protected])
	Mozilla/5.0 (compatible; Ezooms/1.0; [email protected])
	Mozilla/5.0 (compatible; YandexVideo/3.0; +http://yandex.com/bots)
	Mozilla/5.0 (compatible; spbot/4.4.2; +http://OpenLinkProfiler.org/bot )
	findlinks/2.1.3 (+http://wortschatz.uni-leipzig.de/findlinks/)
	Mozilla/5.0 (compatible; MJ12bot/v1.2.5; http://www.majestic12.co.uk/bot.php?+)
	Speedy Spider (Entireweb; Beta/1.2; http://www.entireweb.com/about/search_tech/speedyspider/)
	Mozilla/5.0 (compatible; kazbtbot/0.1; +http://kazbt.com/)
	Orgbybot/OrgbyBot v1.3 (Spider; http://orgby.com/bot/ ; Orgby.com Search Engine)
	Mozilla/5.0 (compatible; aiHitBot-DM/2.0.2 +http://www.aihit.com)
	YahooSeeker-Testing/v3.9 (compatible; Mozilla 4.0; MSIE 5.5; http://search.yahoo.com/)
	Mozilla/5.0 (compatible; Crawlera/1.10.2; UID 40409)
	Mozilla/5.0 (compatible; image.coccoc/1.0; +http://help.coccoc.com/)
	yacybot (/global; amd64 Linux 3.19.2-1-ARCH; java 1.8.0_40; Europe/de) http://yacy.net/bot.html
	Mozilla/5.0 (compatible; evc-batch/2.0.20161017175820)
	datagnionbot (+http://www.datagnion.com/bot.html)
	CopperEgg/RevealUptime/TokyoJP(linode)
	MB-SiteCrawler
	WatchMouse/18990 (http://watchmouse.com/ ; liz)
	HybridBot (hybrid.ru/about. If our bot caused problems please contact us. Contact email: [email protected])
	Mozilla/5.0 (compatible; Crawlera/1.10.2; UID 24522)
	yacybot (/global; amd64 Linux 3.13.0-74-generic; java 1.7.0_91; Europe/en) http://yacy.net/bot.html
	findlinks/1.1.6-beta5 (+http://wortschatz.uni-leipzig.de/findlinks/)
	yacybot (freeworld/global; amd64 Linux 3.13.0-24-generic; java 1.7.0_55; Europe/de) http://yacy.net/bot.html
	Mozilla/5.0 (compatible; OpenfosBot/2.4; +http://www.openfos.com)
	Nusearch Spider (www.nusearch.com)
	WatchMouse/8.4.0.3 (http://watchmouse.com/ ; sesto02.watchmouse.net)
	ICC-Crawler/2.0 (Mozilla-compatible; ; http://www.nict.go.jp/en/univ-com/plan/crawl.html)
	Mozilla/5.0 (compatible; spbot/1.2; +http://www.seoprofiler.com/bot/ )
	Orbiter/1.3 (http://dailyorbit.com/)
	CCBot/2.0
	Mozilla/5.0 (compatible; linkdexbot/2.1; +http://www.linkdex.com/about/bots/)
	Mozilla/5.0 (compatible; DNS-Digger/1.0; +http://www.dnsdigger.com)
	AppEngine-Google; (+http://code.google.com/appengine; appid: s~feedly-social)
	yacybot (/global; x86 Windows 7 6.1; java 1.8.0_71; Europe/de) http://yacy.net/bot.html
	Mozilla/5.0 (Windows Phone 8.1; ARM; Trident/7.0; Touch; rv:11.0; IEMobile/11.0; NOKIA; Lumia 530) like Gecko (compatible; adidxbot/2.0; +http://www.bing.com/bingbot.htm)
	Dex Social Bot
	yacybot (/global; amd64 Linux 4.7.6-200.fc24.x86_64; java 1.8.0_102; Etc/en) http://yacy.net/bot.html
	Mozilla/5.0 (compatible; heritrix/1.14.3 +http://archive.org)
	CRAZYWEBCRAWLER 0.9.10, http://www.crazywebcrawler.com
	Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/50.0.2661.102 Safari/537.36 PTST/277
	Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.1.3; ips-agent) Gecko/20090824 Fedora/1.0.7-1.1.fc4 Firefox/3.5.3
	SafeSearch microdata crawler (https://safesearch.avira.com, [email protected])
	Zemanta Aggregator/0.9 +http://www.zemanta.com
	Mozilla/5.0 (compatible; special_archiver/3.2.0 +http://www.loc.gov/webarchiving/notice_to_webmasters.html)
	Mozilla/5.0 (compatible; GimmeUSAbot/1.0; +https://gimmeusa.com/crawler.html)
	yacybot (freeworld/global; amd64 Windows 7 6.1; java 1.7.0_45; Europe/de) http://yacy.net/bot.html
	Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.1 (KHTML, like Gecko) Chrome/21.0.1180.89 Safari/537.1; 360Spider
	FyberSpider/1.3 (http://www.fybersearch.com/fyberspider.php)
	yacybot (/global; amd64 FreeBSD 10.3-RELEASE; java 1.8.0_77; GMT/en) http://yacy.net/bot.html
	Searchie/1.0 (a Storm-based crawler; https://www.searchie.org; [email protected])
	yacybot (freeworld/global; amd64 Linux 3.1.10-1-desktop; java 1.6.0_22; Europe/de) http://yacy.net/bot.html
	holmes/3.12.4 (http://morfeo.centrum.cz/bot)
	WatchMouse/18990 (http://watchmouse.com/ ; se.watchmouse.com)
	MXT/Nutch-1.10 (http://t.co/GSRLLKex24; informatique at mixdata dot com)
	Mozilla/5.0 (compatible; Shareaholicbot/1.0; +http://www.shareaholic.com/bot)
	ZoomInformation Bot
	Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 5.1; www.alertra.com)
	Mozilla/5.0 (compatible; HomeTags/1.0; http://www.hometags.nl/bot)
	Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.2.17) Gecko/20110515 HeartRails_Capture/1.0.4 (+http://capture.heartrails.com/) Namoroka/3.6.17
	OmniExplorer_Bot/5.20 (+http://www.omni-explorer.com) WorldIndexer
	WatchMouse/18990 (http://watchmouse.com/ ; it)
	Mozilla/5.0 (compatible; imrbot/1.10.8 +http://www.mignify.com)
	Mozilla/5.0 (compatible; spbot/4.0.2; +http://www.seoprofiler.com/bot )
	Readability/1900e6 - http://readability.com/about/
	Mozilla/5.0 (compatible; Dataprovider/6.92; +https://www.dataprovider.com/)
	Mozilla/5.0 (iPhone; CPU iPhone OS 6_0_1 like Mac OS X) AppleWebKit/537.36 (KHTML, like Gecko; Google Page Speed Insights) Version/6.0 Mobile/10A525 Safari/8536.25
	VegeBot (we follow your robots.txt settings before crawling, you can slow down the bot by change the Crawl-Delay parameter in the settings.if you have an enquiry, please email to: [email protected])
	yacybot (webportal-global; amd64 Linux 3.2.0-4-amd64; java 1.7.0_65; Europe/en) http://yacy.net/bot.html
	FeedBucket/1.0 (+http://www.feedbucket.com)
	topster.de HTTP-Header 1.0
	MaxPointCrawler/Nutch-1.10 (maxpoint.crawler at maxpointinteractive dot com)
	Mozilla/5.0 (Windows NT 6.0; rv:45.0) Gecko/20100101 Firefox/45.0 PTST/276
	yacybot (/global; amd64 Windows 7 6.1; java 1.8.0_05; Europe/es) http://yacy.net/bot.html
	NETCRAFT
	Mozilla/5.0 (compatible; WbSrch/1.0; +https://wbsrch.com)
	WebCookies/1.0 (+http://webcookies.info/faq/#agent)
	netEstate Impressumscrawler (+http://www.netestate.de/De/Loesungen/Impressumscrawler)
	msnbot-media/2.0b (+http://search.msn.com/msnbot.htm)
	CopperEgg/RevealUptime/DublinIEUSA
	StormCrawler/1.0 (a Storm-based crawler; https://github.com/DigitalPebble/storm-crawler; [email protected])
	WatchMouse/8.4.0.3 (http://watchmouse.com/ ; uschi02.watchmouse.net)
	Mozilla/5.0 (Linux; Android 5.0.2; SM-G920T Build/LRX22G) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/40.0.2125.111 Mobile Safari/537.36 DareBoost
	Mozilla/5.0 (compatible; DeuSu/0.1.0; +https://deusu.org)
	Mozilla/5.0 (compatible; MJ12bot/v1.4.4 (domain ownership verifier); http://www.majestic12.co.uk/bot.php?+)
	Mozilla/5.0 (compatible; SemrushBot/0.99~bl; +http://www.semrush.com/bot.html)
	WebWatch/Robot_txtChecker
	Mozilla/5.0 (compatible; linkdexbot/2.2; +http://www.linkdex.com/bots/)
	Feedly/1.0 (+http://www.feedly.com/fetcher.html; like FeedFetcher-Google)
	Toweyabot: toweya.com
	Mozilla/5.0 (compatible; Infohelfer/1.4.3; +http://www.infohelfer.de/crawler.php)
	Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.2.1; aggregator:Spinn3r (Spinn3r 3.1); http://spinn3r.com/robot) Gecko/20021130
	FeedCatBot/3.0 (+http://www.feedcat.net/)
	LinkedInBot/1.0 (compatible; Mozilla/5.0; Apache-HttpClient +http://www.linkedin.com), libot/Nutch-1.9 (http://www.linkedin.com; [email protected])
	R6_CommentReader(www.radian6.com/crawler)
	Mozilla/5.0 (compatible; Crawlera/1.10.2; UID 70350)
	Mozilla/5.0 (compatible; MJ12bot/v1.2.3; http://www.majestic12.co.uk/bot.php?+)
	Domain Re-Animator Bot (http://domainreanimator.com) - [email protected]
	Riddler (http://riddler.io/about.html)
	Mozilla/5.0 (compatible; Esribot/1.0; http://www.esrihu.hu/)
	Favicon downloader (http://favicon.netk6.com/)
	findlinks/1.1.5-beta7 (+http://wortschatz.uni-leipzig.de/findlinks/)
	ConveraCrawler/0.9d (+http://www.authoritativeweb.com/crawl)
	Mozilla/5.0 (Windows NT 6.1; rv:6.0) Gecko/20110814 Firefox/6.0 Google favicon
	GetFoundBot (+http://www.getfound.cz/getfoundbot/)
	WatchMouse/8.4.0.3 (http://watchmouse.com/ ; usdal02.watchmouse.net)
	Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/48.0.2564.97 Safari/537.36 Yandex.Translate
	Mozilla/5.0 (compatible; NLNZ_IAHarvester2013 +http://natlib.govt.nz/about-us/current-initiatives/web-harvest-2013)
	COMODOSpider/Nutch-1.2
	Mozilla/5.0 (compatible; DomainMacroCrawler/0.1; +http://domainmacro.com)
	Mozilla/5.0 (compatible; heritrix/3.2.0 +http://www.exif-search.com)
	Mozilla/5.0 (compatible; MagiBot/3.6.2; +http://magi.peak-labs.com/robots.txt)
	Mozilla/5.0 (compatible; Alexabot/1.0; +http://www.alexa.com/help/certifyscan; [email protected])
	psbot-image (+http://www.picsearch.com/bot.html)
	Mozilla/5.0 (Macintosh; Intel Mac OS X 10.9; rv:28.0) Gecko/20100101 Firefox/28.0 (FlipboardProxy/1.6; +http://flipboard.com/browserproxy)
	larbin_2.6.4 ([email protected])
	yacybot (/global; amd64 Windows 7 6.1; java 1.7.0_55; Europe/ru) http://yacy.net/bot.html
	BacklinkCrawler (http://www.backlinktest.com/crawler.html)
	Mozilla/5.0 (Windows NT 6.1; WOW64) adbeat.com/policy AppleWebKit/537.36 (KHTML, like Gecko) Chrome/48.0.2564.116 Safari/537.36
	Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/534.57.2 (KHTML, like Gecko) Version/5.1.7 Safari/534.57.2 PTST/276
	Tools4noobs.com/1.0 Spider
	yacybot (freeworld/global; amd64 Linux 2.6.32-40-generic; java 1.6.0_20; Europe/de) http://yacy.net/bot.html
	Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US) Speedy Spider (http://www.entireweb.com/about/search_tech/speedy_spider/)
	istellabot/t.1
	Mozilla/5.0 (compatible; Lipperhey-Kaus-Australis/5.0; +https://www.lipperhey.com/en/about/)
	Mozilla/5.0 (compatible; Veoozbot/1.0; +http://www.veooz.com/veoozbot.html)
	SemrushBot/Nutch-1.5-SNAPSHOT
	Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)
	Mozilla/5.0 (X11; Linux x86_64; rv:47.0; GTmetrix https://gtmetrix.com/) Gecko/20100101 Firefox/47.0
	CopperEgg/RevealUptime/Oregon(aws)
	yacybot (/global; amd64 Linux 3.16.0-4-amd64; java 1.8.0_102; Europe/en) http://yacy.net/bot.html
	rogerbot/1.0 (http://www.seomoz.org/dp/rogerbot, [email protected])
	yacybot (amd64 Linux 2.6.26-2-amd64; java 1.6.0_20; Europe/en) http://yacy.net/bot.html
	Mozilla/5.0 (Windows NT 6.3; WOW64; Trident/7.0; rv:11.0; BingPreview/1.0b) like Gecko
	Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 5.1; Trident/4.0; .NET CLR 1.1.4322; .NET CLR 2.0.50727; .NET CLR 3.0.04506.30; .NET CLR 3.0.4506.2152; .NET CLR 3.5.30729; MDDS; PTST 2.386)
	Mozilla/5.0 (compatible; RavenCrawler/2.0; +https://raventools.com/seo-website-auditor/)
	Mozilla/5.0 (compatible; YandexMetrika/2.0; +http://yandex.com/bots)
	Mozilla/5.0 (compatible; Scrubby/3.2; +http://seotools.scrubtheweb.com/webpage-analyzer.html)
	dubaiindex (adressendeutschland.de)
	Mozilla/5.0 (compatible; spbot/5.0.2; +http://OpenLinkProfiler.org/bot )
	Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; Netcraft SSL Server Survey - contact [email protected])
	Mozilla/5.0 (compatible; CompSpyBot/1.0; +http://www.compspy.com/spider.html)
	Mozilla/5.0 (compatible; LA2; +http://www.zeerch.com/zeerch2/bot.php)
	yacybot (/global; amd64 Linux 2.6.32-042stab108.8; java 1.7.0_91; America/en) http://yacy.net/bot.html
	updated/0.1-alpha (updated crawler; http://www.updated.com; [email protected])
	CopperEgg/RevealUptime/FrankfurtGermany
	Scrapy/1.1.2 (+http://scrapy.org)
	yrspider (Mozilla/5.0 (compatible; YRSpider; +http://www.yunrang.com/yrspider.html))
	Mozilla/5.0 (compatible; YandexAntivirus/2.0; +http://yandex.com/bots)
	yacybot (freeworld/global; amd64 Windows 7 6.1; java 1.6.0_31; Europe/de) http://yacy.net/bot.html
	Surphace Scout&v4.0 - scout at surphace dot com
	yacybot (freeworld/global; amd64 Linux 2.6.32-5-xen-amd64; java 1.6.0_18; Europe/fr) http://yacy.net/bot.html
	MetaGeneratorCrawler/1.1 (www.metagenerator.info)
	Porkbun/Mustache (Website Analysis; http://porkbun.com; [email protected])
	Mozilla/5.0 (compatible; ScoutJet; +http://www.scoutjet.com/)
	Ruby, link_thumbnailer
	yacybot (freeworld/global; amd64 Windows Server 2008 R2 6.1; java 1.7.0_25; Europe/de) http://yacy.net/bot.html
	yacybot (/global; amd64 Linux 3.13.0-042stab093.4; java 1.7.0_79; Europe/de) http://yacy.net/bot.html
	Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.11 (KHTML, like Gecko) Chrome/23.0.1271.64 Safari/537.11 GotSiteMonitor.com
	Mozilla/5.0 (Linux; Android 4.4; Nexus 5 Build/KRT16M) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/40.0.2125.111 Mobile Safari/537.36 DareBoost
	Mozilla/5.0 (KeepRight OpenStreetMap Checker; http://keepright.at) Gecko/20100101 Firefox/22.0
	Mozilla/5.0 (Windows NT 6.1; WOW64; Trident/7.0; rv:11.0; topster.de Linkchecker 5.0) like Gecko (194.228.205.74)
	Mozilla/5.0 AppleWebkit/537.36 (KHTML, like Gecko) Trident/7.0; rv:11.0, Chrome/42.0.2311.135, Edge/12.10136, http://www.shrinktheweb.com/, Webshot/0.9
	topicbot/1.0 (Mozilla;I;+http://www.topic.bot/contact_page.html)
	DataparkSearch/4.53 (+http://dataparksearch.org/bot)
	Mozilla/5.0 (compatible; Linux; InfegyAtlas/1.0; en-US; [email protected])
	Mozilla/5.0 (X11; Linux x86_64; rv:41.0; GTmetrix https://gtmetrix.com/) Gecko/20100101 Firefox/41.0
	Mozilla/5.0 (compatible; EasouSpider; +http://www.easou.com/search/spider.html)
	Mozilla/5.0 (compatible; parsijoo-update-crawler; +http://www.parsijoo.ir/; [email protected])
	Abrave Spider v4 Robot 1 (http://robot.abrave.co.uk)
	Mozilla/5.0 (compatible; memoryBot/1.21.24 +http://internetmemory.org/en/)
	Mozilla/5.0 (compatible; STINGbot/1.0; +http://136.186.231.16)
	domainsbot (+http://www.domainsbot.com)
	WEPA/3.1 (http://www.wepa.com/; [email protected])
	Mozilla/5.0 (compatible; linkdexbot/2.1; +http://www.linkdex.com/bots/)
	Mozilla/5.0 (compatible; forensiq; +http://www.forensiq.com)
	192.comAgent
	JUST-CRAWLER(+http://www.justsystems.com/jp/tech/crawler/)
	Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm) SitemapProbe
	HuaweiSymantecSpider/[email protected]+(compatible; MSIE 7.0; Windows NT 5.1; Trident/4.0; .NET CLR 2.0.50727; .NET CLR 3.0.4506.2152; .NET CLR ; http://www.huaweisymantec.com/en/IRL/spider)
	Mozilla/5.0 ( Macintosh; Intel Mac OS X 10_10_1 ) AppleWebKit/600.2.5 ( KHTML, like Gecko ) Version/8.0.2 Safari/600.2.5 ( compatible; CloudServerMarketSpider/1.0; +http://cloudservermarket.com/spider.html )
	Mozilla/5.0 (compatible; MJ12bot/v1.3.0; http://www.majestic12.co.uk/bot.php?+)
	Mozilla/5.0 (compatible; AcoonBot/4.10.6; +http://www.acoon.de/robot.asp)
	Scrapy/1.0.1 (+http://scrapy.org)
	seegnifybot/1.0.0 (http://www.seegnify.com/bot)
	omgili/0.5 +http://omgili.com
	Slack-ImgProxy 149 (+https://api.slack.com/robots)
	yacybot (/global; amd64 Linux 3.16.0-4-amd64; java 1.7.0_79; Europe/de) http://yacy.net/bot.html
	www.integromedb.org/Crawler
	Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm
	Mozilla/5.0 (compatible) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/47.0.2526.73 Safari/537.36 [email protected]
	Mozilla/5.0 (compatible; IstellaBot/1.18.81 +http://www.tiscali.it/)
	Gigabot/1.0
	page_test ([email protected])
	istellabot/Nutch-1.10
	Mozilla/5.0 (iPhone; U; CPU iPhone OS 4_1 like Mac OS X; en-us) AppleWebKit/532.9 (KHTML, like Gecko) Version/4.0.5 Mobile/8B117 Safari/6531.22.7 (compatible; Mediapartners-Google/2.1; +http://www.google.com/bot.html)
	ldspider (http://code.google.com/p/ldspider/wiki/Robots)
	yacybot (/global; amd64 Windows 8.1 6.3; java 1.8.0_45; Europe/ru) http://yacy.net/bot.html
	facebookplatform/1.0 (+http://developers.facebook.com)
	Mozilla/5.0 (compatible; RTGI; http://linkfluence.net/)
	yacybot (freeworld/global; amd64 Linux 3.12.43-52.6-default; java 1.8.0_40; Europe/en) http://yacy.net/bot.html
	Mozilla/5.0 (compatible; YandexBot/3.0; MirrorDetector; +http://yandex.com/bots)
	Mozilla/5.0 (compatible; DCPbot/1.2; +http://domains.checkparams.com/)
	WatchMouse/18990 (http://watchmouse.com/ ; d2.watchmouse.com)
	MnoGoSearch/3.3.15
	Mozilla/5.0 (compatible; AboutUsBot/0.9; +http://www.aboutus.org/AboutUsBot)
	Web-sniffer/1.1.0 (+http://web-sniffer.net/)
	Mozilla/4.0 (compatible; Fooooo_Web_Video_Crawl http://fooooo.com/bot.html)
	50.nu/0.01 ( +http://50.nu/bot.html )
	Mozilla/4.0 (compatible; MSIE 7.0; Windows; Windows NT 5.1) BrokenLinkCheck.com/1.1
	Comodo-Certificates-Spider
	Iframely/0.9.8 (+https://iframely.com/;)
	MnoGoSearch/3.3.12
	CopperEgg/RevealUptime/FrankfurtEU
	Python-urllib/2.7 (+http://ella.juls.savba.sk/aranea_about)
	Mozilla/5.0 (compatible; JobKereso; +http://www.kozvetlen-allasok.hu/robot.jsp [email protected])
	Mozilla/5.0 (Windows NT 5.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/49.0.2623.112 Safari/537.36 PTST/201
	Mozilla/5.0 (compatible; Google Keyword Tool; +http://adwords.google.com/select/KeywordToolExternal)
	WinWebBot/1.0; (Balaena Ltd, UK); http://www.balaena.com/winwebbot.html; [email protected];)
	AppleNewsBot
	classbot (+http://allclasses.com)
	yacybot (freeworld/global; amd64 Windows Server 2008 R2 6.1; java 1.6.0_31; America/pt) http://yacy.net/bot.html
	Mozilla/5.0 (iPhone; CPU iPhone OS 7_0 like Mac OS X) AppleWebKit/537.51.1 (KHTML, like Gecko) Version/7.0 Mobile/11A465 Safari/9537.53 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)
	yacybot (/global; arm Linux 3.12.28+; java 1.7.0_71; Europe/en) http://yacy.net/bot.html
	Mozilla/5.0 (compatible; YandexMetrika/2.0; +http://yandex.com/bots mtmon01e.yandex.ru)
	CopperEgg/RevealUptime/TokyoJP
	Mozilla/5.0 (compatible; AhrefsBot/3.1; +http://ahrefs.com/robot/)
	CJNetworkQuality; http://www.cj.com/networkquality
	psbot-page (+http://www.picsearch.com/bot.html)
	DealGates Bot/1.1 by Luc Michalski (http://spider.dealgates.com/bot.html)
	Pingdom GIGRIB (http://www.pingdom.com)
	Mozilla/5.0 (compatible; SEOlyticsCrawler/3.0; +http://crawler.seolytics.net/)
	Mozilla/5.0 (compatible; EventGuruBot/1.0; +http://www.eventguru.com/spider.html)
	Python-urllib/2.7 (+http://aranea.juls.savba.sk/aranea_about.html)
	yacybot (/global; amd64 Windows Vista 6.0; java 1.7.0_55; America/en) http://yacy.net/bot.html
	The Lemur Web Crawler/Nutch-1.3 (Lemur Web Crawler; http://boston.lti.cs.cmu.edu/crawler_12/; [email protected])
	SAMSUNG-SGH-E250/1.0 Profile/MIDP-2.0 Configuration/CLDC-1.1 UP.Browser/6.2.3.3.c.1.101 (GUI) MMP/2.0 (compatible; Googlebot-Mobile/2.1; +http://www.google.com/bot.html)
	Feedfetcher-Nuesbyte; (+http://www.nuesbyte.com)
	copyright sheriff (+http://www.copyrightsheriff.com/)
	yacybot (/global; amd64 Linux 3.2.0-4-amd64; java 1.7.0_75; Europe/en) http://yacy.net/bot.html
	Touche (+http://www.touche.com.ve)
	Panopta v1.1
	Mozilla/5.0 (compatible; parsijoo-bot; +http://www.parsijoo.ir/; [email protected])
	Zookabot/2.2;++http://zookabot.com
	yacybot (/global; amd64 Linux 2.6.32-042stab093.4; java 1.7.0_65; Etc/en) http://yacy.net/bot.html
	it2media-domain-crawler/2.0
	yacybot (/global; amd64 Linux 4.2.0-27-generic; java 1.7.0_95; Europe/en) http://yacy.net/bot.html
	Mozilla/5.0 (Android; Mobile; rv:21.0) Gecko/21.0 Firefox/21.0 commoncrawl.org/research//Nutch-1.7-SNAPSHOT
	Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 5.2; Trident/4.0; Data Center; .NET CLR 1.1.4322; .NET CLR 2.0.50727; .NET CLR 3.0.04506.648; .NET CLR 3.5.21022; .NET CLR 3.0.4506.2152; .NET CLR 3.5.30729; PTST 2.386)
	Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/99.0 Safari/537.36 YottaaMonitor
	CopperEgg/RevealUptime/AtlantaGAUSA
	Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
	yacybot (/global; amd64 Windows 7 6.1; java 1.7.0_55; Europe/fr) http://yacy.net/bot.html
	Mozilla/5.0 (compatible; MJ12bot/v1.2.4; http://www.majestic12.co.uk/bot.php?+)
	Mozilla/5.0 (compatible; Applebot/0.3; +http://www.apple.com/go/applebot)
	ICC-Crawler/2.0 (Mozilla-compatible; ; http://www.nict.go.jp/en/ucri/plan/crawl.html)
	Mozilla/5.0 (compatible; YYSpider; +http://www.yunyun.com/spider.html)
	Mozilla/5.0 (iPhone; CPU iPhone OS 5_0_1 like Mac OS X) (compatible; Yeti-Mobile/0.1; +http://help.naver.com/robots/)
	Mozilla/5.0 (WhatsMyIP.org Text_to_Code_Ratio_Tool) http://whatsmyip.org/ua
	Thumbnail.CZ robot 1.1 (http://thumbnail.cz/why-no-robots-txt.html)
	Google-Calendar-Importer
	Mozilla/5.0 (Windows NT 6.3; WOW64) AppleWebKit/534.34 (KHTML, like Gecko) Websnapr/3.0 Safari/534.34
	Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/50.0.2661.75 Safari/537.36 PTST/276
	WeSEE
	librabot/2.0 (+http://academic.research.microsoft.com/)
	Testomatobot/1.0 (Linux x86_64; +http://www.testomato.com/testomatobot) minicrawler/3.0.1
	Mozilla/5.0 (compatible; MSIE 10.0; Windows NT 6.1; Trident/6.0) PTST/276
	Cliqzbot/0.1 (+http://cliqz.com [email protected])
	Scrubby/3.0 (+http://www.scrubtheweb.com/help/technology.html)
	Pinterest/0.1 +http://pinterest.com/
	Mozilla/5.0 (compatible; Crawlera/1.10.2; UID 52327)
	Comodo-Webinspector-Crawler 2.2.0, http://www.comodorobot.com
	Iframely/0.6.0 (+http://iframely.com/;)
	Mozilla/5.0 (compatible; PaperLiBot/2.1; http://support.paper.li/entries/20023257-what-is-paper-li)
	Mozilla/5.0 (compatible; SiteExplorer/1.0b; +http://siteexplorer.info/)
	Y!J-BSC/1.0 (http://help.yahoo.co.jp/help/jp/blog-search/)
	Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/49.0.2623.112 Safari/537.36 PTST/276
	Mozilla/5.0 (compatible; U; Hootsuite-WebFeed/1.0; mailto:[email protected])
	RankFlex.com Webspider
	TipTop http://feeltiptop.com
	vebidoobot
	Mozilla/5.0 (compatible; Dow Jones Searchbot)
	Mozilla/5.0 (compatible; OrangeBot-Collector/2.0; [email protected])
	Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/534+ (KHTML, like Gecko) MsnBot-Media /1.0b
	Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1.6) Gecko/20070725 Firefox/2.0.0.6 Ara.com.tr AraBot 1.0
	Mozilla/5.0 (iPhone; CPU iPhone OS 9_1 like Mac OS X) AppleWebKit/601.1.46 (KHTML, like Gecko) Version/9.0 Mobile/13B143 Safari/601.1 (compatible; AdsBot-Google-Mobile; +http://www.google.com/mobile/adsbot.html)
	Mozilla/5.0 (compatible; YandexMetrika/2.0; +http://yandex.com/bots mtmon01g.yandex.ru)
	BUbiNG (+http://law.di.unimi.it/BUbiNG.html)
	downnotifier.com monitoring
	A6-Indexer
	Mozilla/5.0 (compatible; htrix/1.14.4 ++https://webarchiv.onb.ac.at/robot.html)
	MetaURI API/2.0 +metauri.com
	Mozilla/5.0 (compatible; MixrankBot; [email protected])
	Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/44.0.2403.155 Safari/537.36 TinEye/1.0 (via http://www.tineye.com/)
	Mozilla/5.0 Project 25499 (project25499.com)
	COMODOspider/Nutch-1.0
	Mozilla/5.0 (compatible; CareerBot/1.1; +http://www.career-x.de/bot.html)
	Uptime.com_(http://uptime.com/)
	AcoonBot/4.10.5 (+http://www.acoon.de)
	BDFetch
	20090717_aa_page ([email protected])
	Metaspinner/1.0 (Metaspinner Search Engine; http://www.meta-spinner.de/; [email protected])
	Mozilla/5.0 (compatible; DBLBot/1.0; +http://www.dontbuylists.com/)
	Mozilla/5.0 (compatible; spbot/2.0; +http://www.seoprofiler.com/bot/ )
	LYT.SR v1.5 +http://lyt.sr/ (running on http://lyt.sr)
	Putin (+http://semanticjuice.com/)
	Mozilla/5.0 (compatible; LA1; +http://www.zeerch.com/bot.php)
	WatchMouse/18990 (http://watchmouse.com/ ; ny)
	bitlybot/2.0
	yacybot (/global; amd64 Linux 3.13.0-45-generic; java 1.7.0_75; Europe/en) http://yacy.net/bot.html
	Webshot/0.9 ( http://www.shrinktheweb.com/ )
	MetaURI API +metauri.com
	Mozilla/5.0 (compatible; ProductoDownloadUrlBot/1.0; +http://www.producto.de/)
	Google-Adwords-Instant (+http://www.google.com/adsbot.html)
	yacybot (freeworld/global; amd64 Linux 3.8.0-19-generic; java 1.7.0_25; Europe/en) http://yacy.net/bot.html
	yacybot (freeworld/global; i386 Linux 2.6.32-5-686; java 1.6.0_18; Europe/en) http://yacy.net/bot.html
	Mozilla/5.0 (Windows NT 6.3; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/42.0.2311.90 Safari/537.36 PTST/277
	Mozilla/5.0 (compatible; DomainSigmaCrawler/0.1; +http://domainsigma.com/robot)
	HyperZbozi.cz Feeder/3.1
	Mozilla/5.0 (Windows; U; Windows NT 5.1; en; rv:1.9.0.13) Gecko/2009073022 Firefox/3.5.2 (.NET CLR 3.5.30729) SurveyBot/2.3 (DomainTools)
	yacybot (/global; amd64 Windows Server 2012 R2 6.3; java 1.7.0_55; Europe/de) http://yacy.net/bot.html
	Google-AdWords-Express
	Mozilla/5.0 (compatible; MJ12bot/v1.4.0; http://www.majestic12.co.uk/bot.php?+)
	SMTBot (similartech.com/smtbot)
	Mozilla/5.0 (compatible; Lipperhey SEO Service; http://www.lipperhey.com/)
	MetaInspector/4.7.2 (+https://github.com/jaimeiniesta/metainspector)
	Mozilla/5.0 (Compatible; Vedma/0.91Beta; +http://www.vedma.ru/bot.htm)
	Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.21 (KHTML, like Gecko) web-capture.net/0.9 Safari/537.21
	Nokia6680/1.0 (4.04.07) SymbianOS/8.0 Series60/2.6 Profile/MIDP-2.0 Configuration/CLDC-1.1 (botmobi find.mobi/bot.html [email protected])
	Mozilla/5.0 (compatible; Exploratodo/1.0; +http://www.exploratodo.com
	iZSearch.com
	Mozilla/5.0 (compatible; YandexFavicons/1.0; +http://yandex.com/bots)
	bl.uk_lddc_bot/3.3.0-SNAPSHOT-2014-10-07T09:33:31Z (+http://www.bl.uk/aboutus/legaldeposit/websites/websites/faqswebmaster/index.html)
	yacybot (/global; amd64 Linux 3.14-0.bpo.1-amd64; java 1.7.0_55; Europe/de) http://yacy.net/bot.html
	HuaweiSymantecSpider/[email protected]+(compatible; MSIE 7.0; Windows NT 5.1; Trident/4.0; .NET CLR 2.0.50727; .NET CLR 3.0.4506.2152; .NET CLR ; http://www.huaweisymantec.com/cn/IRL/spider)
	FAST-WebCrawler/3.7/FirstPage (atw-crawler at fast dot no;http://fast.no/support/crawler.asp)
	Mozilla/5.0 (compatible; DripfeedBot/2.0; +http://dripfeedbookmark.com/bot.html
	Mozilla/5.0 (compatible; heritrix/2.0.2 +http://seekda.com)
	yacybot (freeworld/global; i386 Linux 3.2.0-23-generic-pae; java 1.7.0_03; Europe/en) http://yacy.net/bot.html
	hawkReader/0.9b (Feed Parser; http://www.hawkreader.com; Allow like Gecko)
	ruijie ([email protected])
	CorpusCrawler 2.0.0 (http://corpora.fi.muni.cz/crawler/)
	ExactSeek Crawler (http://www.exactseek.com/)/Nutch-1.4
	WordPress.com; https://botd.wordpress.com
	Mozilla/5.0 (compatible; Qwantify/2.0n; +https://www.qwant.com/)/*
	Mozilla/5.0 (compatible; memoryBot/1.20.268 +http://internetmemory.org/en/)
	ShowyouBot (http://showyou.com/crawler)
	Yoleo Consumer v0.2
	Mozilla/5.0 (compatible; IstellaBot/1.10.2 +http://www.tiscali.it/)
	Mozilla/5.0 (compatible; MSIE or Firefox mutant; not on Windows server; +http://ws.daum.net/aboutWebSearch.html) Daumoa/3.0
	Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/46.0.2490.86 YaBrowser/15.12.0.6151 Safari/537.36 Yandex.Translate
	Mozilla/5.0 (compatible; TeeRaidBot; +https://teeraid.com/bot/)
	Mozilla/5.0 (compatible; Scopia Crawler 1.1; +http://www.scopia.co)
	CopperEgg/RevealUptime/DublinIE(aws)
	yacybot (freeworld/global; amd64 Windows 7 6.1; java 1.7.0_04; Asia/ja) http://yacy.net/bot.html
	RankurBot/Rankur2.1 (http://rankur.com; info at rankur dot com)
	Gaisbot/3.0+([email protected];+http://gais.cs.ccu.edu.tw/robot.php)
	Mozilla/5.0 (compatible; zitebot support [at] zite [dot] com +http://zite.com)
	MetaTagRobot/2.0 (http://www.check-domains.com/website-analysis/website-analyzer.php)
	Virusdie crawler/2.1
	findlinks/2.0.4 (+http://wortschatz.uni-leipzig.de/findlinks/)
	Mozilla/5.0 (compatible; AcoonBot/4.10.7; +http://www.acoon.de/robot.asp)
	Mozilla/5.0 (compatible; DeuSu/5.0.2; +https://deusu.de/robot.html)
	yacybot (-global; amd64 Linux 3.19.0-15-generic; java 1.8.0_45-internal; Europe/de) http://yacy.net/bot.html
	Google_Analytics_Snippet_Validator
	SurcentroBot
	Mozilla/5.0 (compatible; Diffbot/0.1; +http://www.diffbot.com)
	findlinks/1.1.3-beta9 (+http://wortschatz.uni-leipzig.de/findlinks/)
	Mozilla/5.0 (compatible; memoryBot/1.20.199 +http://internetmemory.org/en/)
	Y!J-ASR/0.1 crawler (http://www.yahoo-help.jp/app/answers/detail/p/595/a_id/42716/)
	TwengaBot-Discover (http://www.twenga.fr/bot-discover.html)
	Mozilla/5.0 (bl.uk_lddc_bot; Linux x86_64) PhantomJS/1.9.7 (+http://www.bl.uk/aboutus/legaldeposit/websites/websites/faqswebmaster/index.html)
	panscient.com
	Mozilla/5.0 (compatible; TestCrawler; +http://wiki.github.com/bixo/bixo/bixocrawler; [email protected])
	DataparkSearch/4.37-23012006 ( http://www.dataparksearch.org/)
	GOFORITBOT ( http://www.goforit.com/about/ )
	CopperEgg/RevealUptime/SydneyAustralia
	msnbot-media/1.0 (+http://search.msn.com/msnbot.htm)
	Mozilla/5.0 (compatible; MegaIndex.ru/2.0; +https://www.megaindex.ru/?tab=linkAnalyze)
	MetaCommentBot; http://metacomment.io/about
	Mozilla/5.0 (compatible; YesupBot/1.0; +http://www.yesup.net/bot.html)
	Twikle/1.0 , http://twikle.com , [email protected]
	yacybot (/global; amd64 Windows Server 2008 R2 6.1; java 1.8.0_60; Europe/de) http://yacy.net/bot.html
	Mozilla/4.0 (compatible; Vagabondo/4.0Beta; webcrawler at wise-guys dot nl; http://webagent.wise-guys.nl/; http://www.wise-guys.nl/)
	Comodo-Webinspector-Crawler 2.1
	Mozilla/5.0 (Macintosh; Intel Mac OS X 10_7_5) AppleWebKit/537.11 (KHTML, like Gecko) Chrome/23.0.1271.64 Safari/537.11 KimonoLabs/0.2
	Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1 + FairShare-http://fairshare.cc)
	Scrapy/0.24.5 (+http://scrapy.org)
	Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/534.34 (KHTML, like Gecko) Site-Shot/2.1 (http://www.site-shot.com/) Safari/534.34
	Port Monitor check service 1.0 | http://www.port-monitor.com
	Mozilla/5.0 (compatible; Uptimebot/0.1.43; +http://www.uptime.com/uptimebot)
	yacybot (freeworld/global; amd64 Linux 2.6.32-5-amd64; java 1.6.0_18; Europe/en) http://yacy.net/bot.html
	SolomonoBot/1.04 (http://www.solomono.ru)
	Easy-Thumb (https://www.easy-thumb.net/)
	Mozilla/5.0 (Windows NT 6.0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/49.0.2623.112 Safari/537.36 PTST/276
	nic.it crawler v.1.221115 (7.35.0)
	Mozilla/5.0 (compatible; memorybot/1.24.12 +https://internetmemory.net/en/bots)
	Mozilla/5.0 (compatible; inoreader.com-like FeedFetcher-Google)
	yacybot (/global; amd64 Windows 7 6.1; java 1.7.0_72; Europe/de) http://yacy.net/bot.html
	WatchMouse/8.4.0.3 (http://watchmouse.com/ ; demuc02.watchmouse.net)
	Mozilla/5.0 (X11; Linux x86_64; rv:43.0; GTmetrix https://gtmetrix.com/) Gecko/20100101 Firefox/43.0
	ShopWiki/1.0 ( +http://www.shopwiki.com/wiki/Help:Bot)
	Mozilla/5.0 (compatible; UnisterBot; http://www.bluekiwi.de/misc/imprint)
	Web-sniffer/1.0.31 (+http://web-sniffer.net/)
	woobot
	Mozilla/5.0 (X11; U; Linux Core i7-4980HQ; de; rv:32.0; compatible; JobboerseBot; https://www.jobboerse.com/bot.htm) Gecko/20100401 Firefox/24.0
	AntBot/Ant-Nutch-1.1 (Ant Nutch Crawler; http://www.ant.com; [email protected])
	Mozilla/5.0 (compatible; YandexMedia/3.0; +http://yandex.com/bots)
	Mozilla/5.0 (compatible; Evrinid Iudex 1.0.0; +http://www.evri.com/evrinid)
	adidxbot/1.1 (+http://search.msn.com/msnbot.htm)
	Mozilla/5.0 (compatible; WebThumbnail/2.2; Website Thumbnail Generator; +http://webthumbnail.org)
	Mozilla/5.0 (compatible; Embedly/0.2; +http://support.embed.ly/)
	EasyBib AutoCite (http://content.easybib.com/autocite/)
	yacybot (-global; amd64 Linux 3.2.0-4-amd64; java 1.7.0_65; Europe/en) http://yacy.net/bot.html
	Mozilla/5.0 (compatible; woriobot +http://worio.com)
	Mozilla/4.0 (compatible; Vagabondo/4.0; http://webagent.wise-guys.nl/)
	SniffRSS/0.5beta (+http://www.blogator.com/)
	woobot/1.1
	Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) WordPress.com mShots Safari/537.36
	ADmantX Platform Semantic Analyzer - Turn - ADmantX Inc. - www.admantx.com - [email protected]
	Mozilla/5.0 (compatible; UnisterBot; [email protected])
	Mozilla/5.0 (compatible; electricmonk/3.2.0 +https://www.duedil.com/our-crawler/)
	UptimeDog Robot (www.uptimedog.com)
	Googlebot/2.1 (+http://www.google.com/bot.html)
	Mozilla/5.0 (Windows NT 6.3; WOW64; rv:36.0) Gecko/20100101 Firefox/36.0 (NetShelter ContentScan, contact [email protected] for information)
	Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.0; Trident/4.0; SLCC1; .NET CLR 2.0.50727; Media Center PC 5.0; .NET CLR 3.5.30729; .NET4.0C; .NET CLR 3.0.30729; .NET4.0E) PTST/276
	ServiceUptime.robot
	Mozilla/4.0 (compatible;HostTracker/2.0;+http://www.host-tracker.com/)
	SEO Browser
	Leikibot/1.0 (+http://www.leiki.com)
	yacybot (webportal-global; amd64 Windows 7 6.1; java 1.7.0_04; America/en) http://yacy.net/bot.html
	AppEngine-Google; (+http://code.google.com/appengine; appid: downforeveryoneorjustme)
	Mozilla/5.0 (Windows NT 5.1) BrokenLinkCheck.com/1.1
	Seobility (SEO-Check; http://bit.ly/1dJuuzs)
	Mozilla/5.0 (seoanalyzer; compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)
	Mozilla/5.0 (compatible; Digincore bot; https://www.digincore.com/crawler.html for rules and instructions.
	Site24x7
	websitepulse checker/3.0 (compatible; MSIE 5.5; Netscape 4.75; Linux)
	Mozilla/5.0 (compatible; SemrushBot-SA/0.97; +http://www.semrush.com/bot.html)
	Mozilla/5.0 (compatible; NerdByNature.Bot; http://www.nerdbynature.net/bot)
	Abrave Spider v4 Robot 2 (http://robot.abrave.co.uk)
	HubPages V0.2.2 (http://hubpages.com/help/crawlingpolicy)
	lmspider ([email protected])
	magpie-crawler/1.1 (U; Linux amd64; en-GB; +http://www.brandwatch.net)
	Mozilla/5.0 (compatible; WbSrch/1.1 +http://wbsrch.com
	Mozilla/5.0 (Windows; U; Windows NT 5.1; de; rv:1.9.0.7; Google-SearchByImage) Gecko/2009021910 Firefox/3.0.7
	Mozilla/5.0 (compatible; Najdi.si/3.1)
	larbin_2.6.3 ([email protected])
	Mozilla/5.0 (compatible; aiHitBot/1.1; +http://www.aihit.com/)
	Mozilla/5.0 (Windows NT 6.3; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/50.0.2661.102 Safari/537.36 PTST/277
	Mozilla/5.0 (YahooYSMcm/3.0.0; http://help.yahoo.com)
	yacybot (/global; amd64 Linux 3.16.0-4-amd64; java 1.7.0_79; Europe/en) http://yacy.net/bot.html
	OOZBOT/0.20 ( http://www.setooz.com/oozbot.html ; agentname at setooz dot_com )
	findlinks/2.0.9 (+http://wortschatz.uni-leipzig.de/findlinks/)
	Mozilla/5.0 (compatible; Lipperhey Link Explorer; http://www.lipperhey.com/)
	Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.6; en-US; rv:1.9.2) Gecko/20100115 Firefox/3.6 (FlipboardProxy/1.1; +http://flipboard.com/browserproxy)
	Mozilla/5.0 (Twiceler-0.9 http://www.cuil.com/twiceler/robot.html)
	Mozilla/5.0 (compatible; memoryBot/1.24.11 +http://internetmemory.org/en/)
	RED/1 (https://redbot.org/)
	Twitterbot/1.0
	Mozilla/5.0 (compatible; ptd-crawler; +http://bixolabs.com/crawler/ptd/; [email protected])
	Open Web Analytics Bot 1.5.4
	JikeSpider Mozilla/5.0 (compatible; JikeSpider; +http://shoulu.jike.com/spider.html)
	Mozilla/5.0 (compatible; vkShare; +http://vk.com/dev/Share)
	Mozilla/5.0 (compatible; oBot/2.3.1; +http://www-935.ibm.com/services/us/index.wss/detail/iss/a1029077?cntxt=a1027244)
	elefent/Elefent 1.2 (A friendly web elefent.; http://elefent.eu/; [email protected])
	Genderanalyzer/1.0
	elefent/1.2 (Web Crawler; https://crawler.elefent.net; webmaster at elefent dot net)
	Yandex/1.01.001 (compatible; Win16; H)
	crawler4j for XQuery
	Mozilla/5.0 (compatible; Plukkie/1.4; http://www.botje.com/plukkie.htm)
	LinguaBot/v0.001-dev (MultiLinual Sarch Engine v0.001; LinguaSeek; admin at linguaseek dot com)
	MrakyPraceCzBot/1.0 (http://www.mrakyprace.cz/about-crawler)
	Y!J-BRJ/YATS crawler (http://help.yahoo.co.jp/help/jp/search/indexing/indexing-15.html)
	WatchMouse/8.4.0.3 (http://watchmouse.com/ ; plgdn01.watchmouse.net)
	Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2272.96 Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
	AskQuickly v2 (http://askquickly.org/)
	Mozilla/5.0 (compatible; SISTRIX Crawler; http://crawler.sistrix.net/)
	Miniflux (http://miniflux.net)
	Mozilla/5.0 (compatible; spbot/5.0.1; +http://OpenLinkProfiler.org/bot )
	Mozilla/5.0 (compatible; wmtips.com/1.0; +http://www.wmtips.com/tools/)
	Mozilla/5.0 (compatible; linkdexbot/2.0; +http://www.linkdex.com/bots/)
	Mozilla/2.0 (compatible; Ask Jeeves/Teoma; +http://sp.ask.com/docs/about/tech_crawling.html)
	Google favicon
	Mozilla/5.0(compatible; Sosospider/2.0; +http://help.soso.com/webspider.htm)
	MiaDev/0.0.1 (MIA Bot for research project MIA (www.MIA-marktplatz.de); http://www.mia-marktplatz.de/spider; [email protected])
	Mozilla/5.0 (compatible; proximic; +http://www.proximic.com/info/spider.php)
	Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/33.0.1750.146 Safari/537.36 Specificfeeds- http://www.specificfeeds.com
	Mozilla/5.0 (compatible; socialbm_bot/1.0; +http://spider.socialbm.net)
	wp.com feedbot/1.0 (+https://wp.com)
	Mozilla/5.0 (compatible; CligooRobot/2.0; +http://www.cligoo.de/wk/technik.php)
	Slack-ImgProxy (+https://api.slack.com/robots)
	Mozilla/5.0 (compatible; Online Domain Tools - HTTP Headers Online/1.0; +http://http-headers.online-domain-tools.com)
	Mozilla/5.0 (compatible; SurdotlyBot/1.0; +http://sur.ly/bot.html)
	WGETbot/1.0 (+http://wget.alanreed.org)
	yacybot (/global; amd64 Linux 4.0.9-2-ck; java 1.8.0_51; Europe/de) http://yacy.net/bot.html
	Y!J-BRI/0.0.1 crawler ( http://help.yahoo.co.jp/help/jp/search/indexing/indexing-15.html )
	dlvr.it/1.0 (+http://dlvr.it/) Mozilla/5.0
	yacybot (/global; arm Linux 4.1.13-v7+; java 1.8.0; Etc/de) http://yacy.net/bot.html
	Mozilla/5.0 (compatible; GimmeUSAbot/1.0; +http://gimmeusa.com/crawler.html)
	yacybot (freeworld/global; amd64 Windows 7 6.1; java 1.6.0_29; Europe/fr) http://yacy.net/bot.html
	Link Valet Online 1.2
	Mozilla/5.0 (compatible; METASpider; +http://meta.ua/spider)
	NeutrinoAPI/2.0.1
	thumbshots-de-bot (+http://www.thumbshots.de/)
	Mozilla/5.0 (iPhone; CPU iPhone OS 6_0 like Mac OS X) AppleWebKit/536.26 (KHTML, like Gecko) Version/6.0 Mobile/10A5376e Safari/8536.25 (compatible; Applebot/0.3; +http://www.apple.com/go/applebot)
	argenfybot
	MergadoBot
	Ant/Ant-Nutch-1.1 (Ant Nutch Crawler; http://www.ant.com; [email protected])
	CopperEgg/RevealUptime/SaoPauloBrazil
	Mozilla/4.0 (compatible; Vagabondo/4.0/EU; http://webagent.wise-guys.nl/)
	Mozilla/4.0 (compatible; DAWINCI ANTIPLAG SPIDER)
	Mozilla/5.0 (Windows NT 6.1; WOW64; rv:45.0) Gecko/20100101 Firefox/45.0 PTST/276
	HetrixTools.COM Uptime Monitoring Bot. https://hetrixtools.com/uptime-monitoring-bot.html
	yacybot (/global; amd64 Linux 3.2.0-4-amd64; java 1.7.0_79; Europe/en) http://yacy.net/bot.html
	Mozilla/5.0 (Windows NT 6.1) (compatible; SMTBot/1.0; +http://www.similartech.com/smtbot)
	Mozilla/5.0 (compatible; InbyBot/2.1; +http://inbyapp.com)
	all_web1 ([email protected])
	Mozilla/5.0 (compatible; BusinessSeek.biz_Directory; http://www.businessseek.biz/)
	Wotbox/2.0 ([email protected]; http://www.wotbox.com)
	Mediatoolkitbot ([email protected])
	OpenWebSpider v0.1.2.B (http://www.openwebspider.org/)
	Mozilla/5.0 (compatible; WebCorp/5.0; +http://www.webcorp.org.uk)
	Testomatobot/1.0 (Linux x86_64; +http://www.testomato.com/testomatobot) minicrawler/4.0.0
	fastbot crawler beta 2.0 (+http://www.fastbot.de)
	Sogou Web Spider
	DoCoMo/2.0 P900i(c100;TB;W24H11) (compatible; ichiro/mobile goo;+http://search.goo.ne.jp/option/use/sub4/sub4-1/)
	Mozilla/5.0 (compatible; imagecoccoc/1.0; +http://help.coccoc.com/searchengine)
	XoviBot/1.0
	Mozilla/5.0 (compatible; Online Domain Tools - Online Website Link Checker/1.1; +http://website-link-checker.online-domain-tools.com)
	WatchMouse/8.4.0.3 (http://watchmouse.com/ ; decgn02.watchmouse.net)
	yacybot (i386 Linux 2.6.31-21-generic; java 1.6.0_0; Europe/en) http://yacy.net/bot.html
	ZnajdzFoto/ImageBot 2.0b
	Mozilla/5.0 (compatible; heritrix/3.0.0-SNAPSHOT-20091120.021634 +http://crawler.archive.org)
	Mozilla/5.0 (compatible; ProCogSEOBot/1.0; +http://www.procog.com/ )
	Mozilla/5.0 (compatible; Plukkie/1.6; http://www.botje.com/plukkie.htm)
	yacybot (freeworld/global; amd64 Linux 3.2.0-21-generic; java 1.7.0_03-icedtea; America/en) http://yacy.net/bot.html
	Mozilla/5.0 (iPhone; CPU iPhone OS 8_3 like Mac OS X) AppleWebKit/600.1.4 (KHTML, like Gecko) Version/8.0 Mobile/12F70 Safari/600.1.4 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
	Mozilla/5.0 (compatible; MSIE or Firefox mutant; not on Windows server; + http://tab.search.daum.net/aboutWebSearch.html) Daumoa/3.0
	Mozilla/5.0 (compatible; NLNZ_IAHarvester2014 +http://natlib.govt.nz/about-us/current-initiatives/web-harvest-2014)
	Mozilla/5.0 (compatible; archive.org_bot +http://archive.org/details/archive.org_bot)
	Mozilla/5.0 (compatible; special_archiver/3.3.0 +http://www.loc.gov/webarchiving/notice_to_webmasters.html)
	yacybot (freeworld/global; i386 Linux 3.4.2-linode44; java 1.6.0_27; America/en) http://yacy.net/bot.html
	Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/47.0.2526.111 Safari/537.36 Yandex.Translate
	yacybot (webportal-global; amd64 Linux 3.2.0-4-amd64; java 1.7.0_25; Europe/en) http://yacy.net/bot.html
	FreeWebMonitoring SiteChecker/0.1 (+http://www.freewebmonitoring.com)
	Mozilla/5.0 (compatible; Infohelfer/1.3.0; +http://www.infohelfer.de/crawler.php)
	Kemvibot/1.0 (http://kemvi.com)
	Mozilla/4.0 (compatible; HostTracker.com/1.0;+http://host-tracker.com/)
	AdsBot-Google-Mobile (+http://www.google.com/mobile/adsbot.html) Mozilla (iPhone; U; CPU iPhone OS 3 0 like Mac OS X) AppleWebKit (KHTML, like Gecko) Mobile Safari
	Mozilla/5.0 (compatible; GeliyooBot/1.0; +http://www.geliyoo.com/)
	yacybot (x86 Windows XP 5.1; java 1.6.0_21; Europe/de) http://yacy.net/bot.html
	Mozilla/5.0 (compatible; archive.org_bot/3.1.2 +http://natlib.govt.nz/about-us/current-initiatives/web-harvest-2012)
	Mozilla/5.0 (compatible; StorygizeBot; http://www.storygize.com)
	infohelfer/1.1.0 (http://www.infohelfer.de/)
	yacybot (/global; amd64 Linux 3.16.0-41-generic; java 1.7.0_80; Europe/de) http://yacy.net/bot.html
	SeoCheckBot (FischerNetzDesign Seo Checker, [email protected])
	netEstate NE Crawler (+http://www.website-datenbank.de/)
	Mail.Ru/1.0
	binlar_2.6.3 ([email protected])
	LinkedInBot/1.0 (compatible; Mozilla/5.0; Jakarta Commons-HttpClient/3.1 +http://www.linkedin.com)
	bot.wsowner.com/1.0.0 (+http://wsowner.com/)
	Ronzoobot/1.5 (http://www.ronzoo.com/about/)
	Mozilla/5.0 (compatible; heritrix/3.2.0 +http://www.mixdata.com)
	Mozilla/5.0 (compatible; Yahoo! Slurp China; http://misc.yahoo.com.cn/help.html)
	findlinks/1.1.6-beta6 (+http://wortschatz.uni-leipzig.de/findlinks/)
	Mozilla/5.0 (compatible; LoadTimeBot/0.7; +http://www.load-time.com/bot.html)
	Mozilla/5.0 (compatible; MJ12bot/v1.4.2; http://www.majestic12.co.uk/bot.php?+)
	Mozilla/4.0 (compatible; MSIE 5.01; GomezAgent 2.0; Windows NT)
	Mozilla/5.0 (compatible; MJ12bot/v1.3.2; http://www.majestic12.co.uk/bot.php?+)
	SEOENGBot/1.2 (+http://learn.seoeng.com/seoengbot.htm)
	yacybot (i386 Linux 2.6.31-18-generic; java 1.6.0_0; Europe/en) http://yacy.net/bot.html
	Online Virus Scanner: http://tools.geek-tools.org
	DwnldBot (+http://dwnld.me)
	Mozilla/5.0 (compatible; OpenfosBot/2.1; http://www.openfos.com)
	taptubot *** please read http://www.taptu.com/corp/taptubot ***
	yacybot (freeworld/global; amd64 Linux 3.9.3-1-ARCH; java 1.7.0_21; America/en) http://yacy.net/bot.html
	Testomatobot/1.0 (Linux x86_64; +http://www.testomato.com/testomatobot) minicrawler/4.0.0~beta10
	rogerbot/1.0 (http://www.moz.com/dp/rogerbot, [email protected])
	Mozilla/5.0 (Windows NT 6.1; Win64; x64) KomodiaBot/1.0
	Mozilla/5.0 (compatible; spbot/3.0; +http://www.seoprofiler.com/bot )
	Mozilla/5.0 (compatible; DomainTunoCrawler/0.1; +https://www.domaintuno.com/robot)
	Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Googlebot/2.1; +http://www.google.com/bot.html) Safari/537.36
	yacybot (amd64 Linux 2.6.32-gentoo; java 1.6.0_17; Europe/en) http://yacy.net/bot.html
	Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko; Google Page Speed Insights) Chrome/27.0.1453 Mobile Safari/537.36
	Mozilla/5.0 (compatible; GimmeUSAbot/1.0; +http://gimmeusa-update.com/crawler)
	Cloud mapping experiment. Contact [email protected]
	emefgebot/beta (+http://emefge.de/bot.html)
	DoCoMo/2.0 D903i(c100;TB;W28H20) (compatible; Hatena-Mobile-Gateway/1.2; +http://mgw.hatena.ne.jp/help)
	ichiro/3.0 (http://help.goo.ne.jp/door/crawler.html)
	Mozilla/5.0 (compatible; SecurityResearch.bot; +http://besome1.info/securityresearchbot.html)
	WPScan v2.9 (http://wpscan.org)
	Ronzoobot/1.6 (http://www.ronzoo.com/about/)
	livedoor ScreenShot/0.10
	Mozilla/5.0 (compatible; MSIE 8.0; Windows NT 5.1) KomodiaBot/1.0
	Mozilla/5.0 (compatible; memoryBot/1.20.289 +http://internetmemory.org/en/)
	Mozilla/5.0 (compatible; DomainTunoCrawler/0.1; +http://www.domaintuno.com/robot)
	yacybot (freeworld/global; amd64 Linux 2.6.26-2-amd64; java 1.6.0_18; Europe/de) http://yacy.net/bot.html
	Mozilla/5.0 (compatible; GimmeUSAbot/1.0; +http://gimmeusa.com/crawler)
	Mozilla/5.0 (compatible; spbot/5.0; +http://OpenLinkProfiler.org/bot )
	Mozilla/5.0 (compatible; Webauskunft.org; http://www.webauskunft.org)
	CCResearchBot/1.0 commoncrawl.org/research//Nutch-1.7-SNAPSHOT
	Mozilla/4.0 (compatible; MSIE 5.5; Windows NT 4.0; obot)
	linkdex.com/v2.0
	Mozilla/5.0 (compatible; Crawlera/1.10.2; UID 73393)
	Mozilla/5.0 (compatible; SEODat/0.1 http://crawler.seodat.com)
	yacybot (freeworld/global; amd64 Linux 3.2.0-4-amd64; java 1.6.0_27; Europe/en) http://yacy.net/bot.html
	CorpusCrawler 2.0.12 (http://corpora.fi.muni.cz/crawler/)
	Mozilla/5.0 (compatible; spbot/4.3.0; +http://OpenLinkProfiler.org/bot )
	Mozilla/5.0 (compatible; aiHitBot/2.7; +http://www.aihit.com/)
	Mozilla/5.0 (compatible; SiteAnalyzerBot/4.0; +https://www.site-analyzer.com/)
	DataparkSearch/4.40.1 (+http://www.dataparksearch.org/)
	yacybot (freeworld-global; amd64 Linux 3.0.0-21-generic; java 1.6.0_26; America/en) http://yacy.net/bot.html
	findlinks/1.1.6-beta1 (+http://wortschatz.uni-leipzig.de/findlinks/; YaCy 0.1; yacy.net)
	CopperEgg/RevealUptime/SydneyAU(aws)
	OpenAcoon v4.1.0 (www.openacoon.de)
	semanticbot ([email protected])
	Mozilla/5.0 (compatible; Crawlera/1.10.2; UID 75425)
	Mozilla/5.0 (compatible; Crawlera/1.10.2; UID 11912)
	Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp)
	Mozilla/5.0 (Windows NT 5.1; U; Win64; fr; rv:1.8.1) VoilaBot BETA 1.2 ([email protected])
	yacybot (-global; amd64 Linux 3.13.0-61-generic; java 1.7.0_79; Europe/en) http://yacy.net/bot.html
	CorpusCrawler 2.0.25 (http://corpora.fi.muni.cz/crawler/);Project:CzCorpus
	ichiro/3.0 (http://search.goo.ne.jp/option/use/sub4/sub4-1/)
	R6_FeedFetcher_(www.radian6.com/crawler)
	sosoimagespider+(+http://help.soso.com/soso-image-spider.htm)
	Mozilla/5.0 (X11; Linux x86_64; rv:39.0; GTmetrix http://gtmetrix.com/) Gecko/20100101 Firefox/39.0
	yacybot (freeworld/global; amd64 Linux 3.2.0-24-generic; java 1.6.0_24; Europe/en) http://yacy.net/bot.html
	Mozilla/4.0 (compatible; Vagabondo/4.0; webcrawler at wise-guys dot nl; http://webagent.wise-guys.nl/)
	CommaFeed/2.4.0-SNAPSHOT (https://github.com/Athou/commafeed)
	Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; Google Web Preview Analytics) Chrome/27.0.1453 Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
	yacybot (/global; x86 Windows 7 6.1; java 1.8.0_31; Europe/de) http://yacy.net/bot.html
	Mozilla/5.0 (compatible; woriobot support [at] zite [dot] com +http://zite.com)
	Mozilla/5.0 (compatible; MetaJobBot; http://www.metajob.at/crawler)
	bl.uk_lddc_bot/3.3.0-SNAPSHOT-20140519-0844 (+http://www.bl.uk/aboutus/legaldeposit/websites/websites/faqswebmaster/index.html)
	FFbot/1.0; +http://feedfury.com
	Mozzila/5.0 (compatible; YamanaLab-Robot/1.0; http://www.yama.info.waseda.ac.jp/~crawler/info.html)
	LinqiaRSSBot/1.0 ([email protected])
	FreeWebMonitoring SiteChecker/0.2 (+http://www.freewebmonitoring.com/bot.html)
	Mozilla/5.0 (compatible; Web-Monitoring/1.0; +http://monoid.nic.ru/)
	Mozilla/5.0 (compatible; heritrix/1.12.1b +http://netarkivet.dk/website/info.html)
	Mozilla/5.0 (compatible; SpiderLing (a SPIDER for LINGustic research); http://nlp.fi.muni.cz/projects/biwec/)
	UrlTrends Ranking Updater/2.0 ( http://www.urltrends.com/ )
	yacybot (/global; amd64 Linux 3.16.1-gentoo; java 1.7.0_55; Europe/en) http://yacy.net/bot.html
	Mozilla/5.0 (compatible; gimme60bot/1.0 ; +http://gimme60.com)
	CatchBot/5.0; +http://www.catchbot.com
	elefent/Elefent 1.1 (Friendly web walker.; http://elefent.eu/; [email protected])
	findlinks/2.0.1 (+http://wortschatz.uni-leipzig.de/findlinks/)
	Googlebot-Video/1.0
	Mozilla/5.0 (compatible; XmarksFetch/1.0; +http://www.xmarks.com/about/crawler; [email protected])
	XRL/3.00 (Linux; i686; en-us) (+http://metamark.net/about)
	Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/50.0.2661.87 Safari/537.36 PTST/276
	CorpusCrawler 2.0.8 (http://corpora.fi.muni.cz/crawler/)
	Mozilla/5.0 (compatible; archive.org_bot +http://www.archive.org/details/archive.org_bot)
	Mozilla/5.0 (compatible; OpenindexDeepSpider/Nutch-1.5-dev; +http://openindex.io/spider.html; systemsATopenindexDOTio)
	Ronzoobot/1.3 (http://www.ronzoo.com/about.php)
	MetaInspector/2.2.1 (+https://github.com/jaimeiniesta/metainspector)
	Mozilla/5.0 (compatible; WASALive-Bot ; http://blog.wasalive.com/wasalive-bots/)
	Mozilla/5.0 (compatible; YioopBot; +http://www.yioop.com/bot.php)
	Mozilla/5.0 (compatible; Mozilla/5.0; +http://wiki.github.com/bixo/bixo/bixocrawler; [email protected])
	Mozilla/5.0 (compatible; Kraken/0.1; http://linkfluence.net/; [email protected])
	Mozilla/5.0 (compatible; Qseero; +http://www.q0.com)
	yacybot (-global; amd64 Linux 3.10.0-229.4.2.el7.x86_64; java 1.8.0_45; Europe/en) http://yacy.net/bot.html
	Mozilla/5.0 (compatible; ProCogBot/1.0; +http://www.procog.com/spider.html)
	Mozilla/5.0 (compatible; MJ12bot/v1.4.3; http://www.majestic12.co.uk/bot.php?+)
	Mozilla/5.0 (Anturis Agent)
	Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/534.34 (KHTML, like Gecko) Chromium/31.0.1650.63 Site-Shot/2.1 (http://www.site-shot.com/) Safari/534.34
	webnumbrFetcher/1.0 (http://webnumbr.com/)
	Mozilla/5.0 (compatible; meanpathbot/1.0; +http://www.meanpath.com/meanpathbot.html)
	NIF/1.1 (http://www.newsisfree.com/robot.php users:)
	Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/51.0.2704.103 Safari/537.36 PTST/284
	DoCoMo/2.0 P901i(c100;TB;W24H11) (compatible; ichiro/mobile goo; +http://help.goo.ne.jp/door/crawler.html)
	Iframely/0.8.8 (+https://iframely.com/;)
	yacybot (ppc Mac OS X 10.5.2; java 1.5.0_13; Europe/de) http://yacy.net/bot.html
	Mozilla/5.0 (compatible; UptimeRobot/2.0; http://www.uptimerobot.com/)
	yacybot (/global; amd64 Linux 3.13.0-85-generic; java 1.7.0_95; Europe/en) http://yacy.net/bot.html
	Mozilla/5.0 (compatible; FeedBooster; +http://feeds.qsensei.com)
	yacybot (freeworld/global; amd64 Linux 2.6.32-5-amd64; java 1.6.0_18; Europe/de) http://yacy.net/bot.html
	Mozilla/5.0 (compatible; SputnikFaviconBot/1.2; +http://corp.sputnik.ru/webmaster)
	Mozilla/5.0 (compatible; heritrix/3.1.1; UniLeipzigASV +http://corpora.informatik.uni-leipzig.de/crawler_faq.html)
	Mozilla/5.0 (compatible; OrangeBot/2.0; [email protected])
	Yeti-FeedItemCrawler/1.0 (NHN Corp.; http://help.naver.com/robots/)
	Mozilla/5.0 (compatible; SeznamBot/3.2-test1-1; +http://napoveda.seznam.cz/en/seznambot-intro/)
	Mozilla/5.0 (compatible; heritrix/3.2.0 +http://www.crim.ca)
	yacybot (/global; amd64 Linux 4.0.5-gentoo; java 1.8.0_51; Europe/en) http://yacy.net/bot.html
	pr-cy.ru Screenshot Bot
	GentleSource Short URL Checker ( http://www.gentlesource.com/ )
	crazywebcrawler 0.9.0, http://www.crazywebcrawler.com
	a.pr-cy.ru
	Mozilla/5.0 (iPhone; CPU iPhone OS 7_0 like Mac OS X) AppleWebKit/537.51.1 (KHTML, like Gecko) Version/7.0 Mobile/11A465 Safari/9537.53 (compatible; bingbot/2.0; http://www.bing.com/bingbot.htm)
	findlinks/1.1.6-beta1 (+http://wortschatz.uni-leipzig.de/findlinks/)
	ichiro/5.0 (http://help.goo.ne.jp/door/crawler.html)
	yacybot (/global; amd64 Windows 7 6.1; java 1.8.0_25; Europe/en) http://yacy.net/bot.html
	Yeti/1.1 (Naver Corp.; http://help.naver.com/robots/)
	Technoratibot/8.0
	Setooz/Nutch-1.0 (http://www.setooz.com)
	Mozilla/5.0 (Windows NT 5.2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/49.0.2623.112 Safari/537.36 PTST/201
	semanticbot
	Mozilla/5.0 (compatible; JikeSpider; +http://shoulu.jike.com/spider.html)
	Mozilla/5.0 (compatible; MetamojiCrawler/1.0; +http://www.metamoji.com/jp/crawler.html
	Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1.6) Gecko/20070725 Firefox/2.0.0.6 - James BOT - WebCrawler http://cognitiveseo.com/bot.html
	Mozilla/5.0 (Windows NT 6.1; compatible; BDCbot/1.0; +http://ecommerce.bigdatacorp.com.br/faq.aspx) ppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2272.118 Safari/537.36
	Mozilla/5.0 (compatible; CloudServerMarketSpider/1.0; +http://www.cloudservermarket.com/spider.html)
	Mozilla/5.0 (Windows NT 6.1; WOW64; rv:38.0; Neustar WPM) Gecko/20100101 Firefox/38.0
	Mozilla/5.0 (compatible;+ParchBot/1.0;++http://www.parchmenthill.com/search.htm)
	Sogou-Test-Spider/4.0 (compatible; MSIE 5.5; Windows 98)
	MetaHeadersBot (+http://www.metaheaders.com/bot.html)
	Mozilla/5.0 (Windows; U; Windows NT 6.0; en-GB; rv:1.0; trendictionbot0.4.2; trendiction media ssppiiddeerr; http://www.trendiction.com/bot/; please let us know of any problems; ssppiiddeerr at trendiction.com) Gecko/20071127 Firefox/2.0.0.11
	voyager/2.0 (http://www.kosmix.com/crawler.html)
	Mozilla/5.0 (compatible; YodaoBot/1.0; http://www.yodao.com/help/webmaster/spider/; )
	hivaBot/hivaBot-1.0 (Iranian Search Engine bot; http://yooz.ir; [email protected])
	yacybot (freeworld/global; amd64 Linux 3.2.0-4-amd64; java 1.7.0_55; Europe/de) http://yacy.net/bot.html
	AppEngine-Google; (+http://code.google.com/appengine; appid: s~closure-compiler-hrd)
	larbin_2.5.0 ([email protected])
	Scrapy/0.22.2 (+http://scrapy.org)
	Ronzoobot/1.2 (http://www.ronzoo.com/about.php)
	gooblog/2.0 (http://help.goo.ne.jp/contact/)
	Pingdom.com_bot_version_1.4_(http://www.pingdom.com/)
	Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/535.21 (KHTML, like Gecko) Chrome/19.0.1042.0 Safari/535.21 PagePeeker/2.1; +http://pagepeeker.com/robots/
	WeSEE:Ads/PageBot (http://www.wesee.com/bot/)
	yacybot (amd64 Linux 2.6.18-164.el5; java 1.6.0; Europe/de) http://yacy.net/bot.html
	SemanticScholarBot/1.0 (+http://s2.allenai.org/bot.html)
	Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/535. Safari/535.22+; (+http://immediatenet.com/thumbnail_api.html; free thumbnails)
	Mozilla/5.0 (compatible; Crawlera/1.10.2; UID 85747)
	SBIder/Nutch-1.0-dev (http://www.sitesell.com/sbider.html)
	Scrapy/1.0.4 (+http://scrapy.org)
	SanszBot/1.7(http://www.sansz.org/sanszbot, [email protected]) ([email protected])
	Mozilla/5.0 (compatible; KHTML, like Gecko) Chrome/43.0.2357.81 Safari/537.36 [email protected]
	Mozilla/5.0 (iPhone; U; CPU iPhone OS 4_3 like Mac OS X; en-GB) adbeat.com/policy AppleWebKit/533.17.9 (KHTML, like Gecko) Version/5.0.2 Mobile/8F190 Safari/6533.18.5
	Curious George - www.analyticsseo.com/crawler
	Grammarly/1.0 (http://www.grammarly.com)
	OpenCalaisSemanticProxy
	yacybot (freeworld/global; amd64 Linux 2.6.32-49-server; java 1.6.0_27; Europe/en) http://yacy.net/bot.html
	netEstate RSS crawler (+http://www.rss-directory.info/)
	ipv6-test.com validator
	Mozilla/5.0 (compatible; Qualidator.com Bot 1.0;)
	ichiro/2.0 ([email protected])
	yacybot (freeworld/global; amd64 Linux 2.6.38-14-generic; java 1.6.0_22; Europe/en) http://yacy.net/bot.html
	cg-eye live
	Mozilla/5.0 (compatible; OpenindexSpider; +http://www.openindex.io/en/webmasters/spider.html)
	Mozilla/5.0 (Windows; U; MSIE 9.0; Windows NT 9.0; en-US) AppEngine-Google; (+http://code.google.com/appengine; appid: s~virustotalcloud)
	Mozilla/5.0 (compatible; heritrix/1.14.3.r6601 +http://www.buddybuzz.net/yptrino)
	MetaGeneratorCrawler/1.3.4 (www.metagenerator.info)
	Mozilla/5.0 (Linux; Android 5.0; LG-D855 Build/LRX21R.A1422018487) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/40.0.2214.109 Mobile Safari/537.36 DareBoost
	Mozilla/5.0 (compatible; Nigma.ru/3.0; [email protected])
	Mozilla/5.0 (compatible; Wappalyzer; +https://github.com/AliasIO/Wappalyzer)
	yacybot (/global; x86 Windows XP 5.1; java 1.7.0_71; Europe/de) http://yacy.net/bot.html
	Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/50.0.2661.102 Safari/537.36 PTST/277
	OpenWebSpider v0.1.4 (http://www.openwebspider.org/)
	HubSpot Connect 1.0 (http://dev.hubspot.com/)
	Page Valet/4.1pre5
	msnbot/2.0b (+http://search.msn.com/msnbot.htm)
	amibot - http://www.amidalla.de - [email protected] libwww-perl/5.831
	SETOOZBOT/5.0 ( compatible; SETOOZBOT/0.30 ; http://www.setooz.com/bot.html )
	Mozilla/5.0 (compatible; YoudaoBot/1.0; http://www.youdao.com/help/webmaster/spider/; )
	Mozilla/5.0 (compatible; MSIE or Firefox mutant; not on Windows server;) Daumoa 4.0
	Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.0) LinkCheck by Siteimprove.com
	EdisterBot (http://www.edister.com/bot.html)
	Mozilla/5.0 (iPhone; CPU iPhone OS 6_0 like Mac OS X) AppleWebKit/536.26 (KHTML, like Gecko) Version/6.0 Mobile/10A5376e Safari/8536.25 (compatible; SMTBot/1.0; +http://www.similartech.com/smtbot)
	Tarantula/BETA-0.5 (Tarantula de la Enlle, el mejor buscador en la red; http://www.enlle.com; [email protected])
	Mozilla/5.0 (Windows; U; Windows NT 5.1; fr; rv:1.8.1) VoilaBot BETA 1.2 ([email protected])
	yacybot (freeworld/global; amd64 Linux 2.6.38-8-generic; java 1.6.0_22; Europe/en) http://yacy.net/bot.html
	Mozilla/5.0 (compatible; SemrushBot/0.95; +http://www.semrush.com/bot.html)
	CatchBot/3.0; +http://www.catchbot.com
	Mozilla/5.0 (compatible; uMBot-LN/1.0; mailto: [email protected])
	Mozilla/5.0 (compatible; RukiCrawler/1.0; +http://ruki.rezko.net)
	adbeat_bot
	Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/27.0.1453.116 Safari/537.36 HubSpot Webcrawler
	Mozilla/5.0 (compatible; YandexCatalog/3.0; +http://yandex.com/bots)
	NutchCVS/0.7.1 (Nutch; http://lucene.apache.org/nutch/bot.html; [email protected])
	Statastico/4.0
	facebookexternalhit/1.1 (+http://www.facebook.com/externalhit_uatext.php)
	Mozilla/5.0 (X11; U; Linux x86_64; en-US) AppleWebKit/534.16 (KHTML, like Gecko, Google Keyword Suggestion) Chrome/10.0.648.127 Safari/534.16
	Mozilla/5.0 (iPhone; CPU iPhone OS 7_0 like Mac OS X) AppleWebKit/537.51.1 (KHTML, like Gecko) Version/7.0 Mobile/11A465 Safari/9537.53 BingPreview/1.0b
	EuripBot/2.0 (+http://www.eurip.com) GetRobots
	Mozilla/5.0 (compatible; MJ12bot/v1.2.1; http://www.majestic12.co.uk/bot.php?+)
	Braintree-Webhooks
	Nutch Checksem/Nutch-1.10
	Mozilla/5.0 (compatible; HyperCrawl/0.2; +http://www.seograph.net/bot.html)
	quickobot/quickobot-1 (Quicko Labs; http://quicko.co; robot at quicko dot co)
	Whibse
	Yahoo! Site Explorer Feed Validator http://help.yahoo.com/l/us/yahoo/search/siteexplorer/manage/
	Mozilla/5.0 (compatible; AhrefsBot/5.0; +http://ahrefs.com/robot/)
	Siteshooter B0t 2.1
	GingerCrawler/1.0 (Language Assistant for Dyslexics; www.gingersoftware.com/crawler_agent.htm; support at ginger software dot com)
	Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko; Google Page Speed Insights) Chrome/27.0.1453 Safari/537.36
	eBot / v.0.1e (http://alfa.elchron.cz)
	Mozilla/5.0 (compatible; memoryBot/1.20.247 +http://internetmemory.org/en/)
	yacybot (x86 Windows 2003 5.2; java 1.6.0_20; America/de) http://yacy.net/bot.html
	YandexSomething/1.0
	Scoop
	Mozilla/5.0 (compatible; MegaIndex.ru/2.0; +http://megaindex.com/crawler)
	Mozilla/5.0 (compatible; SputnikBot/2.3; +http://corp.sputnik.ru/webmaster)
	TurnitinBot/2.1 (http://www.turnitin.com/robot/crawlerinfo.html)
	Mozilla/5.0 (compatible; Ask Jeeves/Teoma; +http://about.ask.com/en/docs/about/webmasters.shtml)
	Mozilla/5.0 (compatible; CrawlForMe/3.0.2; [email protected])
	Seologies/0.1 Bot
	www.deadlinkchecker.com XMLHTTP/1.0 Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/46.0.2490.86 Safari/537.36
	dubaiindex (addressendeutschland.de)
	yacybot (/global; x86 Windows 7 6.1; java 1.8.0_40; America/es) http://yacy.net/bot.html
	Mozilla/5.0+(compatible;+PiplBot;++http://www.pipl.com/bot/)
	LinkStats Bot
	Mozilla/5.0 (compatible; adbeat-publisher-description-fetcher; [email protected]; [email protected])
	CISPA Vulnerability Notification (https://notify.mmci.uni-saarland.de)
	librabot/1.0 (+http://search.msn.com/msnbot.htm)
	SeoCheckBot (Seomira Seo Check, http://www.kfsw.de/bot.html)
	yacybot (x86 Windows 2003 5.2; java 1.6.0_16; Europe/de) http://yacy.net/bot.html
	WatchMouse/8.4.0.3 (http://watchmouse.com/ ; itpda01.watchmouse.net)
	yacybot (/global; amd64 Linux 3.13.0-83-generic; java 1.7.0_95; Europe/en) http://yacy.net/bot.html
	BrainbruBot/1.0 (+http://www.brainbru.com/)
	Docoloc
	Mozilla/5.0 (Unknown; Linux x86_64) AppleWebKit/538.1 (KHTML, like Gecko) PhantomJS/2.1.1 Safari/538.1 Prerender (+https://github.com/prerender/prerender)
	dlvr.it/1.0 (+http://dlvr.it/)
	Mozilla/5.0 (compatible; parsijoo-bot; +http://www.parsijoo.ir/; [email protected])
	DataparkSearch/4.35-02122005 ( http://www.dataparksearch.org/)
	Pixray-Seeker/2.0 (Pixray-Seeker; +http://www.pixray.com/pixraybot; [email protected])
	Abrave v5.5 (http://robot.abrave.co.uk)
	Mozilla/5.0 (X11; U; Linux i686 (x86_64); en-US; rv:1.8.1.20) Gecko/20090429 HeartRails_Capture/0.6 (+http://capture.heartrails.com/) BonEcho/2.0.0.20
	yacybot (/global; amd64 Linux 3.19.0-26-generic; java 1.7.0_95; Europe/en) http://yacy.net/bot.html
	MLBot (www.metadatalabs.com/mlbot)
	yacybot (freeworld/global; amd64 Linux 2.6.32-34-server; java 1.6.0_26; Europe/de) http://yacy.net/bot.html
	Mozilla/5.0 (compatible; LinkpadBot/1.08; +http://www.linkpad.ru)
	Mozilla/5.0 (compatible; SearchmetricsBot; http://www.searchmetrics.com/en/searchmetrics-bot/)
	msnbot-NewsBlogs/2.0b (+http://search.msn.com/msnbot.htm)
	ZnajdzFoto/Image 2.0
	Mozilla/5.0 (compatible; Magibot/4.0.0; +http://magi.peak-labs.com/robots.txt)
	Mozilla/5.0 (compatible; spbot/4.0.8; +http://OpenLinkProfiler.org/bot )
	Eurobot/1.2 (http://eurobot.ayell.eu)
	Mozilla/5.0 (compatible; Embedly/0.2; snap; +http://support.embed.ly/)
	Yaanb/1.5.001 (compatible; Win64;+http://www.yaanb.com/company/bot.hmtl)
	yacybot (i386 Linux 2.6.31-16-generic; java 1.6.0_15; Europe/en) http://yacy.net/bot.html
	Mozilla/5.0 (compatible; YioopBot; +http://173.13.143.74/bot.php)
	CopperEgg/RevealUptime/Singapore(aws)
	Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/534.24 (KHTML, like Gecko; Google Web Preview) Chrome/11.0.696 Safari/534.24
	yacybot (freeworld/global; amd64 Windows 8 6.2; java 1.7.0_25; Europe/de) http://yacy.net/bot.html
	Server Density Service Monitoring v2
	Testomatobot/1.0 (Linux x86_64; +http://www.testomato.com/testomatobot) minicrawler/3.0~beta
	TinEye/1.0; +http://www.tineye.com/
	Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.9.0.19; aggregator:Spinn3r (Spinn3r 3.1); http://spinn3r.com/robot) Gecko/2010040121 Firefox/3.0.19
	GigablastOpenSource/1.0
	Googlebot (gocrawl v0.4)
	CSmartCrawler (email: abuse[at]adsadv[dot]com)
	Testomatobot/1.0 (Linux x86_64; +http://www.testomato.com/testomatobot) minicrawler/4.0.0~beta9
	KOCMOHABT (+http://kozmonavt.tk/) AppEngine-Google; (+http://code.google.com/appengine; appid: s~web-explorer)
	Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.2.18) Gecko/20110622 HeartRails_Capture/1.0.5 (+http://capture.heartrails.com/) Namoroka/3.6.18
	PingSpot/1.0 (http://www.pingspot.org/)
	AppEngine-Google; (+http://code.google.com/appengine; appid: s~buzzstream-tools-hrd)
	AddThis.com (http://support.addthis.com/)
	Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko; Google Web Preview) Chrome/27.0.1453 Safari/537.36
	ExactSeekCrawler/1.0
	Yeti/1.0 (NHN Corp.; http://help.naver.com/robots/)
	Mozilla/5.0 (compatible; jpg-newsbot/2.0; +http://vipnytt.no/bot.html)
	g2reader-bot/1.0 (+http://www.g2reader.com/)
	MetaGeneratorCrawler/1.3.12 (www.metagenerator.info)
	QuerySeekerSpider ( http://queryseeker.com/bot.html )
	KDDI-CA31 UP.Browser/6.2.0.7.3.129 (GUI) MMP/2.0 (compatible; ichiro/mobile goo; +http://help.goo.ne.jp/help/article/1142/)
	Mozilla/5.0 (compatible; GrapeshotCrawler/2.0; +http://www.grapeshot.co.uk/crawler.php)
	Mozilla/5.0 (compatible; MSIE 10.0; Windows NT 6.2; Trident/6.0; WOW64; Trident/6.0; BingPreview/1.0b)
	Mozilla/5.0 (compatible; firmilybot/0.3; +http://www.firmily.com/bot.php
	Mozilla/5.0 (X11; U; Linux Core i7-4980HQ; de; rv:32.0; compatible; JobboerseBot; http://www.jobboerse.com/bot.htm) Gecko/20100101 Firefox/38.0
	Mozilla/5.0 (compatible; metager2-verification-bot; +http://metager2.de/technology.php)
	crawler4j (http://code.google.com/p/crawler4j/)
	CirrusExplorer/1.1 (http://www.cireu.com/explorer.php)
	Mozilla/5.0 (compatible; RankvalBot/1.0; +https://rankval.com)
	Mozilla/5.0 (compatible; heritrix/1.14.3 +http://www.accelobot.com)
	Mozilla/5.0 (compatible; BLEXBotTest/1.0; +http://webmeup.com/crawler.html)
	nWormFeedFinder (http://www.nworm.com)
	netEstate NE Crawler (+http://www.sengine.info/)
	Mozila/5.0 (compatible; StudioFACA Search; +http://www.studiofaca.com/)
	Mozilla/5.0 (compatible; Crawlera/1.10.2; UID 92751)
	wikiwix-bot-3.0
	SEOCentro Page Keyword Analyzer v1.2
	yacybot (amd64 Linux 2.6.18-164.15.1.el5xen; java 1.6.0_0; Europe/en) http://yacy.net/bot.html
	Mozilla/5.0 (compatible; bingbot/2.0; http://www.bing.com/bingbot.htm)
	Automattic Analytics Crawler/0.1; http://wordpress.com/crawler/
	SemrushBot/0.92
	http://www.uni-koblenz.de/~flocke/robot-info.txt
	Mozilla/5.0 (compatible; idmarch Automatic.beta/1.3; +http://www.idmarch.org/bot.html)
	Mozilla/5.0 (X11; Linux i686 on x86_64; rv:25.0; GTmetrix http://gtmetrix.com/) Gecko/20100101 Firefox/25.0
	msnbot/2.0b (+http://search.msn.com/msnbot.htm)._
	Stroke.cz (http://stroke.cz)
	Mozilla (+http://web.nli.org.il/sites/NLI/English/collections/israel-collection/internet_archive/Pages/default.aspx)
	audisto.com full crawler 4.33.470 (refer to in robots.txt as audisto, see https://audisto.com/bot)
	http://arachnode.net 2.5
	SurveyBot/2.3 (Whois Source)
	ant.com/Nutch-1.7 (http://ant.com)
	CorpusCrawler 2.0.13 (http://corpora.fi.muni.cz/crawler/)
	Seo Servis - Analyza zdrojoveho kodu
	SEOENGBot/1.2 (+http://www.seoengine.com/seoengbot.htm)
	Mozilla/5.0 (compatible; Genieo/1.0 http://www.genieo.com/webfilter.html)
	Mozilla/5.0 (compatible; evc-batch/2.0.20160708184126)
	HubSpot Marketing Grader
	Mozilla/5.0 (compatible; MSIE 10.0; Windows NT 6.1; Trident/6.0) LinkCheck by Siteimprove.com
	nrsbot/6.0(loopip.com/robot.html)
	Mozilla/5.0 (compatible; BLEXBot/1.0; +http://webmeup.com/crawler.html)
	yacybot (freeworld-global; amd64 Windows 7 6.1; java 1.7.0_02-ea; America/en) http://yacy.net/bot.html
	Mozilla/5.0 (compatible; Blekkobot; ScoutJet; +http://blekko.com/about/blekkobot)
	Mozilla/5.0 (compatible; Topicbot/12.0rc-2; +http://topicbot.awardspace.us/)
	Mozilla/5.0 (compatible; Gimme60bot/1.0; +http://gimme60.com)
	yacybot (i386 Linux 2.6.28-11-generic; java 1.6.0_13; Europe/de) http://yacy.net/bot.html
	Mozilla/5.0 (compatible; DotBot/1.1; http://www.opensiteexplorer.org/dotbot, [email protected])
	Kemvibot/1.0 (http://kemvi.com, [email protected])
	CorpusCrawler 2.0.15 (http://corpora.fi.muni.cz/crawler/)
	dotbot
	Mozilla/5.0 (iPhone; CPU iPhone OS 7_0 like Mac OS X) AppleWebKit/537.51.1 (KHTML, like Gecko) Version/7.0 Mobile/11A465 Safari/9537.53 (compatible; adidxbot/2.0; +http://www.bing.com/bingbot.htm)
	page_viewer ([email protected])
	HootSuite Image proxy ([email protected])
	Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US) Speedy Spider for SpeedyAds (http://www.entireweb.com/about/search_tech/speedy_spider/)
	PINGOMETER_BOT_(HTTPS://PINGOMETER.COM)
	Mozilla/5.0 (compatible; Vagabondo/2.1; webcrawler at wise-guys dot nl; http://webagent.wise-guys.nl/)
	VSE/1.0 ([email protected])
	Scrapy/0.24.0 (+http://scrapy.org)
	Speedy Spider (http://www.entireweb.com)
	Mozilla/5.0 (X11; Linux x86_64; GTmetrix https://gtmetrix.com/) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/51.0.2704.106 Safari/537.36
	WatchMouse/18990 (http://watchmouse.com/ ; hk)
	Mozilla/5.0 (compatible; yoozBot-2.2; http://yooz.ir; [email protected])
	Mozilla/5.0 (compatible; Crawlera/1.10.2; UID 87214)
	Mozilla/5.0 (compatible; discobot/2.0; +http://discoveryengine.com/discobot.html)
	yacybot (freeworld/global; amd64 Windows 7 6.1; java 1.8.0_11; Europe/de) http://yacy.net/bot.html
	Testomatobot/1.0 (Linux x86_64; +http://www.testomato.com/testomatobot) minicrawler/4.0.0~beta6
	yacybot (freeworld/global; amd64 Linux 3.0.0-15-server; java 1.6.0_23; Europe/en) http://yacy.net/bot.html
	Mozilla/5.0 (Macintosh; Intel Mac OS X 10.9; rv:28.0) Gecko/20100101 Firefox/28.0 (FlipboardProxy/1.1; +http://flipboard.com/browserproxy)
	YisouSpider
	Slack-ImgProxy 1.138 (+https://api.slack.com/robots)
	mozilla/5.0 (compatible; webmastercoffee/0.7; +http://webmastercoffee.com/about)
	Mozilla/5.0 (compatible; Uptimebot/0.2.22; +http://www.uptime.com/uptimebot)
	Mozilla/5.0 (compatible; AhrefsBot/1.0; +http://ahrefs.com/robot/)
	Mozilla/5.0 (compatible; Feedspotbot/1.0; +http://www.feedspot.com/fs/bot)
	sg-Orbiter/1.0 (+http://searchgears.de/uber-uns/crawling-faq.html)
	Site24x7 Tools
	NextGenSearchBot 1 (for information visit http://www.zoominfo.com/About/misc/NextGenSearchBot.aspx)
	Aboundex/0.3 (http://www.aboundex.com/crawler/)
	yacybot (freeworld/global; i386 Linux 2.6.37.6-0.5-desktop; java 1.6.0_20; Europe/en) http://yacy.net/bot.html
	Mozilla/5.0 (Windows NT 6.2; WOW64) AppleWebKit/537.4 (KHTML, like Gecko) Chrome/98 Safari/537.4 (StatusCake)
	Mozilla/5.0 (compatible; DomainSONOCrawler/0.1; +http://domainsono.com)
	Mozilla/5.0 (compatible; Online Domain Tools - Server Monitor/1.0; +http://server-monitoring.online-domain-tools.com)
	Mozilla/5.0 (compatible; Konqueror/3.5; Linux) KHTML/3.5.5 (like Gecko) (Exabot-Thumbnails)
	Mozilla/5.0 (compatible; spbot/4.0.4; +http://www.seoprofiler.com/bot )
	Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/27.0.1453 Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
	Factbot 1.09 (see http://www.factbites.com/webmasters.php)
	CopperEgg/RevealUptime/SaoPauloBR(aws)
	CRIM Crawler/Nutch-2.3 (Crawler du Centre de Recherche Informatique de Montr\xc3\xa9al (CRIM))
	Mozilla/5.0 (compatible; heritrix/1.14.3 +http://www.webarchiv.cz)
	Mozilla/5.0 (compatible; WormlyBot; +http://wormly.com)
	yacybot (i386 Linux 2.6.23; java 1.6.0_17; Europe/en) http://yacy.net/bot.html
	Mozilla/5.0 (compatible; Cliqzbot/0.1 +http://cliqz.com/company/cliqzbot)
	URLitor.com
	RavenCrawler
	Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.2.18) Gecko/20130331 HeartRails_Capture/1.0.5 (+http://capture.heartrails.com/) Namoroka/3.6.18
	Mozilla/5.0 (compatible; WebThumbnail/3.x; Website Thumbnail Generator; +http://webthumbnail.org)
	Kyoto-Crawler/2.0 (Mozilla-compatible; kyoto-crawler-contact(at)nlp(dot)kuee(dot)kyoto-u(dot)ac(dot)jp; http://nlp.ist.i.kyoto-u.ac.jp/)
	Mozilla/5.0 (compatible; SputnikImageBot/2.3; +http://corp.sputnik.ru/webmaster)
	WatchMouse/8.4.0.3 (http://watchmouse.com/ ; lvps91-250-96-109.dedicated.hosteurope.de)
	MetaGeneratorCrawler/1.3.3 (www.metagenerator.info)
	LongURL API
	Mozilla/5.0 ( compatible; SETOOZBOT/0.30 ; http://www.setooz.com/bot.html ; agentname at setooz dot_com )
	Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.6; en-US; rv:1.9.2) Gecko/20100115 Firefox/3.6 (FlipboardProxy/2.0; +http://flipboard.com/browserproxy)
	http://tools.geek-tools.org/link-counter/
	SCFCrawler/Nutch-1.8 (Image Crawler for StolenCameraFinder.com; http://www.stolencamerafinder.com/; [email protected])
	Mozilla/5.0 (compatible; Uptimebot/0.2.41; +http://www.uptime.com/uptimebot)
	Mozilla/5.0 (compatible; DomainAppender /1.0; +http://www.profound.net/domainappender)
	Mozilla/5.0 (compatible; hypestat/1.0; +http://www.hypestat.com/bot)
	yacybot (/global; amd64 Linux 3.13.0-042stab093.4; java 1.7.0_79; Europe/en) http://yacy.net/bot.html
	PagesInventory (robot +http://www.pagesinventory.com)
	Sosospider+(+http://help.soso.com/webspider.htm)
	Mozilla/5.0 (compatible; AntBot/1.0; +http://www.ant.com/)
	PagesInventory (robot http://www.pagesinvenotry.com)
	Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; http://www.changedetection.com/bot.html )
	Wotbox/2.01 (+http://www.wotbox.com/bot/)
	Mozilla/5.0 (compatible; bixolabs/1.0; +http://bixolabs.com/crawler/general; [email protected])
	Mozilla/5.0 (compatible; SpiderLing (a SPIDER for LINGustic research); +http://nlp.fi.muni.cz/projects/biwec/)
	CopperEgg/RevealUptime/DallasTXUSA
	yacybot (freeworld/global; amd64 Linux 2.6.32-custom; java 1.6.0_26; Europe/en) http://yacy.net/bot.html
	Bad-Neighborhood Link Analyzer (http://www.bad-neighborhood.com/)
	Mozilla/5.0 (compatible; oBot/2.3.1; +http://filterdb.iss.net/crawler/)
	Pixray-Seeker/2.0 (Pixray-Seeker; http://www.pixray.com/pixraybot; [email protected])
	Cityreview Robot (+http://www.cityreview.org/crawler/)
	yacybot (/global; amd64 Linux 3.16.0-4-amd64; java 1.7.0_95; Europe/en) http://yacy.net/bot.html
	Dlvr.it/1.0 (http://dlvr.it/)
	Zookabot/2.4;++http://zookabot.com
	AddThis.com robot [email protected]
	Mozilla/5.0 (compatible; discoverybot/2.0; +http://discoveryengine.com/discoverybot.html)
	TwengaBot
	Mozilla/5.0 (compatible; 4SeoHuntBot; +http://4seohunt.biz/about.html)
	Web-sniffer.me/1.0.0 (+http://web-sniffer.me/)
	KDDI-CA31 UP.Browser/6.2.0.7.3.129 (GUI) MMP/2.0 (compatible; ichiro/mobile goo; +http://search.goo.ne.jp/option/use/sub4/sub4-1/)
	gosquared-thumbnailer/1.0
	yacybot (/global; amd64 Windows 7 6.1; java 1.7.0_55; Asia/zh) http://yacy.net/bot.html
	BPImageWalker/2.0 (www.bdbrandprotect.com)
	Acoon v4.10.3 (www.acoon.de)
	Mozilla/4.0 (CMS Crawler: http://www.cmscrawler.com)
	WP Engine Install Performance API
	Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.2.24; ips-agent) Gecko/20111107 Ubuntu/10.04 (lucid) Firefox/3.6.24
	Mozilla/5.0 (Windows NT 6.1; WOW64; Trident/7.0; rv:11.0) like Gecko PTST/277
	CamontSpider/1.0 +http://epweb2.ph.bham.ac.uk/user/slater/camont/info.html
	SEOENGBot/1.2 (+http://learn.seoengine.com/seoengbot.htm)
	Y!J-BRJ/YATS crawler (http://listing.yahoo.co.jp/support/faq/int/other/other_001.html)
	CopperEgg/RevealUptime/N.Virginia(aws)
	Mozilla/5.0, Google-AdWords-Express
	Mozilla/5.0 (compatible;YodaoBot-Image/1.0;http://www.youdao.com/help/webmaster/spider/;)
	yacybot (/global; amd64 Linux 3.16.0-4-amd64; java 1.8.0_91; Europe/en) http://yacy.net/bot.html
	yacybot (freeworld/global; amd64 Windows 7 6.1; java 1.7.0_09; Europe/de) http://yacy.net/bot.html
	Acoon v4.10.1 (www.acoon.de)
	Mediatoolkitbot ([email protected])
	Mozilla/5.0 (compatible; Speedy Spider; http://www.entireweb.com/about/search_tech/speedy_spider/)
	Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.2.3) SPEng
	2Bone_LinkChkr/1.0 libwww-perl/6.03
	Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.6; en-US; rv:1.9.2) Gecko/20100115 Firefox/3.6 (FlipboardBrowserProxy/0.0.5; +http://flipboard.com/browserproxy)
	Mozilla/5.0 (compatible; SeznamBot/3.2-test1; +http://napoveda.seznam.cz/en/seznambot-intro/)
	Comodo SSL Checker
	Comodo Spider 1.2
	Who.is Bot
	Mozilla/5.0 (compatible; online-webceo-bot/1.0; +http://online.webceo.com)
	yacybot (freeworld/global; amd64 Windows Server 2008 R2 6.1; java 1.6.0_29; Europe/en) http://yacy.net/bot.html
	psbot/0.1 (+http://www.picsearch.com/bot.html)
	IlTrovatore-Setaccio/1.2 (http://www.iltrovatore.it/aiuto/faq.html)
	Mozilla/5.0 (compatible; Twingly Recon; twingly.com)
	Mozilla/5.0 Moreover/5.1 (+http://www.moreover.com; [email protected])
	Orgbybot/OrgbyBot v1.2 (Spidering the net for Orgby; http://www.orgby.com/ ; Orgby.com Search Engine)
	Browsershots
	AdsBot-Google (+http://www.google.com/adsbot.html)
	Qseero v1.0.0
	Mozilla/5.0 (compatible; ltbot/0.3.6 +http://www.kdsl.tu-darmstadt.de/de/kdsl/research-program/crawling-and-semantic-structuring/)
	yacybot (x86 Windows XP 5.1; java 1.6.0_18; Europe/de) http://yacy.net/bot.html
	visionutils/0.2
	Mozilla/5.0 (compatible; Googlebot/2.1; https://www.deepcrawl.com/bot)
	Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.4 (KHTML, like Gecko; Google Web Preview) Chrome/22.0.1229 Safari/537.4
	Mozilla/5.0 (compatible; Crawlera/1.10.2; UID 8942)
	Mozilla/4.0 (compatible; LinkMarket-Bot v1.2)
	Buzzbot/1.0 (Buzzbot; http://www.buzzstream.com; [email protected])
	WordPress.com; https://public-api.wordpress.com
	Mozilla/5.0 ( compatible; SETOOZBOT/0.30 ; http://www.setooz.com/bot.html )
	CopperEgg/RevealUptime/FremontCAUSA
	Mozilla/5.0 (Windows; U; Windows NT 6.0; en-GB; rv:1.0; trendictionbot0.4.5; trendiction search; http://www.trendiction.de/bot; please let us know of any problems; web at trendiction.com) Gecko/20071127 Firefox/3.0.0.11
	yacybot (/global; amd64 Linux 2.6.32-042stab094.8; java 1.7.0_79; America/en) http://yacy.net/bot.html
	Mozilla/5.0 (compatible; Windows NT 6.1?; ZumBot/1.0; http://help.zum.com/inquiry)
	Mozilla/5.0 (Windows NT 6.1; WOW64; Trident/7.0; rv:11.0; topster.de Linkchecker 5.0) like Gecko (90.177.52.133)
	Mozilla/5.0+(compatible; Monitority/1.0; http://www.monitority.com/)
	yacybot (freeworld/global; x86 Windows 7 6.1; java 1.8.0_25; Europe/de) http://yacy.net/bot.html
	Mozilla/5.0 (compatible; abby/1.0; +http://www.ellerdale.com/crawler.html)
	Mozilla/5.0 (compatible; SemrushBot/1.1~bl; +http://www.semrush.com/bot.html)
	YowedoBot/Yowedo 1.0 (Search Engine crawler for yowedo.com; http://yowedo.com/en/partners.html; [email protected])
	Mozilla/5.0 (compatible; oBot/2.3.1; http://filterdb.iss.net/crawler/)
	Nokia6820/2.0 (4.83) Profile/MIDP-1.0 Configuration/CLDC-1.0 (compatible; Googlebot-Mobile/2.1; +http://www.google.com/bot.html)
	Mozilla/5.0 (compatible; IstellaBot/1.23.15 +http://www.tiscali.it/)
	GetIntent Crawler (http://getintent.com/bot.html)
	Mozilla/5.0 (compatible; Uptimebot/0.1.89; +http://www.uptime.com/uptimebot)
	Cliqzbot
	AboutUsBot
	yacybot (-global; amd64 Linux 3.2.0-4-amd64; java 1.7.0_67; Europe/en) http://yacy.net/bot.html
	Sufog/Nutch-2.2.1 (www.sufog.com; www.sufog.com)
	Mozilla/5.0 (compatible; imbot/0.1 +http://internetmemory.org/en/)
	KOCMOHABT (+http://kozmonavt.tk/) Mozilla/5.0 (compatible; Web Explorer 1) AppEngine-Google; (+http://code.google.com/appengine; appid: s~web-explorer)
	yacybot (/global; amd64 Linux 3.16.0-53-generic; java 1.7.0_85; Europe/en) http://yacy.net/bot.html
	L.webis/0.53 (http://webalgo.iit.cnr.it/index.php?pg=lwebis)
	Mozilla/5.0 (compatible; memoryBot/1.21.23 +http://internetmemory.org/en/)
	ExB Language Crawler 2.1.5 (+http://www.exb.de/crawler)
	Peeplo Screenshot Bot/0.20 ( abuse at peeplo dot_com )
	Mozilla/5.0 (compatible; Qwantify/2.1w; +https://www.qwant.com/)/*
	Experibot_v1 [bit.ly/2bLMJdw]
	WikioFeedBot 1.0 (http://www.wikio.com)
	SalesIntelligent/v1.1 (https://www.salesintelligent.com)
	Mozilla/5.0 (compatible; ExaleadCloudView/5;)
	Mozilla/5.0 (compatible; rogerBot/1.0; UrlCrawler; http://www.seomoz.org/dp/rogerbot)
	Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/44.0.2403.157 Safari/537.36 TinEye/1.0 (via http://www.tineye.com/)
	Mozilla/5.0 (compatible; Search17Bot/1.1; http://www.search17.com/bot.php)
	FollowSite Bot ( http://www.followsite.com/bot.html )
	Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/27.0.1453.116 Safari/537.36 HubSpot Webcrawler - [email protected]
	Mozilla/5.0 (compatible; BlinkaCrawler/1.0; +http://www.blinka.jp/crawler/)
	Mozilla/5.0 (compatible; Qwantify/2.2w; +https://www.qwant.com/)/*
	findlinks/1.1.6-beta2 (+http://wortschatz.uni-leipzig.de/findlinks/)
	yacybot (/global; amd64 Linux 4.4.0-28-generic; java 1.8.0_91; Europe/pl) http://yacy.net/bot.html
	yacybot (/global; amd64 Windows Server 2008 R2 6.1; java 1.8.0_40; Europe/de) http://yacy.net/bot.html
	Mozilla/5.0 (compatible; 200PleaseBot/1.0; +http://www.200please.com/bot)
	WatchMouse/8.4.0.3 (http://watchmouse.com/ ; chzrh01.watchmouse.net)
	Iframely/0.8.8 (+http://iframely.com/;)
	Mozilla/5.0 (compatible; URLAppendBot/1.0; +http://www.profound.net/urlappendbot.html)
	LoadImpactRload/3.2.0 (Load Impact; http://loadimpact.com);
	RSSingBot (http://www.rssing.com)
	RADaR-Bot/Nutch-1.3 (http://radar-bot.com/)
	Ocelli/1.4 (http://www.globalspec.com/Ocelli)
	Mozilla/5.0 (compatible; BuzzSumo; +http://www.buzzsumo.com/bot.html)
	Mozilla/4.0 (compatible; IDwhois/2.0alpha-10102619; Linux; en-US; +http://idwhois.info/about)
	omgilibot/0.4 +http://omgili.com
	yacybot (freeworld/global; amd64 Windows 7 6.1; java 1.6.0_23; Europe/de) http://yacy.net/bot.html
	Y!J-BRW/1.0 crawler (http://help.yahoo.co.jp/help/jp/search/indexing/indexing-15.html)
	Mozilla/5.0 (compatible; AhrefsBot/2.0; +http://ahrefs.com/robot/)
	Mozilla/5.0 (compatible; VideoSurf_bot +http://www.videosurf.com/bot.html)
	Symfony Spider (http://symfony.com/spider)
	Mozilla/5.0 (compatible; Sosospider/2.0; +http://help.soso.com/webspider.htm)
	Mozilla/5.0 (compatible; OpenindexDeepSpider/Nutch-1.5-dev; +http://www.openindex.io/en/webmasters/spider.html)
	ADmantX Platform Semantic Analyzer - ADform - ADmantX Inc. - www.admantx.com - [email protected]
	Mozilla/5.0 (compatible; MJ12bot/v1.3.3; http://www.majestic12.co.uk/bot.php?+)
	cuwhois/1.0 (+http://www.cuwhois.com/)
	Mozilla/5.0 (compatible; memoryBot/1.20.311 +http://internetmemory.org/en/)
	Mozilla/4.0 (compatible; Vagabondo/4.0; webcrawler at wise-guys dot nl; http://webagent.wise-guys.nl/; http://www.wise-guys.nl/)
	yacybot (-global; amd64 Linux 3.16.0-4-amd64; java 1.7.0_75; Europe/en) http://yacy.net/bot.html
	Mozilla/5.0 (compatible; Dazoobot/1.0; http://dazoo.fr)
	Mozilla/5.0 (compatible; Semager/1.4c; +http://www.semager.de/blog/semager-bots/)
	EuripBot/1.1 (+http://www.eurip.com) GetRobots
	yacybot (freeworld/global; amd64 Linux 3.8.0-23-generic; java 1.6.0_27; Pacific/en) http://yacy.net/bot.html
	yacybot (amd64 Linux 2.6.24-23-generic; java 1.6.0_07; Europe/de) http://yacy.net/bot.html
	yacybot (freeworld/global; x86 Windows 2003 5.2; java 1.6.0_32; Europe/de) http://yacy.net/bot.html
	yacybot (freeworld/global; amd64 Linux 3.2.0-4-amd64; java 1.7.0_25; Europe/de) http://yacy.net/bot.html
	socialbm_bot http://spider.socialbm.net
	Mozilla/5.0 (compatible; 007ac9 Crawler; http://crawler.007ac9.net/)
	webinatorbot 1.0; +http://www.webinator.de
	WorldBrewBot/2.1 (+http://www.marketbrew.com/)
	BotOnParade, http://www.bots-on-para.de/bot.html
	ltx71 - (http://ltx71.com/)
	VSB-TUO/1.0 (+http://www.vsb.cz)
	yacybot (/global; amd64 Linux 3.13.0-85-generic; java 1.7.0_101; Europe/en) http://yacy.net/bot.html
	CopperEgg/RevealUptime/TokyoJP(aws)
	Mozilla/5.0 (compatible; etoolsbot/1.0; +https://etools.io/bots/)
	Comodo-Webinspector-Crawler 2.2.2, http://www.comodorobot.com
	Mozilla/5.0 (compatible; TWMBot/0.1; +http://thewebminer.com)
	Mozilla/5.0 (compatible; Plukkie/1.3; http://www.botje.com/plukkie.htm)
	Autonomy
	yacybot (i386 Linux 2.6.30-2-686; java 1.6.0_0; SystemV/en) http://yacy.net/bot.html
	Testomatobot/1.0 (Linux x86_64; +http://www.testomato.com/testomatobot) minicrawler/4.1.0
	CatchBot/1.0; +http://www.catchbot.com
	Sitedomain-Bot(Sitedomain-Bot 1.0, http://www.sitedomain.de/sitedomain-bot/)
	Mozilla/5.0 (compatible; Uptimebot/0.2.42; +http://www.uptime.com/uptimebot)
	yacybot (/global; amd64 Linux 3.2.0-4-amd64; java 1.7.0_03; Europe/en) http://yacy.net/bot.html
	SiteTruth.com site rating system
	SiteUptime.com
	yacybot (i386 Linux 2.6.24-23-generic; java 1.6.0_16; Europe/en) http://yacy.net/bot.html
	UrlTrends Ranking Updater/2.0
	yacybot (/global; amd64 Windows 7 6.1; java 1.7.0_55; Europe/de) http://yacy.net/bot.html
	yacybot (freeworld/global; amd64 Linux 3.5.0-27-generic; java 1.7.0_25; Europe/en) http://yacy.net/bot.html
	GarlikCrawler/1.1 (http://garlik.com/, [email protected])
	Mozilla/5.0 (compatible; ParsijooBot; +http://www.parsijoo.ir/; [email protected])
	Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.1 (KHTML, like Gecko) Chrome/21.0.1180.89 Safari/537.1; 360Spider(compatible; HaosouSpider; http://www.haosou.com/help/help_3_2.html)
	Willow Internet Crawler by Twotrees V2.1
	KrOWLer/0.1.0, matentzn at cs dot man dot ac dot uk
	SimplyFast.info Headers
	yacybot (freeworld/global; amd64 Linux 2.6.32-431.29.2.el6.x86_64; java 1.7.0_65; Europe/en) http://yacy.net/bot.html
	Mozilla/5.0 (compatible; YandexBlogs/0.99; robot; +http://yandex.com/bots)
	msnbot-media/1.1 (+http://search.msn.com/msnbot.htm)
	Mozilla/5.0 (Windows NT 6.1; WOW64; rv:46.0) Gecko/20100101 Firefox/46.0 PTST/278
	Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/534.57.2 (KHTML, like Gecko) Version/5.1.7 Safari/534.57.2 PTST/278
	gipo-crawler/Nutch-1.10 (Global Internet Policy Observatory crawler)
	Mozilla/5.0 (compatible; sukibot_heritrix/3.1.1 +http://suki.ling.helsinki.fi/eng/webmasters.html)
	PayPal IPN ( https://www.paypal.com/ipn )
	Mozilla/5.0 (compatible; KaloogaBot; http://kalooga.com/crawler)
	eCairn-Grabber/1.0 (+http://ecairn.com/grabber) curl/7.15
	Mozilla/5.0 (Windows NT 6.2; WOW64) AppleWebKit/537.15 (KHTML like Gecko; +http://www.diffbot.com) Chrome/24.0.1295.0 Safari/537.15
	ImageEngine/1.0
	DER-bot (+https://goo.gl/9ZibHG)
	yacybot (freeworld/global; amd64 Windows 7 6.1; java 1.7.0_60; Europe/de) http://yacy.net/bot.html
	Mozilla/5.0 (compatible; SemrushBot/0.96.3; +http://www.semrush.com/bot.html)
	Mozilla/5.0 (compatible; Exabot/3.0; +http://www.exabot.com/go/robot)
	Mozilla/5.0 (compatible; YandexDirect/3.0; +http://yandex.com/bots)
	Mozilla/5.0 (compatible; aiHitBot/2.9; +https://www.aihitdata.com/about)
	Linguee Bot (http://www.linguee.com/bot; [email protected])
	MetaGeneratorCrawler/1.3.9 (www.metagenerator.info)
	TelegramBot (like TwitterBot)
	Mozilla/5.0 (compatible; Siteliner/1.0; +http://www.siteliner.com/bot)
	calculon spider/0.1.0 mailto:[email protected]
	yacybot (/global; amd64 Linux 3.13.0-83-generic; java 1.7.0_95; Europe/de) http://yacy.net/bot.html
	PocketParser/2.0 (+https://getpocket.com/pocketparser_ua)
	montastic-monitor http://www.montastic.com
	Mozilla/5.0 (compatible; trovitBot 1.0; +http://www.trovit.com/bot.html)
	Mozilla/5.0 (Windows NT 6.1)+(compatible; SMTBot/1.0; +http://www.similartech.com/smtbot)
	Yaanb/1.5.001 (compatible; Win64;)
	Mozilla/5.0 (compatible; Crawlera/1.10.2; UID 70054)
	Mozilla/5.0 (compatible; spbot/2.0.2; +http://www.seoprofiler.com/bot/ )
	yacybot (freeworld-global; amd64 Linux 3.2.0-4-amd64; java 1.6.0_24; Europe/en) http://yacy.net/bot.html
	Mozilla/5.0 (compatible; ImageFetcher/5.6; +http://images.weserv.nl/)
	CopperEgg/RevealUptime/N.VirginiaUSA
	Huaweisymantecspider (compatible; MSIE 8.0; [email protected])
	OrangeCrawler/Nutch-1.0 ([email protected])
	CopperEgg/RevealUptime/AmsterdamHolland
	Mozilla/5.0 (compatible; SemrushBot-SI/0.97; +http://www.semrush.com/bot.html)
	Mozilla/5.0 (X11; Linux x86_64; GTmetrix https://gtmetrix.com/) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/50.0.2661.75 Safari/537.36
	Mozilla/5.0 (compatible; Genieo/1.0 http://www.genieo.com/webfilter.html) AppEngine-Google; (+http://code.google.com/appengine; appid: s~natmacdevice)
	yacybot (/global; amd64 Linux 3.2.41-042stab094.7; java 1.7.0_75; Europe/en) http://yacy.net/bot.html
	yacybot (x86 Windows Vista 6.1; java 1.6.0_13; Europe/de) http://yacy.net/bot.html
	SmabblerBot/1.0 (http://www.smabbler.com/)
	Mozilla/5.0 (compatible; grapeFX/0.9; [email protected]
	Mozilla/5.0 (compatible; SemrushBot/1~bl; +http://www.semrush.com/bot.html)
	MetaGeneratorCrawler/1.3.14 (www.metagenerator.info)
	oku-taka-lab-bot (Mozilla-compatible; [email protected]; http://lr-www.pi.titech.ac.jp/wp/)
	Mozilla/5.0 (Linux; Android 4.1.2; Galaxy Nexus Build/JZO54K; GTmetrix https://gtmetrix.com/) AppleWebKit/537.22 (KHTML, like Gecko) Chrome/26.0.1410.58 Mobile Safari/537.22
	Mozilla/5.0 (compatible; Feedage/2.0; +http://www.feedage.com/bot.php)
	yacybot (amd64 Linux 2.6.26-2-openvz-amd64; java 1.6.0_12; UTC/en) http://yacy.net/bot.html
	yacybot (-global; amd64 Linux 2.6.32-042stab116.1; java 1.7.0_79; Europe/en) http://yacy.net/bot.html
	Mozilla/5.0 (compatible; DKIMRepBot/1.0; +http://www.dkim-reputation.org)
	Mozilla/5.0 (Windows; U; Windows NT 5.1; zh-CN; rv:1.8.0.11) Firefox/1.5.0.11; 360Spider
	Mozilla/5.0 (compatible; pub-crawler; +http://wiki.github.com/bixo/bixo/bixocrawler; [email protected])
	mfibot/1.1 (http://www.mfisoft.ru/analyst/; <[email protected]>; en-RU)
	Visited by http://tools.geek-tools.org
	peerindex
	Mozilla/5.0 (compatible; janforman.com/2.4; +http://janforman.com/)
	http://www.amagit.com/
	yacybot (freeworld/global; amd64 Linux 2.6.32-40-server; java 1.6.0_20; Europe/en) http://yacy.net/bot.html
	Mozilla/5.0 (iPhone; CPU iPhone OS 8_3 like Mac OS X) AppleWebKit/537.36 (KHTML, like Gecko) Version/8.0 Mobile/12F70 Safari/600.1.4 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
	Mozilla/5.0 (compatible; NLNZ_IAHarvester2014 +https://natlib.govt.nz/publishers-and-authors/web-harvesting/domain-harvest)
	Mozilla/5.0 (compatible; YandexImageResizer/2.0; +http://yandex.com/bots)
	mahonie, neofonie search:robot/search:robot/0.0.1 (This is the MIA Bot - crawling for mia research project. If you feel unhappy and do not want to be visited by our crawler send an email to [email protected]; http://spider.neofonie.de; [email protected])
	Mozilla/5.0 (Linux; Android 5.0; Nexus 5 Build/LRX21O) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/46.0.2490.76 Mobile Safari/537.36 PTST/276
	Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; Trident/5.0; WOW64; Trident/5.0; BingPreview/1.0b)
	Mozilla/5.0 (compatible; OptimizationCrawler/0.2; +http://www.domainoptima.com/bot.html)
	Mozilla/5.0 (compatible; Cloudinary/1.0)
	rogerbot/1.0 (http://moz.com/help/pro/what-is-rogerbot-, [email protected])
	Iframely/0.9.8 (+http://iframely.com/;)
	yacybot (webportal-global; amd64 Linux 2.6.32-5-amd64; java 1.6.0_18; Europe/en) http://yacy.net/bot.html
	Interdose AntiSpamBot/2.0 (+http://www.piep.net)
	CheckMarkNetwork/1.0 (+http://www.checkmarknetwork.com/spider.html)
	Mozilla/5.0 (iPhone; CPU iPhone OS 8_3 like Mac OS X) AppleWebKit/537.36 (KHTML, like Gecko) Version/8.0 Mobile/12F70 Safari/600.1.4 (compatible; Google Search Console)
	rogerbot/1.0 (http://www.seomoz.org/dp/rogerbot, [email protected])
	findlinks/2.5 (+http://wortschatz.uni-leipzig.de/findlinks/)
	Mozilla/5.0 (compatible; MSIE 10.0; Windows NT 6.1; WOW64; Trident/6.0) PTST/277
	yacybot (/global; x86 Windows 2003 5.2; java 1.7.0_67; Europe/de) http://yacy.net/bot.html
	NeutrinoAPI/2.0.3
	Mozilla/5.0 (compatible; spbot/5.0.3; +http://OpenLinkProfiler.org/bot )
	Mozilla/5.0 (compatible; MJ12bot/v1.4.4; http://www.majestic12.co.uk/bot.php?+)
	Site24x7 Agent
	Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.2; WOW64; Trident/6.0; .NET4.0E; .NET4.0C; .NET CLR 3.5.30729; .NET CLR 2.0.50727; .NET CLR 3.0.30729) CrawlerProcess (http://www.PowerMapper.com) /5.7.704.0
	Mozilla/5.0 (compatible; Falconsbot; +http://iws.seu.edu.cn/services/falcons/contact_us.jsp)
	Super Monitoring
	Seznam-Zbozi-robot/3.0
	LinkedInBot/1.0 (compatible; Mozilla/5.0; Jakarta Commons-HttpClient/4.3 +http://www.linkedin.com)
	Mozilla/5.0 (compatible; EveryoneSocialBot/1.0; [email protected] http://everyonesocial.com/)
	Mozilla/5.0 (compatible; IstellaBot/1.01.18 +http://www.tiscali.it/)
	Mozilla/5.0 (compatible; ToutiaoSpider/1.0; http://web.toutiao.com/media_cooperation/;)
	Mozilla/5.0 (compatible; BegunAdvertising/3.0; +http://begun.ru/begun/technology/indexer/)
	Netcraft SSL Server Survey - contact [email protected]
	Mozilla/5.0 (compatible; Qualidator.com SiteAnalyzer 1.0;)
	Mozilla/5.0 (compatible; OrangeBot/2.0; [email protected])
	Mozilla/5.0 (compatible; CloudFlare-AlwaysOnline/1.0; +http://www.cloudflare.com/always-online)
	seebot/1.0.0 (http://www.seegnify.com/bot)
	Slackbot-LinkExpanding 1.0 (+https://api.slack.com/robots)
	Eurobot/1.1 (http://eurobot.ayell.eu)
	Mozilla/5.0 (compatible; IXEbot; +http://medialab.di.unipi.it/IXEbot.html)
	Mozilla/5.0 (GetLinkInfo.com - http://www.getlinkinfo.com)
	Mozilla/5.0 (compatible; spbot/4.0.5; +http://www.seoprofiler.com/bot )
	yacybot (freeworld/global; amd64 Windows 7 6.1; java 1.6.0_31; America/en) http://yacy.net/bot.html
	Mozilla/5.0 (Windows NT 6.1; WOW64; rv:48.0) Gecko/20100101 Firefox/48.0 PTST/314
	Mozilla/5.0 (compatible; AhrefsBot/4.0; +http://ahrefs.com/robot/)
	UXCrawlerBot
	Pixray-Seeker/2.0 (http://www.pixray.com/pixraybot; [email protected])
	Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/50.0.2661.94 Safari/537.36 PTST/276
	FAST-WebCrawler/3.7 (atw-crawler at fast dot no; http://fast.no/support/crawler.asp)
	yacybot (freeworld/global; amd64 Windows 10 10.0; java 1.8.0_60; Europe/de) http://yacy.net/bot.html
	Technoratibot/7.0
	Mozilla/4.0 (compatible; MSIE 5.01; Windows NT 5.0; DomainDB-1.1; http://domaindb.com/crawler/)
	Abrave v6.0 (http://robot.abrave.com)
	Testomatobot/1.0 (Linux x86_64; +http://www.testomato.com/testomatobot) minicrawler/4.0.0~beta11
	DowntimeDetector/1.0 (+http://downforeveryoneorjustme.com)
	Comodo Spider 1.1
	Link Valet Online 1.1
	yacybot (freeworld/global; amd64 Windows XP 5.2; java 1.7.0_04; America/en) http://yacy.net/bot.html
	li_viewer ([email protected])
	Mozilla/5.0 (compatible; Mail.RU/2.0c)
	Mozilla/5.0 (compatible; aiHitBot/1.0; +http://www.aihit.com/)
	Mozilla/5.0 (compatible; Uptimebot/0.2.19; +http://www.uptime.com/uptimebot)
	Mozilla/5.0 (compatible; YandexBot/3.0; +http://yandex.com/bots)
	Mozilla/5.0 (iPhone; CPU iPhone OS 6_0 like Mac OS X) AppleWebKit/536.26 (KHTML, like Gecko) Version/6.0 Mobile/10A5376e Safari/8536.25 ( compatible; CloudServerMarketSpider/1.0; +http://cloudservermarket.com/spider.html )
	yacybot (i386 Linux 2.6.28-13-generic; java 1.6.0_13; Europe/en) http://yacy.net/bot.html
	linguatools-bot/Nutch-1.6 (searching for translated pages; http://www.linguatools.de/linguatoolsbot.html; peter dot kolb at linguatools dot org)
	Mozilla/5.0 (compatible; Lippershey/5.2; +https://www.lipperhey.com/en/about/)
	NextGenSearchBot 69 (for information visit http://www.zoominfo.com/business/nextgensearchbot.html)
	Mozilla/5.0 (compatible; YandexWebmaster/2.0; +http://yandex.com/bots)
	Mozilla/5.0 (compatible; Baiduspider/2.0; +http://www.baidu.com/search/spider.html)
	yacybot (freeworld-global; amd64 Linux 3.2.0-35-generic; java 1.7.0_09; Europe/en) http://yacy.net/bot.html
	peerindex/0.1 (http://www.peerindex.com/; crawler AT peerindex DOT com)
	iqdb/0.2 (+http://iqdb.harry.lu/)
	Mozilla/5.0 (compatible; LuminateBot/1.0; +http://www.luminate.com/bot/)
	Mozilla/5.0 (X11; Linux x86_64) adbeat.com/policy AppleWebKit/537.21 (KHTML, like Gecko) Firefox/34.0 Safari/537.21
	AntBot/1.0 (http://www.ant.com)
	Mozilla/5.0 (compatible; Thumbshots.ru; +http://thumbshots.ru/bot) Firefox/3
	Mozilla/5.0 (compatible; value/1.1; +http://www.aihit.com/)
	Digg Deeper/v1 (http://digg.com/about)
	Elmer, the Thinglink ImageBot (http://www.thinglink.com/help/ThinglinkImageBot)
	rogerbot/1.0 (http://moz.com/help/pro/what-is-rogerbot-, [email protected])
	Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.1 (KHTML, like Gecko) Chrome/21.0.1180.89 Safari/537.1; http://spyonweb.com [email protected]
	BabalooSpider/1.3 (BabalooSpider; http://www.babaloo.si; [email protected])
	Mozilla/5.0 (compatible; dotSemantic/1.0; +http://www.dotsemantic.org)
	TinEye/1.1 (http://tineye.com/crawler.html)
	Mozilla/5.0 (compatible; SemrushBot/0.98~bl; +http://www.semrush.com/bot.html)
	CorpusCrawler 2.0.9 (http://corpora.fi.muni.cz/crawler/)
	aboutthedomain
	Cliqzbot/0.1 (+http://cliqz.com/company/cliqzbot)
	Mozilla/5.0 (compatible; Uptimebot/0.2.43; +http://www.uptime.com/uptimebot)
	LinqiaMetadataDownloaderBot/1.0 ([email protected])
	yacybot (freeworld/global; amd64 Linux 3.2.0-32-generic; java 1.6.0_24; Europe/en) http://yacy.net/bot.html
	NeumobBot/0.2.16 (+http://www.neumob.com/bot)
	Mozilla/5.0 (compatible; Mp3Bot/0.4; +http://mp3realm.org/mp3bot/)
	Miniflux (https://miniflux.net)
	Mozilla/5.0 (iPhone; CPU iPhone OS 8_3 like Mac OS X) AppleWebKit/537.36 (KHTML, like Gecko; Google Page Speed Insights) Version/8.0 Mobile/12F70 Safari/600.1.4
	Digincore crawler bot. See https://www.digincore.com/crawler.html for rules and instructions.
	Mozilla/5.0 (iPhone; CPU iPhone OS 7_0 like Mac OS X) AppleWebKit/537.51.1 (KHTML, like Gecko) Version/7.0 Mobile/11A465 Safari/9537.53 (compatible; adidxbot/2.0; http://www.bing.com/bingbot.htm)
	NerdyBot
	Nutch/2.2.1 (page scorer; http://integralads.com/site-indexing-policy/)
	Mozilla/5.0 (compatible; 008/0.83; http://www.80legs.com/webcrawler.html;) Gecko/2008032620
	Mozilla/4.0 (compatible; LinkMarket-Bot)
	SeoCheckBot (Seo-Check, http://www.kfsw.de/bot.html)
	Mozilla/5.0 (compatible; Whynder Magnet; http://whynder.com/magnet)
	IdeelaborPlagiaat/1
	x28-job-bot; +http://x28.ch/bot.html
	Mozilla/5.0 (compatible; spbot/4.0.7; +https://www.seoprofiler.com/bot )
	Mozilla/5.0 (compatible; RelateIQ Crawler/0.4; +www.relateiq.com; [email protected])
	WebRankSpider/1.37 (+http://ulm191.server4you.de/crawler/)
	Mozilla/5.0 (iPhone; U; CPU iPhone OS 10_0 like Mac OS X; en-us) AppleWebKit/602.1.38 (KHTML, like Gecko) Version/10.0 Mobile/14A5297c Safari/602.1 (compatible; Mediapartners-Google/2.1; +http://www.google.com/bot.html)
	LinkWalker/3.0 (http://www.brandprotect.com)
	Mozilla/5.0 (compatible; discobot/1.0; +http://discoveryengine.com/discobot.html)
	ia_archiver-web.archive.org
	Mozilla/5.0 (compatible; Online Domain Tools - SEO Checker/1.0; +http://online-domain-tools.com)
	Mozilla/5.0 (compatible; AhrefsBot/3.0; +http://ahrefs.com/robot/)
	Mozilla/5.0 (Macintosh; Intel Mac OS X 10_6_8) AppleWebKit/537.13+ (KHTML, like Gecko) Version/5.1.7 Safari/534.57.2; Shoppimon Analyzer (http://www.shoppimon.com/)
	Mozilla/5.0 (compatible; NewShareCounts.com/1.0; +http://newsharecounts.com/crawler)
	Mozilla/5.0 (compatible; Mozilla; +http://wiki.github.com/bixo/bixo/bixocrawler; [email protected])
	Mozilla/5.0 (compatible; CukBot; Not a spammer; http://www.companiesintheuk.co.uk/bot.html)
	yacybot (freeworld/global; amd64 Linux 3.1.10-1.9-default; java 1.6.0_24; Europe/de) http://yacy.net/bot.html
	Nuhk/2.4 (+http://www.neti.ee/cgi-bin/abi/otsing.html)
	elefent/Elefent 1.1 (Friendly web walker.; http://www.elefent.eu/; [email protected])
	HubSpot Links Crawler 1.0 http://www.hubspot.com/
	Mozilla/5.0 (compatible; Siteluxbot/1.0; +https://sitelux.co.uk/)
	CopperEgg/RevealUptime/AmsterdamNL(softlayer)
	Mozilla/5.0 (compatible; spbot/4.0.7; +http://OpenLinkProfiler.org/bot )
	Mediumbot-MetaTagFetcher/0.1 (+https://medium.com/)
	Mozilla/2.0 (compatible; Ask Jeeves/Teoma; +http://about.ask.com/en/docs/about/webmasters.shtml)
	Mozilla/5.0 (compatible; 008/0.85; http://www.80legs.com/webcrawler.html) Gecko/2008032620
	yacybot (/global; amd64 Linux 4.4.0ct; java 1.7.0_91; Europe/de) http://yacy.net/bot.html
	Mozilla/5.0 (compatible; Scrubby/2.2; http://www.scrubtheweb.com/)
	yacybot (/global; amd64 Linux 3.2.0-4-amd64; java 1.8.0_05; Europe/en) http://yacy.net/bot.html
	Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.2.18) Gecko/20110622 HeartRails_Capture/1.0.4 (+http://capture.heartrails.com/) Namoroka/3.6.18
	Mozilla/5.0 (X11; U; Linux i686; de; rv:1.9.0.1; compatible; iCjobs Stellenangebote Jobs; http://www.icjobs.de) Gecko/20100401 iCjobs/3.2.3
	KOCMOHABT (+http://www.KOCMOHABT.tk/) AppEngine-Google; (+http://code.google.com/appengine; appid: s~web-explorer)
	yacybot (freeworld/global; x86 Windows 7 6.1; java 1.6.0_31; Europe/de) http://yacy.net/bot.html
	Mozilla/5.0 (compatible; MSIE 10.0; Windows NT 6.2; WOW64; Trident/6.0) CrawlerProcess (http://www.PowerMapper.com) /5.21.754.0
	Mozilla/5.0 (compatible; Linux x86_64; Mail.RU_Bot/Robots; +http://go.mail.ru/help/robots)
	Mozilla/5.0 (compatible; Googlebot/2.1; +http://import.io)
	ownCloud Server Crawler
	Mozilla/5.0 (compatible; Qwantify/2.1n; +https://www.qwant.com/)/*
	LoadImpactRload/3.1.5 (Load Impact; http://loadimpact.com);
	Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 4.0; Girafabot; girafabot at girafa dot com; http://www.girafa.com)
	OmniExplorer_Bot/6.65a (+http://www.omni-explorer.com) WorldIndexer
	Mozilla/5.0 (compatible; SMTBot/1.0; +http://www.similartech.com/smtbot)
	anonymous coward/1.0 (a Storm-based crawler; https://github.com/DigitalPebble/storm-crawler; [email protected])
	Mozilla/5.0 (compatible; Uptimebot/1.0; +http://www.uptime.com/uptimebot)
	Mozilla/5.0 (compatible; FoundSeoTool/1.0; +http://www.found.co.uk/seo-tool/)
	findlinks/2.1.5 (+http://wortschatz.uni-leipzig.de/findlinks/)
	yacybot (freeworld/global; amd64 Windows NT (unknown) 6.2; java 1.7.0_05; Africa/en) http://yacy.net/bot.html
	Mozilla/5.0 (compatible; memorybot/1.20.71 +http://archivethe.net/en/index.php/about/internet_memory1 on behalf of DNB)
	ownCloud News/5.3.6 (+https://owncloud.org/; 1 subscriber;)
	YottaBot Spider/Nutch-2.3
	Mozilla/5.0 (compatible; WeViKaBot/1.0; +http://www.wevika.de/)
	ThumbShots-Bot (+http://thumbshots.in/bot.html)
	Mnogosearch-3.1.21
	Favicon downloader (+https://favico.be/)
	Mozilla/5.0 (compatible; Page2RSS/0.7; +http://page2rss.com/)
	ExB Language Crawler 2.1.2 (+http://www.exb.de/crawler)
	PrivateSearch/0.1.0 (Polite Nutch Crawler; grierforensics.com)
	nodestackbot/0.1 ([email protected] http://nodestack.com/bot.html)
	iskanie (+http://www.iskanie.com)
	yacybot (/global; amd64 Windows 7 6.1; java 1.7.0_67; Europe/de) http://yacy.net/bot.html
	Mozilla/5.0 (compatible; Dataprovider Site Explorer; http://www.dataprovider.com/)
	Mozilla/5.0 (compatible; akula/12.0rc-2; +http://k311.fd.cvut.cz/)
	Mozilla/5.0 (compatible; bnf.fr_bot; +http://www.bnf.fr/fr/outils/a.dl_web_capture_robot.html)
	Mozilla/5.0 (iPhone; CPU iPhone OS 6_0 like Mac OS X) AppleWebKit/536.26 (KHTML, like Gecko) Version/6.0 Mobile/10A5376e Safari/8536.25 (compatible; Googlebot-Mobile/2.1; +http://www.google.com/bot.html)
	Mozilla/5.0 (compatible; MSIE 10.0; Windows NT 6.1; WOW64; Trident/6.0) PTST/284
	Mozilla/5.0 (compatible; Infohelfer/1.3.3; +http://www.infohelfer.de/crawler.php)
	www.deadlinkchecker.com XMLHTTP/1.0
	Mozilla/5.0 (compatible; TweetmemeBot/3.0; +http://tweetmeme.com/)
	Chrome/49.0.2623.87 (compatible; Wappalyzer; +https://github.com/AliasIO/Wappalyzer)
	Mozilla/5.0 (compatible; MSIE 10.0; Windows NT 6.2; WOW64; Trident/6.0) CrawlerProcess (http://www.PowerMapper.com) /5.11.740.0
	XING-contenttabreceiver/2.0
	Mozilla/5.0 (iPhone; CPU iPhone OS 6_0 like Mac OS X) AppleWebKit/536.26 (KHTML, like Gecko) Version/6.0 Mobile/10A5376e Safari/8536.25 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
	Mozilla/5.0 (compatible; MojeekBot/0.2; http://www.mojeek.com/bot.html#relaunch)
	Pingoscope
	Mozilla/5.0 (compatible; um-LN/1.0; mailto: [email protected])
	Mozilla/5.0 (compatible; Pagespeed/1.1 Fetcher; +http://www.pagespeed.de)
	HTTP-Header-Abfrage/1.0 (http://www.internalscripts.de/werkzeuge/http-header-abfrage.php)
	Mozilla/5.0 (compatible; 80bot/0.71; http://www.80legs.com/spider.html;) Gecko/2008032620
	LoadImpactRload/3.0.3 (Load Impact; http://loadimpact.com);
	Mozilla/5.0 (compatible; Peepowbot/1.0; +http://www.peepow.com/bot.php)
	yacybot (freeworld/global; amd64 Linux 2.6.32-5-amd64; java 1.6.0_26; Atlantic/en) http://yacy.net/bot.html
	WatchMouse/8.4.0.3 (http://watchmouse.com/ ; grath01.watchmouse.net)
	Mozilla/5.0 (compatible; +http://servernfo.com/)
	Mozilla/5.0 (compatible; special_archiver/3.1.1 +http://www.archive.org/details/archive.org_bot)
	CopperEgg/RevealUptime/NewarkNJUSA
	Mozilla/5.0 (compatible; archive.org_bot/3.3.0 +http://www.archive.org/details/archive.org_bot)
	Mozilla/5.0 (compatible; Plukkie/1.1; http://www.botje.com/plukkie.htm)
	Mozilla/5.0 (compatible; musobot/1.0; [email protected]; +http://www.muso.com)
	yacybot (freeworld/global; amd64 Linux 3.10.17-gentooxen; java 1.6.0_31; Europe/nb) http://yacy.net/bot.html
	ResponseCodeTest/1.1
	Mozilla/5.0 (compatible; MSIE 10.0; Windows NT 6.1; Trident/6.0) SiteCheck-sitecrawl by Siteimprove.com
	Acoon v4.9.5 (www.acoon.de)
	Mozilla/5.0 (Windows NT 6.1; WOW64) SkypeUriPreview Preview/0.5
	SiteGuardian/2.0 (Internet Monitoring)
	Mozilla/5.0 (compatible; DCPbot/1.4; +http://domains.checkparams.com/)
	BoardReader Favicon Fetcher /1.0 [email protected]
	baypup/colbert (Baypup; http://www.baypup.com/webmasters; [email protected])
	Mozilla/5.0 (compatible; MonTools.com)
	yacybot (freeworld/global; amd64 Linux 3.2.0-4-amd64; java 1.7.0_03; Etc/de) http://yacy.net/bot.html
	Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/534.51 (KHTML, like Gecko; Google Web Preview) Chrome/12.0.742 Safari/534.51
	linkapediabot (+http://www.linkapedia.com)
	Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.7.12; ips-agent) Gecko/20050922 Fedora/1.0.7-1.1.fc4 Firefox/1.0.7
	BCKLINKS 1.0
	CorpusCrawler 2.0.21 (http://corpora.fi.muni.cz/crawler/);Project:CzCorpus
	Mozilla/5.0 (compatible; Yeti/1.1; +http://help.naver.com/support/robots.html)
	Mozilla/5.0 (compatible; pingbot/2.0; +http://www.pingdom.com/)
	Mozilla/5.0 (compatible; memoryBot/1.20.275 +http://internetmemory.org/en/)
	Mozilla/5.0 (compatible; spbot/1.1; +http://www.seoprofiler.com/bot/ )
	yacybot (/global; amd64 Linux 3.13.0-88-generic; java 1.7.0_101; Europe/en) http://yacy.net/bot.html
	BlogSearch/2 +http://www.icerocket.com/
	Mozilla/5.0 (compatible; LoadTimeBot/0.81; +http://www.load-time.com/bot.html)
	Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9) Gecko/2008052906 Firefox/3.0 (BrowserSpyBot/1.0) (http://browserspy.dk/)
	Mozilla/5.0 (compatible; MojeekBot/0.6; +https://www.mojeek.com/bot.html)
	CopperEgg/RevealUptime/NewarkNJ(linode)
	Automattic Analytics Crawler/0.2; http://wordpress.com/crawler/
	larbin_2.6.3 [email protected]
	HatenaScreenshot
	CopperEgg/RevealUptime/LondonUK
	Mozilla/5.0 (compatible; MojeekBot/0.6; http://www.mojeek.com/bot.html)
	Mozilla/5.0 (compatible; evc-batch/2.0.20160902205649)
	yacybot (freeworld/global; amd64 Linux 3.2.0-23-generic; java 1.6.0_24; Europe/en) http://yacy.net/bot.html
	WordPress.com;
	Mozilla/5.0 (Windows NT 6.1; rv:6.0) Gecko/20110814 Firefox/6.0 Google (+https://developers.google.com/+/web/snippet/)
	Mozilla/5.0 (compatible; ScoutJet; http://www.scoutjet.com/)
	Mozilla/5.0 (compatible; Uptimebot/0.1.97; +http://www.uptime.com/uptimebot)
	WebTarantula.com Crawler
	WatchMouse/8.4.0.3 (http://watchmouse.com/ ; itmil01.watchmouse.net)
	LinqiaScrapeBot/1.0 ([email protected])
	Orbiter (+http://www.dailyorbit.com/bot.htm)
	Mozilla/5.0 (compatible; janforman.com/2.5; +http://janforman.com/)
	InternetSeer.com
	Mozilla/5.0 (compatible; aiHitBot/2.9; +http://www.aihitdata.com/about)
	Mozilla/5.0 (compatible; WebNL; +http://www.web.nl/webmasters/spider.html; [email protected])
	coccoc/1.0 (http://help.coccoc.vn/)
	Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/53.0.2785.116 Safari/537.36 PTST/314
	Mozilla/5.0 (compatible; Googlebot/2.1; startmebot/1.0; +https://start.me/bot)
	Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 5.2; Trident/4.0; .NET CLR 1.1.4322; .NET CLR 2.0.50727; .NET CLR 3.0.4506.2152; .NET CLR 3.5.30729; .NET4.0C; .NET4.0E; PTST 2.386)
	Mozilla/5.0 (compatible; PrivacyAwareBot/1.1; +http://www.privacyaware.org)
	Mozilla/5.0 (compatible; idmarch Automatic.beta/1.4; +http://www.idmarch.org/bot.html)
	Mozilla/5.0 (compatible; tldstat.com/Bot; +http://tldstat.com/bot.html)
	yacybot (/global; amd64 Linux 3.16.0-4-amd64; java 1.7.0_111; Europe/de) http://yacy.net/bot.html
	Mozilla/5.0 (compatible; Crawlera/1.10.2; UID 84846)
	Mozilla/5.0 (compatible; spbot/4.0a; +http://www.seoprofiler.com/bot )
	Mozilla/5.0 (Windows NT 6.1; Win64; x64; +http://www.komodia.com/newwiki/index.php/URL_server_crawler) KomodiaBot/1.0
	WebarooBot (Webaroo Bot; http://64.124.122.252/feedback.html)
	OOZBOT/0.20 ( -- ; http://www.setooz.com/oozbot.html ; agentname at setooz dot_com )
	Mozilla/5.0 (compatible; memoryBot/1.21.16 +http://internetmemory.org/en/)
	FAST-WebCrawler/3.8
	Googlebot-Image/1.0
	enlle punto com/Nutch-1.9
	OpenAcoon v4.10.5 (www.openacoon.de)
	yacybot (freeworld/global; amd64 Linux 3.0.0-21-generic; java 1.6.0_26; America/en) http://yacy.net/bot.html
	Microsearch.ru Bot 1.3;http://microsearch.ru/webmasters
	Mozilla/5.0 (compatible; Google-Site-Verification/1.0)
	Mozilla/5.0 (compatible; Qwantify/2.3w; +https://www.qwant.com/)/2.3w
	rogerbot/1.1 (http://moz.com/help/guides/search-overview/crawl-diagnostics#more-help, [email protected])
	HubSpot Connect 2.0 (http://dev.hubspot.com/)
	Robozilla/1.0
	Mozilla/5.0 (compatible; spbot/2.0.3; +http://www.seoprofiler.com/bot/ )
	Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/53.0.2785.116 Safari/537.36 PTST/314
	facebookexternalhit/1.1
	PercolateCrawler/3.1.30 ([email protected])
	Mozilla/5.0 (compatible; Nmap Scripting Engine; https://nmap.org/book/nse.html)
	Mozilla/5.0 (compatible; pdffillerbot/1.0; +http://www.pdffiller.com/en/support.htm)
	fastbot crawler beta 4.0 (+http://www.fastbot.de)
	SafeDNS search bot/Nutch-1.9 (https://www.safedns.com/searchbot; support [at] safedns [dot] com)
	Mozilla/5.0 (Nekstbot; http://nekst.ipipan.waw.pl/nekstbot/)
	yacybot (freeworld/global; amd64 Linux 2.6.32-5-amd64; java 1.6.0_18; US/en) http://yacy.net/bot.html
	yacybot (freeworld/global; amd64 Linux 3.0.0-12-generic; java 1.6.0_26; Europe/en) http://yacy.net/bot.html
	coccoc/1.0 ()
	larbin_2.6.3 ([email protected])
	yacybot (i386 Linux 2.6.26-2-686; java 1.6.0_0; Europe/de) http://yacy.net/bot.html
	Mozilla/5.0 (Windows NT 6.1; compatible; BDCbot/1.0; +http://ecommerce.bigdatacorp.com.br/faq.aspx) AppleWebKit/535.1 (KHTML, like Gecko) Chrome/13.0.782.107 Safari/535.1
	yacybot (amd64 Linux 2.6.31-22-server; java 1.6.0_18; Asia/en) http://yacy.net/bot.html
	yacybot (/global; amd64 Windows Server 2012 6.2; java 1.8.0_91; Europe/de) http://yacy.net/bot.html
	Mozilla/5.0 (compatible; Netseer crawler/2.0; +http://www.netseer.com/crawler.html; [email protected])
	Mozilla/5.0 (compatible; heritrix/3.1.1 +http://www.mixdata.com)
	Scrapy/0.25.1 (+http://scrapy.org)
	Mozilla/5.0 (compatible; Scopia Crawler 1.2; +http://www.scopia.co)
	cn_ab_viewer ([email protected])
	Mozilla/5.0 (Windows NT 6.3; Trident/7.0; rv 11.0) like Gecko (compatible; Zombiebot/2.1; +http://www.zombiedomain.net/robot/)
	Mozilla/5.0 (compatible; Uptimebot/0.1.38; +http://www.uptime.com/uptimebot)
	yacybot (freeworld/global; amd64 Windows 7 6.1; java 1.7.0_25; Europe/de) http://yacy.net/bot.html
	Python-urllib/2.7 (+http://sketch.juls.savba.sk/aranea_about)
	Mozilla/5.0 (compatible; spbot/4.0; +http://www.seoprofiler.com/bot )
	yacybot (-global; amd64 Windows 8 6.2; java 1.7.0_55; Europe/de) http://yacy.net/bot.html
	yacybot (/global; amd64 Linux 3.2.41-042stab094.7; java 1.7.0_79; Europe/en) http://yacy.net/bot.html
	Mozilla/5.0 (compatible; UASlinkChecker/1.0; +http://user-agent-string.info/UASlinkChecker)
	Y!J-BRO/YFSJ crawler (compatible; Mozilla 4.0; MSIE 5.5; http://help.yahoo.co.jp/help/jp/search/indexing/indexing-15.html; YahooFeedSeekerJp/2.0)
	yacybot (freeworld/global; amd64 Linux 3.10.15-1-MANJARO; java 1.7.0_40; Asia/en) http://yacy.net/bot.html
	Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.2; WOW64; Trident/6.0; .NET4.0E; .NET4.0C; .NET CLR 3.5.30729; .NET CLR 2.0.50727; .NET CLR 3.0.30729) CrawlerProcess (http://www.PowerMapper.com) /5.9.726.0
	Scrubby/2.1 (http://www.scrubtheweb.com/)
	Mozilla/5.0 (compatible; JobKeresoBot; +http://www.kozvetlen-allasok.hu/help.jsp; [email protected]; 5.0
	yacybot (/global; amd64 Windows 7 6.1; java 1.8.0_66; Europe/de) http://yacy.net/bot.html
	Mozilla/5.0 (X11; Linux x86_64 UXCrawlerBot http://zium.co)
	Slackbot-LinkExpanding (+https://api.slack.com/robots)
	yacybot (freeworld/global; amd64 Linux 3.12.15-gentoo; java 1.7.0_55; Europe/en) http://yacy.net/bot.html
	Mozilla/5.0 (compatible; LinkpadBot/1.07; +http://www.linkpad.ru)
	Mozilla/5.0 (compatible; ZB-1; +http://www.zeerch.com/bot.php)
	Mozilla/5.0 (compatible; WbSrch/1.1 +http://wbsrch.com)
	WeSEE:Search
	Mozilla/5.0 (compatible; DCPbot/1.1; +http://domains.checkparams.com/)
	yacybot (webportal-global; amd64 Linux 2.6.32-5-amd64; java 1.6.0_45; Europe/en) http://yacy.net/bot.html
	WordPress.com mShots; http://support.wordpress.com/contact/
	Mozilla/4.0 (compatible; MSIE 5.01; Windows NT 5.0; FeedFinder-2.0; http://bloggz.se/crawler)
	Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.6; en-US; rv:1.9.2) Gecko/20100115 Firefox/3.6 (FlipboardProxy/0.0.5; +http://flipboard.com/browserproxy)
	Mozilla/5.0 (compatible; coccocbot-image/1.0; +http://help.coccoc.com/searchengine)
	Mozilla/5.0 (X11; Linux x86_64; GTmetrix http://gtmetrix.com/) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/31.0.1650.63 Safari/537.36
	https_page_test ([email protected])
	Mozilla/5.0 (Windows; U; Windows NT 5.1; zh-CN; rv:1.8.0.11) Firefox/1.5.0.11 360Spider;
	yacybot (freeworld/global; amd64 Linux 3.2.0-2-amd64; java 1.6.0_24; Europe/en) http://yacy.net/bot.html
	Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_2) AppleWebKit/600.1.4 (KHTML, like Gecko) Safari/600.1.4 (compatible; Princetonbot/1.0; +http://http://tigress-web.princeton.edu/~fy/bot.html)
	Mozilla/5.0 (Windows; U; Windows NT 5.1; zh-CN; rv:1.8.0.11) Gecko/20070312 Firefox/1.5.0.11; 360Spider
	KeywordDensityRobot/0.8 (http://www.seocentro.com/tools/search-engines/keyword-density.html)
	Mozilla/5.0 (compatible; aiHitBot-BP/1.1; +http://www.aihit.com/)
	findlinks/1.1.6-beta4 (+http://wortschatz.uni-leipzig.de/findlinks/)
	Mozilla/5.0 (compatible; Butterfly/1.0; +http://labs.topsy.com/butterfly/) Gecko/2009032608 Firefox/3.0.8
	CopperEgg/RevealUptime/FremontCA(linode)
	Mozilla/5.0 (compatible; NTENTbot; +http://www.ntent.com/ntentbot)
	link_thumbnailer
	KeywordDensityRobot/0.9 (http://www.seocentro.com/tools/search-engines/keyword-density.html)
	Mozilla/5.0 (compatible;WI Job Roboter Spider Version 3;+http://www.webintegration.at)
	Mozilla/5.0 (compatible; Mail.RU/2.0)
	CopperEgg/RevealUptime/N.California(aws)
	Searchie Bot/1.0 (a Storm-based crawler; https://www.searchie.org; [email protected])
	^Nail (http://CaretNail.com)
	MarketBrewBot/2.1 (+http://www.marketbrew.com/)
	Updownerbot (+http://www.updowner.com/bot)
	yacybot (-global; amd64 Windows 8.1 6.3; java 1.7.0_55; Europe/de) http://yacy.net/bot.html
	Yanga WorldSearch Bot v1.1/beta (http://www.yanga.co.uk/)
	Mozilla/5.0 (compatible; bixolabs/1.0; +http://bixolabs.com/crawler/general; [email protected])
	Mozilla/5.0 (compatible; Scopia Crawler 2.0; +http://www.scopia.co)
	asafaweb.com
	KrOWLer/0.0.1, matentzn at cs dot man dot ac dot uk
	Mozilla/5.0 (compatible; alexa site audit/1.0; +http://www.alexa.com/help/webmasters; [email protected])
	findlinks/1.0 (+http://wortschatz.uni-leipzig.de/findlinks/)
	SygolBot http://www.sygol.com
	voltron
	Mozilla/5.0 (compatible; uMBot-FC/1.0; mailto: [email protected])
	Comodo-Webinspector-Crawler 2.2.1, http://www.comodorobot.com
	Superarama.com-Tarama-Botu-v.0.1
	yacybot (freeworld/global; amd64 Linux 3.2.0-3-amd64; java 1.6.0_24; Europe/en) http://yacy.net/bot.html
	Mozilla/5.0 (compatible; SSLBot/1.0; http://www.sslstats.com/sslbot)
	PagesInventory.com (robot http://www.pagesinvenotry.com)
	Mozilla/5.0 (iPhone; U; CPU iPhone OS 4_3 like Mac OS X; en-US) adbeat.com/policy AppleWebKit/533.17.9 (KHTML, like Gecko) Version/5.0.2 Mobile/8F190 Safari/6533.18.5
	WordPress.com Reader
	Mozilla/5.0 (compatible; SEOkicks-Robot +http://www.seokicks.de/robot.html)
	Woko 3.0
	Gigabot/3.0 (http://www.gigablast.com/spider.html)
	ichiro/4.0 (http://help.goo.ne.jp/door/crawler.html)
	bl.uk_lddc_bot/3.1.1 (+http://www.bl.uk/aboutus/legaldeposit/websites/websites/faqswebmaster/index.html)
	Urlstat (http://bit.ly/1dJuuzs)
	yacybot (/global; amd64 Windows 8.1 6.3; java 1.7.0_55; America/ru) http://yacy.net/bot.html
	Bad Neighborhood Header Detector (http://www.bad-neighborhood.com/header_detector.php)
	Mozilla/5.0 (compatible; GeliyooBot/1.0beta; +http://www.geliyoo.com/)
	NalezenCzBot/1.0 (http://www.nalezen.cz/about-crawler)
	WeSEE_Bot:we_help_monitize_your_site (http://www.wesee.com/bot/)
	yacybot (freeworld/global; x86 Windows XP 5.1; java 1.7.0_55; Asia/en) http://yacy.net/bot.html
	studylib download bot, see http://studylib.es/public/bot for additional info
	SemrushBot/0.91
	Jabse.com Crawler v.1.0 www.jabse.com/crawler.php//imagecrawler
	Mozilla/4.0 (compatible; MSIE 5.0; Windows NT; Girafabot; girafabot at girafa dot com; http://www.girafa.com)
	BizwikiBot/0.3 (http://www.bizwiki.com/bizwikibot.htm)
	Mozilla/5.0 (compatible; BIXOCRAWLER; +http://wiki.github.com/bixo/bixo/bixocrawler; [email protected])
	PostPost/1.0 (+http://postpost.com/crawlers)
	Mozilla/5.0 (nomore404.com robot/1.1; +https://nomore404.com/)
	Riddler (http://riddler.io/about)
	Mozilla/5.0 (compatible; socketcrawler; http://nlp.fi.muni.cz/projects/biwec/)
	internetVista monitor (Mozilla compatible)
	2Bone_LinkChecker/1.0 libwww-perl/6.03
	MaxPointCrawler/Nutch-1.1 (maxpoint.crawler at maxpointinteractive dot com)
	DomainStatsBot/1.0 (http://domainstats.io/our-bot)
	KDDI-CA31 UP.Browser/6.2.0.7.3.129 (GUI) MMP/2.0 (compatible; ichiro/mobile goo;+http://search.goo.ne.jp/option/use/sub4/sub4-1/)
	Mergadobot/3.0.2 (+http://mergado.cz)
	DoCoMo/2.0 P900i(c100;TB;W24H11) (compatible; ichiro/mobile goo; +http://search.goo.ne.jp/option/use/sub4/sub4-1/)
	SalesIntelligent/v1.1
	SiteAnalyzerBot
	DoCoMo/2.0 P900i(c100;TB;W24H11) (compatible; ichiro/mobile goo; +http://help.goo.ne.jp/help/article/1142/)
	yacybot (/global; i386 Linux 3.16.3-gentoo; java 1.7.0_55; Australia/en) http://yacy.net/bot.html
	Mozilla/5.0 (compatible; Linux x86_64; Mail.RU_Bot/Img/2.0; +http://go.mail.ru/help/robots)
	Cliqz Bot (+http://www.cliqz.com)
	SalesIntelligent (+https://www.salesintelligent.com)
	Mozilla/5.0 (compatible; memoryBot/1.20.243 +http://internetmemory.org/en/)
	gimme60 (Gimme60 Store ID Bot; gimme60.com)
	Orbiter/1.0 (http://dailyorbit.com/)
	msnbot/1.1 (+http://search.msn.com/msnbot.htm)
	Mozilla/5.0 (compatible; memoryBot/1.20.241 +http://internetmemory.org/en/)
	Mozilla/5.0 (compatible; akula/k311; +http://k311.fd.cvut.cz/)
	Irokez.cz monitoring v1.2 - (http://www.irokez.cz, Irokez.cz, crawl)
	CCBot/2.0 (http://commoncrawl.org/faq/)
	Mozilla/5.0 (compatible; openstat.ru/Bot)
	Mozilla/5.0 (compatible; GigaBot/1.0; +http://www.gigablast.com/ )
	Readability/6a54d4 - http://readability.com/about/
	percbotspider <[email protected]>
	Mozilla/5.0 (compatible; spbot/1.0; +http://www.seoprofiler.com/bot/ )
	HatenaScreenshot/1.0 (checker)
	Mozilla/5.0 (compatible; AcoonBot/4.12.1; +http://www.acoon.de/robot.asp)
	CRAZYWEBCRAWLER 0.9.2, http://www.crazywebcrawler.com
	Mozilla/5.0 (compatible; Online Domain Tools - Online Sitemap Generator/1.1; +http://sitemap-generator.online-domain-tools.com)
	Mozilla/5.0 (compatible; Google-Structured-Data-Testing-Tool +https://search.google.com/structured-data/testing-tool)
	OnPageBot (compatible; Googlebot 2.1; +https://bot.onpage.org/)
	Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/48.0.2564.103 Safari/537.36 Yandex.Translate
	Sogou web spider/4.0
	Mozilla/5.0 (compatible; YandexMetrika/2.0; +http://yandex.com/bots DEV)
	LinkWalker/2.0
	SeoCheck (FischerNetzDesign Seo Checker, [email protected])
	yacybot (freeworld/global; amd64 Linux 3.2.13-1-ARCH; java 1.7.0_03-icedtea; Europe/fr) http://yacy.net/bot.html
	SiteCheck - http://downforeveryoneorjustme.com
	Robots_Tester_http_www.searchenginepromotionhelp.com
	CRAZYWEBCRAWLER 0.9.4, http://www.crazywebcrawler.com
	Mozilla/5.0 (compatible; heritrix/3.1.1 +http://places.tomtom.com/crawlerinfo)
	Mozilla/5.0 (Windows; U; Windows NT 5.0; en-US; rv:1.7.10) Gecko/20050716 Thunderbird/1.0.6 - WebCrawler http://cognitiveseo.com/bot.html
	Mozilla/5.0 (compatible; Cliqzbot/1.0 +http://cliqz.com/company/cliqzbot)
	yacybot (/global; amd64 Linux 4.2.0-gentoo-r1; java 1.8.0_45; Australia/en) http://yacy.net/bot.html
	DomainsDB.net MetaCrawler v.0.9.7c (http://domainsdb.net/)
	Mozilla/4.0 (compatible; MSIE 5.5; Windows NT 5.0; mowser; http://www.mowser.com)
	www.probethenet.com scanner
	Mozilla/5.0 (compatible; Crawlera/1.10.2; UID 6573)
	LCC (+http://corpora.informatik.uni-leipzig.de/crawler_faq.html)
	Imagga Downloader/v0.14.3
	SeopultContentAnalyzer/1.0
	Mozilla/5.0 (compatible; +http://www.hosterstats.com/newgtld-websurvey.php)
	Mozilla/5.0 (compatible; GoogleDocs; apps-spreadsheets; +http://docs.google.com)
	Mozilla/5.0 (compatible;acapbot/0.1)
	Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1;Alibaba.Security.Heimdall)
	HTTPClient/1.0 (2.8.0, ruby 2.0.0 (2014-05-08))
	Jetty/9.3.z-SNAPSHOT
	Jetty/9.3.z-SNAPSHOT, SMUrlExpander
	Jetty/9.3.z-SNAPSHOT, SMViewer
	Liferea/0.x.x (Linux; en_US.UTF-8; http://liferea.sf.net/)
	SynHttpClient-Built/5825
	asynchttp
	SMUrlExpander
	KickFire
	HAAARTLAND http client
	PEAR HTTPRequest class ( http://pear.php.net/ )
	Mozilla/4.0 (vBSEO; http://www.vbseo.com)
	HAARTLAND http client
	JS-Kit URL Resolver http://js-kit.com/
	http://www.checkprivacy.or.kr:6600/RS/PRIVACY_ENFAQ.jsp
	Kml-Google; (+http://code.google.com/apis/kml) gzip
	gvfs/1.20.3
	request.js
	Microsoft URL Control - 6.01.9782
	git/2.1.4
	IPS/1.0
	OxfordCloudService/1.0
	Mozilla/5.0 (compatible; informatique at mixdata dot com/0.6; +http://t.co/GSRLLKex24; informatique at mixdata dot com)
	Mozilla/5.0 (compatible; Feedwind/2.0; +http://feed.mikle.com/support/description/)
	Mozilla/5.0 (compatible; Feedwind/3.0; +http://feed.mikle.com/support/description/)
	Mozilla/5.0 (Macintosh; Intel Mac OS X 1068) AppleWebKit/534.50 (KHTML like Gecko) Version/5.1 Instapaper/4.0 (+http://www.instapaper.com/)
	Mozilla/5.0 (compatible; Keywords Research; http://www.check-domains.com/keyword-research/keywords-adwords.php)
	CAAM/0.1 (CAAM; caam dot crwlr at gmail dot com)
	i2kconnect/"1.7.0"
	resolver/0.1.12 (+http://github.com/andris9/resolver)
	DangDang
	Mozilla/5.0 (compatible; Tetrahedron/1.0; +https://compute.info/tet)
	PritTorrent/0.1
	FeedBunch/1.0 (+http://www.feedbunch.com)
	Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.21 (KHTML, like Gecko) www.HomepageScreenshot.de - Free ScreenShotService/1.0 Safari/537.21
	Mozilla/5.0 (compatible; Licorne Image Snapshot 1.0; +http://www.jednorozec.cz)
	Pinterest/0.2 (+https://www.pinterest.com/bot.html)
	Pinterest/0.1 (+https://www.pinterest.com/bot.html)
	Mozilla/5.0 (iplabel; Windows NT 6.3; WOW64; rv:33.0) Gecko/20100101 Firefox/33.0
	Mozilla/5.0 zgrab/0.x
	Zend_Http_Client
	Zend\Http\Client
	Zao/0.1 (http://www.kototoi.org/zao/)
	YOURLS v1.7.1 +http://yourls.org/ (running on http://flnx.co)
	yoogliFetchAgent/0.1
	Yo-yo
	Mozilla/5.0 (compatible; Yahoo Link Preview; https://help.yahoo.com/kb/mail/yahoo-link-preview-SLN23615.html)
	Mozilla/5.0 (compatible; Yahoo Ad monitoring; https://help.yahoo.com/kb/yahoo-ad-monitoring-SLN24857.html)
	Mozilla/5.0 (iPhone; CPU iPhone OS 7_1 like Mac OS X) AppleWebKit (KHTML, like Gecko) Mobile (compatible; Yahoo Ad monitoring; https://help.yahoo.com/kb/yahoo-ad-monitoring-SLN24857.html)
	xpymep.exe
	xpymep2.exe
	xpymep3.exe
	Xenu Link Sleuth 1.2b
	Xenu Link Sleuth 1.2c
	Xenu Link Sleuth 1.2d
	Xenu Link Sleuth 1.2e
	Xenu Link Sleuth 1.2f
	Xenu Link Sleuth 1.2g
	Xenu Link Sleuth 1.2h
	Xenu Link Sleuth 1.2i
	Xenu Link Sleuth/1.3.7
	Xenu Link Sleuth/1.3.8
	Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/40.0.2214.93 Safari/537.36 XaxisSemanticsClassifier/1.0 http://crystalsemantics.com
	WWW-Mechanize/1.73
	WordPress/4.0.1; http://bestcomp.hol.es
	WordPress/4.3.3; http://homebusinessreviews.online
	WordPress/4.3.3; http://legithomejobs.club
	WordPress/4.4.2; http://cheapautoinsuranceinva.com
	WordPress/4.4.2; http://citycollegeinmiami.com
	WordPress/4.4.2; http://cookingideabook.com
	WordPress/4.4.2; http://oldmanhealth.com
	WordPress/4.4.2; http://sapcrmarchitecture.com
	WordPress/4.4.2; http://www.spar3d.com
	WordPress/4.5.2; http://1.we-careu.xyz
	WordPress/x.x.x.x PHP/4.x.xxMozilla/5.0 (Unknown; Linux x86_64) AppleWebKit/534.34 (KHTML, like Gecko) PhantomJS/1.6.1 Safari/534.34
	Mozilla/5.0 (Macintosh; Intel Mac OS X) Word/14.0.0
	Mozilla/5.0 (Macintosh; Intel Mac OS X) Word/14.25.0
	Mozilla/5.0 (Macintosh; Intel Mac OS X) Word/14.47.0
	Mozilla/5.0 (Macintosh; Intel Mac OS X) Word/14.48.0
	Mozilla/5.0 (Macintosh; Intel Mac OS X) Word/14.50.0
	Mozilla/5.0 (Macintosh; Intel Mac OS X) Word/14.56.0
	Mozilla/5.0 (Macintosh; Intel Mac OS X) Word/14.61.0
	Mozilla/5.0 (Macintosh; Intel Mac OS X) Word/14.62.0
	Mozilla/5.0 (Macintosh; Intel Mac OS X) Word/14.63.0
	Mozilla/5.0 (Macintosh; Intel Mac OS X) Word/14.64.0
	WomlpeFactory/0.1 (+http://www.Womple.com/bot.html)
	Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/534.34 (KHTML, like Gecko) wkhtmltoimage Safari/534.34
	Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/534.34 (KHTML, like Gecko) wkhtmltopdf Safari/534.34
	Mozilla/4.0 (compatible; Win32; WinHttp.WinHttpRequest.5)
	Windows-RSS-Platform/2.0 (IE 11.0; Windows NT 10.0)
	WhatsApp
	WhatsApp/2.12.15/i
	WhatsApp/2.12.16/i
	WhatsApp/2.12.17/i
	WhatsApp/2.12.449 A
	WhatsApp/2.12.453 A
	WhatsApp/2.12.510 A
	WhatsApp/2.12.540 A
	WhatsApp/2.12.548 A
	WhatsApp/2.12.555 A
	WhatsApp/2.12.556 A
	WhatsApp/2.16.1/i
	WhatsApp/2.16.13 A
	WhatsApp/2.16.2/i
	WhatsApp/2.16.42 A
	WhatsApp/2.16.57 A
	WhatWeb/0.4.8-dev
	http://www.almaden.ibm.com/cs/crawler [wf84]
	WeLikeLinks - WeAreNotEvil
	Websquash.com (Add url robot)
	WeCrawlForThePeace - We are not Evil
	Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.21 (KHTML, like Gecko) webscreenie Safari/537.21
	Mozilla/3.0 (compatible; WebMon 1.0.11; Windows XP)
	Mozilla/5.0 (X11; BSD Four) AppleWebKit/534.34 (KHTML, like Gecko) webkit2png Safari/534.34
	WebFetch
	WebIndex
	webcollage/1.114
	webcollage/1.117
	webcollage/1.125
	webcollage/1.129
	webcollage/1.93
	Mozilla/3.0 (compatible; WebCapture 2.0; Auto; Windows)
	W3C_Unicorn/1.0
	W3C_I18n-Checker/1.0
	W3C-mobileOK/DDC-1.0
	W3C-checklink
	W3C-checklink/2.90 libwww-perl/5.64
	W3C-checklink/3.6.2.3 libwww-perl/5.64
	W3C-checklink/4.2 [4.20] libwww-perl/5.803
	W3C-checklink/4.2.1 [4.21] libwww-perl/5.803
	W3C-checklink/4.3 [4.42] libwww-perl/5.805
	W3C-checklink/4.3 [4.42] libwww-perl/5.808
	W3C-checklink/4.3 [4.42] libwww-perl/5.820
	W3C-checklink/4.5 [4.154] libwww-perl/5.823
	W3C-checklink/4.5 [4.160] libwww-perl/5.823
	w3af.org
	VYU2 (GNU; OpenRISC)
	VORTEX/1.2 ( http://marty.anstey.ca/robots/vortex/)
	Vortex/2.2 ( http://marty.anstey.ca/robots/vortex/)
	Vortex/2.2 (+http://marty.anstey.ca/robots/vortex/)
	Mozilla/5.0 (Windows NT 5.1; rv:11.0) Gecko Firefox/11.0 (via ggpht.com GoogleImageProxy)
	URLChecker
	urlresolver
	Upflow/1.0
	Typhoeus - https://github.com/typhoeus/typhoeus
	UdmSearch/3.1.x
	Mozilla/5.0 Tweetminster
	Mozilla/5.0 (compatible; +http://tweetedtimes.com)
	truwoGPS/1.0 (GNU/Linux; U; i686; en-US; +http://www.lan4lano.net/browser.html )
	Traackr.com
	More-Testing (+http://www.theinternetrules.com)
	The Expert HTML Source Viewer (http://www.expert-html.com)
	htmlayout 3.3; above-Windows-7; www.terrainformatica.com )
	WebFuck V2.1 T0PHackTeam www.t0p.xyz
	Symfony2 BrowserKit
	Mozilla/5.0 (compatible; SwiteScraper)
	Summify (Summify/1.0.1; +http://summify.com)
	Stratagems Kumo
	Sqworm/2.9.85-BETA (beta_release; 20011115-775; i686-pc-linux-gnu)
	Sprinklr 2.0
	spray-can/1.3.3
	SortSite/4.4.500.0
	Snoopy v2.0.0
	SMRF URL Expander
	silk/1.0 (+http://www.slider.com/silk.htm)/3.7
	SiteBar/3.3.5 (Bookmark Server; http://sitebar.org/)
	SiteBar/3.3.8 (Bookmark Server; http://sitebar.org/)
	SimplePie/1.3.1 (Feed Parser; http://simplepie.org; Allow like Gecko) Build/20160226060242
	ShortLinkTranslate
	Mozilla/5.0 (compatible; Seznam screenshot-generator 2.1; +http://fulltext.sblog.cz/screenshot/)
	SEOstats 2.0.9 https://github.com/eyecatchup/SEOstats
	semanticdiscovery/0.1
	SearchSight/2.0 (http://SearchSight.com/)
	ScoutURLMonitor/5.9.8
	Robosourcer/1.0
	Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; RetrevoPageAnalyzer; +http://www.retrevo.com/content/about-us)
	RestSharp/105.2.3.0
	Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_5) AppleWebKit/537.78.2 (KHTML, like Gecko) Version/7.0.1(2.5.3) Safari(ReederForMac)
	redback/v0-570-g26f8c96
	RebelMouse/0.1 Mozilla/5.0 (compatible; http://rebelmouse.com) Gecko/20100101 Firefox/7.0.1
	RealPlayer%20Downloader/1569 CFNetwork/720.3.13 Darwin/14.3.0 (x86_64)
	RealPlayer%20Downloader/1701 CFNetwork/760.5.1 Darwin/15.5.0 (x86_64)
	RealPlayer%20Downloader/1725 CFNetwork/720.3.13 Darwin/14.3.0 (x86_64)
	RealPlayer%20Downloader/1750 CFNetwork/720.5.7 Darwin/14.5.0 (x86_64)
	RealPlayer%20Downloader/1750 CFNetwork/760.0.5 Darwin/15.0.0 (x86_64)
	RealPlayer%20Downloader/1750 CFNetwork/760.4.2 Darwin/15.4.0 (x86_64)
	Qirina Hurdler v. 1.05 10.11.01 (+http://www.qirina.com/hurdler.html)
	python-requests/1.1.0 CPython/2.7.6 Linux/3.13.0-58-generic
	python-requests/1.2.0 CPython/2.7.3 Linux/3.2.0-29-virtual
	python-requests/2.1.0 CPython/2.7.3 Linux/3.2.0-4-amd64
	python-requests/2.10.0
	python-requests/2.2.1 CPython/2.7.6 Linux/3.13.0-48-generic
	python-requests/2.2.1 CPython/2.7.6 Linux/3.13.0-83-generic
	python-requests/2.3.0 CPython/2.7.6 Linux/3.13.0-57-generic
	python-requests/2.5.0 CPython/2.7.9 Linux/3.14.20-20.44.amzn1.x86_64
	python-requests/2.6.0 CPython/2.7.3 Linux/3.2.0-4-amd64
	python-requests/2.6.0 CPython/2.7.5 Linux/3.10.0-327.3.1.el7.x86_64
	python-requests/2.7.0 CPython/2.7.6 Linux/3.13.0-85-generic
	python-requests/2.8.0
	python-requests/2.8.1
	python-requests/2.9.1
	Python-httplib2/0.7.4 (gzip)
	Pulsepoint XT3 web scraper
	Mozilla/5.0 (compatible; ProspectB2B/2.0; http://prospectb2b.com)
	Mozilla/5.0 (Macintosh; Intel Mac OS X) PowerPoint/14.39.0
	PostRank/2.0 (postrank.com; 1 subscribers)
	PostRank/2.0 (postrank.com)
	postano
	Ploetz + Zeller (http://www.ploetz-zeller.de) Link Validator v1.0 ([email protected]) for ARIS Business Architect
	Pizilla++ ver 2.45
	phpcrawl
	Pearltrees
	Pcore-HTTP/v0.23.20
	Mozilla/5.0 (Windows; U; Windows NT 6.0; en-US; rv:1.9.0.4) Gecko/2008102920 http://ow.ly web crawler (.NET CLR 3.5.30729)
	Mozilla/5.0 (compatible; Optimizer)
	Mozilla/5.0 [en] (X11, U; OpenVAS 7.0.10)
	JetBrains Omea Reader 1.0.x (http://www.jetbrains.com/omea_reader/)
	JetBrains Omea Reader 2.0 Release Candidate 1 (http://www.jetbrains.com/omea_reader/)
	okhttp/2.5.0
	oegp v. 1.3.0
	Nuzzel
	Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/43.0.2357.134 Safari/537.36 http://notifyninja.com/monitoring
	Notifixious/LinkChecker (http://notifixio.us)
	http://www.nominet.org.uk/privacypolicy
	node-superagent/1.8.3
	node.io
	Kimengi/nineconnections.com
	newspaper/0.0.9.8
	newspaper/0.1.7
	newsme/1.0; [email protected]
	NewsGator/2.0 Bot (http://www.newsgator.com)
	NewsGator/2.5 (http://www.newsgator.com; Microsoft Windows NT 5.1.2600.0; .NET CLR 1.1.4322.2032)
	NetTrack Anonymous Web Statistics https://nettrack.info/support.php
	NetTrack.info Crawler
	NetResearchServer(http://www.look.com)
	NetResearchServer/2.5(loopimprovements.com/robot.html)
	NetResearchServer/2.7(loopimprovements.com/robot.html)
	NetResearchServer/2.8(loopimprovements.com/robot.html)
	NetResearchServer/3.5(loopimprovements.com/robot.html)
	NetResearchServer/4.0(loopimprovements.com/robot.html)
	NetLyzer FastProbe (See http://netlyzer.com/report/www.domain.com for info)
	check_http/v2.0.3 (nagios-plugins 2.0.3)
	MVAClient
	Link Validity Check From: http://www.w3dir.com/cgi-bin (Using: Hot Links SQL by Mrcgiguy.com)
	Mojolicious (Perl)
	mogimogi/1.0
	Mindjet
	Mindjet MindManager
	Microsoft Windows Network Diagnostics
	Microsoft Office Excel 2013
	Microsoft Office Excel 2014
	Microsoft Office Excel 2014 (16.0.6528) Windows NT 10.0
	Microsoft Office Existence Discovery
	Microsoft Office Mobile/15.0
	Microsoft Office OneNote 2013
	Microsoft Office PowerPoint 2013
	Microsoft Office PowerPoint 2013 (15.0.4420) Windows NT 6.2
	Microsoft Office PowerPoint 2013 (15.0.4641) Windows NT 6.2
	Microsoft Office PowerPoint 2013 (15.0.4797) Windows NT 10.0
	Microsoft Office PowerPoint 2013 (15.0.4797) Windows NT 6.2
	Microsoft Office PowerPoint 2013 (15.0.4805) Windows NT 6.1
	Microsoft Office PowerPoint 2013 (15.0.4805) Windows NT 6.2
	Microsoft Office PowerPoint 2013 (15.0.4815) Windows NT 10.0
	Microsoft Office PowerPoint 2014
	Microsoft Office Word 2013
	Microsoft Office Word 2013 (15.0.4623) Windows NT 6.2
	Microsoft Office Word 2013 (15.0.4701) Windows NT 6.1
	Microsoft Office Word 2013 (15.0.4797) Windows NT 10.0
	Microsoft Office Word 2013 (15.0.4805) Windows NT 10.0
	Microsoft Office Word 2013 (15.0.4805) Windows NT 6.1
	Microsoft Office Word 2013 (15.0.4823) Windows NT 10.0
	Microsoft Office Word 2014
	Melvil Rawi/1.0
	marketinggrader.com
	Mandrill-Webhook/1.0
	MailChimp.com
	MagpieRSS/0.72 (+http://magpierss.sf.net)
	http://www.mabontland.com
	lwp-trivial/1.33
	lwp-trivial/1.35
	lwp-trivial/1.36
	lwp-trivial/1.38
	lwp-trivial/1.41
	looksystems.net
	LinkExaminer/1.01 (Windows)
	VLC/2.2.4 LibVLC/2.2.4
	[email protected]
	Mozilla/5.0 (compatible; Jobrapido/1.1; +http://www.jobrapido.com)
	JoBo/1.x (http://www.matuschek.net/jobo.html)
	Jigsaw/2.3.0 W3C_CSS_Validator_JFouffa/2.0
	Mozilla/5.0 (compatible; IODC-Internet Engineering Odysseus 001; +http://iodc.co.uk)
	Mozilla/5.0 (compatible; IODC-Internet Engineering Odysseus 005; +http://iodc.co.uk)
	Mozilla/5.0 (compatible; IODC-Internet Engineering Odysseus 006; +http://iodc.co.uk)
	Mozilla/5.0 (compatible; IODC-Internet Engineering Odysseus 007; +http://iodc.co.uk)
	Mozilla/5.0 (compatible; IODC-Internet Engineering Odysseus 011; +http://iodc.co.uk)
	Mozilla/5.0 (compatible; IODC-Internet Engineering Odysseus 013; +http://iodc.co.uk)
	Mozilla/5.0 (compatible; IODC-Odysseus Survey 29691-100-070315021100-72; +http://iodc.co.uk)
	Mozilla/5.0 (compatible; IODC-Odysseus Survey 29692-100-060415034231-192; +http://iodc.co.uk)
	Mozilla/5.0 (compatible; IODC-Odysseus Survey 31193-100-080615161452-372; +http://iodc.co.uk)
	Mozilla/5.0 (compatible; IODC-Odysseus Survey 37044-100-110516191742-025; +https://iodc.co.uk)
	Mozilla/5.0 (compatible; IODC-Odysseus Survey 41437-100-090715214314-122; +http://iodc.co.uk)
	Integrity/6
	InfoWizards Reciprocal Link System PRO - (http://www.infowizards.com)
	InDesign%20CC/11300 CFNetwork/720.4.4 Darwin/14.4.0 (x86_64)
	inbound.li parser
	InAGist URL Resolver (http://inagist.com)
	igdeSpyder (compatible; igde.ru; +http://igde.ru/doc/tech.html)
	IDG Twitter Links Resolver
	Mozilla/4.5 (compatible; HTTrack 3.0x; Windows 98)
	Jersey/2.19 (HttpUrlConnection 1.7.0_85)
	httpunit/1.x
	httpssites_power/1 CFNetwork/758.3.15 Darwin/15.4.0
	httpscheck (unknown version) CFNetwork/760.5.1 Darwin/15.5.0 (x86_64)
	HttpComponents/1.1
	http_requester/0.1
	HTTP_Request2/@package_version@ (http://pear.php.net/package/http_request2) PHP/5.5.9-1ubuntu4.14
	HTTP_Request2/2.2.0 (http://pear.php.net/package/http_request2) PHP/5.3.29
	HTTP_Request2/2.2.1 (http://pear.php.net/package/http_request2) PHP/5.3.29
	HTTP_Request2/2.2.1 (http://pear.php.net/package/http_request2) PHP/5.5.33
	HTTP_Request2/2.2.1 (http://pear.php.net/package/http_request2) PHP/5.5.35
	HTTP_Request2/2.3.0 (http://pear.php.net/package/http_request2) PHP/5.3.29
	HTTP-Tiny/0.054
	http-kit/2.0
	HTMLParser/2.0
	htdig/3.1.5
	htdig/3.1.5 ([email protected])
	htdig/3.1.5 (root@localhost)
	htdig/3.1.5 ([email protected])
	htdig/3.1.6 ([email protected])
	htdig/3.1.6 ([email protected])
	ht://check/2.0.0-rc1 (Linux 2.6.32-573.18.1.el6.x86_64 x86_64)
	HEADMasterSEO
	Mozilla/4.0 (compatible; grub-client-0.3.0; Crawl your own stuff with http://grub.org)
	Mozilla/4.0 (compatible; grub-client-1.0.4; Crawl your own stuff with http://grub.org)
	Mozilla/4.0 (compatible; grub-client-1.0.5; Crawl your own stuff with http://grub.org)
	Mozilla/4.0 (compatible; grub-client-1.0.6; Crawl your own stuff with http://grub.org)
	Mozilla/4.0 (compatible; grub-client-1.0.7; Crawl your own stuff with http://grub.org)
	Mozilla/4.0 (compatible; grub-client-1.1.1; Crawl your own stuff with http://grub.org)
	Mozilla/4.0 (compatible; grub-client-1.2.1; Crawl your own stuff with http://grub.org)
	Mozilla/4.0 (compatible; grub-client-1.3.1; Crawl your own stuff with http://grub.org)
	Mozilla/4.0 (compatible; grub-client-1.3.7; Crawl your own stuff with http://grub.org)
	Mozilla/4.0 (compatible; grub-client-1.4.3; Crawl your own stuff with http://grub.org)
	Mozilla/4.0 (compatible; grub-client-1.5.3; Crawl your own stuff with http://grub.org)
	Mozilla/4.0 (compatible; grub-client-2.3)
	Mozilla/4.0 (compatible; grub-client-2.6.0)
	Mozilla/4.0 (compatible; grub-client-2.6.1)
	GoSpotCheck 1.0.6 (iPhone; iPhone OS 5.1.1; en_US)
	GoScraper
	GoogleProducer; (+http://goo.gl/7y4SX)
	GoogleHC/
	Google-HTTP-Java-Client/1.17.0-rc (gzip)
	Google-HTTP-Java-Client/1.9.0-alpha
	Mozilla/5.0 (compatible; Google-Apps-Script)
	Go-http-client/1.1
	Go 1.1 package http
	Mozilla/5.0 (Macintosh; Intel Mac OS X 10_6_6) AppleWebKit/534.24 (KHTML, like Gecko) (Contact: [email protected])
	Mozilla/5.0 (compatible; Funnelback) RPT-HTTPClient/0.3-3E
	Mozilla/5.0 (compatible; Snacktory; +https://flynxapp.com)
	Mozilla/3.0 (compatible; Fluffy the spider; http://www.searchhippo.com/; [email protected])
	Flamingo_SearchEngine (+http://www.flamingosearch.com/bot)
	Mozilla/4.0 (compatible; MSIE 5.5; Windows NT 5.0) Fetch API Request
	FeedChecker-Zocle/1.0 (+https://zocle.com/zoclechecker)
	FeedBurner/1.0 (http://www.FeedBurner.com)
	FavOrg
	Faraday v0.9.1
	Mozilla/5.0 (Macintosh; Intel Mac OS X) Excel/14.10.0
	Mozilla/5.0 (Macintosh; Intel Mac OS X) Excel/14.62.0
	ExactSearch
	EventMachine HttpClient
	Mozilla/5.0 (compatible; MSIE 7.0 +http://www.europarchive.org)
	Embed PHP Library
	ECCP/1.2.1+([email protected])
	ec2linkfinder
	Mozilla/5.0 (compatible; EARTHCOM/2.2; +http://enter4u.eu)
	Drupal (+http://drupal.org/)
	Dragonfly File Reader
	DNS-Tools Header-Analyzer
	Dispatch/0.11.1-SNAPSHOT
	[email protected]
	CyberPatrol SiteCat Webbot (http://www.cyberpatrol.com/cyberpatrolcrawler.asp)
	curb
	Mozilla/4.0 (compatible; cron-job.org; http://cron-job.org/abuse/)
	cosmos/0.9_([email protected])
	Mozilla/5.0/Firefox/42.0 (contactbigdatafr at gmail.com)
	Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; .NET CLR 2.0.50727; MAXTHON 2.0); Connect Us: [email protected]
	Pattern/2.6 +http://www.clips.ua.ac.be/pattern
	Mozilla/4.0 (compatible; Cerberian Drtrs Version-3.2-Build-0)
	Mozilla/4.0 (compatible; Cerberian Drtrs Version-3.2-Build-1)
	CC Metadata Scaper http://wiki.creativecommons.org/Metadata_Scraper
	cb crawl (+http://www.companybooknetworking.com)
	CapsuleChecker (http://www.capsulink.com/)
	CakePHP
	Branch-Passthrough
	Branch Metrics API
	boitho.com-dc/0.79 ( http://www.boitho.com/dcbot.html )
	boitho.com-dc/0.81 ( http://www.boitho.com/dcbot.html )
	boitho.com-dc/0.82 ( http://www.boitho.com/dcbot.html )
	boitho.com-dc/0.83 ( http://www.boitho.com/dcbot.html )
	boitho.com-dc/0.85 ( http://www.boitho.com/dcbot.html )
	boitho.com-robot/1.1
	Blackboard Safeassign
	Mozilla/5.0 (Windows NT 6.3; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/42.0.2311.152 Safari/537.36 (BingLocalSearch)
	BIGLOTRON (Beta 2;GNU/Linux)
	Mozilla/5.0 (compatible; bnf.fr_bot; +http://bibnum.bnf.fr/robot/bnf.html)
	Mozilla/4.0 (compatible; B-l-i-t-z-B-O-T)
	Astute SRM
	aria2/1.18.8
	Mozilla/5.0 (compatible; archive-com.com/1.1; +http://archive-com.com/bot)
	Mozilla/4.0 (compatible; Arachmo)
	Apache-HttpClient/4.2.3 (java 1.5)
	Apache-HttpClient/4.2.5 (java 1.5)
	Apache-HttpClient/4.3.1 (java 1.5)
	Apache-HttpClient/4.3.3 (java 1.5)
	Apache-HttpClient/4.3.5 (java 1.5)
	Apache-HttpClient/4.5 (Java/1.8.0_51)
	Apache-HttpClient/4.5 (Java/1.8.0_72)
	Apache-HttpClient/4.5 (Java/1.8.0_74)
	Apache-HttpClient/4.5.1 (Java/1.7.0_79)
	Apache-HttpClient/4.5.1 (Java/1.8.0_40)
	Apache-HttpClient/4.5.2 (Java/1.8.0_72)
	Apache-HttpClient/UNAVAILABLE (java 1.4)
	Mozilla/5.0 (compatible; U; AnyEvent-HTTP/2.22; +http://software.schmorp.de/pkg/AnyEvent)
	Mozilla/5.0 (compatible; U; AnyEvent-HTTP/2.23; +http://software.schmorp.de/pkg/AnyEvent)
	Anemone/0.7.2
	AndroidDownloadManager/6.0.1 (Linux; U; Android 6.0.1; SM-G900F Build/MMB29M)
	AndroidDownloadManager/4.4.2 (Linux; U; Android 4.4.2; SM-T310 Build/KOT49H)
	AndroidDownloadManager/4.4.2 (Linux; U; Android 4.4.2; SM-T210 Build/KOT49H)
	Airmail 1.3.3 rv:237 (Macintosh; Mac OS X 10.9.3; sk_SK)
	Python/3.5 aiohttp/0.21.6
	AHC/1.0
	Mozilla/4.0 (compatible; Advanced Email Extractor v2.61)
	Pulp/1.5.2 (iPad; http://www.acrylicapps.com/pulp/)
	Accoona-AI-Agent/1.1.1 (crawler at accoona dot com)
	Accoona-AI-Agent/1.1.2
	Accoona-AI-Agent/1.1.2 (aicrawler at accoonabot dot com)
	Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1) [email protected]
	scrutiny/6
	RMA/1.0 (compatible; RealMedia)
	PHP/5.5
	NING/1.0
	Mget/0.1.9
	Java/1.4.1_04
	Java/1.6.0
	Java/1.6.0_04
	Java/1.6.0_22
	Java/1.6.0_24
	Java/1.6.0_26
	Java/1.6.0_31
	Java/1.6.0_32
	Java/1.6.0_34
	Java/1.6.0_37
	Java/1.7.0_101
	Java/1.7.0_45
	Java/1.7.0_51
	Java/1.7.0_65
	Java/1.7.0_80
	Java/1.8.0_05
	Java/1.8.0_20
	Java/1.8.0_31
	Java/1.8.0_40
	Java/1.8.0_45
	Java/1.8.0_51
	Java/1.8.0_60
	Java/1.8.0_65
	Java/1.8.0_66
	Java/1.8.0_73
	Java/1.8.0_77
	Java/1.8.0_91
	Java/1.8.0_92
	Goose/1.0.25
	FDM 3.x
	bluefish 0.6 HTML editor
	2ip.ru CMS Detector (http://2ip.ru/cms/)
	masscan/1.0 (https://github.com/robertdavidgraham/masscan)
	VidibleScraper/1.0
	WhereGoes? Redirect Tracer/1.0
	unfurlist (https://github.com/Doist/unfurlist)
	pshtt, https scanning
	gofetch
	COMODO DCV
	CATExplorador/1.0beta (sistemes at domini dot cat; http://domini.cat/catexplorador.html)
	ZmEu
	RSSOwl/2.2.1.201312301314 (Windows; U; en)
	WebClient/1.0
	Microsoft-WebDAV-MiniRedir/10.0.10586
	Microsoft-WebDAV-MiniRedir/10.0.14393
	Microsoft-WebDAV-MiniRedir/5.1.2600
	Microsoft-WebDAV-MiniRedir/6.0.6001
	Microsoft-WebDAV-MiniRedir/6.1.7601
	Microsoft-WebDAV-MiniRedir/6.3.9600
	Microsoft Data Access Internet Publishing Provider Protocol Discovery
	Test Certificate Info
	Needle/1.4.2 (Node.js v6.9.2; linux x64)
	DavClnt
	Mozilla/5.0 (iPhone; CPU iPhone OS 6_0 like Mac OS X) AppleWebKit/536.26 (KHTML, like Gecko) Version/6.0 Mobile/10A5376e Safari/8536.25 mixed-content-scan
	Mozilla/5.0 (compatible; Semiocast HTTP client; http://semiocast.com/)
	Mozilla/5.0 (compatible; The Drop Reaper; +http://tag1consulting.com/dropreaper.html)
	/n software IPWorks HTTP/S Component - www.nsoftware.com
	safe-agent-scanner
	Mozilla/5.0 (compatible; ACHE/Unknown Version; +https://github.com/ViDA-NYU/ache; )
	ANAM (github.com/dutchcoders/anam)
	Dolphin http client/11.2.8(385) (Android)
	Dolphin http client/11.3.4(405) (Android)
	Calypso v/0.01
	-Mozilla/4.0 (compatible; http://search.thunderstone.com/texis/websearch/about.html)
	Mozilla/5.0 (compatible; Tweezler/2.0; http://tweezler.com)
	Jeode/1.x.x
	KeyCDN Perf Test
	Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) YLT Chrome/27.0.1453.110 Safari/537.36
	Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/51.0.2704.84 Safari/537.36 DMBrowser-UV
	Taringa UGC www.taringa.net/shared-link.php
	Mozilla/5.0 (Unknown; Linux x86_64) AppleWebKit/534.34 (KHTML, like Gecko) PingdomTMS/0.8.5 Safari/534.34
	ganarvisitas/1.0 (+http://www.ganarvisitas.com/)
	FeedFetcher-Google; (+http://www.google.com/feedfetcher.html)
	Mozilla/5.0 (compatible) Feedfetcher-Google; (+http://www.google.com/feedfetcher.html)
	Mozilla/5.0 (compatible; Exabot PyExalead/3.0; +http://www.exabot.com/go/robot)
	Mozilla/5.0 PhantomJS (compatible; Seznam screenshot-generator 2.1; +http://fulltext.sblog.cz/screenshot/)
	Mozilla/5.0 (compatible; SeznamBot/3.2-test4; +http://napoveda.seznam.cz/en/seznambot-intro/)
	Mozilla/5.0 (compatible; SeznamBot/3.2-test2; +http://napoveda.seznam.cz/en/seznambot-intro/)
	Mozilla/5.0 (compatible; SeznamBot/3.2-test1; +http://fulltext.sblog.cz/)
	Mozilla/5.0 (compatible; SeznamBot/3.2-test4; +http://fulltext.sblog.cz/)
	Mozilla/5.0 (compatible; SeznamBot/3.2; +http://fulltext.sblog.cz/)
	Seznam-Zbozi-robot/3.2.1
	SeznamBot/2.0 (+http://fulltext.sblog.cz/robot/)
	Seznam-Zbozi-robot/3.2.2
	Mozilla/5.0 (compatible; SeznamBot/3.2-test2; +http://fulltext.sblog.cz/)
	Mozilla/5.0 (compatible; SeznamBot/3.2-test3; +http://fulltext.sblog.cz/)
	Seznam-Zbozi-robot/3.3
	SklikBot/2.0 ([email protected];+http://napoveda.sklik.cz/)
	SeznamBot/3.0 (+http://fulltext.sblog.cz/)
	Mozilla/5.0 (compatible; SeznamBot/3.1-test1; +http://fulltext.sblog.cz/)
	Mozilla/5.0 (Linux; U; Android 4.1.2; cs-cz; Seznam screenshot-generator Build/Q3) AppleWebKit/534.30 (KHTML, like Gecko) Version/4.0 Mobile Safari/534.30
	SeznamBot/3.0-test (+http://fulltext.sblog.cz/), I
	SeznamBot/3.0 (HaF+http://fulltext.sblog.cz/)
	SeznamBot/3.0-test (+http://fulltext.sblog.cz/)
	SeznamBot/3.0-beta (+http://fulltext.sblog.cz/), I
	SeznamBot/3.0-beta (+http://fulltext.sblog.cz/)
	SeznamBot/3.0-alpha (+http://fulltext.sblog.cz/)
	SeznamBot/2.0 (+http://fulltext.seznam.cz/)
	SeznamBot/2.0-Test (+http://fulltext.sblog.cz/robot/)
	Mozilla/5.0 (compatible; Seznam screenshot-generator 2.0; +http://fulltext.sblog.cz/screenshot/)
	Mozilla/5.0 (compatible; FlipboardRSS/1.2; +http://flipboard.com/browserproxy)
	Mozilla/5.0 (Macintosh; Intel Mac OS X 10.11; rv:49.0) Gecko/20100101 Firefox/49.0 (FlipboardProxy/1.2; +http://flipboard.com/browserproxy)
	Mozilla/5.0 (compatible; MJ12bot/v1.4.7; http://mj12bot.com/)
	Mozilla/5.0 (Macintosh; Intel Mac OS X 10_7_2) AppleWebKit/535.7 (KHTML, like Gecko) Chrome/16.0.912.77 Safari/535.7 AppEngine-Google; (+http://code.google.com/appengine; appid: s~feedly-social)
	Mozilla/5.0 (Macintosh; Intel Mac OS X 10_7_2) AppleWebKit/535.7 (KHTML, like Gecko) Chrome/23.0.912.77 Safari/535.7 AppEngine-Google; (+http://code.google.com/appengine; appid: s~feedly-nikon3)
	Mozilla/5.0 (compatible; SemrushBot/1.2~bl; +http://www.semrush.com/bot.html)
	Mozilla/5.0 (compatible; YandexMetrika/2.0; +http://yandex.com/bots mtmon01it.mtrs.yandex.ru)
	Mozilla/5.0 (iPhone; CPU iPhone OS 8_1 like Mac OS X) AppleWebKit/600.1.4 (KHTML, like Gecko) Version/8.0 Mobile/12B411 Safari/600.1.4 (compatible; YandexBot/3.0; +http://yandex.com/bots)
	Mozilla/5.0 (compatible; memoryBot/1.24.61 +http://internetmemory.org/en/)
	Mozilla/5.0 (compatible; memoryBot/1.24.54 +http://internetmemory.org/en/)
	BUbiNG - Research at Brno University of Technology - KNOT group - http://knot.fit.vutbr.cz/crawling/ - Stop? http://law.di.unimi.it/BUbiNG.html
	Mozilla/5.0 (compatible; MSIE 9.0; Firefox/50.1) Daum/4.1
	Mozilla/5.0 (Unknown; Linux x86_64) AppleWebKit/534.34 (KHTML, like Gecko) PhantomJS/1.9.7 Daum Raam Edition b20140820 Safari/534.34
	Sogou Pic Spider/3.0(+http://www.sogou.com/docs/help/webmasters.htm#07)
	yacybot (/global; amd64 Linux 4.4.0-59-generic; java 1.8.0_121; Europe/de) http://yacy.net/bot.html
	yacybot (-global; amd64 Linux 4.4.0-57-generic; java 9-internal; Europe/en) http://yacy.net/bot.html
	yacybot (/global; amd64 Windows 8.1 6.3; java 1.8.0_111; Europe/de) http://yacy.net/bot.html
	yacybot (/global; amd64 Linux 3.16.0-4-amd64; java 1.8.0_111; Europe/en) http://yacy.net/bot.html
	yacybot (/global; amd64 FreeBSD 10.3-RELEASE-p7; java 1.7.0_95; GMT/en) http://yacy.net/bot.html
	yacybot (webportal-global; amd64 Linux 4.4.0-46-generic; java 1.8.0_91; Europe/en) http://yacy.net/bot.html
	yacybot (-global; amd64 Linux 3.14.32-xxxx-grs-ipv6-64; java 1.8.0_111; Europe/de) http://yacy.net/bot.html
	Mozilla/5.0 (compatible; special_archiver/3.3.0 [email protected] +https://archive.org/details/archive.org_bot)
	Mozilla/5.0 (compatible; heritrix/3.3.0-SNAPSHOT-20140702-2247 +http://archive.org/details/archive.org_bot)
	Mozilla/5.0 (compatible; archive.org_bot/heritrix-1.15.4 +http://www.archive.org)
	SeobilityBot (SEO-Check; http://bit.ly/1dJuuzs)
	Testomatobot/1.0 (Linux x86_64; +http://www.testomato.com/testomatobot) minicrawler/5.0.4
	Testomatobot/1.0 (Linux x86_64; +http://www.testomato.com/testomatobot) minicrawler/5.0.3
	Testomatobot/1.0 (Linux x86_64; +http://www.testomato.com/testomatobot) minicrawler/5.0.2
	Testomatobot/1.0 (Linux x86_64; +http://www.testomato.com/testomatobot) minicrawler/5.0.1
	Mozilla/5.0 (compatible; AhrefsBot/5.2; +http://ahrefs.com/robot/)
	Heurekabot-Feed/1.0 (+https://sluzby.heureka.cz/napoveda/heurekabot/)
	Heurekabot-ImageFullText/1.0 (+https://sluzby.heureka.cz/napoveda/heurekabot/)
	Heurekabot-ImageFullText/1.0 (+http://sluzby.heureka.cz/napoveda/heurekabot/)
	Mozilla/5.0 (Windows NT 6.2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/31.0.1650.63 Safari/537.36 QIHU 360SE; 360Spider
	ZoomBot (Linkbot 1.0 http://suite.seozoom.it/bot.html)
	Mozilla/5.0 (compatible; SOLOFIELD/1.0 +http://solofield.net/bot.html)
	Qwantify/1.0
	Mozilla/5.0 (compatible; Yeti/1.1; +http://naver.me/bot)
	Scrapy/1.3.0 (+http://scrapy.org)
	Scrapy/1.0.5.post4+g4b324a8 (+http://scrapy.org)
	Scrapy/1.2.1 (+http://scrapy.org)
	Scrapy/0.16.5 (+http://scrapy.org)
	Mozilla/5.0 (compatible; Dataprovider.com;)
	Mozilla/5.0 (compatible; Dataprovider; https://www.dataprovider.com/spider/)
	Mozilla/5.0 (compatible; SiteExplorer/1.1b; +http://siteexplorer.info/Backlink-Checker-Spider/)
	ICC-Crawler/2.0 (Mozilla-compatible; ; http://ucri.nict.go.jp/en/icccrawler.html)
	DuckDuckBot/1.0; (+http://duckduckgo.com/duckduckbot.html)
	Mozilla/5.0 (compatible; SMTBot/1.0; http://www.similartech.com/smtbot)
	Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/538.1 (KHTML, like Gecko) Chromium/31.0.1650.63 Site-Shot/2.1 (http://www.site-shot.com/) Safari/538.1
	Mozilla/5.0 (Windows NT 6.1; WOW64; Neustar WPM) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/48.0.2564.82 Safari/537.36
	Mozilla/5.0 (iPad; CPU OS 6_1_2 like Mac OS X; Neustar WPM) AppleWebKit/536.26 (KHTML, like Gecko) Version/6.0 Mobile/10A5376e Safari/8536.25
	Mozilla/5.0 (Windows NT 6.0; Neustar WPM) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/48.0.2564.82 Safari/537.36
	Mozilla/5.0 (Windows NT 6.1; Neustar WPM) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/48.0.2564.82 Safari/537.36
	Mozilla/5.0 (Windows NT 6.0; rv:38.0; Neustar WPM) Gecko/20100101 Firefox/38.0
	Mozilla/5.0 (Windows NT 6.1; rv:38.0; Neustar WPM) Gecko/20100101 Firefox/38.0
	Mozilla/5.0 (Windows NT 6.1; Neustar WPM) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/52.0.2743.82 Safari/537.36
	Mozilla/5.0 (Linux; Android 4.1.1; HTC6435LVW Build/JRO03C; Neustar WPM) AppleWebKit/537.22 (KHTML, like Gecko) Chrome/25.0.1364.169 Mobile Safari/537.22
	Mozilla/5.0 (compatible; Whoiswebsitebot/0.1; +http://www.whoiswebsite.net)
	Mozilla/5.0 (Linux; Android 5.0.2; SM-G920T Build/LRX22G) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/54.0.2840 Mobile Safari/537.36 DareBoost
	Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/54.0.2840 Safari/537.36 DareBoost
	Mozilla/5.0 (Linux; Android 5.0; LG-D855 Build/LRX21R.A1422018487) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/54.0.2840 Mobile Safari/537.36 DareBoost
	Mozilla/5.0 (iPhone; CPU iPhone OS 10_0_1 like Mac OS X) AppleWebKit/602.1.50 (KHTML, like Gecko) Version/10.0 Mobile/14A403 Safari/602.1 DareBoost
	Mozilla/5.0 (Linux; Android 4.4.3; HTC One Build/KTU84L) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/54.0.2840 Mobile Safari/537.36 DareBoost
	Mozilla/5.0 (Linux; Android 4.4; Nexus 5 Build/KRT16M) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/54.0.2840 Mobile Safari/537.36 DareBoost
	HubSpot Website Grader ([email protected])
	HubSpot Marketing Grader, HubSpot Marketing Grader
	com.hubspot.spooks.core.UserAgent@753535c0, HubSpot Marketing Grader
	Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/51.0.2704.79 Safari/537.36 (http://www.shrinktheweb.com)
	scrapy-redis (+https://github.com/rolando/scrapy-redis)
	Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/55.0.2883.87 Safari/537.36 PTST/348
	Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/54.0.2840.99 Safari/537.36 PTST/345
	TeamSpeak3-ImageFetcher-1.0
	wf_crawler (http://www.websitefigures.com)
	wf_crawler (http://websitefigures.com)
	Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/55.0.2883.87 Safari/537.36 TinEye/1.0 (via http://www.tineye.com/)
	TinEye-bot/0.61 (see http://www.tineye.com/crawler.html)
	Mozilla/5.0 (compatible; Googlebot/2.1; https://deepcrawl.com/bot)
	Googlebot deepcrawl
	Mozilla/5.0 (compatible; evc-batch/2.0.20170110125844)
	Mozilla/5.0 (compatible; evc-batch/2.0.20161220184139)
	Mozilla/5.0 (compatible; evc-batch/2.0.20161122223033)
	Mozilla/5.0 (compatible; evc-batch/2.0.20161026010401)
	Mozilla/5.0 (X11; Linux x86_64; rv:49.0; GTmetrix https://gtmetrix.com/) Gecko/20100101 Firefox/49.0
	Mozilla/5.0 (Linux; Android 4.3; Galaxy Nexus Build/JWR66Y; GTmetrix https://gtmetrix.com/) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/54.0.2840.68 Mobile Safari/537.36
	Mozilla/5.0 (X11; Linux x86_64; GTmetrix https://gtmetrix.com/) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/54.0.2840.71 Safari/537.36
	Mediumbot-MetaTagFetcher/0.3 (+https://medium.com/)
	Mediumbot-ProxyFetcher/0.1 (+https://medium.com/)
	YOOBI.de - WebBot 1.0
	YOOBI - WebBot 1.0
	Mozilla/5.0 (compatible; heritrix/3.3.0-SNAPSHOT-20160309-0050; UniLeipzigASV +http://corpora.informatik.uni-leipzig.de/crawler_faq.html)
	IAS crawler (page scorer; http://integralads.com/site-indexing-policy/)
	Mozilla/5.0 (compatible; JobKeresoBot; +https://www.kozvetlen-allasok.hu/help.jsp; [email protected]; 5.0
	Mozilla/5.0 (compatible; JobKereso; +https://www.kozvetlen-allasok.hu/robot.jsp [email protected])
	pyspider/0.3.9 (+http://pyspider.org/)
	Mozilla/5.0 (X11; U; Linux x86_64; de-DE; rv:1.9.0.19) Gecko/2010120923 ThumbShotsBot (KFSW 3.0.6-3)
	Mozilla/5.0 (compatible; Discordbot/1.0; +https://discordapp.com)
	Mozilla/5.0 (compatible; Dead Link Checker; http://www.dead-link-checker.com/)
	Mozilla/5.0 (compatible; MSIE 10.0; Windows NT 6.2; WOW64; Trident/6.0) CrawlerProcess (http://www.PowerMapper.com) /5.24.776.0
	Mozilla/4.0 (compatible; BOTW Spider; +http://botw.org)
	TwengaBot-2.0
	Iframely/1.0.0 (+https://iframely.com/;)
	Iframely/0.8.5 (+http://iframely.com/;)
	rogerbot/1.2 (http://moz.com/help/pro/what-is-rogerbot-, [email protected])
	rogerbot/1.0 (http://moz.com/help/pro/what-is-rogerbot-, [email protected]
	Mozilla/5.0 (compatible; Sitemap Generator/1.3; http://www.check-domains.com/sitemap/index.php) Gecko Check-domains)
	mindUpBot
	Mozilla/5.0 (Windows NT 6.1; compatible; BDCbot/1.0; +http://bigweb.bigdatacorp.com.br/faq.aspx) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2272.118 Safari/537.36
	Contacts Crawler (+http://www.scrapinghub.com)
	TA SEO Crawler v0.1 URLData.pm ([email protected])
	Mozilla/5.0 (compatible; scanbot/1.0; +http://dazzlepod.com/ip/)
	ArielisBot/1.1 (+http://arielis.com/
	RedesScrapy/0.24.1 (+http://g2pi.tsc.uc3m.es/es)
	Blackboard Safeassign/0.1 (a Storm-based Blackboard Safeassign web-crawler; https://github.com/DigitalPebble/storm-crawler; [email protected])
	Mozilla/5.0 (Macintosh; Intel Mac OS X 10.11; rv:48.0) Gecko/20100101 Firefox/48.0 glindahl-cocrawler/0.01 (+http://www.pbm.com/~lindahl/glindahl-cocrawler.html)
	botrobin/Nutch-1.13-SNAPSHOT (http://smarter.codes/bot-robin/; [email protected])
	Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/51.0.2704.63 Safari/537.36 (compatible; aylienbot/0.2; +http://www.aylien.com/bot.html)
	GloomarBot/1.1 (https://www.gloomar.com/bot)
	Hatena::Bookmark/2.00 (Hatena::Bookmark; master;)
	Opendi Screenshot Bot [email protected]
	Mozilla/5.0 (compatible; YaK/1.0; http://linkfluence.com/; [email protected])
	Mozilla/5.0 (compatible; KAZ.KZ_Bot/3.0)
	BuckyOHare/1.3 (Googlebot/2.1; +https://hypefactors.com/webcrawler)
	uipbot/1.0 ([email protected])
	gMozilla/5.0 (compatible; Infohelfer/1.4.3; +http://www.infohelfer.de/crawler.php)
	Mozilla/5.0 (compatible; lincobot/0.3; http://www.linksco.com/about/robot.html) AppEngine-Google; (+http://code.google.com/appengine)
	Mozilla/5.0 (compatible; MSIE 10.0; Windows NT 6.2; WOW64; Trident/6.0) MapperCmd/5.24.780.0
	Seeker v.1 http://LookSeek.com/seeker/
	AppManager RPT-HTTPClient/0.3-3E
	LinkAlarm/3.1
	Mozilla/5.0 (X11; U; Linux i686) Firefox/2.0.0.11 wprecon.com survey
	Jeffrey's Exif Viewer (http://regex.info/exif)
	Untiny
	Mozilla/5.0 (Grabify.link - https://grabify.link/expander)
	biNu image cacher ([email protected])
	Mozilla/5.0 (compatible; Google-Youtube-Links)
	ping.blo.gs/2.0
	Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; QQDownload 1.7; .NET CLR 2.0.50727)
	Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.1; Trident/4.0; QQDownload 663; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; .NET4.0C; .NET CLR 1.1.4322)
	Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 5.1; Trident/4.0; QQDownload 760; .NET CLR 2.0.50727; .NET CLR 3.0.4506.2152; .NET CLR 3.5.30729; InfoPath.2; .NET4.0E; .NET4.0C; .NET CLR 1.1.4322)
	Mozilla/5.0 (X11; Linux x86_64; rv:52.0.2) Gecko/20100101 Firefox/52.0.2 Ghost Inspector
	Mozilla/5.0 (X11; Linux x86_64; rv:52.0) Gecko/20100101 Firefox/52.0 Ghost Inspector
	Mozilla/5.0 (X11; Linux x86_64; rv:50.1.0) Gecko/20100101 Firefox/50.1.0 Ghost Inspector
	Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/538.1 (KHTML, like Gecko) QtWebEngine/5.3.0 Safari/538.1 Ghost Inspector
	Mozilla/5.0 (X11; Linux x86_64; rv:45.0.1) Gecko/20100101 Firefox/45.0.1 Ghost Inspector
	Mozilla/5.0 (X11; Linux x86_64; rv:46.0.1) Gecko/20100101 Firefox/46.0.1 Ghost Inspector
	Mozilla/5.0 (X11; Linux x86_64; rv:47.0.2) Gecko/20100101 Firefox/47.0.2 Ghost Inspector
	Mozilla/5.0 (X11; Linux x86_64; rv:48.0.2) Gecko/20100101 Firefox/48.0.2 Ghost Inspector
	Mozilla/5.0 (X11; Linux x86_64; rv:49.0.2) Gecko/20100101 Firefox/49.0.2 Ghost Inspector
	Netpursual/1.0
	Kaspersky Lab CFR link resolver [email protected]
	Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/27.0.1453 Safari/537.36 (compatible; Google-HotelAdsVerifier/2.0)
	b0t
	"echocrawl 2.0"
	masscan/1.0
	Mozilla/5.0 (iPhone; CPU iPhone OS 8_1 like Mac OS X) AppleWebKit/600.1.4 (KHTML, like Gecko) Version/8.0 Mobile/12B411 Safari/600.1.4 (compatible; YandexBot/3.0; +http://yandex.com/bots)
	Mozilla/5.0 (compatible; YandexAccessibilityBot/3.0; +http://yandex.com/bots)
	Mozilla/5.0 (iPhone; CPU iPhone OS 8_1 like Mac OS X) AppleWebKit/600.1.4 (KHTML, like Gecko) Version/8.0 Mobile/12B411 Safari/600.1.4 (compatible; YandexMobileBot/3.0; +http://yandex.com/bots)
	Mozilla/5.0 (compatible; YandexDirectDyn/1.0; +http://yandex.com/bots
	Mozilla/5.0 (compatible; YandexImages/3.0; +http://yandex.com/bots)
	Mozilla/5.0 (compatible; YandexVideo/3.0; +http://yandex.com/bots)
	Mozilla/5.0 (compatible; YandexMedia/3.0; +http://yandex.com/bots)
	Mozilla/5.0 (compatible; YandexBlogs/0.99; robot; +http://yandex.com/bots)
	Mozilla/5.0 (compatible; YandexFavicons/1.0; +http://yandex.com/bots)
	Mozilla/5.0 (compatible; YandexWebmaster/2.0; +http://yandex.com/bots)
	Mozilla/5.0 (compatible; YandexPagechecker/1.0; +http://yandex.com/bots)
	Mozilla/5.0 (compatible; YandexImageResizer/2.0; +http://yandex.com/bots)
	Mozilla/5.0 (compatible; YaDirectFetcher/1.0; Dyatel; +http://yandex.com/bots)
	Mozilla/5.0 (compatible; YandexCalendar/1.0; +http://yandex.com/bots)
	Mozilla/5.0 (compatible; YandexSitelinks; Dyatel; +http://yandex.com/bots)
	Mozilla/5.0 (compatible; YandexMetrika/3.0; +http://yandex.com/bots)
	Mozilla/5.0 (compatible; YandexAntivirus/2.0; +http://yandex.com/bots)
	Mozilla/5.0 (compatible; YandexVertis/3.0; +http://yandex.com/bots)
	Mozilla/5.0 (compatible; YandexBot/3.0; MirrorDetector; +http://yandex.com/bots)
	jetmon/1.0 (Jetpack Site Uptime Monitor by WordPress.com)
	Mozilla/5.0 (compatible; Abonti/0.8 - http://www.abonti.com)
	CloudEndure Scanner ([email protected])
	AFB/3.0 (+http://allloadin.com)
	TLSProbe/1.0 (+https://scan.trustnet.venafi.com/)
	Sens.ai http://sovereign.ai/contact
	eContext/1.0 (eContext Classification Engine)
	Owler
	Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Slack/1.2.6 Chrome/45.0.2454.85 AtomShell/0.34.3 Safari/537.36 Slack_SSB/1.2.6
	Mozilla/5.0 (compatible; ChangesMeter/1.9.1; http://intuiware.com/apps/changes-meter)
	Mozilla/5.0 (compatible; Gookey.co/1.0; +http://gookey.co/)
	TrapitAgent/0.1 (feed processor; +http://trapit.com/about)
	sqlmap/1.0-dev-nongit-201612050a8c (http://sqlmap.org)
	UniversalFeedParser/3.3 +http://feedparser.org/
	TulipChain/5.xx (http://ostermiller.org/tulipchain/) Java/1.x.1_0x (http://apple.com/) Mac_OS_X/10.2.8
	scalaj-http/1.0
	HggH PhantomJS Screenshoter
	fasthttp
	fasthttp, Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/53.0.2785.101 Safari/537.36
	lua-resty-http/0.10 (Lua) ngx_lua/10000
	GIS-LABS:CertMon
	GuzzleHttp/6.2.1 PHP/7.0.15-0ubuntu0.16.04.4
	http-request/v0.7.0 (http://git.io/tl_S2w) node.js/v0.10.29
	intraVnews/1.x
	MailChimp
	G-i-g-a-b-o-t
	Symfony BrowserKit
	VCPP71 http-get 1.0a
	Jaunt/1.2
	Mozilla/5.0 (compatible; Owler/0.4; +; )
	MovableType/x.x
	OMSC
	websitepulse agent/0.1 (compatible; MSIE 5.5; Netscape 4.75; Linux)
	Mozilla/5.0 (compatible; Uptimia; www.uptimia.com)
	ssl-tools.net
	Mozilla/5.0 (compatible; MSIE 10.0; Windows NT 6.1; Trident/6.0; .NET CLR 1.1.4322; .NET CLR 1.0.3705; .NET CLR 2.0.50727; .NET CLR 3.0.04506.648; .NET CLR 3.5.21022; .NET CLR 3.0.4506.2152; .NET CLR 3.5.30729) DMBrowser/2.1
	keycdn-tools/perf
	keycdn-tools/br
	keycdn-tools/h2
	keycdn-tools/perf
	keycdn-tools/br
	keycdn-tools/h2
	KOCMOHABT (+https://kozmonavt.tk/) Mozilla/5.0 (compatible; Web Explorer 2)
	Donuts Content Explorer (www.donuts.domains)
	Netsparker
	Mozilla/5.0+(compatible; MxToolbox/Beta7; http://www.mxtoolbox.com/)
	Apache-HttpAsyncClient/4.1.3 (Java/1.8.0_121)
	Tenon.io Automated Web Accessibility Tester
	ApacheBench/2.3
	Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; Clarsentia)
	Mozilla/5.0 (compatible; historious/2.0; +http://historio.us/)
	Mozilla/5.0 (compatible; uCrawlr/1.0 ; +https://blog.ucoz.ru/upolicy)
	Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)
	Brodie/1.0 (+http://www.15miles.com/)
	Jobg8 URL Monitor
	Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.21 (KHTML, like Gecko) wpif Safari/537.21
	Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.21 (KHTML, like Gecko) imgsizer Safari/537.21
	Mozilla/5.0 (compatible; AppBeat/1.0; +http://www.appbeat.io/)
	Mozilla/5.0 (compatible; Chirp/1.0; +http://www.binarycanary.com)
	Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; WOW64; Trident/5.0; SiteMonitor)
	Mixnode/3.0.1 (+https://www.mixnode.com/; Speed analysis; no data is stored)
	Pingability.com Monitoring Service

Тоже, но с учетом регулярных выражений:

  Скрыть контент

.*Java.*outbrain

008\/

192\.comAgent

2ip\.ru

404checker

^bluefish

^Calypso v\/

^COMODO DCV

^DangDang

^DavClnt

^FDM

^git\/

^Goose\/

^HTTPClient\/

^Java\/

^Jeode\/

^Jetty\/

^Mget

^Microsoft URL Control

^NG\/[0-9\.]

^NING\/

^PHP\/[0-9]

^RMA\/

^Ruby|Ruby\/[0-9]

^scrutiny\/

^VSE\/[0-9]

^WordPress\.com

^XRL\/[0-9]

^ZmEu

a3logics\.in

A6-Indexer

a\.pr-cy\.ru

Abonti\/

Aboundex

aboutthedomain

Accoona-AI-Agent

acoon

acrylicapps\.com\/pulp

adbeat

AddThis

ADmantX

adressendeutschland

Advanced Email Extractor v

agentslug

AHC

aihit

aiohttp\/

Airmail

akula\/

alertra

alexa site audit

Alibaba\.Security\.Heimdall

allloadin\.com

alyze\.info

amagit

AndroidDownloadManager

Anemone

Ant\.com

Anturis Agent

AnyEvent-HTTP\/

ApacheBench\/

Apache-HttpAsyncClient\/

Apache-HttpClient\/

AportWorm\/[0-9]

AppBeat\/[0-9]

AppEngine-Google

Arachmo

arachnode

Arachnophilia

aria2

asafaweb.com

AskQuickly

Astute

asynchttp

autocite

Autonomy

B-l-i-t-z-B-O-T

^b0t$

Backlink-Ceck\.de

Bad-Neighborhood

baidu\.com

baypup\/[0-9]

baypup\/colbert

BazQux

BCKLINKS

BDFetch

BegunAdvertising\/

BigBozz

biglotron

BingLocalSearch

BingPreview

binlar

biNu image cacher

biz_Directory

Blackboard Safeassign

Bloglovin

BlogPulseLive

BlogSearch

Blogtrottr

boitho\.com-dc

BPImageWalker

Braintree-Webhooks

Branch Metrics API

Branch-Passthrough

Brodie\/

Browsershots

BUbiNG

Butterfly\/

BuzzSumo

CAAM\/[0-9]

CakePHP

CapsuleChecker

CaretNail

catexplorador

cb crawl

CC Metadata Scaper

Cerberian Drtrs

CERT\.at-Statistics-Survey

cg-eye

changedetection

ChangesMeter\/

Charlotte

CheckHost

checkprivacy

chkme\.com

Chirp\/[0-9]

CirrusExplorer\/

CISPA Vulnerability Notification

CJNetworkQuality

Clarsentia

clips\.ua\.ac\.be

Cloud mapping experiment

CloudEndure

CloudFlare-AlwaysOnline

Cloudinary\/[0-9]

cmcm\.com

coccoc

CommaFeed

Commons-HttpClient

Comodo SSL Checker

contactbigdatafr

convera

copyright sheriff

Covario-IDS

CrawlForMe\/[0-9]

cron-job\.org

Crowsnest

curb

Curious George

curl

cuwhois\/[0-9]

cybo\.com

DareBoost

DataparkSearch

dataprovider

Daum(oa)?[ \/][0-9]

DeuSu

developers\.google\.com\/\+\/web\/snippet\/

Digg

Dispatch\/

dlvr

DMBrowser

DNS-Tools Header-Analyzer

DNSPod-reporting

docoloc

Dolphin http client\/

DomainAppender

Donuts Content Explorer

dotSemantic

downforeveryoneorjustme

downnotifier\.com

DowntimeDetector

Dragonfly File Reader

drupact

Drupal \(\+http:\/\/drupal\.org\/\)

dubaiindex

EARTHCOM

Easy-Thumb

ec2linkfinder

eCairn-Grabber

ECCP

echocrawl

eContext\/

ElectricMonk

elefent

EMail Exractor

EmailWolf

Embed PHP Library

Embedly

europarchive\.org

evc-batch\/[0-9]

EventMachine HttpClient

Evidon

Evrinid

ExactSearch

ExaleadCloudview

Excel\/

Exif Viewer

Exploratodo

ezooms

facebookexternalhit

facebookplatform

fairshare

Faraday v

fasthttp

Faveeo

Favicon downloader

FavOrg

Feed Wrangler

Feedbin

FeedBooster

FeedBucket

FeedBunch\/[0-9]

FeedBurner

FeedChecker

Feedly

Feedspot

Feedwind\/[0-9]

feeltiptop

Fetch API

Fetch\/[0-9]

Fever\/[0-9]

findlink

findthatfile

FlipboardBrowserProxy

FlipboardProxy

FlipboardRSS

fluffy

flynxapp

forensiq

FoundSeoTool\/[0-9]

free thumbnails

FreeWebMonitoring SiteChecker

Funnelback

G-i-g-a-b-o-t

g00g1e\.net

GAChecker

ganarvisitas\/[0-9]

geek-tools

Genderanalyzer

Genieo

GentleSource

GetLinkInfo

getprismatic\.com

GetURLInfo\/[0-9]

Ghost Inspector

GigablastOpenSource

GIS-LABS

github\.com\/

Go [\d\.]* package http

Go-http-client

gofetch

GomezAgent

gooblog

Goodzer\/[0-9]

Google favicon

Google Keyword Suggestion

Google Keyword Tool

Google Page Speed Insights

Google PP Default

Google Search Console

Google Web Preview

Google-Adwords

Google-Apps-Script

Google-Calendar-Importer

Google-HotelAdsVerifier

Google-HTTP-Java-Client

Google-Publisher-Plugin

Google-SearchByImage

Google-Site-Verification

Google-Structured-Data-Testing-Tool

Google-Youtube-Links

google_partner_monitoring

GoogleDocs

GoogleHC\/

GoogleProducer

Gookey

GoScraper

GoSpotCheck

GoSquared-Status-Checker

gosquared-thumbnailer

grabify

Grammarly

grouphigh

grub-client

GTmetrix

GuzzleHttp

gvfs\/

HAA(A)?RTLAND http client

Hatena

hawkReader

HEADMasterSEO

HeartRails_Capture

heritrix

historious\/

hledejLevne\.cz\/[0-9]

Holmes

HootSuite Image proxy

Hootsuite-WebFeed\/[0-9]

HostTracker

ht:\/\/check

htdig

HTMLParser\/

http-get

HTTP-Header-Abfrage

http-kit

http-request\/

HTTP-Tiny

HTTP_Compression_Test

http_request2

http_requester

HttpComponents

httphr

HTTPMon

PEAR HTTPRequest

httpscheck

httpssites_power

httpunit

HttpUrlConnection

httrack

hosterstats

huaweisymantec

HubPages.*crawlingpolicy

HubSpot Connect

HubSpot Marketing Grader

HyperZbozi.cz Feeder

i2kconnect\/

ichiro

IdeelaborPlagiaat

IDG Twitter Links Resolver

IDwhois\/[0-9]

Iframely

igdeSpyder

IlTrovatore

ImageEngine\/

Imagga

imgsizer

InAGist

inbound\.li parser

InDesign%20CC

infegy

infohelfer

InfoWizards Reciprocal Link System PRO

Instapaper

inpwrd\.com

Integrity

integromedb

internet_archive

InternetSeer

internetVista monitor

intraVnews

IODC

IOI

iplabel

IPS\/[0-9]

ips-agent

IPWorks HTTP\/S Component

iqdb\/

Irokez

isitup\.org

iskanie

iZSearch

janforman

Jaunt\/

Jigsaw

Jobboerse

Jobg8 URL Monitor

jobo

Jobrapido

JS-Kit

Kaspersky Lab CFR link resolver

KeepRight OpenStreetMap Checker

KeyCDN

Keywords Research

KickFire

KimonoLabs\/

Kml-Google

knows\.is

KOCMOHABT

kouio

kulturarw3

KumKie

L\.webis

Larbin

LayeredExtractor

letsencrypt

LibVLC

libwww

Licorne Image Snapshot

Liferea\/

link checker

Link Valet

link_thumbnailer

LinkAlarm\/

linkCheck

linkdex

LinkExaminer

linkfluence

linkpeek

LinkTiger

LinkWalker

Lipperhey

livedoor ScreenShot

LoadImpactPageAnalyzer

LoadImpactRload

LongURL API

looksystems\.net

ltx71

lua-resty-http

lwp-trivial

lycos

LYT\.SR

mabontland

MagpieRSS

Mail.Ru

MailChimp

Mandrill

MapperCmd

marketinggrader

masscan\/[0-9]

Mediapartners-Google

MegaIndex\.ru

Melvil Rawi\/

MergeFlow-PageReader

Metaspinner

MetaURI

Microsearch

Microsoft-WebDAV-MiniRedir

Microsoft Data Access Internet Publishing Provider Protocol

Microsoft Office

Microsoft Windows Network Diagnostics

Mindjet

Miniflux

mixdata dot com

mixed-content-scan

mixnode

Mnogosearch

mogimogi

Mojolicious \(Perl\)

monitis

Monitority\/[0-9]

montastic

MonTools

Moreover

Morning Paper

mowser

MovableType

Mrcgiguy

mShots

MxToolbox\/

MVAClient

nagios

Najdi\.si\/

Needle\/

NETCRAFT

NetLyzer FastProbe

Netpursual

netresearch

NetShelter ContentScan

Netsparker

NetTrack

Netvibes

Neustar WPM

NeutrinoAPI

NewsBlur .*Finder

NewsGator

newsme

newspaper\/

NG-Search

nineconnections\.com

NLNZ_IAHarvester

Nmap Scripting Engine

node-superagent

node\.io

nominet\.org\.uk

Norton-Safeweb

Notifixious

notifyninja

nuhk

nutch

Nuzzel

nWormFeedFinder

Nymesis

Ocelli\/[0-9]

oegp

okhttp

Omea Reader

omgili

OMSC

Online Domain Tools

OpenCalaisSemanticProxy

Openstat\/

OpenVAS

Optimizer

Orbiter

OrgProbe\/[0-9]

Owler

ow\.ly

ownCloud News

OxfordCloudService\/[0-9]

Page Analyzer

Page Valet

page2rss

page_verifier

PagePeeker

Pagespeed\/[0-9]

Panopta

panscient

parsijoo

PayPal IPN

Pcore-HTTP

Pearltrees

peerindex

Peew

PhantomJS\/

PhantomJS Screenshoter

Photon\/

phpcrawl

phpservermon

Pi-Monster

ping\.blo\.gs\/

Pingability

Pingdom

Pingoscope

PingSpot

pinterest\.com

Pizilla

Ploetz \+ Zeller

Plukkie

PocketParser

Pompos

Porkbun

Port Monitor

postano

PostPost

postrank

PowerPoint\/

Priceonomics Analysis Engine

PritTorrent\/[0-9]

Prlog

probethenet

Project 25499

Promotion_Tools_www.searchenginepromotionhelp.com

prospectb2b

Protopage

proximic

pshtt, https scanning

PTST

PTST\/[0-9]+

Pulsepoint XT3 web scraper

Python-httplib2

python-requests

Python-urllib

Qirina Hurdler

QQDownload

Qseero

Qualidator.com SiteAnalyzer

Quora Link Preview

Qwantify

Radian6

RankSonicSiteAuditor

Readability

RealPlayer%20Downloader

RebelMouse

redback\/

Redirect Checker Tool

ReederForMac

request\.js

ResponseCodeTest\/[0-9]

RestSharp

RetrevoPageAnalyzer

Riddler

Rival IQ

Robosourcer

Robozilla\/[0-9]

ROI Hunter

RPT-HTTPClient

RSSOwl

safe-agent-scanner

SalesIntelligent

SauceNAO

SBIder

scalaj-http

Scoop

scooter

ScoutJet

ScoutURLMonitor

Scrapy

ScreenShotService\/[0-9]

Scrubby

search\.thunderstone

SearchSight

Seeker

semanticdiscovery

semanticjuice

Semiocast HTTP client

SEO Browser

Seo Servis

seo-nastroj.cz

Seobility

SEOCentro

SeoCheck

SeopultContentAnalyzer

Server Density Service Monitoring

servernfo\.com

Seznam screenshot-generator

Shelob

Shoppimon Analyzer

ShoppimonAgent\/[0-9]

ShopWiki

ShortLinkTranslate

shrinktheweb

SilverReader

SimplePie

SimplyFast

Site-Shot\/

Site24x7

SiteBar

SiteCondor

siteexplorer\.info

SiteGuardian

Siteimprove\.com

Sitemap(s)? Generator

SiteMonitor

Siteshooter B0t

SiteTruth

sitexy\.com

SkypeUriPreview

Slack\/

slider\.com

slurp

SMRF URL Expander

SMUrlExpander

Snappy

SniffRSS

sniptracker

Snoopy

sogou web

SortSite

sovereign\.ai

spaziodati

Specificfeeds

speedy

SPEng

Spinn3r

spray-can

Sprinklr

sqlmap

spyonweb

Sqworm

SSL Labs

ssl-tools

StackRambler

Statastico\/

StatusCake

Stratagems Kumo

Stroke.cz

StudioFACA

suchen

summify

Super Monitoring

Surphace Scout

SwiteScraper

Symfony2 BrowserKit

SynHttpClient-Built

Sysomos

Symfony BrowserKit

T0PHackTeam

Tarantula\/

Taringa UGC

Tenon\.io

teoma

terrainformatica\.com

Test Certificate Info

Tetrahedron\/[0-9]

The Drop Reaper

The Expert HTML Source Viewer

theinternetrules

theoldreader\.com

Thumbshots

ThumbSniper

TinEye

Tiny Tiny RSS

TLSProbe\/

topster

touche.com

Traackr.com

TrapitAgent

truwoGPS

TulipChain

tweetedtimes\.com

Tweetminster

Tweezler\/

Twikle

Twingly

ubermetrics-technologies

uclassify

uCrawlr\/

UdmSearch

UniversalFeedParser

Untiny

UnwindFetchor

updated

Upflow

Uptimia

URLChecker

URLitor.com

urlresolver

Urlstat

UrlTrends Ranking Updater

Vagabondo

vBSEO

via ggpht\.com GoogleImageProxy

VidibleScraper\/

visionutils

vkShare

voltron

voyager\/

VSAgent\/[0-9]

VSB-TUO\/[0-9]

VYU2

w3af\.org

W3C-checklink

W3C-mobileOK

W3C_I18n-Checker

W3C_Unicorn

wangling

WatchMouse

WbSrch\/

web-capture\.net

Web-Monitoring

Web-sniffer

Webauskunft

WebCapture

WebClient\/

webcollage

WebCookies

WebCorp

WebDoc

WebFetch

WebImages

WebIndex

webkit2png

webmastercoffee

webmon

webscreenie

Webshot

Website Analyzer\/

websitepulse agent

websitepulse[+ ]checker

Websnapr\/

Webthumb\/[0-9]

WebThumbnail

WeCrawlForThePeace

WeLikeLinks

WEPA

WeSEE

wf84

wget

WhatsApp

WhatsMyIP

WhatWeb

WhereGoes\?

Whibse

Whynder Magnet

Windows-RSS-Platform

WinHttpRequest

wkhtmlto

wmtips

Woko

Word\/

WordPress\/

wotbox

WP Engine Install Performance API

wpif

wprecon\.com survey

WPScan

wscheck

WWW-Mechanize

www\.monitor\.us

XaxisSemanticsClassifier

Xenu Link Sleuth

XING-contenttabreceiver\/[0-9]

XmlSitemapGenerator

xpymep([0-9]?)\.exe

Y!J-(ASR|BSC)

Yaanb

yacy

Yahoo Ad monitoring

Yahoo Link Preview

YahooCacheSystem

YahooYSMcm

YandeG

Yandex(?!Search)

yanga

yeti

YLT

Yo-yo

Yoleo Consumer

yoogliFetchAgent

YottaaMonitor

yourls\.org

Zao

Zemanta Aggregator

Zend\\Http\\Client

Zend_Http_Client

zgrab

ZnajdzFoto

ZyBorg

[a-z0-9\-_]*((?<!cu)bot|crawler|archiver|transcoder|spider|uptime|validator|fetcher)

Источник здесь https://github.com/JayBizzle/Crawler-Detect .

 

Супер, спасибо