全球搜索引擎最新蜘蛛/爬虫列表

这里列出了全球搜索引擎蜘蛛/爬虫列表:   Crawler User agent Owner 192.com 192.comAgent 192.com 4anything 4anything.com LinkChecker v2.0 4anything A-Online A-Online Search Aon ABCdatos ABCdatos BotLink/5.xx.xxx#BBL ABCdatos AOL Sqworm/2.9.81-BETA (beta_release; 20011102-760; i686-pc-linux-gnu) AOL ASAHA ASAHA Search Engine Turkey V.001 (http://www.asaha.com/) ASAHA ASPseek ASPSeek/1.2.xa ASPseek ASPSeek/1.2.xxpre ASPSeek/1.2.x ASPSeek/1.2.5 ASPseek/1.2.9d ASPseek/1.2.xx AVSearch AVSearch-1.0(peter.turney@nrc.ca) National Research Council Canada AbachoBot AbachoBOT Abacho AbachoBOT (Mozilla compatible) Aberja Checkoma Aberja Checkoma Aberja Abot abot/0.1 (abot; http://www.abot.com/; abot@abot.com) Abot.com About About/0.1libwww-perl/5.47 About AboutUsBot Mozilla/5.0 (compatible; AboutUsBot/0.9; +http://www.aboutus.org/AboutUsBot) AboutUs Accelobot Mozilla/5.0 (compatible; heritrix/1.8.0 +http://www.accelobot.com) Accelovation Mozilla/5.0 (compatible; heritrix/1.12.0 +http://www.accelobot.com) Accoona Accoona-AI-Agent/1.1.1 (crawler at accoona dot com) Accoona accoona Acoi AcoiRobot Acoi Acoon Robot Acoon Robot v1.50.001 Acoon Acoon-Robot Acoon-Robot v3.00 (http://www.acoon.de/ and http://www.acoon.com/) Acoon Acorn Acorn/Nutch-0.9 (Non-Profit Search Engine; acorn.isara.org; acorn at isara dot o rg) Isara Activtourist Mozilla/4.0 (JemmaTheTourist;http://www.activtourist.com) Activtourist Aesop AESOP_com_SpiderMan Aesop Agada Mozilla/4.0 (agadine3.0) http://www.agada.de/ Agada agadine/1.x.x (+http://www.agada.de) Mozilla/4.0 (agadine3.0) AgentName AgentName/0.1 libwww-perl/5.48 Linkomatic Aibot AIBOT/2.1 By +(http://www.21seek.com/ , A Real artificial intelligence search engine , C hina) 21seek Aicrawler Accoona-AI-Agent/1.1.2 (aicrawler at accoonabot dot com) Accoona Aipbot aipbot/1.0 (aipbot; http://www.aipbot.com/; aipbot@aipbot.com) Aipbot Alacra PortalBSpider/2.0 (spider@portalb.com) Alacra Aladin.de Aladin/3.324 Abacho Aleksika Danmark Aleksika Spider/1.0 (+http://www.aleksika.com/) Aleksika Alexa ia_archiver Alexa AlkalineBOT AlkalineBOT/1.4 (1.4.0326.0 RTM) Vestris AlkalineBOT/1.3 Allesklar.de Allesklar/0.1 libwww-perl/5.46 Allesklar Allrati Allrati/1.1 (+) Unknown Almaden http://www.almaden.ibm.com/cs/crawler [hc5] IBM http://www.almaden.ibm.com/cs/crawler Altavista Scooter-W3-1.0 Altavista Scooter2_Mercator_x-x.0 scooter-venus-3.0.vns Scooter-W3.1.2 Scooter-3.2.NIV Scooter-3.2.JT Scooter-3.2.EX Scooter/2.0 G.R.A.B V1.0 Scooter_trk3-3.0.3 Scooter-3.2.SF0 Scooter/3.3.QA.pczukor Scooter/3.3 Scooter/1.1 (custom) Scooter-3.2.DIL Scooter/1.0 scooter@pa.dec.com Scooter-3.0.FS Scooter-3.0.EU Scooter/2.0 G.R.A.B. X2.0 Scooter/2.0 G.R.A.B. V1.1.0 Scooter-3.0.HD Scooter-3.0.VNS Scooter-3.2.BT Scooter-3.2 Scooter-3.0QI Scooter/3.3_SF Scooter-3.2.snippet Scooter-ARS-1.1-ih Scooter/3.3.vscooter Scooter-3.3dev Scooter-ARS-1.1 Scooter_bh0-3.0.3 Scooter/1.0 Amfibibot Amfibibot/0.06 (Amfibi Robot; http://www.amfibi.com/; agent@amfibi.com) Amfibi Amidalla libwww-perl/5.65 Amidalla Annomille AnnoMille spider 0.1 alpha – http://www.annomille.it/ Annomille AnsearchBot Mozilla/5.0 (compatible; AnsearchBot/1.0; +http://www.ansearch.com.au/) Ansearch AnswerBus AnswerBus (http://www.answerbus.com/) AnswerBus Answerchase PROve AnswerBot 4.0 Answerchase Antibot antibot-V1.3.3.1/debian-linux-sarge Antidot Any Search Info Mozilla/4.0 (Sleek Spider/1.2) Search-Info Anzwers Australia AnzwersCrawl/2.0 (anzwerscrawl@anzwers.com.au;Engine) Anzwers Australia Apexoo Spider Apexoo Spider 1.0 Apexoo Aport Aport Aport Appie appie 1.1 (http://www.walhello.com/) Walhello appie 1.1 (http://www.walhello.com/) Arabulbot Mozilla/5.0 (compatible; arabulbot/1.1; +http://www.arabul.com/bot.html) Arabul ArabyBot ArabyBot (compatible; Mozilla/5.0; GoogleBot; FAST Crawler 6.4; http://www.araby/ .com;) Araby Arachnoidea Arachnoidea (arachnoidea@euroseek.com) Euroseek ArchitextSpider ArchitextSpider Excite Archive.org_bot Mozilla/5.0 (compatible;archive.org_bot/1.7.1; collectionId=316; Archive-It; +ht tp://www.archive-it.org) Archive.org Arexera TECOMAC-Crawler/0.x Arexera X-Crawler Arianna http://www.arianna.it/ Libero Arikus_Spider Arikus_Spider Arikus Asahina Asahina-Antenna/1.x (libhina.pl/x.x ; libtime.pl/x.x) Asahina Asahina-Antenna/1.x Ask 24x Info ask.24x.info Ask 24x Ask Jeeves/Teoma Mozilla/5.0 (compatible; Ask Jeeves/Teoma; +http://about.ask.com/en/docs/about/w ebmasters.shtml) Ask Jeeves Mozilla/2.0 (compatible; Ask Jeeves/Teoma; +http://about.ask.com/en/docs/about/w ebmasters.shtml) Mozilla/2.0 (compatible; Ask Jeeves/Teoma; http://about.ask.com/en/docs/about/we bmasters.shtml) Mozilla/2.0 (compatible; Ask Jeeves/Teoma; +http://sp.ask.com/docs/about/tech_cr awling.html) Asked asked/Nutch-0.8 (web crawler; http://asked.jp/; epicurus at gmail dot com) Asked Askpeter_bot Mozilla/5.0 (compatible; askpeter_bot/3.2; +http://www.askpeter.info) Askpeter Asterias asterias/2.0 Singing Fish Asterias Crawler Mozilla/4.0 (compatible; MSIE 6.0 compatible; Asterias Crawler v4; +http://www.s ingingfish.com/help/spider.html; webmaster@singingfish.com); SpiderThread Revis ion: 3.11 Singingfish Mozilla/4.0 (compatible; MSIE 6.0 compatible; Asterias Crawler v4; +http://www.s ingingfish.com/help/spider.html; webmaster@singingfish.com); SpiderThread Revisi on: 3.10 Astrafind! Mozilla/4.0 (compatible: AstraSpider V.2.1 : astrafind.com) Seeq Atlocal AtlocalBot/1.1 +(http://www.atlocal.com/local-web-site-owner.html) @Local Attentio Attentio/Nutch-0.9-dev (Attentio’s beta blog crawler; http://www.attentio.com/; info@att entio.com) Attentio Augurnet Swiss augurfind Augurnet Swiss augurnfind V-1.x Axada axadine/ (Axadine Crawler; http://www.axada.de/; ) Axada Axandra Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; IBP; .NET CLR 1.1.4322) Axandra Axmo AxmoRobot – Crawling your site for better indexing on http://www.axmo.com/ search engine . Axmo Ay-Up FastBug http://www.ay-up.com/ Ay-up BE Internet Search Engine Blaiz-Bee/2.00.8222 (BE Internet Search Engine http://www.rawgrunt.com/) Rawgrunt Ba.be Mozilla/4.72 [en] (BACS http://www.ba.be/) BA BaBoom Web Portal BaboomBot/1.x.x (+http://www.baboom.us) Baboum BabalooSpider BabalooSpider/1.2 (BabalooSpider; http://www.babaloo.si/; spider@babaloo.si) Babaloo Backlink-Check Backlink-Check.de (+http://www.backlink-check.de/bot.html) Backlink-Check Baiduspider Baiduspider+(+http://www.baidu.com/search/spider.htm) Baidu.com Baiduspider+(+http://www.baidu.com/search/spider_jp.html) Balihoo Bloodhound/Nutch-0.9 (Testing Crawler for Research – obeys robots.txt and robots meta tags ; http://balihoo.com/index.aspx; robot at balihoo dot com) Balihoo TestCrawler/Nutch-0.9 (Testing Crawler for Research ; http://balihoo.com/index.a spx; tgautier at balihoo dot com) BanBot BanBots/1.2 (spider@banbots.com) Banbot BeamMachine BeamMachine/0.5 (dead link remover of http://www.beammachine.net/) BeamMachine Beauty (Cosmoty) beautybot/1.0 (+http://www.uchoose.de/crawler/beautybot/) uCHOOSE BebopBot BebopBot/2.5.1 ( crawler http://www.apassion4jazz.net/bebopbot.html ) Apassion4jazz BecomeBot Mozilla/5.0 (compatible; BecomeBot/1.83; MSIE 6.0 compatible; +http://www.become .com/site_owners.html) BecomeBot Mozilla/5.0 (compatible; BecomeBot/3.0; MSIE 6.0 compatible; +http://www.become. com/site_owners.html) BecomeJPBot Mozilla/5.0 (compatible; BecomeJPBot/2.3; MSIE 6.0 compatible; +http://www.becom e.co.jp/site_owners.html) Become BeijingCrawler BeijingCrawler Unknown BigClique BigCliqueBOT/1.03-dev (bigclicbot; http://www.bigclique.com/; bot@bigclique.com) BigClique Biglotron BIGLOTRON (Beta 2;GNU/Linux) Biglotron Bigsearch Bigsearch.ca/Nutch-0.9-dev (Bigsearch.ca Internet Spider; http://www.bigsearch.c/ a/; info@enhancededge.com) Bigsearch Bigsearch.ca/Nutch-1.0-dev (Bigsearch.ca Internet Spider; http://www.bigsearch.c/ a/; info@enhancededge.com) BilBasen FAST Enterprise Crawler 6 used by BilBasen ApS (michael@bilinfo.dk) Bilinfo BilgiBetaBot BilgiBetaBot/0.8-dev (bilgi.com (Beta) ; http://lucene.apache.org/nutch/bot.html ; nutch-agent@lucene.apache.org) Bilgi BilgiBot BilgiBot/1.0(beta) (http://www.bilgi.com/; bilgi at bilgi dot com) Bilgi Bisnisseek Custom Spider http://www.bisnisseek.com/ /1.0 Bisnisseek Bitacle Robot Bitacle bot/1.1 Bitacle Bitacle Robot (V:1.0;) (http://www.bitacle.com/) Blaiz Enterprises Blaiz-Bee/1.0 (+http://www.blaiz.net) Blaiz Enterprises Blaiz-Bee Blaiz-Bee/2.00.5502 (+http://www.blaiz.net) Blaiz Blaiz-Bee/2.00.5622 ( http://www.blaiz.net/) Blitzsuche BlitzBOT@tricus.net RP ONLINE Mozilla/4.0 (compatible; B_L_I_T_Z_B_O_T) BlitzBOT@tricus.net (Mozilla compatible) BlogRefsBot Mozilla/5.0 (compatible; BlogRefsBot/0.1; http://www.blogrefs.com/about/bloggers ) BlogRefs BlogSearch BlogSearch/1.0 +http://www.icerocket.com/ IceRocket BlogSearch/1.x +http://www.icerocket.com/ BlogWatcher blogWatcher_Spider/0.1 (http://www.lr.pi.titech.ac.jp/blogWatcher/) Okumura Group Blogbot Naamah 1.0a/Blogbot (http://blogbot.de/) Blogbot Naamah 1.0.1/Blogbot (http://blogbot.de/) Blogdex BlogBot/1.x Massachusetts Institute of Technology Blogdimension BlogBot Blogdimension/Alpha2 (Blogdimension BlogBot; http://www.blogdimension.com/) Blogdimension Bloglines Bloglines Title Fetch/1.0 (http://www.bloglines.com/) Bloglines Bloglines/3.1 (http://www.bloglines.com/) Bloglines-Images Bloglines-Images/0.1 (http://www.bloglines.com/) Bloglines BlogzIce BlogzIce/1.0 +http://www.icerocket.com/ IceRocket BlogzIce/1.0 (+http://icerocket.com; rhodes@icerocket.com) Boitho boitho.com-robot/1.x (http://www.boitho.com/bot.html) Boitho boitho.com-dc boitho.com-dc/0.xx (http://www.boitho.com/dcbot.html) boitho.com-robot/1.x Bot bot/1.0 (bot; http://; bot@bot.bot) Unknown BotSeer Mozilla 4.0(compatible; BotSeer/1.0; +http://botseer.ist.psu.edu) Penn State College of Information Sciences and Technology Botmobi Nokia6300/2.0 (05.50) Profile/MIDP-2.0 Configuration/CLDC-1.1 (botmobi http://fi/ nd.mobi/bot.html abuse@mtld.mobi) Find.mobi BravoBrian bSTOP BStop.BravoBrian.it Agent Detector BravoBrian BravoBrian SpiderEngine MarcoPolo BrightCrawler BrightCrawler (http://www.brightcloud.com/brightcrawler.asp) Brightcloud Bruinbot BruinBot (+http://webarchive.cs.ucla.edu/bruinbot.html) University of California Btbot BTbot/0.x (+http://www.btbot.com/btbot.html) Btbot BuildCMS crawler BuildCMS crawler (http://www.buildcms.com/crawler) BuildCMS BuiltWith Mozilla/5.0 (compatible; BuiltWith/0.1; +http://builtwith.com/bot.html) BuiltWith BullsEye/Intelliseek BullsEye Intelliseek BurstFindCrawler BurstFindCrawler/1.1 (crawler.burstfind.com; http://crawler.burstfind.com/; crawl er@burstfind.com) Burstfind Buscaplus Buscaplus Robi/1.0 (http://www.buscaplus.com/robi/) Buscaplus CEA larbin_2.6_basileocaml (basile.starynkevitch@cea.fr) CEA CMP libwww-perl/5.41 CMP United Business Media CUPS PrivacyFinder Cache Bot v1.0 PrivacyBird Camcrawler Camcrawler (+http://www.camdiscover.com/crawler.html) Sensation Internet Services CanadianContent Search RoboCrawl (http://www.canadiancontent.net/) CanadianContent RoboCrawl (http://www.canadiancontent.net/) Carleson carleson/1.0 Cosmix Catall Spider Catall Spider Catall Catall-Spider Catall-Spider/3.3.3(http://www.catall.de/) Catall CazoodleBot CazoodleBot/0.1 (CazoodleBot Crawler; http://www.cazoodle.com/; mqbot@cazoodle.co m) Cazoodle CazoodleBot/CazoodleBot-0.1 (CazoodleBot Crawler; http://www.cazoodle.com/cazood lebot; cazoodlebot@cazoodle.com) Ccubee ccubee/x.0 Empyreum Changedetection Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; http://www.changedetect/ ion.com/bot.html ) Changedetection Charlotte Mozilla/5.0 (compatible; Charlotte/1.0b; http://www.searchme.com/support/) Searchme Mozilla/5.0 (compatible; Charlotte/1.1; http://www.searchme.com/support/) Mozilla/5.0 (compatible; Charlotte/1.0b; charlotte@betaspider.com) Christcentral Mozilla/4.0 (compatible; ChristCrawler.com, ChristCrawler@ChristCENTRAL.com) Christcentral ChristCRAWLER 2.0 CipinetBot CipinetBot (http://www.cipinet.com/bot.html) Cipinet CipinetBot/1.0 (http://www.cipinet.com/bot.html) CjLogbot Mozilla/5.0 (compatible; CjLogbot 1.0; +http://www.cjlog.com/bot) CjLog Claymont Search Claymont.com Claymont Search CloakDetect CloakDetect/0.9 (+http://fulltext.seznam.cz/) Seznam Clushbot Clushbot/3.xx-Ajax (+http://www.clush.com/bot.html) Clush Clushbot/2.x (+http://www.clush.com/bot.html) Mozilla/5.0 (Clustered-Search-Bot/1.0; support@clush.com; http://www.clush.com/)   Clushbot/3.x-BinaryFury (+http://www.clush.com/bot.html) Clushbot/3.xx-Peleus (+http://www.clush.com/bot.html) Clushbot/3.31-BinaryFury (+http://www.clush.com/bot.html) Clushbot/3.xx-Hector (+http://www.clush.com/bot.html) Cnet robot Mozilla/4.6 [en] (http://www.cnet.com/) Search.com CoBITSProbe CoBITSProbe Academia Sinica Cobion Mozilla/4.0 (compatible; MSIE 5.5; Windows NT 4.0; obot) Cobion Mozilla/4.0 (compatible; MSIE 5.5; Windows NT 4.0; QXW03018) oBot ((compatible;Win32)) Combine Combine/2.0 http://combine.it.lth.se/ Combine Combine/2.0 Combine/3 http://combine.it.lth.se/ Cometrics-bot cometrics-bot, http://www.cometrics.de/ Cometrics Cometsystems Crawler (cometsearch@cometsystems.com) Cometsystems Crawler (cometsearch@cometsystems.com) Comperio FAST Enterprise Crawler 6 used by Comperio AS (sts@comperio.no) Comperio Compete.com larbin_2.2.0 (crawl@compete.com) Compete Inc Computerorgs htdig/3.1.6 (http://computerorgs.com/) Computerorgs.com Comrite Comrite/0.7.1 (Nutch; http://lucene.apache.org/nutch/bot.html; nutch-agent@lucen e.apache.org) Comrite ConveraCrawler ConveraCrawler/0.9e ( http://www.authoritativeweb.com/crawl) Convera ConveraCrawler/0.9d (+http://www.authoritativeweb.com/crawl) Converas RetrievalWare CrawlConvera0.1 (CrawlConvera@yahoo.com) Convera ConveraCrawler/0.2 Convera Internet Spider V6.x ConveraMultiMediaCrawler/0.1 (+http://www.authoritativeweb.com/crawl) CoolBot CoolBot SuchMaschine21 Cortina Vision Research Lab image spider at vision.ece.ucsb.edu Vision Research Lab CougarSearch CougarSearch/0.1 (+http://www.cougarsearch.com/faq.shtml) CougarSearch Cowbot Cowbot-0.1.x (NHN Corp. / +82-2-3011-1954 / nhnbot@naver.com) Naver Cowbot-0.1 (NHN Corp. / +82-2-3011-1954 / nhnbot@naver.com) CrawlerBoy CrawlerBoy Pinpoint.com Motricity Crawling jpeg Mozilla/5.0 (compatible; Crawling jpeg; http://www.yama.info.waseda.ac.jp/) Yamana Laboratory – Waseda University Japan Crawllybot Crawllybot/0.1 (Crawllybot; +http://www.crawlly.com; crawler@crawlly.com) Crawlly Croccrawler CrocCrawler vx.3 [en] (http://www.croccrawler.com/) (X11; I; Linux 2.0.44 i686) Croccrawler.com CsCrawler Hi! I’m CsCrawler, my homepage: http://www.kde.cs.uni-kassel.de/lehre/ss2005/goo glespam/crawler.html RPT-HTTPClient/0.3-3 University of Kassel Csci_b659/0.13 csci_b659/0.13 Indiana University School of Informatics Cuasar Cuasarbot/0.9b http://www.cuasar.com/spider_beta/ Cuasar CurryGuide CurryGuide SiteScan 1.1 CurryGuide CyberAlerts Mozilla/3.0 (compatible; Webinator-indexer.cyberalert.com/2.56) CyberAlerts Cydral CydralSpider/1.x (Cydral Web Image Search; http://www.cydral.com/) Cydral CydralSpider CydralSpider/2.4 (Cydral Image Search; http://www.cydral.com/) Cydral CydralSpider/2.2 (Cydral Image Search; http://www.cydral.com/) DAUM RSS Robot ELI/20070402:2.0 (DAUM RSS Robot, Daum Communications Corp.; +http://ws.daum.net /aboutkr.html) Daum DAUM Web Robot Mozilla/4.0 (compatible; MSIE is not me; DAUMOA/1.0.0; DAUM Web Robot; Daum Comm unications Corp., Korea) Daum Mozilla/4.0 (compatible; MSIE is not me; DAUMOA/1.0.1; DAUM Web Robot; Daum Comm unications Corp., Korea) Mozilla/4.0 (compatible; MSIE enviable; DAUMOA/1.0.1; DAUM Web Robot; Daum Commu nications Corp., Korea; +http://ws.daum.net/aboutkr.html) Mozilla/4.0 (compatible; MSIE enviable; DAUMOA 2.0; DAUM Web Robot; Daum Communi cations Corp., Korea; +http://ws.daum.net/aboutkr.html) DNS-Digger Mozilla/5.0 (compatible; DNS-Digger/1.0; +http://www.dnsdigger.com) Dnsdigger DailyOrbit Orbiter/T-2.0 (+http://www.dailyorbit.com/bot.htm) DailyOrbit DataFountains DataFountains/DMOZ Feature Vector Corpus Creator (http://ivia.ucr.edu/useragents .shtml) University of California DataFountains/DMOZ Downloader DataSpear Spider Bot DataSpear/1.0 (Spider; http://www.dataspear.com/spider.html; spider@dataspear.co m) DataSpear DataSpearSpiderBot/0.2 (DataSpear Spider Bot; http://dssb.dataspear.com/bot.html ; dssb@dataspear.com) DataparkSearch DataparkSearch/4.xx (http://www.dataparksearch.org/) dpSearch DataparkSearch/4.47 (+http://dataparksearch.org/bot) DaviesBot DaviesBot/1.7 (http://www.wholeweb.net/) Wholeweb Daypop daypopbot/0.x Daypop DbDig dbDig(http://www.prairielandconsulting.com/) Connections De.com Mozilla/5.0 (compatible; de/1.13.2 +http://www.de.com) De.com DeepIndexer DeepIndexer.ca Deepindex Deepak-USC/ISI deepak-USC/ISI University of Southern California Deepindex DeepIndex (http://www.en.deepindex.com/) Deepindex DeepIndex Deepindex V2 Denmex Websearch Denmex websearch (http://search.denmex.com/) Denmex Websearch DepSpid Mozilla/4.0 (compatible; DepSpid/5.03; +http://about.depspid.net) DepSpid Dev-spider2 dev-spider2.searchpsider.com/1.3b Searchspider DiaGem Japan DiaGem/1.1 (http://www.skyrocket.gr.jp/diagem.html) DiaGem Japan Die Kraehe -DIE-KRAEHE- META-SEARCH-ENGINE/1.1 http://www.die-kraehe.de/ Die Kraehe Diggit Digger/1.0 JDK/1.3.0rc3 Diggit Direct Hit Mozilla/2.0 (compatible; EZResult — Internet Search Engine) Teoma Disco-crawl disco/Nutch-1.0-dev (experimental crawler; http://www.discoveryengine.com/; disco-crawl@ discoveryengine.com) Discoveryengine disco/Nutch-0.9 (experimental crawler; http://www.discoveryengine.com/; disco-crawl@disc overyengine.com) Ditto DittoSpyder Ditto DoCoMo DoCoMo/1.0/Nxxxi/c10 NTT DoCoMo DoCoMo/1.0/Nxxxi/c10/TB DoCoMo/2.0 P900iV(c100;TB;W24H11) Dodgebot dodgebot/experimental Agmlab DotBot Mozilla/5.0 (compatible; DotBot/1.1; http://www.dotnetdotcom.org/, crawler@dotne tdotcom.org) Dotnetdotcom DotBot/1.0.1 Doubanbot Doubanbot/1.0 (bot@douban.com http://www.douban.com/) Douban Download-Tipp Download-Tipp Linkcheck (http://download-tipp.de/) Download-Tipp EyeCatcher (Download-tipp.de)/1.0 Drecombot Drecombot/1.0 (http://career.drecom.jp/bot.html) Drecom Japan DtSearchSpider dtSearchSpider dtSearch Dumbot Dumbot(version 0.1 beta) DumbFind.com Dumbot(version 0.1 beta – http://www.dumbfind.com/dumbot.html) Dumbot(version 0.1 beta – dumbfind.com) E-SocietyRobot e-SocietyRobot(http://www.yama.info.waseda.ac.jp/~yamana/es/) Yamana Laboratory E-StyleISP eStyleSearch 4 (compatible; MSIE 6.0; Windows NT 5.0) e-StyleISP EApolloBot eApolloBot/1.0 (eApollo search engine robot; http://www.eapollo.com/; eapollo at global-opto dot com) EApollo EMPAS_ROBOT EMPAS_ROBOT Empas ESISmartSpider ESISmartSpider smart-spider.com Earthcom EARTHCOM.info/1.xbeta [www.earthcom.info] Earthcom.info Mozilla/5.0 (compatible; EARTHCOM/2.2; +http://enter4u.eu) EARTHCOM.info/1.x [www.earthcom.info] Mozilla/5.0 (compatible; EARTHCOM.info/2.01; http://www.earthcom.info/) EasyDL EasyDL/3.04 http://keywen.com/Encyclopedia/Bot Keywen EasyDL/3.xx EasyDL/3.xx http://keywen.com/Encyclopedia/Bot Echo.com Mozilla/4.0 (compatible; MSIE 5.0; Windows 95) TrueRobot; 1.5 Echo.com Echo.fr EchO!/2.0 Echo.fr Egothor Mozilla/5.0 (compatible; egothor/8.0g; +http://ego.ms.mff.cuni.cz/) Charles University in Prague Egotobot EgotoBot/4.8 (+http://www.egoto.com/about.htm) Egoto.com Elfbot elfbot/1.0 (+http://www.uchoose.de/crawler/elfbot/) uCHOOSE Elsop LinkScan/9.0g Unix LinkScan LinkScan/x.x Unix LinkScan/11.0beta2 Unix EmeraldShield.com Web Spider EmeraldShield.com Web Spider (http://www.emeraldshield.com/webbot.aspx) Emeraldshield Enfish Tracker Enfish Tracker Enfish Enoola enoola (http://www.enoola.com/) Enoola Enterprise Search Enterprise_Search/1.0.xxx Innerprise Search/1.0 (http://www.innerprise.net/es-spider.asp) Enterprise_Search/1.00.xxx;MSSQL (http://www.innerprise.net/es-spider.asp) Enterprise_Search/1.0 ES.NET_Crawler/2.0 (http://search.innerprise.net/) Entireweb WorldLight Entireweb Mozilla/4.0 (compatible; SpeedySpider; http://www.entireweb.com/) Speedy Spider (Beta/x.x; speedy@entireweb.com) Speedy_Spider (http://www.entireweb.com/) Speedy Spider (http://www.entireweb.com/about/search_tech/speedy_spider/) Envolkspider envolk/1.7 (+http://www.envolk.com/envolkspiderinfo.php) Envolk envolk[ITS]spider/1.6(+http://www.envolk.com/envolkspider.html) envolk/1.7 (+http://www.envolk.com/envolkspiderinfo.html) EroCrawler EroCrawler EroCrawler Eruvo-bot eruvo-bot 4.8.1 (http://www.eruvo.com/) Eruvo EuripBot EuripBot/0.4 (+http://www.eurip.com) PreCheck Eurip.com EuripBot/0.2 (+http://www.eurip.com) GetRobots EuripBot/0.4 (+http://www.eurip.com) GetFile EuripBot/0.5 (+http://www.eurip.com) PreCheck Euro-spider Euro-Spider Shopping 1.0 Euro-spider Evaal Evaal/0.7.1 (Evaal; http://search.evaal.com/bot.html; bot@evaal.com) Evaal EvaalSE EvaalSE – bot@evaal.com Evaal Eventax eventax/1.3 (eventax; http://www.eventax.de/; info@eventax.de) Eventax Everest-Vulcan Everest-Vulcan Inc./0.1 (R&D project; host=e-1-24; http://everest.vulcan.com/cra wlerhelp) Vulcan Exabot Exabot/2.0 Exalead Mozilla/5.0 (compatible; Exabot/3.0; +http://www.exabot.com/go/robot) Mozilla/5.0 (compatible; Exabot Test/3.0; +http://www.exabot.com/go/robot) Exalead NG/MimeLive Client (convert/http/0.120) Exabot/3.0 ExaBotTest/3.0 ExaBotTest/2.0 NG/2.0 Exabot-Test/1.0 Mozilla/5.0 (compatible; Konqueror/3.2; Linux) (KHTML, like Gecko) Exabot-Images Mozilla/5.0 (compatible; Exabot-Images/3.0; +http://www.exabot.com/go/robot) Exalead Exabot-Images/1.0 NG/4.0.1229 ExactSEEK ExactSeek Crawler/0.1 ExactSEEK exactseek.com eseek-larbin_2.6.2 (crawler@exactseek.com) exactseek-pagereaper-2.63 (crawler@exactseek.com) exactseek-crawler-2.63 (crawler@exactseek.com) ExactSeek_Spider ExactSeek_Spider ExactSeek Excalibur Excalibur Internet Spider V6.5.4 Convera Execrawl Execrawl/1.0 (Execrawl; http://www.execrawl.com/; bot@execrawl.com) Execrawl FAST-WebCrawler FAST-WebCrawler/2.1-pre5 (crawler@fast.no; http://www.fast.no/faq/faqfastwebsear ch/faqfastwebcrawler.html) FAST FAST-WebCrawler/2.2-pre3 (crawler@fast.no; http://www.fast.no/faq/faqfastwebsear ch/faqfastwebcrawler.html) FAST-WebCrawler/2.2-pre4 (crawler@fast.no; http://www.fast.no/faq/faqfastwebsear ch/faqfastwebcrawler.html) FAST-WebCrawler/2.2-pre2 (crawler@fast.no; http://www.fast.no/faq/faqfastwebsear ch/faqfastwebcrawler.html) FAST-WebCrawler/3.7/FirstPage (atw-crawler at fast dot no;http://fast.no/support /crawler.asp) FAST-WebCrawler/2.1-pre13 (crawler@fast.no; http://www.fast.no/faq/faqfastwebsea rch/faqfastwebcrawler.html) FAST-WebCrawler/2.2-pre1 (crawler@fast.no; http://www.fast.no/faq/faqfastwebsear ch/faqfastwebcrawler.html) FAST-WebCrawler/2.2-pre5 (crawler@fast.no; http://www.fast.no/faq/faqfastwebsear ch/faqfastwebcrawler.html) FAST-WebCrawler/2.2-pre8 (crawler@fast.no; http://www.fast.no/faq/faqfastwebsear ch/faqfastwebcrawler.html) FAST-WebCrawler/2.1-pre10 (crawler@fast.no; http://www.fast.no/faq/faqfastwebsea rch/faqfastwebcrawler.html) FAST-WebCrawler/2.1-pre14 (crawler@fast.no; http://www.fast.no/faq/faqfastwebsea rch/faqfastwebcrawler.html) FAST-WebCrawler/2.1-pre12 (crawler@fast.no; http://www.fast.no/faq/faqfastwebsea rch/faqfastwebcrawler.html) FAST-WebCrawler/2.1-pre11 (crawler@fast.no; http://www.fast.no/faq/faqfastwebsea rch/faqfastwebcrawler.html) FAST-WebCrawler/2.2-pre9 (crawler@fast.no; http://www.fast.no/faq/faqfastwebsear ch/faqfastwebcrawler.html) FAST-WebCrawler/2.1-pre6 (crawler@fast.no; http://www.fast.no/faq/faqfastwebsear ch/faqfastwebcrawler.html) FAST-WebCrawler/2.1-pre7 (crawler@fast.no; http://www.fast.no/faq/faqfastwebsear ch/faqfastwebcrawler.html) FAST-WebCrawler/2.0.10 (crawler@fast.no; http://www.fast.no/faq/faqfastwebsearch /faqfastwebcrawler.html) FAST-WebCrawler/2.1-pre4 (crawler@fast.no; http://www.fast.no/faq/faqfastwebsear ch/faqfastwebcrawler.html) FAST-WebCrawler/2.0.9 (crawler@fast.no; http://www.fast.no/faq/faqfastwebsearch/ faqfastwebcrawler.html) FAST-WebCrawler/2.1.prealpha.2000-04-07.1 (ashen@looksmart.net) FAST Enterprise Crawler/6.4.18 (crawler@fast.no) FAST-SoccerCrawler/2.2-pre-cvs (oyvinda@fast.no; http://www.fast.no/faq/faqfastw ebsearch/faqfastwebcrawler.html) FAST-WebCrawler/2.1-pre2 (ashen@looksmart.net) FAST-WebCrawler/2.1.pre.2000-04-18.1 (crawler@fast.no; http://www.fast.no/faq/fa qfastwebsearch/faqfastwebcrawler.html) fastlwspider/1.0 FAST-WebCrawler/2.1.pre.2000-04-14.1 (ashen@looksmart.net) FDSE Mozilla/4.0 (compatible; FDSE robot) Abadoor FaXobot Faxobot/1.0 Faxo Factbot factbot : http://www.factbites.com/robots Factbites Factbot 1.09 (see http://www.factbites.com/webmasters.php) Fast Search PycURL FAST Fastbot fastbot crawler beta 2.0 (+http://www.fastbot.de) Fastbot Favo.eu crawler favo.eu crawler/0.6 (http://www.favo.eu/) Favo Feed24 Feed24.com Feed24 FeedChecker FeedChecker/0.01 University of Tokyo Feedfetcher-Google Feedfetcher-Google; (+http://www.google.com/feedfetcher.html) Google Feedster Crawler Feedster Crawler/3.0; Feedster, Inc. Feedster Felix Felix – Mixcat Crawler (+http://mixcat.com) MixCat Filangy Filangy/1.01 (Filangy; http://www.filangy.com/filangyinfo.jsp?inc=robots.jsp; fi langy-agent@filangy.com) Filangy FindLinks http://wortschatz.uni-leipzig.de/findlinks/ University of Leipzig findlinks/1.1.1 (+http://wortschatz.uni-leipzig.de/findlinks/) findlinks/1.1.1-a2 (+http://wortschatz.uni-leipzig.de/findlinks/) findlinks/1.0.9 (+http://wortschatz.uni-leipzig.de/findlinks/) findlinks/0.901 (+http://wortschatz.uni-leipzig.de/findlinks/) findlinks/1.1.4-beta1 ( http://wortschatz.uni-leipzig.de/findlinks/) findlinks/1.1.1-a5 (+http://wortschatz.uni-leipzig.de/findlinks/) Findexa Crawler Findexa Crawler (http://www.findexa.no/gulesider/article26548.ece) Findexa FineBot FineBot Finesearch Firefly Firefly/1.0 (compatible; Mozilla 4.0; MSIE 5.5) Fireball Firefly/1.0 FirstGov FirstGov.gov Search – POC:firstgov.webmasters@gsa.gov U.S.Government Firstsbot firstsbot Firstsfind Flapbot Flapbot/0.7.2 (Flaptor Crawler; http://www.flaptor.com/; crawler at flaptor perio d com) Flaptor Flatlandbot great-plains-web-spider/flatlandbot (Flatland Industries Web Spider; http://www/. flatlandindustries.com/flatlandbot.php; jason@flatlandindustries.com) Flatland Industries great-plains-web-spider/gpws (Flatland Industries Web Spider; http://www.flatlan/ dindustries.com/flatlandbot.php; jason@flatlandindustries.com) flatlandbot/flatlandbot (Flatland Industries Web Spider; http://www.flatlandindu/ stries.com/flatlandbot.php; jason@flatlandindustries.com) FlickBot FlickBot 2.0 RPT-HTTPClient/0.3-3 DivX.com Fluffy the spider Mozilla/3.0 (compatible; Fluffy the spider; http://www.searchhippo.com/; info@se archhippo.com) Searchhippo FnooleBot FnooleBot/2.5.2 (+http://www.fnoole.com/addurl.html) Fnoole Folkd.com Spider Folkd.com Spider/0.1 beta 1 (http://www.folkd.com/) Folkd ForAll.pl-Crawler ForAll.pl-Crawler/1.0 ForAll Francis Francis/1.0 (francis@neomo.de http://www.neomo.de/) Neomo FreshNotes crawler FreshNotes crawler< report problems to crawler-at-freshnotes-dot-com FreshNotes FreshNotes crawler, report problems to crawler-at-freshnotes-dot-com Freshmeat freshmeat.net URL validator/1.1 Freshmeat FuchsBot FuchsBot +http://www.fuchsbot.tld FuchsBot FurlBot Mozilla/4.0 compatible FurlBot/Furl Search 2.0 (FurlBot; http://www.furl.net/; wn .furlbot@looksmart.net) Furl FuseBulb FuseBulb.Com FuseBulb FyberSpider FyberSpider (+http://www.fybersearch.com/fyberspider.php) FyberSearch GAIS Robot GAIS Robot/1.0B2 Seed GEXTEST-00393 gsa-crawler (Enterprise; GEXTEST-00393; gsasymbiosys@gmail.com,xeonbox4@gmail.co m) Unknown GPU p2p crawler Mozilla/4.0 (compatible; GPU p2p crawler http://gpu.sourceforge.net/search_engin e.php) GPU GSiteCrawler GSiteCrawler/v1.20 rev. 273 (http://gsitecrawler.com/) GSiteCrawler Gaaz gazz/x.x (gazz@nttrd.com) Infobee Gaisbot Gaisbot/3.0+(robot06@gais.cs.ccu.edu.tw;+http://gais.cs.ccu.edu.tw/robot.php) Gais Gaisbot/3.0+(robot@gais.cs.ccu.edu.tw;+http://gais.cs.ccu.edu.tw/robot.php) Gaisbot/3.0+(indexer@gais.cs.ccu.edu.tw;+http://gais.cs.ccu.edu.tw/robot.php) GalaxyBot Mozilla/4.0 (compatible; http://www.galaxy.com/) Galaxy Mozilla/4.0 (compatible; MSIE 5.0; www.galaxy.com/galaxybot.html) GalaxyBot/1.0 (http://www.galaxy.com/galaxybot.html) Gamekitbot gamekitbot/1.0 (+http://www.uchoose.de/crawler/gamekitbot/) Uchoose GammaSpider GammaSpider/1.0 Gammasite GenieKnows geniebot wgao@genieknows.com GenieKnows larbin_2.6.3 (wgao@genieknows.com) Mozilla/5.0 wgao@genieknows.com Mozilla/5.0 (wgao@genieknows.com) GeonaBot GeonaBot 1.x; http://www.geona.com/ Geona Georgia Institute of Technology larbin_2.6.2 (listonATccDOTgatechDOTedu) Georgia Institute of Technology Geourl Mozilla/5.0 (compatible; geourl/2.0b16 – http://geourl.org/bot) Geourl GigaBaz Brainbot gigabaz/3.1x (baz@gigabaz.com; http://gigabaz.com/gigabaz/) Gigabaz MicroBaz Gigabot Gigabot/2.0; http://www.gigablast.com/spider.html Gigablast Gigabot/3.0 (http://www.gigablast.com/spider.html) Gigabot/2.0att Gigabot/2.0/gigablast.com/spider.html Gigabot/2.0 Girafabot Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 4.0; Girafabot; girafabot at giraf a dot com; http://www.girafa.com/) Girafa GlobalQueue Look.com Multi-mode GnodSpider GNODSPIDER (http://www.gnod.net/) Gnod GoForIt GOFORITBOT ( http://www.goforit.com/about/ ) GoForIt GoForIt.com Goblin Goblin/0.9 (http://www.goguides.org/) GoGuides Goblin/0.9.x (http://www.goguides.org/goblin-info.html) Gonzo1 gonzo1[P] +http://www.suchen.de/popups/faq.jsp T-info Gonzo2 gonzo2[P] mailto:crawleradmin.t-info@telekom.de T-info gonzo1[P] mailto:crawleradmin.t-info@telekom.de gonzo2[P] +http://www.suchen.de/faq.html Goo (Japan) Mozilla/3.0 (Slurp.so/Goo; slurp@inktomi.com; http://www.inktomi.com/slurp.html)     Google-Adsense Mediapartners-Google Google Mediapartners-Google/2.1 ( http://www.googlebot.com/bot.html) Mediapartners-Google/2.1 Google-Image Googlebot-Image/1.0 Google Googlebot-Image/1.0 (+http://www.googlebot.com/bot.html) Googlebot-Image/1.0 ( http://www.googlebot.com/bot.html) Google-Sitemaps Google-Sitemaps/1.0 Google Google-WAP Nokia-WAPToolkit/1.2 googlebot(at)googlebot.com Google Google WAP Proxy/1.0 GoogleBot Mozilla/5.0 (compatible; Googlebot/2.1; http://www.google.com/bot.html) Google Mozilla/4.0 (MobilePhone SCP-5500/US/1.0) NetFront/3.0 MMP/2.0 FAKE (compatible; Googlebot/2.1; http://www.google.com/bot.html) Googlebot/Test ( http://www.googlebot.com/bot.html) Googlebot/2.1 ( http://www.googlebot.com/bot.html) Mozilla/4.0 (MobilePhone SCP-5500/US/1.0) NetFront/3.0 MMP/2.0 (compatible; Goog lebot/2.1; http://www.google.com/bot.html) Googlebot/1.0 (googlebot@googlebot.com http://googlebot.com/) Googlebot/1.0 (googlebot@googlebot.com) Googlebot/2.1w (+http://googlebot.com/bot.html) Googlebot/2.1 ( http://www.google.com/bot.html) Googlebot/1.0 Googlebot/2.0 (+http://googlebot.com/bot.html) Googlebot-w/2.1 (+http://googlebot.com/bot.html) Googlebot/2.0 beta (googlebot@googlebot.com) Googlebot/2.1 (+http://www.googlebot.com/bot.html) Googlebot/2.1 (+http://www.google.com/bot.html) Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html) Googlebot-Mobile Generic Mobile Phone (compatible; Googlebot-Mobile/2.1; +http://www.google.com/b ot.html) Google KDDI-CA33 UP.Browser/6.2.0.10.4 (GUI) MMP/2.0 (compatible; Googlebot-Mobile/2.1; +http://www.google.com/bot.html) Nokia6820/2.0 (4.83) Profile/MIDP-1.0 Configuration/CLDC-1.0 (compatible; Google bot-Mobile/2.1; +http://www.google.com/bot.html) Greaterera Mozilla/5.0 (compatible; heritrix/1.7.0 +http://www.greaterera.com/) Greaterera GrigorBot GrigorBot 0.8 (http://www.grigor.biz/bot.html) Grigor Gromit Gromit/1.0 Australasian Legal Information Institute Grub-client Mozilla/4.0 (compatible; grub-client-1.4.3; Crawl your own stuff with http://gru/ b.org) Grub Gsa-crawler gsa-crawler (Enterprise; GIX-03519; cknuetter@stubhub.com) IBM gsa-crawler (Enterprise; GIX-04637; rex_li@trend.com.tw) Gulliver Gulliver/1.2 Northernlight Gulliver/1.3 GulperBot Mozilla/5.0 [en] (compatible; Gulper Web Bot 0.2.4 www.ecsl.cs.sunysb.edu/~maxim /cgi-bin/Link/GulperBot) University of New-York Gulper Web Bot 0.2.4 (www.ecsl.cs.sunysb.edu/~maxim/cgi-bin/Link/GulperBot) Gungho-crawler Gungho/0.08004 (http://code.google.com/p/gungho-crawler/wiki/Index) Gungho GurujiBot GurujiBot/1.0 (+http://www.guruji.com/WebmasterFAQ.html) Guruji GurujiBot/1.0 (+http://www.guruji.com/en/WebmasterFAQ.html) Harvest-NG Harvest-NG/1.0.2 Harvest-NG Hatena Antenna Hatena Antenna/0.4 (http://a.hatena.ne.jp/help#robot) Hatena Antenna HatenaScreenshot HatenaScreenshot/1.0 (checker) Hatena HatenaScreenshot/1.0 (checker) Hbtronix.spider hbtronix.spider.2 — http://hbtronix.de/spider.php Hbtronix HeinrichderMiragoRobot HeinrichderMiragoRobot (http://www.miragorobot.com/scripts/deinfo.asp) Mirago Helix Helix/1.x (+http://www.sitesearch.ca/helix/) SiteSearch HenriLeRobotMirago HenriLeRobotMirago (http://www.miragorobot.com/scripts/frinfo.asp) Mirago HenryTheMiragoRobot HenryTheMiragoRobot (http://www.miragorobot.com/scripts/mrinfo.asp) Mirago Heritrix mozilla/5.0 (compatible; heritrix/1.3.0 +http://archive.crawler.org) University of Washington archive.org_bot Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; heritrix/1.3.0 +http://www.cs .washington.edu/research/networking/websys/) Heritrix L3S Mozilla/5.0 (compatible; heritrix/1.5.0 +http://www.l3s.de/~kohlschuetter/projec ts/crawling/) L3S Research Center Heritrix/1.4.0 Mozilla/5.0 (compatible; heritrix/1.4.0 +http://www.chepi.net) Chepi Hermits Search Mozilla/5.0 (compatible; Hermit Search. Com; +http://www.hermitsearch.com) Hermits Search Hiiglespider Hiiglespider/0.1, Hiigle.com, http://hiigle.com/spider Hiigle Hitwise Spider Hitwise Spider v1.0 http://www.hitwise.com/ Hitwise Holmes holmes/3.10.1 (OnetSzukaj/5.0; +http://szukaj.onet.pl) Szukaj.onet holmes/3.11 (http://morfeo.centrum.cz/bot) holmes/3.11 (OnetSzukaj/5.0; +http://szukaj.onet.pl) holmes/x.x holmes/3.9 (OnetSzukaj/5.0; +http://szukaj.onet.pl) HolmesBot HolmesBot (http://holmes.ge/) Holmes HomePageSearch HomePageSearch(hpsearch.uni-trier.de) HomePageSearch Homerbot Homerbot: http://www.homerweb.com/ Homerweb Honda-Search Honda-Search/0.7.2 (Nutch; http://lucene.apache.org/nutch/bot.html; search@honda -search.com) Honda-Search Hoowwwer HooWWWer/2.1.0 (+http://cosco.hiit.fi/search/hoowwwer/ | mailto:crawler-info%3Catroot@localhost) ht:/dig Htdig/3.1.6 htdig/3.1.6 (unconfigured@htdig.searchengine.maintainer) Acad?mie de Toulouse I1searchbot i1searchbot/2.0 (i1search web crawler; http://www.i1search.com/; crawler@i1search .com) I1search ICC-Crawler ICC-Crawler(Mozilla-compatible;http://kc.nict.go.jp/icc/crawl.html;icc-crawl-con tact(at)ml(dot)nict(dot)go(dot)jp) NICT ICC-Crawler(Mozilla-compatible; http://kc.nict.go.jp/icc/crawl.html; icc-crawl(a t)ml(dot)nict(dot)go(dot)jp) ICC-Crawler(Mozilla-compatible; http://kc.nict.go.jp/icc/crawl.html; icc-crawl-c ontact(at)ml(dot)nict(dot)go(dot)jp) ICCrawler ICCrawler – ICjobs (http://www.icjobs.de/bot.htm) ICCenter ICRA_Label_spider ICRA_label_spider/x.0 Icra IDBot Mozilla/5.0 (compatible; IDBot/1.0; +http://www.id-search.org/bot.html) Id-search IIITBOT IIITBOT/1.1 (Indian Language Web Search Engine; http://webkhoj.iiit.net/; pvvpr a t iiit dot ac dot in) Webkhoj INGRID Mozilla/3.0 (INGRID/3.0 MT; webcrawler@NOSPAMexperimental.net; http://webmaster/. ilse.nl/jsp/webmaster.jsp) Ilse IP2MapBot IP2MapBot/1.1 http://www.ip2map.com/ Ip2Map IPiumBot IPiumBot laurion(dot)com Laurions IRLbot IRLbot/2.0 (compatible; MSIE 6.0; http://irl.cs.tamu.edu/crawler) Texas A&M University IRLbot/3.0 (compatible; MSIE 6.0; http://irl.cs.tamu.edu/crawler) IRLbot/1.0 (+http://irl.cs.tamu.edu/crawler) IWAgent IWAgent/ 1.0 – http://www.brandprotect.com/ Brandprotect Iaskspider2 iaskspider2 (iask@staff.sina.com.cn) Sina Ichiro ichiro/1.0 (ichiro@nttr.co.jp) Goo ichiro/2.0 (http://help.goo.ne.jp/door/crawler.html) ichiro/1.0 (ichiro@nttr.co.jp) IconSurf IconSurf/2.0 favicon monitor (see http://iconsurf.com/robot.html) IconSurf IconSurf/2.0 favicon finder (see http://iconsurf.com/robot.html) Icsbot icsbot-0.1 International Christian school of Seoul Ideare ideare – SignSite/1.x Ideare IlTrovatore IlTrovatore/1.2 (IlTrovatore; http://www.iltrovatore.it/bot.html; bot@iltrovator e.it) IlTrovatore Ilial/Nutch ilial/Nutch-0.9 (Ilial, Inc. is a Los Angeles based Internet startup company. Fo r more information please visit http://www.ilial.com/crawler; http://www.ilial.c/ om/crawler; crawl@ilial.com) Ilial ilial/Nutch-0.9-dev Ilse Mozilla/3.0 (Vagabondo/2.0 MT; webcrawler@NOSPAMexperimental.net; http://aanmeld/ en.ilse.nl/?aanmeld_mode=webhints) Ilse Mozilla/3.0 (INGRID/3.0 MT; webcrawler@NOSPAMexperimental.net; http://aanmelden/. ilse.nl/?aanmeld_mode=webhints) ImageWalker ImageWalker/2.0 (http://www.bdbrandprotect.com/) Bdbrandprotect IncyWincy NetResearchServer/x.x(loopimprovements.com/robot.html) LoopImprovements IncyWincy page crawler(webmaster@loopimprovements.com,http://www.loopimprovement s.com/robot.html) IncyWincy(http://www.loopimprovements.com/robot.html) IncyWincy/2.1(loopimprovements.com/robot.html) IncyWincy data gatherer(webmaster@loopimprovements.com,http://www.loopimprovemen ts.com/robot.html) IncyWincy (Look) IncyWincy(http://www.look.com/) Look IndexTheWeb IndexTheWeb.com Crawler7 IndexTheWeb Indonesia Interactive Mozilla/4.0 (compatible; MSIE 4.0; Windows NT; Site Server 3.0 Robot) Indonesia Interactive Indonesia Interactive InelaBot InelaBot/0.2 (+http://inelegant.org/bot) Inelegant Inet Library Inet library Inet Library InfoFly InfoFly/1.0 (http://www.versions-project.org/) Versions-project InfoLab robot Mozilla/5.0 (compatible; heritrix/1.10.2 +http://i.stanford.edu/) Stanford University InfoSec Search Bot RedCell/0.1 (InfoSec Search Bot (Coming Soon); http://www.telegenetic.net/bot.ht ml; lhall@telegenetic.net) Telegenetic Infoseek InfoSeek Sidewinder/0.9 Go Inria larbin_2.2.1_de_Viennot (Laurent.Viennot@inria.fr) Inria Insitor Search robot Insitor.com search and find world wide! Insitor Insitornaut Insitornaut Insitor Internet Ninja Internet Ninja x.0 Dream Train Internet Internetseer InternetSeer.com Internetseer Iprospect Mozilla/3.0 (compatible; Webinator-DEV01.home.iprospect.com/2.56) Iprospect IpselonBot IpselonBot/0.xx-beta (Ipselon; http://www.ipselon.com/; ipselonbot@ipselon.com) Ipselon Iseekbot iSEEKbot/iSEEKbot-0.9-dev (http://beta.iseek.com/iseekbot.html; bot at iseek dot com) Iseek Ishida Lab larbin_2.2.2 (sugayama@lab7.kuis.kyoto-u.ac.jp) Kyoto University It-bot IlTrovatore-Setaccio/1.2 (It-bot;compatible;MSIE 6.0;Mozilla/4.0; http://www.ilt/ rovatore.it/bot.html; bot@iltrovatore.it) IlTrovatore Jabot Jabot/7.x.x (http://odin.ingrid.org/) ODIN Directory Jabot/6.x (http://odin.ingrid.org/) Jambot Jambot/0.2.1 (Jambot; http://www.jambot.com/blog/static.php?page=webmaster-robot ; crawler@jambot.com) Jambot Jambot/0.1.1 (Jambot; http://www.jambot.com/blog; crawler@jambot.com) Jayde Crawler Jayde Crawler. http://www.jayde.com/ Jayde Jeanie jeanie/3.3.3(www.sidedc.net/;compatible;MSIE 6.0;Windows NT 5.51) Sidedc Jetbot Jetbot/1.0 JetEye Jobs.de-Robot Mozilla/5.0 (compatible; jobs.de-Robot http://www.jobs.de/; jobsde@jobscout24.de) ( newsexpress e-mail: newsexpress-l@neofonie.de http://www.neofonie.de/loesunge n/search/robot.html ) Neofonie Jongaimpi jongaimpi/2.10 (jonga; http://www.jonga.co.za/; info@jonga.co.za) Jonga Jyxobot Jyxobot/1 Jyxo Jyxobot/x K2 Spider k2spider Verity KAIST AITrc Crawler KAIST AITrc Crawler AITrc KFSW-Bot KFSW-Bot (Version: 1.01, powered by KFSW, http://www.kfsw.de/) KFSW KIT_Fireball KIT_Fireball/2.0 Dino-online KSbot KSbot/1.0 (KnowledgeStorm crawler; http://www.knowledgestorm.com/resources/conte nt/crawler/index.html; crawleradmin@knowledgestorm.com) Knowledgestorm KakleBot KakleBot – www.kakle.com/0.1 (KakleBot – http://www.kakle.com/; http:// www.kakle.com/bo t.html; support@kakle.com) akle KaloogaBot kalooga/KaloogaBot (Kalooga; http://www.kalooga.com/info.html?page=crawler; craw ler@kalooga.com) kalooga kalooga/KaloogaBot (Kalooga; http://www.kalooga.com/; info@kalooga.com) Kasparek Firefox_1.0.6 (kasparek@naparek.cz) Czech Technical University Prague Keegeebot Keegeebot/2.1 (+http://www.keegee.com/keegee/bot.html) Keegee Kenjin Spider Kenjin Spider Kenjin Kevin Kevin http://dznet.com/kevin/ Dznet.com Kevin http://websitealert.net/kevin/ KicktooBot kicktooBotV1.1 kictooBot@kictoo.com Kicktoo Kinja-imagebot kinja-imagebot (http://www.kinja.com/) Kinja Kinjabot kinjabot (http://www.kinja.com/) Kinja KnowItAll KnowItAll(knowitall@cs.washington.edu) University of Washington Knowledge.com Knowledge.com/0.x knowledge.com Krugle Krugle/Krugle,Nutch/0.8+ (Krugle web crawler; http://www.krugle.com/crawler/info .html; webcrawler@krugle.com) Krugle Kulokobot kulokobot http://www.kuloko.com/ kuloko@backweave.com Kuloko kuloko-bot/0.x Kulturarw kulturarw3/0.1 National Library of Sweden Kumm KummHttp/1.1 (compatible; KummClient; Linux rulez) Sanoma Kyluka crawl Mozilla/5.0 (compatible; Kyluka crawl; http://www.kyluka.com/crawl.html; crawl@k yluka.com) Kyluka LECodeChecker LECodeChecker/3.0 libgetdoc/1.0 Linkexchange LNSpiderguy LNSpiderguy Lexis-Nexis LapozzBot LapozzBot/1.5 (+http://robot.lapozz.hu) Lapozz LapozzBot/1.4 (+http://robot.lapozz.hu) LapozzBot/1.4 ( http://robot.lapozz.com/) Larbin_2.6.3 larbin_2.6.3 larbin2.6.3@unspecified.mail Unknown larbin_2.6.3 marzia.polito@intel.com Lawinfo-crawler lawinfo-crawler/Nutch-0.9-dev (Crawler for lawinfo.com pages; http://www.lawinfo/ .com; webmaster@lawinfo.com) Lawinfo Lemur Consulting larbin_2.6.2 (tom@lemurconsulting.com) Lemur Consulting Lexibot Mata Hari/2.00 BrightPlanet LexiBot/1.00 Liafa larbin_2.2.2_guillaume (guillaume@liafa.jussieu.fr) Liafa LibWeb libWeb/clsHTTP — hiongun@kt.co.kr Korea Telecom LibertyW LibertyW (+http://www.lw01.com) LibertyW LibertyW (+http://www.libertyw.eu) LijitSpider LijitSpider/Nutch-0.9 (Reports crawler; http://www.lijit.com/; info(a)lijit(d)co m) Lijit LinkWalker LinkWalker Seven TwentyFour Linknzbot linknzbot LinkNZ Links2Go Mozilla/3.01 (Compatible; Links2Go Similarity Engine) Links2Go Links4US-Crawler Links4US-Crawler, (+http://links4us.com/) Links4US LinksManager.com_bot Mozilla/5.0 (compatible; LinksManager.com_bot +http://linksmanager.com/linkcheck er.html) Unknown Llaut Llaut/1.0 (http://mnm.uib.es/~gallir/llaut/bot.html) Universitat de les Illes Balears Lmspider lmspider (lmspider@scansoft.com) Nuance LocalBot LocalBot/1.0 ( http://www.localbot.co.uk/) LocalBot Lockstep Spider Lockstep Spider/1.0 Lockstep Look.com NetResearchServer(http://www.look.com/)   LookdirBot LookdirBot Lookdir Lovel Lovel as 1.0 ( +http://www.everatom.com) Everatom Ltaa_web_crawler larbin_2.6.3 (ltaa_web_crawler@groupes.epfl.ch) Ecole Polytechnique F?d?rale de Lausanne Luchs.at URL checker luchs.at URL checker Luchs Lycos_Spider Lycos_Spider_(modspider) Lycos Mozilla/4.0 (compatible; MSIE 5.0; Windows 98;Lycos_Spider_(T-Rex) ; Lycos_Spide r_(T-Rex) ) Mozilla/4.0 (compatible; MSIE 5.0; Windows 98;Lycos_Spider_Beta2(T-Rex) ; Lycos_ Spider_Beta2(T-Rex) ) Lycos_Spider_(T-Rex) MJ12bot MJ12bot/v1.0.8 (http://majestic12.co.uk/bot.php?+) Majestic 12 MJ12bot/v1.2.0 (http://majestic12.co.uk/bot.php?+) MJ12bot/v1.1.1 (http://majestic12.co.uk/bot.php?+) Mozilla/5.0 (compatible; MJ12bot/v1.2.3; http://www.majestic12.co.uk/bot.php?+) MJ12bot/vx.x.x (http://www.majestic12.co.uk/projects/dsearch/mj12bot.php) MJ12bot/v1.1.2 (http://majestic12.co.uk/bot.php?+) MJ12bot/v1.0.7 (http://majestic12.co.uk/bot.php?+) MQBOT MQBOT/Nutch-0.9-dev (MQBOT Nutch Crawler; http://falcon.cs.uiuc.edu/; mqbot@cs.ui uc.edu) University of Illinois MSN Bot msnbot-media/1.1 (+http://search.msn.com/msnbot.htm) MSN msnbot/1.1 (+http://search.msn.com/msnbot.htm) msnbot/0.3 (+http://search.msn.com/msnbot.htm) msnbot/0.9 (+http://search.msn.com/msnbot.htm) msnbot/1.0 (+http://search.msn.com/msnbot.htm) msnbot-Products/1.0 (+http://search.msn.com/msnbot.htm) msnbot-media/1.0 (+http://search.msn.com/msnbot.htm) Mozilla/4.0 (compatible; MSIE 6.0; Windows NT; MS Search 4.0 Robot) MSNBOT_Mobile MSNBOT_Mobile MSMOBOT Mozilla/2.0 (compatible; MSIE 4.02; Windows CE; Default) MSN MSRBOT MSRBOT (http://research.microsoft.com/research/sv/msrbot) Microsoft MSRBot MSRBOT (http://research.microsoft.com/research/sv/msrbot/) Microsoft MSRBOT (http://research.microsoft.com/research/sv/msrbot/ MaSagool MaSagool/1.0 (MaSagool; http://sagool.jp/; info@sagool.jp) Sagool Mail.Ru Mail.Ru/1.0 Mail.Ru Mainseek_Bot Mozilla/5.0 (compatible;MAINSEEK_BOT) Mainseek Mammoth Mozilla/5.0 (+http://www.sli-systems.com/) Mammoth/0.1 SLI Systems mammoth/1.0 (+http://www.sli-systems.com/) Mozilla/5.0 (+http://www.eurekster.com/mammoth) Mammoth/0.1 MantraAgent MantraAgent LookSmart Mariner Mariner/5.1b [de] (Win95; I ;Kolibri gncwebbot) Kolibri Martini Martini LookSmart MARTINI Marvin Marvin v0.3 Health On the Net Fondation Masterseek MasterSeek Masterseek Maxbot Spider/maxbot.com admin@maxbot.com Maxbot Maxomobot maxomobot/dev-20051201 (maxomo; http://67.102.134.34:4047/MAXOMO/MAXOMObot.html; maxomobot@maxomo.com) Maxomo MediaCrawler MediaCrawler-1.0 (Experimental) Media Find MediaSearch MediaSearch/0.1 http://www.fi/ Mediater Rechercher libwww/5.3.2 Mediater MegaSheep MegaSheep v1.0 (http://www.searchuk.com/ internet sheep) Search UK Megaglobe Crawler Mozilla/5.0 (compatible; Megaglobe Crawler/1.0; http://www.megaglobe.com/) Megaglobe Melbot WebSpider Melbot WebSpider & RSS News Crawler http://www.melbot.info/ (V.2.42 by A.I.C.E.) Melbot Mercator Mercator-1.x Altavista Mercator-Scrub-1.1 Mercator-2.0 Merl.com larbin_2.1.1 larbin2.1.1@somewhere.com Mitsubishi Electrical Research Lab MetaGer_PreChecker MetaGer_PreChecker0.1 MetaGer Metacarta Mozilla/5.0 (compatible; heritrix/1.5 +http://www.metacarta.com) Metacarta Mozilla/4.0 (compatible; MSIE 5.01; Windows NT 5.0) (samualt9@bigfoot.com) Metaeuro Web Crawler Metaeuro Web Crawler/0.2 (MetaEuro Web Search Clustering Engine; http://www.meta/ euro.com; crawler at metaeuro dot com) Metaeuro Metager-Linkchecker MetaGer-LinkChecker Metager MetagerBot MetagerBot/0.8-dev (MetagerBot; http://metager.de/; ) Metager Metaquerier MQbot http://metaquerier.cs.uiuc.edu/crawler University of Illinois MQbot metaquerier.cs.uiuc.edu/crawler Metaspinner Metaspinner/0.01 (Metaspinner; http://www.meta-spinner.de/; support@meta-spinner .de/) Metaspinner Metatagsdir metatagsdir/0.7 (+http://metatagsdir.com/directory/) Metatagsdir Microsoft Small Business Indexer Microsoft Small Business Indexer Microsoft Microsoft URL Control Microsoft URL Control – 6.00.8862 Unknown Microsoft URL Control – 6.01.9782 Miggibot A1 Sitemap Generator/1.0 (+http://www.micro-sys.dk/products/sitemap-generator/) miggibot/2006.01.24 Micro-sys Mirar Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; Mirar Search Indexing Agent s ee getmirar.com) Mirar Missigua Locator 1.9 Missigua Locator 1.9 Unknown Misterbot Misterbot-Nutch/0.7.1 (Misterbot-Nutch; http://www.misterbot.fr/; admin@misterbot .fr) Misterbot Miva Miva (AlgoFeedback@miva.com) Miva MnoGoSearch MnogoSearch/3.2.xx MnoGoSearch Mo College 1.9 Mo College 1.9 Unknown Moget moget/x.x (moget@goo.ne.jp) Goo Mogimogi mogimogi/1.0 Goo MojeekBot Mozilla/5.0 (compatible; MojeekBot/2.0; http://www.mojeek.com/bot.html) Mojeek MojeekBot/0.x (archi; http://www.mojeek.com/bot.html) Monogol http://www.monogol.de/ Monogol Moreoverbot Moreoverbot/5.00 (+http://www.moreover.com) Moreover Morris Morris – Mixcat Crawler (+http://mixcat.com) MixCat Mowserbot Mozilla/4.0 (compatible; MSIE 5.5; Windows NT 5.0; mowserbot; http://www.mowser/. com/bot) Mowser MozDex mozDex/0.04-dev (mozDex; http://www.mozdex.com/en/bot.html; spider@mozdex.com) MozDex Mozdex Mozdex/0.06-dev (Mozdex; http://www.mozdex.com/bot.html; spider@mozdex.com) MozDex MultiText MultiText/0.1 Virginia Polytechnic Institute and State University Multicrawler multicrawler (+http://sw.deri.org/2006/04/multicrawler/robots.html) SWSE multicrawler ( http://sw.deri.org/2006/04/multicrawler/robots.html) MuscatFerret Mozilla/3.0 (compatible; MuscatFerret/1.6.x; claude@euroferret.com) Euro Ferret Mozilla/3.0 (compatible; MuscatFerret/1.5.4; claude@euroferret.com) Mozilla/3.0 (compatible; MuscatFerret/1.5; olly@muscat.co.uk) MusicWalker MusicWalker2.0 (+http://www.somusical.com) SoMusical! My-bytebot my-bytebot/1.0 (+http://spider.my-byte.de/info) My-byte MyFamilyBot Mozilla/4.0 (compatible; MyFamilyBot/1.0; http://www.ancestry.com/learn/bot.aspx ; SearchBot@MyFamilyInc.com) Ancestry Mylinea Mylinea.com Crawler 2.0 Mylinea NASA Search 1.0 NASA Search 1.0 Unknown NCSA NCSA Beta 1 (http://vias.ncsa.uiuc.edu/viasarchivinginformation.html) National Center for Supercomputing Applications NG-SearchBot NG-Search/0.90 (NG-SearchBot; http://www.ng-search.com/; ) NG-Search NG-Search/0.9.8 (http://www.ng-search.com/) NII larbin_2.6.2 (hamasaki@grad.nii.ac.jp) National Institut of Informatics (Japan) NLCrawler Mozilla/5.0 (compatible; NLCrawler/2.0.27; Linux 2.6.3-7; i686; en_US)KHTML/3.4. 89 (like Gecko) Northern Light NMG Spider NMG Spider/0.3 (szukanko.com) Szukanko NTT Directory nttdirectory_robot/0.9 (super-robot@super.navi.ocn.ne.jp) NTT Directory NWSpider NWSpider 0.9 NWSpider Nabot nabot_1.0 Naver NABOT/5.0 Najdi Mozilla/5.0 (compatible; InterseekWeb/3.x) Najdi Nameprotect NPBot (http://www.nameprotect.com/botinfo.html) Nameprotect NPBot-1/2.0 NP/0.1 (NP; http://www.nameprotect.com/; npbot@nameprotect.com) Nationaldirectory NationalDirectoryAddURL/1.0 Nationaldirectory NationalDirectory-WebSpider/1.3 NatzanBot NatzanBot, http://www.natzan.com/ Natzan NaverBot NaverBot-1.0 (NHN Corp. / +82-2-3011-1954 / nhnbot@naver.com) Naver NaverBot_dloader/1.5 dloader(NaverRobot)/1.0 NaverBot-1.0 (NHN Corp. / +82-31-784-1989 / nhnbot@naver.com) Mozilla/4.0 (compatible; NaverBot/1.0; http://help.naver.com/delete_main.asp) NavissoBot NavissoBot/1.7 (+http://navisso.com/) Navisso NavissoBot Nebulla Nebullabot/2.2 (http://bot.nebulla.info/) Nebulla Nebullabot/2.2 (http://bot.nebulla.de/) Nebulla/V1.0 Spider (http://spider.nebulla.de/) Nelian Pty Ltd – Spider Nelian Pty Ltd – Spider v2.1 ( http://pcaccessoriesparts.com/ ) Pcaccessoriesparts NetNose Mozilla/4.0 (compatible; MSIE 5.0; NetNose-Crawler 2.0; A New Search Experience: http://www.netnose.com/) NetNose NetResearchServer NetResearchServer/4.0(loopimprovements.com/robot.html) Loopimprovements NetSprint — 2.0 NetSprint — 2.0 Wirtualna Polska NetWhatCrawler NetWhatCrawler/0.06-dev (NetWhatCrawler from NetWhat.com; http://www.netwhat.com/ ; support@netwhat.com) Netwhat NetinfoBot NetinfoBot/1.0 (http://netinfo.bg/netinfobot.html) NetinfoBot Netluchs Netluchs/0.8-dev ( ; http://www.netluchs.de/; ___don’t___spam_me_@netluchs.de) Netluchs Netprospector Netprospector JavaCrawler Netprospector Netscape Robozilla/1.0 DMOZ NextGenSearchBot NextGenSearchBot 1 (for information visit http://about.zoominfo.com/About/NextGe nSearchBot.aspx) Zoominfo NextGenSearchBot 1 (for information visit http://www.eliyon.com/NextGenSearchBot ) NextGenSearchBot 1 (for information visit http://www.zoominfo.com/NextGenSearchB ot) NextopiaBot NextopiaBOT (+http://www.nextopia.com) distributed crawler client beta v1.1 Nextopia NextopiaBOT (+http://www.nextopia.com) distributed crawler client beta v0.x NimbleCrawler Mozilla/5.0 (Windows;) NimbleCrawler 2.0.0 obeys UserAgent NimbleCrawler For pro blems contact: crawler@healthline.com Healthline Mozilla/5.0 (Windows;) NimbleCrawler 1.12 obeys UserAgent NimbleCrawler For prob lems contact: crawler@healthline.com Mozilla/5.0 (Windows; U; Windows NT 5.0; en-US; rv:1.7.7) NimbleCrawler 1.11 obe ys UserAgent NimbleCrawler For problems contact: crawler_at_dataalchemy.com Mozilla/5.0 (Windows;) NimbleCrawler 2.0.1 obeys UserAgent NimbleCrawler For pro blems contact: crawler@healthline.com Noago Spider Noago Spider Noago NokodoBot NokodoBot/1.x (+http://nokodo.com/bot.htm) Nokodo Norbert Norbert the Spider(Burf.com) Burf Noxtrumbot noxtrumbot/1.0 (crawler@noxtrum.com) Noxtrum Noyona noyona_0_1 Noyona Nsyght nsyght.com/Nutch-0.9 (nsyght.com; search.nsyght.com) Nsyght NuSearch Spider nuSearch Spider http://www.nusearch.com/ (compatible ; MSIE 4.01) NuSearch Nutch Nutch Lucene NutchCVS/0.0x-dev (Nutch; http://www.nutch.org/docs/bot.html; nutch-agent@lists. sourceforge.net) NutchOrg/0.0x-dev (Nutch; http://www.nutch.org/docs/bot.html; nutch-agent@lists. sourceforge.net) NutchCVS/0.06-dev (Nutch; http://www.nutch.org/docs/en/bot.html; nutch-agent@lis ts.sourceforge.net) NutchCVS/0.8-dev NutchCVS/0.8-dev (Nutch running at UW; http://www.nutch.org/docs/en/bot.html; sy crawl@cs.washington.edu) University of Washington NutchEC2Test NutchEC2Test/Nutch-0.9-dev (Testing Nutch on Amazon EC2.; http://lucene.apache.o/ rg/nutch/bot.html; ec2test at lucene.com) Amazon Obidos-bot obidos-bot (just looking for books.) Weblog bookwatch Object Sciences Corp. Mozilla/4.0 (compatible; MSIE 5.5; Windows NT 5.0; T312461) RPT-HTTPClient/0.3-3 E   Objects Search ObjectsSearch/0.01-dev (ObjectsSearch;http://www.ObjectsSearch.com/bot.html; sup port@thesoftwareobjects.com) SAIC Ocelli Ocelli/1.x (http://www.globalspec.com/Ocelli) GlobalSpec Ocelli/1.3 (http://www.globalspec.com/Ocelli) Octora Octora Beta – http://www.octora.com/ Octora Octora Beta Bot Octora Beta Bot – http://www.octora.com/ Octora OmniExplorer OmniExplorer_Bot/1.0x (+http://www.omni-explorer.com) Internet Categorizer OmniExplorer OmniExplorer_Bot/1.1x (+http://www.omni-explorer.com) Torrent Crawler OmniExplorer_Bot/x.xx (+http://www.omni-explorer.com) WorldIndexer OmniExplorer_Bot/1.0x (+http://www.omni-explorer.com) Job Crawler OnetSzukaj Onet.pl SA, http://szukaj.onet.pl/ Szukaj Mozilla/5.0 (compatible; OnetSzukaj/5.0; +http://szukaj.onet.pl) OpenISearch OpenISearch/1.x (http://www.openisearch.com/) OpenISearch OpenPortal4U DigOut4U Arisem OpenTaggerBot OpenTaggerBot (http://www.opentagger.com/opentaggerbot.htm) OpenTagger OpenTextSiteCrawler OpenTextSiteCrawler/2.9.2 OpenText OpenWebSpider OpenWebSpider/x OpenWebSpider Openfind Openbot/3.0+(robot-response@openfind.com.tw;+http://www.openfind.com.tw/robot.ht ml) Openfind Openfind Robot/1.1A2 Openfind data gatherer, Openbot/3.0+(robot-response@openfind.com.tw;+http://www. openfind.com.tw/robot.html) OpidooBot OpidooBOT (larbin2.6.3@unspecified.mail) Opidoo Oracle Oracle Ultra Search Oracle OrangeSpider OrangeSpider Orangeslicer Orbiter Orbiter/T-2.0 (+http://www.dailyorbit.com/bot.htm) DailyOrbit Overture-WebCrawler Overture-WebCrawler/3.8/Fresh (atw-crawler at fast dot no; http://fast.no/suppor t/crawler.asp) FAST OzMonitor Mozilla/5.0 (Linux; fr) Firefox (ozMonitor Free) OzMonitor Ozelot ozelot/2.7.3 (Search engine indexer; www.flying-cat.de/ozelot; ozelot@flying-cat .de) Flying-cat PJspider PJspider/3.0 (pjspider@portaljuice.com; http://www.portaljuice.com/) Nextopia PWeBot PWeBot/1.2 Inspector (http://www.programacionweb.net/robot.php) Programacionweb Mozilla/5.0 (compatible; PWeBot/3.1; http://www.programacionweb.net/robot.php) Page-store Mozilla/5.0 (compatible; heritrix/1.12.1 +http://www.page-store.com) Page-store PageBitesHyperBot PageBitesHyperBot/600 (http://www.pagebites.com/) PageBites Page_verifier page_verifier http://www.securecomputing.com/goto/pv Securecomputing Pagebull Pagebull http://www.pagebull.com/ Pagebull Pages Jaunes FAST Enterprise Crawler 6 used by Pages Jaunes (crawladmin@gmail.com) Pages Jaunes FAST Enterprise Crawler 6 used by Pages Jaunes (pvincent@pagesjaunes.fr) FAST Enterprise Crawler 6 used by Pages Jaunes (fastadmin@pagesjaunes.fr) PalmeraBot Mozilla/5.0 (compatible; PalmeraBot; http://www.links24h.com/help/palmera) Versi on 0.001 Links24h Pandora Mozilla/5.0 (compatible; heritrix/1.5.0-200506231921 +http://pandora.nla.gov.au/ crawl.html) National Library of Australia ParaSite ParaSite/1.0b (http://www.ianett.com/parasite/) Ianett Patwebbot Patwebbot (http://www.herz-power.de/technik.html) Patsearch Peerbot PEERbot http://www.peerbot.com/ Peerbot Petitsage http://www.petitsage.fr/ site detector 0.4 Petitsage PicoSearch PicoSearch/1.0 Pico Search PictureOfInternet PictureOfInternet/3.0 (http://www.malfunction.org/poi; erik@malfunction.org) Malfunction Piffany Piffany_Web_Spider_v0.x Piffany Piffany_Web_Scraper_v0.x Pilot Hitlist Marketwave Hit List Pilot Hitlist Pingdom Pingdom GIGRIB v1.1 (http://www.pingdom.com/) Pingdom PipeLiner pipeLiner/0.xx (PipeLine Spider; http://www.pipeline-search.com/webmaster.html) Pipeline Pita Pita Stanford University PlantyNet_WebRobot PlantyNet_WebRobot_V1.9 dhkang@plantynet.com PlantyNet PlantyNet_WebRobot_V1.9 yangsam@plantynet.com PluckFeedCrawler PluckFeedCrawler/2.0 (compatible; Mozilla 4.0; MSIE 5.5; http://www.pluck.com/; 1 subscribers) Pluck Pmoz Mozilla/5.0 (compatible; pmoz.info ODP link checker; +http://pmoz.info/doc/botin fo.htm) Pmoz Podtech Mozilla/5.0 (compatible; MSIE 6.0; Podtech Network; crawler_admin@podtech.net) Podtech Pogodak Mozilla/5.0 (compatible; pogodak.ba/3.x) Pogodak Polybot polybot 1.0 (http://cis.poly.edu/polybot/) Polytechnic University Brooklyn Pompos Pompos Iliad (Free) Pompos/1.3 http://dir.com/pompos.html Pompos/1.x pompos@iliad.fr Pompos/1.1 http://dir.com/pompos.html Pompos/1.2 http://dir.com/pompos.html Pompos/1.x http://dir.com/pompos.html PopJapanSearch Robot/www.pj-search.com PopJapanSearch Popdexter Popdexter/1.0 Popdex PrivacyFinder/1.1 PrivacyFinder/1.1 AT&T Privacy Bird Privacy Preferences ProWebguide ProWebGuide Link Checker (http://www.prowebguide.com/) ProWebguide Probe! PROBE! (Probert Encyclopaedia Research Robot V1.0 http://www.probertencyclopaedi/ a.com) Probert Encyclopaedia Psbot psbot/0.1 (+http://www.picsearch.com/bot.html) Picsearch Psycheclone psycheclone Unknown Pubblisito info@pubblisito.com– (http://www.pubblisito.com/) il Sud dei Motori di Ricerca Pubblisito Pumpkin blogsearchbot-pumpkin-3 Artofcomputing Python-urllib Python-urllib/1.16 Unknown QPCreep QPCreep Test Rig ( We are not indexing, just testing ) Quepasa QihooBot Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; QihooBot 1.0 qihoobot@qihoo.n et) Qihoo Quantcastbot Mozilla/5.0 (compatible; Quantcastbot/1.0; http://www.quantcast.com/) Quantcast Qube qube/2.0 (+http://qube.qelix.com/v2) Qube QuepasaCreep QuepasaCreep v0.9.1x Quepasa QuepasaCreep ( crawler@quepasacorp.com ) QueryN Metasearch QueryN Metasearch QueryN Qweerybot QweeryBot/3.01 ( http://qweerybot.qweery.nl/) Qweery Qweery_robot.txt_CheckBot/3.01 (http://qweerybot.qweery.com/) QweeryBot/3.02 ( http://qweerybot.qweery.nl/) R6_CommentReader R6_CommentReader_(www.radian6.com/crawler) Radian6 R6_FeedFetcher R6_FeedFetcher_(www.radian6.com/crawler) Radian6 RAMPyBot RAMPyBot/0.8-dev (Nutch; http://lucene.apache.org/nutch/bot.html; nutch-agent@lu cene.apache.org) GiveRamp RAMPyBot – www.giveRAMP.com/0.1 (RAMPyBot – http://www.giveramp.com/; http://www.giveram/ p.com/bot.html; support@giveRAMP.com) RSS One Engine RSS One Engine/0.72 (+http://www.rss-one.com) RSS One Rambot ramBot xtreme x.x Intersearch RankMeter Web Snooper SearchUtilities Rational SiteCheck Rational SiteCheck (Windows NT) GSInnova Reaper Reaper/2.0x (+http://www.sitesearch.ca/reaper) Marty Anstey Reaper [2.03.10-031204] (http://www.sitesearch.ca/reaper/) RedBot RedBot/redbot-1.0 (Rediff.com Crawler; redbot at rediff dot com) Rediff RedCarpet/1.2 RedCarpet/1.2 (http://www.redcarpet-inc.com/robots.html) Pronto RedCell RedCell/0.1 (RedCell; telegenetic.net/bot.html; lhall_at_telegenetic.net) Telegenetic RedKernel RedKernel WWW-Spider 2/0 (+http://www-spider.redkernel-softwares.com/) RedKernel RoboPal RoboPal (http://www.findpal.com/) FindPal Rotondo Rotondo/3.1 libwww/5.3.1 QualiGo RufusBot RufusBot (Rufus Web Miner; http://www.webaroo.com/rooSiteOwners.html) Webaroo RutterBot RutterBot(+http://www.aktienbetreuer.de/bot.html) Aktienbetreuer SAIT sait/Nutch-0.9 (SAIT Research; http://www.samsung.com/) Samsung SBIder SBIder/0.8-dev (SBIder; http://www.sitesell.com/sbider.html; http://support.site/ sell.com/contact-support.html) Sitesell SBIder/SBIder-0.8.2-dev (http://www.sitesell.com/sbider.html) SBIder/0.7 (SBIder; http://www.sitesell.com/sbider.html; http://support.sitesell/ .com/contact-support.html) SandCrawler SandCrawler – Compatibility Testing Microsoft Savvybot savvybot/0.2 WebSavvy ScanWeb ScanWeb Eserver Schibstedsokbot schibstedsokbot (compatible; Mozilla/5.0; MSIE 5.0; FAST FreshCrawler 6; +http:/ /www.schibstedsok.no/bot/) Schibsted ScholarUniverse ScholarUniverse/0.8 (Nutch;+http://scholaruniverse.com/bot.jsp; fetch-agent@scho laruniverse.com) ScholarUniverse Scirus-crawler FAST Enterprise Crawler 6 / Scirus scirus-crawler@fast.no; http://www.scirus.com/ /srsapp/contactus/ Scirus ScollSpider ScollSpider/2.0 (+http://www.webwobot.com/ScollSpider.php) Webwobot Mozilla/3.0 (compatible; ScollSpider; http://www.webwobot.com/) ScorpionBot Fooky.com/ScorpionBot/ScoutOut;http://www.fooky.com/scorpionbots Fooky ScoutAnt ScoutAnt/0.1; +http://www.ant.com/what_is_ant.com/ Ant Scrubby Scrubby/2.x (http://www.scrubtheweb.com/) Scrub the web Mozilla/5.0 (compatible; Scrubby/2.2; http://www.scrubtheweb.com/) Sdcresearchlabs-testbot sdcresearchlabs-testbot/0.8-dev (www.shopping.com/bot.html; http://lucene.apache/ .org/nutch/bot.html; researchbot@shopping.com) Shopping Search-Engine-Studio Search-Engine-Studio Xtreeme Search-Info Spider-Sleek/2.0 (+http://search-info.com/linktous.html) Search-Info Search.ch search.ch V1.4 Search CH search.ch V1.4.2 (spiderman@search.ch; http://www.search.ch/) Search4free lwp-trivial/1.34 Search4free SearchByUsa SearchByUsa/2 (SearchByUsa; http://www.SearchByUsa.com/bot.html; info@SearchByUs a.com) Search4USA SearchEngineWorlds ( Robots.txt Validator http://www.searchengineworld.com/cgi-bin/robotcheck.cgi )   SearchEngineWorlds SearchExpress Spider SearchExpress Spider0.99 SearchExpress SearchScout ClariaBot/1.0 SearchScout Diamond/1.0 DiamondBot SearchSight SearchSight/2.0 (http://SearchSight.com/) SearchSight SearchSpider Searchspider/1.2 (SearchSpider; http://www.searchspider.com/; webmaster@searchspi der.com) SearchSpider SearchSpider.com/1.1 SearchdayBot SearchdayBot Searchday Searchguild SearchGuild_DMOZ_Experiment (chris@searchguild.com) SearchGuild Searchit-Now Robot Searchit-Now Robot/2.2 (+http://www.searchit-now.co.uk) Searchit-Now Searchmee! Spider Searchmee! Spider v0.98a Searchmee! Seekbot Seekbot/1.0 (http://www.seekbot.net/bot.html) RobotsTxtFetcher/1.2 SeekPort Seekbot/1.0 (http://www.seekbot.net/bot.html) HTTPFetcher/2.2 Seekbot/1.0 (http://www.seekbot.net/bot.html) HTTPFetcher/0.3 Seekbot/1.0 (http://www.seekbot.net/bot.html) RobotsTxtFetcher/1.0 (XDF) Seeker.lookseek Seeker.lookseek.com Lookseek Semager Semager/1.0 (http://www.semager.de/) Semager Semager/1.1 (http://www.semager.de/blog/semager-bots/) Sensis.com.au Web Crawler Sensis.com.au Web Crawler (search_commentsatsensisdotcomdotau) Sensis SeznamBot SeznamBot/2.0-test (+http://fulltext.sblog.cz/) Seznam SeznamBot/1.0 (+http://fulltext.seznam.cz/) SeznamBot/1.0 SharewarePlaza Agent-SharewarePlazaFileCheckBot/2.0+(+http://www.SharewarePlaza.com) SharewarePlaza Sherlock sherlock/1.0 Indiana University School of Informatics Shim-Crawler Shim-Crawler(NICT)(Mozilla-compatible; http://www.logos.ic.i.u-tokyo.ac.jp/crawl er/; crawler@logos.ic.i.u-tokyo.ac.jp) Chikayama-Taura laboratory Shim-Crawler(Mozilla-compatible; http://www.logos.ic.i.u-tokyo.ac.jp/crawler/; c rawl@logos.ic.i.u-tokyo.ac.jp) Shim-Crawler(Mozilla-compatible; http://www.logos.ic.i.u-tokyo.ac.jp/crawl/; cra wl@logos.ic.i.u-tokyo.ac.jp) ShopWiki ShopWiki/1.0 ( +http://www.shopwiki.com/) Littlewiki ShopWiki/1.0 ( +http://www.shopwiki.com/wiki/Help:Bot) Shoula Shoula.com Crawler 2.0 Shoula! Search SietsCrawler SietsCrawler/1.1 (+http://www.siets.biz) Siets Siigle Orumcex Siigle Orumcex v.001 Turkey (http://www.siigle.com/) Siigle Silk/1.0 silk/1.0 Slider silk/1.0 (+http://www.slider.com/silk.htm)/3.7 Sirketcebot Sirketcebot/v.01 (http://www.sirketce.com/bot.html) Sirketce Site Server 3.0 Robot Mozilla/4.0 (compatible; MSIE 4.0; Windows NT; Site Server 3.0 Robot) ACR American College of Radiology SiteBar SiteBar/3.3.8 (Bookmark Server; http://sitebar.org/) SiteBar SiteBaseBot Mozilla/5.0 (compatible; SiteBaseBot/1.0; http://dir.sitebase.ru/) SiteBase SiteSpider SiteSpider +(http://www.SiteSpider.com/) SiteSpider SiteTruth SiteTruth.com site rating system SiteTruth SitiDiBot +SitiDi.net/SitiDiBot/1.0 (+Have Good Day) SitiDi.net Skampy Mozilla/4.0 (compatible; MSIE 6.0; MSIE 5.5; Windows NT 5.1) Skampy/0.9.x [en] Skaffe Skizzle User-Agent: Mozilla/4.0 (SKIZZLE! Distributed Internet Spider v1.0 – http://www.skizzle/ .com) Skizzle SkreemRBot Mozilla/5.0 (compatible; SkreemRBot +http://skreemr.com) SkreemR Slider Slider_Search_v1-de Slider Slurp (Yahoo) Mozilla/5.0 (compatible; Yahoo! DE Slurp; http://help.yahoo.com/help/us/ysearch/ slurp) Yahoo Slurp China (Yahoo) Mozilla/5.0 (compatible; Yahoo! Slurp China; http://misc.yahoo.com.cn/help.html)   Yahoo Slurp Inktomi Mozilla/5.0 (Slurp/si; slurp@inktomi.com; http://www.inktomi.com/slurp.html) Hotbot-Lycos,NBCi etc. Mozilla/5.0 (Slurp/cat; slurp@inktomi.com; http://www.inktomi.com/slurp.html) Mozilla/3.0 (Slurp/si; slurp@inktomi.com; http://www.inktomi.com/slurp.html) Mozilla/3.0 (Slurp/cat; slurp@inktomi.com; http://www.inktomi.com/slurp.html) Slurp Inktomi (Yahoo) Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slu rp) Yahoo Mozilla/5.0 (Macintosh; U; PPC Mac OS X Mach-O; en-US; rv:1.8.1.5) Gecko/2007071 3 Firefox/2.0.0.5 Mozilla/5.0 (compatible; Yahoo! Slurp/3.0; http://help.yahoo.com/help/us/ysearch /slurp) Mozilla/5.0 (X11; U; Linux i686 (x86_64); en-US; rv:1.8.1.4) Gecko/20071214 BonE cho/2.0.0.4 Slurpy Verifier Slurpy Verifier/1.0 Yahoo SmiffyDCMetaSpider SmiffyDCMetaSpider/1.0 Smiffysplace Snapbot Snapbot/1.0 Hotlinks Snoopy Snoopy v1.2 Unknown SnykeBot SnykeBot/0.6 (http://www.snyke.com/) Snyke SoftHypermarket SoftHypermarketFileCheckBot/1.0+(+http://www.softhypermaket.com) SoftHypermarket Sogou web spider Sogou web spider/3.0(+http://www.sogou.com/docs/help/webmasters.htm#07) Sogou Sohu-search sohu-search Sohu Somewhere Mozilla (Mozilla@somewhere.com) Somewhere Spam Bot Mozilla/2.0 (compatible; NEWT ActiveX; Win32) Unknown Speedfind speedfind ramBot xtreme 8.1 Lotse Speedy Spider Speedy Spider (Entireweb; Beta/1.3; http://www.entireweb.com/about/search_tech/s peedyspider/) Entireweb Speedy Spider (Entireweb; Beta/1.2; http://www.entireweb.com/about/search_tech/s peedyspider/) Speedy Spider (Beta/1.0; http://www.entireweb.com/) Speedy Spider (Entireweb; Beta/1.0; http://www.entireweb.com/about/search_tech/s peedyspider/) Speedy Spider (http://www.entireweb.com/about/search_tech/speedyspider/) Speedy Spider (Entireweb; Beta/1.1; http://www.entireweb.com/about/search_tech/s peedyspider/) Sphere Scout Sphere Scout&v4.0 – scout at sphere dot com Sphere Sphsearch FAST Enterprise Crawler 6 used by Singapore Press Holdings (crawler@sphsearch.sg ) Singapore Press Holdings SpiderMonkey Mouse-House/7.4 (spider_monkey spider info at www.mobrien.com/sm.shtml) SpiderMonkey SpiderMonkey/7.0x (SpiderMonkey.ca info at http://spidermonkey.ca/sm.shtml) SplatSearch libwww-perl/5.45 Splat Spock Crawler Spock Crawler (http://www.spock.com/crawler) Spock Sproose sproose/0.1-alpha (sproose crawler; http://www.sproose.com/bot.html; crawler@spr oose.com) Sproose sproose/1.0beta (sproose bot; http://www.sproose.com/bot.html; crawler@sproose.c om) StackRambler StackRambler/2.0 Rambler StackRambler/x.x StackRambler/2.0 (MSIE incompatible) Steeler Steeler/1.x (http://www.tkl.iis.u-tokyo.ac.jp/~crawler/) University of Tokyo Steeler/3.3 (http://www.tkl.iis.u-tokyo.ac.jp/~crawler/) Strategic Board Bot Strategic Board Bot (+http://www.strategicboard.com) Strategic Board Suchbaer suchbaer.de (CrawlerAgent v0.103) Suchbaer suchbaer.de Suchpad suchpad/1.0 (+http://www.suchpad.de) Suchpad SummizeBot Mozilla/5.0 (compatible; SummizeBot +http://www.summize.com) Summize SuperSnooper Robot@SuperSnooper.Com SuperSnooper SurveyBot SurveyBot/2.3 (Whois Source) Domaintools Susie !Susie (http://www.sync2it.com/susie) Sync2it Swooglebot Swooglebot/2.0. (+http://swoogle.umbc.edu/swooglebot.html) University of Maryland SycaBoT SycaBoT/1.0 SycaroX SycaBoT-Audio SycaBoT-Audio SycaroX SycaBoT-Image SycaBoT-Image SycaroX SycaBoT-Programme SycaBoT-Programme SycaroX SycaBoT-Video SycaBoT-Video SycaroX Sycrawl NutchCVS/0.7.1 (Nutch running at UW; http://www.nutch.org/docs/en/bot.html; sycr awl@cs.washington.edu) University of Washington Sygol <http://www.sygol.com/> http://www.sygol.com/ Sygol SygolBot SygolBot http://www.sygol.com/ Sygol SygolBot http://www.sygol.net/ SynooBot SynooBot/Mozilla/5.0 (compatible; Synoobot/0.9; http://www.synoo.com/search/bot. html) Synoo Mozilla/5.0 (compatible; Synoobot/0.9; http://www.synoo.com/search/bot.html) Syntryx Syntryx ANT Scout Chassis Pheromone; Mozilla/4.0 compatible crawler Syntryx Szukacz Szukacz/1.x (robot; www.szukacz.pl/jakdzialarobot.html; szukacz@proszynski.pl) Szukacz Szukacz/1.5 (robot; www.szukacz.pl/jakdzialarobot.html; info@szukacz.pl) Szukacz/1.5 (robot; www.szukacz.pl/html/jak_dziala_robot.html; info@szukacz.pl) TAGword Tagword (http://tagword.com/dmoz_survey.php) TAGword DMOZ survey TCDBOT TCDBOT/Nutch-0.8 (PhD student research;”http://www.tcd.ie/; mcgettrs at t c d dot IE)” Trinity College Dublin Tags2dir tags2dir.com/0.8 (+http://tags2dir.com/directory/) Tags2dir TailRank Robot Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.2.1; aggregator:TailRank; http://ta/ ilrank.com/robot) Gecko/20021130 TailRank Talkro Talkro Web-Shot/1.0 (E-mail: webshot@daumsoft.com, Home: http://222.122.15.190/w ebshot) DaumSoft TargetSeek Mozilla/4.0 (compatible; MSIE 6.0; TargetSeek/1.0; +http://www.targetgroups.net/ TargetSeek.html) Targetgroups Technoratibot Technoratibot/0.7 Technorati Tecomi Bot Tecomi Bot (http://www.tecomi.com/bot.htm) Tecomi Teemer Teemer (NetSeer, Inc. is a Los Angeles based Internet startup company.; http://w/ ww.netseer.com/crawler.html; crawler@netseer.com) Netseer TelenetDigger TelenetDigger/1.0; htdig/3.1.6 (webdev@staff.telenet.be) Telenet Teoma teoma_agent1 Teoma Teoma MP teomaagent crawler-admin@teoma.com teomaagent1 [crawler-admin@teoma.com] Mozilla/2.0 (compatible; Ask Jeeves/Teoma) Teradex Mapper Teradex Mapper; mapper@teradex.com; http://www.teradex.com/ Teradex TeragramCrawler TeragramCrawler Teragram TeragramWebcrawler TeragramWebcrawler/1.0 Teragram TerrawizBot TerrawizBot/1.0 (+http://www.terrawiz.com/bot.html) Terrawiz TheSuBot TheSuBot/0.1 (http://www.thesubot.de/) TheSuBot Theme Spider Theme Spider (+http://www.themespider.com/spider.html) Themespider Theophrastus Mozilla/5.0 (compatible; Theophrastus/1.1; http://users.cs.cf.ac.uk/N.A.Smith/th eophrastus.php) N.A.Smith Thumbshots-de-bot thumbshots-de-Bot (Version: 1.02, powered by http://www.thumbshots.de/) Thumbshots Thunderstones Webinator Mozilla/2.0 (compatible; T-H-U-N-D-E-R-S-T-O-N-E) Thunderstones TimboBot timboBot/0.9 http://www.breakingblogs.com/timbo_bot.html Breakingblogs TivraSpider tivraSpider/1.0 (crawler@tivra.com) AT&T Tkensaku Tkensaku/x.x(http://www.tkensaku.com/q.html) Tkensaku ToileBot Mozilla/5.0 (+http://www.toile.com/) ToileBot/0.1 La Toile du Qu?bec Topodia Topodia/1.2-dev (Topodia – Crawler for HTTP content indexing; http://www.topodia/ .com/; support@topodia.com) Topodia Toutatis Toutatis x-xx.x (hoppa.com) Hoppa Toutatis x.x (hoppa.com) Toutatis x.x-x Traazibot traazibot/testengine (+http://www.traazi.de) Traazi Trampelpfad Trampelpfad-Spider Trampelpfad Trampelpfad-Spider-v0.1 TrendTech http://www.trendtech.dk/spider.asp) TrendTech Truveo Mozilla/5.0 (compatible; heritrix/1.4t http://www.truveo.com/) Truveo Turnitin TurnitinBot Turnitin TurnitinBot TurnitinBot/x.x (http://www.turnitin.com/robot/crawlerinfo.html) Turnitin Turnpike Emporium Turnpike Emporium LinkChecker/0.1 Turnpike Emporium Directory TutorGigBot TutorGig/1.5 (+http://www.tutorgig.com/crawler) TutorGig Tutorial Crawler 1.4 (http://www.tutorgig.com/crawler) Twiceler Twiceler www.cuill.com/twiceler/robot.html Cuill Mozilla/5.0 (Twiceler-0.9 http://www.cuil.com/twiceler/robot.html) Twiceler-0.9 http://www.cuil.com/twiceler/robot.html Twiceler www.cuil.com/twiceler/robot.html Twiceler-0.9 http://www.cuill.com/twiceler/robot.html Mozilla/5.0 (Twiceler-0.9 http://www.cuill.com/twiceler/robot.html) TygoBot TygoProwler Tygo TygoBot TÜzilla Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0; ODP entries t_st; http://tuez/ illa.de/t_st-odp-entries-agent.html) TÜzilla UKWizz Mackster( http://www.ukwizz.com/ ) UKWizz UKWizz/Nutch-0.8.1 (UKWizz Nutch crawler; http://www.ukwizz.com/) UN OCHA FAST Enterprise Crawler 6 used by suckling@un.org (UN OCHA) UN OCHA UOLCrawler UOLCrawler (soscrawler@uol.com.br) UOL URL-Spider Mozilla/5.0 URL-Spider URL-Spider URL_Spider_Pro URL_Spider_Pro/x.x+(http://www.innerprise.net/usp-spider.asp) Innerprise URL_Spider_Pro/x.x URL Spider Pro/x.xx (innerprise.net) USyd-NLP-Spider USyd-NLP-Spider (http://www.it.usyd.edu.au/~vinci/bot.html) University of Sydney Ultraseek Ultraseek Go UnChaosBot UnChaosBot, From Chaos To Order, UnChaos Hybrid Web Search Engine at http://www.unc/ haos.com (info@unchaos.com) UnCHAOS http://www.unchaos.com/“> UnChaos Bot, Hybrid Web Search Engine. ( vadim_gonchar@unchaos.com) http://www.unchaos.com/“> UnChaos , From Chaos To Order, Hybrid Web Search Engine. (vadim_gonchar@unchaos.com) Uni-koblenz http://www.uni-koblenz.de/~flocke/robot-info.txt University of Koblenz Unido-bot unido-bot, http://mobicom.cs.uni-dortmund.de/bot.html University of Dortmund Updated updated/0.1beta (updated.com; http://www.updated.com/; crawler@updated.om) Updated Updated crawler updated/0.1-alpha (updated crawler; http://www.updated.com/; crawler@updated.com)   Updated Uptimebot Uptimebot Uptimebot UptimeBot(http://www.uptimebot.com/) Urlchecker1.0 Mozilla/4.0 (compatible; http://www.euro-directory.com/; urlchecker1.0) Euro Directory Urlfan-bot urlfan-bot/1.0; +http://www.urlfan.com/site/bot/350.html Urlfan VBSEO Mozilla/4.0 (vBSEO; http://www.vbseo.com/) VBSEO VIProbot VIPr/0.4 (VIProbot; http://www.vipsolutions.hu/; info@vipsolutions.hu) Vipsolutions VMBot VMBot/0.7.2 (VMBot; http://www.VerticalMatch.com/; vmbot@tradedot.com) VerticalMatch VMBot/0.9 (VMBot; http://www.verticalmatch.com/; vmbot@tradedot.com) VWBOT VWBOT/Nutch-0.9-dev (VWBOT Nutch Crawler; http://vwbot.cs.uiuc.edu;+vwbot@cs.uiu/ c.edu University of Illinois Vacobot Vacobot; (+http://vaco.ws/bot.html) Vaco Vagabondo Vagabondo/3.0 (webagent at wise-guys dot nl) WiseGuys Mozilla/4.0 (compatible; Vagabondo/4.0Beta; webcrawler at wise-guys dot nl; http ://webagent.wise-guys.nl/) Mozilla/4.0 (compatible; Vagabondo/2.2; webcrawler at wise-guys dot nl; http://w/ ebagent.wise-guys.nl/) Mozilla/4.0 (compatible; Vagabondo/4.0Beta; webcrawler at wise-guys dot nl; htt p://webagent.wise-guys.nl/; http://www.wise-guys.nl/) Mozilla/4.0 (compatible; Vagabondo/4.0; webcrawler at wise-guys dot nl; http:// webagent.wise-guys.nl/; http://www.wise-guys.nl/) Vagabondo-WAP Vagabondo-WAP/2.0 (webcrawler at wise-guys dot nl; http://webagent.wise-guys.nl/ )/1.0 Profile WiseGuys Vakes Vakes/0.01 (Vakes; http://www.vakes.com/; search@vakes.com) Vakes Verizon fast enterprise crawler 6 used by verizon superpages powered by fast (kevin.watt ers@fastsearch.com) Verizon Superpages FAST Enterprise Crawler 6 used by Verizon Superpages Powered By FAST (crawler_ad min@superpages.com) Vermut Mozilla/5.0 (compatible; heritrix/@VERSION@ +http://vermut.aol.com) AOL mozilla/5.0 (compatible; vermut +http://www.aol.com) Mozilla/5.0 (compatible; vermut +http://vermut.aol.com) Versus versus 0.2 (+http://versus.integis.ch) Hochschule f?r Technik Rapperswil Versus Crawler versus crawler eda.baykan@epfl.ch Ecole Polytechnique F?d?rale de Lausanne VeryGoodSearch VeryGoodSearch.com.DaddyLongLegs VeryGoodSearch Verzamelgids verzamelgids.nl – Networking4all Bot/x.x Verzamelgids Verzamelgids/2.2 (http://www.verzamelgids.nl/) Vespa Crawler (Yahoo) Vespa Crawler Yahoo! Norway Visbot Visbot/1.0 (+http://www.visvo.com/bot.html;bot@visvo.com) Visvo VisBot/2.0 (Visvo.com Crawler; http://www.visvo.com/bot.html; bot@visvo.com) Vlad Mozilla/5.0 (compatible; vlad/1.1; +http://www.freshnotes.com/vlad.html) Freshnotes VoilaBot KE_1.0/2.0 libwww/5.2.8 Voila.fr Mozilla/5.0 (Windows; U; Windows NT 5.1; fr; rv:1.8.1) VoilaBot BETA 1.2 (suppor t.voilabot@orange-ftgroup.com) Mozilla/4.0 (compatible; MSIE 5.0; Windows 95) VoilaBot; 1.6 Mozilla/5.0 (Windows; U; Windows NT 5.1; fr; rv:1.8.1) VoilaBot BETA 1.2 (http:/ /www.voila.com/) Mozilla/4.0 (compatible; MSIE 5.0; Windows 95) VoilaBot BETA 1.2 (http://www.voi/ la.com/) Vonna.com b o t Mozilla/4.0 (compatible; MSIE 4.01; Vonna.com b o t) Vonna Vortex/2.2 Vortex/2.2 (+http://marty.anstey.ca/robots/vortex/) Marty Anstey Voyager cfetch/1.0 Kosmix voyager-hc/1.0 voyager/1.0 Vspider vspider Verity vspider/3.x W3C-Validator W3C_Validator/1.432.2.10 W3C W3C-checklink W3C-checklink/4.2 [4.20] libwww-perl/5.803 W3C W3SiteSearch Crawler W3SiteSearch Crawler_v1.1 http://www.w3sitesearch.de/ W3SiteSearch W8net Mozilla/5.0 usww.com-Spider-for-w8.net usww WEP Search WEP Search 00 Unknown WIRE WIRE/0.x (Linux; i686; Bot,Robot,Spider,Crawler) CWR WIRE/0.11 (Linux; i686; Bot,Robot,Spider,Crawler,aromano@cli.di.unipi.it) WISEbot WISEbot/1.0 (WISEbot@koreawisenut.com; http://wisebot.koreawisenut.com/) Koreawisenut http://www.fi/ crawler http://www.fi/ crawler, contact crawler@www.fi http://www.fi/ WWWeasel Robot WWWeasel Robot v1.00 (http://wwweasel.de/) World Wide Weasel Wadaino.jp-crawler wadaino.jp-crawler 0.2 (http://wadaino.jp/) Wadaino Wavefire Wavefire/0.8-dev (Wavefire; http://www.wavefire.com/; info@wavefire.com) Wavefire Waypath Waypath development crawler – info at waypath dot com Waypath Waypath Scout v2.x – info at waypath dot com WeRelateBot WeRelateBot/0.9 (WeRelate; http://www.werelate.org/wiki/WeRelate:Bot; dallan@wer elate.org) WeRelate WeatherBot WeatherBot v1.4 http://www.ezweather.net/ EZ Weather WebAlta Crawler WebAlta Crawler/1.3.15 (http://www.webalta.ru/bot.html) (Windows; U; Windows NT 5.1; ru-RU) WebAlta WebAlta Crawler/1.3.33 (http://www.webalta.net/ru/about_webmaster.html) (Windows ; U; Windows NT 5.1; ru-RU) WebAlta Crawler/1.2.1 (http://www.webalta.ru/bot.html) WebAlta Crawler/1.3.18 (http://www.webalta.net/ru/about_webmaster.html) (Windows ; U; Windows NT 5.1; ru-RU) WebBOT Carnegie_Mellon_University_WebCrawler,http://www.andrew.cmu.edu/~brgordon/webbot /index.html Carnegie Mellon University WebCorp WebCorp/1.0 WebCorp WebFindBot WebFindBot(http://www.web-find.com/) web-find WebGobbler webGobbler/1.x.x WebGobbler WebRankSpider WebRankSpider/1.37 (+http://ulm191.server4you.de/crawler/) WebRankSpider WebSearch WebSearch.COM.AU/3.0.1 (The Australian Search Engine; http://websearch.com.au/; S earch@WebSearch.COM.AU) WebSearch WebSearchBench WebSearchBench WebCrawler v0.1(Experimental) University of Dortmund WSB, http://websearchbench.cs.uni-dortmund.de/ WSB WebCrawler V1.0 (Beta), cl@cs.uni-dortmund.de WebSearchBench WebCrawler V1.0 (Beta), Prof. Dr.-Ing. Christoph Lindemann, Unive rsität Dortmund, cl@cs.uni-dortmund.de, http://websearchbench.cs.uni-dortmund.de/ / WebSearchBench WebCrawler V1.0 (Beta), Prof. Dr.-Ing. Christoph Lindemann, Unive rsit?t Dortmund, cl@cs.uni-dortmund.de, http://websearchbench.cs.uni-dortmund.de/ / WebStat WebStat/1.0 (Unix; beta; 20040314) University of South Carolina WebVac WebVac (webmaster@pita.stanford.edu) Stanford University WebarooBot WebarooBot (Webaroo Bot; http://64.124.122.252/feedback.html) Webaroo WebarooBot (Webaroo Bot; http://www.webaroo.com/rooSiteOwners.html) Webbandit webbandit/4.xx.0 SoftwareSolutions Webbot Mozilla/5.0 (compatible; Webbot/0.1; http://www.webbot.ru/bot.html) Webbot webbot(+http://webbot.com/bot.htm) Webclipping Webclipping.com Webclipping Webcrawl webcrawl.net Webcrawl webcrawl.net Webduniabot Mozilla/5.0 (compatible; Webduniabot/1.0; +http://search.webdunia.com/bot.aspx) Webdunia Webglimpse Webglimpse 2.xx.x (http://webglimpse.net/) Webglimpse Weblog Attitude Diffusion Weblog Attitude Diffusion 1.0 Los Alamos National Laboratory Webmeasurement-bot webmeasurement-bot, http://rvs.informatik.uni-leipzig.de/ University of Leipzig Webmeasurement-crawler webmeasurement-crawler, http://mobicom.cs.uni-dortmunde.de/webmeasurement/ University of Dortmund WebsiteWorth WebsiteWorth v1.0 Sootle Webspinne Webspinne/1.0 webmaster@webspinne.de Webspinne Websquash Websquash.com (Add url robot) Websquash Webster Webster v0.3 ( http://webster.healeys.net/ ) Webster.healeys Webverzeichnis Webverzeichnis.de – Telefon: 01908 / 26005 Webverzeichnis Whoiam Mozilla/5.0 whoiam [http://www.axxus.de/] Axxus Willow Willow Internet Crawler by Twotrees V2.1 Twotrees WinME Mozilla/4.0 (compatible; MSIE 5.0; Windows ME) Opera 5.11 [en] ? WinkBot WinkBot/0.06 (Wink.com search engine web crawler; http://www.wink.com/Wink:WinkB ot; winkbot@wink.com) Wink WiseGuys Mozilla/3.0 (Vagabondo/2.0 MT; webcrawler@NOSPAMwise-guys.nl; http://webagent.wi/ se-guys.nl/) WiseGuys Vagabondo/2.0 MT Mozilla/3.0 (Vagabondo/1.x MT; webagent@wise-guys.nl; http://webagent.wise-guys/. nl/) Vagabondo/1.x MT (webagent@wise-guys.nl) Vagabondo/2.0 MT (webagent at wise-guys dot nl) Vagabondo/2.0 MT (webagent@NOSPAMwise-guys.nl) Worio Mozilla/5.0 (compatible; heritrix/1.6.0 http://www.worio.com/) Worio Worio bot Mozilla/5.0 (compatible; woriobot heritrix/1.10.0 +http://worio.com) Worio Mozilla/5.0 (compatible; worio bot heritrix/1.10.0 +http://worio.com) Worio heritrix bot worio heritrix bot (+http://worio.com/) Worio WorldWideWeb-X/3.1 WorldWideWeb-X/3.1 (+http://www.worldwideweb-x.com/) WorldWideWeb Wotbox Wotbox/alpha0.x.x (bot@wotbox.com; http://www.wotbox.com/) Java/1.4.1_02 Wotbox Wotbox/alpha0.6 (bot@wotbox.com; http://www.wotbox.com/) Wume_crawler wume_crawler/1.1 (http://wume.cse.lehigh.edu/~xiq204/crawler/) Lehigh University Wwlib/Linux Wwlib/Linux Wolverhampton Web Library Wwwster wwwster/1.4 (Beta, mailto:gue@cis.uni-muenchen.de) Centrum f?r Informations- und Sprachverarbeitung XP5 Project XP5 [2.03.07-111203] Marty Anstey Xdefine egothor/3.0a (+http://www.xdefine.org/robot.html) Xdefine Xirq xirq/0.1-beta (xirq; http://www.xirq.com/; xirq@xirq.com) Xirq Xyleme SA France cosmos/0.9_(robot@xyleme.com) Xyleme cosmos/0.8_(robot@xyleme.com) Y!J Mozilla/4.0 (compatible; Y!J; for robot study; keyoshid) Yahoo Japan Y!J/1.0 (http://help.yahoo.co.jp/help/jp/search/indexing/indexing-15.html) Yacy yacy (http://www.yacy.net/; v20040602; i386 Linux 2.4.26-gentoo-r13; java 1.4.2_06; MET/ en) Yacy Yacybot yacybot (x86 Windows XP 5.1; java 1.5.0_06; Europe/de) yacy.net Yacy yacybot (i386 Linux 2.6.17-gentoo-r7; java 1.5.0_08; GMT/en) yacy.net yacybot (i386 Linux 2.6.17-2-686; java 1.5.0_06; Europe/de) yacy.net Yahoo Japan robot DoCoMo/2.0/SO502i (compatible; Y!J-SRD/1.0; http://help.yahoo.co.jp/help/jp/sear ch/indexing/indexing-27.html) Yahoo Japan Mozilla/4.0 (compatible; Yahoo Japan; for robot study; kasugiya) DoCoMo/2.0 SH902i (compatible; Y!J-SRD/1.0; http://help.yahoo.co.jp/help/jp/sear ch/indexing/indexing-27.html) Y!J-SRD/1.0 Yahoo Search Japan robot Y!J-BSC/1.0 (http://help.yahoo.co.jp/help/jp/search/indexing/indexing-15.html) Yahoo Japan Yahoo Search Marketing crawler Mozilla/4.0 (compatible; crawlx, crawler@trd.overture.com) Yahoo YahooYSMcm/2.0.0 Yahoo! Mindset Yahoo! Mindset Yahoo! Mindset Yahoo-Blogs Yahoo-Blogs/v3.9 (compatible; Mozilla 4.0; MSIE 5.5; http://help.yahoo.com/help/ us/ysearch/crawling/crawling-02.html ) Yahoo Yahoo-MMAudVid Yahoo-MMAudVid/2.0(mms dash mm aud vid crawler dash support at yahoo dash inc.co m ;Mozilla 4.0 compatible; MSIE 7.0;Windows NT 5.0; .NET CLR 2.0) Yahoo Yahoo-MMCrawler Yahoo-MMCrawler/3.x (mms dash mmcrawler dash support at yahoo dash inc dot com) Yahoo Yahoo-Test/4.0 Yahoo-Test/4.0 Yahoo Yahoo-VerticalCrawler Yahoo-VerticalCrawler-FormerWebCrawler/3.9 crawler at trd dot overture dot com; http://www.alltheweb.com/ Yahoo YahooFeedSeeker YahooFeedSeeker/2.0 (compatible; Mozilla 4.0; MSIE 5.5; http://publisher.yahoo.c/ om/rssguide) Yahoo YahooFeedSeeker/1.0 (compatible; Mozilla 4.0; MSIE 5.5; http://my.yahoo.com/s/pu blishers.html) YahooSeeker YahooSeeker/CafeKelsa-dev (compatible; Konqueror/3.2; FreeBSD ;cafekelsa-dev-web master@yahoo-inc.com ) (KHTML, like Gecko) Yahoo YahooSeeker/M1A1-R2D2 SIE-SX1/05 UP.Browser/7.0.0.1.181 (GUI) MMP/2.0 Profile/MIDP-2.0 Configuration/C LDC-1.1 (compatible;YahooSeeker/M1A1-R2D2;mobile-search-customer-care AT yahoo-i nc dot com) Yahoo LG-C1500 UP.Browser/6.2.3 (GUI) MMP/1.0 (compatible;YahooSeeker/M1A1-R2D2;mobile -search-customer-care AT yahoo-inc dot com) MOT-V975/81.33.02I MIB/2.2.1 Profile/MIDP-2.0 Configuration/CLDC-1.1 (compatible ;YahooSeeker/M1A1-R2D2; http://help.yahoo.com/help/us/ysearch/crawling/crawling– 01.html) SonyEricssonP910c/R2A SEMC-Browser/Symbian/3.0 Profile/MIDP-2.0 Configuration/CL DC-1.0 (compatible;YahooSeeker/M1A1-R2D2;mobile-search-customer-care AT yahoo-in c dot com) Nokia6682/2.0 (3.01.1) SymbianOS/8.0 Series60/2.6 Profile/MIDP-2.0 configuration /CLDC-1.1 UP.Link/6.3.0.0.0 (compatible;YahooSeeker/M1A1-R2D2;http://help.yahoo. com/help/us/ysearch/crawling/crawling-01.html) SonyEricssonP910c/R2A SEMC-Browser/Symbian/3.0 Profile/MIDP-2.0 Configuration/CL DC-1.0 (compatible;YahooSeeker/M1A1-R2D2; http://help.yahoo.com/help/us/ysearch/ crawling/crawling-01.html) SGH-Z130 SHP/VPP/R5 SMB3.1 SMM-MMS/1.1.0 profile/MIDP-2.0 configuration/CLDC-1.0 (compatible;YahooSeeker/M1A1-R2D2;mobile-search-customer-care AT yahoo-inc dot com) SIE-SX1/05 UP.Browser/7.0.0.1.181 (GUI) MMP/2.0 Profile/MIDP-2.0 Configuration/C LDC-1.1 (compatible;YahooSeeker/M1A1-R2D2; http://help.yahoo.com/help/us/ysearch /crawling/crawling-01.html) Nokia6682/2.0 (3.01.1) SymbianOS/8.0 Series60/2.6 Profile/MIDP-2.0 configuration /CLDC-1.1 UP.Link/6.3.0.0.0 (compatible;YahooSeeker/M1A1-R2D2; http://help.yahoo/ .com/help/us/ysearch/crawling/crawling-01.html) Nokia6682/2.0 (3.01.1) SymbianOS/8.0 Series60/2.6 Profile/MIDP-2.0 configuration /CLDC-1.1 UP.Link/6.3.0.0.0 (compatible;YahooSeeker/M1A1-R2D2;mobile-search-cust omer-care AT yahoo-inc dot com) SGH-Z130 SHP/VPP/R5 SMB3.1 SMM-MMS/1.1.0 profile/MIDP-2.0 configuration/CLDC-1.0 (compatible;YahooSeeker/M1A1-R2D2; http://help.yahoo.com/help/us/ysearch/crawli ng/crawling-01.html) LG-C1500 UP.Browser/6.2.3 (GUI) MMP/1.0 (compatible;YahooSeeker/M1A1-R2D2; http: //help.yahoo.com/help/us/ysearch/crawling/crawling-01.html) Nokia9500 (compatible;YahooSeeker/M1A1-R2D2;mobile-search-customer-care AT yahoo -inc dot com) MOT-V975/81.33.02I MIB/2.2.1 Profile/MIDP-2.0 Configuration/CLDC-1.1 (compatible ;YahooSeeker/M1A1-R2D2;mobile-search-customer-care AT yahoo-inc dot com) Yandex Mozilla/4.0 (compatible; MSIE 5.0; YANDEX) Yandex Yandex/1.01.001 (compatible; Win16; I) Yarienavoir yarienavoir.net/0.2 Yarienavoir Yellopet spider.yellopet.com – http://www.yellopet.com/ Yellopet Yeti Yeti/1.0 (NHN Corp.; http://help.naver.com/robots/) Naver Yeti Yeti/0.01 (nhn/1noon, yetibot@naver.com, check robots.txt daily and follows it) Yggdrasil yggdrasil/Nutch-0.9 (yggdrasil biorelated search engine; www dot biotec dot tu m inus dresden do de slash schroeder; heiko dot dietze at biotec dot tu minus dres den dot de) GoPubMed YodaoBot Mozilla/5.0 (compatible; YodaoBot/1.0; http://www.yodao.com/help/webmaster/spide r/; ) Yodao YodaoBot/1.0 (http://www.yodao.com/help/webmaster/spider/; ) YooW! YooW!/1.8.9 RC1 (+http://www.yoow.eu) YooW! Yoogli yoogliFetchAgent/0.1 Yoogli Yoono Mozilla/5.0 (compatible; Yoono; http://www.yoono.com/) Yoono Yoono web-crawler yoono/1.0 web-crawler/1.0 Yoono YottaCars YottaCars_Bot/4.12 (+http://www.yottacars.com) Car Search Engine YottaCars YottaShopping YottaShopping_Bot/4.12 (+http://www.yottashopping.com) Shopping Search Engine YottaShopping Z-Add Link Checker Z-Add Link Checker (http://w3.z-add.co.uk/linkcheck/) Z-add Zao Crawler Zao-Crawler Kototoi Zao-Crawler 0.2b Zao/0.1 (http://www.kototoi.org/zao/) ZeBot ZeBot_www.ze.bz (ze.bz@hotmail.com) ZE.BZ ZeBot_lseek.net (bot@ze.bz) Zealbot Mozilla/4.0 (compatible; Zealbot 1.0) LookSmart Zearchit Zearchit Zearchit Zedzo.digest zedzo.digest/0.1 (http://www.zedzo.com/) Zedzo Zerxbot zerxbot/Version 0.6 libwww-perl/5.79 Zerx Zetbot DeepIndex ( http://www.zetbot.com/ ) Zetbot Zeus ZBot/1.00 (icaulfield@zeus.com)   Zeus xxxxx Webster Pro V2.9 Win32 Zeus ThemeSite Viewer Webster Pro V2.9 Win32 ZipppBot ZipppBot/0.xx (ZipppBot; http://www.zippp.net/; webmaster@zippp.net) Zipp ZIPPPCVS/0.xx (ZipppBot/.xx;http://www.zippp.net; webmaster@zippp.net) Zippy Zippy v2.0 – Zippyfinder.com Zippyfinder ZoomSpider ZoomSpider – wrensoft.com WrenSoft Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0; ZoomSpider.net bot; .NET CLR 1.1.4322) Zspider zspider/0.9-dev http://feedback.redkolibri.com/ Redkolibri ZyBorg (LookSmart) ZyBorg LookSmart ZyBorg (Wisenut) Mozilla/4.0 compatible ZyBorg/1.0 (wn-16.zyborg@looksmart.net; http://www.wisenu/ tbot.com) Wisenut ZyBorg/1.0 (ZyBorg@WISEnut.com; http://www.wisenut.com/)
If you enjoyed this post, make sure you subscribe to my RSS feed!

匿名进行回复 取消回复

电子邮件地址不会被公开。 必填项已用*标注