Seit dem 16.12.2009 habe ich die Online-Anzeige dieses HP-Forums auf 12 Stunden eingestellt; sodass mir dies ermöglicht, auch sämtliche Suchmaschinenbots zu erfassen, die im System noch nicht registriert sind, um sie zu registrieren und in dieser Liste aufzunehmen.
Die Bots geben dabei in der Regel jeweils in ihrer Kennung ihre Homepage an. Wer für seine Webseite auch so eine Liste erstellen möchte, kann bei den zahlreichen Browserkennungen jeweils bei nachfolgenden drei Webseiten nachschauen. Siehe hierzu auch hier: Klick!
http://user-agent-string.info/
http://user-agent-string.info/list-of-ua
http://www.botsvsbrowsers.com/Archives
Http://friendfeed.com/about/bot
http://Anonymouse.org
http://about.ask.com/en/docs/about/webmasters.shtml
http://ahrefs.com/robot/
http://archivethe.net/en/index.php/about/internet_memory1
http://bimeon.com
http://bixolabs.com/crawler/general
http://bsalsa.com
http://cliqz.com/
http://code.google.com/appengine
http://cognitiveseo.com/bot.html
http://crawler.archive.org/
http://crawler.sistrix.net/
http://desktop.google.com
http://developer.yahoo.com/yql/provider
http://developers.facebook.com/
http://discoveryengine.com/discobot.html
http://discoveryengine.com/discoverybot.html
http://domains.checkparams.com/
http://duckduckgo.com/duckduckbot.html
http://emefge.de/bot.html
http://endb-consolidated.aihit.com/
http://epweb2.ph.bham.ac.uk/user/slater/camont/info.html
http://flipboard.com/browserproxy
http://fulltext.sblog.cz/
http://games-online.cjb.net
http://garlik.com
http://geohasher.gotdns.org
http://glutenfreepleasure.com/
http://go.mail.ru/help/robots
http://gsitecrawler.com
http://help.goo.ne.jp/door/crawler.html
http://help.naver.com/robots/
http://help.yahoo.com/
http://help.yahoo.com/help/us/ysearch/slurp
http://help.zum.com/inquiry
http://huaweisymantec.com
http://inagist.com/
http://infegy.com/
http://intelligiants.com/bot.html
http://jetsli.de/crawler
http://jigsaw.w3.org/css-validator/
http://js-kit.com/
http://kc.nict.go.jp/project1/crawl.html
http://knowmore.com/bots
http://labs.topsy.com/butterfly/
http://linkfluence.net/
http://megaindex.com/crawler
http://mergeflow.net/info/pagereader
http://metager2.de/technology.php
http://mozshot.nemui.org/
http://netcomber.com/
http://netcraft.com
http://openindex.io/spider.html
http://pastebin.de/25277
http://pear.php.net/package/http_request2
http://queryseeker.com/bot.html
http://robotgenius.net
http://search.msn.com/msnbot.htm
http://search.yahoo.com/yahooseeker.html
http://showyou.com/crawler
http://spinn3r.com/robot
http://spotinfluence.com/
http://summify.com
http://support.embed.ly/
http://thumbshots.in/bot.html
http://thumbsniper.com/
http://tweetedtimes.com
http://twikle.com/
http://unshort.me/about.html
http://validator.w3.org/
http://validator.w3.org/mobile/
http://variohost24.de
http://variohost24.net
http://w.moreover.com
http://webalgo.iit.cnr.it/index.php?pg=lwebis
http://webmeup.com/crawler.html
http://wizard.ae.krakow.pl/~jb/bot
http://worio.com
http://wortschatz.uni-leipzig.de/findlinks
http://www.80legs.com/spider.html
http://www.MIA-marktplatz.de/
http://www.Nutch.de
http://www.abonti.com
http://www.acoon.de/robot.asp
http://www.admantx.com/
http://www.alexa.com/help/webmasters
http://www.amfibi.com/cabot
http://www.anonsphere.com/
http://www.answerbus.com
http://www.ant.com/
http://www.apple.com/go/applebot
http://www.archive.org/details/archive.org_bot
http://www.askpeter.info/bot.html
http://www.axxus.de/
http://www.backlinktest.com/crawler.html
http://www.baidu.com/search/spider.htm
http://www.bing.com/bingbot.htm
http://www.bitvo.com/
http://www.blogscope.net/
http://www.bohble.de/pfadzurbotseite/bot.html
http://www.botje.com/plukkie.htm
http://www.brandwatch.net
http://www.caddo.de/bot.html
http://www.career-x.de/bot.html
http://www.cityreview.org/crawler/
http://www.cligoo.de/wk/technik.php
http://www.cliqz.com/
http://www.cmscrawler.com
http://www.commoncrawl.org/bot.html
http://www.crowsnest.tv/
http://www.crystalsemantics.com/user-agent/
http://www.cuil.com/twiceler/robot.html
http://www.dotnetdotcom.org
http://www.e-ditor.com
http://www.ellerdale.com/crawler.html
http://www.entireweb.com/about/search_tech/speedy_spider
http://www.eurip.com
http://www.eventguru.com/spider.html
http://www.evri.com/evrinid
http://www.exabot.com/go/robot
http://www.example.com
http://www.facebook.com/externalhit_uatext.php
http://www.fairshare.cc (http://www.golem.de/0903/65729.html)
http://www.fastbot.de/
http://www.findxbot.com/
http://www.float.com
http://www.genieo.com/webfilter.html
http://www.gigablast.com/spider.html
http://www.gnip.com/
http://www.google.com/bot.html
http://www.google.com
http://www.grapeshot.co.uk/crawler.php
http://www.gsmchinois.com/cubot-a6589-mtk6589-quad-core-android-4-1-3g-gps-498
http://www.haosou.com/help/help_3_2.html
http://www.hubspot.com/
http://www.icjobs.de
http://www.infohelfer.de/
http://www.jomjaibot.com
http://www.kalooga.com/info.html?page=crawler
http://www.kfsw.de/bot.html
http://www.kosmix.com/html/crawler.html
http://www.linguee.com/bot
http://www.linkbutler.de/spider
http://www.linkdex.com/about/bots/
http://www.load-time.com/bot.html
http://www.m-brain.com/neliveto
http://www.madaali.de/pfadzurbotseite/bot.html
http://www.majestic12.co.uk/bot.php
http://www.meanpath.com/meanpathbot.html
http://www.metadatalabs.com/mlbot
http://www.micro-sys.dk/products/sitemap-generator
http://www.mignify.com/
http://www.mister-wong.de/search/
http://www.mojeek.com/bot.html
http://www.nerdbynature.net/bot
http://www.netseer.com/crawler.html
http://www.omgili.com/Crawler.html
http://www.openindex.io/en/webmasters/spider.html
http://www.picmole.com
http://www.picsearch.com/bot.html
http://www.pixray.com/
http://www.proximic.com
http://www.psychoterapia-dda.pl/
http://www.puritysearch.net/
http://www.radian6.com/crawler
http://www.ruky.de/bot.html
http://www.scoutjet.com
http://www.search17.com/bot.php
http://www.searchme.com/support/
http://www.searchmetrics.com/en/searchmetrics-bot/
http://www.semager.de/blog/semager-bots/
http://www.semantissimo.de
http://www.sengine.info/
http://www.seodiver.com/bot
http://www.seokicks.de/robot.html
http://www.seomoz.org/dp/rogerbot
http://www.seoprofiler.com/bot/
http://www.sitebot.org/robot
http://www.snap.com
http://www.solomono.ru
http://www.teesoft.info
http://www.thumbshots-server.com/bot/
http://www.thumbshots.de
http://www.toshiba.co.jp/rdc/about/crawl_info.htm
http://www.tropiait.net Kein Bot.
http://www.turnitin.com/robot/crawlerinfo.html
http://www.twenga.fr/bot-discover.html
http://www.twitmunin.com
http://www.ubermetrics-technologies.com/
http://www.unister.de/
http://www.vbseo.com
http://www.voila.com
http://www.w3sitesearch.de
http://www.warebay.com/bot.html
http://www.web.nl/webmasters/spider.html
http://www.webinator.de/
http://www.webkicks.de
http://www.wiamond.com/zeige/bot.html
http://www.wotbox.com
http://www.youdao.com/help/webmaster/spider/
http://www.yunrang.com/yrspider.html
http://yacy.net/bot.html
http://yandex.com/bots
https://safesearch.avira.com/
ttp://blog.suggy.com/was-ist-suggy/suggy-webcrawler
Ab hier (noch) nicht alphabetisch geordnet, sowie eventuell doppelt:
http://corpora.informatik.uni-leipzig.de/crawler_faq.html
http://www.webinator.de/
http://www.seograph.net/bot.html
http://www.ubermetrics-technologies.com/
http://izsearch.com/
http://www.datagnion.com/bot.html
http://crawler.seolytics.net/
http://sur.ly/bot.html
http://www.scopia.co/
http://www.yahoo-help.jp/app/answers/detail/p/595/a_id/42716/
http://nlp.fi.muni.cz/projects/biwec/
http://www.metajob.de/crawler
http://getintent.com/
https://deusu.de/robot.html
http://www.similartech.com/smtbot
http://domainstats.io/our-bot
http://www.picsearch.com/bot.html
http://www.sogou.com/docs/help/webmasters.htm
http://newsharecounts.com/crawler
http://socialrank.io/about
http://www.linkdex.com/bots/
amp-cloud.de - Bot
http://www.analyticsseo.com/crawler
http://siteexplorer.info/Backlink-Checker-Spider/
http://linkfluence.com/