# robots.txt generated at http://www.mcanerin.com # robots.txt generated for http://www.thetiger.de User-agent: Googlebot User-agent: Googlebot-Image User-agent: MSNBot User-agent: Slurp User-agent: yahoo-mmcrawler User-agent: psbot User-agent: asterias User-agent: yahoo-blogs/v3.9 Disallow: User-agent: Teoma User-agent: Gigabot User-agent: Scrubby User-agent: Robozilla User-agent: Nutch User-agent: baiduspider Disallow: / User-agent: ia_archiver Disallow: # Some added Stuff from http://www.tech-faq.com/bad-robots.shtml User-agent: BackDoorBot/1.0 User-agent: Black.Hole User-agent: BlackWidow User-agent: BlowFish User-agent: BlowFish/1.0 User-agent: Bookmark search tool User-agent: Bot mailto:craftbot@yahoo.com User-agent: BotALot User-agent: BotRightHere User-agent: BuiltBotTough User-agent: Bullseye User-agent: Bullseye/1.0 User-agent: BunnySlippers User-agent: Cegbfeieh User-agent: CheeseBot User-agent: CherryPicker User-agent: CherryPickerElite/1.0 User-agent: CherryPickerSE/1.0 User-agent: ChinaClaw User-agent: Copernic User-agent: CopyRightCheck User-agent: Crescent User-agent: Crescent Internet ToolPak HTTP OLE Control v.1.0 User-agent: Custo User-agent: DISCo User-agent: DISCo Pump 3.0 User-agent: DISCo Pump 3.2 User-agent: DISCoFinder User-agent: DittoSpyder User-agent: Download Demon User-agent: Download Demon/3.2.0.8 User-agent: Download Demon/3.5.0.11 User-agent: EirGrabber User-agent: EmailCollector User-agent: EmailSiphon User-agent: EmailWolf User-agent: EroCrawler User-agent: Express WebPictures User-agent: Express WebPictures (www.express-soft.com) User-agent: ExtractorPro User-agent: EyeNetIE User-agent: FairAd Client User-agent: Flaming AttackBot User-agent: FlashGet User-agent: FlashGet WebWasher 3.2 User-agent: Foobot User-agent: FrontPage User-agent: FrontPage [NC,OR] User-agent: Gaisbot User-agent: GetRight User-agent: GetRight/2.11 User-agent: GetRight/3.1 User-agent: GetRight/3.2 User-agent: GetRight/3.3 User-agent: GetRight/3.3.3 User-agent: GetRight/3.3.4 User-agent: GetRight/4.0.0 User-agent: GetRight/4.1.0 User-agent: GetRight/4.1.1 User-agent: GetRight/4.1.2 User-agent: GetRight/4.2 User-agent: GetRight/4.2b (Portuguxeas) User-agent: GetRight/4.2c User-agent: GetRight/4.3 User-agent: GetRight/4.5 User-agent: GetRight/4.5a User-agent: GetRight/4.5b User-agent: GetRight/4.5b1 User-agent: GetRight/4.5b2 User-agent: GetRight/4.5b3 User-agent: GetRight/4.5b6 User-agent: GetRight/4.5b7 User-agent: GetRight/4.5c User-agent: GetRight/4.5d User-agent: GetRight/4.5e User-agent: GetRight/5.0beta1 User-agent: GetRight/5.0beta2 User-agent: GetWeb! User-agent: Go!Zilla User-agent: Go!Zilla (www.gozilla.com) User-agent: Go!Zilla 3.3 (www.gozilla.com) User-agent: Go!Zilla 3.5 (www.gozilla.com) User-agent: Go-Ahead-Got-It User-agent: GrabNet User-agent: Grafula User-agent: HMView User-agent: HTTrack User-agent: HTTrack 3.0 User-agent: HTTrack [NC,OR] User-agent: Harvest User-agent: Harvest/1.5 User-agent: Image Stripper User-agent: Image Sucker User-agent: Indy Library User-agent: Indy Library [NC,OR] User-agent: InfoNaviRobot User-agent: InterGET User-agent: Internet Ninja User-agent: Internet Ninja 4.0 User-agent: Internet Ninja 5.0 User-agent: Internet Ninja 6.0 User-agent: Iron33/1.0.2 User-agent: JOC Web Spider User-agent: JennyBot User-agent: JetCar User-agent: Kenjin Spider User-agent: Kenjin.Spider User-agent: Keyword Density/0.9 User-agent: Keyword.Density User-agent: LNSpiderguy User-agent: LeechFTP User-agent: LexiBot User-agent: LinkScan/8.1a Unix User-agent: LinkScan/8.1a.Unix User-agent: LinkWalker User-agent: LinkextractorPro User-agent: MIDown tool User-agent: MIIxpc User-agent: MIIxpc/4.2 User-agent: MSIECrawler User-agent: Mass Downloader User-agent: Mass Downloader/2.2 User-agent: Mata Hari User-agent: Mata.Hari User-agent: Microsoft URL Control User-agent: Microsoft URL Control - 5.01.4511 User-agent: Microsoft URL Control - 6.00.8169 User-agent: Microsoft.URL User-agent: Mister PiX User-agent: Mister PiX version.dll User-agent: Mister Pix II 2.01 User-agent: Mister Pix II 2.02a User-agent: Mister.PiX User-agent: NICErsPRO User-agent: NPBot User-agent: NPbot User-agent: Navroad User-agent: NearSite User-agent: Net Vampire User-agent: Net Vampire/3.0 User-agent: NetAnts User-agent: NetAnts/1.10 User-agent: NetAnts/1.23 User-agent: NetAnts/1.24 User-agent: NetAnts/1.25 User-agent: NetMechanic User-agent: NetSpider User-agent: NetZIP User-agent: NetZip Downloader 1.0 Win32(Nov 12 1998) User-agent: NetZip-Downloader/1.0.62 (Win32; Dec 7 1998) User-agent: NetZippy+(http://www.innerprise.net/usp-spider.asp) User-agent: Octopus User-agent: Offline Explorer User-agent: Offline Explorer/1.2 User-agent: Offline Explorer/1.4 User-agent: Offline Explorer/1.6 User-agent: Offline Explorer/1.7 User-agent: Offline Explorer/1.9 User-agent: Offline Explorer/2.0 User-agent: Offline Explorer/2.1 User-agent: Offline Explorer/2.3 User-agent: Offline Explorer/2.4 User-agent: Offline Explorer/2.5 User-agent: Offline Navigator User-agent: Offline.Explorer User-agent: Openbot User-agent: Openfind User-agent: Openfind data gatherer User-agent: Oracle Ultra Search User-agent: PageGrabber User-agent: Papa Foto User-agent: PerMan User-agent: ProPowerBot/2.14 User-agent: ProWebWalker User-agent: Python-urllib User-agent: QueryN Metasearch User-agent: QueryN.Metasearch User-agent: RMA User-agent: Radiation Retriever 1.1 User-agent: ReGet User-agent: RealDownload User-agent: RealDownload/4.0.0.40 User-agent: RealDownload/4.0.0.41 User-agent: RealDownload/4.0.0.42 User-agent: RepoMonkey User-agent: RepoMonkey Bait & Tackle/v1.01 User-agent: SiteSnagger User-agent: SlySearch User-agent: SmartDownload User-agent: SmartDownload/1.2.76 (Win32; Apr 1 1999) User-agent: SmartDownload/1.2.77 (Win32; Aug 17 1999) User-agent: SmartDownload/1.2.77 (Win32; Feb 1 2000) User-agent: SmartDownload/1.2.77 (Win32; Jun 19 2001) User-agent: SpankBot User-agent: Sqworm/2.9.85-BETA (beta_release; 20011115-775; i686-pc-linux User-agent: SuperBot User-agent: SuperBot/3.0 (Win32) User-agent: SuperBot/3.1 (Win32) User-agent: SuperHTTP User-agent: SuperHTTP/1.0 User-agent: Surfbot User-agent: Szukacz/1.4 User-agent: Teleport User-agent: Teleport Pro User-agent: Teleport Pro/1.29 User-agent: Teleport Pro/1.29.1590 User-agent: Teleport Pro/1.29.1634 User-agent: Teleport Pro/1.29.1718 User-agent: Teleport Pro/1.29.1820 User-agent: Teleport Pro/1.29.1847 User-agent: TeleportPro User-agent: Telesoft User-agent: The Intraformant User-agent: The.Intraformant User-agent: TheNomad User-agent: TightTwatBot User-agent: Titan User-agent: True_Robot User-agent: True_Robot/1.0 User-agent: TurnitinBot User-agent: TurnitinBot/1.5 User-agent: URL Control User-agent: URL_Spider_Pro User-agent: URLy Warning User-agent: URLy.Warning User-agent: VCI User-agent: VCI WebViewer VCI WebViewer Win32 User-agent: VoidEYE User-agent: WWW-Collector-E User-agent: WWWOFFLE User-agent: Web Image Collector User-agent: Web Sucker User-agent: Web.Image.Collector User-agent: WebAuto User-agent: WebAuto/3.40 (Win98; I) User-agent: WebBandit User-agent: WebBandit/3.50 User-agent: WebCapture 2.0 User-agent: WebCopier User-agent: WebCopier v.2.2 User-agent: WebCopier v2.5 User-agent: WebCopier v2.6 User-agent: WebCopier v2.7a User-agent: WebCopier v2.8 User-agent: WebCopier v3.0 User-agent: WebCopier v3.0.1 User-agent: WebCopier v3.2 User-agent: WebCopier v3.2a User-agent: WebEMailExtrac.* User-agent: WebEnhancer User-agent: WebFetch User-agent: WebGo IS User-agent: WebLeacher User-agent: WebReaper User-agent: WebReaper [info@webreaper.net] User-agent: WebReaper [webreaper@otway.com] User-agent: WebReaper v9.1 - www.otway.com/webreaper User-agent: WebReaper v9.7 - www.webreaper.net User-agent: WebReaper v9.8 - www.webreaper.net User-agent: WebReaper vWebReaper v7.3 - www,otway.com/webreaper User-agent: WebSauger User-agent: WebSauger 1.20b User-agent: WebSauger 1.20j User-agent: WebSauger 1.20k User-agent: WebStripper User-agent: WebStripper/2.03 User-agent: WebStripper/2.10 User-agent: WebStripper/2.12 User-agent: WebStripper/2.13 User-agent: WebStripper/2.15 User-agent: WebStripper/2.16 User-agent: WebStripper/2.19 User-agent: WebWhacker User-agent: WebZIP User-agent: WebZIP/2.75 (http://www.spidersoft.com) User-agent: WebZIP/3.65 (http://www.spidersoft.com) User-agent: WebZIP/3.80 (http://www.spidersoft.com) User-agent: WebZIP/4.0 (http://www.spidersoft.com) User-agent: WebZIP/4.1 (http://www.spidersoft.com) User-agent: WebZIP/4.21 User-agent: WebZIP/4.21 (http://www.spidersoft.com) User-agent: WebZIP/5.0 User-agent: WebZIP/5.0 (http://www.spidersoft.com) User-agent: WebZIP/5.0 PR1 (http://www.spidersoft.com) User-agent: WebZip User-agent: WebZip/4.0 User-agent: WebmasterWorldForumBot User-agent: Website Quester User-agent: Website Quester - www.asona.org User-agent: Website Quester - www.esalesbiz.com/extra/ User-agent: Website eXtractor User-agent: Website eXtractor (http://www.asona.org) User-agent: Website.Quester User-agent: Webster Pro User-agent: Webster.Pro User-agent: Wget User-agent: Wget/1.5.2 User-agent: Wget/1.5.3 User-agent: Wget/1.6 User-agent: Wget/1.7 User-agent: Wget/1.8 User-agent: Wget/1.8.1 User-agent: Wget/1.8.1+cvs User-agent: Wget/1.8.2 User-agent: Wget/1.9-beta User-agent: Widow User-agent: Xaldon WebSpider User-agent: Xaldon WebSpider 2.5.b3 User-agent: Xenu's User-agent: Xenu's Link Sleuth 1.1c User-agent: Zeus User-agent: Zeus 11389 Webster Pro V2.9 Win32 User-agent: Zeus 11652 Webster Pro V2.9 Win32 User-agent: Zeus 18018 Webster Pro V2.9 Win32 User-agent: Zeus 26378 Webster Pro V2.9 Win32 User-agent: Zeus 30747 Webster Pro V2.9 Win32 User-agent: Zeus 32297 Webster Pro V2.9 Win32 User-agent: Zeus 39206 Webster Pro V2.9 Win32 User-agent: Zeus 41641 Webster Pro V2.9 Win32 User-agent: Zeus 44238 Webster Pro V2.9 Win32 User-agent: Zeus 51070 Webster Pro V2.9 Win32 User-agent: Zeus 51674 Webster Pro V2.9 Win32 User-agent: Zeus 51837 Webster Pro V2.9 Win32 User-agent: Zeus 63567 Webster Pro V2.9 Win32 User-agent: Zeus 6694 Webster Pro V2.9 Win32 User-agent: Zeus 71129 Webster Pro V2.9 Win32 User-agent: Zeus 82016 Webster Pro V2.9 Win32 User-agent: Zeus 82900 Webster Pro V2.9 Win32 User-agent: Zeus 84842 Webster Pro V2.9 Win32 User-agent: Zeus 90872 Webster Pro V2.9 Win32 User-agent: Zeus 94934 Webster Pro V2.9 Win32 User-agent: Zeus 95245 Webster Pro V2.9 Win32 User-agent: Zeus 95351 Webster Pro V2.9 Win32 User-agent: Zeus 97371 Webster Pro V2.9 Win32 User-agent: Zeus Link Scout User-agent: b2w/0.1 User-agent: cosmos User-agent: eCatch User-agent: eCatch/3.0 User-agent: hloader User-agent: httplib User-agent: humanlinks User-agent: larbin User-agent: larbin (samualt9@bigfoot.com) User-agent: larbin samualt9@bigfoot.com User-agent: larbin_2.6.2 (kabura@sushi.com) User-agent: larbin_2.6.2 (larbin2.6.2@unspecified.mail) User-agent: larbin_2.6.2 (listonATccDOTgatechDOTedu) User-agent: larbin_2.6.2 (vitalbox1@hotmail.com) User-agent: larbin_2.6.2 kabura@sushi.com User-agent: larbin_2.6.2 larbin2.6.2@unspecified.mail User-agent: larbin_2.6.2 larbin@correa.org User-agent: larbin_2.6.2 listonATccDOTgatechDOTedu User-agent: larbin_2.6.2 vitalbox1@hotmail.com User-agent: libWeb/clsHTTP User-agent: lwp-trivial User-agent: lwp-trivial/1.34 User-agent: moget User-agent: moget/2.1 User-agent: pavuk User-agent: pcBrowser User-agent: searchpreview User-agent: spanner User-agent: suzuran User-agent: tAkeOut User-agent: toCrawl/UrlDispatcher User-agent: turingos User-agent: webfetch/2.1.0 User-agent: wget Disallow: / # Some added Stuff from http://www.user-agents.org/index.shtml User-agent: 8484 Boston Project v 1.0 User-agent: Atomz/1.0 User-agent: atSpider/1.0 User-agent: autoemailspider User-agent: Der große BilderSauger 2.00u User-agent: DISCo Pump x.x User-agent: Download Demon/3.x.x.x User-agent: Download Druid 2.x User-agent: Download Express 1.0 User-agent: Download Master User-agent: Download Ninja 3.0 User-agent: Download Wonder User-agent: Download-Tipp Linkcheck (http://download-tipp.de/) User-agent: Download.exe(1.1) (+http://www.sql-und-xml.de/freeware-tools/) User-agent: DSurf15a 01 User-agent: DSurf15a 71 User-agent: DSurf15a 81 User-agent: DSurf15a VA User-agent: EmailWolf 1.00 User-agent: ESurf15a 15 User-agent: FSurf15a 01 User-agent: Full Web Bot 0416B User-agent: Full Web Bot 0516B User-agent: Full Web Bot 2816B User-agent: PBrowse 1.4b User-agent: PEval 1.4b User-agent: Pockey-GetHTML/4.12.0 (Win32; GUI; ix86) User-agent: Pockey-GetHTML/x.xx User-agent: Pockey/x.x.x User-agent: Pockey7.x.x(WIN32GUI) User-agent: Poirot User-agent: Port Huron Labs User-agent: Production Bot 0116B User-agent: Production Bot 2016B User-agent: Production Bot DOT 3016B User-agent: PSurf15a 11 User-agent: PSurf15a 51 User-agent: PSurf15a VA User-agent: psycheclone User-agent: RSurf15a 41 User-agent: RSurf15a 51 User-agent: RSurf15a 81 User-agent: Shareaza v1.x.x.xx User-agent: SharewarePlazaFileCheckBot/1.0+(+http://www.SharewarePlaza.com) User-agent: Snapbot/1.0 User-agent: snap.com beta crawler v0 User-agent: sogou spider User-agent: Sogou web spider/3.0* User-agent: sohu agent User-agent: sohu-search User-agent: SSurf15a 11 User-agent: Under the Rainbow 2.2 User-agent: WebVulnCrawl.blogspot.com/1.0 libwww-perl/5.803 User-agent: Wells Search II User-agent: WEP Search 00 Disallow: / # Alles andere an komischen Bots, gefunden in der Serverlog ... User-agent: Xaldon WebSpider 2.7.b6 User-agent: FAST Enterprise Crawler 6* User-agent: IRLbot/3.0* User-agent: Sogou webspider/3.0* User-agent: ichiro/2.0* Disallow: / #Hier bei den Verzeichnissen (nach "Disallow:") einfach die Verzeichnisse ändern. Das #hier bedeutet, das die Spiders (Google etc.) NICHT in diese Verzeichnisse dürfen... User-agent: * Disallow: /cgi-bin/ Disallow: /_1/ Disallow: /_2/ Disallow: /_3/ Disallow: /_4/ Disallow: /_5/ Disallow: /_6/ Disallow: /_aTV/ # Der Code ist laut "http://tool.motoricerca.info/robots-checker.phtml" komplett fehlerfrei, also VALID ;D