| User Agent | | Date Added |
| ChristCrawler.com |  | 11/02/2004 16:14:07 |
| A Christian internet spider that searches web sites to find Christian Related material |
| cIeNcIaFiCcIoN.nEt | | 11/02/2004 16:17:52 |
| Robot encargado de la indexación de las páginas para www.cienciaficcion.net |
| CindooSpider | | 22/02/2009 23:52:09 |
|
| CipinetBot | | 8/02/2004 15:25:43 |
|
| Climate Change Spider | | 8/02/2004 15:26:47 |
|
| cmsworldmap.com | | 4/02/2011 12:39:31 |
Search Engine for CMS System from linkfactory.dk
fell for bad bot trap
|
| Collective | | 11/02/2004 16:21:03 |
Collective is the most cleverest Internet search engine, With all found urls guaranteed to have your search terms.
Collective is a highly configurable program designed to interrogate online search engines and online databases, it will ignore web pages that lie about there content, and dead url's, it can be super strict, it searches each web page it finds for your search terms to ensure those terms are present, any positive urls are added to a html file for your to view at any time even before the program has finished. Collective can wonder the web for days if required. |
| Conceptbot | | 11/02/2004 16:26:22 |
| The Conceptbot spider is used to research concept-based search indexing techniques. It uses a breadth first search to spread out the number of hits on a single site over time. The spider runs at irregular intervals and is still under construction. |
| CoolBot | | 11/02/2004 16:27:43 |
| The CoolBot robot is used to build and maintain the directory of the german search engine Suchmaschine21. |
| Cowbot | | 8/02/2004 15:27:55 |
|
| Crawlson | | 10/09/2020 14:28:39 |
|
| CrazyWebCrawler | | 24/09/2015 23:55:45 |
|
| Cusco | | 11/02/2004 16:42:30 |
| The Cusco robot is part of the CUCE indexing sistem. It gathers information from several sources: HTTP, Databases or filesystem. At this moment, it's universe is the .pt domain and the information it gathers is available at the Portuguese search engine Cusco http://www.cusco.pt/. |
| DeepIndex | | 8/02/2004 15:29:32 |
|
| Desert Realm Spider | | 8/02/2004 15:30:24 |
| The spider indexes fantasy and science fiction sites by using a customizable keyword algorithm. Only home pages are indexed, but all pages are looked at for links. Pages are visited randomly to limit impact on any one webserver. |
| DeuSu | | 9/10/2015 20:50:57 |
| DeuSu is my personal project. It is a search-engine which has its very own search-index. To create this search-index, over a billion web-pages have to be crawled. The DeuSu robot is the software used to do this. |
| Die Blinde Kuh | | 11/02/2004 0:10:22 |
| The robot is use for indexing and proofing the registered urls in the german language search engine for kids. Its a non-commercial one-woman-project of Birgit Bachmann living in Hamburg, Germany. |
| DIE-KRAEHE | | 8/02/2004 15:28:58 |
|
| DienstSpider | | 11/02/2004 16:54:43 |
| Indexing and searching the NCSTRL(Networked Computer Science Technical Report Library) and ERCIM Collection |
| Digger | | 11/02/2004 16:56:06 |
| indexing web sites for the Diggit! search engine |
| Digital Integrity Robot | | 11/02/2004 17:03:37 |
|
| Direct Hit Grabber | | 11/02/2004 17:04:55 |
| Direct Hit Grabber indexes documents and collects Web statistics for the Direct Hit Search Engine (available at www.directhit.com and our partners' sites) |
| DittoSpyder | | 8/09/2006 0:13:47 |
| Ditto image search engine |
| DNAbot | | 11/02/2004 17:07:07 |
| A search robot in 100 java, with its own built-in database engine and web server . Currently in Japanese. |
| DragonBot | | 11/02/2004 17:10:23 |
| Collects web pages related to East Asia |
| DuckAssistBot | | 29/10/2024 16:17:50 |
|
| DuckDuckBot | | 10/09/2015 23:29:37 |
|
| DuckDuckPreview | | 25/09/2015 13:35:06 |
|
| DWCP (Dridus' Web Cataloging Project) | | 11/02/2004 17:11:45 |
| The DWCP robot is used to gather information for Dridus' Web Cataloging Project, which is intended to catalog domains and urls (no content). |
| Eco-Portal Spider | | 8/02/2004 15:33:26 |
|
| ELFINBOT | | 11/02/2004 17:17:04 |
| ELFIN is used to index and add data to the "Lets Find It Now Search Engine" (http://letsfinditnow.com). The robot runs every 30 days. |
| EMPAS_ROBOT | | 8/02/2004 15:30:46 |
|
| Environmental Sustainability Spider | | 8/02/2004 15:34:26 |
|
| EroCrawler | | 8/09/2006 0:17:58 |
| adult search engine |
| ES.NET | | 5/07/2005 11:02:08 |
| Innerprise develops full-text indexing search engine software technology enabling search for your Web site, Intranet, or the Web. Advanced crawler features ensure that only documents you want indexed are indexed. Key features provide support for common file types, secure servers, multiple servers, and complete automation through built-in schedulers. |
| Esther | | 11/02/2004 22:45:21 |
| This crawler is used to build the search database at http://search.falconsoft.com/ |
| EuroSeek Arachnoidea | | 10/02/2004 1:19:00 |
|
| Evliya Celebi | | 11/02/2004 22:46:23 |
| crawles pages under ".tr" domain or having turkish character encoding (iso-8859-9 or windows-1254) |
| ExactSeek Crawler | | 8/02/2004 19:40:22 |
ExactSeek.com is an internet search engine and directory that receives and indexes over 30,000 new site submissions daily. To date, more than 2 million web sites have been indexed and added to the ExactSeek database and another 2 to 3 million web sites will be added in the near future. Our goal is to not index the Web but to provide searchers with a "quality" database of between 4 and 5 million web sites. In addition to standard web search results, ExactSeek also offers targeted searches of specialized databases. Currently, visitors can use niche search engines to find newsletters, articles, mp3 files, images, and comparison shopping sites.
Uses UAs: "eseek-crawler" and "exactseek-crawler*" |
| Exalead | | 8/02/2004 15:35:38 |
|
| Excite ArchitextSpider | | 10/02/2004 1:19:52 |
| Its purpose is to generate a Resource Discovery database, and to generate statistics. The ArchitextSpider collects information for the Excite and WebCrawler search engines. |
| EZResult | | 10/02/2004 1:30:10 |
|
| FAST | | 8/02/2004 15:36:30 |
| Crawler for alltheweb.com |
| FastBug | | 8/02/2004 15:37:06 |
|
| FastCrawler | | 11/02/2004 22:49:58 |
| FastCrawler is used to build the databases for search engines used by 1klik.dk and it's partners |
| FeedFetcher-Google | | 13/04/2008 22:39:11 |
|
| Feedster Crawler | | 8/02/2004 15:39:37 |
|
| FemtosearchBot | | 21/12/2018 13:26:33 |
| fell for bad bot trap |
| Findexa Crawler | | 2/08/2007 23:00:00 |
| Norwegian search engine |
| Findxbot | | 17/02/2015 22:04:25 |
| Findx bot is a web scraping bot used by the search engine findx. Findx's goal is to create an independent European search engine with a strong focus on privacy and user choice. Findx bot scrapes sites to include in its index and help direct users to those sites. |
