| User Agent | | Date Added |
| ReGet |  | 9/09/2006 0:18:57 |
|
| retriever | | 12/09/2006 0:38:02 |
|
| ripper | | 12/09/2006 0:38:13 |
|
| RoboFox | | 27/07/2005 23:27:49 |
| scheduled utility to download and database a domain |
| Robot Francoroute | | 11/02/2004 23:17:13 |
| Part of the RISQ's Francoroute project for researching francophone. Uses the Accept-Language tag and reduces demand accordingly |
| SBL-BOT | | 17/10/2014 12:30:55 |
| SoftByte Labs BlackWidow |
| Shai'Hulud | | 31/07/2005 23:40:57 |
Used to build mirrors for internal use.
This robot finds its roots in a research project at RDTeX Perspective Projects Group in 1996. |
| SiteCopy | | 6/02/2004 22:20:31 |
| SiteCopy arkiverer din hjemmeside og sikrer virksomhedens digitale historie. |
| SiteSnagger | | 9/09/2006 0:25:17 |
|
| SiteSucker | | 6/02/2004 22:21:24 |
| SiteSucker can be used to make local copies of your web sites for easy maintenance. SiteSucker can "localize" the files it downloads, allowing you to browse a site off-line. |
| SMPU | | 3/05/2006 21:41:15 |
Referer: http://www.norhaus.com/smpu.html
SMPU is a HTTP/1.0 URI parser and spider. The purpose of SMPU is resource collection and web site analysis.
- SMPU does not request any page more than once on any crawl.
- We will send you any information we have collected by request.
What does it do?
More often than not SMPU is used as a download utility, as it can recursively download some (or all) resources on a website. If you are seeing many requests that are all different then your server's contents are being either wholly or partially mirrored by the user.
If you are seeing occasional requests the chances are SMPU is being used as a spider to traverse the internet looking for something, and found a reference to your site.
What can I make it do?
Plenty of things, as a download util it's pretty good but it is more powerful as an analysis tool. You should familiarise yourself with the arguments for an idea of what it can do. It is free to download, and if you are a regular command prompt user it's a pretty useful tool to have around. |
| stripper | | 12/09/2006 0:40:24 |
|
| sucker | | 12/09/2006 0:40:06 |
|
| Sunrise XP | | 30/06/2007 0:24:38 |
| Sunrise XP converts web sites and newsfeeds to Plucker documents for offline reading on your handheld. |
| SuperBot | | 9/09/2006 0:28:11 |
| By using SuperBot to save your important and frequently used websites directly on your PC, you will ensure that you never lose access to vital Web information. |
| SuperHTTP | | 9/09/2006 0:30:21 |
| A full featured personal web spider. Use it to download entire websites for offline viewing, collect images or other file types off the internet, grab entire collections of files, Query search engines, or check links on your web site. |
| Teleport Webspiders | | 6/02/2004 22:27:40 |
|
| thief | | 12/09/2006 0:40:48 |
|
| thieves | | 12/09/2006 0:40:58 |
|
| vb wininet | | 9/02/2004 23:24:21 |
vbcode example to fetch page from web server.
Proof of monitoring available (log files)
NOT VERIFIED: probably also used in Igrabber Internet content grabber (http://www.aldostools.com/igrabber.html). Website did not exist anymore, google cache was still available:
A powerful internet content grabber that lets you automatically download web server contents off the net, extract specific information and integrate data from multiple pages for your later review, reduce download size or for transfer to portable devices, by simply placing all the 'Content Filters' you want fetched into one directory and pressing a button, using the command line or directly from the Favorites Links using the integration with the new AWeb (Aldo's Web Server).
Includes AWeb 1.5 letting browse filtered sites using filter templates. To filter images enter from MSIE address bar an URL like: http://localhost/filter=filter.dat/http://www.anyweb.com/. |
| Web Downloader | | 6/02/2004 22:29:24 |
|
| Web Sucker | | 11/09/2006 23:13:40 |
|
| WebAuto | | 24/05/2004 11:13:21 |
| WebAuto/3.42ƒÀ1 (WinNT; I) |
| WebCopier | | 6/02/2004 22:30:13 |
| Use our products to record websites and store them locally until you are ready to view them. |
| WebDownloader for X | | 6/02/2004 22:31:18 |
|
| Webdup | | 6/02/2004 22:38:09 |
|
| WebFetch | | 11/09/2006 23:05:01 |
|
| WebLeacher | | 11/09/2006 23:07:37 |
|
| WebMiner | | 6/02/2004 22:33:34 |
| WebMiner finds specific Internet content online and transfers the files to your local hard drive. |
| WeBoX | | 7/07/2010 11:55:20 |
|
| WebReaper | | 11/09/2006 23:08:37 |
| WebReaper is web crawler or spider, which can work its way through a website, downloading pages, pictures and objects that it finds so that they can be viewed locally, without needing to be connected to the internet. The sites can be saved locally as a fully-browsable website which can be viewed with any browser (such as Internet Explorer, Netscape, Opera, etc), or they can be saved into the Internet Explorer cache and viewed using IE's offline mode as if the you'd surfed the sites 'by hand'. |
| WebSauger | | 11/09/2006 23:11:07 |
|
| Website Downloader | | 19/02/2004 14:11:13 |
| Bad written bot: doesn't do URL decoding! |
| WebSite Extractor | | 6/02/2004 22:45:40 |
| Website Extractor gives you complete control over including and excluding downloads from individual servers, folders, URLs, and files using keywords. |
| Website Quester | | 2/05/2006 1:30:36 |
Referer: http://FindExe.com/search.htm
This is website extractor. Aggressive bot.
|
| WebStripper | | 6/02/2004 22:36:10 |
| Use WebStripper offline browser to quickly download web sites and pages onto your hard disk. WebStripper creates an exact copy of the site which you can browse offline whenever you want without having to connect to the internet. |
| WebWhacker | | 11/09/2006 23:20:04 |
| Copy entire web sites to your hard drive. |
| WebZIP | | 6/02/2004 22:37:50 |
| Save reference material, online books, e-zines, news articles and other Web content to your hard drive and browse them anywhere, anytime - fast! |
| WWWOFFLE | | 12/02/2004 0:02:21 |
|
| Xaldon WebSpider | | 6/02/2004 22:42:00 |
| Xaldon WebSpider 2 durchsucht automatisch gemäß den Vorgaben des Benutzers das World-Wide-Web und speichert die gesammelten Informationen lokal ab. |
| Xtreeme SiteXpert | | 9/02/2004 20:19:04 |
| With SiteXpert, without any HTML / JavaScript knowledge, you can quickly create a variety of cross-browser navigation systems without having to worry about compatibility between browsers. You can also create a search engine (hosted by your own web server or for CDROM distribution). The program will automatically crawl through your web site or local disk in search for documents. SiteXpert comes with 10 navigation system types and over 90 graphical schemes. |
