Home
User Agents
New Agents
List All
My User Agent
Add New
User-Agents Database
Edit User Agent: URL Spider Pro
Name:
robots.txt:
N/A
Yes
No
DataMiningCommercial
DataMiningPublic
DownloadManager
EmailHarvesting
GuestbookSpamming
HackTool
ImageDownload
IndexingTool
Monitoring
OfflineBrowser
OtherBad
Trademark
ValidationTool
LinkChecking
Browser
MediaPlayer
Proxy
Adware
BrowserExtension
Spyware
Editing
Device
NewsFeed
SearchEngine
FilteringSoftware
SoftwareComponent
Translation
SEO
MailClient
AI
Key:
META tags:
N/A
Yes
No
URL:
User Agents (separate each entry with a CRLF):
IP Ranges (separate each entry with a CRLF):
Information:
URL Spider Pro is designed for creating a small Google-like search engine. You simply configure (through keyword and domain filters) the spider for the type of information it should index and it will crawl through the Web collecting only documents matching that configuration. You can also instruct it to crawl through one or more specific domains, making it easy to add site search functionality to your Web site.
FromHeader
User Agents - Search
Enter keyword or user agent:
AQTRON
i
X
Serious about Security
Copyright © 2025 AQTRONiX. All rights reserved.