Home
User Agents
New Agents
List All
My User Agent
Add New
User-Agents Database
Edit User Agent: Scrapy
Name:
robots.txt:
N/A
Yes
No
DataMiningCommercial
DataMiningPublic
DownloadManager
EmailHarvesting
GuestbookSpamming
HackTool
ImageDownload
IndexingTool
Monitoring
OfflineBrowser
OtherBad
Trademark
ValidationTool
LinkChecking
Browser
MediaPlayer
Proxy
Adware
BrowserExtension
Spyware
Editing
Device
NewsFeed
SearchEngine
FilteringSoftware
SoftwareComponent
Translation
SEO
MailClient
AI
Key:
META tags:
N/A
Yes
No
URL:
User Agents (separate each entry with a CRLF):
Scrapy/0.15 (+http://scrapy.org) Scrapy/0.17.0-222-ge1f4144 (+http://scrapy.org)
IP Ranges (separate each entry with a CRLF):
Information:
Scrapy is a fast high-level screen scraping and web crawling framework, used to crawl websites and extract structured data from their pages. It can be used for a wide range of purposes, from data mining to monitoring and automated testing. fell for bad bot trap!
FromHeader
User Agents - Search
Enter keyword or user agent:
AQTRON
i
X
Serious about Security
Copyright © 2025 AQTRONiX. All rights reserved.
