Malicious User-Agent Detection in Logs

AttackPatterns Web Application Scanning / Bot Activity

What This Means

Identify malicious User-Agent strings in web server logs. Detect vulnerability scanners, bots, and attack tools by analyzing User-Agent patterns targeting your web infrastructure.

Example Log

-- Access log showing suspicious User-Agent strings:
203.0.113.55 "GET /admin HTTP/1.1" 404 0 "-" "sqlmap/1.7.2"
203.0.113.56 "GET / HTTP/1.1" 200 5120 "-" "Nikto/2.1.6"
203.0.113.57 "GET /wp-login.php HTTP/1.1" 404 0 "-" "WPScan v3.8.24"
203.0.113.58 "GET / HTTP/1.1" 200 5120 "-" "python-requests/2.28.0"
203.0.113.59 "GET /.env HTTP/1.1" 403 0 "-" "Go-http-client/1.1"
203.0.113.60 "GET / HTTP/1.0" 200 5120 "-" "-"

Indicators of Suspicious Activity

User-Agent strings identifying known scanning tools (sqlmap, nikto, WPScan, Acunetix, Nessus)
Generic HTTP library User-Agents (python-requests, Go-http-client, curl, Java)
Empty or missing User-Agent headers
User-Agent strings claiming to be old browsers (IE 6/7/8) from non-legacy IPs
Spoofed Googlebot User-Agent from non-Google IP addresses
User-Agent strings containing exploit payloads or shell commands

How to Investigate

Parse access logs and group requests by User-Agent string
Identify known malicious or scanning tool User-Agents in your logs
Verify claimed search engine bots using reverse DNS lookup
Check the behavior of suspicious User-Agents (what URIs they request, error rates)
Correlate malicious User-Agents with IP addresses for further blocking
Track new or unusual User-Agent strings appearing in your logs

Recommended Mitigations

Block known scanning tool User-Agent strings at the WAF or web server level
Implement bot detection that analyzes behavior, not just User-Agent strings
Require JavaScript execution challenge for suspicious User-Agents
Block requests with empty User-Agent headers
Verify search engine bot claims using reverse DNS
Deploy a bot management solution for sophisticated bot detection

Scan This Log Instantly

Paste a suspicious log line below and get an instant AI-powered security assessment.

0 / 2000

What is 9 + 3?

Need a Full Investigation?

Scan entire log files, detect attack patterns, reconstruct timelines, and generate a full investigation report.

Run Smart Scan

Related Log Types

Related Attack Patterns

Frequently Asked Questions

Can I rely on User-Agent strings for security? ▼

No. User-Agent strings are trivially spoofed. They are useful for detecting unsophisticated scanners but should not be your primary defense. Combine User-Agent analysis with behavioral detection and IP reputation.

Which User-Agent strings should I block? ▼

Block known tools: sqlmap, nikto, WPScan, Acunetix, Nessus, dirbuster, gobuster. Also consider blocking empty User-Agents and generic library clients from IPs that are not your own integrations.

How do I verify a Googlebot claim? ▼

Perform a reverse DNS lookup on the IP. Legitimate Googlebot IPs resolve to crawl-xxx-xxx-xxx-xxx.googlebot.com. If the reverse DNS does not match, the User-Agent is spoofed.