Web Bot Detection

How Web Bot Detection Works

Incoming Ad Traffic → [ Data Collection ] → [ Analysis Engine ] → [ Action ] ┬─ Allow (Legitimate User)
                     │                 │                   │          └─ Block (Fraudulent Bot)
                     │                 │                   │
                  (IP, User-Agent,   (Heuristics,        (Filter,
                   Behavior)          Signatures)         Challenge)

Diagram Element Breakdown

Incoming Ad Traffic

This represents the flow of all visitors—both human and bot—who click on a digital ad and are directed to the advertiser’s website or landing page. It is the starting point of the detection pipeline.

Data Collection

This stage represents the system’s process of gathering identifying information from each visitor. Key data points like IP address, user-agent strings, and behavioral patterns are collected here to be used as evidence for analysis.

Analysis Engine

This is the brain of the operation. The engine processes the collected data using various techniques, such as comparing it against known fraud signatures (e.g., blacklisted IPs), applying heuristic rules (e.g., impossible travel speed), and analyzing behavioral biometrics to differentiate bots from humans.

Action

This is the final, defensive step. Based on the analysis, the system takes a specific action. Legitimate traffic is allowed to pass, while fraudulent traffic is mitigated through blocking, filtering, or issuing a challenge (like a CAPTCHA), thereby protecting the advertiser’s budget.

🧠 Core Detection Logic

Example 1: IP Reputation and Filtering

This logic checks the visitor’s IP address against known blacklists of proxy servers, data centers, and previously identified malicious actors. It’s a first line of defense that quickly blocks traffic from sources with a poor reputation, which are often used to mask the origin of bot traffic.

FUNCTION checkIpReputation(ipAddress):
  IF ipAddress IN knownBadIpList THEN
    RETURN "BLOCK"
  ELSEIF ipAddress IN vpnOrProxyList THEN
    RETURN "FLAG_AS_SUSPICIOUS"
  ELSE
    RETURN "ALLOW"
  ENDIF

Example 2: User-Agent Validation

This technique inspects the user-agent string sent with each request. Bots often use generic, outdated, or inconsistent user agents that don’t match known legitimate browser signatures. This logic flags or blocks traffic with suspicious user-agent strings that deviate from common patterns, indicating non-human activity.

FUNCTION validateUserAgent(userAgentString):
  IF userAgentString IS EMPTY OR userAgentString IS GENERIC_BOT_UA THEN
    RETURN "BLOCK"
  ELSEIF userAgentString NOT IN knownBrowserSignatures THEN
    RETURN "FLAG_AS_SUSPICIOUS"
  ELSE
    RETURN "ALLOW"
  ENDIF

Example 3: Behavioral Heuristics (Click Velocity)

This logic analyzes the timing and frequency of user actions, such as the time between a page loading and an ad being clicked. A human user typically takes a few seconds to orient themselves, while a bot might click instantaneously. Rules based on abnormally high click velocity or frequency help identify automated behavior.

FUNCTION checkClickVelocity(session):
  timeSincePageLoad = session.clickTimestamp - session.pageLoadTimestamp
  
  IF timeSincePageLoad < 1_SECOND THEN
    RETURN "BLOCK"
  ELSEIF session.clicksPerMinute > 30 THEN
    RETURN "FLAG_AS_SUSPICIOUS"
  ELSE
    RETURN "ALLOW"
  ENDIF

📈 Practical Use Cases for Businesses

• Campaign Shielding – Real-time bot detection blocks fraudulent clicks on PPC campaigns the moment they happen, preventing ad budgets from being wasted on traffic that has no chance of converting. This directly protects marketing spend and improves ROI.
• Data Integrity – By filtering out non-human traffic, businesses ensure their analytics platforms (like Google Analytics) reflect genuine user engagement. This leads to more accurate metrics, such as conversion rates and bounce rates, enabling better strategic decisions.
• Lead Generation Quality – For businesses running lead-generation campaigns, bot detection filters out fake form submissions. This prevents sales teams from wasting time and resources on fraudulent leads and keeps the CRM database clean and reliable.
• Improved Return on Ad Spend (ROAS) – By ensuring that ad spend is directed only toward legitimate human users, businesses can achieve a higher return on their investment. Clean traffic leads to higher-quality interactions and a greater likelihood of conversions for the same budget.

Example 1: Geofencing Rule

This pseudocode demonstrates a common use case where a business wants to ensure ad clicks are coming from its target geographic regions. Clicks originating from unexpected or known high-fraud locations are automatically blocked to protect the ad campaign budget.

FUNCTION enforceGeofencing(visitorIp):
  visitorCountry = getCountryFromIp(visitorIp)
  allowedCountries = ["USA", "Canada", "UK"]

  IF visitorCountry NOT IN allowedCountries THEN
    ACTION: BlockRequest("Traffic from this region is not allowed.")
    RETURN FALSE
  ENDIF
  
  RETURN TRUE

Example 2: Session Scoring Logic

This example shows how a system can score a visitor’s session based on multiple risk factors. A session with a high score is deemed fraudulent and blocked. This is more sophisticated than a single rule, as it aggregates evidence to make a more accurate decision and reduce false positives.

FUNCTION calculateSessionRisk(session):
  riskScore = 0

  IF session.ipType == "Data Center" THEN
    riskScore = riskScore + 40
  ENDIF

  IF session.hasHeadlessBrowserFingerprint == TRUE THEN
    riskScore = riskScore + 50
  ENDIF
  
  IF session.timeOnPage < 2_SECONDS THEN
    riskScore = riskScore + 15
  ENDIF

  IF riskScore > 80 THEN
    ACTION: BlockSession("High-risk session detected.")
  ENDIF

🐍 Python Code Examples

This Python function simulates checking for rapid, repeated clicks from the same IP address within a short time frame. This is a common pattern for simple click fraud bots and helps in identifying non-human velocity.

CLICK_TIMESTAMPS = {}
TIME_WINDOW_SECONDS = 60
CLICK_LIMIT = 20

def is_abnormal_click_frequency(ip_address):
    """Checks if an IP address exceeds a click frequency threshold."""
    import time
    current_time = time.time()
    
    if ip_address not in CLICK_TIMESTAMPS:
        CLICK_TIMESTAMPS[ip_address] = []
    
    # Remove old timestamps
    CLICK_TIMESTAMPS[ip_address] = [t for t in CLICK_TIMESTAMPS[ip_address] if current_time - t < TIME_WINDOW_SECONDS]
    
    # Add current click
    CLICK_TIMESTAMPS[ip_address].append(current_time)
    
    # Check limit
    if len(CLICK_TIMESTAMPS[ip_address]) > CLICK_LIMIT:
        print(f"Blocking IP {ip_address} for excessive clicks.")
        return True
        
    return False

# Example Usage
is_abnormal_click_frequency("198.51.100.5") # Returns False
# ...imagine 20 more clicks from the same IP in under 60 seconds...
is_abnormal_click_frequency("198.51.100.5") # Would eventually return True

This code filters a list of incoming web requests by checking for suspicious user-agent strings. Bots often use generic or known malicious user agents, which can be easily filtered out to block low-quality traffic.

def filter_suspicious_user_agents(requests):
    """Filters out requests with known bad or missing user agents."""
    SUSPICIOUS_AGENTS = ["bot", "spider", "scraper", "HeadlessChrome"]
    legitimate_requests = []
    
    for request in requests:
        user_agent = request.get("user_agent", "").lower()
        is_suspicious = False
        if not user_agent:
            is_suspicious = True
        else:
            for keyword in SUSPICIOUS_AGENTS:
                if keyword in user_agent:
                    is_suspicious = True
                    break
        
        if not is_suspicious:
            legitimate_requests.append(request)
        else:
            print(f"Filtered out request from {request['ip']} with UA: {request.get('user_agent')}")
            
    return legitimate_requests

# Example Usage
traffic_log = [
    {"ip": "203.0.113.1", "user_agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36"},
    {"ip": "198.51.100.1", "user_agent": "AhrefsBot/7.0"},
    {"ip": "203.0.113.2", "user_agent": "Python-urllib/3.9 (bot)"}
]
clean_traffic = filter_suspicious_user_agents(traffic_log)
# clean_traffic will contain only the first request

Types of Web Bot Detection

• Signature-Based Detection – This method identifies bots by matching their attributes against a known database of malicious signatures. Signatures can include IP addresses, user-agent strings, and request headers associated with known botnets or scraping tools. It is effective against known threats but struggles with new or sophisticated bots.
• Behavioral Analysis – This approach focuses on *how* a visitor interacts with a website, rather than *who* they are. It analyzes patterns like mouse movements, click speed, navigation paths, and session duration to distinguish human behavior from the more predictable, rapid actions of a bot.
• Fingerprinting – This technique involves collecting a detailed set of parameters from a visitor’s device and browser, such as screen resolution, installed fonts, browser plugins, and operating system. This unique “fingerprint” can identify bots that try to mask their identity and track them across different sessions and IP addresses.
• Challenge-Based Detection – This method actively challenges a visitor to prove they are human, most commonly through a CAPTCHA test. While effective, it can create friction for legitimate users and may be solved by advanced bots, so it is often used as a secondary validation method.