Protecting websites from bots and methods to bypass them

Modern websites aim to protect their data from unauthorized collection through various methods. The reasons for restricting access to their data can vary: the desire to monetize data through paid APIs, the need to protect information from competitors, or simply not wanting to share data with outsiders.

To determine whether a request is from a real user or a bot, websites use various techniques. The primary methods of protection include:

1. HTTP Header Inspection: The order of headers, their case, and the presence of specific headers can indicate whether the request was sent by a real browser or a script. It’s important to note that many automated libraries, such as Python Requests or Scrapy, can be easily identified due to header discrepancies with real browsers.
   Example: A real browser might send a header like User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.124 Safari/537.36, while a bot might use something like User-Agent: python-requests/2.25.1.
2. IP Address Analysis: Using databases of IP addresses belonging to data centers, proxy servers, and Tor networks helps websites quickly identify suspicious requests. If IP addresses change frequently or do not match the user’s location, it raises suspicions.
   Example: If a user regularly changes IP addresses, moving from one region to another in a short time, this may indicate the use of a proxy service.
3. JavaScript Execution: Sites can use JavaScript to perform various tasks, such as checking time delays before sending a response. This helps identify automated requests, as scripts often respond much faster than humans.
   Example: A real user might interact with a form and submit data after a few seconds, while a bot could do it instantly.
4. Fingerprinting: Collecting data about users through JavaScript, such as fonts, installed software, graphics card characteristics, and other system parameters, creates a unique “fingerprint” of the user’s device. This technique allows precise identification of devices without using cookies.
   Example: Two devices with the same set of fonts and screen resolution are likely to be identified as the same device.

To bypass these protections, automated browsers like Puppeteer, Playwright, and Selenium have been developed. These tools can mimic real user behavior, including interacting with JavaScript and correctly displaying HTTP headers. However, even these tools can be detected if additional measures are not taken to conceal their use. Additional bypass methods include:

1. Using Residential Proxies: These proxies provide real IP addresses that belong to physical users, making them difficult to detect.
   Example: Unlike data center proxies, residential proxies can be tied to specific regions and ISPs, making them more reliable for bypassing geographical restrictions.
2. Emulating Real Actions: Automated scripts can be configured to emulate real user actions, such as mouse movements, clicks, and delays between actions.
   Example: A script can be set up with random delays between clicks and text inputs to mimic human behavior.
3. Bypassing Captchas: Various methods exist for bypassing captchas, such as using image recognition services or automatic speech recognition for audio captchas.
   Example: To solve reCAPTCHA v2, a service that provides captcha solutions for a small fee can be used by sending the image for recognition by real people.
4. Analyzing Mobile Applications: Mobile applications often have their own APIs that may be less protected. Tools like Charles Proxy and Frida allow traffic analysis of mobile applications and data extraction.
   Example: Charles Proxy allows intercepting and analyzing HTTPS requests sent by mobile applications, helping to identify and exploit vulnerabilities in their APIs.

Popular bot protection systems like Cloudflare, PerimeterX, and Datadome use all the above methods. They provide multi-layered protection by analyzing headers, IP addresses, network fingerprints, and JavaScript interactions.

Examples of Protection Systems and Bypass Methods

1. Cloudflare:
   • Protection Methods: IP address checks, JavaScript tests, captchas.
   • Bypass Methods: Using Puppeteer to bypass JavaScript tests, employing residential proxies to avoid IP blocking.
2. PerimeterX:
   • Protection Methods: Fingerprinting, behavioral analysis.
   • Bypass Methods: Complex scripts that mimic real user behavior, changes in browser fingerprinting.
3. Datadome:
   • Protection Methods: Network packet analysis, HTTP header inspection.
   • Bypass Methods: Spoofing network data, using automated browsers.

Thus, data protection on websites is a complex and multi-layered process. On the other hand, developers of automated data collection systems constantly seek new ways to bypass these protections, creating an endless cat-and-mouse game between data defenders and those trying to access it.

Need data from a site protected by Cloudflare or a captcha? See website scraping with bot protection (in Ukrainian).