The internet is extensive, and the amount of data online only continues to grow. Web and Application Programming Interfaces (API) scraping bots crawl the internet, harvesting and aggregating a corpus of information; if it’s online, they will scan it. Scraping bots mimic human behavior, allowing the bot to “pass” for a legitimate user or even an official API client.
There are many legitimate uses for web and API scraping bots— security research, search engine results, IP address mapping, vulnerability scanning, and more. However, the persistent nature of web scraping bots can also make them a nuisance, especially when they repeatedly scrape APIs.
What is web and API scraping?
Web scraping bots automatically crawl the internet, sending HTTP requests and receiving HTML page content in return. Because APIs return information in structured formats like XML or JSON they can be easier for bots to scrape, which incentivizes bots to target APIs.
Are scraping bots legal?
At first glance, the legality of web and API scraping bots may seem questionable—they’re accessing content that they are not intended to access and using the content for unauthorized purposes. The difference between legitimate and malicious web scraping bots lies in how the bot interacts with the website and how the information is used.
Discerning legitimate web scraping bots from malicious ones can be challenging, but there are differences. Legitimate bots are identifiable as belonging to the organization for which they gather information. For example, Googlebot identifies itself in its HTTP User-Agent header as belonging to Google and uses a defined set of source IP network blocks. Other bots are authenticated with an API user and key, which is like a unique password assigned to that specific bot.
Legitimate web scraping bots also abide by the rules each website sets forth in its ‘robots.txt’ file, which spells out which portions of the website can be crawled. Malicious web scrapers may attempt to mimic a valid HTTP User-Agent header but will likely ignore the rules set forth by a website.
Impact of scraping bots on APIs.
Many modern businesses rely on APIs to allow third parties to interface with their data and services. However, they don’t want API scraping bots to make iterative requests of their API because they cannot control the information the bot will receive, the frequency at which it scrapes for information, or what it will do with the information. Because of this, most publicly available or “open” APIs, such as weather predictions or the U.S. government Open Data initiative, require third parties to authenticate with the API using an API key requiring registration. In contrast, when bots scrape an API that does not require registration and authentication, the API owners have a limited set of resources to block that misbehaving API client.
Whether a scraping bot is poorly programmed or intentionally malicious, it can cause API server overload, resulting in slower response times or service interruptions. These misbehaving web and API scraping bots can significantly impact API Performance. Businesses should take steps to prevent API scraping to avoid application performance issues as web and API scraping bots are resource intensive.
Why bots love APIs (and why businesses hate API scraping bots).
While businesses may not want a bot continuously scraping their APIs, it’s important to understand that bots love APIs.
APIs respond in structured formats that are easy for web scraping bots to parse, such as JSON, XML, etc. The bot can easily extract the information it needs from an API, and because API requests and responses are smaller, the bot can do this faster than it would compared to the HTML content on a website. APIs also lack the client-side protections that websites can use to detect (and block) a bot. In fact, most API traffic is from friendly bots because of these advantages.
Businesses don’t want to serve traffic to non-friendly bots—after all, bots aren’t paying customers. From needing to add extra capacity in their API and website to added service provider costs, API scraping bots can be expensive for businesses. All of which underscores the need for bot management to block malicious bots while allowing friendly ones.
Malicious scraping bots.
Malicious bots can cause a variety of problems for businesses, including intellectual property theft, inventory hoarding, account takeovers, credential stuffing, and even fraud. No business wants a malicious bot crawling their website. Some examples of malicious web scraping bots include:
Inventory hoarding bots.
These bots systematically map product information, including SKUs, descriptions, and prices on eCommerce websites. This data can be used as competitive intelligence to undercut competitors or create unfair market conditions.
Scalping bots.
Have you ever wanted a limited-edition pair of sneakers, a video game system, or airline tickets and been unable to purchase them? Scaping bots acquire items and then resell them at a markup, often at auction sites.
Credential stuffing bots.
These bots quickly and repeatedly try to gain access to an account, usually online banking accounts but also eCommerce or rewards points sites, by checking known username and password combinations. Once a valid combination is detected, the bot herder logs in as the user, locks them out of the account, and proceeds to withdraw available funds or cash-out in any way that they can.
Phishing toolkit bots.
These bots create look-alike website logins that are used in phishing campaigns. By using the web page layout and images from the real site, users are unable to identify that the phishing site is malicious and will steal their login credentials.
Content scraping bots.
Everything is content, and threat actors know this. Content web scraping bots download articles, images, or videos in bulk to repost them elsewhere without permission, thereby violating copyright laws and adversely impacting the content owner.
Email scraping bots.
Also called email harvesting bots, these web scraping bots specifically scrape email addresses and compile them in lists to be used in email phishing attacks.
DDoS bots.
Some malicious bots will scrape APIs and websites excessively to overload the servers and the infrastructure that supports them. The result is an availability disruption for other users and a potential loss of income for the duration of the outage.
Vulnerability Scanning bots.
Malicious bots are also handy reconnaissance tools. Threat actors routinely use bots to scan the internet, search for exposed services, and identify targets of opportunity. Because of their automated nature, bots can scan many APIs and websites on the internet, evaluate their application stack, and identify outdated versions and vulnerabilities that can be exploited.
How to stop web and API scraping.
As with all things on the internet, not all bots are bad. Many helpful bots improve our daily internet browsing experience. The heart of bot management is blocking bad bots while allowing friendly bots, some of which may be web or API scraping bots. While it isn’t feasible to entirely block scraping bots, there are methods businesses can employ to manage how these bots interact with their web presence:
Behavior-based detection.
Web scraping bots attempt to look human, but behavior-based detection solutions can identify suspicious behaviors. For example, high click rates, rapid page navigation, or uniform time intervals between actions may suggest bot activity.
Implement rate controls.
Rate controls limit the requests a single IP or user can make within a specific period and can be applied to the network, application, and API levels. However, rate limits can affect legitimate users and even APIs if a bot scrapes the API excessively.
IP blocklists for cloud service providers.
A lot of scraping bots are run on virtual machines inside Cloud Service Providers (CSPs). IP blocklists block all traffic from known malicious IP addresses from reaching your website. IP blocklists can be individual IP addresses or blocks of IP addresses. While IP blocklists can be an effective tactic to block malicious activities, they should be layered with other security controls as threat actors can spoof their IP addresses.
Deploy an API and bot management protection solution.
Protecting your online presence means protecting your web application and your API. Vercara’s UltraAPI and UltraAPI Bot Manager allow businesses to understand the entirety of their API landscape and detect malicious bot activity before it impacts web performance. UltraAPI Bot Manager can also enforce workflows, for example, forcing users to go to the login page before submitting the login form and receiving information.
Restrict your API.
All traffic looks the same to an API— this is part of the challenge of protecting APIs and why bot management is so critical for API security. If your business currently uses an open API, ensure you require authentication. Vercara’s UltraAPI Bot Manager helps businesses identify open parts of their API and dynamically block unauthenticated API requests.
CAPTCHA.
Completely Automated Public Turing test to tell Computers and Humans Apart (CAPTCHA) challenges can help differentiate between human users and web scrapers. CAPTCHA challenges are effective at blocking automated web scraper bots but can impact a user’s experience on the site if overused.
Stop bot attacks in their tracks.
Web scraping isn’t inherently bad, but as with any other type of automated internet activity, businesses need to ensure the reliability and availability of their online presence. Modern businesses need reliable, scalable protection from automated bot attacks. Anti-scraping solutions should be able to identify and prevent website and API scraping based on known behavioral patterns and analytics.
To learn how Vercara’s UltraAPI Bot Manager can help you detect and manage bots, visit our solutions page.
