If you’ve followed the news much lately, particularly as it relates to new and emerging technologies, you’ve probably heard a thing or two about bad bots or automated threats. While the issue with bots is not new and emerging, the companies popping up attempting to solve the problem are.
What do we mean by bad bots?
When we hear bots, most of us think of the Google crawler that reads our website to help rank us on search engine results. Or maybe we think of chat bots when we’re shopping online and someone chimes in on a chat window to help us out. Yes, these are indeed bots. Bots are essentially automated programs (scripts) that run tasks on the internet. However, most of us don’t know that there are just as many, if not more bad bots on the Internet than there are good bots.
Bad bots are designed to act as similar to a human as possible. Let me put this into perspective. Let’s say you opened your laptop, pulled up your browser and started surfing the internet for an hour. Maybe you were just reading the news, your favorite magazine articles, looking at crazy memes, gambling online, viewing adult content (sshhh), shopping, or whatever tickles your fancy. The point is, you can create a bot that will mimic every single page you viewed, link you clicked, fields you may have typed into, etc…so essentially, on the surface, if we videotaped your computer screen next to a bots screen, you wouldn’t have a clue which is which.
What are bad bots doing and who is affected?
Bad bots are doing a variety of things, which this website outlines. Take a look at our Botnet Attacks page. If you want the in depth overview, check out the OWASP Top 20 Automated Threat Handbook
Attacks vary depending on the industry you’re in. The most common threats that are seen include web scraping, spamming, ad fraud, account and credit card fraud, and skewed analytics.
For example, if you run an Ecommerce site, the most common and critical attacks are scraping of prices and inventory data, account and card fraud, and skewed analytics. Scraping your key product data enables fraudsters and competitors to use this against you. They’ll either create a duplicate site with your information or they’ll leverage your prices and inventories to undercut you on prices. This drives traffic away from you and places you in a worse position in search results. They’ll also try to crack into accounts with username and password combinations, as well as credit card credentials. If the marketing team is seeing a bunch of bogus traffic hitting their site, it makes it very difficult to track conversions and drive key business decisions.
If you’re a digital publisher, ad fraud is the big one. Conversion rates are everything and if you have bots loading ad impressions or clicking on ads, those analytics will be all messed up. It looks terrible to advertisers and throws off their numbers making it difficult to execute on key business decisions. This is becoming more of a concern for publishers and many are now focused on delivering quality over quantity.
Real Estate companies care about skewed analytics and scraping. One of the biggest revenue generators in the real estate listing world is lead generation. These sites generate home buyer leads for their agents. They don’t want bots filling in bogus information and messing up the quality of those leads. These sites also need to comply with the rules under the National Association of Realtors. The quality of listing data is key, so they don’t want bots scraping this data and using it on another website.
The travel industry is another big one. This is one of the most price competitive industries there is. The last thing they want is having their flight and price data stolen and used against them. If I’m a site like Orbitz for example, and I scrape Expedia for prices on the same flights I’ve listed, I can use that data to undercut Expedia and get the booking.
I can go on and on about the various threats that businesses see, but you get the gist. Basically, any site on the Internet is a good target for bots. There just happen to be bigger targets than others. At the end of the day though, if you have valuable content on your site, you’re a target. And even if you have a login with some valuable information on the other side, a bot can run a ton of automated attempts at guessing username/password combinations and crack in. One step further, once they are in, they can run the same attempts against credit card credentials.
Here’s an example of a very professional botnet tool that allows you to run an automated script, specifically designed for cashing out.
Cashing out is essentially when a hacker uses valid credit card credentials to steal money or goods online. This tool shows just how easy it is to run a bot. You just plug in a few fields of information and click submit.
How do we stop these automated threats?
You can’t. I wouldn’t recommend starting an online business. Kidding. So here I will outline some methods to prevent these threats. Keep in mind, there is no 100%, sure fire way of blocking them all, but you can still do a damn good job.
By far the most effective way of dealing with this issue is to use a bot mitigation company. Yes, there are actually companies dedicated to solving this issue and it’s become it’s own industry. You will find that some of these providers focus on specific industries and threats and some just block bots all together.
You will want to figure out what’s best for your needs. Some providers only monitor the traffic and give you visibility into your bot activity. Some will block the bots all together, or at least give you the option of how you want to respond to a bad bot in real-time. Take a look at their detection methods as this will play a big part in determining their accuracy of detection. There is usually some type of fingerprint involved. A basic fingerprint may just use the IP and header information, while a more in depth fingerprint uses many data points about the requestors browser to create a cookie. This cookie will follow the requestor where it goes, even it bounces around different IP addresses. This goes a long way in bot mitigation.
Don’t get tricked into thinking your firewall or WAF will do the trick. Both of these tools are reactive in nature and can only block simple or stupid bots. Most bots these days are super smart and insanely hard to detect. They will fly right under a WAF’s radar. A WAF is basically there to patch holes in some bad code that was written in your application. They will protect you from those code vulnerabilities like SQL injections and cross-site scripting being the most common. This has nothing to do with automated bots acting like humans. You can plug rules into your WAF, rate limit, and block known bad actors based on IP and header information, but that’s about it. Most bots are being used across hundreds or thousands of IPs, so blocking based on IPs alone is pointless. If the bot makes 2 requests from 1,000 IPs, rate limiting does nothing. As it relates to headers, these can be easily manipulated by the attacker. It’s very easy to make it look like you’re coming from a real browser, like the latest version of Chrome.
So the point being, look into bot mitigation vendors that focus on this exact problem of automated threats. Hopefully this article gives you some helpful insight. Please take some time to read through the rest of our site to stay informed. If you don’t feel that you have a bot problem, you’re either really lucky, or you just don’t know. Whether you consider bot mitigation as a means to solve a known problem or you’re buying an insurance policy, it’s something to take very seriously. Heck, if OWASP writes a handbook on it, you know it’s real.
Leave a Reply