The First Rule of the Law of Web Scraping


The first rule of the law of web scraping is this: Details matter.

As I’ve written before, there are dozens of laws associated with web scraping. As such, no single “three-part test” can tell you whether what you’re doing is legal or not. The only way to properly assess that is to look into the details. You need to review all of the websites that you’re scraping, assess the specific techniques you’re using to scrape them, and then apply that analysis to the laws governing those websites in the relevant jurisdictions where that are most likely to be applied to you.

There are some web-scraping practices that are totally benign and there are others that are nearly guaranteed to attract trouble. And, of course, the vast majority fall somewhere in between on that spectrum.

Are you a researcher or a commercial web scraper? Are you scraping publicly available data or beneath a login? Are you scraping personally identifying information or not?  Is the terms of use on the site a browsewrap, a sign-in wrap, or a clickwrap? Is there a captcha that you have to solve in order to scrape? Is the site you’re scraping located in California or Florida? Or perhaps the EU?

The assessment of the risk level associated with your web scraping changes in response to all of those questions. And there are many other questions that are relevant to determining your web-scraping risk.

The details matter. Our firm has reviewed the web-scraping practices of many companies, and I’m often surprised by how my initial assessment of the client’s risk changes after I do a deep dive into the details of what they’re doing. There have been many instances where my initial instinct was to think that a prospective client was doing something low risk, only to work with the client and to find out the opposite was true. And there are other times when my initial instinct is to consider something very risky and then eventually determine that what they were doing was actually relatively low risk.  

This is a complex and evolving area of law. As one of a handful of lawyers in this country who works in this area of law as the majority of his professional practice, it never ceases to amaze me what I see.

But the more I work in this area, the clearer it becomes to me that, when it comes to web scraping and the law, details matter.