Transcription
There's only two types of people in this world: the conformist API consumer and the renegade web scraper.
It's always been a bit of a gray area to write code that automatically extracts valuable data from websites. But will you go to jail if you do it? You're stealing, right? To jail! I refuse to go back to jail, but I'm literally shaking right now because the US District Court in Delaware just ruled that Booking.com violated the Computer Fraud and Abuse Act by scraping the Ryan Air website.
Booking tried to countersue for defamation because Ryan Air hurt the corporation's feelings by calling them an "OTA pirate" or "online travel agency pirate." Not surprisingly, the court rejected that claim. But this is not the first time that web scrapers have battled in court.
In today's video, we'll look at the legal and ethical concerns in this case and find out if my web scraping tutorials will land you in jail.
It is July 26, 2024, and you're watching The Code Report. In this weird cybernetic post-human world, data has become more valuable than gold. Google and Meta are both trillion-dollar companies built almost exclusively by collecting data from users to be exploited by advertisers.
For the last 30 years, humans have voluntarily and involuntarily filled the worldwide web with exabytes of data about our lives, and much of this data is freely available through the lens of a web browser. But the problem is that the raw data is trapped inside a rock of HTML code, rendering it useless for data analysis.
The data becomes valuable, though, when it's mined into a clean universal format like JSON or CSV that can be fed into machine learning tools like XGBoost or sold to data brokers on the black market. And that's where web scraping comes in.
I've made tutorials about how to do it at an industrial scale using tools like Puppeteer, which can render the content on a website programmatically, click on buttons, fill out forms, and ultimately extract valuable data.
Despite content being freely available, like Amazon product listings for example, many website owners don't like to be scraped and forbid this behavior in their terms of service and robots.txt file. But this file is like the no smoking signs on an airplane; it doesn't actually stop anything from happening. You can easily enjoy a cigarette on a plane, just like the good old days.
However, the flight attendant might come over to you and tell you to stop doing that and ban you from the airline. Website owners can do the same thing by banning your IP address if they suspect you're a scraper.
Now, to bypass this ban, a smoker can change his identity, reboard the plane, and take a few more drags. Web scrapers can do the same thing by rotating their IP addresses via residential proxy networks, making it look like the traffic is coming from normal people around the world.
But this brings up some pretty deep moral and ethical concerns. On one hand, the data is freely available, and it's not exactly theft because the website owner hasn't lost anything. It's just like piracy is not theft. Not everyone agrees with this philosophy, though. One could argue that exploiting someone else's data for profit could violate that person's copyrights.
In the case of Booking versus Ryan Air, Booking was not only scraping the website but also reselling Ryan Air tickets for a profit without their authorization. Not only was there web scraping going on, but also the intent to defraud.
Now, if you're a scraper, the law you'll want to talk to your lawyer about is the Computer Fraud and Abuse Act, or CFAA, which was enacted all the way back in 1986. In 2015, a company called 3Taps was scraping data from Craigslist for a website called PadMapper. Craigslist was not too happy about this and blocked their IP addresses and sent a cease-and-desist letter. But the company continued to scrape until they were sued.
The court sided with Craigslist and set the precedent that online hosts can use the CFAA to protect public data. Ultimately, 3Taps agreed to never scrape Craigslist again and pay $1 million.
The bottom line is that if a company tells you to stop scraping, you have to stop, or you could be sued. But in 2019, there was another high-profile case: HiQ Labs versus LinkedIn. They scraped data from LinkedIn, trying to predict when an employee would leave their job. LinkedIn also sent them a cease-and-desist. However, this time the court ruled in favor of HiQ, allowing them to access LinkedIn's public data. That decision was ultimately affirmed by the Supreme Court.
Just a couple of weeks ago, scrapers got another big win. If you're building an AI tool like GitHub Copilot and scrape a bunch of open-source code to train it, you should be good to go. This lawsuit that says Copilot ignores the licenses and violates the rights of possibly millions of software developers was dismissed by a judge with prejudice, which means it can't be refiled.
And that brings us back to our original question: Will I go to jail for scraping a website? Well, I'm no lawyer, but if you're accessing publicly available data and not defrauding anyone, the chances of going to jail are extremely low.
The only thing to worry about is being sued by a mega corporation that cripples you financially for the rest of your life.
This has been The Code Report. Thanks for watching, and I will see you in the next one.