The digital age all but demands your business tap into big data to remain competitive, but web scraping tools can be difficult to wield effectively. After all, websites are constantly deploying more advanced countermeasures against bots, including IP blocking, CAPTCHA, honeypots, and more. Even as the nature of web design evolves to favor dynamic content presentation, basic scrapers are increasingly befuddled searching for simple HTML buried under elaborate JavaScript wizardry.
But with Zenscrape, none of those challenges are insurmountable. In fact, effective web scraping has never been easier–and you don’t even need to know how to code. So, how does it work? We explore Zenscrape’s customizable, streamlined SaaS offering in our review below.
What is web scraping, and how can it help your business?
You may not need this primer on web scraping–after all, since you’re on this page, there’s a fair chance you’re already in the know. But just so we don’t leave anyone behind during the course of this review, let’s cover some of the basics.
In essence, web scraping is the process of parsing and collecting data that is publicly available on one or many websites. You’ve probably already heard about Google’s “spiders”, which “crawl” all over the internet to assess what sorts of information websites are serving up to potential visitors. This amalgamation of data allows Google to make intelligent decisions about which search results to return. (It’s the same process that brought you to this webpage after typing in “best web scraper api”!)
Example use-cases
So how can web scraping help your business? Let us count the ways! Actually, that’s probably way beyond the scope of this article, so let’s just zoom in on a few possible use-cases:
- Comparison shopping – Imagine you’re selling a specific product on your webstore, and are considering running a sale. In order to know how deeply to cut prices, it will help to know what the competition is doing. You can spend ages manually searching other online stores, or you can use a web scraper to do it for you. Better yet, why not have the web scraper constantly monitor your competitors, alerting you in real time as their prices and inventory rise and fall? That’d be one heck of an advantage.
- Tracking stocks – You run a financial advisory site, and your visitors need up-to-date information on specific stock movements. Do you hire a massive team to monitor the marketplace, or do you let a computer do it for you? Obviously, the latter is (far) more efficient and productive. Such is the power of a web scraper!
- Managing your reputation online – People have been talking about your business, but odds are you’re not going to be a fly on the wall of every public forum where reviews of your outstanding service take place. Similarly, one of your customers may be venting about a problem that you could solve–if only you knew they were venting on that Facebook group or subreddit. With a web scraper, you essentially have a radar constantly scanning the Internet for mentions of your brand, allowing you to paint a clearer picture of what people are saying about your business, and where they’re having those conversations.
Why Zenscrape is awesome
With a little imagination, it’s easy to dream up all kinds of uses for web scraping. So why should you choose Zenscrape specifically? In a word: usability. The visionary minds behind Zenscrape understand that you have a massive need for data, but may not have an advanced IT degree or the budget to hire someone in-house who has one.
To wit, Zenscrape promises incredible results without demanding you write a single line of code. So how does it deliver on that claim? We investigate below.
The Zenscrape API
Let’s start off by saying something that may seem entirely contradictory to our previous claim: if you’re looking to perform DIY web scraping with Zenscrape’s API, you’ll need to know some coding. But just bear with us a moment–Zenscrape’s business model is tailored to people who want their cake, and to eat it, too.
Speaking less poetically, this means that Zenscrape provides deeply customizable tech in the form of their API; but more importantly, they offer comprehensive support to help you use and get the most out of that tech. You tell them what you want it to do, and they’ll pull the levers and squash the bugs behind the scenes to make it happen. It’s dead simple–and requires zero coding knowledge to communicate your needs.
So with our technophobic jitters firmly in check, let’s talk tech for a minute.
Under the hood
Essentially, it starts with your API key, which is automatically generated for you and can be rotated as necessary. From there, your website or app can invoke the Zenscrape API with a simple GET request. The base URL to which you’ll target that request looks like:
https://app.zenscrape.com/api/v1/get?apikey=APIKEY
There are a few parameters which define where and how your scraper collects data, including:
- url – Where on the web you want to source your data
- location – Where you want to locate your Zenscrape proxy (more on this later)
- render – Using a headless browser is extremely useful for cutting through UI “fluff” that stylizes most modern web pages, and getting to the raw HTML data you need. This option counts against your request volume (more on this later)
- premium – Some websites are particularly hard to scrape, and using local proxies can help you get around common countermeasures to prevent bots like Zenscrape from collecting data. This option also counts against your request volume
- keep_headers – Another optional parameter designed to dodge common stumbling blocks like cookies and user agents
Once you’ve got your target set (we’ll use http://toscrape.com as an example), output will look something like this:
<!DOCTYPE html>
<html lang="en">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
<title>Scraping Sandbox</title>
<link href="./css/bootstrap.min.css" rel="stylesheet">
<link href="./css/main.css" rel="stylesheet">
</head>
<body>
<div class="container">
<div class="row">
<div class="col-md-1"></div>
<div class="col-md-10 well">
<img class="logo" src="img/sh-logo.png" width="200px">
<h1 class="text-right">Web Scraping Sandbox</h1>
</div>
</div>
<div class="row">
<div class="col-md-1"></div>
<div class="col-md-10">
<h2>Books</h2>
<p>A <a href="http://books.toscrape.com">fictional bookstore</a> that desperately wants to be scraped. It's a safe place for beginners learning web scraping and for developers validating their scraping technologies as well. Available at: <a href="http://books.toscrape.com">books.toscrape.com</a></p>
<div class="col-md-6">
<a href="http://books.toscrape.com"><img src="./img/books.png" class="img-thumbnail"></a>
</div>
<div class="col-md-6">
<table class="table table-hover">
<tr><th colspan="2">Details</th></tr>
<tr><td>Amount of items </td><td>1000</td></tr>
<tr><td>Pagination </td><td>✔</td></tr>
<tr><td>Items per page </td><td>max 20</td></tr>
<tr><td>Requires JavaScript </td><td>✘</td></tr>
</table>
</div>
</div>
</div>
<div class="row">
<div class="col-md-1"></div>
<div class="col-md-10">
<h2>Quotes</h2>
<p><a href="http://quotes.toscrape.com/">A website</a> that lists quotes from famous people. It has many endpoints showing the quotes in many different ways, each of them including new scraping challenges for you, as described below.</p>
<div class="col-md-6">
<a href="http://quotes.toscrape.com"><img src="./img/quotes.png" class="img-thumbnail"></a>
</div>
<div class="col-md-6">
<table class="table table-hover">
<tr><th colspan="2">Endpoints</th></tr>
<tr><td><a href="http://quotes.toscrape.com/">Default</a></td><td>Microdata and pagination</td></tr>
<tr><td><a href="http://quotes.toscrape.com/scroll">Scroll</a> </td><td>infinite scrolling pagination</td></tr>
<tr><td><a href="http://quotes.toscrape.com/js">JavaScript</a> </td><td>JavaScript generated content</td></tr>
<tr><td><a href="http://quotes.toscrape.com/tableful">Tableful</a> </td><td>a table based messed-up layout</td></tr>
<tr><td><a href="http://quotes.toscrape.com/login">Login</a> </td><td>login with CSRF token (any user/passwd works)</td></tr>
<tr><td><a href="http://quotes.toscrape.com/search.aspx">ViewState</a> </td><td>an AJAX based filter form with ViewStates</td></tr>
<tr><td><a href="http://quotes.toscrape.com/random">Random</a> </td><td>a single random quote</td></tr>
</table>
</div>
</div>
</div>
</div>
</body>
</html>
If that made your eyes glaze over, don’t worry about it. (For the record, this is just the HTML that constitutes the body of a relatively simple website. A “real” site will yield more complex, dynamic results). With the Zenscrape API properly configured, your website or app will be able to pick out the important bits, and return useful results over time. And, getting to that point is where Zenscrape–as a SaaS company–really shines. Let’s explore their service offering further…
Zenscrape Services
You don’t have to know how to manage APIs directly to get advanced functionality out of Zenscrape; you simply need to have a goal in mind, and their expert team will take care of the rest. For any data that you could possibly want to parse, Zenscrape will happily construct a custom solution for you. Here’s a rundown of their most popular packages:
Automated Sales Intelligence
If you run a business, you already understand the value of market research. Zenscrape aids in this endeavor by helping you collect hard data, enabling you to assemble smarter customer segments. Datasets can include contact information for the customers of both your business and those of your competitors, among others.
Brand Monitoring
We’ve touched upon this already, but it bears mention how deeply Zenscrape can delve into the perception of your brand. Scraping everything from social media conversations to site reviews can help you construct a clear impression of where your brand messaging and customer support is succeeding, and where it can be improved. Zenscrape will work closely with you to define the most important metrics, and build out your API to track them.
Review Aggregation
Part and parcel with brand management is review aggregation. Not everyone will think to write reviews directly onto your site, but may do so on public forums like Yelp, Google, and others. Zenscrape will help pull all of those reviews together, allowing you to display them where they are most needed: Your storefront.
Price & Product Monitoring
Don’t let the market run circles around you! With Zenscrape, dynamic price, inventory, and product feature monitoring helps you build and retain your competitive edge. If there’s room to expand your margins, Zenscrape’s data-driven approach will help you achieve it. Monitor your market in real time, visualize your standing amidst competitors, and make more accurate predictions with Zenscrape.
Hiring Data
One of the most sophisticated ways to understand emerging markets is looking at hiring trends. Zenscrape gives you a summarized view of even the most crowded job boards and career pages. This can also give you valuable insights into the strategies of competitors as they grow or attenuate certain departments.
Machine Learning
Building reliable deep learning models requires data–and lots of it. Zenscrape helps you train your machine learning model by accumulating vast troves of highly targeted data of any type.
Custom Solutions
All of the above service categories are merely popular suggestions of what you can do with Zenscrape. The actual breadth of possibilities is truly endless, and their support team is ready to help you craft the perfect web scraper for your needs. With Zenscrape, technology is no longer a limiting factor in gathering the data you need to accelerate your business.
Simply reach out to Zenscrape for a free consultation call. Together, you’ll brainstorm your goals and solutions, from which Zenscrape will return some sample data after a few days. Once you’ve fine-tuned your requirements and agreed to the quote, Zenscrape will build out the perfect solution and begin returning your precious data in short order.
Zenscrape’s Data Extraction Tool
So we’ve talked at length about what you can do with the Zenscrape API (plus what you can ask their all-star customer service to help you do), but what if you’re looking for a simpler solution? Enter the Zenscrape Data Extraction Tool.
The cure for the common code is a GUI, and Zenscrape has expertly crafted a streamlined data extraction tool that runs primarily off mouse-clicks, rather than keystrokes. Getting started scraping has really never been easier–once you log in, you’re greeted with a single unassuming button that simply prompts “Create Scraper”.
Once you’ve done that, you’ll enter a nickname for your shiny new bot, then input the URL you want the scraper to target. Let’s use Yahoo Finance as an example, with the goal of monitoring Dow Futures.
After a couple seconds, Zenscrape will fetch data from Yahoo Finance, presenting all on-page elements precisely the same as any human visitor would see. From there, we want to “add a selector”, which is a fancy way of saying “choose some data on the page you want to monitor”. Give it a nickname, specify the type of information (in this case, it’s text), then select the page element that displays the value of the Dow Futures.
Finally, we need to configure how your Zenscrape bot exports the data it collects. Obviously, we want to turn it on, so select an “Active” status, choose your preferred file format (we’re using Excel), then specify the time interval in which Zenscrape re-checks Yahoo Finance for the Dow Futures. Once this is done, hit save!
After giving your bot some time to work its magic, navigate to “Results” in the sidebar. You’ll find a list of all the scrapers you’ve created, plus a handy “Download All” button. Once you click this, you’ll harvest all that sweet data Zenscrape has been diligently collecting for you, in an easy-to-parse format that you can use as you like. Simple, right?
Pricing
Zenscrape is customizable, right down to the payment plans. By default, there are five payment tiers, though it is possible (and encouraged) to negotiate a more tailored subscription.
Free
When Zenscrape says “free”, they mean it. There’s no upfront cost, no trial period, nor any obligation to provide payment information for them to keep on file. You don’t get the incredible onboarding of the other tiers, but with 1,000 monthly requests (limit 1 concurrent request), JS rendering, geotargeting, and all the standard proxies, a knowledgeable user will be able to get great mileage out of the free plan if limited data scraping is required.
Small – $8.99/month
The smallest paid tier is a significant step up. You get 50,000 monthly requests and can run any number of them concurrently, plus JS rendering, geotargeting, standard proxies and premium proxies. Just keep in mind that running those proxies counts against your total monthly requests (standard=5 requests, premium=20 requests). If the numbers alone weren’t enough, you get concierge onboarding, which is in our opinion where the real value lies in Zenscrape’s SaaS offering.
Medium – $24.99/month
The most popular tier, you get a ton of horsepower for a modest monthly price. We’re talking a quarter-million monthly requests, JS rendering, geotargeting, and every available proxy. The higher request limit means you can be more aggressive with your proxy usage, allowing you to parse much more difficult-to-obtain data. As with the Small tier, you get customized onboarding to best meet your needs.
Large – $79.99/month
Enough to cover the needs of most SMBs, the Large tier offers everything in the Medium tier, but raises the monthly cap to a staggering 1,000,000 requests.
Very Large – $199.99/month
Got a truly data-hungry business? Consider the Very Large tier, which gives you ample headroom to perform the most sophisticated scraping en masse. The monthly subscription fee isn’t insignificant, but represents a 50% discount over the Medium price point at 3,000,000 requests. What’s more, the concierge onboarding becomes even more valuable as your scraping scales.
Conclusion
At this point, it’s clear to see that Zenscrape is whatever you need it to be in terms of collating data from the web. Its sophisticated API helps you plumb the depths of your market and brand, allowing you to improve the value proposition of your business to your customers. However, Zenscrape has also made it easy to get up and running without advanced coding knowledge, with their streamlined Data Extraction Tool. Whether you’re comfortable with DIY scraping or require customized consultation to build out the perfect data collection system, Zenscrape has you covered at a competitive price point.
Has our review piqued your interest in Zenscrape? What data drives your business? Reach out to us in a comment below.