Advanced Tutorials

How to write a QA script to ingest and parse outputs from multiple scrapers

Step 1: Setup

In this tutorial we are going to go over the steps to do QA (Quality Assurance) on the output from your scrapers so you can validate that the data you scrape is correct. We do this by creating another “QA” scraper that will ingest the output from your initial scraper and perform QA on it using rules that you define.

For this example, we are going to be creating a QA scraper for the, “Simple Ebay scraper” that we previously created in, “Coding Tutorials.” Let’s create an empty directory and name it ‘ebay-scraper-qa’

$ mkdir ebay-scraper-qa

Next let’s go into the directory and initialize it as a Git repository

$ cd ebay-scraper-qa
$ git init .
Initialized empty Git repository in /Users/workspace/ebay-scraper-qa/.git/

Create a file called Gemfile inside the ebay-scraper-qa directory with the following line. This will add the dh_easy-qa Gem, which is the Gem that handles the QA.

gem 'dh_easy-qa'

Now run the following command in the ebay-scraper-qa directory to install this gem:

bundle

Next let’s make a yaml file that will contain the configuration for the QA. Create a file called dh_easy.yaml inside the ebay-scraper-qa directory with the following contents:

qa:
  scrapers:
    ebay-scraper:
      - listings

This is the configuration we use to tell the QA what scrapers to use and which collections within each scraper to perform QA on. In this example, we are going to be doing QA on our ebay-scraper scraper and, more specifically, the listings collection. You can add as many scrapers and as many collections here as you want, but keep in mind that the same QA validation rules will be applied to all of them. Basically, whatever scrapers and collections you add should have similar output. If you have another scraper that has different output and different validation rules, you will need to create a separate QA scraper.

Step 2: Seeding

Now, let’s create a seeder script. This is where we will iterate through the scrapers that we just defined and seed them to be fetched by DataHen. In this example we will just seed one scraper.

Create the seeder directory:

mkdir seeder

Next, let’s create a ruby script called seeder.rb in the seeder directory with the following content:

config_path = File.expand_path('dh_easy.yaml', Dir.pwd)
config = YAML.load(File.open(config_path))['qa']
config['scrapers'].each do |scraper_name, collections|
  if collections
    pages << {
      url: "https://fetchtest.datahen.com/?scraper=#{scraper_name}",
      method: "GET",
      page_type: "qa",
      vars: {
        scraper_name: scraper_name,
        collections: collections
      }
    }
  end
end

This will create a page on DataHen with the scraper (ebay-scraper) and collection (listings) that we want to validate saved in the vars. We will eventually create a parser that will send these values to the dh_easy-qa gem which will load the relevant data from DataHen and perform QA on it using rules that we will eventually define as well.

Let’s go back into the root directory of the project

$ cd ..
$ ls -alth
total 0
drwxr-xr-x  10 johndoe  staff   320B 26 Nov 16:19 .git
drwxr-xr-x   3 johndoe  staff    96B 26 Nov 16:15 seeder
drwxr-xr-x   1 johndoe  staff    57B 26 Nov 16:15 Gemfile
drwxr-xr-x   4 johndoe  staff   128B 26 Nov 16:15 .
drwxr-xr-x  10 johndoe  staff   320B 26 Nov 15:59 ..

Now that we’ve created the seeder script, let’s see if there are any syntax error in it by trying the seeder script.

$ hen seeder try ebay-example-qa seeder/seeder.rb
Trying seeder script
=========== Seeding Executed ===========
----------------------------------------
Would have saved 1 out of 1 Pages
[
  {
    "url": "https://fetchtest.datahen.com/?scraper=ebay-scraper",
    "method": "GET",
    "page_type": "qa",
    "force_fetch": true,
    "vars": {
      "scraper_name": "ebay-scraper",
      "collections": [
        "listings"
      ]
    }
  }
]

Looks like our seeder script test was successful. Let’s now create the config file to tell DataHen about our seeder. Create a config.yaml file in the root project directory with the following content:

seeder:
 file: ./seeder/seeder.rb
 disabled: false # Optional. Set it to true if you want to disable execution of this file

The config above simply tells DataHen where the seeder file is, so that it can be executed.

Let’s now commit our files with git and push them to Github. Add all of the current files with the following commands:

$ git add .
$ git commit -m 'create inital qa files'
[master (root-commit) 7632be0] create initial qa files
 1 file changed, 5 insertions(+)
 create mode 100644 seeder/seeder.rb
 create mode 100644 Gemfile
 create mode 100644 Gemfile.lock
 create mode 100644 dh_easy.yaml
 create mode 100644 config.yaml

Next, let’s push it to an online git repository provider. In this case let’s push this to Github. In the example below it is using our git repository, you should push to your own repository.

$ git remote add origin https://github.com/DataHenOfficial/ebay-scraper-qa.git
$ git push -u origin master
Counting objects: 4, done.
Delta compression using up to 8 threads.
Compressing objects: 100% (2/2), done.
Writing objects: 100% (4/4), 382 bytes | 382.00 KiB/s, done.
Total 4 (delta 0), reused 0 (delta 0)
remote:
remote: Create a pull request for 'master' on GitHub by visiting:
remote:      https://github.com/DataHenOfficial/ebay-scraper-qa/pull/new/master
remote:
To https://github.com/DataHenOfficial/ebay-scraper-qa.git
 * [new branch]      master -> master
Branch 'master' set up to track remote branch 'master' from 'origin'.

Congratulations, you’ve successfully seeded the scraper collections we want to validate with QA and are ready to define some validation rules, which we will do in the next step.

Step 3: QA Validation Rules

Now that we have seeded a page to DataHen with info about the scraper that we want to perform QA on, we can define some validation rules. Add some lines to the dh_easy.yaml file so that it looks like the following:

qa:
  scrapers:
    ebay-example:
      - listings
  individual_validations:
    url:
      required: true
      type: Url
    title:
      required: true
      type: String

Basically this will tell the QA gem to perform the following validations on the listings collection of our ebay-example scraper. It will make sure the url and the title are both present and it will make sure that the url output is actually a url and that the title output is a string. If any of these validations fail, the failure will be returned in a summary and the specific listing will be returned with the corresponding failure.

Next we need to create a parser, which will parse the page that has our scraper (ebay-example) and collection (listings) info saved in vars, and use the dh_easy-qa gem to perform QA. In our seeder we set the page_type to qa, so lets create a parser named qa.rb inside a folder named parsers.

First lets create the parsers directory:

mkdir parsers

Next create a file called qa.rb inside this parsers directory with the following lines:

require 'dh_easy/qa'
DhEasy::Qa::Validator.new.validate_internal(page['vars'], outputs)

Here we are telling the QA gem to validate internal scrapers on DataHen and are passing the details of these scrapers inside the vars. We also pass the outputs array which is a special reserved word in DataHen that is an array of job output to be saved. This way the QA gem will be able to save the QA output to DataHen for you to see.

Let’s retrieve the GID of the page that we seeded earlier so we can try it out locally. Run the following command in the project root directory.

hen scraper page list ebay-example-qa
[
 {
  "gid": "fetchtest.datahen.com-1767f1fa6b7302b4a618b16b470fc1d2",
  "job_id": 9793,
  "job_status": "active",
  "status": "parsing_failed",
  "fetch_type": "standard",
  "page_type": "qa",
  "priority": 0,
  "method": "GET",
  "url": "https://fetchtest.datahen.com/?scraper=ebay-example",
  "effective_url": "https://fetchtest.datahen.com/?scraper=ebay-example",
  "headers": null,
  "cookie": null,
  "body": null,
  "created_at": "2019-08-09T21:44:18.709737Z",
  "no_redirect": false,
  "ua_type": "desktop",
  "freshness": "2019-08-09T21:44:18.735754Z",
  "fresh": true,
  "parsing_at": null,
  "parsing_failed_at": "2019-08-09T22:05:30.684121Z",
  "parsed_at": null,
  "parsing_try_count": 3,
  "parsing_fail_count": 3,
  "fetched_at": "2019-08-09T21:45:10.312099Z",
  "fetching_try_count": 1,
  "to_fetch": "2019-08-09T21:44:18.73264Z",
  "fetched_from": "web",
  "response_checksum": "9d650deb8d3fd908de452f27e148293d",
  "response_status": "200 OK",
  "response_status_code": 200,
  "response_proto": "HTTP/1.1",
  "content_type": "text/html; charset=utf-8",
  "content_size": 555,
  "vars": {
   "collections": [
    "listings"
   ],
   "scraper_name": "ebay-example"
  },
  "failed_response_status_code": null,
  "failed_response_headers": null,
  "failed_response_cookie": null,
  "failed_effective_url": null,
  "failed_at": null,
  "failed_content_type": null,
  "force_fetch": false
 }
]

We can see the scraper name and collection are both present in “vars,” but what we are interested in is the gid which will look something like, “fetchtest.datahen.com-1767f1fa6b7302b4a618b16b470fc1d2.” We can use the gid to try out our parser, which will perform the QA, on this page.

Run the following command, replacing the <gid> part with your gid value:

hen parser try ebay-example-qa parsers/qa.rb <gid>

The output should look something like:

Trying parser script
getting Job Page
1
2
validating scraper: ebay-example
Validating collection: listings
data count 42
=========== Parsing Executed ===========
----------------------------------------
Would have saved 1 out of 1 Outputs
[
  {
    "pass": "true",
    "_collection": "ebay-example_listings_summary",
    "total_items": 42
  }
]

This means that our validation rules in dh_easy.yaml have passed for each of the 42 items. This also means that you have successfully performed basic QA on your scraper! We will look at more advanced settings in the next section.

Additional Validation Rules

These are examples of all the available validation rules. You use them by adding them to dh_easy.yaml nested under ‘individual_validations.’ For example, here is a validation rule that makes sure a field named, “Title” is present, is a string, and has a length of 10.

qa:
  individual_validations:
    Title:
      required: true
      type: String
      length: 10

Here are all the possible validation rules and values:


length: 5

Validates the length of a field value. The field value can be an Integer, Float, or a String.


type: String

Validates that a value is a String.


type: Integer

Validates that a value is an Integer. The value can be a number in a String. Examples that would pass: 10, ‘10’, ‘1,000’.


type: Float

Validates that a value is a Float (a number with a decimal point). The value can be a number with decimal points in a String. Examples that would pass: 2.0, ‘3.14’, ‘99.99’.


type: Date
format: '%d-%m-%Y'

Validates that a value is a date. A format is required using Ruby strftime.


type: Url

Validates that a value is a valid url.


value:
  equal: 'test'

You can also add validations that validate the value of a field itself. For example, the above validation will validate that a field is equal to the string, ‘test’.


value:
  less_than: 5

You can also verify that a field is less than something.


value:
  greater_than: 100

You can also verify that a field is more than something.


value:
  regex: "^(Monday|Tuesday|Wednesday|Thursday|Friday|Saturday|Sunday)$"

Validates the value of a field using a regular expression. For example, you could validate that a value is a phone number or a day of the week like the above example. Regex uses case ignore by default.


title:
  value:
    equal: 'Test title'
    if:
      search_input:
        value:
          equal: 'Search'

You can also implement conditions on value validations. For instance, the above example validates that the value of a field named, ‘title’ has a value equal to ‘Test title’ only if the value of the field named, ‘search_input’ has a value equal to, ‘Search.’ If statements currently support value checks with the same options as a normal value check (less_than, greater_than, regex, and equal).


title:
  required: true
  if:
    search_input:
      value:
        regex: '(One|Two|Three)'

You can also implement an if condition on ‘required.’ The above example will only check if the ‘title’ field is required if the field named, ‘search_input’ has a value equal to: ‘One,’ ‘Two,’ or ‘Three.’


title:
  required: true
  if:
    and:
      -
       field1:
         value:
           regex: '(One|Two)'
      -
       field2:
         value:
           less_than: 100

If conditions can also take ‘and’ and ‘or’ operators. The above example shows a validation that will only check if the ‘title’ field is required if the field named, ‘field1’ has a value equal to: ‘One’ or ‘Two’ and the field named, ‘field2’ has a value that is less than 100.

Group Validations

In addition to these individual validations you can also perform more complicated validations on the data as a whole. For example, you may want to ensure that a specific field has ranked values and are in order. To add group validations create a file named group_validations.rb in your QA scraper root directory with the following (data is an array of the items you are performing QA on):

module GroupValidations
  def validate_count
    errors[:group_validation] = { failure: 'count' } if data.count > 100
  end
end

This example would create an error if the total number of items is greater than 100. Let’s look at another another example.

module GroupValidations
  def validate_count
    errors[:group_validation] = { failure: 'count' } if data.count > 100
  end

  def validate_unique_ids
    ids = data.collect{|item| item['id'] }
    errors[:group_validation] = { failure: 'unique_ids' } if ids.uniq.count != ids.count
  end
end

This would add another validation that checks to make sure all item ‘id’ values are unique. You can edit these examples to create your own.

Thresholds

Thresholds are useful if you want to suppress errors based on frequency. You can suppress errors on the basis of the number of errors relative to the number of items you are performing QA on. The threshold itself is a number between 0 and 1 where 1 means that if any error occurs, the error will show. A threshold of 0 means we are ignoring all errors. There are multiple ways you can set a threshold which we will go through below.

We can set a threshold on a per field basis which will apply to all scrapers. This can be done by adding “threshold” to the dh_easy.yaml file to a specific field just like a rule. For example, the following will add a threshold that will only show errors on the “Title” field for every scraper if the occurance rate is above 60%.

qa:
  individual_validations:
    Title:
      threshold: 0.6
      required: true
      type: String
      length: 10

We can also set thresholds on a per field and a per scraper scraper basis at the same time. Using the “Title” field example, this means you could have a threshold of 0.6 for the Title on one scraper and have 1.0 for another scraper. In order to implement different thresholds for individual scrapers you can create a file called thresholds.yaml in your scraper root directory. Here is an example of a thresholds.yaml file that would apply different thresholds for a scraper named ebay1 and a scraper named ebay2.

---
ebay1:
   Title:
     threshold: 0.6
     required: true
     type: String
ebay2:
   Title:
     threshold: 1.0
     required: true
     type: String

External sources

In addition to performing validation on scrapers that run on DataHen (internal sources) you can also perform validation on external sources. For example, if you have a scraper that runs somewhere else, you can validate it by ingesting a json endpoint. Here is an example seeder for an external source:

pages << {
   url: "http://dummy.restapiexample.com/api/v1/employees",
   method: "GET",
   force_fetch: true,
   freshness: Time.now.iso8601,
   vars: {
     collection_id: "employees-1"
   }
}

This seeder could be expanded to seeding multiple endpointpoints by loading a YAML file and iterating through like in Step 2 above. After seeding our external json endpoint we can now write a parser such as the following:

require 'typhoeus'
require 'json'
require 'dh_easy/qa'

collection_name = page['vars']['collection_id']
json = JSON.parse(content)
qa = DhEasy::Qa::Validator.new(json, {})
qa.validate_external(outputs, collection_name)

We can also implement thresholds with external sources by loading a thresholds yaml and passing it into the validator options. We can update our parser so that it looks like the following:

require 'typhoeus'
require 'json'
require 'yaml'
require 'dh_easy/qa'

collection_name = page['vars']['collection_id']
file_path = File.expand_path('thresholds.yaml', Dir.pwd)
thresholds = YAML.load(File.open(file_path))
options = { 'thresholds' => thresholds[collection_name] }

json = JSON.parse(content)
qa = DhEasy::Qa::Validator.new(json, options)
qa.validate_external(outputs, collection_name)