Scraper Maintenance workflow

  1. Look at the job outputs and/or job logs

  2. If there is an error, pause or cancel the affected job on the DataHen scraper

  3. Go to the scraper directory locally

  4. Modify the seeder script or parser scripts locally

  5. Run the seeder script or parser script locally against some fetched job pages

  6. Deploy the scraper

  7. Resume the scrape job, or run another scrape by creating another scrape job.

  8. Check the job outputs on DataHen