Scraper Rules

Scraper rules are the configuration instructions that tell a web scraper where to go and what to extract at each crawl depth level.

First, select the crawl depth level at which the actions will be executed. Crawl depth level is the number of links the program must follow to navigate from the start page to the current page. All start pages are at depth level 0. Pages linked directly from any start page have a depth level of 1. A page's depth level is always one greater than the page it was opened from. If a page is level 1, pages opened from it are level 2, and so on.

There are two ways to add a depth level: by clicking the "Add" button or by clicking the "Duplicate" button. A click on the "Add" button just adds a new level that contains no actions. A click on the "Duplicate" button adds a new level and copies all actions from the current level to the new one. You will be able to edit the level by clicking the "Edit" button. You can delete the level by selecting it and and clicking the "Delete" button.

Then, click the "Add" button to add new scraper actions, In the window that appears, wait for the page to load, and then select the links you want the program to follow and the text you want the program to extract. You can edit scraper actions later by clicking the "Edit" button.

You can add additional actions by clicking the "Add Additional Actions" button.

Options

• Any deeper level - if this option is enabled, all actions configured for the current depth level will also be executed on all pages at deeper levels. It allows you to define actions for a single level and have them automatically applied to all subsequent deeper levels. Use this option when the link and text positions are consistent across all deeper-level pages. For example: If links to product details are always located in the same positions on every page with search results you only need to specify these link positions once on the first page.
• Maximum Crawl Depth - it is the deepest level where the program stops the scraping process.
• Crawling Order - it is the order of loading pages:
  • Breadth-First Crawling - if this option is selected, pages of lower level are loaded first.
  • Depth-First Crawling - if this option is selected, pages of higher level are loaded first.