Rabobank Web Crawler


The main goal of the Newsreader is to provide Rabobank (and its subsidiaries) Employees with a searchable collection of relevant Banking and Fintech articles. For that purpose, the Newsreader will provide the following functionalities.

  • The Newsreader will crawl full text articles, to enable full text keyword search for the user and to allow for specific filters in the search result (e.g. business domain).
  • The Newsreader will provide end users with only the title and snippet (first couple of lines) of the article, sufficient to decide whether or not to read the full article.
  • The Newsreader will reroute the end user to the original article on the website of origin, to read the full article.

Crawler Policy

  • We will notify you each time we crawl your website, leaving a link to our policy, including the name of the user agent, the interval settings and contact details.
  • The crawler will adhere to the directives in the robots.txt file (if available) and will not crawl disallowed pages.
  • The crawler will access each site in a page by page manner, with some intervals.
  • We will adhere to your further (written) instructions, and implement these within a reasonable timeframe after your notice.
  • We will stop crawling your site altogether after your explicit (written) request, and implement this within a reasonable timeframe after your notice.