Documentation for crwlr / crawler (v0.1)

Attention: You're currently viewing the documentation for v0.1 of the crawler package.
This is not the latest version of the package.
If you didn't navigate to this version intentionally, you can click here to switch to the latest version.

Response Cache

This is still kind of a half-baked feature, which is why I currently don't really want to document it. If you're curious, you can have a look at the getFromCache method in the HttpLoader.

The purpose of this feature: when you're working on a crawler, you will probably run it again and again. Performing actual HTTP requests during all the test runs makes the runs unnecessary slow and produces a lot of network traffic and traffic on the crawled servers. Using the response cache during development allows for very fast test runs, once the needed pages have been loaded for the first time.