PHP Composer Packages for Crawler and Scraper Development

crwlr.software is a collection of open source PHP composer packages that provide the necessary tools to build web crawlers and scrapers. The crawler package contains everything and helps you build crawlers as fast as possible. There are also sub-packages that you can use standalone.

Get started now!

you@computer:~/Projects/MyCrawler

Packages

crwlr / crawler

The main package of this collection, providing kind of a framework and a lot of ready to use, so-called steps, that you can use to build your own web crawlers and scrapers with.

crwlr / url

The Swiss Army knife for urls. Parses urls to components (scheme, host, domain, path,...). You can access and modify url components, compare components of different urls and resolve relative to absolute urls. Also supports internationalized domain names.

crwlr / robots-txt

Use this library within crawler and scraper programs to parse robots.txt files and check if your crawler user-agent is allowed to load certain paths.

Latest Blog Posts

What's new in crwlr / crawler v0.4

2022-05-10

Last friday version 0.4 of the crawler package was released with some pretty useful improvements. Read what's shipped with this new minor update.

» Read more

What's new in crwlr / crawler v0.2 and v0.3

2022-04-30

There are already two new 0.x versions of the crawler package. Here a quick summary of what's new in versions 0.2 and 0.3.

» Read more

Release of crwlr / crawler v0.1.0

2022-04-18

After months of hard work, today I'm finally releasing the first version (v0.1.0) of the crwlr / crawler package. Here some information on what it is, its state and current and future features.

» Read more

Prevent Homograph Attacks using the crwlr / url Package

2022-01-19

Homograph attacks are using internationalized domain names (IDN) for malicious links including domains that look like trusted organizations. You can use the crwlr Url class to detect and monitor urls containing IDNs in your user's input.

» Read more

Why I start crwlr.software

2018-04-15

This is just a short introduction to what crwlr.software is and will become in the future and why you may like it.

» Read more