Robots Exclusion Standard/Protocol Parser

Use this library within crawler/scraper programs to parse robots.txt files and check if your crawler user-agent is allowed to load certain paths.

Table of contents

Requirements

Requires PHP version 7.4 or above.

Installation

Install the latest version with:

composer require crwlr/robots-txt

Usage

use Crwlr\RobotsTxt\RobotsTxt;

$robotsTxtContent = file_get_contents('https://www.crwlr.software/robots.txt');
$robotsTxt = RobotsTxt::parse($robotsTxtContent);

$robotsTxt->isAllowed('/packages', 'MyBotName');

You can also check with an absolute url.
But attention: the library won't (/can't) check if the host of your absolute url is the same as the robots.txt file was on (because it doesn't know the host where it's on, you just give it the content).

$robotsTxt->isAllowed('https://www.crwlr.software/packages', 'MyBotName');