Analyze robots.txt with Python Standard Library

If haven't searched both "python" and "robots.txt" in the same input box, I would not ever know that Python Standard Library could parse robots.txt with urllib.robotparser.

But the official document of urllib.robotparser doesn't go into detail. With the document, you could check whether a url can be fetch with a robot with robot_parser_inst.can_fetch(user_agent, url) if you are building a crawler bot yourself. But if you want to do some statistics about robots.txt, like what most disallowed path for Googlebot are, from the document not only you don't know how to do it, but also you don't know whether the lib is able to do it.

Surely the lib can do a lot more things, and this article could be supplementary to the official document.

Parse downloaded file

If you download a robots.txt file, and you want to store/cache it and parse it later. You are not going to use the .read() method from the official document but the .parse() method.

Note that you need to pass an iterable of lines to the .parse() method, for example

1fd = open('robots.txt')
2rfp = RobotFileParser()
3rfp.parse(fd.readlines())
4# or as below:
5# rfp.parse(fd.read().splitlines())

After parsing, you may use .can_fetch() method to check with a user agent can crawl a path.

Entry and RuleLine

Beyond checking whether a crawler bot can fetch a url, there is more work need to be addressed on robots.txt files, e.g., data analysis, statistics, etc.. So we need abilities to iterate the rules parsed from robots.txt file. The document of urllib.robotparser doesn't go into detail about the topic. Entry and 'RuleLine' are internal classes that could help us with the abilities disclose the statistics of robots.txt files.

An Entry object is a list of rules with regard to those specified in User-agent. Each Entry contains a .rulelines attribute, which is a list of RuleLine objects. A RuleLine object describes the path and the allow or disallow state of the rule. A parsed RobotFileParser object may contain several Enriy objects, where the entry of user agent '*' is .default_entry, and the rest of the Entry objects are in the .entries list attribute.

Below is an example of Entry and RuleLine objects in a robots.txt, tagged in the comments:

1User-agent: Googlebot  # .entries[0]
2Allow: /blogs/  # .entries[0].rulelines[0]
3Disallow: /  # .entries[0].rulelines[1]
4
5User-agent: *  # .default_entry
6Disallow: /  # .default_entry.rulelines[0]

To iterate over all the rules in a robots.txt file, consider the following code snippet:

1def iterate_rules(robots_content):
2    rfp = RobotFileParser()
3    rfp.parse(robots_content.splitlines())
4    entries = [rfp.default_entry, *rfp.entries]\
5              if rfp.default_entry else rfp.entries
6    for entry in entries:
7        for ruleline in entry.rulelines:
8            yield (entry.useragents, ruleline.path, ruleline.allowance)

The wildcard in Allow/Disallow

The section is not much related with urllib.robotparser, but the protocol of robots.txt itself. In a robots.txt file, you may use '*' as a wildcard symbol in User-agent directives, like

1User-agent: *

But how about use '*' in Allow and Disallow directives? Actually, that depends. The protocol of robots.txt has two major version. The previous version as stated in https://www.robotstxt.org/orig.html has no support of wildcard symbols in Allow/Disallow directives, and urllib.robotparser in Python is implemented based on the version. The newer version as stated in rfc9309 introduced the support of '$' and '*' symbols, Googlebot and Bingbot both support this version.

It is kind of outdated to use the .can_fetch() method in Python Standard Library to check whether a url is crawable. But if you want to do some analysis on a bunch of robots.txt files, urllib.robotparser is still a straightforward yet solid choice.

Monkey patch for urllib.robotparser

We can monkey patch urllib.robotparser to support * and $ as wildcards in Allow and Disallow directives. The code is as below(or in github gist):

 1'''
 2A monkey patch for urllib.robotparser to support * and $ in robots.txt.
 3'''
 4import re
 5import urllib.parse
 6from urllib.robotparser import RobotFileParser, Entry, RuleLine
 7
 8
 9def get_robots_pattern(path):
10    ending = '.*?'
11    if path.endswith('$'):
12        path = path[:-1]
13        ending = '$'
14    parts = path.split('*')
15    parts = map(urllib.parse.quote, map(re.escape, parts))
16    return '.*?'.join(parts) + ending
17
18
19def _rule_line__init__(self, path, allowance):
20    if path == '' and not allowance:
21        # an empty value means allow all
22        allowance = True
23    path = urllib.parse.urlunparse(urllib.parse.urlparse(path))
24    self.pattern = re.compile(get_robots_pattern(path))
25    self.path = path
26    self.allowance = allowance
27
28
29def _rule_line_applies_to(self, filename):
30    return True if self.pattern.match(filename) else False
31
32
33RuleLine.__init__ = _rule_line__init__
34RuleLine.applies_to = _rule_line_applies_to
35
36__all__ = ['RobotFileParser']

Import RobotFileParser from the code above to use the patched version.