SCP09: robots.txt ignored by default

What it does

Reports ROBOTSTXT_OBEY being missing or set to False in a setting module (e.g. settings.py).

Why is this bad?

The default behavior of a Scrapy project should be to respect robots.txt.

Any choice to ignore it [1] should be made on a per-spider basis (e.g. in custom_settings).

[1]

There can be legitimate reasons to ignore robots.txt, such as syntax errors.

Example

ROBOTSTXT_OBEY = False

Use instead:

ROBOTSTXT_OBEY = True