robots.txt file parsing and checking for R. Contribute to ropensci/robotstxt development by creating an account on GitHub.