Metadata-Version: 2.4
Name: RobotsTxt_TheHunterDog
Version: 0.0.3
Description-Content-Type: text/markdown

# RobotsTxt

## Description
The project is a robots.txt parser. The parser is written in Python, easy to use and has small improvements for typos. The project is open source and encourages contributions and ideas for improvements.

## Usage
The usage of the package is simple. first you have to init the parser.
```python
USER_AGENT = "GOOGLEBOT"
RobotsTxt_TheHunterDog.initRobotsTxt(USER_AGENT, robots_txt_url = "https://www.google.com/robots.txt")
```
when it has been initailized the allowed and disallowd list has been parsed and stored. Each get request must be checked against the list. this is being done with the built in make get request function.
```python
USER_AGENT = "GOOGLEBOT"
RobotsTxt_TheHunterDog.main.makeRequest("https://www.google.com", USER_AGENT)
```

The user agent is required for each request, so that the user agent is supplied to the server.
