How to scrape a whole website's robots.txt file properly? or Best way to scrape a whole website's rob

Quick note: Some sites hide rules in sitemaps or dynamic endpoints.

So scrape whole website robot.txt how to isn’t always enough.

I cross-check with "robots-txt-checker" tools online to see if rules are being followed.

Python’s urllib.robotparser is built-in but kinda basic.