Scraping whole website (robots.txt) is a hot debate. Legally, robots.txt isn’t binding—it’s more of a courtesy.
But ethically? If the site didn’t block it, they might not *care*, but still—don’t be a jerk.
For tools, I’d recommend Puppeteer if you need JS rendering.
But ethically? If the site didn’t block it, they might not *care*, but still—don’t be a jerk.
For tools, I’d recommend Puppeteer if you need JS rendering.