Hey! Great thread. I’ve been working with web crawlers to get the utility rate API for a while now, and yeah, it’s a bit of a headache sometimes. One tool I swear by is Scrapy—it’s super flexible and handles a lot of the heavy lifting for you.
Also, if you’re scraping, check out BeautifulSoup for parsing HTML. It’s a lifesaver when dealing with messy websites.
For proxies, I’ve had good luck with Bright Data—they’re pricey but worth it if you’re doing a lot of scraping.
And yeah, always check for APIs first. I found that some utility companies hide their APIs in their JavaScript files, so inspect the network tab in your browser dev tools. You might get lucky!
Also, if you’re scraping, check out BeautifulSoup for parsing HTML. It’s a lifesaver when dealing with messy websites.
For proxies, I’ve had good luck with Bright Data—they’re pricey but worth it if you’re doing a lot of scraping.
And yeah, always check for APIs first. I found that some utility companies hide their APIs in their JavaScript files, so inspect the network tab in your browser dev tools. You might get lucky!