Hey! I’ve been scraping airnow data too, and yeah, it can be a pain. 😅
I switched from BeautifulSoup to Scrapy for webscraping airnow, and it’s been a game-changer. Handles pagination way better, and you can set up middleware to avoid rate limits.
Also, check out the `requests-html` library if you want something lighter than Selenium but still need to handle dynamic content. It’s been super helpful for me.
Good luck!
I switched from BeautifulSoup to Scrapy for webscraping airnow, and it’s been a game-changer. Handles pagination way better, and you can set up middleware to avoid rate limits.
Also, check out the `requests-html` library if you want something lighter than Selenium but still need to handle dynamic content. It’s been super helpful for me.
Good luck!