Honestly, I gave up on making ChatGPT crawl websites directly. Instead, I use Screaming Frog to scrape the site first, then feed the text to ChatGPT.
It’s extra steps, but at least it’s reliable. Plus, you get full control over what content gets pulled.
It’s extra steps, but at least it’s reliable. Plus, you get full control over what content gets pulled.
