Hey there! I’ve been using wget scrape urls for a while now. One pitfall I’ve seen is not setting `--robots=off` when needed. Some sites block based on robots.txt, so be careful with that.
For saving files, I use `-r` for recursive downloads and `-np` to avoid parent directories. Keeps things clean!
Also, check out `--no-clobber` to avoid overwriting files. Super handy.
For saving files, I use `-r` for recursive downloads and `-np` to avoid parent directories. Keeps things clean!
Also, check out `--no-clobber` to avoid overwriting files. Super handy.