What's the best curl command to download from a URL list file? or How do I use the curl command to do

14 Replies, 1144 Views

"What's the best curl command to download from a URL list file?"

Hey folks!

I've got a text file with a bunch of URLs, and I need to download 'em all using curl. I know there's a way to do it with a single curl command to download from url list file, but I'm kinda stuck on the exact syntax.

Tried googling, but some answers are way too complicated. Anyone got a simple, working example? Like, do I use `-K` or `xargs` or something else?

Also, bonus points if you know how to save the files with their original names. Thanks in advance!

(PS: If this has been asked before, sorry—my search-fu is weak today.)
Hey! The easiest way to do this is with `xargs` and curl. Just run:

```bash
xargs -n 1 curl -O < urls.txt
```

This will download each URL in the file and save it with the original filename (thanks to `-O`).

If you wanna speed things up, try `-P` flag with xargs for parallel downloads, like `-P 4` for 4 at once.

Also, if you're dealing with a ton of URLs, check out `aria2c`—it's way faster for bulk downloads.
For a pure curl command to download from url list file, you can use `-K` to read the file directly. Like this:

```bash
curl -K urls.txt
```

But your file needs to have each URL prefixed with `url = ` (e.g., `url = https://example.com/file1`).

If you don’t wanna edit the file, stick with `xargs`. Btw, `wget` also has a `-i` flag for this, if you’re open to alternatives.
Man, I feel you—curl syntax can be a pain. Here’s what I use:

```bash
while read url; do curl -O "$url"; done < urls.txt
```

Super simple. Loops through each line, downloads with original names.

Pro tip: Add `-L` to follow redirects if some URLs bounce around. And maybe `-s` to silence the output if it’s spammy.
If you’re on macOS or Linux, this one-liner works like a charm:

```bash
cat urls.txt | xargs -I % curl -O %
```

The `-I %` lets you handle weird chars in URLs better.

Also, if you wanna log errors, toss in `-f` to fail silently and redirect errors to a file. Handy for big batches.
Not sure why everyone’s overcomplicating this. Just use:

```bash
curl $(cat urls.txt) -O
```

Boom. Done.

But fair warning—it’ll choke if your file has spaces or special chars. For bulletproofing, the `while read` loop is safer.
For a more structured approach, try this:

```bash
curl -Z --parallel-max 5 -O $(<urls.txt)
```

The `-Z` enables parallel downloads, and `--parallel-max` limits concurrent connections.

Also, if you’re on Windows, check out `curl` via WSL or use PowerShell’s `Invoke-WebRequest`.

---
Wow, thanks everyone! Tried the `xargs -n 1 curl -O` one and it worked perfectly.

Quick follow-up: Some URLs are failing with 403 errors—any way to retry those automatically? Also, is there a flag to skip existing files so I don’t re-download?

Y’all saved me hours of googling. Cheers!



Users browsing this thread: 1 Guest(s)