There are many times in our browsing experience we want to download images or video-chunks that are in a particular sequence. I have used software that does the same Link Sequence Downloader. I know the same is possible using curl command but didn’t spend time learning it. Today I explored and wanted to document my learning here.
Reading curl’s manpage, this capability is titled as “Globbing”. It gives the following options to replace parts of the URL.
- With a List of different names by using {}
This command will download three files (one.txt, two.txt, three.txt).
Note: In the curl command if the output option is not provided then it will simply print the file content in the terminal. So remember to use -O (uppercase O) which instructs curl to use the same remote-file-name for the downloaded file.
# List of name (downloads 3 files)
curl -O "https://raptor.in/developer/download-sequence/different-names/{one,two,three}.txt"- With sequences of alphanumeric series by using []
# Numeric sequence (downloads 3 files)
curl -O "https://raptor.in/developer/download-sequence/sequence-alphanumeric/8k-background-gradient-[9-11].jpg"
# Numeric sequence with leading zeros (downloads 3 files)
curl -O "https://raptor.in/developer/download-sequence/sequence-alphanumeric/8k-background-gradient-[009-011].jpg"
# Alphabet sequence (downloads 3 files)
curl -O "https://raptor.in/developer/download-sequence/sequence-alphanumeric/8k-background-gradient-[a-c].jpg"- With multiple sequences in the url (nesting not supported)
In this example, the curl command should download 27 files. Here we are downloading file named SectionA.txt, SectionB.txt and SectionC.txt across Vol1-3 for years 1999-2001.
If we use -O (uppercase O) then the output will have only three files (SeciontA-C.txt for Vol 3 for year 2000). The problem here is, the files that we are downloading from different paths have the same filenames. So even though the curl command downloads all 27 files when it outputs it is getting overwritten. To avoid this we can use-o(lowercase O) followed by the filename you prefer.
Since we are using sequences in the URL to download the files, curl command internally converts these sequences into variables and these variables can be accessed with the following syntax #1 contains the value of the 1st sequence, #2 contains value of 2nd sequence and so on. In this example the value of #1 would be 1999-2000 for the 1st sequence. Now this command will download 27 files and output 27 files.
# Using -O will overwrite existing files (downloads 3 files)
curl https://raptor.in/developer/download-sequence/multiple-sequence-alphanumeric/\[1999-2001\]/Vol\[1-3\]/Section\{A,B,C\}.txt -o "#1-Vol#2-Section#3.txt"
# Use -o to avoid output overwrite (downloads 27 files)
curl https://raptor.in/developer/download-sequence/multiple-sequence-alphanumeric/\[1999-2001\]/Vol\[1-3\]/Section\{A,B,C\}.txt -o "#1-Vol#2-Section#3.txt"- With step counter to get every Nth number or letter
This curl command will download incremental sequence of file from the given starting number. In the first example, 20 is the starting number and increments to 22, 24, 26, so it downloads 4 files. In the second example, 21 is the starting number and increments to 23, 25, so it downloads 3 files.
# Step counter - Downloads 4 files (20, 22, 24, 26)
https://raptor.in/developer/download-sequence/stepcounter-alphanumeric/8k-background-gradient-[20-26:2].jpg
# Step counter - Downloads 3 files (21, 23, 25)
https://raptor.in/developer/download-sequence/stepcounter-alphanumeric/8k-background-gradient-[21-26:2].jpgUsing Remote File Name instead of custom
While downloading, if you want to use the same file name as remote file name using the `–remote-name` option
curl -O --remote-name "https://raptor.in/developer/download-sequence/stepcounter-alphanumeric/8k-background-gradient-[20-26:2].jpg"Reference
