cURL, short for “Client for URLs”, is a command line tool for transferring data using various protocols. This article aims at providing the commands that are widely used and as a handy reference.
Table of Contents
- Download a Single File
- Save the cURL Output to a File
- Hide Progress Bar
- Fetch Multiple Files at a Time
- Follow HTTP Location Headers with -L Option
- Continue/Resume a Previous Download
- Use a Proxy with or Without Authentication
- Query HTTP Headers
- Upload Files to FTP Server
- Download Files from FTP Server
- List/Download Using Ranges
- More Information Using Verbose and Trace Option
- Send Mail Using SMTP Protocol
- HTTP/2 Support Check
- Simulate HTTP Methods
- Make a POST Request with Parameters
- Conclusion
You can invoke curl command from your terminal without thinking about ways to install it, as it comes pre-installed on most Linux based Operating Systems.
There are a vast amount of use-cases for curl, such as:
- FTP upload
- Proxy support
- SSL connections
- HTTP post
cURL also supports the use of all the following protocols: DICT, FILE, FTP, FTPS, GOPHER, HTTP, HTTPS, IMAP, IMAPS, LDAP, LDAPS, POP3, POP3S, RTMP, RTSP, SCP, SFTP, SMB, SMBS, SMTP, SMTPS, TELNET, and TFTP.
Download a Single File
The following command will get the content of the URL and display it in the STDOUT (i.e on your terminal).
curl https://www.gnu.orgTo store the output in a file, you an redirect it as shown below. This will also display some additional download statistics.
curl https://www.gnu.org > gnu-org.htmlSave the cURL Output to a File
We can save the result of the cURL command to a file by using -o/-O options.
-o(lowercase o) the result will be saved in the filename provided in the command line-O(uppercase O) the filename in the URL will be taken and it will be used as the filename to store the result
curl -o my-gettext.html https://www.gnu.org/software/gettext/manual/gettext.html As a result, now the page gettext.html will be saved in the file named my-gettext.html. Also you can note that when running cURL with -o option, it displays the progress meter for the download as follows.
When you use cURL -O, it will save the content in the file named ‘gettext.html’ itself in the local machine.
curl -O http://www.gnu.org/software/gettext/manual/gettext.htmlNote: When curl has to write the data to the terminal, it disables the Progress Meter, to avoid confusion in printing. We can use >, -o, -O options to move the result to a file.
Hide Progress Bar
cURL by default shows progress bar. To hide it -s (--silent)option can be used.
curl -s -O http://www.gnu.org/software/gettext/manual/gettext.htmlIf for some reason that does not work on your platform, you could always redirect stderr to /dev/null:
curl -O http://www.gnu.org/software/gettext/manual/gettext.html 2>/dev/nullFetch Multiple Files at a Time
Of course, we can download multiple files in a single shot by specifying the URLs on the command line.
curl -O https://www.gnu.org/software/gettext/manual/html_node/index.html -O https://www.gnu.org/software/gettext/manual/gettext.html Follow HTTP Location Headers with -L Option
However by default cURL doesn’t follow the HTTP Location headers. It is also termed as Redirects. When a requested web page is moved to another place, then an HTTP Location header will be sent as a Response and it will have where the actual web page is located.
We can insists cURL to follow the redirection using -L option, as shown below.
curl -L https://www.google.comContinue/Resume a Previous Download
Using cURL -C option, you can continue a download which was stopped already for some reason. This will be helpful when you download large files, and the download got interrupted.
If we say -C -, then cURL will find from where to start resuming the download. We can also give an offset -C <offset>. The given offset bytes will be skipped from the beginning for the source file.
Start a big download using curl, and press Ctrl-C to stop it in between the download.
curl -O https://www.gnu.org/software/gettext/manual/gettext.htmlUsing curl -C -, we can continue the download from where it left off earlier.
curl -C - -O https://www.gnu.org/software/gettext/manual/gettext.htmlUse a Proxy with or Without Authentication
If you are behind a proxy server listening on port 8080 at proxy.yourdomain.com, do:
curl -x proxy.yourdomain.com:8080 -U user:password -O https://www.gnu.org/software/gettext/manual/gettext.html where you can skip -U user:password if your proxy does not require authentication.
Query HTTP Headers
HTTP headers allow the remote web server to send additional information about itself along with the actual request. In addition to, this provides the client with details on how the request is being handled.
To query the HTTP headers from a website, do:
curl -I https://www.gnu.orgHTTP/1.1 200 OK
Date: Mon, 13 Jul 2020 21:22:32 GMT
Server: Apache/2.4.7
Content-Location: home.html
Vary: negotiate,accept-language,Accept-Encoding
TCN: choice
Strict-Transport-Security: max-age=63072000
Access-Control-Allow-Origin: (null)
Accept-Ranges: bytes
Cache-Control: max-age=0
Expires: Mon, 13 Jul 2020 21:22:32 GMT
Content-Type: text/html
Content-Language: enUpload Files to FTP Server
cURL can also be used to upload files to the FTP server with -T option.
curl -u ftpuser:ftppass -T myfile.txt ftp://ftp.server.comAs a result, the above command will upload the file named myfile.txt to the FTP server. You can also upload multiple files at a same time using the range operations.
curl -u ftpuser:ftppass -T "{file1,file2}" ftp://ftp.server.comOptionally we can use . to get the input from STDIN and transfer to the remote.
curl -u ftpuser:ftppass -T - ftp://ftp.server.com/mynewfile.txtThe above command will get the input from the user from Standard Input and save the contents in the ftp server under the name mynewfile.txt.
You can provide one -T for each URL and the pair specifies what to upload where.
Download Files from FTP Server
cURL can also be used to download files from FTP servers. If the given FTP path is a directory, by default it will list the files under the specific directory.
curl -u ftpuser:ftppass -O ftp://ftp_server/public_html/mysql.phpThe above command will download the mysql.php file from the ftp server and save it in the local directory.
curl -u ftpuser:ftppass -O ftp://ftp_server/public_html/Here, the given URL refers to a directory. So cURL will list all the files and directories under the given URL.
List/Download Using Ranges
cURL supports ranges to be given in the URL. When a range is given, files matching within the range will be downloaded. It will be helpful to download packages from the FTP mirror sites.
curl http://ftp.us.debian.org/debian/pool/main/[a-z]/The above command will list out all the packages from a-z ranges in the terminal.
More Information Using Verbose and Trace Option
You can get to know what is happening using the -v option. This option enable the verbose mode and it will print the details.
curl -v https://www.gnu.orgThe about command will output the following:
* Trying 209.51.188.148:443...
* Connected to www.gnu.org (209.51.188.148) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
* CAfile: /etc/ssl/certs/ca-certificates.crt
CApath: none
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES128-GCM-SHA256
* ALPN, server did not agree to a protocol
* Server certificate:
* subject: CN=emacs.org
* start date: Jun 17 09:07:40 2020 GMT
* expire date: Sep 15 09:07:40 2020 GMT
* subjectAltName: host "www.gnu.org" matched cert's "www.gnu.org"
* issuer: C=US; O=Let's Encrypt; CN=Let's Encrypt Authority X3
* SSL certificate verify ok.
> GET / HTTP/1.1
> Host: www.gnu.org
> User-Agent: curl/7.71.1
> Accept: */*
>
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< Date: Mon, 13 Jul 2020 21:56:04 GMT
< Server: Apache/2.4.7
< Content-Location: home.html
< Vary: negotiate,accept-language,Accept-Encoding
< TCN: choice
< Strict-Transport-Security: max-age=63072000
< Access-Control-Allow-Origin: (null)
< Accept-Ranges: bytes
< Cache-Control: max-age=0
< Expires: Mon, 13 Jul 2020 21:56:04 GMT
< Transfer-Encoding: chunked
< Content-Type: text/html
< Content-Language: en
<
...Send Mail Using SMTP Protocol
cURL can also be used to send mail using the SMTP protocol. You should specify the from-address, to-address, and the mailserver ip-address as shown below.
curl --mail-from [email protected] --mail-rcpt [email protected] smtp://mailserver.com Once the above command is entered, it will wait for the user to provide the data to mail. Once you’ve composed your message, type . (period) as the last line, which will send the email immediately.
HTTP/2 Support Check
If you have the latest cURL release, you can use the --http2 option to check if a particular URL supports the new HTTP/2 protocol. Therefore, if the site does support HTTP/2, you will see HTTP/2.0 200 in the header instead of HTTP/1.1 200.
curl -I --http2 https://www.opensource.comSimulate HTTP Methods
The GET method is used to retrieve resources from a particular URL. The simple curl https://www.gnu.org/ command will use GET as the default HTTP method, however it can also be specified using --request GET or -X GET.
curl --request GET https://www.gnu.orgThe POST method is used to post information to a web server (e.g. a comment on a forum). This can be specified using --request POST or -X POST.
curl --request POST https://yourwebsite.comThe DELETE method deletes the resource from the web server associated with a specific URL. This can be specified using --request DELETE or -X DELETE.
curl --request DELETE https://yourwebsite.comThe PUT method creates or replaces a resource based on the data the client submits to the web server. (e.g creating a new web page or updating an existing one). This can be specified using --request PUT or -X PUT.
curl --request PUT https://yourwebsite.comMake a POST Request with Parameters
The following command will send the animal1 and animal2 parameters, along with their corresponding values, to https://yourdomain.com/animals.php
curl --request POST --data "animal1=cat&animal2=dog" https://yourdomain.com/animals.phpYou can use this tip to simulate the behavior of a regular HTML form.
Conclusion
In this post, we explained what cURL is and what you can do with some basic commands. To learn more about cURL, you can visit the project’s website.
