cURL, short for “Client for URLs”, is a command line tool for transferring data using various protocols. This article aims at providing the commands that are widely used and as a handy reference.
You can invoke curl
command from your terminal without thinking about ways to install it, as it comes pre-installed on most Linux based Operating Systems.
There are a vast amount of use-cases for curl, such as:
- FTP upload
- Proxy support
- SSL connections
- HTTP post
cURL also supports the use of all the following protocols: DICT, FILE, FTP, FTPS, GOPHER, HTTP, HTTPS, IMAP, IMAPS, LDAP, LDAPS, POP3, POP3S, RTMP, RTSP, SCP, SFTP, SMB, SMBS, SMTP, SMTPS, TELNET, and TFTP.
Download a Single File
The following command will get the content of the URL and display it in the STDOUT (i.e on your terminal).
curl https://www.gnu.org
To store the output in a file, you an redirect it as shown below. This will also display some additional download statistics.
curl https://www.gnu.org > gnu-org.html
Save the cURL Output to a file
We can save the result of the cURL command to a file by using -o
/-O
options.
- -o (lowercase o) the result will be saved in the filename provided in the command line
- -O (uppercase O) the filename in the URL will be taken and it will be used as the filename to store the result
curl -o my-gettext.html https://www.gnu.org/software/gettext/manual/gettext.html
As a result, now the page gettext.html
will be saved in the file named my-gettext.html
. Also you can note that when running cURL with -o
option, it displays the progress meter for the download as follows.
When you use cURL -O
, it will save the content in the file named ‘gettext.html’ itself in the local machine.
curl -O http://www.gnu.org/software/gettext/manual/gettext.html
Note: When curl has to write the data to the terminal, it disables the Progress Meter, to avoid confusion in printing. We can use >
|-o
|-O
options to move the result to a file.
Fetch Multiple Files at a Time
Of course, we can download multiple files in a single shot by specifying the URLs on the command line.
curl -O https://www.gnu.org/software/gettext/manual/html_node/index.html -O https://www.gnu.org/software/gettext/manual/gettext.html
Follow HTTP Location Headers with -L option
However by default cURL doesn’t follow the HTTP Location headers. It is also termed as Redirects. When a requested web page is moved to another place, then an HTTP Location header will be sent as a Response and it will have where the actual web page is located.
We can insists curl to follow the redirection using -L option, as shown below.
curl -L https://www.google.com
Continue/Resume a Previous Download
Using curl -C
option, you can continue a download which was stopped already for some reason. This will be helpful when you download large files, and the download got interrupted.
If we say -C -
, then curl will find from where to start resuming the download. We can also give an offset -C <offset>
. The given offset bytes will be skipped from the beginning for the source file.
Start a big download using curl, and press Ctrl-C to stop it in between the download.
curl -O https://www.gnu.org/software/gettext/manual/gettext.html
Using curl -C -
, we can continue the download from where it left off earlier.
curl -C - -O https://www.gnu.org/software/gettext/manual/gettext.html
Use a Proxy with or without Authentication
If you are behind a proxy server listening on port 8080 at proxy.yourdomain.com, do:
curl -x proxy.yourdomain.com:8080 -U user:password -O https://www.gnu.org/software/gettext/manual/gettext.html
where you can skip -U user:password
if your proxy does not require authentication.
Query HTTP Headers
HTTP headers allow the remote web server to send additional information about itself along with the actual request. In addition to, this provides the client with details on how the request is being handled.
To query the HTTP headers from a website, do:
curl -I https://www.gnu.org
HTTP/1.1 200 OK
Date: Mon, 13 Jul 2020 21:22:32 GMT
Server: Apache/2.4.7
Content-Location: home.html
Vary: negotiate,accept-language,Accept-Encoding
TCN: choice
Strict-Transport-Security: max-age=63072000
Access-Control-Allow-Origin: (null)
Accept-Ranges: bytes
Cache-Control: max-age=0
Expires: Mon, 13 Jul 2020 21:22:32 GMT
Content-Type: text/html
Content-Language: en
Upload Files to FTP Server
cURL can also be used to upload files to the FTP server with -T option.
curl -u ftpuser:ftppass -T myfile.txt ftp://ftp.server.com
As a result, the above command will upload the file named myfile.txt
to the FTP server. You can also upload multiple files at a same time using the range operations.
curl -u ftpuser:ftppass -T "{file1,file2}" ftp://ftp.server.com
Optionally we can use .
to get the input from STDIN and transfer to the remote.
curl -u ftpuser:ftppass -T - ftp://ftp.server.com/mynewfile.txt
The above command will get the input from the user from Standard Input and save the contents in the ftp server under the name mynewfile.txt
.
You can provide one -T
for each URL and the pair specifies what to upload where.
Download Files from FTP server
cURL can also be used to download files from FTP servers. If the given FTP path is a directory, by default it will list the files under the specific directory.
curl -u ftpuser:ftppass -O ftp://ftp_server/public_html/mysql.php
The above command will download the mysql.php
file from the ftp server and save it in the local directory.
curl -u ftpuser:ftppass -O ftp://ftp_server/public_html/
Here, the given URL refers to a directory. So cURL will list all the files and directories under the given URL.
List/Download using Ranges
cURL supports ranges to be given in the URL. When a range is given, files matching within the range will be downloaded. It will be helpful to download packages from the FTP mirror sites.
curl http://ftp.us.debian.org/debian/pool/main/[a-z]/
The above command will list out all the packages from a-z
ranges in the terminal.
More Information using Verbose and Trace Option
You can get to know what is happening using the -v option. -v option enable the verbose mode and it will print the details.
curl -v https://www.gnu.org
The about command will output the following:
* Trying 209.51.188.148:443...
* Connected to www.gnu.org (209.51.188.148) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
* CAfile: /etc/ssl/certs/ca-certificates.crt
CApath: none
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES128-GCM-SHA256
* ALPN, server did not agree to a protocol
* Server certificate:
* subject: CN=emacs.org
* start date: Jun 17 09:07:40 2020 GMT
* expire date: Sep 15 09:07:40 2020 GMT
* subjectAltName: host "www.gnu.org" matched cert's "www.gnu.org"
* issuer: C=US; O=Let's Encrypt; CN=Let's Encrypt Authority X3
* SSL certificate verify ok.
> GET / HTTP/1.1
> Host: www.gnu.org
> User-Agent: curl/7.71.1
> Accept: */*
>
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< Date: Mon, 13 Jul 2020 21:56:04 GMT
< Server: Apache/2.4.7
< Content-Location: home.html
< Vary: negotiate,accept-language,Accept-Encoding
< TCN: choice
< Strict-Transport-Security: max-age=63072000
< Access-Control-Allow-Origin: (null)
< Accept-Ranges: bytes
< Cache-Control: max-age=0
< Expires: Mon, 13 Jul 2020 21:56:04 GMT
< Transfer-Encoding: chunked
< Content-Type: text/html
< Content-Language: en
<
...
Send Mail using SMTP Protocol
cURL can also be used to send mail using the SMTP protocol. You should specify the from-address, to-address, and the mailserver ip-address as shown below.
curl --mail-from blah@test.com --mail-rcpt foo@test.com smtp://mailserver.com
Once the above command is entered, it will wait for the user to provide the data to mail. Once you’ve composed your message, type .
(period) as the last line, which will send the email immediately.
HTTP/2 support check
If you have the latest curl release, you can use the --http2
option to check if a particular URL supports the new HTTP/2 protocol. Therefore, if the site does support HTTP/2, you will see HTTP/2.0 200
in the header instead of HTTP/1.1 200
.
curl -I --http2 https://www.opensource.com
Simulate HTTP methods
The GET
method is used to retrieve resources from a particular URL. The simple curl https://www.gnu.org/
command will use GET
as the default HTTP method, however it can also be specified using --request GET
or -X GET
.
curl --request GET https://www.gnu.org
The POST
method is used to post information to a web server (e.g. a comment on a forum). This can be specified using --request POST
or -X POST
.
curl --request POST https://yourwebsite.com
The DELETE
method deletes the resource from the web server associated with a specific URL. This can be specified using --request DELETE
or -X DELETE
.
curl --request DELETE https://yourwebsite.com
The PUT
method creates or replaces a resource based on the data the client submits to the web server. (e.g creating a new web page or updating an existing one). This can be specified using --request PUT
or -X PUT
.
curl --request PUT https://yourwebsite.com
Make a POST request with Parameters
The following command will send the animal1
and animal2
parameters, along with their corresponding values, to https://yourdomain.com/animals.php
curl --request POST --data "animal1=cat&animal2=dog" https://yourdomain.com/animals.php
You can use this tip to simulate the behavior of a regular HTML form.