Most Widely Used And Popular cURL Commands In Practice

cURL, short for “Client for URLs”, is a command line tool for transferring data using various protocols. This article aims at providing the commands that are widely used and as a handy reference.

You can invoke curl command from your terminal without thinking about ways to install it, as it comes pre-installed on most Linux based Operating Systems.

There are a vast amount of use-cases for curl, such as:

• FTP upload
• Proxy support
• SSL connections
• HTTP post

cURL also supports the use of all the following protocols: DICT, FILE, FTP, FTPS, GOPHER, HTTP, HTTPS, IMAP, IMAPS, LDAP, LDAPS, POP3, POP3S, RTMP, RTSP, SCP, SFTP, SMB, SMBS, SMTP, SMTPS, TELNET, and TFTP.

Download a single file

The following command will get the content of the URL and display it in the STDOUT (i.e on your terminal).

curl https://www.gnu.org

To store the output in a file, you an redirect it as shown below. This will also display some additional download statistics.

curl https://www.gnu.org > gnu-org.html

Save the cURL output to a file

We can save the result of the cURL command to a file by using -o/-O options.

• -o (lowercase o) the result will be saved in the filename provided in the command line
• -O (uppercase O) the filename in the URL will be taken and it will be used as the filename to store the result

curl -o my-gettext.html https://www.gnu.org/software/gettext/manual/gettext.html 

As a result, now the page gettext.html will be saved in the file named my-gettext.html. Also you can note that when running cURL with -o option, it displays the progress meter for the download as follows.

When you use cURL -O, it will save the content in the file named ‘gettext.html’ itself in the local machine.

curl -O http://www.gnu.org/software/gettext/manual/gettext.html

Note: When curl has to write the data to the terminal, it disables the Progress Meter, to avoid confusion in printing. We can use >, -o, -O options to move the result to a file.

Fetch multiple files at a time

Of course, we can download multiple files in a single shot by specifying the URLs on the command line.

curl -O https://www.gnu.org/software/gettext/manual/html_node/index.html -O https://www.gnu.org/software/gettext/manual/gettext.html 

Follow HTTP location headers with -L option

However by default cURL doesn’t follow the HTTP Location headers. It is also termed as Redirects. When a requested web page is moved to another place, then an HTTP Location header will be sent as a Response and it will have where the actual web page is located.

We can insists curl to follow the redirection using -L option, as shown below.

curl -L https://www.google.com

Continue/Resume a previous download

Using curl -C option, you can continue a download which was stopped already for some reason. This will be helpful when you download large files, and the download got interrupted.

If we say -C -, then curl will find from where to start resuming the download. We can also give an offset -C <offset>. The given offset bytes will be skipped from the beginning for the source file.

Start a big download using curl, and press Ctrl-C to stop it in between the download.

curl -O https://www.gnu.org/software/gettext/manual/gettext.html

Using curl -C -, we can continue the download from where it left off earlier.

curl -C - -O https://www.gnu.org/software/gettext/manual/gettext.html

Use a proxy with or without authentication

If you are behind a proxy server listening on port 8080 at proxy.yourdomain.com, do:

curl -x proxy.yourdomain.com:8080 -U user:password -O https://www.gnu.org/software/gettext/manual/gettext.html 

where you can skip -U user:password if your proxy does not require authentication.

Query HTTP headers

HTTP headers allow the remote web server to send additional information about itself along with the actual request. In addition to, this provides the client with details on how the request is being handled.

To query the HTTP headers from a website, do:

curl -I https://www.gnu.org

HTTP/1.1 200 OK
Date: Mon, 13 Jul 2020 21:22:32 GMT
Server: Apache/2.4.7
Content-Location: home.html
Vary: negotiate,accept-language,Accept-Encoding
TCN: choice
Strict-Transport-Security: max-age=63072000
Access-Control-Allow-Origin: (null)
Accept-Ranges: bytes
Cache-Control: max-age=0
Expires: Mon, 13 Jul 2020 21:22:32 GMT
Content-Type: text/html
Content-Language: en

Upload files to FTP server

cURL can also be used to upload files to the FTP server with -T option.

curl -u ftpuser:ftppass -T myfile.txt ftp://ftp.server.com

As a result, the above command will upload the file named myfile.txt to the FTP server. You can also upload multiple files at a same time using the range operations.

curl -u ftpuser:ftppass -T "{file1,file2}" ftp://ftp.server.com

Optionally we can use . to get the input from STDIN and transfer to the remote.

curl -u ftpuser:ftppass -T - ftp://ftp.server.com/mynewfile.txt

The above command will get the input from the user from Standard Input and save the contents in the ftp server under the name mynewfile.txt.

You can provide one -T for each URL and the pair specifies what to upload where.

Download files from FTP server

cURL can also be used to download files from FTP servers. If the given FTP path is a directory, by default it will list the files under the specific directory.

curl -u ftpuser:ftppass -O ftp://ftp_server/public_html/mysql.php

The above command will download the mysql.php file from the ftp server and save it in the local directory.

curl -u ftpuser:ftppass -O ftp://ftp_server/public_html/

Here, the given URL refers to a directory. So cURL will list all the files and directories under the given URL.

List/Download using ranges

cURL supports ranges to be given in the URL. When a range is given, files matching within the range will be downloaded. It will be helpful to download packages from the FTP mirror sites.

curl http://ftp.us.debian.org/debian/pool/main/[a-z]/

The above command will list out all the packages from a-z ranges in the terminal.

More information using verbose and trace option

You can get to know what is happening using the -v option. -v option enable the verbose mode and it will print the details.

curl -v https://www.gnu.org

The about command will output the following:

*   Trying 209.51.188.148:443...
* Connected to www.gnu.org (209.51.188.148) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/certs/ca-certificates.crt
  CApath: none
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES128-GCM-SHA256
* ALPN, server did not agree to a protocol
* Server certificate:
*  subject: CN=emacs.org
*  start date: Jun 17 09:07:40 2020 GMT
*  expire date: Sep 15 09:07:40 2020 GMT
*  subjectAltName: host "www.gnu.org" matched cert's "www.gnu.org"
*  issuer: C=US; O=Let's Encrypt; CN=Let's Encrypt Authority X3
*  SSL certificate verify ok.
> GET / HTTP/1.1
> Host: www.gnu.org
> User-Agent: curl/7.71.1
> Accept: */*
> 
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< Date: Mon, 13 Jul 2020 21:56:04 GMT
< Server: Apache/2.4.7
< Content-Location: home.html
< Vary: negotiate,accept-language,Accept-Encoding
< TCN: choice
< Strict-Transport-Security: max-age=63072000
< Access-Control-Allow-Origin: (null)
< Accept-Ranges: bytes
< Cache-Control: max-age=0
< Expires: Mon, 13 Jul 2020 21:56:04 GMT
< Transfer-Encoding: chunked
< Content-Type: text/html
< Content-Language: en
< 
...

Send mail using SMTP protocol

cURL can also be used to send mail using the SMTP protocol. You should specify the from-address, to-address, and the mailserver ip-address as shown below.

curl --mail-from [email protected] --mail-rcpt [email protected] smtp://mailserver.com 

Once the above command is entered, it will wait for the user to provide the data to mail. Once you’ve composed your message, type . (period) as the last line, which will send the email immediately.

HTTP/2 support check

If you have the latest curl release, you can use the --http2 option to check if a particular URL supports the new HTTP/2 protocol. Therefore, if the site does support HTTP/2, you will see HTTP/2.0 200 in the header instead of HTTP/1.1 200.

curl -I --http2 https://www.opensource.com

Simulate HTTP methods

The GET method is used to retrieve resources from a particular URL. The simple curl https://www.gnu.org/ command will use GET as the default HTTP method, however it can also be specified using --request GET or -X GET.

curl --request GET https://www.gnu.org

The POST method is used to post information to a web server (e.g. a comment on a forum). This can be specified using --request POST or -X POST.

curl --request POST https://yourwebsite.com

The DELETE method deletes the resource from the web server associated with a specific URL. This can be specified using --request DELETE or -X DELETE.

curl --request DELETE https://yourwebsite.com

The PUT method creates or replaces a resource based on the data the client submits to the web server. (e.g creating a new web page or updating an existing one). This can be specified using --request PUT or -X PUT.

curl --request PUT https://yourwebsite.com

curl --request POST --data "animal1=cat&animal2=dog" https://yourdomain.com/animals.php