This article explains the curl command in Linux and how to use it with examples based on best practices.
Table of Contents
- What is cURL?
- Download a Single File
- Save the cURL Output to a File
- Hide Progress Bar
- Fetch Multiple Files at a Time
- Follow HTTP Location Headers with -L Option
- Continue/Resume a Previous Download
- Use a Proxy with or Without Authentication
- Query HTTP Headers
- Upload Files to FTP Server
- Download Files from FTP Server
- List/Download Using Ranges
- More Information Using Verbose and Trace Option
- Send Mail Using SMTP Protocol
- HTTP/2 Support Check
- Simulate HTTP Methods
- Make a POST Request with Parameters
- Conclusion
What is cURL?
cURL, short for “Client URL,” is a command-line tool for transferring data using various protocols. It is an important Linux tool often used for connection troubleshooting.
At its most basic, cURL allows you to communicate with a server by defining the location in the form of a URL and the data you want to transmit. You can invoke the curl command from your terminal without thinking about ways to install it, as it comes pre-installed on most Linux-based operating systems.
There are a vast amount of use-cases for cURL, such as:
- FTP upload
- Proxy support
- SSL connections
- HTTP post
cURL also supports the use of all the following protocols: DICT, FILE, FTP, FTPS, GOPHER, HTTP, HTTPS, IMAP, IMAPS, LDAP, LDAPS, POP3, POP3S, RTMP, RTSP, SCP, SFTP, SMB, SMBS, SMTP, SMTPS, TELNET, and TFTP.
Download a Single File
The following command will get the URL’s content and display it in the STDOUT (i.e., on your terminal).
curl https://www.gnu.orgTo store the output in a file, you can redirect it, as shown below. This will also display some additional download statistics.
curl https://www.gnu.org > gnu-org.htmlSave the cURL Output to a File
We can save the result of the cURL command to a file using -o/-O options.
-
-o(lowercase o): the result will be saved in the filename provided in the command line -
-O(uppercase O): the filename in the URL will be taken, and it will be used as the filename to store the result
curl -o my-gettext.html https://www.gnu.org/software/gettext/manual/gettext.html As a result, the page gettext.html will be saved in the file named my-gettext.html. Also, you can note that running curl with the -o option displays the progress meter for the download as follows.
When you use cURL -O, it will save the content in the file named ‘gettext.html’ itself in the local machine.
curl -O http://www.gnu.org/software/gettext/manual/gettext.htmlNote: When curl has to write the data to the terminal, it disables the progress meter to avoid confusion in printing. We can use >, -o, -O options to move the result to a file.
Hide Progress Bar
cURL, by default, shows a progress bar. To hide it -s (--silent) option can be used.
curl -s -O http://www.gnu.org/software/gettext/manual/gettext.htmlIf for some reason, that does not work on your platform, you could always redirect stderr to /dev/null:
curl -O http://www.gnu.org/software/gettext/manual/gettext.html 2>/dev/nullFetch Multiple Files at a Time
Of course, we can download multiple files in a single shot by specifying the URLs on the command line.
curl -O https://www.gnu.org/software/gettext/manual/html_node/index.html -O https://www.gnu.org/software/gettext/manual/gettext.html Follow HTTP Location Headers with -L Option
However, by default, cURL doesn’t follow the HTTP location headers, also termed redirects. When a requested web page is moved to another place, an HTTP location header will be sent as a response, and it will have where the actual web page is located.
We can insist cURL follow the redirection using the -L option, as shown below.
curl -L https://www.google.comContinue/Resume a Previous Download
Using the cURL -C option, you can continue a download that was stopped already for some reason. This will be helpful when you download large files, and the download gets interrupted.
If we say -C -, then cURL will find from where to start resuming the download. We can also give an offset -C <offset>. The given offset bytes will be skipped from the beginning for the source file.
Start a big download using curl, and press Ctrl-C to stop it in between the download.
curl -O https://www.gnu.org/software/gettext/manual/gettext.htmlUsing curl -C -, we can continue the download from where it left earlier.
curl -C - -O https://www.gnu.org/software/gettext/manual/gettext.htmlUse a Proxy with or Without Authentication
If you are behind a proxy server listening on port 8080 at proxy.yourdomain.com, do:
curl -x proxy.yourdomain.com:8080 -U user:password -O https://www.gnu.org/software/gettext/manual/gettext.html where you can skip -U user:password if your proxy does not require authentication.
Query HTTP Headers
HTTP headers allow the remote web server to send additional information about itself and the actual request. In addition, this provides the client with details on how the request is being handled.
To query the HTTP headers from a website, do:
curl -I https://www.gnu.orgHTTP/1.1 200 OK
Date: Mon, 13 Jul 2020 21:22:32 GMT
Server: Apache/2.4.7
Content-Location: home.html
Vary: negotiate,accept-language,Accept-Encoding
TCN: choice
Strict-Transport-Security: max-age=63072000
Access-Control-Allow-Origin: (null)
Accept-Ranges: bytes
Cache-Control: max-age=0
Expires: Mon, 13 Jul 2020 21:22:32 GMT
Content-Type: text/html
Content-Language: enUpload Files to FTP Server
cURL can also upload files to the FTP server with the -T option.
curl -u ftpuser:ftppass -T myfile.txt ftp://ftp.server.comAs a result, the above command will upload the file named myfile.txt to the FTP server. You can also upload multiple files simultaneously using the range operations.
curl -u ftpuser:ftppass -T "{file1,file2}" ftp://ftp.server.comOptionally we can use . to get the input from STDIN and transfer it to the remote.
curl -u ftpuser:ftppass -T - ftp://ftp.server.com/mynewfile.txtThe above command will get the input from the user from Standard Input and save the contents in the ftp server under the name mynewfile.txt.
You can provide one -T for each URL, and the pair specifies where to upload.
Download Files from FTP Server
cURL can also be used to download files from FTP servers. However, if the given FTP path is a directory, cURL will list the files under the specified directory.
curl -u ftpuser:ftppass -O ftp://ftp_server/public_html/mysql.phpThe above command will download the mysql.php file from the ftp server and save it in the local directory.
curl -u ftpuser:ftppass -O ftp://ftp_server/public_html/Here, the given URL refers to a directory. So cURL will list all the files and directories under the given URL.
List/Download Using Ranges
cURL supports ranges to be given in the URL. When a range is given, files matching within the range will be downloaded. It will be helpful to download packages from the FTP mirror sites.
curl http://ftp.us.debian.org/debian/pool/main/[a-z]/The above command will list all the packages from a-z ranges in the terminal.
More Information Using Verbose and Trace Option
You can get to know what is happening using the -v option. This option enables the verbose mode, and it will print the details.
curl -v https://www.gnu.orgThe about command will output the following:
* Trying 209.51.188.148:443...
* Connected to www.gnu.org (209.51.188.148) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
* CAfile: /etc/ssl/certs/ca-certificates.crt
CApath: none
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES128-GCM-SHA256
* ALPN, server did not agree to a protocol
* Server certificate:
* subject: CN=emacs.org
* start date: Jun 17 09:07:40 2020 GMT
* expire date: Sep 15 09:07:40 2020 GMT
* subjectAltName: host "www.gnu.org" matched cert's "www.gnu.org"
* issuer: C=US; O=Let's Encrypt; CN=Let's Encrypt Authority X3
* SSL certificate verify ok.
> GET / HTTP/1.1
> Host: www.gnu.org
> User-Agent: curl/7.71.1
> Accept: */*
>
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< Date: Mon, 13 Jul 2020 21:56:04 GMT
< Server: Apache/2.4.7
< Content-Location: home.html
< Vary: negotiate,accept-language,Accept-Encoding
< TCN: choice
< Strict-Transport-Security: max-age=63072000
< Access-Control-Allow-Origin: (null)
< Accept-Ranges: bytes
< Cache-Control: max-age=0
< Expires: Mon, 13 Jul 2020 21:56:04 GMT
< Transfer-Encoding: chunked
< Content-Type: text/html
< Content-Language: en
<
...Send Mail Using SMTP Protocol
cURL can also be used to send mail using the SMTP protocol. As shown below, you should specify the from-address, to-address, and the mail server IP address.
curl --mail-from [email protected] --mail-rcpt [email protected] smtp://mailserver.com Once the above command is entered, it will wait for the user to provide the data to mail. So when you’ve composed your message, type “.” (period) as the last line, which will send the email immediately.
HTTP/2 Support Check
If you have the latest cURL release, you can use the --http2 option to check if a particular URL supports the new HTTP/2 protocol. Therefore, if the site does support HTTP/2, you will see HTTP/2.0 200 in the header instead of HTTP/1.1 200.
curl -I --http2 https://www.opensource.comSimulate HTTP Methods
The GET method is used to retrieve resources from a particular URL. For example, the simple curl https://www.gnu.org/ command will use GET as the default HTTP method. However, it can also be specified using --request GET or -X GET.
curl --request GET https://www.gnu.orgThe POST method posts information to a web server (e.g., a comment on a forum). This can be specified using --request POST or -X POST.
curl --request POST https://yourwebsite.comThe DELETE method deletes the resource associated with a specific URL from the web server. This can be specified using --request DELETE or -X DELETE.
curl --request DELETE https://yourwebsite.comThe PUT method creates or replaces a resource based on the data the client submits to the web server. (e.g., creating a new web page or updating an existing one). This can be specified using --request PUT or -X PUT.
curl --request PUT https://yourwebsite.comMake a POST Request with Parameters
The following command will send the animal1 and animal2 parameters, along with their corresponding values, to https://yourdomain.com/animals.php
curl --request POST --data "animal1=cat&animal2=dog" https://yourdomain.com/animals.phpYou can use this tip to simulate the behavior of a regular HTML form.
Conclusion
We explained what the curl command is. The examples in this article are straightforward, but they showcase the most commonly used cURL use cases and are intended to help you understand how the curl command works on Linux.
To learn more about cURL, you can visit the project’s website.
