FreeBSD Curl Command | How to Use Curls

As a web developer or systems administrator, understanding HTTP requests and how to interact with APIs is critical, especially when it comes to automating requests and debugging. In such a scenario, you might want to execute a quick HTML request from the terminal and this is where CURL comes in to meet your needs.

Curl is free and open source software that can be used to exchange data with servers using one of the many supported protocols, such as HTTP, HTTPS, FTP, FTP, SFTP. Because Curl is a command line tool, it can be used as a command line script to automate repetitive tasks such as downloading or exchanging data and information between electronic devices.

Client CURL allows us to exchange data between the client and server via a command line interface (CLI). By simply specifying the server URL address and the data to be sent and requested will be immediately transferred. Not only that, curl can also be used in cars, television sets, routers, printers, audio equipment, cell phones, tablets, settop boxes, media players and is an Internet transfer engine for thousands of software applications in more than ten billion installations.

In this article, we will introduce the CURL command, explore how and why it is used, and show some common examples of the CURL command and their use cases.

List of contents:
1. Installing CURL on FreeBSD
2. Using Basic CURL Commands
3. Download Files With CURL
4. Perform a POST Request
5. Send JSON or XML Data
6. Using Multipart/Form-data Content Type
7. Set Request Headers



1. Installing CURL on FreeBSD

If you use the FreeBSD server, CURL can be installed via the FreeBSD ports and also via the pkg package. In this article we will explain how to install CURL with the FreeBSD ports system.

root@ns1:~ # cd /usr/ports/ftp/curl
root@ns1:/usr/ports/ftp/curl #
make config
root@ns1:/usr/ports/ftp/curl #
make install clean

After the installation is complete, check the version of CURL that has been installed.

root@ns1:~ # curl --version
curl 8.1.2 (amd64-portbld-freebsd13.1) libcurl/8.1.2 OpenSSL/1.1.1t zlib/1.2.13 libpsl/0.21.2 (+libidn2/2.3.4) libssh2/1.11.0 nghttp2/1.53.0
Release-Date: 2023-05-30
Protocols: dict file ftp ftps gopher gophers http https imap imaps pop3 pop3s rtsp scp sftp smtp smtps telnet tftp
Features: alt-svc AsynchDNS GSS-API HSTS HTTP2 HTTPS-proxy IPv6 Kerberos Largefile libz NTLM NTLM_WB PSL SPNEGO SSL threadsafe TLS-SRP UnixSockets


From the script display above, we can see that the version of CURL used is curl 8.1.2


2. Using Basic CURL Commands

Similar to other command line applications such as git, wget, perl and fetch, CURL also has a basic syntax that can be used as a reference for running CURL. The basic syntax of CURL is very simple and relatively easy to remember. Below is an example of basic CURL syntax.

curl [OPTIONS] [URL]


After knowing the basic CURL commands, now we practice with the google.com website. The CURL script below is used to display page content and to render pages from the google.com web site.

root@ns1:~ # curl http://www.google.com
root@ns1:~ #
curl https://www.google.com
root@ns1:~ #
curl -I https://www.google.com

After you try the scripts above, you will see the output of each script. Pay attention to any differences in the output of the script above.

When called with the above script, curl prints the body of the response sent by the server to standard output. In this case, the content that will be displayed is the HTML code of the page. Another example, suppose we want to count the number of times the word "google" appears in the source code of the page returned by the request from the example script above.

root@ns1:~ # curl -s https://www.google.com | grep -i -o google | wc -w
115


3. Download Files With CURL

Curl command file utility support for downloading and uploading files. Curl is useful for a lot of work with system administration, web development for calling web services, and so on. Suppose we want to download the latest version of Firefox from the official Mozilla website. To save the tarball locally as firefox.tar.bz2, run the following script.

root@ns1:~ # curl --output firefox.tar.gz https://download-installer.cdn.mozilla.net/pub/firefox/releases/106.0.1/linux-x86_64/en-US/firefox-106.0.1.tar.bz2
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
64 75.1M 64 48.1M 0 0 1794k 0 0:00:42 0:00:27 0:00:15 1599k


As an alternative to options --output, it can be replaced with --remote-name or -O. This will cause the file to be saved with its remote name, firefox-106.0.1.tar.bz2, run the script below.

root@ns1:~ # curl --remote-name 'https://download-installer.cdn.mozilla.net/pub/firefox/releases/106.0.1/linux-x86_64/en-US/firefox-106.0.1.tar.bz2'
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
12 75.1M 12 9727k 0 0 2594k 0 0:00:29 0:00:03 0:00:26 2594k


Basic CURL syntax for downloading files
To download files using curl use the below syntax. Options -O is used to save files on the local system with the same name on the remote system.

root@ns1:~ # curl -O https://www.monsta.com.br/admin/uploader/uploads/Configurando_o_Servidor_DNS_Unbound.pdf
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 198k 100 198k 0 0 52918 0 0:00:03 0:00:03 --:--:-- 52918


Download File and Save With Different Name
With CURL you can also save the downloaded file with a different name on the local machine. Use -o followed by a new file name to download and save the file with a different name. In this example, we will download the file unbound.pdf and rename it to serverdnsunbound.pdf.

root@ns1:~ # curl -o serverdnsunbound.pdf https://www.bortzmeyer.org/unbound.pdf
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 75422 100 75422 0 0 34565 0 0:00:02 0:00:02 --:--:-- 34581


Downloading Multiple Files at Once
One of the advantages of CURL is that it can download many files at once. The example below will download the files unbound.pdf, unbound-windows-manual-02.pdf and opengate1.5.39.tar.gz.

root@ns1:~ # curl -O https://www.bortzmeyer.org/unbound.pdf -O https://punoqun.net/documentation/unbound/unbound-windows-manual-02.pdf -O https://opengate.osdn.jp/download/opengate1.5.39.tar.gz
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 75422 100 75422 0 0 36376 0 0:00:02 0:00:02 --:--:-- 36383
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 865k 100 865k 0 0 328k 0 0:00:02 0:00:02 --:--:-- 328k
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 106k 100 106k 0 0 40534 0 0:00:02 0:00:02 --:--:-- 40542


After the download process is complete, check the download results in the /root folder with the ls command, because when downloading we are active in the root@ns1 folder:~ #

root@ns1:~ # ls
.bash_history .k5login .profile .wget-hsts unbound.pdf
.cshrc .local .shrc
opengate1.5.39.tar.gz
.history .login .ssh
unbound-windows-manual-02.pdf


Limit Download Speed
Almost the same as the wget command, CURL also has the ability to limit file download speed. The following is an example of a script to limit file download speed with CURL.

root@ns1:~ # curl --limit-rate 100k -O https://opengate.osdn.jp/download/opengate1.5.39.tar.gz
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 106k 100 106k 0 0 42754 0 0:00:02 0:00:02 --:--:-- 42768



4. Perform a POST Request

The POST http method is used to create new resources on the server. POST requests usually include data, which can be sent in various ways. The easiest way to include data in a POST request with CURL is to use the --data (-d) or --data-urlencode option. The data sent must be urlen encoded (percent encoded). If we use the former, we need to encode it first; if we use the latter (available in the latest versions of CURL), instead, the data will be encoded automatically.

root@ns1:~ # curl --data-urlencode "name=John Smith" --data-urlencode "age=30" https://httpbin.org/post
{
"args": {},
"data": "",
"files": {},
"form": {
"age": "30",
"name": "John Smith"
},
"headers": {
"Accept": "*/*",
"Content-Length": "22",
"Content-Type": "application/x-www-form-urlencoded",
"Host": "httpbin.org",
"User-Agent": "curl/8.1.2",
"X-Amzn-Trace-Id": "Root=1-64c5b2a5-0c4b5fc569f1fb7c3408e18d"
},
"json": null,
"origin": "36.90.8.254",
"url": "https://httpbin.org/post"
}


In the example above, we sent a request to the site https://httpbin.org, which can be used to test the request. The server returns a response that reflects the data sent in the request.


5. Send JSON or XML Data

In some cases we may need to send a JSON string as data. In such cases, we need to do two things:
a. Set the Content-Type header parameter to application/json explicitly
b. Pass a JSON string as an argument to the --data option.

Here is an example of such a request:

root@ns1:~ # curl --header "Content-Type: application/json" --data '{"name": "John Smith", "age": 30 }' https://httpbin.org/post
{
"args": {},
"data": "{\"name\": \"John Smith\", \"age\": 30 }",
"files": {},
"form": {},
"headers": {
"Accept": "*/*",
"Content-Length": "34",
"Content-Type": "application/json",
"Host": "httpbin.org",
"User-Agent": "curl/8.1.2",
"X-Amzn-Trace-Id": "Root=1-64c5b4a4-1b6c2b9e56e669d92761a69b"
},
"json": {
"age": 30,
"name": "John Smith"
},
"origin": "36.90.8.254",
"url": "https://httpbin.org/post"
}


The same principle must be applied if we want to send XML data, for example.

root@ns1:~ # curl --header "Content-Type: text/xml" --data '<person><name>John Smith</name><age>30</age></person>' https://httpbin.org/post
{
"args": {},
"data": "<person><name>John Smith</name><age>30</age></person>",
"files": {},
"form": {},
"headers": {
"Accept": "*/*",
"Content-Length": "53",
"Content-Type": "text/xml",
"Host": "httpbin.org",
"User-Agent": "curl/8.1.2",
"X-Amzn-Trace-Id": "Root=1-64c5b4f6-3297a7c61dca2a4423d8ab7f"
},
"json": null,
"origin": "36.90.8.254",
"url": "https://httpbin.org/post"
}




6. Using Multipart/Form-data Content Type

To make a POST request and include data using the multipart/form-data content type, we can run Curl with the --form (-F) option. This is what we want to use, for example when we need to upload files. This is the same request we made in the previous example, but done with the following content types.

root@ns1:~ # curl --form "name=John Smith" --form "age=30" https://httpbin.org/post
{
"args": {},
"data": "",
"files": {},
"form": {
"age": "30",
"name": "John Smith"
},
"headers": {
"Accept": "*/*",
"Content-Length": "243",
"Content-Type": "multipart/form-data; boundary=------------------------be26f7f1fcf94068",
"Host": "httpbin.org",
"User-Agent": "curl/8.1.2",
"X-Amzn-Trace-Id": "Root=1-64c5b5dc-22b1c1d749f6e0ed4b2c6b11"
},
"json": null,
"origin": "36.90.8.254",
"url": "https://httpbin.org/post"
}




7. Set Request Headers

Header parameters are used to provide additional information into the request, some examples are Keep-Alive which is used to control how many connections should be kept open or User-Agent, which is used to identify the application type and operating system, or Accept , used to determine the MIME type received by the client. To set header parameters with Curl we can use the -H option (short for --header).

root@ns1:~ # curl --header "Accept: text/xml" --header "User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:105.0) Gecko/20100101 Firefox/105.0" https://httpbin.org/get
{
"args": {},
"headers": {
"Accept": "text/xml",
"Host": "httpbin.org",
"User-Agent": "Mozilla/5.0 (X11; Linux x86_64; rv:105.0) Gecko/20100101 Firefox/105.0",
"X-Amzn-Trace-Id": "Root=1-64c5b772-5d6573380a68c65042b1d669"
},
"origin": "36.90.8.254",
"url": "https://httpbin.org/get"
}


To set the user agent that performs the request, Curl provides a shortcut via the --user-agent (-A) option.

root@ns1:~ # curl --user-agent "User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:105.0) Gecko/20100101 Firefox/105.0" https://httpbin.org/get
{
"args": {},
"headers": {
"Accept": "*/*",
"Host": "httpbin.org",
"User-Agent": "User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:105.0) Gecko/20100101 Firefox/105.0",
"X-Amzn-Trace-Id": "Root=1-64c5b7e3-74c46169149208f9465e53f1"
},
"origin": "36.90.8.254",
"url": "https://httpbin.org/get"
}



Curl is a useful utility for making GET, POST, HEADER, and many other requests to remote servers. It's even helpful for quickly downloading remote files like wget and git. This tutorial is only a few examples of using CURL, you can read other articles which discuss CURL in more depth.
Iwan Setiawan

I Like Adventure: Mahameru Mount, Rinjani Mount I Like Writer FreeBSD

Post a Comment

Previous Post Next Post