$ # You can use a config file:$ cat << EOF > /sandbox/.curlcoutput-dir /tmpshow-errorsilentEOF$ curl --config /sandbox/.curlrc https://httpbin/uuid$ # Multiple queries (useful if there're a lot of queries).$ curl https://httpbin.org/get --url-query "name=Alice" --url-query "age=25"$ # You can call multiple urls$ curl https://httpbin.org/uuid https://httpbin.org/uuid$ # Include response headers.$ curl -i https://httpbin.org/uuidHTTP/2 200date: Tue, 30 Jul 2024 07:36:21 GMTcontent-type: application/jsoncontent-length: 53server: gunicorn/19.9.0access-control-allow-origin: *access-control-allow-credentials: true{ "uuid": "d8bf60e9-5f69-46d6-aee3-5cc08af4d639"}$ # Returns only headers (-I / --head).$ curl --head https://httpbin.org/uuidHTTP/2 200date: Tue, 30 Jul 2024 07:36:21 GMTcontent-type: application/jsoncontent-length: 53server: gunicorn/19.9.0access-control-allow-origin: *access-control-allow-credentials: true$ # Extract specific response information (there are > 50 variables).$ # Extract response status and content type:$ curl -w "status: %{response_code}\ntype: %{content_type}" https://httpbin.org/status/429status: 429type: text/html; charset=utf-8$ # Extract response headers:$ curl -w "date: %header{date}\nlength: %header{content-length}" https://httpbin.org/status/429date: Tue, 30 Jul 2024 07:21:26 GMTlength: 0$ # Download files.$ # `--create-dirs` will create the directories if they do not exist.$ curl --output-dir /tmp/some/place --create-dirs -O https://httpbin.org/uuid$ # Upload files.$ curl --upload-file /tmp/some/place/uuid https://httpbin.org/put$ # Retry.$ curl --retry 3 https://httpbin.org/unstable --retry-max-time 2 --retry-delay 1$ # Limit number of concurrent requests.$ # Possible values: n/s, n/m, n/h, n/d (change n with a number).$ curl --rate 3/s 'https://httpbin.org/anything/[1-9].txt'$ # Limit bandwidth usage.$ curl --limit-rate 1k https://httpbin.org/get$ # Timeouts.$ curl --max-time 0.5 https://httpbin.org/delay/1curl: (28) Operation timed out after 501 milliseconds with 0 bytes received$ curl --connect-timeout 0.5 https://httpbin.org/delay/1$ # Pass credentials from a file.$ cat << EOF > /tmp/.netrcmachine httpbin.orglogin alicepassword cheeseEOF$ curl --netrc-file /tmp/.netrc https://httpbin.org/basic-auth/alice/cheese$ # Post data.$ curl -d "foo=bar" -d "age=25" https://httpbin.org/post$ # Post data from file.$ echo "Alice, age 25" > /tmp/data.txt$ curl -d '@/tmp/data.txt' -H "content-type: text/plain" https://httpbin.org/post$ # URL encode the request$ curl --data-urlencode "Name: Foo Bar" https://httpbin.org/post$ # Post form.$ curl -f "foo=bar" https://httpbin.org/post$ # Redirect with endless loop protection:$ curl -L --max-redirs 3 https://httpbin.org/redirect/10curl: (47) Maximum (3) redirects followed$ # Enable cookie with `--cookie` / `-b`.$ # Persist cookies between calls with `--cookie-jar` / `-c`.$ curl -b "" -c /tmp/cookies https://httpbin.org/cookies/set?sessionid=123456'$ # Enable HSTS.$ curl --hsts /tmp/hsts -o /dev/null https://httpbin.org/get
I recently watched the 3.5-hour workshop Mastering the curl command line by Daniel Stenberg, the author of curl. The video was awesome and I learned a ton of things, so I wanted a (shortened) text version for future reference. Here it is.
I’ve also included some interactive examples, so you can try out different curl commands as you read.
The curl project
Curl started in 1996 (back then it was called httpget). It was created by Rafael Sagula, who later transferred the project to Daniel. Daniel renamed the project to curl in 1998. In 2000, he extracted the networking functionality into the libcurl library, and curl itself became a command-line wrapper around it.
Curl stands for “client for URLs”. It’s a tool for client-side internet transfers with URLs. An internet transfer is an upload or download using a specific protocol (curl supports quite a few), where the endpoint is identified by a URL.
Curl is open sourced under the (slightly modified) MIT License. It accepts contributions from anyone, no paperwork required. There are about 15 people who can accept PRs. The lead (and only full-time) developer is Daniel.
Curl supports a crazy number of protocols, from HTTP, FTP and TELNET to IMAP, LDAP and GOPHER. It runs on 92 operating systems and has over 20 billion installations worldwide.
Curl + libcurl is about 160K lines of code. Curl is released every 8 weeks and has about 250 releases as of August 2023.
After the release, there is a 10-day cool-down, followed by a 21-day feature window, followed by a 25-day feature freeze.
Curl performs internet transfers, where it acts as a client, uploading or downloading data from a remote server. The data can be anything: text, images, audio, video, and so on.
Curl supports both unauthenticated protocols (such as HTTP or FTP) and their authenticated counterparts (such as HTTPS or FTPS). Data transferred over an unauthenticated protocol can be intercepted and tampered with, so it’s better to always use authenticated ones. Curl can disable server verification with an --insecure flag, but you better not do this in production.
Curl currently has about 250 options, and the number is growing at a rate of ≈10/year.
There are short command line options (single dash):
curl -V
And long options (double dash):
curl --version
All options are available in “long” format, but only some of them have “short” counterparts.
The number of options keeps growing.
Some options are of boolean type. You can turn such options on:
curl --silent http://httpbin/uuid
Or off:
curl --no-silent http://httpbin/uuid
Some options accept arguments:
curl --output /tmp/uuid.json http://httpbin/uuid
Arguments that contain spaces should be enclosed in quotes:
Curl accepts any number of URLs, each of which requires a destination — stdout or a file. For example, this command saves the first UUID to /tmp/uuid1.json and the second UUID to tmp/uuid2.json:
Curl automatically expands glob expressions in URLs into multiple specific URLs.
For example, this command requests three different paths (al, bt, gm), each with two different parameters (num=1 and num=2), for a total of six URLs:
curl --output-dir /tmp -o "out_#1_#2.txt" http://httpbin/anything/{al,bt,gm}?num=[1-2]&& ls /tmp
You can disable globbing with the --globoff option if []{} characters are valid in your URLs. Then curl will treat them literally.
Use --parallel (-Z) to tell curl process URLs concurrently.
Config file
As the number of options increases, the curl command becomes harder to decipher. To make it more readable, you can prepare a config file that lists one option per line (-- is optional):
output-dir /tmpshow-errorsilent
By default, curl reads the config from $HOME/.curlrc, but you can change this with the --config (-K) option:
Now that we understand how curl handles URLs and options, let’s talk about specific features.
Version
--version (-V) knows everything about the installed version of curl:
curl -V
It lists (line by line) ➊ versions of curl itself and its dependencies, ➋ the release date, ➌ available protocols, and ➍ enabled features.
Verbose
--verbose (-v) makes curl verbose, which is useful for debugging:
curl -v http://httpbin/uuid
If --verbose is not enough, try --trace (the single - sends the trace output to stdout):
curl --trace - http://httpbin/uuid
Or --trace-ascii:
curl --trace-ascii - http://httpbin/uuid
Use --write-out (-w) to extract specific information about the response. It supports over 50 variables. For example, here we extract the status code and response content type:
--remote-name (-O) tells curl to save the output to a file specified by the URL (specifically, by the part after the last /). It’s often used together with --output-dir, which tells curl where exactly to save the file:
You can use --max-filesize (in bytes) to limit the allowed response size, but often it isn’t known in advance, so it may not work.
Retry
Sometimes the remote host is temporarily unavailable. To deal with these situations, curl provides the --retry [num] option. If a request fails, curl will try it again, but no more than num times:
curl -i --retry 3 http://httpbin/unstable
(this URL fails 50% of the time)
You can set the maximum time curl will spend retrying with --retry-max-time (in seconds) or the delay between retries with --retry-delay (also in seconds):
For curl, “request failed” means one of the following HTTP codes: 408, 429, 500, 502, 503 or 504. If the request fails with a “connection refused” error, curl will not retry. But you can change this with --retry-connrefused, or even enable retries for all kinds of problems with --retry-all-errors.
Uploads
Curl is often used to download data from the server, but you can also upload it. Use the --upload-file (-T) option:
To stop slow transfers, set the minimum allowed download speed (in bytes per second) with --speed-limit. By default, curl checks the speed in 30 seconds intervals, but you can change this with --speed-time.
For example, allow no less than 10 bytes/sec during a 3-second interval:
Another thing to limit is the number of concurrent requests (e.g. if you download a lot of files). Use --rate for this. It accepts seconds, minutes, hours or days:
There are also some network connection-level settings.
Timeouts
To limit the maximum amount of time curl will spend interacting with a single URL, use --max-time (in fractional seconds):
curl --max-time 0.5 http:/httpbin/delay/1
(this one fails)
Instead of limiting the total time, you can use --connect-timeout to limit only the time it takes to establish a network connection:
curl --connect-timeout 0.5 http:/httpbin/delay/1
(this one works fine)
Credentials
You almost never want to pass the username and password in the curl command itself. One way to avoid this is to use the .netrc file. It specifies hostnames and credentials for accessing them:
machine httpbin
login alice
password cheese
machine example.com
login bob
password nuggets
Pass the --netrc option to use the $HOME/.netrc file, or --netrc-file to use a specific one:
When curl exits, it returns a numeric value to the shell. For success, it’s 0, and for errors, there are about 100 different values.
For example, here is an exit status 7 (failed to connect to host):
curl http://httpbin:1313/get
You can access the exit status through the $? shell variable.
HTTP
Curl is mostly used to work with HTTP, so let’s talk about it.
HTTP/1.x is a plain-text protocol that describes the communication between the client and the server. The client sends messages like this:
POST /anything/chat HTTP/1.1host: httpbingo.orgcontent-type: application/jsonuser-agent: curl/7.87.0{ "message": "Hello!"}
The first line is a request line. The method (POST) defines the operation the client wants to perform. The path (/anything/chat) is the URL of the requested resource (without the protocol, domain and port). The version (HTTP/1.1) indicates the version of the HTTP protocol.
Next lines (until the empty line) are headers. Each header is a key-value pair that tells the server some useful information about the request. In our case it’s the hostname of the server (httpbingo.org), the type of the content (application/json) and the client’s self-identification (user-agent).
Finally, there is the actual data that the client sends to the server.
The first line is a status line. The version (HTTP/1.1) indicates the version of the HTTP protocol. The status code (200) tells whether the request was successful or not, and why (there are many status codes for different situations). The status message is a human-readable description of the status code (HTTP/2 does not have it).
Next lines (until the empty line) are headers. Similar to request headers, these provide useful information about the response to the client.
Finally, there is the actual data that the server sends to the client.
The HTTP protocol is stateless, so any state must be contained within the request itself, either in the headers or in the body.
HTTP/2, the successor to HTTP/1.1, is a binary protocol. However, curl displays HTTP/2 messages in plain text (just like HTTP/1.1), so we can safely ignore this fact for our purposes.
HTTP method
Curl supports all HTTP methods (sometimes called verbs).
GET (the default one, requires no options):
curl http://httpbin/get
HEAD (-I/--head, returns headers only):
curl -I http://httpbin/head
POST (-d/--data for data or -F/--form for HTTP form):
Typically, status codes 2xx (specifically 200) are considered “success”, while 4xx are treated as client-side errors and 5xx as server-side errors. But curl doesn’t care about codes: to it, every HTTP response is a success:
curl http://httpbin/status/503 && echo OK
To make curl treat 4xx and 5xx codes as errors, use --fail (-f):
curl -f http://httpbin/status/503 && echo OK
To print the response code, use --write-out with the response_code variable:
Response body, sometimes called payload, is what curl outputs by default:
curl http://httpbin/get
You can ask the server to compress the data with --compressed, but curl will still show it as uncompressed:
curl --compressed http://httpbin/get
(note how the Accept-Encoding request header has changed)
Ranges
To ask the server for a piece of data instead of the whole thing, use the --range (r) option. This will cause curl to request the specified byte range.
For example, here we request 50 bytes starting with the 100th byte:
curl --range 100-150 http://httpbin/range/1024
Note that the server may ignore the ask and return the entire response.
If you are downloading data from a server, you can also use --continue-at (-C) to continue the previous transfer at the specified offset:
curl --continue-at 1000 http://httpbin/range/1024
HTTP versions
By default, curl uses HTTP/1.1 for the http scheme and HTTP/2 for https. You can change this with flags:
--http0.9
--http1.0
--http1.1
--http2
--http3
To find out which version the server supports, use the http_version response variable:
Conditional requests are useful when you want to avoid downloading already downloaded data (assuming it is not stale). Curl supports two different conditions: file timestamp and etag.
Timestamp conditions use --time-cond (-z).
Download the data only if the remote resource is newer (condition holds):
Etag conditions are a bit more involved. An etag is a value returned by the server that uniquely identifies the current version of the requested resource. It is often a hash of the data.
To checks an etag, curl must first to save it with --etag-save:
Timestamp conditions rely on the Last-Modified response header, so if the server does not provide it, the resource will always be considered newer. The same goes for etag conditions and the Etag response header.
HTTP POST
POST sends data to the server. By default, it’s a set of key-value pairs encoded in a single string with a application/x-www-form-urlencoded Content-Type header.
You can use --data (-d) to specify individual key-value pairs (or the entire string):
curl -d name=alice -d age=25 http://httpbin/post
To send data from a file, use @ with a file path. Use --header (-H) to change the Content-Type header with according to the file contents:
To URL-encode data (escape all symbols not allowed in URLs), use --data-urlencode:
curl --data-urlencode "Name: Alice Barton" http://httpbin/post
Multipart formpost
POST can send data as a sequence of “parts” with a multipart/form-data content type. It’s often used for HTML forms that contain both text fields and files.
Each part has a name, headers, and data. Parts are separated by a “mime boundary”:
--------------------------d74496d66958873e
Content-Disposition: form-data; name="person"
anonymous
--------------------------d74496d66958873e
Content-Disposition: form-data; name="secret"; filename="file.txt"
Content-Type: text/plain
contents of the file
--------------------------d74496d66958873e--
To construct multipart requests with curl, use --form (F). Each of these options adds a part to the request:
A redirect is when the server, instead of returning the requested resource, tells the client that the resource is located elsewhere (as indicated by the Location header). A redirect always has a 3xx response code.
Curl does not follow redirects by default, it returns the response as is:
curl -i http://httpbin/redirect/1
To make curl follow redirects, use --follow (-L):
curl -L http://httpbin/redirect/1
To protect against endless loop redirects, use --max-redirs:
curl -L --max-redirs 3 http://httpbin/redirect/10
HTTP PUT
The PUT method is often used to send files to the server. Use --upload-file (-T) for this:
Sometimes PUT is used for requests in REST APIs. For these, use --request (-X) to set the method and --data (-d) to send the data:
curl -X PUT -H "content-type: application/json"
-d '{"name": "alice"}'
http://httpbin/put
Cookies
The HTTP protocol is stateless. Cookies are an ingenious way around this:
The server wants to associate some state with the client session.
The server returns that state in the Set-Cookie response header.
The client recognizes the cookies and sends them back with each request in the Cookie request header.
Each cookie has an expiration date — either explicit one or “end of session” one (for browser clients, this is often when the user closes the browser).
Curl ignores cookies by default. To enable them, use the --cookie (-b) option. To make curl persist cookies between calls, use --cookie-jar (-c).
Subsequent curl calls with -b /tmp/cookies will send the sessionid cookie back to the server.
Curl automatically discards cookies from the cookie jar when they expire (this requires an explicit expiration date set by the server). To discard session-based cookies, use --junk-session-cookies (-j):
curl -j -b /tmp/cookies http://httpbin/get
Alternative services
The Alt-Svc HTTP response header indicates that there is another network location (an alternative service) that the client can use for future requests.
To enable alternative services, use --alt-svc. This tells curl to store the services in the specified file and consider them for future requests.
The HTTP Strict-Transport-Security response header (also known as HSTS) informs the client that the server should only be accessed via HTTPS, and that any future attempts to access it via HTTP should automatically be converted to HTTPS.
To make curl respect HSTS, use --hsts. This tells curl to store HSTS-enabled servers in the specified file and automatically convert http → https when accessing them.
Since the article is getting huge, I’m going to skip some parts of the video:
Curl supports protocols like SCP, SFTP, email (IMAP/POP3/SMTP), MQTT, TELNET, DICT, and WebSocket (experimental). Daniel goes over them in the video (FTP in detail, the rest very briefly).
Curl fully supports TLS (Transport Layer Security, the underlying protocol for HTTPS). You can set a specific minimal allowed TLS version, read certificates from a file, or use client certificates for mutual authentication.
Curl supports proxies (applications that act as intermediaries between the client and the server). There are HTTP and SOCKS proxies, and of course curl supports both, with or without authentication.
Curl is 25 years old in 2023. It has been growing and improving all these years, and the rate of development is only increasing. The Internet is constantly changing, with new protocols and ways of doing transfers popping up all the time. Curl needs to keep up, so it will continue to grow and add exciting new features!
There are two great resources if you want to dig deeper:
