Learn HTTP in about a week!
Learning should be fun and quick; no need to spend ages in just one single course.
What exactly is chunked encoding in HTTP?
When the size of data to be sent over HTTP is not known beforehand, you can consider using chunked transfer encoding.
HTTP has evolved a lot over the ages. With each new version, the issues of the previous versions got addressed and many new, shiny features got introduced as well. At the moment, the topic of discussion is the issue of sending a payload without knowing its size beforehand.
In HTTP 1.0, this issue was handled by a connection cutoff. However, this was an inefficient approach, for the same connection couldn't then be used to make another request.
HTTP 1.1 fixed it by introducing something called chunked encoding. Let's spare a few minutes to understand what exactly is chunked encoding and why is it used.
What is chunked encoding?
At its core, chunked encoding is when an HTTP message's body is sent in small chunks. This is indicated by the Transfer-Encoding HTTP header, set to the value chunked.
Transfer-Encoding: chunkedEach chunk has a special representation to allow the receiver to determine how many bytes are present in the chunk and then extract those bytes easily.
Chunked encoding was introduced in HTTP 1.1 to provide a more sensible approach to deliver HTTP messages where the body's size is not known beforehand. The previous solution, in HTTP 1.0, was to sever the connection. The receiver would take the disconnection as a signal for the body's end.
But, as you can expect, this method was inefficient. Most importantly, the connection could NOT be reused for multiple requests. With the advent of chunked encoding, the issue got alleviated and now the connection didn't need to be severed.
Transfer-Encoding vs. Content-Encoding
Bear in mind that Transfer-Encoding is different from Content-Encoding. They can easily be confused to mean the same thing.
Transfer-Encoding specifies how the bytes of the body are being transferred to the end receiver; NOT how the bytes themselves are encoded. Content-Encoding, on the other hand, specifies the compression scheme used in the message, that is, how the bytes of the body are encoded themselves.
As an example, a message might be both, chunk-encoded and compressed at the same time.
The format of chunked encoding
Chunked encoding has a dedicated format to it. When you wish to deliver a message in the chunk-encoded format, first and foremost you must set Transfer-Encoding to chunked and then prepare to send each and every chunk, one by one.
Speaking of which, each chunk is composed of two main things:
- The chunks's size.
- The chunk's actual data.
The size is required so that the receiver is able to read the chunk's following data reliably. Note that the size is a hexadecimal number. Both the chunk size and data end with \r\n — the carriage-return (CR) and line-feed (LF) characters (often denoted as CRLF).
For example, let's suppose that you wish to send the data "Hello world" in a chunk. Here's how the chunk would look:
B\r\n
Hello world\r\nIn the first line, B represents the chunk's size, in hexadecimal. B means 11, so the chunk's data is 11 bytes in size. The second line contains the entire 11 bytes of data: "Hello world".
\r\n because they denote a new line together and this new line is already shown in the message. If I were to truly depict the message, it would be B\r\nHello world\r\n.The very last chunk to signal the end of the entire stream of chunks has a size of 0 and no data. Something like this:
0\r\n
\r\nBasically, there's an empty new line following the 0.
Prefer Content-Length if possible
Generally, it's a good idea to use Content-Length, if it's possible, in place of chunked encoding. But why?
you ask.
Prevents chunk overhead
Well, for one, Content-Length prevents all the overhead of chunked encoding (that is, the size information of each chunk in addition to the CRLFs).
Provides the total size
Secondly, Content-Length provides useful size information to the receiver which is often leveraged to determine how much portion has been received thus far.
For example, when you download a file via the browser, the fact that the browser displays something like "10% completed" is only because the server sent back the size of the file via a Content-Length header. Without this header, the browser can only show you how much it has downloaded so far, not the total size.
Anyways, to restate it: Make sure to include Content-Length if you can. Use chunked encoding only when the data is truly being generated dynamically and computing its total size is impossible or impractical.
