Mini HTTP Guide for Developers

• developers, Guide

9 min. (1855 words)

mini-http-guide-for-developers

Frameworks often hide/abstract parts of HTTP away. I think this is often a bit of a shame: it hides what’s possible with HTTP, and so can lead to effects on engineering decisions.

This short guide aims to rectify that. It details a few of the most common and useful parts of HTTP, and is aimed for developers with some experience making or receiving HTTP requests. [In this post, the term HTTP is used to refer to the bytes of the HTTP protocol, which is the same if those bytes are sent over plain TCP, or through a TLS tunnel. The term HTTPS is used only when necessary to distinguish HTTP over TLS.]

Initialising the connection

Say we ask our HTTP client to make a GET request to the URL https://example.com/the/path. Firstly, there isn’t no such a thing as a URL in HTTP: it’s just a shorthand that the client parses and uses the different components at various points in the process.

  • The client resolves example.com to an IP address, say to 1.2.3.4. Note that typically in many cases this would involve sending the string example.com unencrypted across the network.
  • Initiate a TCP connection to the IP address 1.2.3.4 on port 443: if no port is specified in an HTTPS URL, port 443 is assumed. For HTTP URLs, port 80 is assumed.
  • Initiate a TLS connection over the top of this TCP connection. This again would uses the domain example.com, in both SNI, and verification of the subsequently supplied certificate. In many cases, the domain name example.com would be transmitted unencrypted across the internet.
  • Then start the HTTP request/response process. This would use both the domain example.com, as well as /the/path. This process is detailed below.

The HTTP request

The client sends the bytes of the HTTP request message over the TLS connection. This is made of a request-line containing the method and the path, followed by a number of header key:value pair lines, a blank line, and then the body. In this case, the body is 0-bytes long, which is typical for GET requests.

GET /the/path HTTP/1.1rn
host: example.comrn
rn

A “line” ends with the two characters rn. The visual line breaks in the examples shown here are for ease of comprehension, and are not characters that are transmitted.

The HTTP response

The server would then respond with a status-line, some headers, a blank line, and the body of the response.

HTTP/1.1 200 OKrn
content-length: 21rn
rn
The bytes of the body

Notable headers

The important parts of HTTP are the headers: bits of metadata sent before the body of the message [usually].

Host header

Usually HTTP clients add a host header automatically from the supplied URL. It has two common uses.

  • CDNs or reverse proxies use the host header to determine how to route requests onwards.
  • Application server code uses the host header in a best-effort attempt to determine the domain the HTTP client used to make the request. Depending on the configuration of intermediate proxies, this can mean the application server may not be able to correctly determine what the original domain was.

Content-length header

HTTP is sent over TCP [or TCP TLS], which surfaces as a stream of bytes in client code. A “stream of bytes” means that the receiver could just receive a single byte at a time, perhaps even with seconds of delay between each. The receiver has no way to know if it has received all of the bytes, or the connection is just a bit slow. For this reason, requests and responses [that can have a body], can supply a header that tells the other end how many bytes are part of the body of the message. This is the content-length header.

Often HTTP clients add this automatically if they know at the time of starting sending the HTTP message how many bytes will be sent in the body.

Transfer-encoding header

A HTTP message sender may want to send a body, but it does not know at the start how many bytes make up that body. One option is that it can wait until it knows how many bytes, and set the content-length header appropriately. However, this may involve having to buffer all the bytes in memory, which may not be possible or desirable.

An alternative is to use transfer-encoding: chunked. With this header, the body of the message is sent in chunks, each prefixed by the number of bytes in that chunk [as it happens, in hexadecimal]. This means the body transfer can be started without knowing how many bytes in total will be sent. Common chunk sizes are between 8kb and 64kb. Often HTTP clients “do the chunking” themselves, adding the chunk header before each chunk as needed.

However, it is usually better to avoid transfer-encoding: chunked and instead set a content-length header. The receiver can use this in various ways, such as to or be able to allocate resources needed at the start of downloading the body, or estimate time remaining. If the receiver needs to know how many bytes are in the body, using transfer-encoding: chunked may be forcing it to buffer the entire body in memory before it can process it further.

Wonderfully, you can often still stream bodies with a correctly set content-length, but you may need to go to a bit of effort to find the right value. For example, to stream a file you may need to query the file system explicity to find the length of the file before starting to fetch its bytes.

Connection header

HTTP/1.1 by default keeps connections open after a HTTP request/response, so they can be used for subsequent request/response, and avoid the overhead of new TCP [or TCP TLS] connections. This referred to as persistant connections, and is often a good thing, but has downsides.

Usually servers would only keep the connections alive for a certain period of time, and then close them. This means there is a race condition: a server could have closed the connection from their point of view, but the client not be aware of this and attempt to re-use the connection, send its bytes [but the server wouldn’t process them], and only later some time would the client would be aware of an error condition. The client may not know if it’s safe to retry the request or not. For example, a client may have no way of determining if a POST errored before or after it was processed by the server. If designing an API, you may wish to implement some sort of unique idempotency-key for such requests. With this, the client can safely retry requests that have failed from its point of view, while the server knows not to reprocess any duplicates, and can still return the response corresponding to the original request.

Another downside is that if you don’t end up re-using the connection, resources would continue to be used needlessly on both the client and the server.

If you want a smaller chance of issues like this, you may explicitly set a connection: close header. If you can deal with such issues, you may wish to design the system to take better advantage of persistant connections. For example, instead of choosing to have multiple S3 buckets each on a different domain, you choose to have one, to take better advantage of per-domain HTTP persistent connections and speed up S3 requests/responses.

X-forwarded-proto

This is a modern header: it is often added to requests by HTTP-aware intermediate CDNs or reverse proxies. If the proxy has received an HTTPS connection, it can add x-forwarded-proto: https, and otherwise adds x-forwarded-proto: http.

Without this header, the application server behind a reverse proxy would have no mechanism to know if the client made its request via HTTP or HTTPS. This may be important if you would like to respond to HTTP requests with redirects to HTTPS URLs.

X-forwarded-for

Often an application server would like to know the IP address of the client. However, if the client connects to a reverse proxy, and then the reverse proxy connects to the application server, the application server only has details of that final TCP connection. From its point of view, its TCP client is the reverse proxy. This is often not helpful.

The solution to this is that each intermediate proxy adds (to) the x-forwarded-for header in the request, setting the IP address that its incoming TCP connection is from. If there is already an x-forwarded-for header on its incoming HTTP request, it appends the IP address to this in a comma separated list before forwarding the HTTP request onwards.

This means that the application server can receive an x-forwarded-for with a long list of IP addresses in it, for example x-forwarded-for: 1.2.3.4, 5.6.7.8, 9.10.11.12. Because each server adds to the value of the existing x-forwarded-for header supplied by a potentially untrustworthy client, care must be taken before trusting any particular value in this list.

For example, you may have an application accessible behind a CDN, which adds an x-forwarded-for, so in the application server you may be tempted to trust the first IP in x-forwarded-for. However, the CDN would append to any existing values in x-forwarded-for. This means that an evil client can send a request with an existing x-forwarded-for header, set with some IP, and trick the application into thinking the client is at that IP. Knowing this, you may choose to use the last IP address in the list, thinking that this can be trusted. However, this may also not be a good choice: often applications are accessible both from the CDN, but also directly, even if just via an IP address. An evil client could connect to this with a spoofed x-forwarded-for header, and again trick the application.

Solutions to this trust issue involve only using the last N values of x-forwarded-for, where you have a mechanism to ensure that those N hops a) definitely involved certain infrastructure and b) you trust that infrastruture to manipulate any existing x-forwarded-for in a certain way.

Summary: Reconstructing URLs

Reconstructing the URL that a client used involves multiple parts of the HTTP request: the path of the start-line, the host header, as well as the x-forwarded-proto header. For all this to work, intermediate proxies must be appropriately configured.

Summary: Streaming

HTTP is often enough for streaming: you may not need anything fancier. If you can determine the full length of the body, set the content-length header; otherwise, use transfer-encoding: chunked.

Summary: HTTP is leaky

All non-trivial abstractions, to some degree, are leaky.

Joel Spolsky

HTTP is a leaky abstraction, exposing information on the lower-level TCP [or TCP TLS] connection via the x-forwarded-* headers; giving the ability to control that connection via the connection header; and requiring one of content-length or transfer-encoding headers to make up for the fact that TCP doesn’t have any concept of message length.

If you want to take full advantage of HTTP, you should be aware of these; compensate for them; and even be able to leverage them when needed to avoid unnecessary time, memory, code, or infrastructure use.

124 comments

  1. I all the time used to read paragraph in news papers but now as I am a user
    of web therefore from now I am using net for
    posts, thanks to web.

  2. Hi! Would you mind if I share your blog with my facebook group?

    There’s a lot of people that I think would really appreciate your content.
    Please let me know. Many thanks

  6. Do you mind if I quote a few of your posts as long as I provide credit
    and sources back to your weblog? My blog site is in the very same
    area of interest as yours and my users would certainly benefit from some of
    the information you provide here. Please let me know if this ok with you.
    Appreciate it!

    Also visit my blog – wwwjoker388net, Teri,

  7. Have you ever considered about including a little bit more than just
    your articles? I mean, what you say is valuable and everything.
    Nevertheless imagine if you added some great photos or videos to give your
    posts more, “pop”! Your content is excellent but with images and video clips, this site
    could undeniably be one of the best in its field. Amazing blog!

    Also visit my homepage :: id akaun test lpe88

  10. Hey just wanted to give you a quick heads up and let you know
    a few of the images aren’t loading properly.
    I’m not sure why but I think its a linking issue.
    I’ve tried it in two different web browsers and both show the same outcome.

    Feel free to surf to my page … download live22 android

  12. Hi there I am so delighted I found your blog, I really found you by mistake, while I was browsing on Aol for something else, Regardless I am here now and would just like to say thanks a lot for a incredible post and a all
    round thrilling blog (I also love the theme/design),
    I don’t have time to go through it all at the moment but
    I have saved it and also added your RSS feeds, so when I have time I will be back to read a lot more,
    Please do keep up the excellent job.

    My web-site … gem club suncity

  13. Please let me know if you’re looking for a author for your blog.
    You have some really great posts and I think I would be a good asset.
    If you ever want to take some of the load off, I’d really like to
    write some articles for your blog in exchange for a link back to mine.

    Please send me an email if interested. Cheers!

    Here is my webpage; game 918kaya

  19. Excellent post. I was checking continuously this blog and I’m inspired!
    Extremely helpful info specifically the final part
    🙂 I maintain such information a lot. I used to be looking for this certain info for a very lengthy time.
    Thanks and good luck.

    Stop by my site – frun-test.sakura.ne.jp

  22. My spouse and i got absolutely delighted when Jordan could complete his investigations from your precious recommendations he received through your
    site. It’s not at all simplistic to simply find yourself
    giving away secrets and techniques that others might have been trying to sell.
    And we all keep in mind we have got the blog owner to appreciate because of that.
    The most important illustrations you’ve made, the simple web site navigation, the relationships your site help to foster – it is most amazing, and it’s
    assisting our son and the family imagine that this article is
    brilliant, and that is exceedingly important. Many thanks for the whole lot!

    Feel free to surf to my website Clinton

  28. Everything posted made a lot of sense. But, what about
    this? what if you added a little content? I mean, I don’t want to tell you how to run your blog, however suppose you added a post
    title that grabbed folk’s attention? I mean Mini HTTP Guide
    for Developers – Pavvy Designs is kinda vanilla.
    You might look at Yahoo’s front page and watch how they write article headlines to grab people interested.

    You might add a video or a pic or two to grab readers excited about everything’ve written. In my opinion, it would
    bring your posts a little livelier.

    Also visit my blog http://frun-test.sakura.ne.jp/userinfo.php?uid=121810

  30. What’s up all, here every person is sharing such familiarity, thus it’s good to read this
    web site, and I used to go to see this web site everyday.

    Also visit my homepage: Arnette

  32. I think this is among the most important information for me.

    And i’m glad reading your article. But wanna remark on few
    general things, The website style is great, the articles is
    really great : D. Good job, cheers

    Here is my blog … kebe.top

  33. Wow, incredible blog layout! How lengthy have you ever been running a
    blog for? you made blogging look easy. The overall glance of your website is great, let alone the content![X-N-E-W-L-I-N-S-P-I-N-X]I simply could not go away your web
    site before suggesting that I actually loved the usual info a person provide in your guests?
    Is going to be again ceaselessly in order to
    check up on new posts.

    my page – https://www.diablo.moe/

  36. I must point out my passion for your kind-heartedness in support of visitors who require assistance with that subject.
    Your real dedication to passing the solution all
    through came to be particularly interesting and has in most cases enabled men and women much like me
    to achieve their targets. The warm and helpful key points signifies this much a
    person like me and still more to my office workers.
    Thank you; from each one of us.

    Here is my web-site – lubertsi.net

  38. I feel that is one of the so much vital info for me. And i am glad studying your article.
    But wanna observation on few common issues, The website taste is wonderful, the articles is in point
    of fact great :D. Excellent job, cheers.

    My web page; mpc-install.com

  40. I think this is one of the most vital information for me. And i’m glad reading your article.
    But want to remark on few general things, The web site style is wonderful, the
    articles is really nice : D. Good job, cheers

    Here is my blog post clubriders.men

  46. When someone writes an article he/she keeps the image of a user in his/her brain that how a user can be aware
    of it. Thus that’s why this post is outstdanding. Thanks!

    Also visit my homepage: chengdian.cc

  50. I’m amazed, I must say. Seldom do I encounter a blog that’s equally educative
    and amusing, and without a doubt, you have hit the nail on the head.
    The issue is something that too few men and women are speaking
    intelligently about. Now i’m very happy that I found this during my search for something concerning this.

    Also visit my web-site :: http://www.aniene.net/

  51. obviously like your web site however you have to test the spelling on several of your posts.

    A number of them are rife with spelling problems and I in finding it very bothersome to tell the truth however I will surely come back again.

    Also visit my site: http://www.1stanapa.ru

  52. Please let me know if you’re looking for a article author for
    your site. You have some really good posts and I feel I would be
    a good asset. If you ever want to take some of the load
    off, I’d really like to write some material for your blog
    in exchange for a link back to mine. Please blast me an e-mail if interested.
    Cheers!

    my page https://kebe.top

  55. Hello there, just became alert to your blog through Google, and found that it’s really informative.
    I?m going to watch out for brussels. I?ll be grateful if
    you continue this in future. Lots of people will be benefited from your writing.
    Cheers!

    My web page mpc-install.com

  60. Interesting blog! Is your theme custom made or did you download it from somewhere?
    A theme like yours with a few simple tweeks would really make my blog stand out.

    Please let me know where you got your theme. Thanks a lot

    My web site Dick Fuel

  65. Attractive element of content. I simply stumbled upon your
    weblog and in accession capital to say that I get actually enjoyed account your blog posts.
    Anyway I will be subscribing in your augment or even I achievement you
    get right of entry to constantly quickly.

  67. Thanks for one’MosQiller S Review marvelous posting!
    I genuinely enjoyed reading it, you are a great author.I will make certain to bookmark your blog and will often come back down the road.
    I want to encourage yourself to continue your great work, have a nice evening!

  69. I was suggested this blog by my cousin. I’m not sure whether this post is written by him as
    no one else know such detailed about my difficulty. You are
    wonderful! Thanks!

  70. Today, I went to the beach with my kids. I found a sea shell and gave it to my 4 year old
    daughter and said “You can hear the ocean if you put this to your ear.” She placed the shell to her ear and screamed.
    There was a hermit crab inside and it pinched her ear.
    She never wants to go back! LoL I know this is totally off topic but I had to
    tell someone!

  71. hello!,I love your writing so much! share we be in contact
    extra about your article on AOL? I require an expert in this area to unravel my problem.

    Maybe that’s you! Taking a look forward to peer you.

  72. Heya! I understand this is somewhat off-topic but I needed to ask.
    Does managing a well-established blog like yours require
    a lot of work? I am brand new to operating a blog but I do
    write in my diary everyday. I’d like to start
    a blog so I can easily share my experience and feelings
    online. Please let me know if you have any ideas or tips for brand new aspiring blog owners.

    Appreciate it!

  74. You could definitely see your skills in the work you write.
    The arena hopes for even more passionate writers such as you who are not afraid to mention how they believe.
    At all times follow your heart.

  75. Very nice post. I simply stumbled upon your blog and wished to mention that
    I’ve truly enjoyed surfing around your weblog posts. In any case I will be subscribing for your feed and
    I’m hoping you write once more soon!

  79. It’s perfect time to make a few plans for the longer term and it is time to be happy.
    I have learn this submit and if I may I desire to suggest you some interesting things or suggestions.
    Perhaps you could write subsequent articles regarding this article.
    I want to read even more things approximately it!

    Here is my web site; Maribel

  84. I’m truly enjoying the design and layout of your website.
    It’s a very easy on the eyes which makes it much more
    enjoyable for me to come here and visit more often.
    Did you hire out a designer to create your theme?
    Fantastic work! scoliosis surgery https://0401mm.tumblr.com/ scoliosis surgery

  87. Thank you for your website post. Thomas and I are actually saving to
    buy a new e-book on this subject and your post has made us all to save money.

    Your thinking really clarified all our concerns. In fact, above
    what we had acknowledged just before we came upon your fantastic blog.

    My partner and i no longer have doubts and
    also a troubled mind because you have really attended to our own needs above.
    Thanks

    Feel free to visit my site :: Reneaux Serum

  88. Helpful information. Lucky me I found your website by accident, and I’m shocked
    why this twist of fate did not happened in advance!
    I bookmarked it.

    Here is my homepage: Hilton

  89. Thank you for the good writeup. It in fact was a amusement account it.
    Look advanced to more added agreeable from you! By the way, how can we communicate?

    Check out my page: drug use

  92. I don’t know whether it’s just me or if perhaps everyone else encountering issues with your
    site. It appears as though some of the written text within your content are
    running off the screen. Can someone else please comment and let me know if this is happening
    ways to have better sex them as well?
    This could be a problem with my web browser because I’ve
    had this happen previously. Many thanks

  96. Pretty portion of content. I just stumbled upon your website and in accession capital to say that I get
    in fact loved account your weblog posts. Any way I will be subscribing for your feeds and even I success you
    get right of entry to constantly rapidly.

  102. I was more than happy to find this web site. I want to to thank you for ones time due to this fantastic read!!
    I definitely really liked every bit of it
    and i also have you saved to fav to look at new things in your web
    site.

  104. Odd this publish is totaly unrelated to what I used to be looking google for, nevertheless it was once indexed at the first page. I assume your doing one thing right if Google likes you enough to place you at the first page of a non related search.

  107. I was curious if you ever considered changing the structure of your blog?
    Its very well written; I love what youve got to say. But maybe you could
    a little more in the way of content so people could connect with it better.
    Youve got an awful lot of text for only having one or 2 images.
    Maybe you could space it out better?

  110. you’re actually a excellent webmaster. The website loading velocity is amazing.
    It seems that you are doing any unique trick. In addition, The contents are masterpiece.

    you’ve performed a great job on this topic!

  111. Hi there, i read your blog from time to time and i own a similar one and i
    was just wondering if you get a lot of spam remarks?
    If so how do you reduce it, any plugin or anything you can advise?
    I get so much lately it’s driving me mad so any assistance
    is very much appreciated.

  113. Howdy! This article could not be written any better!

    Looking through this article reminds me
    of my previous roommate! He continually kept preaching about
    this. I’ll send this post to him. Pretty sure he’ll have a great read.
    I appreciate you for sharing!

  114. you’re really a good webmaster. The web site loading speed is
    incredible. It kind of feels that you’re doing any unique trick.
    Moreover, The contents are masterpiece. you’ve done a excellent
    activity on this subject!

  115. Hello, Neat post. There is a problem along with your website in web
    explorer, may test this? IE still is the market chief and a huge section of folks will omit your fantastic writing due to
    this problem.

  120. Hi there colleagues, how is all, and what you want to say about this piece of writing, in my view its
    actually awesome designed for me.

  121. A fascinating discussion is definitely worth comment.
    I do think that you need to write more about this subject,
    it might not be a taboo matter but typically people don’t discuss these subjects.
    To the next! All the best!!

  122. My brother recommended I might like this web site. He
    was entirely right. This post truly made my day. You cann’t imagine simply
    how much time I had spent for this info! Thanks!

  123. I really like what you guys are up too. Such clever work and exposure!

    Keep up the fantastic works guys I’ve added you guys to my own blogroll.

Leave a Reply

Your email address will not be published.