Go’s standard library HTTP server supports HTTP/2 by default. It has great documentation, and a great demo page [code]. In this post, I will first show Go’s HTTP/2 server capabilities, and explain how to consume them as clients. Then, I will present h2conn, a library that simplifies full-duplex communication over an HTTP/2 connection.

The code in this post is available at posener/h2demo.

HTTP/2 server

Let’s create an HTTP/2 server in Go! According to the HTTP/2 documentation, everything is automatically configured for us, we don’t even need to import Go’s standard library http2 package:

This package is low-level and intended to be used directly by very few people. Most users will use it indirectly through the automatic use by the net/http package (from Go 1.6 and later). For use in earlier Go versions see ConfigureServer. (Transport support requires Go 1.6 or later)

HTTP/2 enforces TLS. In order to achieve this we first need a private key and a certificate. On Linux, the following command does the job. Run it and follow the prompted questions.

openssl req -newkey rsa:2048 -nodes -keyout server.key -x509 -days 365 -out server.crt

The command will generate two files: server.key and server.crt.

• server.key: Contains our server private key - it should remain private and secret in production systems. This key will be used to encrypt HTTPS responses, which could be decrypted with our server public key.
• server.crt: The server certificate - represents the server’s identity and contains the server’s public key. This file can be shared publicly and its content is sent to the client as part of the TLS handshake.

Now, for the server code, in its simplest form, we will just use Go’s standard library HTTP server and enable TLS with the generated SSL files.

package main

import (
	"log"
	"net/http"
)

func main() {
	// Create a server on port 8000
	// Exactly how you would run an HTTP/1.1 server
	srv := &http.Server{Addr: ":8000", Handler: http.HandlerFunc(handle)}

	// Start the server with TLS, since we are running HTTP/2 it must be
	// run with TLS.
	// Exactly how you would run an HTTP/1.1 server with TLS connection.
	log.Printf("Serving on https://0.0.0.0:8000")
	log.Fatal(srv.ListenAndServeTLS("server.crt", "server.key"))
}

func handle(w http.ResponseWriter, r *http.Request) {
	// Log the request protocol
	log.Printf("Got connection: %s", r.Proto)
	// Send a message back to the client
	w.Write([]byte("Hello"))
}

No TLS? The H2C (HTTP/2 Cleartext) protocol is HTTP/2 with no TLS. The standard library will support it only from Go 1.12. But currently the external package x/net/http2/h2c can be used.

Edit: The standard library won’t include the H2C handler, it will remain in the x/net/http2/h2c package.

HTTP/2 Client

In go, the standard http.Client is used for HTTP/2 requests as well. The only difference is the usage of http2.Transport instead of http.Transport in the client’s Transport field.

Our generated server certificate is “self signed”, which means it was not signed by a known certificate authority (CA). This will cause our client not to trust it:

package main

import (
	"fmt"
	"net/http"
)

const url = "https://localhost:8000"

func main() {
	_, err := http.Get(url)
	fmt.Println(err)
}

Let’s try to run it:

$ go run h2-client.go 
Get https://localhost:8000: x509: certificate signed by unknown authority

In the server logs, we will also see the that the client (the remote) had an error:

http: TLS handshake error from [::1]:58228: remote error: tls: bad certificate

To solve this, we can configure the client with a custom TLS configuration. We will add the server certificate file to the client “certificate pool”, since we trust this one even though it was not signed by a known CA.

We will also add an option to choose between HTTP/1.1 and HTTP/2 transports according to a command line flag.

Here is the code (view on github):

package main

import (
	"crypto/tls"
	"crypto/x509"
	"flag"
	"fmt"
	"io/ioutil"
	"log"
	"net/http"

	"golang.org/x/net/http2"
)

const url = "https://localhost:8000"

var httpVersion = flag.Int("version", 2, "HTTP version")

func main() {
	flag.Parse()
	client := &http.Client{}

	// Create a pool with the server certificate since it is not signed
	// by a known CA
	caCert, err := ioutil.ReadFile("server.crt")
	if err != nil {
		log.Fatalf("Reading server certificate: %s", err)
	}
	caCertPool := x509.NewCertPool()
	caCertPool.AppendCertsFromPEM(caCert)

	// Create TLS configuration with the certificate of the server
	tlsConfig := &tls.Config{
		RootCAs: caCertPool,
	}

	// Use the proper transport in the client
	switch *httpVersion {
	case 1:
		client.Transport = &http.Transport{
			TLSClientConfig: tlsConfig,
		}
	case 2:
		client.Transport = &http2.Transport{
			TLSClientConfig: tlsConfig,
		}
	}

	// Perform the request
	resp, err := client.Get(url)
	if err != nil {
		log.Fatalf("Failed get: %s", err)
	}
	defer resp.Body.Close()
	body, err := ioutil.ReadAll(resp.Body)
	if err != nil {
		log.Fatalf("Failed reading response body: %s", err)
	}
	fmt.Printf(
		"Got response %d: %s %s\n",
		resp.StatusCode, resp.Proto, string(body))
}

This time we get the proper response:

$ go run h2-client.go 
Got response 200: HTTP/2.0 Hello

On the server logs we will see the right log line: Got connection: HTTP/2.0!

But what happens when we try to use HTTP/1.1 transport?

$ go run h2-client.go -version 1
Got response 200: HTTP/1.1 Hello

Our server has nothing specific for HTTP/2, so it supports HTTP/1.1 connections. This is important for backward compatibility. Additionally, the server log indicates that the connection was HTTP/1.1: Got connection: HTTP/1.1.

HTTP/2 Advanced Features

We created an HTTP/2 client-server connection, and we are enjoying the benefits of a secured and efficient connection. But HTTP/2 provides more features, let’s investigate them!

Server Push

HTTP/2 enables server push which “constructs a synthetic request using the given target”.

This can be easily implemented in the server handler (view on github):

func handle(w http.ResponseWriter, r *http.Request) {
	// Log the request protocol
	log.Printf("Got connection: %s", r.Proto)

	// Handle 2nd request, must be before push to prevent recursive calls.
	// Don't worry - Go protect us from recursive push by panicking.
	if r.URL.Path == "/2nd" {
		log.Println("Handling 2nd")
		w.Write([]byte("Hello Again!"))
		return
	}

	// Handle 1st request
	log.Println("Handling 1st")

	// Server push must be before response body is being written.
	// In order to check if the connection supports push, we should use
	// a type-assertion on the response writer.
	// If the connection does not support server push, or that the push
	// fails we just ignore it - server pushes are only here to improve
	// the performance for HTTP/2 clients.
	pusher, ok := w.(http.Pusher)
	if !ok {
		log.Println("Can't push to client")
	} else {
		err := pusher.Push("/2nd", nil)
		if err != nil {
			log.Printf("Failed push: %v", err)
		}
	}

	// Send response body
	w.Write([]byte("Hello"))
}

A word about the http.Pusher implementation: I must admit that the design of type-assertion to check if the connection supports server push is a weird choice and it is not clear to me. I assume that it is adopted for backward compatibility with Go 1.1, but I wonder if there could be a nicer way to add this capability. The same goes for the http.Flusher implementation, that will be discussed below.

Consuming Server Push

Let’s re-run the server, and test the clients.

For HTTP/1.1 client:

$ go run ./h2-client.go -version 1
Got response 200: HTTP/1.1 Hello

Server logs will show:

Got connection: HTTP/1.1
Handling 1st
Can't push to client

The HTTP/1.1 client transport connection results in an http.ResponseWriter that does not implement the http.Pusher, this makes sense. In our server code we can choose what to do in the case of this kind of client.

For HTTP/2 client:

go run ./h2-client.go -version 2
Got response 200: HTTP/2.0 Hello

The server logs will show:

Got connection: HTTP/2.0
Handling 1st
Failed push: feature not supported

That’s weird. Our client with HTTP/2 transport only got the first “Hello” response. The log indicates that the connection implements the http.Pusher interface - but once we actually invoke the Push() function - it fails.

I found this StackOverflow thread with an example how to enable server push for go clients. Apparently, the HTTP/2 client transport sets an HTTP/2 setting flag that indicates that the push is disabled. There are this Github issue and this proposed change set that suppose to enable HTTP/2 push, but it seems to hang there for quite a long time.

So currently, there is no option to consume the server Push with a Go client.

As a side note, google-chrome, as a client, can handle server push.

The server logs will show what we expect, the handler was called twice, with paths / and /2nd, even though the client actually made only one request with path /:

Got connection: HTTP/2.0
Handling 1st
Got connection: HTTP/2.0
Handling 2nd

Full Duplex Communication

The Go HTTP/2 demo page has an echo example, which demonstrates a full-duplex communication between server and client.

Let’s test it first with CURL:

$ curl -i -XPUT --http2 https://http2.golang.org/ECHO -d hello
HTTP/2 200 
content-type: text/plain; charset=utf-8
date: Tue, 24 Jul 2018 12:20:56 GMT

HELLO 

We configured curl to use HTTP/2, and sent a PUT /ECHO with “hello” as the body. The server returned an HTTP/2 200 response with “HELLO” as the body. But we didn’t do anything sophisticated here, it looks like a good old HTTP/1.1 half-duplex communication with different header. Let’s dig into this, and investigate how we can use the HTTP/2 full-duplex capabilities.

Server Implementation

A simplified version of the HTTP echo handler (a one that does not capitalizes the response) is below. It uses the http.Flusher interface, that HTTP/2 adds to the http.ResponseWriter.

type flushWriter struct {
	w io.Writer
}

func (fw flushWriter) Write(p []byte) (n int, err error) {
	n, err = fw.w.Write(p)
	// Flush - send the buffered written data to the client
	if f, ok := fw.w.(http.Flusher); ok {
		f.Flush()
	}
	return
}

func echoCapitalHandler(w http.ResponseWriter, r *http.Request) {
	// First flash response headers
	if f, ok := w.(http.Flusher); ok {
		f.Flush()
	}
	// Copy from the request body to the response writer and flush
	// (send to client)
	io.Copy(flushWriter{w: w}, r.Body)
}

The server copies everything from the request body reader to a “flush writer” that writes to the ResponseWriter and Flush() it. Again, we see the awkward type-assertion style implementation, the flush operation sends the buffered data to the client.

Notice that this is full-duplex, the server reads a line and write-flushes a line, repeatedly, in one HTTP handler call.

Go Client Implementation

I tried to figure out how an HTTP/2 enabled go client would use this endpoint, and found this Github issue. Brad Fitzpatrick suggests something similar to the following code. It is pretty “low level”, so I added explanations in comments.

const url = "https://http2.golang.org/ECHO"

func main() {
    // Create a pipe - an object that implements `io.Reader` and `io.Writer`. 
    // Whatever is written to the writer part will be read by the reader part.
	pr, pw := io.Pipe()
	
    // Create an `http.Request` and set its body as the reader part of the
    // pipe - after sending the request, whatever will be written to the pipe,
    // will be sent as the request body.
    // This makes the request content dynamic, so we don't need to define it
    // before sending the request.
	req, err := http.NewRequest(http.MethodPut, url, ioutil.NopCloser(pr))
	if err != nil {
		log.Fatal(err)
	}
	
    // Send the request
	resp, err := http.DefaultClient.Do(req)
	if err != nil {
		log.Fatal(err)
	}
	log.Printf("Got: %d", resp.StatusCode)
	
	// Run a loop which writes every second to the writer part of the pipe
	// the current time.
	go func() {
		for {   
			time.Sleep(1 * time.Second)
			fmt.Fprintf(pw, "It is now %v\n", time.Now())
		}
	}()
	
    // Copy the server's response to stdout.
	_, err = io.Copy(os.Stdout, res.Body)
	log.Fatal(err)
}

The example is pretty interesting. We create a request with a “dynamic” body - not something straight forward for the average Go programmer. Then, we receive a “dynamic” response body - this is more “normal”, yet not too common. In HTTP/1.1 those kind of requests and responses would be used to send or receive stream of data. But here, the response stream starts before the request stream is finished. After each time we send data through the pipe to the request, data is returned from the server in the response body. That’s great - we just got full duplex communication between two Go processes over an HTTP/2 connection.

The down side in this example is the API - the standard library provides us with powerful tools, but low level knowledge is needed in order to use them.

Full-duplex Communication with posener/h2conn

h2conn is a tiny library that is supposed to improve the user experience of HTTP/2 full duplex communication.

For example, the above Go echo example client could be written as the following (view on github):

const url = "https://http2.golang.org/ECHO"

func main() {
	// Create a client, that uses the HTTP PUT method.
	c := h2conn.Client{Method: http.MethodPut}
	
	// Connect to the HTTP/2 server
	// The returned conn can be used to:
	//   1. Write - send data to the server.
	//   2. Read - receive data from the server.
	conn, resp, err := c.Connect(context.Background(), url)
	if err != nil {
		log.Fatal(err)
	}
	defer conn.Close()
	log.Printf("Got: %d", resp.StatusCode)
	
	// Send time periodically to the server
	go func() {
		for {
			time.Sleep(1 * time.Second)
			fmt.Fprintf(conn, "It is now %v\n", time.Now())
		}
	}()
	
	// Read responses from the server to the stdout.
	_, err = io.Copy(os.Stdout, conn)
	if err != nil {
		log.Fatal(err)
	}
}

Server code can also be simplified. The following handler implements the same echo server from above (view on github):

func echo(w http.ResponseWriter, r *http.Request) {
	// Accept returns a connection to the client  that can be used:
	//   1. Write - send data to the client
	//   2. Read - receive data from the client
	conn, err := h2conn.Accept(w, r)
	if err != nil {
		log.Printf(
			"Failed creating connection from %s: %s",
			r.RemoteAddr, err)
		http.Error(w,
			http.StatusText(http.StatusInternalServerError),
			http.StatusInternalServerError)
		return
	}
	defer conn.Close()

	// Send back to the client everything that we receive
	io.Copy(conn, conn)
}

For more examples, go to the example page in the github repo.

Recap

Go enables HTTP/2 connection with server push and full-duplex communication, which also supports HTTP/1.1 connection with the standard library’s standard TLS server - that’s amazing. As for the standard library HTTP client, it does not support server push, but supports full-duplex communication with the standard library’s standard http.Client Here I introduced a tool that makes full-duplex communication easier, which is available in the posener/h2conn package.