Writing a hypermedia app in Go
Table of Contents
This is the second post in a series:
- Writing safe hypermedia apps for a resource-constrained world, which introduces the series
- this post
- Writing a hypermedia app in Rust
Overview of the app #
The example application in Hypermedia Systems is a simple CRUD application that manages a list of contacts. It illustrates how a Hypermedia-Driven Application can provide a good user interface as an alternative to single-page applications. With a tiny JavaScript library (htmx) and some declarative attributes on HTML elements, all sorts of user interactions can fetch fragments of HTML from the server and modify only certain parts of the page. For example, as we page through the list of contacts, or delete a contact from the list, the displayed list is updated without reloading the page.
Let’s implement this app in Go, with two objectives:
- Make it more realistic by adding an internationalised UI, a relational database with a full-text search index, better input validation, cursor-based pagination, and UI tests.
- Make it easy to maintain, with a minimum of boilerplate, and an emphasis on compile-time checks to prevent bugs.
We’ll start with a SQLite database, and switch to PostgreSQL in a later step. You can find the complete source code of both versions of the app here:
Libraries #
Our first task is to choose some libraries.
HTML templates #
Our user interface is based on HTML templates. There’s a layout template that provides a header and shows informational messages (like ‘Contact added’) in response to user actions. Other templates, such as a form for editing a contact, can be embedded in this layout. There are also templates for fragments of HTML that replace one or more parts of the currently displayed page. For example, when you click on a pagination link, the list of contacts changes, along with the pagination links.
So we’ll need a template engine. Go’s standard library has
html/template, but its templates aren’t type-safe. There are
basically two kinds of type-safe template engines: the kind that lets you write something close
to standard HTML, and the kind that lets you express the structure of an HTML document in a
programming language. I prefer the first kind, because the closer the template syntax is to
HTML, the easier it will be for a UX designer, or anyone who knows HTML, to work on the
templates without having to know Go. I’ve chosen templ, which is easy to
use, compiles templates to Go code, and has good editor support. To use it, we write templates
as functions whose bodies are HTML, and run templ generate to compile them to Go functions. We
can call these functions from our HTTP endpoint implementations.
Internationalisation #
We’d like our app to be available in multiple languages; in this example, I’ve implemented support for English and French. To facilitate maintenance, we want to avoid having different versions of the same template for different languages, or cluttering up the templates with different translations of the same messages. So we need a library that allows us to call a function from a template and get the translation of a message in the target language.
One important consideration is what the library’s designers chose to use as a message identifier. Some libraries (like gettext) use the whole message text in the default language as the message identifier. The limitations of this approach are well expressed in the social contract of Project Fluent:
First of all, it means that any change to the source string invalidates all translations of the string. This severely increases the burden on the developers to never alter messages in the source language as it results in all translations having to be updated.
Secondly, it makes it harder to introduce multiple messages with the same source string which should be translated differently.
Therefore, I prefer libraries that use unique message identifiers chosen by developers.
Another important criterion in choosing a library is how it handles plurals. While plurals are relatively simple in English (there are just two categories, singular and plural), many languages have more categories, along with syntactic rules that require a sentence to be adjusted in more complex ways depending on the number expressed. A good internationalisation library should implement the Unicode CLDR plural rules, which support over 200 languages.
I was interested in trying Project Fluent, but it looks as if that project may have been abandoned. So I’ve chosen go-18n, which has the features we need. To avoid clutter, I’ve written a function like this for each message, providing the text in English, the default language:
|
|
We can call this function directly from a template. The goi18n command-line program extracts
the messages from our source code into text files, where we can then translate them. At run
time, we can access a message translation via a Localizer for the target language; we’ll see
below how we get the correct one.
Since we’re internationalising the app, we can also improve the cross-cultural validity of our field names: instead of ‘first name’ and ‘last name’, which can lead to misunderstandings (in Chinese names, the family name comes first), we’ll follow the recommendation in the FOAF ontology and use ‘given name’ and ‘family name’.
Database access and schema migration #
Go’s standard library provides the database/sql package, but
it requires you to write your own mappings between query results and Go structs. Maintaining
these mappings is cumbersome, and it’s easy to make mistakes that won’t be caught at compile
time. We could use an object-relational mapping
(ORM) library, but I think
it’s better to write our SQL ourselves. ORMs can produce inefficient queries.1 Writing our
own SQL allows us to take full advantage of the features of modern SQL
and its opportunities for query optimisation, particularly by
designing queries and indexes together, and by refining queries in light of the execution plans
of the database’s query optimiser. (See my post Optimising PostgreSQL queries with an open
dataset for an example.)
I think the ideal is to have a library that provides type safety for SQL queries, so queries are
checked at compile time and automatically return Go structs. The
sqlc package does just this (see its introductory blog
post for a more detailed explanation). To use sqlc, we put
our SQL queries in text files. When we run sqlc generate on the command line, sqlc connects to
the database to analyse our queries, and generates a Go package with a type-safe function
corresponding to each query.
For example, suppose we misspell a column name (familyname instead of family_name) in an SQL
query:
|
|
We’ll get an error from sqlc generate:
|
|
Here we’ll also have sqlc use prepared statements to prevent SQL injection attacks.
Database schemas change over time, so we need a tool that manages schema migrations. There are several good options, and here I’ve chosen goose.
HTTP server framework #
Go’s standard library has net/http for writing HTTP servers, but I’ve chosen the Gin framework, which implements type-safe mappings between requests and Go structs, as well as flexible input validation via the validator package. And we can easily integrate templ into Gin.
UI testing #
If we were using the SPA approach, our server would have a JSON API, and we could write tests that check whether it returns the correct data for various requests. But our application returns HTML. So we’ll write tests that are analogous to JSON API tests, focusing on the functionality of our HTML responses. Using the goquery package, we’ll parse the HTML responses and use CSS selectors to check the contents of specific elements. This allows the graphic design to change without breaking our tests.
Database design #
(If you’ve already read the Rust version of this post, you can skip this section.)
Our database is very simple. There’s just one table, contact:
|
|
But there are a few details to think about.
I’ve defined the contact table as STRICT. By
default, SQLite has a very flexible type
system, which some people like. I prefer
to get errors when there’s a bug in my program. With the STRICT option, SQLite does strict
type checking at run time, in addition to the compile-time checks that sqlc provides.
Nullable columns add a bit of complexity to our code, so to keep things simple for this example,
we’ll allow only the phone column to be null.
The app should display the contact list sorted in some order; we’ll use ORDER BY family_name, given_name, email. To make this efficient, we’ll add an index with that sort order:
|
|
Since the app requires email addresses to be unique, I’ve put a UNIQUE constraint on the
email field, and we’ll need a query like this:
|
|
To make this query efficient, we’ll put an index on the email column:
|
|
We want a full-text search index that allows us to find contacts whose given name or family name
matches a string. We’ll let the user type the first letters of given_name and/or family_name
and get a list of matching contacts. Fortunately, SQLite has a full-text search module, FTS5. To
use it, we create a virtual table to store the index:
|
|
We’ll use triggers to keep the table
contact_fts up to date with the table contact. And that completes our database
schema.
Application architecture #
For this simple CRUD app, a
model-view-controller
architecture is sufficient. The views are HTML templates, and the controllers are HTTP
routes. The main type in the model is Contact:
|
|
The database abstraction layer is just a struct called Database that contains the connection
pool and has methods like this:
|
|
Input validation #
What should happen when the user submits a form to create a contact? We’ll have to validate all
the fields, and be able to collect multiple validation errors and report them all back to the
user. The form has a <span class="error"> under each input field for displaying these errors.
Gin allows us to bind the form fields to the fields of a struct, call a validation function for
each field, and get a slice of validation errors. Since a contact that hasn’t been saved yet
doesn’t have an ID, let’s make another struct for this:
|
|
The form tag specifies the form field to be bound to a struct field, and binding lists the
validators to be run for the field. We can use the
validator package’s built-in validators as well as
our own. Here I’ve made custom validators called name, phone, and email, which use
Unicode regular expressions. Since this is an
internationalised application, we want to support:
- all the Unicode characters that could appear in a person’s name
- any phone number that could be used in any country
- email address internationalisation
For example, my simple attempt at a name regex accepts groups of Unicode letters and marks, separated by spaces and/or punctuation:
|
|
So if your given name is 曼玉-فاتن, all is well. We’ll make a custom validator function using this regex:
|
|
When the app starts, we’ll register the function with the validator package:
|
|
Error handling #
Our error handling strategy distinguishes between errors that are the client’s fault and errors that are the server’s fault:
|
|
If an error is the client’s fault, we’ll just report it to the user, e.g. by displaying ‘Page
not found’. If it’s the server’s fault (e.g. it’s a DatabaseError), we’ll display ‘Internal
server error’ and log the error details.
Routes and languages #
Before we define HTTP endpoints, we should decide how the app will know which language to use.
The Accept-Language request header allows the browser to provide a ranked list of the
languages that the user prefers, but we also have to allow the user to select a
language
just for this app:
This header serves as a hint when the server cannot determine the target content language otherwise (for example, use a specific URL that depends on an explicit user decision). The server should never override an explicit user language choice. The content of
Accept-Languageis often out of a user’s control (when traveling, for instance). A user may also want to visit a page in a language different from the user interface language.
Let’s put a language switcher in the page header, and put a BCP 47 language
tag at the beginning of every endpoint’s path.
For the root path, we can use the Accept-Language header, and redirect to a path containing a
language tag.
With the go-i18n package, we initialise an i18n.Bundle on startup; it contains the message
keys and translations. This bundle will be part of the application’s shared state, along with
the Database. This is our AppState struct:
|
|
If we implement our routes as methods with *AppState as their receiver, we can set up a Gin
router like this:
|
|
We can use Gin’s binding and validation to validate the :lang path parameter as well. I’ve
written a little function called GetLocalizer, which returns the Localizer for the language
we got from the path, if any, and falls back to using the Accept-Language header.
Let’s not make one of those annoying websites whose language switcher takes you back to the home
page. We can write a function, URLsWithLangs, that takes the current URL path, adjusts it for
every available language, and returns a slice of the resulting paths. In the page header, we
display those URLs as links.
Now, if we write a template called Show that takes a Contact as a parameter, our
contactGet route will look like this:
|
|
Pagination #
(If you’ve already read the Rust version of this post, you can skip this section and the next.)
The pagination in Hypermedia Systems is based on page
numbers,
which is inefficient (the database always has to retrieve all the pages preceding the one you
asked for) and can produce unexpected results if the data is being modified while you’re paging
through it. A better approach is cursor-based
pagination.
When returning a page of results, the database layer can return two cursors, called prev and
next, which the view can present as links to the previous and next pages. The ‘Previous page’
link contains the prev cursor as the parameter before, and the Next page link contains the
next cursor as the parameter after. There are a few tricks involved in making this work.
Our cursor will contain the values of the columns family_name, given_name, and email,
which are the same ones we use to sort contacts. Since we already have an index on those three
columns, in that order, the database can efficiently retrieve a page of rows before or after a
cursor.
We’d like to avoid returning a prev cursor if we’ve reached the first page, or a next cursor
if we’ve reached the last one. The solution is as follows: in a ‘page after’ request, if our
maximum page size is N, we request N + 1 rows. If we get them, we know a next page exists, so we
return a next cursor based on the last row in the returned page. If we were given an after
cursor in the request, we can assume there’s a previous page, so the prev cursor corresponds
to the first row in the returned page.
Our SQL queries for getting a page of contacts have to take into account the fact that names aren’t necessarily unique. So our ‘page after’ query looks like this (with a Common Table Expression so we don’t have to pass the same parameters more than once):
|
|
For ‘page before’ queries, it’s the same thing but in reverse order.
Full-text search queries #
There isn’t much left to do for full-text search. The /:lang/contacts endpoint receives a
query string consisting of one or more words or word prefixes. In the database layer, we parse
this string and adapt it to the FTS5 query syntax. For example, G O'Mal becomes "G"* + "O'Mal"*, and will match a contact whose name is Grace O’Malley. We return a maximum of one
page of full-text search results, so the route passes nil cursors to the template, which
removes the pagination links via an htmx out-of-band
swap.
Integration tests #
There’s nothing remarkable about this application’s unit tests (unless your name happens to be
曼玉-فاتن), but the integration tests are more interesting. I’ve written two sets of integration
tests: one for the database layer, and one for the HTTP endpoints. In both cases, each test runs
with a freshly initialised SQLite database containing test data from a fixture. As mentioned
above, the HTTP endpoint tests parse the HTML responses and check the contents of specific
elements. We use a Gin router in test mode, and
net/http/httptest records the router’s responses, which
we can parse using goquery. For example, to test a
GET request to /:lang/contacts/:id/edit, the endpoint for editing an existing contact, we
can do something like this:
|
|
Since we’re just looking at the text in labels and form fields using simple CSS selectors, tests like this are unlikely to be affected by changes in the graphic design of the page.
PostgreSQL #
We don’t need to do much to adapt our app to use PostgreSQL instead of SQLite.
Full-text search #
(If you’ve already read the Rust version of this post, you can skip this section.)
Like SQLite, Postgres has built-in full-text
search, and it’s even easier to set
up. We can store tsvector values (preprocessed documents) in a generated
column
in the contact table, with a GIN index to
speed up queries:
|
|
The SQL query to find matching rows uses the full-text match operator, @@:
|
|
The syntax of the string that we pass to to_tsquery just needs minor adjustments to work with
Postgres: if the user types G O'Mal, we pass the argument 'G':* & 'O''Mal':*.
Integration tests with Testcontainers #
Running our integration tests with Postgres takes a bit more work. To run each test with a freshly initialised database as before, we can use Testcontainers and run Postgres in an ephemeral Docker container. We’ll run our integration tests synchronously, so we only need one database per test run. The Testcontainers postgres module has a useful feature for this: we can initialise the database with our migrations and our fixture, then save a snapshot of the database. After each test completes, we restore the database to the snapshot so it’s ready for the next test. The code for this is too long to quote here, but you’ll find it in integration/common.go.
Conclusion #
By using a declarative validation framework, and deterministic code generation in a statically typed language, we’ve kept boilerplate to a minimum, and improved the chances that if we make a mistake, we’ll get an error at compile time rather than at run time. This should help keep our code maintainable and prevent bugs.
Next, let’s write the same app in Rust.