Rust on Benjamin Geer

Writing a hypermedia app in Rust

Mon, 05 Oct 2026 07:02:00 +0000

This is the third post in a series:

Writing safe hypermedia apps for a resource-constrained world, which introduces the series
Writing a hypermedia app in Go
this post

Overview of the app #

The example application in Hypermedia Systems is a simple CRUD application that manages a list of contacts. It illustrates how a Hypermedia-Driven Application can provide a good user interface as an alternative to single-page applications. With a tiny JavaScript library and some declarative attributes on HTML elements, all sorts of user interactions can fetch fragments of HTML from the server and update only certain parts of the page. For example, as we page through the list of contacts, or delete a contact from the list, the list changes without reloading the page.

Let’s implement this app in Rust, with two objectives:

Make it more realistic by adding an internationalised UI, a relational database with a full-text search index, better input validation, cursor-based pagination, and UI tests.
Make it easy to maintain, with a minimum of boilerplate, and an emphasis on compile-time checks to prevent bugs.

We’ll start with a SQLite database, and switch to PostgreSQL in a later step. You can find the complete source code of both versions of the app here:

Libraries #

Our first task is to choose some libraries.

HTML templates #

Our user interface is based on HTML templates. There’s a layout template that provides a header and shows informational messages (like ‘Contact added’) in response to user actions. Other templates, such as a form for editing a contact, can be embedded in this layout. There are also templates for fragments of HTML that replace one or more parts of the currently displayed page. For example, when you click on a pagination link, the list of contacts changes, along with the pagination links.

So we’ll need a template engine, preferably a type-safe one. There are basically two kinds of type-safe template engines: the kind that lets you write something close to standard HTML, and the kind that lets you express the structure of an HTML document in a programming language. I prefer the first kind, because the closer the template syntax is to HTML, the easier it will be for a UX designer, or anyone who knows HTML, to work on the templates without having to know Rust.

I’ve chosen Askama, which is easy to use, and compiles templates to Rust code. Since its syntax is based on Jinja, editor tooling that works with Jinja will also work with Askama. To use it, we write our templates in HTML files; for each template, we write a Rust struct definition that contains the template’s variables and refers to the HTML file via a macro. In our HTTP endpoint implementations, we call that struct’s render method to generate HTML.

Internationalisation #

We’d like our app to be available in multiple languages; in this example, I’ve implemented support for English and French. To facilitate maintenance, we want to avoid having different versions of the same template for different languages, or cluttering up the templates with different translations of the same messages. So we need a library that allows us to call a function from a template and get the translation of a message in the target language.

One important consideration is what the library’s designers chose to use as a message identifier. Some libraries (like gettext) use the whole message text in the default language as the message identifier. The limitations of this approach are well expressed in the social contract of Project Fluent:

First of all, it means that any change to the source string invalidates all translations of the string. This severely increases the burden on the developers to never alter messages in the source language as it results in all translations having to be updated.

Secondly, it makes it harder to introduce multiple messages with the same source string which should be translated differently.

Therefore, I prefer libraries that use unique message identifiers chosen by developers.

Another important criterion in choosing a library is how it handles plurals. While plurals are relatively simple in English (there are just two categories, singular and plural), many languages have more categories, along with syntactic rules that require a sentence to be adjusted in more complex ways depending on the number expressed. A good internationalisation library should implement the Unicode CLDR plural rules, which support over 200 languages.

I was interested in trying Project Fluent, but it looks as if that project may have been abandoned. So I’ve chosen karatepe, which has the features we need. Karatepe provides a simple DSL for defining messages and writing translations, and a macro that compiles message definitions into Rust functions. For example, here’s a message definition that takes a numerical parameter amount:

`1`	`contacts_count(amount: ..)`

And here’s its English translation:

contacts_count(amount) = ({amount} total {amount => {
    one => contact
    _ => contacts
  }})

The message definition is compiled to a Rust function, contacts_count, which we can call directly from a template. At run time, we can access a message translation via a Translation for the target language; we’ll see below how we get the correct one.

Since we’re internationalising the app, we can also improve the cross-cultural validity of our field names: instead of ‘first name’ and ‘last name’, which can lead to misunderstandings (in Chinese names, the family name comes first), we’ll follow the recommendation in the FOAF ontology and use ‘given name’ and ‘family name’.

Database access and schema migration #

We could use an object-relational mapping (ORM) library for database access, but I think it’s better to write our SQL ourselves. ORMs can produce inefficient queries.¹ Writing our own SQL allows us to take full advantage of the features of modern SQL and its opportunities for query optimisation, particularly by designing queries and indexes together, and by refining queries in light of the execution plans of the database’s query optimiser. (See my post Optimising PostgreSQL queries with an open dataset for an example.)

I think the ideal is to have a library that provides type safety for SQL queries, so queries are checked at compile time and automatically return Rust structs. The SQLx crate does just this, and can return our own structs by deriving its FromRow trait.

For example, suppose we misspell a column name (familyname instead of family_name) in an SQL query:

let contact: Contact = sqlx::query_as!(
    Contact,
    "SELECT id, familyname, given_name, phone, email
    FROM contact WHERE id = $1",
    id
)
.fetch_one(&mut *tx)
.await?;

By default, the query_as! macro will connect to the development database at compile time to check the query, and we’ll get a compile error:

1
2

error: error returned from database: (code: 1) no such column: familyname
   --> src/db.rs:236:23

SQLx also uses prepared statements to prevent SQL injection attacks, and provides a command-line tool for managing database schema migrations.

HTTP server framework #

We’ll use axum, which conveniently implements type-safe mappings between requests and Rust structs via Serde, and is easy to use with Askama.

UI testing #

If we were using the SPA approach, our server would have a JSON API, and we could write tests that check whether it returns the correct JSON for various requests. But our application returns HTML. So we’ll write tests that are analogous to JSON API tests, focusing on the functionality of our HTML responses. Using the dom_query crate, we’ll parse the HTML responses and use CSS selectors to check the contents of specific elements. This allows the graphic design to change without breaking our tests.

Database design #

(If you’ve already read the Go version of this post, you can skip this section.)

Our database is very simple. There’s just one table, contact:

CREATE TABLE IF NOT EXISTS contact (
  id INTEGER PRIMARY KEY,
  family_name TEXT NOT NULL,
  given_name TEXT NOT NULL,
  phone TEXT,
  email TEXT UNIQUE NOT NULL
) STRICT;

But there are a few details to think about.

I’ve defined the contact table as STRICT. By default, SQLite has a very flexible type system, which some people like. I prefer to get errors when there’s a bug in my program. With the STRICT option, SQLite will do strict type checking at run time, in addition to the compile-time checks that sqlc provides.

Nullable columns add a bit of complexity to our code, so to keep things simple for this example, we’ll allow only the phone column to be null.

The app should display the contact list sorted in some order; we’ll use ORDER BY family_name, given_name, email. To make this efficient, we’ll add an index with that sort order:

1
2

CREATE INDEX IF NOT EXISTS idx_contact_sort
ON contact (family_name, given_name, email);

Since the app requires email addresses to be unique, we have a UNIQUE constraint on the email field, and we’ll need a query like this:

`1`	`SELECT 1 FROM contact WHERE email = ?;`

To make this query efficient, we’ll put an index on the email column:

`1`	`CREATE INDEX IF NOT EXISTS idx_contact_email ON contact (email);`

We want a full-text search index that allows us to find contacts whose name matches a string. We’ll let the user type the first letters of given_name and/or family_name and get a list of matching contacts. Fortunately, SQLite has a full-text search module, FTS5. To use it, we create a virtual table to store the index:

CREATE VIRTUAL TABLE IF NOT EXISTS contact_fts USING fts5(
    family_name,
    given_name,
    phone UNINDEXED,
    email UNINDEXED,
    content = 'contact',
    content_rowid = 'id'
);

We’ll use database triggers to keep the FTS5 table up to date with the content table. And that completes our database schema.

Application architecture #

For this simple CRUD app, a model-view-controller architecture is sufficient. The views are HTML templates, and the controllers are HTTP routes. The main type in the model is a struct called Contact, which we’ll get to in a moment. The database abstraction layer is just a struct called Database that contains the connection pool and has methods like this:

1
2
3

pub async fn get(&self, id: ContactId) -> Result<Contact, DatabaseError> {
    // ...
}

Parsing, not validating #

We’ll apply Alexis King’s principle, Parse, don’t validate. In short, instead of representing, say, an email address as a &str, we use the newtype pattern and create a type for it, called Email, which can only be constructed from valid input. This has several beneficial effects:

All the functions in the database layer only accept newtypes and structs composed of newtypes, so it’s impossible for them to receive invalid values.
Since it’s impossible to construct an invalid value, we never have to wonder whether a value has already been validated, or risk validating it twice.
Having to pass an &Email to a function instead of a &str can prevent certain mistakes, such as using the wrong variable in a function call, or getting the order of the parameters wrong.
Function parameters become clearer: after: &Cursor tells us more than after: &str.

We’ll use the nutype crate to define our newtypes. It offers a variety of built-in validation functions, e.g. for validating with a regular expression. Since this is an internationalised application, we’ll use Unicode regular expressions to support:

all the Unicode characters that could appear in a person’s name
any phone number that could be used in any country
email address internationalisation

For example, my simple attempt at a name regex accepts groups of Unicode letters and marks, separated by spaces and/or punctuation:

`1`	`^[\p{L}\p{M}]+(?:(?:\p{P}\p{Zs}\|[\p{P}\p{Zs}])[\p{L}\p{M}]+)*\p{P}?$`

So if your given name is 曼玉-فاتن, all is well.

Now we can define a newtype:

#[nutype(
    validate(regex = NAME_REGEX, len_char_max = 50),
    derive(Debug, Display, PartialEq, Clone, AsRef, Serialize),
    derive_unchecked(sqlx::Type),
)]
#[sqlx(transparent)]
pub struct GivenName(String);

This generates a constructor, GivenName::try_new, which takes a string and returns a Result<GivenName, GivenNameError>. We’ve also specified that we want SQLx to be able to construct a GivenName without validation, because we assume that the data in the database has already been validated. We also have the option of not trusting the database, and parsing GivenName in the normal way in the database layer.

After defining a few other newtypes in the same way, we can define Contact:

#[derive(Debug, PartialEq, sqlx::FromRow)]
pub struct Contact {
    pub id: ContactId,
    pub family_name: FamilyName,
    pub given_name: GivenName,
    pub phone: Option<Phone>,
    pub email: Email,
}

When the user is creating a new contact, it doesn’t have an ID yet, so we also define ContactContent:

#[derive(Debug, PartialEq, Serialize)]
pub struct ContactContent {
    pub family_name: FamilyName,
    pub given_name: GivenName,
    pub phone: Option<Phone>,
    pub email: Email,
}

What should happen when the user submits a form to create a contact? We don’t want axum to deserialise the form data directly to a ContactContent, because that would stop parsing at the first error. Instead, we want to collect multiple validation errors and report them all back to the user. The form has a <span class="error"> under each input field for displaying these errors. We can represent the unparsed form input as a struct like this:

use serde_with::{NoneAsEmptyString, serde_as};

#[serde_as]
#[derive(Debug, Default, Deserialize)]
pub struct ContactForm {
    pub family_name: String,
    pub given_name: String,

    #[serde_as(as = "NoneAsEmptyString")]
    pub phone: Option<String>,

    pub email: String,
}

We’d like to parse a ContactForm and get either a ContactContent or a collection of errors. There isn’t yet a way to automate this completely, but we can write a short function to do it using the frunk crate of functional programming tools. We’ll use its Validated type, which collects the results of several operations. If there are no errors, Validated::into_result produces an HList, a statically typed list whose elements can have different types. Given an HList, we can easily construct our ContactContent. If any errors occurred, we get them in a Vec.

use frunk::prelude::*;
use frunk_core::hlist_pat;

impl ContactForm {
    pub fn parse(&self) -> Result<ContactContent, Vec<ParseError>> {
        let validated = FamilyName::try_new(&self.family_name)
            .map_err(ParseError::from)
            .into_validated()
            + GivenName::try_new(&self.given_name).map_err(ParseError::from)
            + self.phone.as_ref()
                .map(|phone| Phone::try_new(phone)
                    .map_err(ParseError::from))
                .transpose()
            + Email::try_new(&self.email).map_err(ParseError::from);

        validated
            .into_result()
            .map(
                |hlist_pat!(family_name, given_name, phone, email)|
                    ContactContent {
                        family_name,
                        given_name,
                        phone,
                        email,
                    },
            )
    }
}

Our ParseError type contains the original error (GivenNameError, etc.), so we can easily determine which fields had errors. We can then pass the ContactForm and a collection of translated error messages to the template.

If validation was successful, we can pass the ContactContent to Database::add, which looks like this:

pub async fn add(&self, contact: &ContactContent) -> Result<(), Error> {
    let mut tx = self.pool.begin().await?;

    sqlx::query!(
        r#"INSERT INTO contact (family_name, given_name, phone, email)
            VALUES ($1, $2, $3, $4)"#,
        &contact.family_name,
        &contact.given_name,
        &contact.phone,
        &contact.email
    )
    .execute(&mut *tx)
    .await?;

    tx.commit().await?;
    Ok(())
}

Error handling #

Whose fault is it? #

Our error handling strategy distinguishes between errors that are the client’s fault and ones that are the server’s fault. We can use the thiserror crate to simplify wrapping errors in other errors:

use thiserror::Error;

#[derive(Error, Debug)]
pub enum Error {
    #[error("Client error: {0}")]
    Client(#[from] ClientError),

    #[error("Server error: {0}")]
    Server(ServerError),
}

#[derive(Error, Debug)]
pub enum ClientError {
    #[error("Page not found")]
    NotFound,

    // ...
}

#[derive(Error, Debug)]
pub enum ServerError {
    #[error("Database error: {0}")]
    Database(#[from] DatabaseError),

    // ...
}

#[derive(Error, Debug)]
pub enum DatabaseError {
    #[error("{0}")]
    Sqlx(#[from] sqlx::Error),
}

impl<E> From<E> for Error
where
    E: Into<ServerError>,
{
    fn from(err: E) -> Self {
        Error::Server(err.into())
    }
}

If an error is the client’s fault, we’ll just report it to the user, e.g. by displaying ‘Page not found’. If it’s the server’s fault (e.g. it’s a DatabaseError), we’ll display ‘Internal server error’ and log the error details.

Centralising error handling #

I like to centralise error handling so it doesn’t clutter up the rest of the code. With axum, a route handler returns a Result whose error type is anything that implements the IntoResponse trait. Therefore, with a bit of plumbing, we can use the ? operator in our route handler functions to log the error if necessary and return an error page using our error.html template. We make an error type called ErrorResponse that just holds an HTTP status code and some HTML, and we implement IntoResponse for it. The we write a method on our Error type that produces an ErrorResponse using the template, with a translated error message, and logs the error if it’s the server’s fault. This method has to take a &Translation parameter so the template can get the error message in the target language:

impl Error {
    pub fn to_error_response(self, t: &Translation) -> ErrorResponse {
        // ...
    }
}

Finally, we write an extension trait for Result to convert any error into an ErrorResponse:

pub trait ResultErrorTranslationExt<T, E>
where
    E: std::error::Error,
    Error: From<E>,
{
    fn tr_err(self, t: &Translation) -> Result<T, ErrorResponse>;
}

impl<T, E> ResultErrorTranslationExt<T, E> for Result<T, E>
where
    E: std::error::Error,
    Error: From<E>,
{
    fn tr_err(self, t: &Translation) -> Result<T, ErrorResponse> {
        self.map_err(|err| Error::from(err).to_error_response(t))
    }
}

Now we can write route handler functions that return Result<Html<String>, ErrorResponse> and handle errors very concisely. For example, if the db.get method can return DatabaseError, we can call it like this:

1
2

let t: &Translation = // ...
let contact = app.db.get(id).await.tr_err(t)?;

Routes and languages #

Before we define HTTP endpoints, we should decide how the app will know which language to use. The Accept-Language request header allows the browser to provide a ranked list of the languages that the user prefers, but we also have to allow the user to select a language just for this app:

This header serves as a hint when the server cannot determine the target content language otherwise (for example, use a specific URL that depends on an explicit user decision). The server should never override an explicit user language choice. The content of Accept-Language is often out of a user’s control (when traveling, for instance). A user may also want to visit a page in a language different from the user interface language.

Let’s put a language switcher in the page header, and put a BCP 47 language tag at the beginning of every endpoint’s path. For the root path, we can use the Accept-Language header, and redirect to a path containing a language tag.

With the karatepe crate, we load a Translation for each language on startup. We’ll keep these in a HashMap (with icu::locale::LanguageIdentifier as the key type) in the application’s shared state, along with the Database. This is our AppState struct:

pub struct AppState {
    pub db: Database,
    pub translations: Arc<HashMap<LanguageIdentifier, Translation>>,
}

axum’s Router can hold onto this state for us so the route handler functions can access it. We set it up like this:

pub fn init_router(app: AppState) -> Router {
    Router::new()
        .route("/", get(index)) // checks Accept-Language
        .route(
            "/{lang}/contacts",
            get(contacts::contacts_get).delete(contacts::contacts_delete),
        )
        .route(
            "/{lang}/contacts/{id}",
            get(contacts::contact_get).delete(contacts::contact_delete),
        )
        // ...
        .with_state(app)
}

axum provides extractors that handler functions can use to extract and parse data in the request. We use the Path extractor to parse the language code as a LanguageIdentifier (which implements serde::Deserialize), so we don’t need to make our own newtype for this. Then we just need a little function called translation that takes a LanguageIdentifier and returns the corresponding Translation from the AppState.

Let’s not make one of those annoying websites whose language switcher takes you back to the home page. We can write a function, uris_with_langs, that takes the current URL path, adjusts it for every available language, and returns a Vec of the resulting paths. In the page header, we display those URLs as links.

Our template for displaying a contact is called show.html, and its variables are in struct ShowContactTemplate. So our contact_get route now looks like this:

pub async fn contact_get(
    State(app): State<AppState>,
    Path((lang, id)): Path<(LanguageIdentifier, ContactId)>,
    uri: Uri,
) -> Result<Html<String>, ErrorResponse> {
    let t = app.translation(&lang);
    let contact = app.db.get(id).await.tr_err(t)?;

    let template = ShowContactTemplate {
        t,
        lang,
        lang_uris: localisation::uris_with_langs(uri).tr_err(t)?,
        contact,
    };

    Ok(Html(template.render().tr_err(t)?))
}

Pagination #

(If you’ve already read the Go version of this post, you can skip this section and the next.)

The pagination in Hypermedia Systems is based on page numbers, which is inefficient (the database always has to retrieve all the pages preceding the one you asked for) and can produce unexpected results if the data is being modified while you’re paging through it. A better approach is cursor-based pagination. When returning a page of results, the database layer can return two cursors, called prev and next, which the view can present as links to the previous and next pages. There are a few tricks involved in making this work.

Our cursor will contain the values of the columns family_name, given_name, and email, which are the same ones we use to sort contacts. Since we already have an index on those three columns, in that order, the database can efficiently retrieve a page of rows before or after a cursor.

We’d like to avoid returning a prev cursor if we’ve reached the first page, or a next cursor if we’ve reached the last one. The solution is as follows: in a ‘page after’ request, if our maximum page size is N, we request N + 1 rows. If we get that many rows, we know a next page exists, so we return a next cursor based on the last row in the returned page. If we were given a next cursor in the request, we can assume there’s a previous page, so the first row in the returned page is the prev cursor.

Our SQL queries for getting a page of contacts have to take into account the fact that given names and family names aren’t necessarily unique, although email addresses are. So our ‘page after’ query looks like this (with a Common Table Expression so we don’t have to pass the same parameters more than once):

WITH vars AS (SELECT ? AS family_name, ? AS given_name, ? AS email)
SELECT id, family_name, given_name, phone, email
FROM contact
WHERE (family_name = (SELECT family_name FROM vars)
       AND given_name = (SELECT given_name FROM vars)
       AND email > (SELECT email FROM vars))
OR (family_name = (SELECT family_name FROM vars)
    AND given_name > (SELECT given_name FROM vars))
OR family_name > (SELECT family_name FROM vars)
ORDER BY family_name, given_name, email
LIMIT ?;

Full-text search queries #

There isn’t much left to do for full-text search. The /{lang}/contacts endpoint receives a query string consisting of one or more words or word prefixes. We parse this string and adapt it to the FTS5 query syntax. For example, G O'Mal becomes "G"* + "O'Mal"*, and will match a contact whose name is Grace O’Malley. We return a maximum of one page of full-text search results, so the route passes None cursors to the template, which removes the pagination links via an htmx out-of-band swap.

Integration tests #

There’s nothing remarkable about this application’s unit tests (unless your name happens to be 曼玉-فاتن), but the integration tests are more interesting. I’ve written two sets of integration tests: one for the database layer, and one for the HTTP endpoints. In both cases, we use sqlx::test, which creates a new database for each test and initialises it with a fixture, so the tests are isolated from each other. As mentioned above, the HTTP endpoint tests parse the HTML responses and check the contents of specific elements. We use the same axum router as in the application, and parse the responses using dom_query. For example, to test a GET request to /{lang}/contacts/{id}/edit, the endpoint for editing an existing contact, we can do something like this:

#[sqlx::test(fixtures("test-contacts"))]
async fn edit_contact_get(pool: SqlitePool) {
    let server = init_server(pool).await;
    let response = server.get("/en/contacts/1/edit").await;

    assert_eq!(response.status_code(), StatusCode::OK);
    let doc = Document::from(response.text());

    let email_label = doc.select("label[for='email']").text().to_string();
    assert_eq!(email_label, "Email");

    let email = doc.select("#email").attr("value").unwrap().to_string();
    assert_eq!(email, "foo1@bar.com");

    // ...
}

Since we’re just looking at the text in labels and form fields using simple CSS selectors, test like this are unlikely to be affected by changes in the graphic design of the page.

PostgreSQL #

We don’t need to do much to adapt our app to use PostgreSQL instead of SQLite.

Full-text search #

(If you’ve already read the Go version of this post, you can skip this section.)

Like SQLite, Postgres has built-in full-text search, and it’s even easier to set up. We can store tsvector values (preprocessed documents) in a generated column in the contact table, with a GIN index to speed up queries:

CREATE TABLE IF NOT EXISTS contact (
  id BIGINT GENERATED ALWAYS AS IDENTITY PRIMARY KEY,
  family_name TEXT NOT NULL,
  given_name TEXT NOT NULL,
  phone TEXT,
  email TEXT UNIQUE NOT NULL,
  textsearch tsvector GENERATED ALWAYS AS (
    to_tsvector(
      'simple',
      COALESCE(family_name, '') || ' ' || COALESCE(given_name, '')
    )
  ) STORED
);

CREATE INDEX textsearch_idx ON contact USING GIN (textsearch);

The SQL query to find matching rows uses the full-text match operator, @@:

SELECT id, family_name, given_name, phone, email
FROM contact, to_tsquery($1) query
WHERE query @@ textsearch
ORDER BY ts_rank_cd(textsearch, query)
LIMIT $2;

The syntax of the string that we pass to to_tsquery just needs minor adjustments to work with Postgres: if the user types G O'Mal, we pass the argument 'G':* & 'O''Mal':*.

Integration tests with Testcontainers #

We can use Testcontainers to run Postgres in an ephemeral Docker container. #[sqlx:test] runs the tests in parallel, and isolates them from each other by creating a new database in the same container for each test.

There’s a small problem to solve here: #[sqlx:test] generates code that reads the test database URL from an environment variable or an .env file, but we don’t know that URL until after Testcontainers starts the Docker container. The solution, shown in an example project by Matilda Smeds, is to use the linktime crate to define a module initialisation function that runs before the tests do. This function starts the test container and sets the DATABASE_URL environment variable via std::env::set_var.

Conclusion #

By following the principle of ‘Parse, don’t validate’, and by using deterministic code generation that takes advantage of the guarantees provided by Rust’s type system, we’ve kept boilerplate to a minimum, and improved the chances that if we make a mistake, we’ll get an error at compile time rather than at run time. This should help keep our code maintainable and prevent bugs.

See the N+1 selects problem and selecting all columns without thinking about it. ↩︎

Writing safe hypermedia apps for a resource-constrained world

Mon, 05 Oct 2026 07:00:00 +0000

When I was developing web applications in the 1990s, it was common for the server to generate HTML pages using templates, and a lot of template engines were written for this (including one by me). Starting in the early 2000s, Single-Page Applications (SPA) became widespread. In an SPA, the server mainly returns JSON, and a JavaScript (or TypeScript) framework generates the HTML in the browser. I’ve worked on some applications like this, and it struck me that they’re cumbersome to maintain, partly because the JavaScript frameworks are huge and complex and change rapidly.

Nowadays there is increasing dissatisfaction with the SPA approach, not only because of the maintenance burden, but also because these applications have to load megabytes of JavaScript over the network before they can do anything, and this makes them slow, especially on slow network connections. Meanwhile, HTML has become more powerful, making JavaScript less necessary. As a result, there is renewed interest in generating HTML on the server.

One of the drawbacks of generating HTML on the server in the 1990s was the need to reload the whole page whenever anything changed. Now that limitation is gone, thanks to tiny JavaScript libraries like htmx. By adding declarative attributes to HTML elements, we can allow all sorts of user interactions to fetch fragments of HTML from the server and replace only the parts of the page that need to change. Applications that work this way are called Hypermedia-Driven Applications, and there’s a book about them, called Hypermedia Systems.

The book shows how to construct a contact management application using htmx, and the source code is provided in Python. Since the book’s focus is on htmx, the server code is very simple: it only supports English, contacts are stored in a file rather than in a database, there’s very little input validation, pagination uses page numbers, and there are no tests.

In a resource-constrained world, it makes sense to write applications like this in languages that compile to machine code and use relatively little memory.¹ We’d like to have the option of using a very lightweight database like SQLite or a more powerful one like PostgreSQL. To help prevent bugs and security vulnerabilities, we can favour statically typed languages with an emphasis on built-in safety features. And we’d like our application to be easy to maintain.

Given these preferences, let’s make the example app from Hypermedia Systems more realistic. We’ll give it an internationalised UI, a relational database with a full-text search index, better input validation, cursor-based pagination, and UI tests. In the next few posts, I’ll present two implementations, one in Go and one in Rust, with either SQLite or PostgreSQL. I’ll explain the design choices I’ve made in each version, and cover some interesting techniques that have wider applications. I’m not a UX designer, so I won’t try to improve on the user interface described in the book, but I’ll have a few words to say about facilitating the work of UX designers. I won’t cover most of the details that are explained in Hypermedia Applications, but you can read these articles without having read the book. Let’s think about some techniques we can use to write web applications that are safe, resource-efficient, and easy to maintain:

(Each article can be read independently of the other.)

The relationship between energy efficiency and the choice of programming language is an active area of research; see Energy Efficiency of Programming Languages. ↩︎

Comparing the performance of SQL queries

Thu, 29 Jan 2026 18:25:18 +0100

I wrote a little Rust program, sqlstopwatch, to compare the performance of different SQL queries. I was inspired by the article Benchmarking SQL published by the jOOQ project, where they write the test code in SQL. I wanted something similar, but with a few additional requirements:

A command-line program with a simple TUI showing a progress bar.
The program should support PostgreSQL, MySQL, and SQLite.
It should read the SQL queries and test parameters from a configuration file specified on the command line.
It should print the results as a table in the terminal or save them as a CSV file that I can use to generate a chart.

The TUI was easy to make with Ratatui. To interact with the database, I used the async database library sqlx, which provides a bundled SQLite library.

I considered using TOML for the config file, but the file has to specify the database credentials. In testing environments, credentials may have to be loaded from environment variables or from files on a mounted file system, but TOML doesn’t support carrying out those sorts of operations. Moreover, you might want to generate a number of similar queries and compare them, as I did in this database optimisation project. Therefore, it’s better if the config file is a program in a scripting language. My exercise in writing an interpreter for a scripting language had given me a new appreciation for the Lua virtual machine. So the config file is a Lua script, which I find more convenient to write in Fennel, a Lisp dialect that compiles to Lua. Thanks to the mlua library, it was very easy to embed a Lua interpreter in the program, and to use Serde to convert Lua tables into Rust structs.

Writing a Bittorrent client in Rust

Thu, 15 Jan 2026 15:16:01 +0100

Overview #

To continue learning Rust, I decided to implement a client for Bittorrent, the communication protocol for peer-to-peer file sharing. The Bittorrent clients I’ve used have complex GUIs that seem to be designed for users who download and upload torrents all day, rather than for the occasional user like me, who just wants to get one file. And like many GUI applications, Bittorrent clients tend to accumulate files that take up disk space in obscure directories. I do many things in the terminal, and when I need to download a file, my reflex is to use curl. So I thought, why not make something like curl for Bittorrent? Thinking about curl’s progress meter, it occurred to me that I could do something similar with the TUI library Ratatui, which I’d been wanting to try.

A Bittorrent client seemed like a good opportunity to learn about asynchronous programming in Rust. I had read a lot of criticism of Rust’s async, but I found it pleasant to work with (using the Tokio runtime), and certainly more comfortable than the comparable frameworks that I’d used in Scala (Akka and Cats Effect). I particularly like that an async function can be run in an existing task or in its own task, unlike Scala’s Future, which is assigned a thread and starts running as soon as it’s constructed. Rust’s approach seems more conducive to refactoring and maintainability. At the same time, I’m glad not to have to deal with the complexity of using monads as in Cats Effect.

The result is Sayaca, named after a South American bird that plays an important role in seed dispersal (in Bittorrent terminology, ‘seeding’ means uploading).

Design #

My first task was to implement a parser and formatter for Bencode, the data format used in the Bittorrent protocol. Conceptually it’s similar to JSON, but it’s designed for binary data rather than text. I wanted to try using a parser combinator library, and had no trouble implementing the parser using nom. Since I wanted to be able to serialise and deserialise Rust structs as Bencode, I added Serde support. I’ve published the resulting crate as bside.

The main difficulty in designing the client was the messy state of the Bittorrent specifications. There are ambiguities (the word ‘piece’ means two different things) and implementations have diverged from the specs (e.g. peer IDs are not used to identify peers). A great deal of knowledge about the protocol is not in the official specs, but in other documents scattered around the Web. There is a long list (not updated since 2018) of draft protocol extensions that are supported by many clients. A so-called encryption extension (which actually seems to describe a type of obfuscation) seems to be widely implemented, but as far as I can tell, since 2023, its specification has been preserved only in the Internet Archive’s Wayback Machine. There is a draft BitTorrent Protocol Specification v2 (not updated since 2017), which attempts to correct some of the original protocol’s flaws, such as its reliance on the SHA-1 hash function (which has not been considered secure since 2005). I could have wandered for a long time in a labyrinth of documents and in the source code of existing clients. To keep things simple, I decided to implement only the original BitTorrent Protocol Specification and the IPv6 tracker extension.

When saving data to disk, a Bittorrent client should perform well with large files, be able to interrupt a download and resume it later (which means that it has to note which pieces have been downloaded), and handle torrents that include multiple files. To meet these requirements, and to keep things transparent for the user, I chose to save downloaded data in an intermediate file (with the extension .sayaca), in the same directory where the completed file(s) will be saved. The intermediate file contains a bit field indicating which pieces have been downloaded (the same bit field that we have to send to peers anyway). Sayaca uses memory mapping to read and write this file; this should provide good I/O performance with large files. When the download is complete, it copies the data from the intermediate file into the destination file(s), and the intermediate file is deleted on exit if it is no longer needed.

Since a Bittorrent client is I/O-bound, I decided to use an event-driven architecture in which one task does most of the work, as in Redis. This main task delegates all the network I/O to other tasks, which it communicates with via channels. This was simple to implement, and had the advantage of making it easy to write unit tests for the main task.

Much of the work that a Bittorrent client does amounts to a sort of bookkeeping. In Sayaca, this job is the responsibility of the Bookkeeper. For example, in the recommended algorithm for selecting file pieces to request, the client tries to request the rarest pieces first. This requires a data structure that keeps track of the pieces that each peer has, as well has how rare each piece is, among other things. This can be done in different ways; I chose to use an ordered map, ordered by rarity, to store the IDs of pieces and the sets of peers that have them. The Bookkeeper can then easily find a piece, randomly chosen from among the rarest pieces, along with a randomly chosen peer that has that piece and is accepting requests from us. Thus, much of the client’s work involves updating the Bookkeeper when new information arrives and querying it to find out what to do next.

I like TUIs, and it was easy to make a simple one with Ratatui. But since the client is a library that communicates with its UI via channels, it would be straightforward to wrap the client in a different UI, e.g. a more accessible CLI.

Alternatives to this approach #

I chose to use Tokio because I wanted to learn about it and about async, but I think the program probably wouldn’t have been very different if I had used threads instead, with the same communication via channels.

Writing a virtual machine in Rust

Sun, 12 Oct 2025 00:00:00 +0000

Overview #

To help me learn Rust, I decided to implement an interpreter for Lox, the scripting language in Robert Nystrom’s book Crafting Interpreters. I’ve been interested in scripting languages that are designed to be embedded in applications, and this seemed like a good way to learn more about how such languages are made.

Nystrom’s book walks you through two different implementations of the interpreter: a tree-walk interpreter in Java, and a bytecode virtual machine in C. I was more interested in the bytecode VM. Over the years, I’d written a couple of tree-walk interpreters, starting with Apache FreeMarker, as well as an optimising source-to-source compiler with type inference. But I hadn’t implemented a bytecode VM yet. The idea of it brought back good memories of doing assembly-language programming on the Apple II when I was fifteen years old. The instruction set for Nystrom’s virtual machine is simpler than the instruction set of the Apple II’s Motorola 6502 CPU, but the basic idea is the same. So I skipped the part of his book about the tree-walk interpreter and went directly to the part about the bytecode VM.

The result is OrganicLox. The design is broadly similar to the one in the book, but I’ve tried to write idiomatic Rust, with reasonable performance and no unsafe code. Functionally, OrganicLox should be identical to Nystrom’s clox: at least, it passes his 246 automated tests.

I decided against writing my own garbage collector, because everyone seems to agree that garbage collectors are particularly difficult to implement in Rust. To keep things simple, I used reference counting instead (via Rc). On the main branch, this means that cycles of object properties won’t be collected. On the with_dumpster_gc branch, I’ve replaced Rc with the dumpster crate, which uses reference counting but also deals with cycles.

My favourite parts #

I liked this book very much: Nystrom’s clear explanations, useful illustrations, easygoing writing style, and humour made the book a pleasure to read.

For me, some of the most interesting parts of his design, which draws heavily on the design of the Lua VM, involved the implementation of local variables and closures in a stack-based VM. I enjoyed learning that the compiler associates local variables with specific positions on the stack, so the VM only needs to know about stack positions, not about the variable names they’re associated with.

The implementation of upvalues, the variables that are captured by closures (and may need to be heap-allocated as a result), was tricky to get right. While fixing a bug, I was delighted to realise that a recursive call to a function defined in a local scope requires the function’s closure to be one of its own upvalues.

What I wished for #

Nystrom alternates between top-down and bottom-up explanations. The bottom-up explanations made it more difficult to translate his design into Rust, since I had to guess where he was going. Several times I had to refactor a lot of code because I had guessed incorrectly. Since implementing Lox in all sorts of programming languages seems to be a popular exercise, it would have been helpful to have more high-level notes about the design.

It was fun to learn about Vaughan Pratt’s top-down operator precedence parsing technique and to see that it’s actually possible to generate bytecode without first building an AST. But perhaps it would have been more useful to produce an AST and learn something about writing a second pass to optimise the bytecode.

Performance #

The main optimisation I did at the outset was to have the scanner intern all identifiers (using string_interner), so neither the compiler nor the VM need to do any string comparisons on identifiers.

After finishing Chapter 24, I spent a bit of time looking at performance, following the advice in The Rust Performance Book. I ran benchmarks with criterion, did some profiling with perf and hotspot, adjusted the build configuration, and inlined some functions.

I then looked at some other Rust implementations of Lox, and noticed that everyone seems to have found it difficult to approach the performance of Nystrom’s clox, and that the implementations that succeeded in doing so contained a lot of unsafe code, which I wanted to avoid. I highly recommend Manuel Cerón’s blog post about his impressive, determined, and partly successful attempt to match the speed of clox in Rust by writing more and more unsafe code.

For comparison, here are the results of some of Nystrom’s benchmarks run on these implementations:

Nystrom’s clox
The safe version of Cerón’s loxido
The unsafe version of loxido
OrganicLox with Rc
OrganicLox with the dumpster garbage collector

OrganicLox with Rc is two to four times slower than clox, which is perhaps in the typical range for interpreters written in Rust compared to interpreters written in C. The version with dumpster is much slower. It seems clear that a fast garbage collector is essential for a high-performance VM.

How much do these differences matter? Benchmarks are designed to simulate extreme scenarios, e.g. by using long loops or deep recursion, or by creating a lot of work for the garbage collector, so they’re not necessarily representative of real use cases. The scripts run by applications with an embedded interpreter are often short and do very little work. We might get a more realistic idea of how a VM performs in such a scenario by running Nystrom’s integration test suite, which runs 246 short Lox scripts and checks their output. Here’s a comparison:

In an application where every CPU cycle counts, the differences in performance between these different implementations could be very significant. But in other use cases, they might not be.

Alternatives to this approach #

It’s tempting to try adding a just-in-time compiler (JIT) to improve the performance of the interpreter, either by writing one from scratch or by using an existing backend like LLVM or Cranelift. Examples in Rust include Roto and Dora. But even if you do that, it’s not easy to achieve high performance. In these benchmarks, Python’s new JIT has no measurable effect on performance (see also these Dora benchmark results).

That’s one reason to consider using an existing interpreter that already has years of engineering invested in it, instead of creating a new one. Lua is one popular option, but it isn’t fast enough for some applications. The widely used LuaJIT VM is an order of magnitude faster than the standard Lua VM for some use cases, but it’s a tracing JIT, which has advantages and disadvantages. The Lua developers have an interesting paper in which they discuss the limitations of this approach, and propose a different solution (an additional statically typed language, a bit like Cython).

For some use cases, if embedded scripts don’t do much work, their performance might not be critical. If they do so much work that their performance is insufficient, another option is not to embed an interpreter at all. Cloudflare, which had invested heavily for over 20 years in scripts using LuaJIT embedded in OpenResty, recently replaced them with Rust code.