Benjamin Geer

Writing a hypermedia app in Rust

Mon, 05 Oct 2026 07:02:00 +0000

This is the third post in a series:

Writing safe hypermedia apps for a resource-constrained world, which introduces the series
Writing a hypermedia app in Go
this post

Overview of the app #

The example application in Hypermedia Systems is a simple CRUD application that manages a list of contacts. It illustrates how a Hypermedia-Driven Application can provide a good user interface as an alternative to single-page applications. With a tiny JavaScript library and some declarative attributes on HTML elements, all sorts of user interactions can fetch fragments of HTML from the server and update only certain parts of the page. For example, as we page through the list of contacts, or delete a contact from the list, the list changes without reloading the page.

Let’s implement this app in Rust, with two objectives:

Make it more realistic by adding an internationalised UI, a relational database with a full-text search index, better input validation, cursor-based pagination, and UI tests.
Make it easy to maintain, with a minimum of boilerplate, and an emphasis on compile-time checks to prevent bugs.

We’ll start with a SQLite database, and switch to PostgreSQL in a later step. You can find the complete source code of both versions of the app here:

Libraries #

Our first task is to choose some libraries.

HTML templates #

Our user interface is based on HTML templates. There’s a layout template that provides a header and shows informational messages (like ‘Contact added’) in response to user actions. Other templates, such as a form for editing a contact, can be embedded in this layout. There are also templates for fragments of HTML that replace one or more parts of the currently displayed page. For example, when you click on a pagination link, the list of contacts changes, along with the pagination links.

So we’ll need a template engine, preferably a type-safe one. There are basically two kinds of type-safe template engines: the kind that lets you write something close to standard HTML, and the kind that lets you express the structure of an HTML document in a programming language. I prefer the first kind, because the closer the template syntax is to HTML, the easier it will be for a UX designer, or anyone who knows HTML, to work on the templates without having to know Rust.

I’ve chosen Askama, which is easy to use, and compiles templates to Rust code. Since its syntax is based on Jinja, editor tooling that works with Jinja will also work with Askama. To use it, we write our templates in HTML files; for each template, we write a Rust struct definition that contains the template’s variables and refers to the HTML file via a macro. In our HTTP endpoint implementations, we call that struct’s render method to generate HTML.

Internationalisation #

We’d like our app to be available in multiple languages; in this example, I’ve implemented support for English and French. To facilitate maintenance, we want to avoid having different versions of the same template for different languages, or cluttering up the templates with different translations of the same messages. So we need a library that allows us to call a function from a template and get the translation of a message in the target language.

One important consideration is what the library’s designers chose to use as a message identifier. Some libraries (like gettext) use the whole message text in the default language as the message identifier. The limitations of this approach are well expressed in the social contract of Project Fluent:

First of all, it means that any change to the source string invalidates all translations of the string. This severely increases the burden on the developers to never alter messages in the source language as it results in all translations having to be updated.

Secondly, it makes it harder to introduce multiple messages with the same source string which should be translated differently.

Therefore, I prefer libraries that use unique message identifiers chosen by developers.

Another important criterion in choosing a library is how it handles plurals. While plurals are relatively simple in English (there are just two categories, singular and plural), many languages have more categories, along with syntactic rules that require a sentence to be adjusted in more complex ways depending on the number expressed. A good internationalisation library should implement the Unicode CLDR plural rules, which support over 200 languages.

I was interested in trying Project Fluent, but it looks as if that project may have been abandoned. So I’ve chosen karatepe, which has the features we need. Karatepe provides a simple DSL for defining messages and writing translations, and a macro that compiles message definitions into Rust functions. For example, here’s a message definition that takes a numerical parameter amount:

`1`	`contacts_count(amount: ..)`

And here’s its English translation:

contacts_count(amount) = ({amount} total {amount => {
    one => contact
    _ => contacts
  }})

The message definition is compiled to a Rust function, contacts_count, which we can call directly from a template. At run time, we can access a message translation via a Translation for the target language; we’ll see below how we get the correct one.

Since we’re internationalising the app, we can also improve the cross-cultural validity of our field names: instead of ‘first name’ and ‘last name’, which can lead to misunderstandings (in Chinese names, the family name comes first), we’ll follow the recommendation in the FOAF ontology and use ‘given name’ and ‘family name’.

Database access and schema migration #

We could use an object-relational mapping (ORM) library for database access, but I think it’s better to write our SQL ourselves. ORMs can produce inefficient queries.¹ Writing our own SQL allows us to take full advantage of the features of modern SQL and its opportunities for query optimisation, particularly by designing queries and indexes together, and by refining queries in light of the execution plans of the database’s query optimiser. (See my post Optimising PostgreSQL queries with an open dataset for an example.)

I think the ideal is to have a library that provides type safety for SQL queries, so queries are checked at compile time and automatically return Rust structs. The SQLx crate does just this, and can return our own structs by deriving its FromRow trait.

For example, suppose we misspell a column name (familyname instead of family_name) in an SQL query:

let contact: Contact = sqlx::query_as!(
    Contact,
    "SELECT id, familyname, given_name, phone, email
    FROM contact WHERE id = $1",
    id
)
.fetch_one(&mut *tx)
.await?;

By default, the query_as! macro will connect to the development database at compile time to check the query, and we’ll get a compile error:

1
2

error: error returned from database: (code: 1) no such column: familyname
   --> src/db.rs:236:23

SQLx also uses prepared statements to prevent SQL injection attacks, and provides a command-line tool for managing database schema migrations.

HTTP server framework #

We’ll use axum, which conveniently implements type-safe mappings between requests and Rust structs via Serde, and is easy to use with Askama.

UI testing #

If we were using the SPA approach, our server would have a JSON API, and we could write tests that check whether it returns the correct JSON for various requests. But our application returns HTML. So we’ll write tests that are analogous to JSON API tests, focusing on the functionality of our HTML responses. Using the dom_query crate, we’ll parse the HTML responses and use CSS selectors to check the contents of specific elements. This allows the graphic design to change without breaking our tests.

Database design #

(If you’ve already read the Go version of this post, you can skip this section.)

Our database is very simple. There’s just one table, contact:

CREATE TABLE IF NOT EXISTS contact (
  id INTEGER PRIMARY KEY,
  family_name TEXT NOT NULL,
  given_name TEXT NOT NULL,
  phone TEXT,
  email TEXT UNIQUE NOT NULL
) STRICT;

But there are a few details to think about.

I’ve defined the contact table as STRICT. By default, SQLite has a very flexible type system, which some people like. I prefer to get errors when there’s a bug in my program. With the STRICT option, SQLite will do strict type checking at run time, in addition to the compile-time checks that sqlc provides.

Nullable columns add a bit of complexity to our code, so to keep things simple for this example, we’ll allow only the phone column to be null.

The app should display the contact list sorted in some order; we’ll use ORDER BY family_name, given_name, email. To make this efficient, we’ll add an index with that sort order:

1
2

CREATE INDEX IF NOT EXISTS idx_contact_sort
ON contact (family_name, given_name, email);

Since the app requires email addresses to be unique, we have a UNIQUE constraint on the email field, and we’ll need a query like this:

`1`	`SELECT 1 FROM contact WHERE email = ?;`

To make this query efficient, we’ll put an index on the email column:

`1`	`CREATE INDEX IF NOT EXISTS idx_contact_email ON contact (email);`

We want a full-text search index that allows us to find contacts whose name matches a string. We’ll let the user type the first letters of given_name and/or family_name and get a list of matching contacts. Fortunately, SQLite has a full-text search module, FTS5. To use it, we create a virtual table to store the index:

CREATE VIRTUAL TABLE IF NOT EXISTS contact_fts USING fts5(
    family_name,
    given_name,
    phone UNINDEXED,
    email UNINDEXED,
    content = 'contact',
    content_rowid = 'id'
);

We’ll use database triggers to keep the FTS5 table up to date with the content table. And that completes our database schema.

Application architecture #

For this simple CRUD app, a model-view-controller architecture is sufficient. The views are HTML templates, and the controllers are HTTP routes. The main type in the model is a struct called Contact, which we’ll get to in a moment. The database abstraction layer is just a struct called Database that contains the connection pool and has methods like this:

1
2
3

pub async fn get(&self, id: ContactId) -> Result<Contact, DatabaseError> {
    // ...
}

Parsing, not validating #

We’ll apply Alexis King’s principle, Parse, don’t validate. In short, instead of representing, say, an email address as a &str, we use the newtype pattern and create a type for it, called Email, which can only be constructed from valid input. This has several beneficial effects:

All the functions in the database layer only accept newtypes and structs composed of newtypes, so it’s impossible for them to receive invalid values.
Since it’s impossible to construct an invalid value, we never have to wonder whether a value has already been validated, or risk validating it twice.
Having to pass an &Email to a function instead of a &str can prevent certain mistakes, such as using the wrong variable in a function call, or getting the order of the parameters wrong.
Function parameters become clearer: after: &Cursor tells us more than after: &str.

We’ll use the nutype crate to define our newtypes. It offers a variety of built-in validation functions, e.g. for validating with a regular expression. Since this is an internationalised application, we’ll use Unicode regular expressions to support:

all the Unicode characters that could appear in a person’s name
any phone number that could be used in any country
email address internationalisation

For example, my simple attempt at a name regex accepts groups of Unicode letters and marks, separated by spaces and/or punctuation:

`1`	`^[\p{L}\p{M}]+(?:(?:\p{P}\p{Zs}\|[\p{P}\p{Zs}])[\p{L}\p{M}]+)*\p{P}?$`

So if your given name is 曼玉-فاتن, all is well.

Now we can define a newtype:

#[nutype(
    validate(regex = NAME_REGEX, len_char_max = 50),
    derive(Debug, Display, PartialEq, Clone, AsRef, Serialize),
    derive_unchecked(sqlx::Type),
)]
#[sqlx(transparent)]
pub struct GivenName(String);

This generates a constructor, GivenName::try_new, which takes a string and returns a Result<GivenName, GivenNameError>. We’ve also specified that we want SQLx to be able to construct a GivenName without validation, because we assume that the data in the database has already been validated. We also have the option of not trusting the database, and parsing GivenName in the normal way in the database layer.

After defining a few other newtypes in the same way, we can define Contact:

#[derive(Debug, PartialEq, sqlx::FromRow)]
pub struct Contact {
    pub id: ContactId,
    pub family_name: FamilyName,
    pub given_name: GivenName,
    pub phone: Option<Phone>,
    pub email: Email,
}

When the user is creating a new contact, it doesn’t have an ID yet, so we also define ContactContent:

#[derive(Debug, PartialEq, Serialize)]
pub struct ContactContent {
    pub family_name: FamilyName,
    pub given_name: GivenName,
    pub phone: Option<Phone>,
    pub email: Email,
}

What should happen when the user submits a form to create a contact? We don’t want axum to deserialise the form data directly to a ContactContent, because that would stop parsing at the first error. Instead, we want to collect multiple validation errors and report them all back to the user. The form has a <span class="error"> under each input field for displaying these errors. We can represent the unparsed form input as a struct like this:

use serde_with::{NoneAsEmptyString, serde_as};

#[serde_as]
#[derive(Debug, Default, Deserialize)]
pub struct ContactForm {
    pub family_name: String,
    pub given_name: String,

    #[serde_as(as = "NoneAsEmptyString")]
    pub phone: Option<String>,

    pub email: String,
}

We’d like to parse a ContactForm and get either a ContactContent or a collection of errors. There isn’t yet a way to automate this completely, but we can write a short function to do it using the frunk crate of functional programming tools. We’ll use its Validated type, which collects the results of several operations. If there are no errors, Validated::into_result produces an HList, a statically typed list whose elements can have different types. Given an HList, we can easily construct our ContactContent. If any errors occurred, we get them in a Vec.

use frunk::prelude::*;
use frunk_core::hlist_pat;

impl ContactForm {
    pub fn parse(&self) -> Result<ContactContent, Vec<ParseError>> {
        let validated = FamilyName::try_new(&self.family_name)
            .map_err(ParseError::from)
            .into_validated()
            + GivenName::try_new(&self.given_name).map_err(ParseError::from)
            + self.phone.as_ref()
                .map(|phone| Phone::try_new(phone)
                    .map_err(ParseError::from))
                .transpose()
            + Email::try_new(&self.email).map_err(ParseError::from);

        validated
            .into_result()
            .map(
                |hlist_pat!(family_name, given_name, phone, email)|
                    ContactContent {
                        family_name,
                        given_name,
                        phone,
                        email,
                    },
            )
    }
}

Our ParseError type contains the original error (GivenNameError, etc.), so we can easily determine which fields had errors. We can then pass the ContactForm and a collection of translated error messages to the template.

If validation was successful, we can pass the ContactContent to Database::add, which looks like this:

pub async fn add(&self, contact: &ContactContent) -> Result<(), Error> {
    let mut tx = self.pool.begin().await?;

    sqlx::query!(
        r#"INSERT INTO contact (family_name, given_name, phone, email)
            VALUES ($1, $2, $3, $4)"#,
        &contact.family_name,
        &contact.given_name,
        &contact.phone,
        &contact.email
    )
    .execute(&mut *tx)
    .await?;

    tx.commit().await?;
    Ok(())
}

Error handling #

Whose fault is it? #

Our error handling strategy distinguishes between errors that are the client’s fault and ones that are the server’s fault. We can use the thiserror crate to simplify wrapping errors in other errors:

use thiserror::Error;

#[derive(Error, Debug)]
pub enum Error {
    #[error("Client error: {0}")]
    Client(#[from] ClientError),

    #[error("Server error: {0}")]
    Server(ServerError),
}

#[derive(Error, Debug)]
pub enum ClientError {
    #[error("Page not found")]
    NotFound,

    // ...
}

#[derive(Error, Debug)]
pub enum ServerError {
    #[error("Database error: {0}")]
    Database(#[from] DatabaseError),

    // ...
}

#[derive(Error, Debug)]
pub enum DatabaseError {
    #[error("{0}")]
    Sqlx(#[from] sqlx::Error),
}

impl<E> From<E> for Error
where
    E: Into<ServerError>,
{
    fn from(err: E) -> Self {
        Error::Server(err.into())
    }
}

If an error is the client’s fault, we’ll just report it to the user, e.g. by displaying ‘Page not found’. If it’s the server’s fault (e.g. it’s a DatabaseError), we’ll display ‘Internal server error’ and log the error details.

Centralising error handling #

I like to centralise error handling so it doesn’t clutter up the rest of the code. With axum, a route handler returns a Result whose error type is anything that implements the IntoResponse trait. Therefore, with a bit of plumbing, we can use the ? operator in our route handler functions to log the error if necessary and return an error page using our error.html template. We make an error type called ErrorResponse that just holds an HTTP status code and some HTML, and we implement IntoResponse for it. The we write a method on our Error type that produces an ErrorResponse using the template, with a translated error message, and logs the error if it’s the server’s fault. This method has to take a &Translation parameter so the template can get the error message in the target language:

impl Error {
    pub fn to_error_response(self, t: &Translation) -> ErrorResponse {
        // ...
    }
}

Finally, we write an extension trait for Result to convert any error into an ErrorResponse:

pub trait ResultErrorTranslationExt<T, E>
where
    E: std::error::Error,
    Error: From<E>,
{
    fn tr_err(self, t: &Translation) -> Result<T, ErrorResponse>;
}

impl<T, E> ResultErrorTranslationExt<T, E> for Result<T, E>
where
    E: std::error::Error,
    Error: From<E>,
{
    fn tr_err(self, t: &Translation) -> Result<T, ErrorResponse> {
        self.map_err(|err| Error::from(err).to_error_response(t))
    }
}

Now we can write route handler functions that return Result<Html<String>, ErrorResponse> and handle errors very concisely. For example, if the db.get method can return DatabaseError, we can call it like this:

1
2

let t: &Translation = // ...
let contact = app.db.get(id).await.tr_err(t)?;

Routes and languages #

Before we define HTTP endpoints, we should decide how the app will know which language to use. The Accept-Language request header allows the browser to provide a ranked list of the languages that the user prefers, but we also have to allow the user to select a language just for this app:

This header serves as a hint when the server cannot determine the target content language otherwise (for example, use a specific URL that depends on an explicit user decision). The server should never override an explicit user language choice. The content of Accept-Language is often out of a user’s control (when traveling, for instance). A user may also want to visit a page in a language different from the user interface language.

Let’s put a language switcher in the page header, and put a BCP 47 language tag at the beginning of every endpoint’s path. For the root path, we can use the Accept-Language header, and redirect to a path containing a language tag.

With the karatepe crate, we load a Translation for each language on startup. We’ll keep these in a HashMap (with icu::locale::LanguageIdentifier as the key type) in the application’s shared state, along with the Database. This is our AppState struct:

pub struct AppState {
    pub db: Database,
    pub translations: Arc<HashMap<LanguageIdentifier, Translation>>,
}

axum’s Router can hold onto this state for us so the route handler functions can access it. We set it up like this:

pub fn init_router(app: AppState) -> Router {
    Router::new()
        .route("/", get(index)) // checks Accept-Language
        .route(
            "/{lang}/contacts",
            get(contacts::contacts_get).delete(contacts::contacts_delete),
        )
        .route(
            "/{lang}/contacts/{id}",
            get(contacts::contact_get).delete(contacts::contact_delete),
        )
        // ...
        .with_state(app)
}

axum provides extractors that handler functions can use to extract and parse data in the request. We use the Path extractor to parse the language code as a LanguageIdentifier (which implements serde::Deserialize), so we don’t need to make our own newtype for this. Then we just need a little function called translation that takes a LanguageIdentifier and returns the corresponding Translation from the AppState.

Let’s not make one of those annoying websites whose language switcher takes you back to the home page. We can write a function, uris_with_langs, that takes the current URL path, adjusts it for every available language, and returns a Vec of the resulting paths. In the page header, we display those URLs as links.

Our template for displaying a contact is called show.html, and its variables are in struct ShowContactTemplate. So our contact_get route now looks like this:

pub async fn contact_get(
    State(app): State<AppState>,
    Path((lang, id)): Path<(LanguageIdentifier, ContactId)>,
    uri: Uri,
) -> Result<Html<String>, ErrorResponse> {
    let t = app.translation(&lang);
    let contact = app.db.get(id).await.tr_err(t)?;

    let template = ShowContactTemplate {
        t,
        lang,
        lang_uris: localisation::uris_with_langs(uri).tr_err(t)?,
        contact,
    };

    Ok(Html(template.render().tr_err(t)?))
}

Pagination #

(If you’ve already read the Go version of this post, you can skip this section and the next.)

The pagination in Hypermedia Systems is based on page numbers, which is inefficient (the database always has to retrieve all the pages preceding the one you asked for) and can produce unexpected results if the data is being modified while you’re paging through it. A better approach is cursor-based pagination. When returning a page of results, the database layer can return two cursors, called prev and next, which the view can present as links to the previous and next pages. There are a few tricks involved in making this work.

Our cursor will contain the values of the columns family_name, given_name, and email, which are the same ones we use to sort contacts. Since we already have an index on those three columns, in that order, the database can efficiently retrieve a page of rows before or after a cursor.

We’d like to avoid returning a prev cursor if we’ve reached the first page, or a next cursor if we’ve reached the last one. The solution is as follows: in a ‘page after’ request, if our maximum page size is N, we request N + 1 rows. If we get that many rows, we know a next page exists, so we return a next cursor based on the last row in the returned page. If we were given a next cursor in the request, we can assume there’s a previous page, so the first row in the returned page is the prev cursor.

Our SQL queries for getting a page of contacts have to take into account the fact that given names and family names aren’t necessarily unique, although email addresses are. So our ‘page after’ query looks like this (with a Common Table Expression so we don’t have to pass the same parameters more than once):

WITH vars AS (SELECT ? AS family_name, ? AS given_name, ? AS email)
SELECT id, family_name, given_name, phone, email
FROM contact
WHERE (family_name = (SELECT family_name FROM vars)
       AND given_name = (SELECT given_name FROM vars)
       AND email > (SELECT email FROM vars))
OR (family_name = (SELECT family_name FROM vars)
    AND given_name > (SELECT given_name FROM vars))
OR family_name > (SELECT family_name FROM vars)
ORDER BY family_name, given_name, email
LIMIT ?;

Full-text search queries #

There isn’t much left to do for full-text search. The /{lang}/contacts endpoint receives a query string consisting of one or more words or word prefixes. We parse this string and adapt it to the FTS5 query syntax. For example, G O'Mal becomes "G"* + "O'Mal"*, and will match a contact whose name is Grace O’Malley. We return a maximum of one page of full-text search results, so the route passes None cursors to the template, which removes the pagination links via an htmx out-of-band swap.

Integration tests #

There’s nothing remarkable about this application’s unit tests (unless your name happens to be 曼玉-فاتن), but the integration tests are more interesting. I’ve written two sets of integration tests: one for the database layer, and one for the HTTP endpoints. In both cases, we use sqlx::test, which creates a new database for each test and initialises it with a fixture, so the tests are isolated from each other. As mentioned above, the HTTP endpoint tests parse the HTML responses and check the contents of specific elements. We use the same axum router as in the application, and parse the responses using dom_query. For example, to test a GET request to /{lang}/contacts/{id}/edit, the endpoint for editing an existing contact, we can do something like this:

#[sqlx::test(fixtures("test-contacts"))]
async fn edit_contact_get(pool: SqlitePool) {
    let server = init_server(pool).await;
    let response = server.get("/en/contacts/1/edit").await;

    assert_eq!(response.status_code(), StatusCode::OK);
    let doc = Document::from(response.text());

    let email_label = doc.select("label[for='email']").text().to_string();
    assert_eq!(email_label, "Email");

    let email = doc.select("#email").attr("value").unwrap().to_string();
    assert_eq!(email, "foo1@bar.com");

    // ...
}

Since we’re just looking at the text in labels and form fields using simple CSS selectors, test like this are unlikely to be affected by changes in the graphic design of the page.

PostgreSQL #

We don’t need to do much to adapt our app to use PostgreSQL instead of SQLite.

Full-text search #

(If you’ve already read the Go version of this post, you can skip this section.)

Like SQLite, Postgres has built-in full-text search, and it’s even easier to set up. We can store tsvector values (preprocessed documents) in a generated column in the contact table, with a GIN index to speed up queries:

CREATE TABLE IF NOT EXISTS contact (
  id BIGINT GENERATED ALWAYS AS IDENTITY PRIMARY KEY,
  family_name TEXT NOT NULL,
  given_name TEXT NOT NULL,
  phone TEXT,
  email TEXT UNIQUE NOT NULL,
  textsearch tsvector GENERATED ALWAYS AS (
    to_tsvector(
      'simple',
      COALESCE(family_name, '') || ' ' || COALESCE(given_name, '')
    )
  ) STORED
);

CREATE INDEX textsearch_idx ON contact USING GIN (textsearch);

The SQL query to find matching rows uses the full-text match operator, @@:

SELECT id, family_name, given_name, phone, email
FROM contact, to_tsquery($1) query
WHERE query @@ textsearch
ORDER BY ts_rank_cd(textsearch, query)
LIMIT $2;

The syntax of the string that we pass to to_tsquery just needs minor adjustments to work with Postgres: if the user types G O'Mal, we pass the argument 'G':* & 'O''Mal':*.

Integration tests with Testcontainers #

We can use Testcontainers to run Postgres in an ephemeral Docker container. #[sqlx:test] runs the tests in parallel, and isolates them from each other by creating a new database in the same container for each test.

There’s a small problem to solve here: #[sqlx:test] generates code that reads the test database URL from an environment variable or an .env file, but we don’t know that URL until after Testcontainers starts the Docker container. The solution, shown in an example project by Matilda Smeds, is to use the linktime crate to define a module initialisation function that runs before the tests do. This function starts the test container and sets the DATABASE_URL environment variable via std::env::set_var.

Conclusion #

By following the principle of ‘Parse, don’t validate’, and by using deterministic code generation that takes advantage of the guarantees provided by Rust’s type system, we’ve kept boilerplate to a minimum, and improved the chances that if we make a mistake, we’ll get an error at compile time rather than at run time. This should help keep our code maintainable and prevent bugs.

See the N+1 selects problem and selecting all columns without thinking about it. ↩︎

Writing a hypermedia app in Go

Mon, 05 Oct 2026 07:01:00 +0000

This is the second post in a series:

Writing safe hypermedia apps for a resource-constrained world, which introduces the series
this post
Writing a hypermedia app in Rust

Overview of the app #

The example application in Hypermedia Systems is a simple CRUD application that manages a list of contacts. It illustrates how a Hypermedia-Driven Application can provide a good user interface as an alternative to single-page applications. With a tiny JavaScript library (htmx) and some declarative attributes on HTML elements, all sorts of user interactions can fetch fragments of HTML from the server and modify only certain parts of the page. For example, as we page through the list of contacts, or delete a contact from the list, the displayed list is updated without reloading the page.

Let’s implement this app in Go, with two objectives:

Make it more realistic by adding an internationalised UI, a relational database with a full-text search index, better input validation, cursor-based pagination, and UI tests.
Make it easy to maintain, with a minimum of boilerplate, and an emphasis on compile-time checks to prevent bugs.

We’ll start with a SQLite database, and switch to PostgreSQL in a later step. You can find the complete source code of both versions of the app here:

Libraries #

Our first task is to choose some libraries.

HTML templates #

So we’ll need a template engine. Go’s standard library has html/template, but its templates aren’t type-safe. There are basically two kinds of type-safe template engines: the kind that lets you write something close to standard HTML, and the kind that lets you express the structure of an HTML document in a programming language. I prefer the first kind, because the closer the template syntax is to HTML, the easier it will be for a UX designer, or anyone who knows HTML, to work on the templates without having to know Go. I’ve chosen templ, which is easy to use, compiles templates to Go code, and has good editor support. To use it, we write templates as functions whose bodies are HTML, and run templ generate to compile them to Go functions. We can call these functions from our HTTP endpoint implementations.

Internationalisation #

First of all, it means that any change to the source string invalidates all translations of the string. This severely increases the burden on the developers to never alter messages in the source language as it results in all translations having to be updated.

Secondly, it makes it harder to introduce multiple messages with the same source string which should be translated differently.

Therefore, I prefer libraries that use unique message identifiers chosen by developers.

I was interested in trying Project Fluent, but it looks as if that project may have been abandoned. So I’ve chosen go-18n, which has the features we need. To avoid clutter, I’ve written a function like this for each message, providing the text in English, the default language:

func InfoContactCount(loc *i18n.Localizer, count int) string {
  return loc.MustLocalize(&i18n.LocalizeConfig{
    DefaultMessage: &i18n.Message{
      ID:    "InfoContactCount",
      One:   "({{.Count}} total contact)",
      Other: "({{.Count}} total contacts)",
    },
    TemplateData: map[string]any{
      "Count": count,
    },
    PluralCount: count,
  })
}

We can call this function directly from a template. The goi18n command-line program extracts the messages from our source code into text files, where we can then translate them. At run time, we can access a message translation via a Localizer for the target language; we’ll see below how we get the correct one.

Database access and schema migration #

Go’s standard library provides the database/sql package, but it requires you to write your own mappings between query results and Go structs. Maintaining these mappings is cumbersome, and it’s easy to make mistakes that won’t be caught at compile time. We could use an object-relational mapping (ORM) library, but I think it’s better to write our SQL ourselves. ORMs can produce inefficient queries.¹ Writing our own SQL allows us to take full advantage of the features of modern SQL and its opportunities for query optimisation, particularly by designing queries and indexes together, and by refining queries in light of the execution plans of the database’s query optimiser. (See my post Optimising PostgreSQL queries with an open dataset for an example.)

I think the ideal is to have a library that provides type safety for SQL queries, so queries are checked at compile time and automatically return Go structs. The sqlc package does just this (see its introductory blog post for a more detailed explanation). To use sqlc, we put our SQL queries in text files. When we run sqlc generate on the command line, sqlc connects to the database to analyse our queries, and generates a Go package with a type-safe function corresponding to each query.

For example, suppose we misspell a column name (familyname instead of family_name) in an SQL query:

`1`	`SELECT id, familyname, given_name, phone, email FROM contact WHERE id = ?;`

We’ll get an error from sqlc generate:

`1`	`sql/queries.sql:1:1: sqlite3: SQL logic error: no such column: familyname`

Here we’ll also have sqlc use prepared statements to prevent SQL injection attacks.

Database schemas change over time, so we need a tool that manages schema migrations. There are several good options, and here I’ve chosen goose.

HTTP server framework #

Go’s standard library has net/http for writing HTTP servers, but I’ve chosen the Gin framework, which implements type-safe mappings between requests and Go structs, as well as flexible input validation via the validator package. And we can easily integrate templ into Gin.

UI testing #

If we were using the SPA approach, our server would have a JSON API, and we could write tests that check whether it returns the correct data for various requests. But our application returns HTML. So we’ll write tests that are analogous to JSON API tests, focusing on the functionality of our HTML responses. Using the goquery package, we’ll parse the HTML responses and use CSS selectors to check the contents of specific elements. This allows the graphic design to change without breaking our tests.

Database design #

(If you’ve already read the Rust version of this post, you can skip this section.)

Our database is very simple. There’s just one table, contact:

CREATE TABLE IF NOT EXISTS contact (
  id INTEGER PRIMARY KEY,
  family_name TEXT NOT NULL,
  given_name TEXT NOT NULL,
  phone TEXT,
  email TEXT UNIQUE NOT NULL
) STRICT;

But there are a few details to think about.

I’ve defined the contact table as STRICT. By default, SQLite has a very flexible type system, which some people like. I prefer to get errors when there’s a bug in my program. With the STRICT option, SQLite does strict type checking at run time, in addition to the compile-time checks that sqlc provides.

Nullable columns add a bit of complexity to our code, so to keep things simple for this example, we’ll allow only the phone column to be null.

The app should display the contact list sorted in some order; we’ll use ORDER BY family_name, given_name, email. To make this efficient, we’ll add an index with that sort order:

1
2

CREATE INDEX IF NOT EXISTS idx_contact_sort
ON contact (family_name, given_name, email);

Since the app requires email addresses to be unique, I’ve put a UNIQUE constraint on the email field, and we’ll need a query like this:

`1`	`SELECT 1 FROM contact WHERE email = ?;`

To make this query efficient, we’ll put an index on the email column:

`1`	`CREATE INDEX IF NOT EXISTS idx_contact_email ON contact (email);`

We want a full-text search index that allows us to find contacts whose given name or family name matches a string. We’ll let the user type the first letters of given_name and/or family_name and get a list of matching contacts. Fortunately, SQLite has a full-text search module, FTS5. To use it, we create a virtual table to store the index:

CREATE VIRTUAL TABLE IF NOT EXISTS contact_fts USING fts5(
    family_name,
    given_name,
    phone UNINDEXED,
    email UNINDEXED,
    content = 'contact',
    content_rowid = 'id'
);

We’ll use triggers to keep the table contact_fts up to date with the table contact. And that completes our database schema.

Application architecture #

For this simple CRUD app, a model-view-controller architecture is sufficient. The views are HTML templates, and the controllers are HTTP routes. The main type in the model is Contact:

type Contact struct {
  ID         int64
  FamilyName string
  GivenName  string
  Phone      string
  Email      string
}

The database abstraction layer is just a struct called Database that contains the connection pool and has methods like this:

func (db *Database) Get(
  ctx context.Context,
  id int64,
) (*model.Contact, error) {
  // ...
}

Input validation #

What should happen when the user submits a form to create a contact? We’ll have to validate all the fields, and be able to collect multiple validation errors and report them all back to the user. The form has a <span class="error"> under each input field for displaying these errors. Gin allows us to bind the form fields to the fields of a struct, call a validation function for each field, and get a slice of validation errors. Since a contact that hasn’t been saved yet doesn’t have an ID, let’s make another struct for this:

type ContactContent struct {
  FamilyName string `form:"family_name" binding:"required,name"`
  GivenName  string `form:"given_name" binding:"required,name"`
  Phone      string `form:"phone" binding:"omitempty,phone"`
  Email      string `form:"email" binding:"required,email"`
}

The form tag specifies the form field to be bound to a struct field, and binding lists the validators to be run for the field. We can use the validator package’s built-in validators as well as our own. Here I’ve made custom validators called name, phone, and email, which use Unicode regular expressions. Since this is an internationalised application, we want to support:

all the Unicode characters that could appear in a person’s name
any phone number that could be used in any country
email address internationalisation

For example, my simple attempt at a name regex accepts groups of Unicode letters and marks, separated by spaces and/or punctuation:

`1`	`^[\p{L}\p{M}]+(?:(?:\p{P}\p{Zs}\|[\p{P}\p{Zs}])[\p{L}\p{M}]+)*\p{P}?$`

So if your given name is 曼玉-فاتن, all is well. We’ll make a custom validator function using this regex:

var nameValidator validator.Func = func(fl validator.FieldLevel) bool {
  field := fl.Field().String()
  return len(field) <= 50 && NameRegex.MatchString(field)
}

When the app starts, we’ll register the function with the validator package:

1
2
3

if validate, ok := binding.Validator.Engine().(*validator.Validate); ok {
  validate.RegisterValidation("name", nameValidator)
}

Error handling #

Our error handling strategy distinguishes between errors that are the client’s fault and errors that are the server’s fault:

type ClientError struct {
  Cause error
}

// ...

type NotFoundError struct{}

type ServerError struct {
  Cause error
}

type DatabaseError struct {
  Cause error
}

// ...

Routes and languages #

This header serves as a hint when the server cannot determine the target content language otherwise (for example, use a specific URL that depends on an explicit user decision). The server should never override an explicit user language choice. The content of Accept-Language is often out of a user’s control (when traveling, for instance). A user may also want to visit a page in a language different from the user interface language.

With the go-i18n package, we initialise an i18n.Bundle on startup; it contains the message keys and translations. This bundle will be part of the application’s shared state, along with the Database. This is our AppState struct:

type AppState struct {
  Database   *db.Database
  I18NBundle *i18n.Bundle
}

If we implement our routes as methods with *AppState as their receiver, we can set up a Gin router like this:

engine := gin.Default()
engine.GET("/", app.index) // checks Accept-Language
engine.GET("/:lang/contacts", app.contactsGet)
engine.GET("/:lang/contacts/:id", app.contactGet)
// ...

We can use Gin’s binding and validation to validate the :lang path parameter as well. I’ve written a little function called GetLocalizer, which returns the Localizer for the language we got from the path, if any, and falls back to using the Accept-Language header.

Let’s not make one of those annoying websites whose language switcher takes you back to the home page. We can write a function, URLsWithLangs, that takes the current URL path, adjusts it for every available language, and returns a slice of the resulting paths. In the page header, we display those URLs as links.

Now, if we write a template called Show that takes a Contact as a parameter, our contactGet route will look like this:

type LanguagePathParams struct {
  Lang string `uri:"lang" binding:"required,bcp47_strict_language_tag"`
}

type IDPathParams struct {
  ID int64 `uri:"id" binding:"required"`
}

type ContactPathParams struct {
  LanguagePathParams
  IDPathParams
}

func (state *AppState) contactGet(ctx *gin.Context) {
  var pathParams ContactPathParams

  if err := ctx.ShouldBindUri(&pathParams); err != nil {
    state.notFound(ctx)
    return
  }

  loc := localisation.GetLocalizer(state.I18NBundle, pathParams.Lang, ctx)

  contact, err := state.Database.Get(ctx, pathParams.ID)
  if err != nil {
    errorPage(ctx, pathParams.Lang, loc, err)
    return
  }

  urlsWithLangs, err := localisation.URLsWithLangs(ctx.Request.URL)
  if err != nil {
    errorPage(ctx, pathParams.Lang, loc, err)
    return
  }

  ctx.HTML(http.StatusOK, "", templates.Show(pathParams.Lang, loc, contact, urlsWithLangs))
}

Pagination #

(If you’ve already read the Rust version of this post, you can skip this section and the next.)

We’d like to avoid returning a prev cursor if we’ve reached the first page, or a next cursor if we’ve reached the last one. The solution is as follows: in a ‘page after’ request, if our maximum page size is N, we request N + 1 rows. If we get them, we know a next page exists, so we return a next cursor based on the last row in the returned page. If we were given an after cursor in the request, we can assume there’s a previous page, so the prev cursor corresponds to the first row in the returned page.

Our SQL queries for getting a page of contacts have to take into account the fact that names aren’t necessarily unique. So our ‘page after’ query looks like this (with a Common Table Expression so we don’t have to pass the same parameters more than once):

WITH vars AS (SELECT ? AS family_name, ? AS given_name, ? AS email)
SELECT id, family_name, given_name, phone, email
FROM contact
WHERE (family_name = (SELECT family_name FROM vars)
       AND given_name = (SELECT given_name FROM vars)
       AND email > (SELECT email FROM vars))
OR (family_name = (SELECT family_name FROM vars)
    AND given_name > (SELECT given_name FROM vars))
OR family_name > (SELECT family_name FROM vars)
ORDER BY family_name, given_name, email
LIMIT ?;

For ‘page before’ queries, it’s the same thing but in reverse order.

Full-text search queries #

There isn’t much left to do for full-text search. The /:lang/contacts endpoint receives a query string consisting of one or more words or word prefixes. In the database layer, we parse this string and adapt it to the FTS5 query syntax. For example, G O'Mal becomes "G"* + "O'Mal"*, and will match a contact whose name is Grace O’Malley. We return a maximum of one page of full-text search results, so the route passes nil cursors to the template, which removes the pagination links via an htmx out-of-band swap.

Integration tests #

There’s nothing remarkable about this application’s unit tests (unless your name happens to be 曼玉-فاتن), but the integration tests are more interesting. I’ve written two sets of integration tests: one for the database layer, and one for the HTTP endpoints. In both cases, each test runs with a freshly initialised SQLite database containing test data from a fixture. As mentioned above, the HTTP endpoint tests parse the HTML responses and check the contents of specific elements. We use a Gin router in test mode, and net/http/httptest records the router’s responses, which we can parse using goquery. For example, to test a GET request to /:lang/contacts/:id/edit, the endpoint for editing an existing contact, we can do something like this:

func TestEditContactGet(t *testing.T) {
  server := InitServer(t, "test-contacts")

  recorder := httptest.NewRecorder()
  request, _ := http.NewRequest("GET", "/en/contacts/1/edit", nil)
  server.ServeHTTP(recorder, request)

  test.Equals(t, recorder.Code, http.StatusOK)
  doc, err := goquery.NewDocumentFromReader(recorder.Result().Body)
  if err != nil {
    t.Fatalf("Failed to parse HTTP response: %v", err)
  }

  familyNameLabel := doc.Find("label[for='family_name']").Text()
  test.Equals(t, familyNameLabel, "Family name")

  familyName, _ := doc.Find("#family_name").Attr("value")
  test.Equals(t, familyName, "Aaa")

  // ...
}

Since we’re just looking at the text in labels and form fields using simple CSS selectors, tests like this are unlikely to be affected by changes in the graphic design of the page.

PostgreSQL #

We don’t need to do much to adapt our app to use PostgreSQL instead of SQLite.

Full-text search #

(If you’ve already read the Rust version of this post, you can skip this section.)

CREATE TABLE IF NOT EXISTS contact (
  id BIGINT GENERATED ALWAYS AS IDENTITY PRIMARY KEY,
  family_name TEXT NOT NULL,
  given_name TEXT NOT NULL,
  phone TEXT,
  email TEXT UNIQUE NOT NULL,
  textsearch tsvector GENERATED ALWAYS AS (
    to_tsvector(
      'simple',
      COALESCE(family_name, '') || ' ' || COALESCE(given_name, '')
    )
  ) STORED
);

CREATE INDEX textsearch_idx ON contact USING GIN (textsearch);

The SQL query to find matching rows uses the full-text match operator, @@:

SELECT id, family_name, given_name, phone, email
FROM contact, to_tsquery($1) query
WHERE query @@ textsearch
ORDER BY ts_rank_cd(textsearch, query)
LIMIT $2;

The syntax of the string that we pass to to_tsquery just needs minor adjustments to work with Postgres: if the user types G O'Mal, we pass the argument 'G':* & 'O''Mal':*.

Integration tests with Testcontainers #

Running our integration tests with Postgres takes a bit more work. To run each test with a freshly initialised database as before, we can use Testcontainers and run Postgres in an ephemeral Docker container. We’ll run our integration tests synchronously, so we only need one database per test run. The Testcontainers postgres module has a useful feature for this: we can initialise the database with our migrations and our fixture, then save a snapshot of the database. After each test completes, we restore the database to the snapshot so it’s ready for the next test. The code for this is too long to quote here, but you’ll find it in integration/common.go.

Conclusion #

By using a declarative validation framework, and deterministic code generation in a statically typed language, we’ve kept boilerplate to a minimum, and improved the chances that if we make a mistake, we’ll get an error at compile time rather than at run time. This should help keep our code maintainable and prevent bugs.

Next, let’s write the same app in Rust.

See the N+1 selects problem and selecting all columns without thinking about it. ↩︎

Writing safe hypermedia apps for a resource-constrained world

Mon, 05 Oct 2026 07:00:00 +0000

When I was developing web applications in the 1990s, it was common for the server to generate HTML pages using templates, and a lot of template engines were written for this (including one by me). Starting in the early 2000s, Single-Page Applications (SPA) became widespread. In an SPA, the server mainly returns JSON, and a JavaScript (or TypeScript) framework generates the HTML in the browser. I’ve worked on some applications like this, and it struck me that they’re cumbersome to maintain, partly because the JavaScript frameworks are huge and complex and change rapidly.

Nowadays there is increasing dissatisfaction with the SPA approach, not only because of the maintenance burden, but also because these applications have to load megabytes of JavaScript over the network before they can do anything, and this makes them slow, especially on slow network connections. Meanwhile, HTML has become more powerful, making JavaScript less necessary. As a result, there is renewed interest in generating HTML on the server.

One of the drawbacks of generating HTML on the server in the 1990s was the need to reload the whole page whenever anything changed. Now that limitation is gone, thanks to tiny JavaScript libraries like htmx. By adding declarative attributes to HTML elements, we can allow all sorts of user interactions to fetch fragments of HTML from the server and replace only the parts of the page that need to change. Applications that work this way are called Hypermedia-Driven Applications, and there’s a book about them, called Hypermedia Systems.

The book shows how to construct a contact management application using htmx, and the source code is provided in Python. Since the book’s focus is on htmx, the server code is very simple: it only supports English, contacts are stored in a file rather than in a database, there’s very little input validation, pagination uses page numbers, and there are no tests.

In a resource-constrained world, it makes sense to write applications like this in languages that compile to machine code and use relatively little memory.¹ We’d like to have the option of using a very lightweight database like SQLite or a more powerful one like PostgreSQL. To help prevent bugs and security vulnerabilities, we can favour statically typed languages with an emphasis on built-in safety features. And we’d like our application to be easy to maintain.

Given these preferences, let’s make the example app from Hypermedia Systems more realistic. We’ll give it an internationalised UI, a relational database with a full-text search index, better input validation, cursor-based pagination, and UI tests. In the next few posts, I’ll present two implementations, one in Go and one in Rust, with either SQLite or PostgreSQL. I’ll explain the design choices I’ve made in each version, and cover some interesting techniques that have wider applications. I’m not a UX designer, so I won’t try to improve on the user interface described in the book, but I’ll have a few words to say about facilitating the work of UX designers. I won’t cover most of the details that are explained in Hypermedia Applications, but you can read these articles without having read the book. Let’s think about some techniques we can use to write web applications that are safe, resource-efficient, and easy to maintain:

(Each article can be read independently of the other.)

The relationship between energy efficiency and the choice of programming language is an active area of research; see Energy Efficiency of Programming Languages. ↩︎

What makes a good extension language?

Thu, 07 May 2026 11:01:09 +0200

‘Scripting languages are designed for gluing’, wrote John Ousterhout, the creator of Tcl, in 1998.¹ What Ousterhout had in mind was writing an application in a scripting language and having it delegate certain functions to components written in system programming languages. A similar idea inspired the wrapper generator SWIG, developed for scientific computing.² Nowadays, Python is widely used in this way, thanks to its vast ecosystem of modules written in system programming languages.

But there is another way to combine a scripting language with a system programming language: embed an interpreter in an application. In 1994, Paul Dubois asked us to imagine a program ‘whose user interface is a programming language’.³ As the creators of Lua wrote in 1996:

There is increasing demand for customizable applications. As applications became more complex, customization with simple parameters became impossible: users now want to make configuration decisions at execution time; users also want to write macros and scripts to increase productivity. In response to these needs, there is an important trend nowadays to split complex systems in two parts: kernel and configuration. The kernel implements the basic classes and objects of the system, and is usually written in a compiled, statically typed language, like C or Modula-2. The configuration part, usually written in an interpreted, flexible language, connects these classes and objects to give the final shape to the application.⁴

Several examples come to mind: Basic in Excel (an example of of end-user programming), JavaScript in web browsers and in Visual Studio Code, Emacs Lisp, and Lua in many programs, such as Redis, Neovim, pandoc, and game engines. Embedded languages are often called extension languages, a term that suggests that the host application only delegates a few tasks to scripts, but in some cases, most of the application’s functionality is implemented in the interpreted language (about 1.5 million lines of code in Emacs), and the kernel is much smaller. Andy Wingo, a contributor to the extension language Guile, wrote in 2009:

Emacs, when it was first created in its GNU form in 1984, was a new take on the problem of ‘how to make a program’. The Emacs thesis is that it is delightful to create composite programs based on an orthogonal kernel written in a low-level language together with a powerful, high-level extension language.

What are the requirements for an embedded language? The authors of the Lua paper list a few, focusing on the design of the language itself. Here I would like to ask: are they the same as the requirements for a glue language? One difference can be found in something outside language design, namely the language’s library ecosystem. Python’s rich ecosystem is important for making it a good glue language, but Lua is widely used as an extension language even though its library ecosystem is much smaller and less active. This is partly because embedded scripts often rely on functionality provided by the host application, and don’t need many libraries. Sometimes, as with web browsers and Emacs, a library ecosystem develops that’s specific to the application.

Wingo explains that the success of Emacs inspired the creation of Guile, a Scheme dialect intended to be a more general-purpose extension language for the GNU project. While Guile has found some niches, it hasn’t been widely adopted. Is that because it’s a Lisp? The authors of the Lua paper wrote: ‘Lisp cannot be called user-friendly when it comes to customization. Its syntax is rather crude for non-programmers.’ The designers of Emacs Lisp disagreed; in their view,

The language should be accessible to a wide audience, so that as many people as possible can adapt Emacs to their own needs, without being dependent on the availability of someone with technical expertise. For example, the Introduction to Programming in Emacs Lisp tutorial targets users with no programming experience.⁵

I’m inclined to think that use of extension languages is driven more by the popularity of the applications that support them than by their syntax. When Microsoft chose Basic as the extension language for Excel in 1993, people who wanted to extend Excel learned Basic because they had to. In 1995, Netscape created JavaScript as the extension language for its web browser, and people who wanted to write scripts that ran in web browsers then learned JavaScript because they had to. As a result, twenty years later, a large number of programmers knew JavaScript. When Microsoft then developed Visual Studio Code and needed an extension language, they could have chosen Basic again, but this time they chose JavaScript, no doubt because many developers already knew it.

Lua doesn’t have such an application to drive its adoption, but it has something else. I suspect that many of the developers who use Lua do so not because they have to, but because they have strict requirements concerning performance and/or executable size, and therefore choose LuaJIT, which is used in ‘network switches, set-top boxes and other embedded devices’. (If you want those features but you also want to program in Lisp, there’s Fennel, a Lisp that compiles to Lua.)

Extension languages are often designed to be embedded in C programs, and need substantial glue code (‘bindings’) to integrate them into programs written in other languages. A lack of convenient bindings therefore becomes an obstacle to using the extension language. For example, there are good libraries for embedding Python and Lua in Rust programs, but nothing comparable for Guile. (Instead, there are several Scheme interpreters implemented in Rust.) Lack of IDE support can be another obstacle.

In short, what makes a good extension language can depend as much on its environment (the host application’s functionality and constraints) and ecosystem (including libraries, JITs, bindings, and IDE support) as on the language itself.

Bibliography #

Beazley, David M., and Peter S. Lomdahl. 1996. “Lightweight Computational Steering of Very Large Scale Molecular Dynamics Simulations.” In Proceedings of the 1996 ACM/IEEE Conference on Supercomputing. IEEE Computer Society. https://doi.org/10.1145/369028.369136.

Dubois, Paul F. 1994. “Making Applications Programmable.” Computers in Physics 8 (1): 70–73. https://doi.org/10.1063/1.4823265.

Ierusalimschy, Roberto, Luiz Henrique de Figueiredo, and Waldemar Celes Filho. 1996. “Lua—An Extensible Extension Language.” Software: Practice and Experience 26 (6): 635–52. https://www.lua.org/spe.html.

Monnier, Stefan, and Michael Sperber. 2020. “Evolution of Emacs Lisp.” Proceedings of the ACM on Programming Languages 4 (HOPL). https://doi.org/10.1145/3386324.

Ousterhout, John K. 1998. “Scripting: Higher Level Programming for the 21st Century.” IEEE Computer 31 (3): 23–30. https://www.tcl-lang.org/doc/scripting.html.

Ousterhout, ‘Scripting’. ↩︎
Beazley and Lomdahl, ‘Lightweight Computational Steering of Very Large Scale Molecular Dynamics Simulations’. ↩︎
Dubois, ‘Making Applications Programmable’. ↩︎
Ierusalimschy et al., ‘Lua—An Extensible Extension Language’. ↩︎
Monnier and Sperber, ‘Evolution of Emacs Lisp’. ↩︎

Optimising PostgreSQL queries with an open dataset

Tue, 10 Mar 2026 13:49:30 +0100

Overview #

This post is a brief illustration of some database optimisation techniques, using the BRÉF (Base Révisée des Élu·es de France / Revised database of representatives elected in France), which contains data about representatives elected in France from 1948 to 2020. The BRÉF is based on French public data, in which a team of researchers has corrected many errors and inconsistencies. It’s published as a PostgreSQL database that consists of several tables and offers many opportunities for joins and optimisations.

Here I’ll be using this database to develop and optimise a SQL query with several joins, to get information about mayors and the places where they were elected. I’ll build up the query from simpler parts, which I’ll optimise as I go along. For this exercise, I’ll assume that we can’t change the database schema, and that we’re willing to pay some cost, in storage space and update performance, to improve query performance (particularly considering that the database isn’t very large and that updates are rare). Therefore I’ll focus on what we can do by creating indexes. To identify and test potential optimisations, we’ll have two sources of information:

PostgreSQL’s query plans, which we can get by using its EXPLAIN command.
Performance tests run with my sqlstopwatch tool, which runs queries many times and reports the average response time of each query.

Optimising a simple query #

In the terminology used in the BRÉF,

a mandate is an elected position (member of parliament, senator, etc.), while a function is an executive role that an individual holds during a mandate. For example, a person could have a mandate as a city councillor, and at the same time have the function of mayor on the council.

Hence there is a Mandate table and a Function table, and we’ll start with those.

Let’s start by finding all the functions of type MAIRE (mayor). We’ll use EXPLAIN ANALYZE to have Postgres run the query, then tell us the query plan (EXPLAIN) along with how long it took (ANALYZE).

EXPLAIN ANALYZE
SELECT "MandateId"
FROM "Function"
WHERE "FunctionType" = 'MAIRE'

QUERY PLAN
Seq Scan on "Function"  (cost=0.00..5596.96 rows=113829 width=4)
  (actual time=0.010..17.567 rows=114192 loops=1)
  Filter: (("FunctionType")::text = 'MAIRE'::text)
  Rows Removed by Filter: 158845
Planning Time: 0.217 ms
Execution Time: 20.136 ms

The query plan is a tree, and each node has an estimated cost in arbitrary units. Note that ‘the cost of an upper-level node includes the cost of all its child nodes’. Seq Scan means that the database scans the whole Function table looking for matching strings. To avoid this, let’s create an index on the FunctionType column and run EXPLAIN ANALYZE with the same query again. (In this exercise we’ll just use B-Tree indexes, which the CREATE INDEX command uses by default.)

1
2
3

CREATE INDEX index_function ON "Function" (
    "FunctionType"
)

The result is a different query plan:

QUERY PLAN
Bitmap Heap Scan on "Function"  (cost=1286.60..4893.46 rows=113829 width=4)
  (actual time=3.764..11.449 rows=114192 loops=1)
  Recheck Cond: (("FunctionType")::text = 'MAIRE'::text)
  Heap Blocks: exact=822
  ->  Bitmap Index Scan on index_function
      (cost=0.00..1258.14 rows=113829 width=0)
      (actual time=3.589..3.589 rows=114192 loops=1)
        Index Cond: (("FunctionType")::text = 'MAIRE'::text)
Planning Time: 0.314 ms
Execution Time: 13.551 ms

Postgres has done a bitmap scan, which Postgres developer Tom Lane explained in this mailing list post.

A bitmap scan fetches all the tuple-pointers from the index in one go, sorts them using an in-memory “bitmap” data structure, and then visits the table tuples in physical tuple-location order…. If the bitmap gets too large we convert it to “lossy” style, in which we only remember which pages contain matching tuples instead of remembering each tuple individually. When that happens, the table-visiting phase has to examine each tuple on the page and recheck the scan condition to see which tuples to return.

This is more efficient than the full table scan, but we’re still paying the cost of fetching the tuples (rows) from the table and rechecking the scan condition. Since we’re only reading the MandateId column, we can put it in the index as well, to create a covering index for this query:

CREATE INDEX index_function ON "Function" (
    "FunctionType"
)
INCLUDE (
    "MandateId"
)

With that, we get a different query plan:

QUERY PLAN
Index Only Scan using index_function on "Function"
  (cost=0.42..4490.35 rows=115003 width=4)
  (actual time=0.027..7.354 rows=114192 loops=1)
  Index Cond: ("FunctionType" = 'MAIRE'::text)
  Heap Fetches: 0
Planning Time: 0.184 ms
Execution Time: 9.025 ms

Now Postgres is able to do an index-only scan, because all the columns we need are in the index. It looks like an improvement, but let’s run a performance test to check. Let’s test several queries that search for various functions, including mayor. We’ll run the test first with no index, then with each of the two indexes above.

SELECT "MandateId"
FROM "Function"
WHERE "FunctionType" = 'MAIRE'
FETCH FIRST 750 ROWS ONLY

SELECT "MandateId"
FROM "Function"
WHERE "FunctionType" = 'MAIRE DELEGUE'
FETCH FIRST 750 ROWS ONLY

SELECT "MandateId"
FROM "Function"
WHERE "FunctionType" = 'PRESIDENT DU CONSEIL REGIONAL'
FETCH FIRST 750 ROWS ONLY

SELECT "MandateId"
FROM "Function"
WHERE "FunctionType" = 'PREMIER ADJOINT AU MAIRE'
FETCH FIRST 750 ROWS ONLY

Clearly, having an index is much better than not having one. Compared to the index that just contains FunctionType, the index that includes MandateId (and thus enables an index-only scan) doesn’t make a big difference for some values of FunctionType, but it’s more than twice as fast for MAIRE.

Sorting #

We’d like to get the results sorted by function start date:

EXPLAIN ANALYZE
SELECT "MandateId"
FROM "Function"
WHERE "FunctionType" = 'MAIRE'
ORDER BY "FunctionStartDate"

With no index, the database says this query takes about twice as long as it did without the sort:

QUERY PLAN
Sort  (cost=15354.29..15644.28 rows=115995 width=8)
      (actual time=32.564..37.109 rows=114192 loops=1)
  Sort Key: "FunctionStartDate"
  Sort Method: external merge  Disk: 2016kB
  ->  Seq Scan on "Function"
      (cost=0.00..5596.96 rows=115995 width=8)
      (actual time=0.016..17.797 rows=114192 loops=1)
        Filter: (("FunctionType")::text = 'MAIRE'::text)
        Rows Removed by Filter: 158845
Planning Time: 0.398 ms
Execution Time: 40.132 ms

Interestingly, if we use the index from the previous section with this query, it doesn’t change the query plan this time. Postgres has decided that our index isn’t likely to provide any benefit for this query. To make the index worth using here, let’s add the FunctionStartDate column to it, so the entries for each function type are sorted by start date in the index:

CREATE INDEX index_function ON "Function" (
    "FunctionType",
    "FunctionStartDate"
)
INCLUDE (
    "MandateId"
)

QUERY PLAN
Index Only Scan using index_function on "Function"
  (cost=0.42..4743.89 rows=115995 width=8)
  (actual time=0.052..9.744 rows=114192 loops=1)
  Index Cond: ("FunctionType" = 'MAIRE'::text)
  Heap Fetches: 0
Planning Time: 0.432 ms
Execution Time: 12.275 ms

Now once again we have an index-only scan, which looks more efficient. Let’s run a performance test comparing the previous index with this one. We’ll use the queries from the previous section, with ORDER BY "FunctionStartDate" added to each one.

It seems that the more rows to be sorted, the greater the advantage of using an index that’s already sorted in the order that the ORDER BY clause requests. The advantage is very significant for MAIRE (114,192 rows) and PREMIER ADJOINT AU MAIRE (37,511 rows), but less significant in the other cases, which return far fewer rows.

Adding joins #

Now that we have MandateId, we can get foreign keys from the corresponding row of the Mandate table, and do joins to get more information about the person who was elected and the place where the election occurred. Let’s start with the municipality (commune). The Area table contains data about different sorts of administrative divisions, including municipalities. Before we can find out more about the municipality, we need its ID from the Mandate table. So let’s start with that and just look at the query plan, without running the query:

EXPLAIN
SELECT
    func."FunctionStartDate",
    m."AreaId"
FROM 
    "Function" func,
    "Mandate" m
WHERE
    func."FunctionType" = 'MAIRE'
    AND func."MandateId" = m."MandateId"
ORDER BY func."FunctionStartDate"
FETCH FIRST 750 ROWS ONLY

QUERY PLAN
Limit  (cost=0.85..968.89 rows=750 width=36)
  ->  Nested Loop  (cost=0.85..147087.20 rows=113957 width=36)
        ->  Index Only Scan using index_function on "Function" func
            (cost=0.42..4664.34 rows=113957 width=8)
              Index Cond: ("FunctionType" = 'MAIRE'::text)
        ->  Index Scan using "pk_Mandat" on "Mandate" m
            (cost=0.43..1.25 rows=1 width=36)
              Index Cond: ("MandateId" = func."MandateId")

The database is still doing an Index Only Scan for the Function table, but it’s doing an Index Scan for the Mandate table, using that table’s primary key index. This means that it traverses the index until it finds the matching MandateId, then fetches the corresponding row from the Mandate table to get the AreaId. We can add an index containing AreaId to get an index-only scan:

`1`	`CREATE INDEX index_mandate ON "Mandate" ("MandateId") INCLUDE ("AreaId")`

Now the query plan looks like this:

QUERY PLAN
Limit  (cost=0.85..505.54 rows=750 width=10)
  ->  Nested Loop  (cost=0.85..76310.71 rows=113401 width=10)
        ->  Index Only Scan using index_function on "Function" func
            (cost=0.42..4643.26 rows=113401 width=8)
              Index Cond: ("FunctionType" = 'MAIRE'::text)
        ->  Index Only Scan using index_mandate on "Mandate" m
            (cost=0.43..0.63 rows=1 width=10)
              Index Cond: ("MandateId" = func."MandateId")

It looks like the cost has gone down. Let’s run a performance test, once again with different values for FunctionType, to compare the primary key index with the covering index. In both cases we’ll keep our best-performing index on Function.

The covering index doesn’t make a huge difference, but it still seems to be worth using.

Now that we have the AreaId of the municipality, we can get its name and its official geographical code (which is distinct from its postcode). We’d also like to get the name of the person who was elected.

SELECT
    func."FunctionStartDate",
    area."AreaCode" as code_commune,
    area."AreaName" as commune,
    indiv."FirstName",
    indiv."LastName"
FROM "Function" func
JOIN "Mandate" m ON func."MandateId" = m."MandateId"
JOIN "Area" area ON m."AreaId" = area."AreaId"
JOIN "Individual" indiv ON m."IndividualId" = indiv."IndividualId"
WHERE func."FunctionType" = 'MAIRE'
ORDER BY func."FunctionStartDate"
FETCH FIRST 100 ROWS ONLY

Since we have several joins, I’m using the JOIN keyword to make the query a bit easier to read. (This doesn’t have any effect on the query plan.) And since the query returns more columns now, I’m selecting fewer rows so the performance test results aren’t dominated by the time it takes to return the query results.

For the join between Mandate and Individual, we’ll add IndividualId to our index on Mandate:

1
2
3

CREATE INDEX index_mandate ON "Mandate" ("MandateId") INCLUDE (
    "AreaId", "IndividualId"
)

Then we’ll add covering indexes on Area and Individual for our query:

CREATE INDEX index_area ON "Area" ("AreaId") INCLUDE ("AreaCode", "AreaName");

CREATE INDEX index_indiv ON "Individual" ("IndividualId") INCLUDE (
    "FirstName", "LastName"
);

With this, we get the following query plan:

QUERY PLAN
Limit  (cost=1.57..132.35 rows=100 width=38)
  ->  Nested Loop  (cost=1.57..148131.29 rows=113274 width=38)
        ->  Nested Loop  (cost=1.15..96170.50 rows=113274 width=34)
              ->  Nested Loop  (cost=0.85..82875.64 rows=113274 width=22)
                    ->  Index Only Scan using index_function on "Function" func
                        (cost=0.42..4636.71 rows=113274 width=8)
                          Index Cond: ("FunctionType" = 'MAIRE'::text)
                    ->  Index Only Scan using index_mandate on "Mandate" m
                        (cost=0.43..0.69 rows=1 width=22)
                          Index Cond: ("MandateId" = func."MandateId")
              ->  Memoize  (cost=0.30..0.32 rows=1 width=24)
                    Cache Key: m."AreaId"
                    Cache Mode: logical
                    ->  Index Only Scan using index_area on "Area" area
                        (cost=0.29..0.31 rows=1 width=24)
                          Index Cond: ("AreaId" = (m."AreaId")::text)
        ->  Index Only Scan using index_indiv on "Individual" indiv
            (cost=0.42..0.46 rows=1 width=28)
              Index Cond: ("IndividualId" = (m."IndividualId")::text)

All the data comes from the indexes; the tables aren’t involved at all. If we don’t create those last two indexes, we get Index Scan rather than Index Only Scan on Area and Individual. Let’s compare the performance of this query with and without those two indexes.

There’s very little difference, but let’s keep the indexes.

Let’s now compare the performance of this last query with no indexes (apart from the primary key indexes that were created by default) and with all the indexes we’ve made:

This query is 14 to 24 times faster with the indexes than without them.

More joins #

Let’s try to get the name of the department that the municipality belongs to. We can add a join on the Inclusion table to find the Area that represents the department:

EXPLAIN
SELECT
    func."FunctionStartDate",
    area1."AreaCode" as code_commune,
    area1."AreaName" as commune,
    area2."AreaName" as departement,
    indiv."FirstName",
    indiv."LastName"
FROM "Function" func
JOIN "Mandate" m ON func."MandateId" = m."MandateId"
JOIN "Area" area1 ON m."AreaId" = area1."AreaId"
JOIN "Inclusion" incl on m."AreaId" = incl."IncludedAreaId"
JOIN "Area" area2 on incl."IncludingAreaId" = area2."AreaId"
JOIN "Individual" indiv ON m."IndividualId" = indiv."IndividualId"
WHERE func."FunctionType" = 'MAIRE'
AND area2."AreaType" = 'Département'
ORDER BY func."FunctionStartDate"
FETCH FIRST 100 ROWS ONLY

QUERY PLAN
Limit  (cost=35292.57..35304.24 rows=100 width=51)
  ->  Gather Merge  (cost=35292.57..35351.14 rows=502 width=51)
        Workers Planned: 2
        ->  Sort  (cost=34292.55..34293.18 rows=251 width=51)
              Sort Key: func."FunctionStartDate"
              ->  Nested Loop  (cost=30010.19..34282.96 rows=251 width=51)
                    ->  Parallel Hash Join  (cost=30009.76..34167.82 rows=251 width=47)
                          Hash Cond: (func."MandateId" = m."MandateId")
                          ->  Parallel Index Only Scan using index_function on "Function" func
                              (cost=0.42..3980.29 rows=47235 width=8)
                                Index Cond: ("FunctionType" = 'MAIRE'::text)
                          ->  Parallel Hash  (cost=29971.32..29971.32 rows=3042 width=47)
                                ->  Hash Join  (cost=2914.37..29971.32 rows=3042 width=47)
                                      Hash Cond: ((m."AreaId")::text = (area1."AreaId")::text)
                                      ->  Parallel Seq Scan on "Mandate" m
                                          (cost=0.00..24881.12 rows=572112 width=22)
                                      ->  Hash  (cost=2911.27..2911.27 rows=248 width=43)
                                            ->  Nested Loop
                                                (cost=967.16..2911.27 rows=248 width=43)
                                                  ->  Hash Join
                                                      (cost=966.88..2832.29 rows=248 width=19)
                                                        Hash Cond:
                                                        ((incl."IncludingAreaId")::text =
                                                         (area2."AreaId")::text)
                                                        ->  Seq Scan on "Inclusion" incl
                                                            (cost=0.00..1607.70
                                                             rows=98170 width=15)
                                                        ->  Hash
                                                            (cost=965.40..965.40
                                                             rows=118 width=19)
                                                              ->  Seq Scan on "Area" area2
                                                                  (cost=0.00..965.40
                                                                   rows=118 width=19)
                                                                    Filter:
                                                                    (("AreaType")::text =
                                                                    'Département'::text)
                                                  ->  Index Only Scan using index_area
                                                      on "Area" area1
                                                      (cost=0.29..0.32 rows=1 width=24)
                                                        Index Cond:
                                                        ("AreaId" =
                                                         (incl."IncludedAreaId")::text)
                    ->  Index Only Scan using index_indiv on "Individual" indiv
                        (cost=0.42..0.46 rows=1 width=28)
                          Index Cond: ("IndividualId" = (m."IndividualId")::text)

What has happened here? Postgres has decided that a hash join would be more efficient than adding another nested loop. Instead of using our index on the Mandate table, it plans to read about 500,000 rows into a hash table in memory. It uses the index sorted by date on Function, but since a hash table isn’t sorted, it then has to sort the results again. If we run EXPLAIN ANALYZE, we can see that the query takes about 600 ms to run; just getting the department’s name has made our query hundreds of times slower.

We have several options:

Give the query planner a better option than the hash join.
Optimise the hash join by selecting fewer columns (which isn’t really an option in this case) or fewer rows (we could, for example, add a WHERE condition like func."FunctionStartDate" < DATE '1975-01-01').
Do the join in the application. In this case, since there are only 101 departments in France, and they change very rarely, the application could read them on startup. Each municipality’s geographical code (which we already have from the previous query) contains the code of its department, so it would be easy for the application to look up the department in memory.

Let’s try the first option. We can add this index:

1
2
3

CREATE INDEX index_inclusion ON "Inclusion" ("IncludedAreaId") INCLUDE (
    "IncludingAreaId"
);

Now the query plan is back to nested loops:

QUERY PLAN
Limit  (cost=1002.32..10605.10 rows=100 width=51)
  ->  Nested Loop  (cost=1002.32..58907.08 rows=603 width=51)
        ->  Gather Merge  (cost=1001.89..58630.47 rows=603 width=47)
              Workers Planned: 2
              ->  Nested Loop  (cost=1.87..57560.84 rows=251 width=47)
                    ->  Nested Loop  (cost=1.58..57476.99 rows=272 width=41)
                          ->  Nested Loop  (cost=1.28..53851.27 rows=107699 width=37)
                                ->  Nested Loop  (cost=0.85..36596.43 rows=47235 width=22)
                                      ->  Parallel Index Only Scan using index_function
                                          on "Function" func
                                          (cost=0.42..3980.29 rows=47235 width=8)
                                            Index Cond: ("FunctionType" = 'MAIRE'::text)
                                      ->  Index Only Scan using index_mandate on "Mandate" m
                                          (cost=0.43..0.69 rows=1 width=22)
                                            Index Cond: ("MandateId" = func."MandateId")
                                ->  Memoize  (cost=0.43..0.46 rows=2 width=15)
                                      Cache Key: m."AreaId"
                                      Cache Mode: logical
                                      ->  Index Only Scan using index_inclusion
                                          on "Inclusion" incl
                                          (cost=0.42..0.45 rows=2 width=15)
                                            Index Cond: ("IncludedAreaId" = (m."AreaId")::text)
                          ->  Memoize  (cost=0.30..0.34 rows=1 width=19)
                                Cache Key: incl."IncludingAreaId"
                                Cache Mode: logical
                                ->  Index Scan using "pk_Territoire" on "Area" area2
                                    (cost=0.29..0.33 rows=1 width=19)
                                      Index Cond:
                                      (("AreaId")::text = (incl."IncludingAreaId")::text)
                                      Filter: (("AreaType")::text = 'Département'::text)
                    ->  Index Only Scan using index_area on "Area" area1
                        (cost=0.29..0.31 rows=1 width=24)
                          Index Cond: ("AreaId" = (m."AreaId")::text)
        ->  Index Only Scan using index_indiv on "Individual" indiv
            (cost=0.42..0.46 rows=1 width=28)
              Index Cond: ("IndividualId" = (m."IndividualId")::text)

The cost is still rather high, though, and EXPLAIN ANALYZE says the query takes about 12 ms. A performance test shows that it can be 10-20 times slower than our previous query:

It would be more efficient to keep the previous query, and do the join for the departments in the application.

Conclusion #

This exercise has demonstrated a few important points about database optimisation:

Each time we change a query, we have to check the query plan again, and run a performance test, to see what effect the change has on performance.
Different queries may need different indexes.
An index-only scan can provide the best query performance in some cases, but we have to test it to find out.
Sometimes the best optimisation is to request less data.

You can find all the scripts and configuration I used for this exercise in this git repository.

To learn more about this subject, I recommend Markus Winand’s website Use the Index, Luke!, and his book SQL Performance Explained.

Comparing the performance of SQL queries

Thu, 29 Jan 2026 18:25:18 +0100

I wrote a little Rust program, sqlstopwatch, to compare the performance of different SQL queries. I was inspired by the article Benchmarking SQL published by the jOOQ project, where they write the test code in SQL. I wanted something similar, but with a few additional requirements:

A command-line program with a simple TUI showing a progress bar.
The program should support PostgreSQL, MySQL, and SQLite.
It should read the SQL queries and test parameters from a configuration file specified on the command line.
It should print the results as a table in the terminal or save them as a CSV file that I can use to generate a chart.

The TUI was easy to make with Ratatui. To interact with the database, I used the async database library sqlx, which provides a bundled SQLite library.

I considered using TOML for the config file, but the file has to specify the database credentials. In testing environments, credentials may have to be loaded from environment variables or from files on a mounted file system, but TOML doesn’t support carrying out those sorts of operations. Moreover, you might want to generate a number of similar queries and compare them, as I did in this database optimisation project. Therefore, it’s better if the config file is a program in a scripting language. My exercise in writing an interpreter for a scripting language had given me a new appreciation for the Lua virtual machine. So the config file is a Lua script, which I find more convenient to write in Fennel, a Lisp dialect that compiles to Lua. Thanks to the mlua library, it was very easy to embed a Lua interpreter in the program, and to use Serde to convert Lua tables into Rust structs.

Writing a Bittorrent client in Rust

Thu, 15 Jan 2026 15:16:01 +0100

Overview #

To continue learning Rust, I decided to implement a client for Bittorrent, the communication protocol for peer-to-peer file sharing. The Bittorrent clients I’ve used have complex GUIs that seem to be designed for users who download and upload torrents all day, rather than for the occasional user like me, who just wants to get one file. And like many GUI applications, Bittorrent clients tend to accumulate files that take up disk space in obscure directories. I do many things in the terminal, and when I need to download a file, my reflex is to use curl. So I thought, why not make something like curl for Bittorrent? Thinking about curl’s progress meter, it occurred to me that I could do something similar with the TUI library Ratatui, which I’d been wanting to try.

A Bittorrent client seemed like a good opportunity to learn about asynchronous programming in Rust. I had read a lot of criticism of Rust’s async, but I found it pleasant to work with (using the Tokio runtime), and certainly more comfortable than the comparable frameworks that I’d used in Scala (Akka and Cats Effect). I particularly like that an async function can be run in an existing task or in its own task, unlike Scala’s Future, which is assigned a thread and starts running as soon as it’s constructed. Rust’s approach seems more conducive to refactoring and maintainability. At the same time, I’m glad not to have to deal with the complexity of using monads as in Cats Effect.

The result is Sayaca, named after a South American bird that plays an important role in seed dispersal (in Bittorrent terminology, ‘seeding’ means uploading).

Design #

My first task was to implement a parser and formatter for Bencode, the data format used in the Bittorrent protocol. Conceptually it’s similar to JSON, but it’s designed for binary data rather than text. I wanted to try using a parser combinator library, and had no trouble implementing the parser using nom. Since I wanted to be able to serialise and deserialise Rust structs as Bencode, I added Serde support. I’ve published the resulting crate as bside.

The main difficulty in designing the client was the messy state of the Bittorrent specifications. There are ambiguities (the word ‘piece’ means two different things) and implementations have diverged from the specs (e.g. peer IDs are not used to identify peers). A great deal of knowledge about the protocol is not in the official specs, but in other documents scattered around the Web. There is a long list (not updated since 2018) of draft protocol extensions that are supported by many clients. A so-called encryption extension (which actually seems to describe a type of obfuscation) seems to be widely implemented, but as far as I can tell, since 2023, its specification has been preserved only in the Internet Archive’s Wayback Machine. There is a draft BitTorrent Protocol Specification v2 (not updated since 2017), which attempts to correct some of the original protocol’s flaws, such as its reliance on the SHA-1 hash function (which has not been considered secure since 2005). I could have wandered for a long time in a labyrinth of documents and in the source code of existing clients. To keep things simple, I decided to implement only the original BitTorrent Protocol Specification and the IPv6 tracker extension.

When saving data to disk, a Bittorrent client should perform well with large files, be able to interrupt a download and resume it later (which means that it has to note which pieces have been downloaded), and handle torrents that include multiple files. To meet these requirements, and to keep things transparent for the user, I chose to save downloaded data in an intermediate file (with the extension .sayaca), in the same directory where the completed file(s) will be saved. The intermediate file contains a bit field indicating which pieces have been downloaded (the same bit field that we have to send to peers anyway). Sayaca uses memory mapping to read and write this file; this should provide good I/O performance with large files. When the download is complete, it copies the data from the intermediate file into the destination file(s), and the intermediate file is deleted on exit if it is no longer needed.

Since a Bittorrent client is I/O-bound, I decided to use an event-driven architecture in which one task does most of the work, as in Redis. This main task delegates all the network I/O to other tasks, which it communicates with via channels. This was simple to implement, and had the advantage of making it easy to write unit tests for the main task.

Much of the work that a Bittorrent client does amounts to a sort of bookkeeping. In Sayaca, this job is the responsibility of the Bookkeeper. For example, in the recommended algorithm for selecting file pieces to request, the client tries to request the rarest pieces first. This requires a data structure that keeps track of the pieces that each peer has, as well has how rare each piece is, among other things. This can be done in different ways; I chose to use an ordered map, ordered by rarity, to store the IDs of pieces and the sets of peers that have them. The Bookkeeper can then easily find a piece, randomly chosen from among the rarest pieces, along with a randomly chosen peer that has that piece and is accepting requests from us. Thus, much of the client’s work involves updating the Bookkeeper when new information arrives and querying it to find out what to do next.

I like TUIs, and it was easy to make a simple one with Ratatui. But since the client is a library that communicates with its UI via channels, it would be straightforward to wrap the client in a different UI, e.g. a more accessible CLI.

Alternatives to this approach #

I chose to use Tokio because I wanted to learn about it and about async, but I think the program probably wouldn’t have been very different if I had used threads instead, with the same communication via channels.

Mona Ja‘bub, Leading Society Towards Change

Sun, 14 Dec 2025 00:00:00 +0000

[English version of the text published in Spanish.¹]

Mona Salem Said Ja‘bub, قيادة المجتمع نحو التغيير: التجربة التربوية لثورة ظفار (1969-1992) [Leading Society towards Change: The Educational Experience of the Dhofar Revolution (1969-1992)], Beirut: مركز دراسات الوحدة العربية [Centre for Arab Unity Studies], first edition 2010, second edition 2023, 368 pages.

This first book by Omani historian Mona Ja‘bub has earned praise in the Arab world for its approach to the social and cultural history of the Dhofar revolution in Oman. Published by a prestigious Arab publishing house specialised in social science, the book was initially withdrawn by the Omani authorities from the Muscat International Book Fair in 2012, but it returned the next day, a sign of the relaxation of censorship. The book focuses on gender issues and on the circulation of ideas. Through an analysis of historical events little known outside the Arab world, it makes an original contribution to the study of the global reception of Maoism and Marxism in the 1960s and 1970s.

From 1968 to 1975, a Marxist-Leninist revolution took place in the province of Dhofar in the Sultanate of Oman, against the Sultan Said and his successor Qaboos, who were dependent on the British Empire. At that time, Dhofar had a subsistence economy, and its population lived in extreme poverty. On the coast there was fishing, on the plains there was some agriculture, mainly cultivation of incense trees, and in the highlands cattle were raised. Even after oil exports began in 1967, the state collected taxes without providing any services to the population. Modern medicine was unknown, the only school was reserved for the Sultan’s court, and most of the inhabitants of Dhofar, and nearly all the women, were illiterate. In the Dhofari tribal system, land belonged to the tribes, not to individuals; some tribes enjoyed more prestige than others and had the privilege of bearing arms. At the bottom of the social hierarchy were people that the tribes and the Sultan had enslaved.

An armed Dhofari rebellion inspired by Arab nationalism, launched in 1965 by Dhofaris living in Kuwait and supported by the Nasser regime in Egypt, was discredited by the defeat of Egypt and its allies in the June 1967 war with Israel. Several months later, a group of Dhofari militants travelled to China to be trained as revolutionaries. They returned home convinced by Maoism, on a ship loaded with weapons and food. All this enabled Dhofari communists to take control of what came to be called the Popular Front for the Liberation of Oman and the Arab Gulf. (Since it changed its name several times, I will simply call it the Popular Front.) This turn to the left was strengthened by the unfailing support that the Dhofari revolution received from South Yemen, Oman’s neighbour, which in 1970 became the only communist country in the Arab world.

However, the application of Marxist theory to the local context was controversial. When the Popular Front commissioned a Bahraini Marxist to write a report on the class structure in Dhofar, the report concluded that there were no social classes in Dhofar, only tribes. The Popular Front’s leaders refused to accept this conclusion, and the report was not published. They acknowledged that there was no proletariat or bourgeoisie in Dhofar, but claimed that the tribes were in fact classes.

The Popular Front banned slavery in the liberated areas. It abolished tribal ownership of land, and all land became public property. Under the impetus of feminist activists within its ranks, it made women’s liberation one of the pillars of its doctrine. Women therefore took up arms in the revolutionary army, women’s literacy was made a priority, and polygamy was abolished.

But the militants believed that the social transformation they aspired to required not only decrees imposed by force, but also an education system capable of instilling in its pupils values and behaviours that went against tradition. Other researchers have examined the political aspects of the Dhofari revolution, but the originality of Mona Ja‘bub’s book lies in its focus on the educational activities of the Popular Front, particularly the primary school and the secondary school that it established in Yemen, near the Omani border. Ja‘bub interviewed many former activists and former pupils of these schools, and this enables her to paint an intimate portrait of an attempt to adapt Marxist theory to a context that was very different from the industrial economy that Marx had in mind.

The idea that Dhofaris belonged to the Arab nation was a fundamental part of the Popular Front’s ideology. Standard Arabic, which they saw as the national language of the Arabs, was therefore a central aspect of the culture that the revolutionary teachers sought to transmit to their pupils. In the Arab world, many different Arabic dialects, some of which are mutually unintelligible, are used in everyday life, while Standard Arabic, which few people can speak, is used mainly in writing and in formal discourse. However, the pupils spoke a Shehri, a distant cousin of Arabic, and apparently did not understand Standard Arabic or any Arabic dialect. At first, the teachers dealt with this situation by requiring the pupils to speak only in Standard Arabic (which would have been seen as strange in all other Arab countries) and by forbidding them to speak in Shehri, even outside the classroom; offenders were punished. If the pupils needed a word that they did not know in Arabic, they could only use gestures to express their meaning. The teachers told the pupils that this was part of their duty to their Arab homeland. Ja‘bub shows that the pupils not only obeyed these rules out of fear, but also truly internalised them. A former pupil recalls that one day, while talking to the cook at the school canteen, she inadvertently used a Shehri word, and immediately felt ashamed. She reported herself to the school administration, and only felt better after having apologised in front of all the other pupils. Later, a new head teacher repealed the ban on Shehri, and implemented a policy of promoting Dhofari culture.

The schools of the Popular Front were sorely lacking in resources during their early years. At first, there were no buildings, only tents. With no textbooks available, pupils copied their lessons by hand. For four years, the only book available in sufficient quantities for teaching reading and writing in Arabic was a translation of Mao Zedong’s Little Red Book, of which China supplied many copies with each delivery of weapons. It was also the Front’s most important political textbook, alongside texts by Marx, Engels, Lenin, Stalin, Ho Chi Minh, and Che Guevara. Mao was by far the most admired figure among the Dhofari people, and many wore Mao badges, not only, according to the author, because he sent them weapons and food, but also because his ideas corresponded to their concerns, because he had a well-developed conception of anticolonial struggle, and because he wrote for peasants.

The Marxism that was taught was nevertheless filtered in one respect: at least during the early years of the revolution, the Front’s leaders and teachers did not mention the Marxist critique of religion, probably to avoid conflict with a population composed of very devout Muslims. On the contrary, although they did not offer any religious instruction, they asserted that there was no contradiction between socialism and Islam. The author agrees with them on this point. In her view, before the revolution, Dhofari society was already based on a sort of tribal communism in which the state was nearly absent. She asserts that the population of the Arabian peninsula was already comfortable with the idea of the rule of the proletariat, since, she says, this idea appears in the Qur’an, and she cites this verse: ‘We wished to favour those who were oppressed in the land, to make them leaders, the ones to survive’ (28:5)². This has been one of the Islamic Left’s favourite verses since the 1970s.

In 1970, the UK replaced the Sultan of Oman with his son, Qaboos. While maintaining his father’s authoritarianism, Qaboos tried not only to defeat the revolution on the battlefield, but also to win over the population, by promoting economic development, building state schools, restoring some of the tribes’ privileges, proclaiming ceaselessly that Islam was the basis of his rule, and claiming that the Popular Front was against Islam. According to Ja‘bub, while this accusation was false for a long time, the Front’s leaders, believing that they had gained the population’s complete confidence, eventually said openly that religion is the opium of the people. In response, their own soldiers rebelled against them, and this led to a schism within the Popular Front. In Ja‘bub’s opinion, if the revolutionaries had continued their selective use of Marxist thought, adapting it to the local society and to its Islamic beliefs, they could have had more success than all the other leftist movements in the Arab world.

After the military defeat of the revolution in 1975, what happened to the social changes that it had brought about? Ja‘bub’s answers to this question are among the most interesting parts of the book. The revolutionaries encouraged and celebrated marriages between members of tribes and formerly enslaved people. The former pupils that Ja‘bub interviewed remember being delighted at these marriages. During the revolution, words associated with racism became repulsive to them, and they even refused to use the word ‘black’ to refer to a person’s skin colour. They proudly told the author that, since they were convinced at the time that tribalism was unjust and reactionary, they spontaneously stopped writing the name of their tribe when they wrote their name. But Ja‘bub observes that, when she met them for the first time, the first question they asked her was ‘What tribe do you belong to?’ and that that their conversation was peppered with remarks about the quality of this or that person’s tribe. Similarly, women educated by the Popular Front are now among the most conservative Dhofari women and the most attached to the outward signs of patriarchal tradition. On one hand, they have become doctors, teachers, and managers, but on the other hand, they have rejected the feminism that the Popular Front advocated and practised.

In Ja‘bub’s view, Marxist theory cannot be applied as is to the tribal social order that existed in Dhofar before the revolution. Nevertheless, there was social injustice, and many of the founders and leaders of the revolution belonged to the non-tribal social categories that suffered from this injustice. This book, which has become an important reference for historical research on the Arabian peninsula, is therefore an important contribution to debates about the relevance of Marxist concepts in different social contexts. It also provides a fascinating glimpse of a population’s ability to sincerely adopt revolutionary values in a quasi-state constructed by militants, only to abandon them with the arrival of the following state.

Bibliography #

Geer, Benjamin. 2025. “Mona Salem Said Ja‘bub, Qiyadat al-mujtama‘ nahw al-taghyir: al-tajriba al-tarbawiyya li-thawrat Dhufar (1969-1992) [Conducir la sociedad hacia el cambio: la experiencia educativa de la revolución de Dhofar].” Prismas. Revista de historia intelectual 29 (1): 412–14. https://doi.org/10.48160/18520499prismas29.1626.

Geer, ‘Mona Ja‘bub, Qiyadat al-mujtama‘ nahw al-taghyir’. ↩︎
M. A. S. Abdel Haleem’s translation. ↩︎

Writing a virtual machine in Rust

Sun, 12 Oct 2025 00:00:00 +0000

Overview #

To help me learn Rust, I decided to implement an interpreter for Lox, the scripting language in Robert Nystrom’s book Crafting Interpreters. I’ve been interested in scripting languages that are designed to be embedded in applications, and this seemed like a good way to learn more about how such languages are made.

Nystrom’s book walks you through two different implementations of the interpreter: a tree-walk interpreter in Java, and a bytecode virtual machine in C. I was more interested in the bytecode VM. Over the years, I’d written a couple of tree-walk interpreters, starting with Apache FreeMarker, as well as an optimising source-to-source compiler with type inference. But I hadn’t implemented a bytecode VM yet. The idea of it brought back good memories of doing assembly-language programming on the Apple II when I was fifteen years old. The instruction set for Nystrom’s virtual machine is simpler than the instruction set of the Apple II’s Motorola 6502 CPU, but the basic idea is the same. So I skipped the part of his book about the tree-walk interpreter and went directly to the part about the bytecode VM.

The result is OrganicLox. The design is broadly similar to the one in the book, but I’ve tried to write idiomatic Rust, with reasonable performance and no unsafe code. Functionally, OrganicLox should be identical to Nystrom’s clox: at least, it passes his 246 automated tests.

I decided against writing my own garbage collector, because everyone seems to agree that garbage collectors are particularly difficult to implement in Rust. To keep things simple, I used reference counting instead (via Rc). On the main branch, this means that cycles of object properties won’t be collected. On the with_dumpster_gc branch, I’ve replaced Rc with the dumpster crate, which uses reference counting but also deals with cycles.

My favourite parts #

I liked this book very much: Nystrom’s clear explanations, useful illustrations, easygoing writing style, and humour made the book a pleasure to read.

For me, some of the most interesting parts of his design, which draws heavily on the design of the Lua VM, involved the implementation of local variables and closures in a stack-based VM. I enjoyed learning that the compiler associates local variables with specific positions on the stack, so the VM only needs to know about stack positions, not about the variable names they’re associated with.

The implementation of upvalues, the variables that are captured by closures (and may need to be heap-allocated as a result), was tricky to get right. While fixing a bug, I was delighted to realise that a recursive call to a function defined in a local scope requires the function’s closure to be one of its own upvalues.

What I wished for #

Nystrom alternates between top-down and bottom-up explanations. The bottom-up explanations made it more difficult to translate his design into Rust, since I had to guess where he was going. Several times I had to refactor a lot of code because I had guessed incorrectly. Since implementing Lox in all sorts of programming languages seems to be a popular exercise, it would have been helpful to have more high-level notes about the design.

It was fun to learn about Vaughan Pratt’s top-down operator precedence parsing technique and to see that it’s actually possible to generate bytecode without first building an AST. But perhaps it would have been more useful to produce an AST and learn something about writing a second pass to optimise the bytecode.

Performance #

The main optimisation I did at the outset was to have the scanner intern all identifiers (using string_interner), so neither the compiler nor the VM need to do any string comparisons on identifiers.

After finishing Chapter 24, I spent a bit of time looking at performance, following the advice in The Rust Performance Book. I ran benchmarks with criterion, did some profiling with perf and hotspot, adjusted the build configuration, and inlined some functions.

I then looked at some other Rust implementations of Lox, and noticed that everyone seems to have found it difficult to approach the performance of Nystrom’s clox, and that the implementations that succeeded in doing so contained a lot of unsafe code, which I wanted to avoid. I highly recommend Manuel Cerón’s blog post about his impressive, determined, and partly successful attempt to match the speed of clox in Rust by writing more and more unsafe code.

For comparison, here are the results of some of Nystrom’s benchmarks run on these implementations:

Nystrom’s clox
The safe version of Cerón’s loxido
The unsafe version of loxido
OrganicLox with Rc
OrganicLox with the dumpster garbage collector

OrganicLox with Rc is two to four times slower than clox, which is perhaps in the typical range for interpreters written in Rust compared to interpreters written in C. The version with dumpster is much slower. It seems clear that a fast garbage collector is essential for a high-performance VM.

How much do these differences matter? Benchmarks are designed to simulate extreme scenarios, e.g. by using long loops or deep recursion, or by creating a lot of work for the garbage collector, so they’re not necessarily representative of real use cases. The scripts run by applications with an embedded interpreter are often short and do very little work. We might get a more realistic idea of how a VM performs in such a scenario by running Nystrom’s integration test suite, which runs 246 short Lox scripts and checks their output. Here’s a comparison:

In an application where every CPU cycle counts, the differences in performance between these different implementations could be very significant. But in other use cases, they might not be.

Alternatives to this approach #

It’s tempting to try adding a just-in-time compiler (JIT) to improve the performance of the interpreter, either by writing one from scratch or by using an existing backend like LLVM or Cranelift. Examples in Rust include Roto and Dora. But even if you do that, it’s not easy to achieve high performance. In these benchmarks, Python’s new JIT has no measurable effect on performance (see also these Dora benchmark results).

That’s one reason to consider using an existing interpreter that already has years of engineering invested in it, instead of creating a new one. Lua is one popular option, but it isn’t fast enough for some applications. The widely used LuaJIT VM is an order of magnitude faster than the standard Lua VM for some use cases, but it’s a tracing JIT, which has advantages and disadvantages. The Lua developers have an interesting paper in which they discuss the limitations of this approach, and propose a different solution (an additional statically typed language, a bit like Cython).

For some use cases, if embedded scripts don’t do much work, their performance might not be critical. If they do so much work that their performance is insufficient, another option is not to embed an interpreter at all. Cloudflare, which had invested heavily for over 20 years in scripts using LuaJIT embedded in OpenResty, recently replaced them with Rust code.

Calculating centrality indices with the Paris public transport network

Fri, 29 Aug 2025 00:00:00 +0000

I thought it would be interesting to try some basic network analysis of the Paris métro (rapid transit) and RER (commuter rail) systems, starting with some centrality indices. These are metrics that rank the nodes in a graph according to their importance, for some definition of importance. For example, in a transport network, the centrality of a station could give an indication of the likelihood that passengers will pass through that station as part of their itinerary. A public transport authority could use this information to gauge how inconvenient it would be if particular stations were closed, e.g. because of flooding or maintenance, and to help it identify the most effective ways of improving the resilience of the network. Here I’ll use Python, networkx, and SQLite.

Constructing the graph #

Here I’ve chosen to combine the métro and RER into a single graph. The first challenge is to construct this graph as an edge list and a node list. The agencies that manage the Paris public transport networks don’t provide such a dataset, but they do provide a GTFS dataset representing the train schedules for the next 30 days. (Thanks to Oliver Blanthorn for pointing this out.) The GTFS data represents the trips (sequences of stops) that trains are scheduled to take along their routes (métro or RER lines). To construct the edge list for the network, we can traverse each trip, storing an edge between each stop and the next one. Some trips go in one direction, others go in the opposite direction, and many don’t cover the whole line, but by traversing them all, we should build up a complete directed graph of the network. We’ll have to traverse about 64,000 trips, containing a total of nearly 1.5 million stops, and a lot of the data will be redundant for our purposes, but this doesn’t matter as long as we can process it in a reasonable amount of time.

We therefore have to implement an ETL (Extract, transform, load) process. I wrote a shell script that downloads the GTFS data, as well as several files from a dataset called Référentiel des lignes de transport en commun d’île-de-France, which we will also need. Since all these files are in CSV format, and use unique identifiers to link together the data in different files, it will be easier to work with this data if we use a relational database. In this shell script, I import the necessary files into an intermediate SQLite database, and create some database indexes to speed up the queries in the transformation step. This takes about 21 seconds on my laptop, and the size of the resulting database is 2.1G.

Finally, I’ve written a Python script for the transform and load phases. It reads the intermediate database and builds the output database, which just contains a table of edges and a table of nodes. Since the output database is small (132K), the script creates it in memory for better performance, and writes it to disk when it’s complete. This script takes about 8 seconds to run.

In the Référentiel, links between the métro and and RER networks are represented by zones de correspondance and pôles d’échange, which can include multiple stations connected by corridors. To include these relations in the graph, I’ve chosen to add edges between all the stations in the same group.

In the output database, the graph is a directed multigraph, because two stations can be connected by more than one line. In such cases, we can see those two stations as having a stronger connection than two stations connected by only one line. Other things being equal, more lines mean more passengers travelling between those two stations. We’ll keep this information about mutiple edges when we do our calculations.

Drawing parts of the graph #

First let’s draw some of the data to see if it looks correct. We can write a function that draws a transport network graph. Here we don’t care about the geographical positions of the stations, we just want to see that we have the right stations with the right links between them. To help us see which line is which, we can use the colours defined for the transport network map, which are given in the GTFS dataset, plus black for the edges I added for the zones de correspondance and the pôles d’échange.

%config InlineBackend.figure_formats = ['svg']
import sqlite3
import pandas as pd
import networkx as nx
import numpy as np
import matplotlib.pyplot as plt
import itertools as it


def draw_graph(G, title, height, label_x_offset):
    plt.figure(figsize=(12, height))
    plt.margins(x=0.18, y=0)
    plt.title(title)

    pos = nx.nx_agraph.graphviz_layout(G, prog="dot")

    # See https://networkx.org/documentation/stable/auto_examples/drawing/plot_multigraphs.html
    connectionstyle = [f"arc3,rad={r}" for r in it.accumulate([0.15] * 4)]

    nx.draw_networkx_nodes(G, pos, node_size=30, node_color="#000000")

    edge_colors = [
        f"#{G[source_node_id][target_node_id][edge_key]["route_color"]}"
        for source_node_id, target_node_id, edge_key in G.edges
    ]

    nx.draw_networkx_edges(
        G, pos, edge_color=edge_colors, connectionstyle=connectionstyle
    )

    label_pos = {
        node_id: (node_pos[0] + label_x_offset, node_pos[1])
        for node_id, node_pos in pos.items()
    }

    text = nx.draw_networkx_labels(
        G,
        label_pos,
        labels=nx.get_node_attributes(G, "node_name"),
        horizontalalignment="left",
        verticalalignment="bottom",
        font_size=6,
    )

    for _, t in text.items():
        t.set_rotation(15)

Let’s try it with the métro lines 1 and 4, and the RER A:

DATABASE = "../../../notebook-data/paris-rail-data/paris-graph.db"

conn = sqlite3.connect(f"file:{DATABASE}?mode=ro")

nodes = pd.read_sql_query(
    """select distinct
        nodes.node_id, nodes.node_name
    from nodes
    inner join edges on
        nodes.node_id = edges.source_node_id or
        nodes.node_id = edges.target_node_id
    where
        (edges.route_type = 1 and edges.route_name = '1') or
        (edges.route_type = 1 and edges.route_name = '4') or
        (edges.route_type = 2 and edges.route_name = 'A') or
        (edges.route_type is null and
            (edges.route_name = 'PdE Châtelet - Les Halles' or
            edges.route_name = 'ZdC Gare de Lyon'))""",
    conn,
    index_col="node_id",
)

edges = pd.read_sql_query(
    """select * from edges where
        (edges.route_type = 1 and edges.route_name = '1') or
        (edges.route_type = 1 and edges.route_name = '4') or
        (edges.route_type = 2 and edges.route_name = 'A') or
        (edges.route_type is null and
            (edges.route_name = 'PdE Châtelet - Les Halles' or
            edges.route_name = 'ZdC Gare de Lyon'))""",
    conn,
).replace({np.nan: None})

G = nx.from_pandas_edgelist(
    edges,
    source="source_node_id",
    target="target_node_id",
    edge_attr=True,
    create_using=nx.MultiDiGraph(),
)

node_attributes = nodes.to_dict(orient="index")
nx.set_node_attributes(G, node_attributes)

draw_graph(
    G,
    title="Métro lines 1 and 4 and the RER A",
    height=20,
    label_x_offset=6,
)

That looks correct.

Plotting results #

The centrality functions in networkx return a dictionary whose keys are node IDs and whose values are centrality scores. Let’s define a function that takes such a dictionary and plots a bar chart of the top 25 stations in descending order of centrality.

def centrality_chart(centrality_dict, title):
    names_and_scores = [
        (node_attributes[item[0]]["node_name"], item[1])
        for item in centrality_dict.items()
    ]

    top_25 = sorted(names_and_scores, key=lambda t: t[1], reverse=True)[:25]

    top_25_df = pd.DataFrame(
        top_25,
        columns=["Station", "Centrality score"],
    ).set_index("Station")

    ax = top_25_df.plot.barh(title=title)
    ax.invert_yaxis()

Degree centrality #

First let’s read the whole graph:

nodes = pd.read_sql_query(
    "select * from nodes",
    conn,
    index_col="node_id",
)

edges = pd.read_sql_query(
    "select * from edges",
    conn,
).replace({np.nan: None})

conn.close()

G = nx.from_pandas_edgelist(
    edges,
    source="source_node_id",
    target="target_node_id",
    edge_attr=True,
    create_using=nx.MultiDiGraph(),
)

node_attributes = nodes.to_dict(orient="index")
nx.set_node_attributes(G, node_attributes)
print(f"Loaded {len(G.nodes)} nodes and {len(G.edges)} edges.")

Loaded 563 nodes and 1363 edges.

The simplest centrality index is degree centrality: for each node, we can count its inbound links (for in-degree centrality) or its outbound links (for out-degree centrality), and normalise the result by dividing by $n - 1$, where $n$ is the number of nodes. In our case, the results are the same for in-degree and out-degree centrality.

1
2

in_degree_centrality_dict = nx.in_degree_centrality(G)
centrality_chart(in_degree_centrality_dict, "In-degree centrality")

We can see immediately that the stations with the highest scores are mostly métro stations rather than RER stations. One possible explanation could be the apparently more radial structure of the RER network compared to the métro. Besides being located on more than one line, several of these stations (such as Havre-Caumartin) belong to a pôle d’échange.

Eigenvector centrality #

With eigenvector centrality, each node gets a score that is proportional to the sum of the scores of its neighbours. This way, a node can get a high score by having many neighbours and/or by having high-scoring neighbours. Given an undirected graph of $n$ nodes, for a node $i$ whose set of neighbours is $M(i)$, the eigenvector centrality $x_i$ is defined as

$$ x_i = \frac{1}{\lambda} \sum_{j \in M(i)} x_j $$

where $\lambda$ is a constant. We can rewrite this equation using the network’s adjacency matrix $A$, in which $a_{ij}$ is 1 if there is an edge that links nodes $i$ and $j$, and 0 otherwise:

$$ x_i = \frac{1}{\lambda} \sum_{j = 1}^n a_{ij} x_j $$

This is equivalent to

$$ \mathbf{x} = \frac{1}{\lambda} A\mathbf{x} $$

where $\mathbf{x}$ is a vector whose elements are the centrality scores. We can write this as

$$ A\mathbf{x} = \lambda\mathbf{x} $$

which means that $\lambda$ is an eigenvalue of $A$ and $\mathbf{x}$ is the corresponding eigenvector. Given the eigenvalues and eigenvectors of the matrix, choosing the appropriate eigenvector is simple, because (according to the Perron–Frobenius theorem), for a matrix (like the adjacency matrix) that has no negative elements, only the eigenvector corresponding to the largest eigenvalue can be chosen to have no negative elements. With a directed graph like our transport network, we can choose a left eigenvector or a right eigenvector. Usually the left eigenvector is chosen (and this is what networkx does), so that a node’s score is based on its inbound edges rather than its outbound edges. This doesn’t make much difference in our case, though, because nearly every inbound edge in our network has a matching outbound edge.

When eigenvector centrality is used with a directed graph, the graph must be strongly connected. This is true of our transport network, because every station is reachable from every other station:

`1`	`nx.is_strongly_connected(G)`

True

Here we use the networkx.eigenvector_centrality_numpy function, which first constructs the adjacency matrix. In a directed multigraph such as this one, if there are $n$ edges in the same direction between two nodes, networkx represents them as the value $n$ in the matrix, which is equivalent to a single edge with weight $n$. It then gets the relevant eigenvector, and returns a dictionary containing each node ID and the corresponding element of the eigenvector.

1
2

eigenvector_centrality_dict = nx.eigenvector_centrality_numpy(G)
centrality_chart(eigenvector_centrality_dict, "Eigenvector centrality")

If we compare these results to the degree centrality results, we can see that Havre-Caumartin, which is in the middle of the six-station Paris Saint-Lazare - Opéra pôle d’échange, gets a higher score because it’s adjacent to the two highly-ranked stations Saint-Lazare and Opéra. Several other stations, such as Saint-Augustin, Auber, and Haussmann Saint-Lazare, also get higher scores for the same reason.

Betweenness centrality #

Betweenness centrality counts the number of times a node is on one of the shortest paths (i.e., those that pass through as few nodes as possible) between each pair of other nodes. For a node $v$, it is defined as

$$ c_B(v) =\sum_{s \neq v \neq t} \frac{\sigma(s, t|v)}{\sigma(s, t)} $$

where $s$ and $t$ are any other nodes in the network, $\sigma(s, t)$ is the number of shortest paths from $s$ to $t$, and $\sigma(s, t|v)$ is the number of those paths that pass through $v$. In a transport network, a station with a high value for $c_B(v)$ is one that many passengers are likely to pass through if they choose one of the shortest paths to their destination.

1
2

betweenness_centrality_dict = nx.betweenness_centrality(G)
centrality_chart(betweenness_centrality_dict, "Betweenness centrality")

Most of the stations on this list are RER stations rather than métro stations, which makes sense, because the RER covers more distance between stops, so if the same journey can be done via métro or via RER, the shortest path will usually be via RER. It’s not surprising to see Châtelet les Halles at the top of the list, given its geographically central location and the large number of lines that pass through its pôle d’échange. It also looks like there are other configurations that can give a station a relatively high rank. For example, Juvisy is on a long section of the RER C where few stations have connections to other lines, and it connects to a similar section of the RER D, so it seems likely that many shortest paths have to pass through it.

Conclusion #

This has just been a brief exploration of a few centrality indices for this network, with some examples of how different definitions of a node’s importance result in different rankings. All the scripts used here can be found in this blog’s Git repository, along with the Jupyter notebook from which I generated this page. Please contact me if you have any questions or suggestions.

The words ‘Turk’, ‘Arab’, and ‘Greek’ in the Ottoman Empire in 1899

Fri, 04 Apr 2025 00:00:00 +0000

While reading Khaled Khalifa’s beautiful, elegiac novel No One Prayed Over Their Graves, I noticed something that may not matter in the context of the novel, but is worth thinking about if you’re interested in the history of social categories that are taken for granted today. The narrator talks about a party thrown by a Christian woman in Istanbul to welcome the new century on the night of 31 December 1899:

He liked Maggie’s idea of welcoming a unique moment with a raucous party that she had begun preparing a month ahead of time. She invited Greek musician friends and a few of her old friends, mostly Christian Turks working in small public roles, or Greeks over sixty who were still lighthearted and loved dancing. They would reminisce about their past with deep grief and plot how to return to their villages in Greece, but year by year they mouldered away in the big city, turning their hands over and remembering that most of their villages had been burned to the ground by Ottoman troops. (p. 142)¹

Elsewhere he talks about a Christian maid from Iraq and a Muslim maid working in the same house in Aleppo near the end of the 19th century:

Before they fell asleep, Margot would remember her youth and tell Um Kheir about the young men who used to wait for her to pass so they could have the boon of a single glance from her…. Hundreds of times Margot retold stories about Muslim youths, Kurds and Arabs and Turks, who had falen in love with her…. (p. 102)

Were the words ‘Turk’, ‘Arab’ and ‘Greek’ used in this way in the Ottoman Empire in 1899? I was sceptical, remembering things like this observation by historian Keith David Watenpaugh about the inhabitants of Aleppo in 1918:

With few exceptions, the question of who was an Arab and who was a Turk rarely arose in the city and would have been considered absurd. Most urbane, literate Aleppines would have bristled at being designated mere ‘Arabs’ or ‘Turks’, terms which in pre-nationalist consciousness connoted backward desert dwellers or rough country people.²

Sociologist Murat Ergin says something similar:

The term “Turk” in the Ottoman Empire was broadly used as a derogatory term referring to unrefined nomads in Anatolia…. [T]he cosmopolitan subjects of the empire living in large cities never considered themselves Turks. This was about to change toward the end of the nineteenth century….³

As for the meaning of the word ‘Greek’ in Anatolia, historian Anthony Bryer writes:

At the beginning of the [19th] century a Pontic Christian might describe himself in the old way as a Douberites, Phytianos or Tsitenos first, and then as a ‘Roman’ (Rum) Orthodox subject of the sultan; by the end of the century he was calling himself a Greek….⁴

So perhaps the meanings of these words were changing, at least for some people, in 1899. In any case, I would like to read a novel that focuses on how social categories changed in that period, and on what was lost with the spread of nationalist concepts.

Bibliography #

Bryer, Anthony. 1991. “The Pontic Greeks Before the Diaspora.” Journal of Refugee Studies 4 (4): 315–34. https://doi.org/10.1093/jrs/4.4.315.

Ergin, Murat. 2017. “Is the Turk a White Man?”: Race and Modernity in the Making of Turkish Identity. Brill.

Watenpaugh, Keith David. 2005. “Cleansing the Cosmopolitan City: Historicism, Journalism and the Arab Nation in the Post-Ottoman Eastern Mediterranean.” Social History 30 (1): 1–24. https://doi.org/10.1080/030710242000337260.

These quotations and page numbers are from Leri Price’s translation of the novel. For the original Arabic text and page numbers, see the Arabic version of this article. ↩︎
Watenpaugh, ‘Cleansing the Cosmopolitan City’, 9. ↩︎
Ergin, “Is the Turk a White Man?”, 71. ↩︎
Bryer, ‘The Pontic Greeks before the Diaspora’, 327. ↩︎

The Sound of Crying

Thu, 06 Mar 2025 00:00:00 +0000

In 2024 I did the English subtitles for the short film Mal partum (English title: The Sound of Crying), a beautiful documentary about postnatal depression. Émilie D. used her personal experience to make a film that provides practical knowledge that we can all use.

Shady Lewis, On the Greenwich Line

Fri, 10 Jan 2025 00:00:00 +0000

The novel On the Greenwich Line, by the Egyptian writer Shady Lewis, came out in 2019. Since reading it in Arabic in 2023, I have given the original Arabic version or the French translation to several of my friends (there’s also a German translation), and now I will be giving other friends the English translation that is due out next month. With great sensitivity, Lewis has written a very funny comic novel about immigration, racism, and bureaucracy in London, where he is a social worker. In a lecture at the Institut du Monde Arabe in Paris in April 2023, he said that when the news is so full of horrors every day that we become inured to them, humour can revive our feelings. The treatment did me a lot of good.

Sherif Younis, The Sacred March

Thu, 26 Sep 2024 00:00:00 +0000

[English version of the text published in Spanish.¹]

Sherif Younis, الزحف المقدس: مظاهرات التنحي وتشكل عبادة ناصر [The Sacred March: The Demonstrations Against Nasser’s Resignation and the Formation of His Cult], 1st edition, Cairo, Dar Mirit, 2005. 2nd revised edition, Beirut and Cairo, Dar al-Tanwir, 2012, 196 pages.

In June 1967, when Gamal Abdel Nasser’s military dictatorship in Egypt suffered a crushing defeat in the Six-Day War against Israel, Nasser immediately announced his resignation. Shortly afterwards, crowds demonstrated for two days across Egypt to beg him to change his mind, and Nasser responded to the outcry by remaining in office. In The Sacred March, the Egyptian historian Sherif Younis asks what led the Egyptian people to call for Nasser to remain in power. How did an authoritarian regime that was ruthless towards the population, and enforced political passivity by means of imprisonment and torture, manage to inspire such fervour? Some historians of Nasserism believe that these demonstrations were a response to the sincere emotions of the demonstrators, while others argue that they were organised by the regime. Taking an original approach to the study of Nasserism, Younis departs from these two interpretations. For him, the demonstrations in support of the leader, which were most likely spontaneous, were also a product of the ideology that the regime had put in place after the military coup of 1952.

A specialist in modern Egyptian history, Younis is one of the most important Egyptian historians of recent decades. To some extent, his work reflects a Foucauldian approach. He has translated classics such as Hayden White’s Metahistory (1973) from English into Arabic, as well as fundamental works on Egyptian history such as Khaled Fahmy’s All the Pasha’s Men (1997) and Timothy Mitchell’s Rule of Experts (2002). His research and translations often fuel intellectual debates in Egypt and the Arab world in general. Given the author’s importance, The Sacred March is undoubtedly one of the most significant contributions in recent decades to debates in Arabic on Nasserism.

The historiography of Nasserism is often apologetic or, in some cases, hagiographic, defending the Nasser regime on the grounds of its anti-colonialism, its Arab nationalism, or its allegedly socialist policies. Sometimes this historiography attempts to analyse Nasserism using popular theories such as Gramscian Marxism or modernisation theory, without managing to explain why the regime took one form rather than another, and why it lasted for almost two decades despite repeated failures. Unlike these approaches, Younis’s study focuses on ideology, and sets out to analyse Nasserism’s fundamental concepts. Through a detailed reading of primary sources (including official speeches, propaganda, and press articles by Nasserist intellectuals), Younis argues against Marxist analyses that associate Nasserism with a social class. On the contrary, in his view, the key to understanding this regime is that it did not represent any particular class or group apart from its own elite. This Nasserist elite consisted of a small clique of officers who, led by Nasser, took power in 1952 and put an end to British rule and the Egyptian monarchy. As Younis explains, the members of this group had diverse political preferences (such as Islamism or communism) and were united by a vague patriotism and an attraction to authoritarianism.

The author observes that with the creation of the Free Officers in September 1949, Nasser and his group renounced all links with political parties and organisations in Egypt, and committed themselves to strict independence from all forces in the political field. In their view, the very existence of political divisions among the Egyptian people was harmful and benefited the colonial powers. The Nasserist elite saw its first task as emptying the public arena of any political organisation that represented only part of the people and thus endangered the nation. By associating division with corruption, the regime presented itself as a means of purifying the people. Younis shows that Nasserism conceived of the people as a collection of atomised individuals, who were expected to line up behind Nasser as in a military parade. This is what the official discourse called the ‘sacred march’: the officers were supposed to embody the conscience and will of an ideal people. According to Younis, in this ‘theological’ conception of politics, two meanings of the term ‘people’ coexisted. On the one hand, there was an ideal and invisible people which, like a god, transmitted its revelation to the officers in a direct and inexplicable way. On the other hand, there was the actually existing people which, weak and corrupt, needed to be guided and transformed into an ideal people. The official ideology presented Nasser as the indispensable intermediary between these two phases of the people, i.e., as a prophet.

For Younis, this conception of Nasser became believable for the population mainly because of the regime’s foreign policy, which was attributed exclusively to the leader, and which aimed to fulfil some of the ambitions of the nationalist movement: to put an end to the humiliation of colonisation and to restore the glory of the homeland. The most important example of this foreign policy was the announcement of the nationalisation of the Suez Canal in 1956 and the failure of the Israeli, British and French invasion that followed.

Younis shows that Nasser’s authority was based on the idea that only the leader could maintain direct communication with the people. In theory, this communication had a supernatural quality; because of it, the representative institutions abolished by the regime were supposed to be superfluous. Political passivity and the loss of freedoms were seen as the price the Egyptian people had to pay for national dignity. According to Younis, this ideology helped to create a feeling of dependence among the population, which explains the demonstrations against Nasser’s resignation after Egypt’s defeat in the 1967 war. The emotional bond between the demonstrators and Nasser resulted from the regime’s restrictions on the political field (which stemmed from its rejection of a multi-party system), the fulfilment of some of the demands of the nationalist movement, and the effective dissemination of an official theology based on the cult of Nasser. Thus, accustomed to the idea that only Nasser could act and that they were merely spectators and admirers, the demonstrators demanded that ‘the only father they knew’ should remain in office.

Younis continued to develop this analysis in his book نداء الشعب: تاريخ نقدي للإيديولوجيا الناصرية [The Call of the People : A Critical History of Nasserist Ideology] (754 pages), published in 2012, in which he examines in detail the historical roots of this ideology, along with the institutions and policies that contributed to its spread and evolution from the 1952 coup until Nasser’s death in 1970. But The Sacred March already makes an important contribution to our understanding of Egyptian nationalism and the similarities between nationalism and religion. In particular, the use of the category ‘prophet’ to analyse the symbolic domination exercised by a dictator perceived as a ‘national hero’, as well as the idea that nationalism implies a deification of the people (or the nation), encourages us to explore a range of tools from the sociology of religion to analyse the social and ideological structure of nationalism, in the Egyptian context and beyond. This would also be an opportunity to look at another god, the ‘homeland’, to which Younis has paid less attention.

Bibliography #

Geer, Benjamin. 2022. “Sherif Younis, Al-zahf al-muqaddas: muzaharat al-tanahhi wa-tashakkul ‘ibadat Nasir [La Marcha Sagrada: las manifestaciones contra la dimisión de Nasser y la formación de su culto].” Prismas. Revista de historia intelectual 26 (2): 369–70. https://doi.org/10.48160/18520499prismas26.1350.

Geer, ‘Sherif Younis, Al-zahf al-muqaddas’. ↩︎

How to make Arabic PDFs with Markdown and pandoc

Sat, 21 Sep 2024 00:00:00 +0000

Here’s one way to write Arabic articles in Markdown format and typeset them as PDF files. This assumes you’re using Linux or a similar environment, and are comfortable with the command line. The basic idea is to to use pandoc to convert the Markdown to $\mathrm{\LaTeX}$ markup, and tell pandoc to run LuaTeX to generate the PDF.

You’ll need:

A Markdown editor that handles Arabic text well. I recommend Obsidian, which is user-friendly and has good built-in right-to-left text support. You can add the Obsidian RTL Plugin to get a few improvements.
A good Arabic font that’s convenient for text editing, such as Noto Naskh Arabic.
Some good Arabic fonts for your PDFs, such as Amiri for body text and Cairo for headings.
The LuaTeX typesetter, which you can get by installing TeXlive. If you’re not sure what you need, just install the whole TeXlive distribution.
pandoc
GNU Make
An ordinary text editor for writing Makefiles and the like, such as Visual Studio Code.

Top-level directory #

Obsidian expects you to put Markdown files in a directory tree that it configures as a ‘vault’. You can have multiple vaults, but it stores its own configuration and plugins separately in each vault.

Create a directory that will contain Arabic articles. In Obsidian, choose ‘Open folder as vault’ and open the folder you created. This just means that it creates a hidden subdirectory called .obsidian; don’t put anything in that subdirectory.

I prefer to make a subdirectory for each article, and to keep shared configuration in the top-level directory.

Shared pandoc variables #

In the top-level directory you created, create a file variables.md, containing pandoc variables to be used for all the articles.

---
documentclass: article
classoption: a4paper
colorlinks: true
indent: true
---

Shared LaTeX formatting #

Create a file header.tex, containing LaTeX formatting commands to be used for all the articles. Here I’m following the advice in the Babel documentation, and I’ve added some commands to make the output look better, by:

Increasing the default text size (Scale=1.2).
Putting more space between lines (baselinestretch).
Using the titling and titlesec packages to increase the sizes of titles and headings.

\usepackage[arabic, provide=*, bidi=basic, layout=counters.tabular]{babel}
\babelfont{rm}[Renderer=Harfbuzz, Scale=1.2]{Amiri}
\babelfont{sf}[Renderer=Harfbuzz, Scale=1.2]{Cairo}
\babelfont{tt}[Renderer=Harfbuzz, Scale=1.2]{DejaVu Sans Mono}

\renewcommand{\baselinestretch}{2}

\usepackage{titling}
\usepackage{titlesec}

\pretitle{
  \begin{center}
    \normalfont\sffamily\Huge
}
\posttitle{
  \end{center}
}

\titleformat{\section}
  {\normalfont\sffamily\large}
  {\thesection}
  {1em}{}[]

\titleformat{\subsection}
  {\normalfont\sffamily}
  {\thesection}
  {1em}{}[]

Shared Makefile rules #

Create a file called rules.mk, which contains the shared Makefile rules for all the articles:

.PHONY: all clean

INPUT = $(wildcard *.md)
OUTPUT = $(patsubst %.md, %.pdf, $(INPUT))

all: $(OUTPUT)

%.pdf: %.md ../header.tex ../variables.md
	pandoc --pdf-engine lualatex --include-in-header ../header.tex -o $@ ../variables.md $<

clean:
	rm -f $(OUTPUT) *~

Articles #

For each article, make a subdirectory, e.g. article-01, and put the following Makefile in it:

`1`	`include ../rules.mk`

Use Obsidian to write the article in Markdown in the same subdirectory. At the top of the Markdown file, put the pandoc variables that are specific to that article (these are called ‘properties’ in Obsidian), e.g.:

---
title: "هذا عنوان المقال"
author: "أنا"
date: "فبراير 2024"
---

To generate the PDF, open a terminal, change to the article-01 directory, and type make. Your PDF should be generated in the same directory.

كوكبنا

Fri, 20 Sep 2024 15:00:00 +0000

An Arabic-language podcast that I started in 2021 on ecology, migration, and inequality, and would like to continue someday: كوكبنا.شبكة.

Tondauer

Fri, 20 Sep 2024 14:00:00 +0000

In 2020 I started a project called Tondauer, to make Creative Commons licensed editions of sheet music. It happened because of the Covid-19 lockdown that year: I was learning to play Felix Mendelssohn’s Prelude Op. 104a, No. 2 (MWV U 123), because it suited my mood of restless frustration at the time.

I started with a 19th-century edition that could be downloaded for free. Then I bought an edition with fingerings, and wondered why it had different notes. I looked in vain for the composer’s manuscript, but found only an earlier draft. I had been reading R. Larry Todd’s biography of Mendelssohn, so I emailed him to ask if he knew where the manuscript of the final version was, and he replied that it hadn’t been found. This meant that the only sources were the first editions, which were published after the composer’s death.

I had learned something about how critical editions are made while working in the field of digital humanities. So I decided to make my own critical edition. Over several months, as libraries and archives gradually reopened, I got digital reproductions of the first German and British editions. The two editors had worked together to publish them simultaneously, but they didn’t have the same notes, either. In my edition, I tried to make a plausible explanation of what had happened, and reconstruct the composer’s intentions as well as possible. In the end I tried to make the edition I wished I had had, and added fingerings with the help of Penelope Roskell.

More recently I’ve been working with musicologist Florence Launay on editions of forgotten works by female composers, starting with Armande de Polignac’s Preludes for Piano, which had survived only in a single copy of the first edition, preserved by her descendants. Florence had photocopied it, and pianists had been recording the pieces using photocopies of that photocopy. Now that we’ve published a corrected edition with Florence’s preface, I hope more pianists will enjoy playing these Preludes.

SocioResources

Fri, 20 Sep 2024 13:00:00 +0000

My old blog (2013-2021) on the sociology of autonomy and the autonomy of sociology.

أبحاث لفتت نظري

Fri, 20 Sep 2024 12:00:00 +0000

My old Arabic blog (2009-2014) on which I summarised research that I thought was interesting: أبحاث لفتت نظري.

About

Fri, 20 Sep 2024 00:00:00 +0000

I’m a software developer and ex-academic (sociology, conceptual history, cognitive linguistics, Arabic culture, digital humanities). I live in Paris.

A short essay about my professional life, written in 2018: An Accidental Career.

A more detailed interview from 2017: Surviving sociology in Egypt and elsewhere.

Open-source software #

Codeberg

My current open-source software projects are on Codeberg.

The source code of this website.

Academic work #

ORCID

Open-access versions of all my publications are linked from my ORCID profile. In some cases you have to click on Show more detail to display the URL of the open-access version.

Contact information #

Signal: benjamingeer.11

Matrix: @benjamingeer:matrix.org

Email: ben@benjamingeer.eu

Mastodon: @benjamingeer@piaille.fr

Fanny Hensel – September

Fri, 22 Sep 2023 00:00:00 +0000

In 1841, Fanny Hensel (1805–1847) wrote Das Jahr (The Year), a large-scale piano cycle with a movement for each month of the year. She published ‘September’ in 1846 as part of her Op. 2, but Das Jahr was first published in its entirety in 1989. Subsequently, a magnificent revised autograph manuscript was discovered, in which each movement is copied on paper of a different colour, illustrated with a drawing by Hensel’s husband Wilhelm, and preceded by a poetic epigram. Furore Verlag has published a critical edition.

Considering the music, poetry, and illustrations together, Marian Wilson Kimber sees the work as depicting not simply the cycle of the year, but also the stages of an individual’s life.¹

The epigram for ‘September’ is from Goethe’s poem An den Mond (To the Moon):

Fließe, fließe, lieber Fluß
Nimmer werd’ ich froh.

Translation:

Flow, flow, dear river
Never will I be happy.

There are countless interpretations of the poem. For Tobias Klein:

The retreat into nature and especially into the solitude of the night is presented as consolation in the face of the disappointments and frustrations to which the individual is exposed in society…. The river appears in this context as a symbol of change and transience, as well as of ambivalence, as it can be both destructive … and invigorating….²

The piece displays ‘an artful application of the three-hand technique’, ‘in which a melody played by the thumbs in the middle register of the piano was surrounded by material assigned to the other fingers above and below, to create the illusion of three hands at the keyboard’.³ The harpist Elias Parish-Alvars (1808–1849) invented this technique, the pianist Sigismund Thalberg (1812–1871) transferred it to the piano,⁴ and it became widely used in the 1830s. In ‘September’, the upper voice seems to suggest the flowing water of the river.

The movement’s ingenious harmonic development⁵ is an example of how Hensel’s work anticipated the avant-garde of the second half of the 19th century.⁶

Bibliography #

Klein, Tobias. n.d. “Interpretation: An den Mond – Johann Wolfgang von Goethe.” https://lyrik.antikoerperchen.de/johann-wolfgang-von-goethe-an-den-mond,textbearbeitung,372.html.

Manning, Laurence. 2015. “Fanny Hensel, compositrice de l’avenir ? Anticipations du langage musical wagnérien dans l’œuvre pour piano de la maturité de Hensel.” Les Cahiers de la Société québécoise de recherche en musique 16 (1–2): 121. https://doi.org/10.7202/1039618ar.

Todd, R. Larry. 2010. Fanny Hensel: The Other Mendelssohn. Oxford University Press.

Walker, Alan. 1987. Franz Liszt: The Virtuoso Years, 1811–1847. Cornell University Press.

Wilson Kimber, Marian. 2008. “Fanny Hensel’s Seasons of Life: Poetic Epigrams, Vignettes, and Meaning in Das Jahr.” Journal of Musicological Research 27 (4): 359–95. https://doi.org/10.1080/01411890802384409.

Kimber, ‘Fanny Hensel’s Seasons of Life’. ↩︎
Klein, ‘Interpretation: An den Mond’. ↩︎
Todd, Fanny Hensel, 264, 233. ↩︎
Walker, Franz Liszt, 234. ↩︎
Todd, Fanny Hensel, 275–77. ↩︎
Manning, ‘Fanny Hensel, compositrice de l’avenir’. ↩︎

Chopin, Waltz, Op. 64, No. 2

Sat, 20 May 2023 00:00:00 +0000

Frédéric Chopin, Waltz, Op. 64, No. 2.

Benjamin Geer, piano.

Recorded 20 May 2023.

J. S. Bach, tr. Busoni – Ich ruf’ zu dir, Herr

Sun, 19 Sep 2021 00:00:00 +0000

J. S. Bach, Ich ruf’ zu dir, Herr, piano transcription by Ferruccio Busoni.

Benjamin Geer, piano.

Recorded 8 July 2021.

Trying to Be a Jazz Guitarist

Sun, 11 Apr 2021 00:00:00 +0000

I started playing jazz guitar around the time I went to university. I wanted to combine the improvisational tradition of jazz with non-tonal harmony and large-scale structures. But I thought I should start with bebop.

In 1995–96, while working as an English language teaching assistant in France, I had the opportunity to play in an amateur big band as well as a quintet:

Here are some earlier recordings, from 1991:

When I was about 30 I stopped playing, mainly out of frustration from my own slow progress and from the difficulty of trying to be a musician while needing to earn a living in some other way. I wish I had kept at it.