Embedding files in Go binaries

I recently ported github-to-omnifocus to Go. One new thing I learned is that it’s easy to embed files into Go binaries, extending Go’s “single file” deployment simplicity to include required non-Go resources.

In this case, I needed to call JXA scripts from Go. In github-to-omnifocus, these are required to interface with Omnifocus itself; they create, read, update and complete tasks. I wanted a single binary executable still, and feared I’d need to embed the JXA scripts as strings within the binary. That would make them hard to debug as I’d not be able to run them standalone. Fortunately, Go 1.16 introduced compiler support for embedding files into Go binaries, alongside an API to read them at runtime.

This is handled using the embed package. The way that embed works is that you sit the files to embed alongside your source file, and use a compiler directive inside your code to tell the compiler to embed the file. Within your code, the embedded resource can be read using a variable associated with the resource.

Embedding a single file

import "embed"

//go:embed hello.txt
var f embed.FS

The go:embed compiler directive tells the compiler to embed the file hello.txt that is in the same directory as the Go file. This file is made available to your code through the f variable that follows the go:embed directive.

To use the file, one uses the embed.FS variable to “read” the data:

data, _ := f.ReadFile("hello.txt")
print(string(data))

Embedding the contents of a directory

To embed a directory, the go:embed directive uses the path to the directory. Finding this, the compiler will embed all the files in the directory. In github-to-omnifocus, I have a directory of JXA scripts in a jxa folder alongside my Go file. In omnifocus.go, I embed them using:

import "embed"

var (
	//go:embed jxa
	jxa embed.FS
)

I can then read each file using a file path, including the original jxa folder:

jsCode, _ := jxa.ReadFile("jxa/ofaddnewtask.js")

If you are curious, this code shows how I execute a script by calling the osascript tool. I feed the script JSON and return JSON from it. That makes it easy to serialise input parameters and deserialise return values on the Go side. Overall, it’s relatively painless. In fact, you have to do the same thing in node.js: the JXA runtime is completely separate from the node.js one, even though both are running JavaScript!

More complex scenarios

The go:embed directive accepts patterns, and multiple file patterns, matched using path.Match. There can be multiple lines of go:embed before the variable that is used to access them. The example in the embed package tells you what you need to know:

import "embed"

// content holds our static web server content.
//go:embed image/* template/*
//go:embed html/index.html
var content embed.FS

In addition, a single file can be embedded “directly”: if the variable type is string or []byte instead of embed.FS and the go:embed is a single file reference, the compiler will ensure the variable is initialised with the contents of the file:

import _ "embed"

//go:embed hello.txt
var s string

Note the use of _ in the import line, which tells the compiler not to balk at the “unused” embed package – though it isn’t in the source file, it is used to initialise the variable so needs to be there. A wart, nothing more.

I was pleasantly surprised by the ease of embedding and using files within Go. The API feels very well thought out, and very concise. I can see how it’d make it easy to embed all the resources of a web service within the code, as shown by the more complex example.

Alongside Go, I write Python. Since I started to write Go four years ago, I’ve always been impressed by its deployment story: copy a single binary to where you want to run it. This is bliss compared to deploying a Python application. The simple, yet effective, way that Go allows embedding resources extends this simplicity in deployment. While I’d like to learn more about the inner workings – to understand how embedding large amounts of files, or large files, would affect performance and resource usage – for now, I’m content to leave that for another day.