Hugo & Git

How I am using Hugo and Git to build, publish, and manage, this website and its public repositories.

I have spent a while now learning Hugo, the static website generator, and I have set up this website where I can publish some of my projects and thoughts. This post describes how I have set up this website using Hugo and Git so that it automatically rebuilds on changes to the website and so that readers can download my public repositories and check out my work. One of the goals I have had is to use Git in its distributed way with public read-access and private write-access without having to resort to use GitHub or similar services. A big selling point of Git is that it is distributed and easy to serve.

The Website Repository

This website is obviously stored in a repository. The second obvious thing is you don’t develop and write changes directly to the live website. Development happens in a separate clone from the live repository. The live repository for this website is stored at /srv/http/example.com. Normally when hugo build is executed it creates the directory public and this directory is the root of the website served by the web server.

It would however be nice to be able to push changes to the live copy, have it rebuild the website, and automatically publish the new changes. Usually when pushing in Git it is recommended to use bare repositories but bare repositories do not have an active work tree¹. This means that the actual tracked files are not available which is a problem since the files must be available for Hugo to build the website. But there is an alternative. The live repository can be a regular (non-bare) clone configured using:

$ git config set receive.denyCurrentBranch updateInstead

This enables Git to update the working tree directly on push. We can also recognize that it would be a bad idea to have this enabled by default. Suddenly your working tree in your repository could be changed by anyone with remote access.

 receive.denyCurrentBranch
	 If set to true or "refuse", git-receive-pack will deny a ref update to
	 the currently checked out branch of a non-bare repository. Such a push
	 is  potentially  dangerous because it brings the HEAD out of sync with
	 the index and working tree. If set to "warn", print a warning of  such
	 a  push  to  stderr, but allow the push to proceed. If set to false or
	 "ignore", allow such pushes with no message. Defaults to "refuse".

	 Another option is "updateInstead" which will update the  working  tree
	 if  pushing  into the current branch. This option is intended for syn‐
	 chronizing working directories when one side is not easily  accessible
	 via  interactive ssh (e.g. a live web site, hence the requirement that
	 the working directory be clean). This mode also comes  in  handy  when
	 developing  inside  a  VM  to test and fix code on different Operating
	 Systems.

	 By default, "updateInstead" will refuse the push if the  working  tree
	 or the index have any difference from the HEAD, but the push-to-check‐
	 out hook can be used to customize this. See githooks(5).

I then created a build script that runs on the post-receive hook. A problem is that the build can fail and it would be great if the currently live website is not destroyed in that case. So let’s rethink the public directory. public should not be a regular directory but a symbolic link to the most recent successful build. When the build script is executed Hugo should build the website into a build directory and, if successful, the old public link should be updated to the new build. This solves the problem of the build failing and it also solves the problem of updating the live contents without entering a weird in-between transition state². Updating a single symbolic link is a lot faster and simpler than updating multiple individual files.

#!/bin/sh

#
# This script creates a build in the directory 'builds' and, if successful,
# symlinks some extra stuff into the build before finally creating a link
# 'public' to the new build. After this script is finished 'public' is ready to
# be published.
#
# The build uses the short git commit hash as name.
#
# NOTE: The build directory will eventually fill up with old builds and should
#       be manually cleaned.
#

#
# NOTE: To run this script on every push, put
#
#   [receive]
#       denyCurrentBranch = updateInstead
#
#   [hook "build-website"]
#       event = post-receive
#       command = cd .. && GIT_DIR=".git" ./build.sh
#
# in .git/config.
#
# NOTE: When pushing the return value corresponds to the push, not the deploy.
#
# Read the hook output carefully to make sure that the deploy actually
# succeeded. If the push succeeds it does not mean that the deploy succeeded.
#

# Exit with non-zero if any of the following commands exits with non-zero.
set -e

# Get the short hash value for HEAD.
HASH="$(git rev-parse --short HEAD)"

#
# Increase COUNTER until a file with the name "$HASH-$COUNTER" is not found.
#
# NOTE: This is to prevent overwriting an existing build with the same commit
# hash. Most of the time this is not a problem but if the user tries to rebuild
# the most recent commit it would otherwise overwrite the live website.
#

COUNTER=1
while true; do
    TARGET="builds/$HASH-$COUNTER"
    if [ ! -e "$TARGET" ]; then
        break;
    fi
    COUNTER=$((COUNTER + 1))
done

echo "[$TARGET] Building website..."

hugo build --destination="$TARGET"

# Symlink extra stuff into the build target directory
ln -s "/srv/http/tmp"               "$TARGET"
ln -s "/srv/http/repos/public/"     "$TARGET/repos"
ln -s "/srv/http/braider/webpage"   "$TARGET/braider/prototype"

# Finalise the build by linking 'public' to the new build.
ln -sfn "$TARGET" public

echo "[$TARGET] ...website built and deployed successfully"

Note that the build script also writes some extra links into the build directory. More on those later.

It is often recommended that the post-receive hook should notify a build system which runs the build asynchronously because Git does not close its connection with the server until after the hook has finished. This is not a problem in this case since the build is fast. I also like having the hook output whether the deploy was successful or not. Unfortunately though the return value for the push depends on the success of the push and not the deploy.

Public Repositories

I have made the decision that the repositories that I have published should be independent of Hugo. Publishing new repositories or pushing updates to existing repositories should not require a rebuild of the website. Perhaps you would like to rebuild on every push in case you have some public README available but in that case I would probably look into Gitea instead.

Because of these requirements I cannot use the Hugo directories for static content or assets. If the content in these directories were updated then a new build would be required to publish the updates. The repositories cannot be put in public because it is now a link that updates after every build.

I instead created the directory /srv/http/repos/public for public repositories such as sigmar.git, pdftoc.git, and offsetof.git. It is not inside the served public link so a symbolic link to it is created when the website is being built. Because it is a simple symbolic link to a directory I can edit the contents of the public repositories and the changes will be automatically reflected on the live website. No rebuilding is necessary!

The public repositories are served read-only using the Git Dumb HTTP protocol. This is simple to setup. I just do a bare clone of the repository that I want to serve into /srv/http/repos/public and I don’t forget to enable and run the default post-update hook that runs git update-server-info. Since these are bare repositories without a working tree it is not necessary to configure denyCurrentBranch.

$ cd /srv/http/repos/public
$ git clone --bare <repo>
$ cd <repo>.git/hooks
$ mv post-update.sample post-update
$ ./post-update

The Git Dumb HTTP protocol offers public read-access but no write-access. Private write-access to these repositories is instead achieved through SSH (or the Local protocol if on the same machine). For example, if I want to push changes to a repository I can clone the public repository, make changes, and push the changes using SSH. It is the same repository just two different protocols to reach it!

$ git clone https://example.com/repos/<repo>.git
<make changes and commit>
$ git push host:/srv/http/repos/public/<repo>.git master

This obviously requires SSH to be setup and that the user has write-access to /srv/http/repos/public. One caveat to keep in mind is that the default remote origin will use the public read-only HTTP protocol after clone. You would probably want to change this to use the private SSH protocol instead of specifying the full URL in the command as in the example above.

You could make the argument that the repositories I have is part of the website and that they should be part of it explicitly since several of my posts link to them. If I ever move this website I also must remember to move all of the repositories. I have considered adding the repositories as submodules but that seems like it could be messy.

The Braider Repository

Special consideration was needed for the Braider repository. The Braider repository is not public except for the prototype. Because of this it is not next to the other repositories. Like the other repositories I also did not want to have to rebuild this website when updating the prototype.

But this is similar to this website! Think of Braider as a separate website and just link to it. I made a regular clone of its repository into /srv/http/braider and, like this website’s repository, I have configured it with updateInstead to enable pushing directly to it. In the repository is a directory website that contains the public prototype. Now we just have to add a link in public to it in the build script.

Conclusion

I now have a website that is capable of publishing repositories and thoughts. This post became more Git-heavy and less Hugo than originally intended. Perhaps I have some thoughts on Hugo to share in the future.

This is probably the easiest way to setup and use Git without having to resort to using central services and it is probably an underestimated and underused feature of Git. That said, however, GitHub does provide actual value in the form of issue tracking, communication, discovery, etc. But perhaps consider using GitHub in addition to a personal server like this. It can serve as a mirror and a backup of your repositories and you can probably use hooks to keep them in sync. This is probably even more important in recent times as GitHub have been suffering from outages and high profile exits.

Using locally hosted services such as GitLab and Gitea is also interesting. In my brief research it does seem like GitLab is a massive resource hog, whereas Gitea is much lighter. Another interesting approach is described in Chapter 4.4 for private repositories with a team of contributors using a shared locked-down user account over SSH.

References

The Pro Git book on git-scm is a good resource for Git. Especially relevant for this post is Chapter 4: “Git on the Server” and, to a lesser extent, Chapter 5: “Distributed Git”.