mvidell.se

Offsetof

Mon, 18 May 2026 00:21:58 +0200

A common problem in programming is parsing different key-value pairs. These keys and values could easily be read into a map of some sort and each value lookup-ed when needed. But often these key-value pairs follow a schema. They have a known structure to them.

{
	"instrument_id": 10,
	"name": "ACME",
	"long_name": "ACME Corporation"
}

C has a way to represent structured data. Its the common struct.

struct InstrumentInfo {  // (1)
	i64 instrument_id;
	String name,
	String long_name,
};

The problem however is how to write the structured JSON data into the structured C data. Without any meaningful type introspection in C it becomes quite hard to map the keys to the fields in the struct. You could have a bunch of if-statements to check but that gets old quite quickly especially when you consider that you probably have multiple different structures to parse.

Enter offsetof(3).

NAME
     offsetof - offset of a structure member

LIBRARY
     Standard C library (libc, -lc)

SYNOPSIS
     #include <stddef.h>

     size_t offsetof(type, member);

DESCRIPTION
     The macro offsetof() returns the offset of the field member from the start
     of the structure type.

	 ...

offsetof is great. offsetof(InstrumentInfo, name) can be used to find the the byte offset of the field name in the struct InstrumentInfo. But it does not get us all the way there. offsetof cannot be used on an arbitrary string at runtime. It only calculates the offsets for compile-time constants.

Let’s declare some structs.

struct Schema;

struct Key {
	// The name of the key and field.
	String key;
	
	// Offset of the field in the struct given by offsetof.
	i64 offset;
	
	// A schema for the corresponding value.
	Schema *schema;
};

enum Type {
	INTEGER, OBJECT, STRING,  // ...
};

struct Schema {
	Type type;
	
	// An array of keys.
	Key *keys;
	i64 num_keys;
	
	// The size of the struct representing the object.
	i64 size;
};

Each Schema has a list of keys and a size of the struct it represents. Each Key has the name of the key, its offset into the struct, and a separate schema describing its paired value.

Let’s define some schemas.

// Schemas for "primitive" types
Schema i64_schema = { .type = INTEGER };
Schema String_schema = { .type = STRING };

// Schema for InstrumentInfo
Key InstrumentInfo_keys[] = {
	make_key3(InstrumentInfo, instrument_id, &i64_schema),
	make_key3(InstrumentInfo, name, &String_schema),
	make_key3(InstrumentInfo, long_name, &String_schema),
};
make_schema(InstrumentInfo);

In the section above some primitive schemas are defined and a schema InstrumentInfo_schema is defined from the InstrumentInfo struct. An array of keys is defined using the macro make_key3 which uses the stringizing operator # to convert the field name to a string constant and calls offsetof on the given field. The make_schema macro defines InstrumentInfo_schema with the defined keys IntrumentInfo_keys using the token-pasting operator ##.

#define make_key3(Struct, property, json_schema)    \
    {                                               \
        .key = S(#property),                        \
        .offset = offsetof(Struct, property),       \
        .schema = json_schema,                      \
    }

#define make_schema(Struct)                         \
    Schema Struct ## _schema = {                    \
        .type = OBJECT,                             \
        .keys = Struct ## _keys,                    \
        .num_keys = countof(Struct ## _keys),       \
        .size = sizeof(Struct),                     \
    }

But this is quite silly. Why are we defining a Schema for InstrumentInfo when the schema already exist? We wrote the schema when we declared the struct in (1)! We should use that.

A key insight is that InstrumentInfo_keys and InstrumentInfo_schema can and should be automatically generated at compile-time. If the schemas are written into a header file schemas.h then a simple parser (emphasis on simple) can be executed using the header file as input at build time. This parser would extract all of the structs and their fields in the header and output a separate file containing the definitions for _keys and _schema.

Parsing Values

Let’s consider parsing values.

// Read into (compile-time) known struct.
InstrumentInfo info = {};
parse_value(stream, &InstrumentInfo_schema, (char*) &info);

// Read into arbitrary struct given by schema.
char *buffer = (char*) malloc(schema->size);
parse_value(stream, schema, buffer);

parse_value takes an input stream of tokens, a schema, and a buffer as parameters. An underlying assumption is that the data is structured and it is known what is expected, therefore the parse function is given a schema as a parameter. The buffer is a pointer to a location that is at least schema->size large and the parsed values are written to it.

void parse_value(Stream *stream, Schema *schema, char *buffer) {
	switch (schema->type) {
	case INTEGER: return parse_integer(stream, schema, buffer);
	case  OBJECT: return parse_object(stream, schema, buffer);
	case  STRING: return parse_string(stream, schema, buffer);
	default: error();
	}
}

parse_value simply switches on schema type.

void parse_integer(Stream *stream, Schema *schema, char *buffer) {
	// Expect integer value from the input stream.
	i64 integer = expect_integer(stream);
	i64 *field = (i64*) buffer;
	*field = integer;
}

In the case of integers, strings, and other primitives the value is just read from the input stream and written directly into the given buffer. It checks that the read value actually is an integer and that the read data conforms to the expected schema.

void parse_object(Stream *stream, Schema *schema, char *buffer) {
	// Loop over the input stream for as long as the object is open
	while (!object_ended(stream)) {
		// Read key from stream and lookup in schema->keys
		Key *key = lookup(expect_key(stream), schema);
		if (!key) error();
		
		// This is where the parsed value should be stored.
		char *field = &buffer[key->offset];
		
		// Call parse_value with the key's schema and field.
		parse_value(stream, key->schema, field);
	}
}

When parsing objects it reads the key and parses the next value according to the key’s schema. The storage for the value is inside the given buffer. Then continue with the next key-value pair.

The example can easily be extended to cover nested objects.

struct Price {
	String price;
	i64 decimals;
};

struct PriceInfo {
	Price last;
	Price open;
	Price close;
	i64 tick_timestamp;
};

struct UnderlyingDetails {
	InstrumentInfo info;
	PriceInfo price_info;
};

// Define the _keys and _schema variables...

Demo

Check out demo with git clone https://mvidell.se/offsetof.git. Build it with make.

The demo features

two large JSON files parsed using different schemas.
more types including arrays.
both input and output types for on-the-fly conversions, e.g. a floating point number in JSON can be stored as a String in the struct.
a flag for each key whether the value should be stored directly or as a pointer.
a simple recursive descent parser for header files that generates the schemas at build time.
an object equality function according to schema in a function similar to parse_value.
pdjson, a streaming JSON parser (more of a lexer), by Chris Wellons.
memory arenas inspired by Casey Muratori’s N+2 programmer and Chris Wellons’s blog.

Possible Extensions

Use macro magic to define _keys and _schemas instead of the parser.
Expected keys, i.e. keys that must be in the read object or its an error.
Default values for keys and a flag to check whether the key was found in the object.
Arrays that support more than one type of values.
Infer the schema automatically from JSON examples.

Conclusion

offsetof can clearly be used for the purpose of parsing structured data into structured data.

Are there any downsides? The structs in the demo can be quite large, some nearing 1 KiB in size. The sizes can be somewhat reduced by storing the data indirectly using pointers but on the other hand, should you? The data has to be stored somewhere and storing it as a map probably uses an even larger memory footprint, with worse locality, and with more expensive lookups compared to structs. Using and chasing too many pointers will convert the struct to a linked list with linked list performance (serial cache misses).

One worry is that there is some hidden footgun or undefined behaviour somewhere but it seems fine. UBSan complained initially about misaligned pointers but that was solved by tweaking the arena memory allocations.

On the other hand we should wonder why it is so hard to read structured data into structured data. We should expect programming languages for the 21st century to be able to solve this much easier using type introspection, reflective programming, and/or metaprogramming.

Livestream - README

Wed, 13 May 2026 12:33:04 +0200

📋

Summary

In this post I go through some of the implementation details of the orchestrator that I implemented as a workaround. For full context on the problems that I try to solve see the previous article Livestream - Save the Frames.

My Workaround

I have implemented a simple orchestrator that starts and maintains multiple processes of yt-dlp that redundantly download the stream. These processes are started using --abort-on-unavailable and, if a yt-dlp process exits, the orchestrator will waitpid(2) on it and start a new worker. As it automatically restart workers the user is no longer required to babysit the download. Through careful selection of flags it can even start download a new video (with different id) should the streamer stop the stream and restart it on their end.

This is obviously not a great solution. The underlying assumption is that if one process encounters a problem downloading a fragment the other processes might be spared. The solution requires a lot of extra storage and a lot of extra network bandwidth. It might not even work that well if the stream encounters a problem though at least it will restart the download should it encounter a problem.

Implementation Details

Maintaining workers

The main loop of the orchestrator maintains two sets of (up to) N workers; the main workers and the outgoing workers. The main workers are the workers that are currently downloading the stream. After the main workers have been running for T seconds they are marked as outgoing and a new set of main workers are spawned to take over. This is, in part, to keep the sizes of the video files manageable. After the outgoing workers have been running for D seconds (overlapping with the newly spawned main workers) they are signalled to terminate. First with SIGINT to allow them to cleanly exit and, after an additional delay, a SIGKILL is sent to make sure that they are terminated.

For each iteration of the main loop the main workers are checked to see that they are still running. If any workers have exited unexpectedly a new worker is spawned to take its place. If too many workers unexpectedly exits within the duration T the orchestrator will exit and terminate all workers. The streamer probably ended the stream.

Starting Workers

The orchestrator creates a new directory for each worker. It then forks and the worker cd:s into the directory and executes yt-dlp using exec(3).

The workers need to know what to download and instead of giving the URL to the stream it is better to give the URL to the streamer. Using some selected options to yt-dlp it is possible to tell it to only download the current active livestream. This also solves the problem if the streamer restarts the stream. The most important flags are --lazy-playlist which prevents yt-dlp to first generate (a potentially long) playlist and --break-match-filter is_live which tells yt-dlp to exit if the next video in the playlist is not live.

Blocking, unblocking, and polling for signals

It is important that the user is able to cleanly shutdown the orchestrator and workers. When the orchestrator receives a SIGINT signal it will start the teardown process and signal its workers to terminate. It is important for the orchestrator though that the SIGINT is captured synchronously. It would be bad if the main loop could be interrupted at any point during its execution leading to an inconsistent state. E.g. a worker could have been spawned but a signal interrupted the orchestrator before the number of workers was increased. Therefore the orchestrator must block SIGINT using sigprocmask(2) and, in order to not block the main loop, poll for signals using sigtimedwait(2) to guarantee the correct state of the program.

It is important for the forked workers to unblock SIGINT otherwise it will be blocked for yt-dlp after exec(3). Otherwise the orchestrator could not tell its workers to terminate.

Download

Download the orchestrator using git clone https://mvidell.se/livestream.git and compile it (on Linux) using make.

Help

Usage: livestream [OPTIONS] URL

Options
    -h, --help      Print this help and exit.
    --test          Run the test_worker instead of yt-dlp.
                    Make sure it is in $PATH.

    Any other options are passed to yt-dlp.

Examples
    livestream https://www.youtube.com/@USER/streams
    livestream -f <code> -r <rate> https://www.youtube.com/@USER/streams

Default Options
    yt-dlp --lazy-playlist --max-downloads 1
    --skip-playlist-after-errors 1
    --break-match-filter is_live
    --abort-on-unavailable-fragments
    --fixup never

Livestream - Save the Frames

Wed, 13 May 2026 12:32:58 +0200

📋

Summary

In this post I go through the difficulties I have had when downloading ongoing streams from YouTube using yt-dlp. yt-dlp is a big project with a lot of switches and features and, while I have not explored every single feature and option it has to offer, it is quite bad that something as simple as downloading an ongoing stream is such a hard problem using the go-to download everything from YouTube downloader. In the end I present my workaround which is to have multiple processes of yt-dlp in parallell downloading the stream for redundancy. I wrote a simple orchestrator in C/C++ that automates this.

Some working assumptions

The download has to be done while the stream is live as the stream is expected to be removed after it has ended (more common than you think).
The stream can experience hiccups, lost fragments, false positive end of stream detected, etc.
Some frames are lost between the streamer and YouTube. These can probably never be saved and are irrevocably lost forever.
Frames are more often lost between YouTube and me and these are the frames I want to save. These unavailable fragments are often frequently available for download on retry (but after failing the default retries).
The solution should be hands off. The user should not have to babysit the download.

Lets look at some scenarios

Just use yt-dlp $URL to download the stream

This is the no-brain starter method but it is not a great solution. The problem with this is that this only downloads the stream from the current position. How are you going to download the beginning of the stream if it started hours ago?

Download the stream from the beginning with yt-dlp –live-from-start $URL

This will download the stream from the beginning up to to the current fragment and will even continue until the end of stream. But what will you do when a fragment 4321 of 5000+ fails to download? Several times I have encountered fragments that failed to download the first time (failing all default retries) but was able to be downloaded after the download was restarted. The problem here is that the entire video has to be re-downloaded in this case. What if it is fragment 4322 that has a problem in the re-download? Am I supposed to restart the download from the beginning again?

Add –keep-fragments option

Using this option it will keep the fragments downloaded as separate files and it will skip downloading fragments that have been previously downloaded if the process is restarted. But there are several problems. If the download is interrupted before the end of the stream it will merge all fragments into an incomplete video. At this point the download should be restarted as soon as possible, it should not spend a long time (and a lot of storage space) building an incomplete video. Furthermore, the merged video will (if combined with –live-from-start) prevent yt-dlp from downloading the stream again because yt-dlp will think that it has already downloaded it. The user is still required to babysit the download and to interrupt it if any fragments fail to download.

–abort-on-unavailable-fragments

This will abort (i.e. exit the download) instead of continuing if it fails to download a fragment. It is sometimes possible to restart the download (why? when? how? I have no idea). While using this you still have to be actively monitoring the download because if a fragment fails then it obviously won’t download the rest of the stream and the download has to be restarted manually.

–abort-on-unavailable-fragments and –keep-fragments

Using –abort-on-unavailable-fragments it will stop the download when a fragment is unavailable. This, in combination with –keep-fragments, might be good for easier restarts (still manual restarts though). This combination I have not been able to test thoroughly. If a fragment is irrevocably lost will this actually manage to download the rest of the stream?

Use yt-dlp –fragment-retries infinite $URL

I haven’t tested this either but I would assume that this will hammer on fragment 4321 infinity times. This is good if the fragment become available in the future but some fragments are irrevocably lost and will never become available. Will this option then prevent the download of the remainder of the stream?

Use yt-dlp –fragment-retries $X $URL where $X < infinite

This is obviously also not sufficient. The unavailable fragments could be unavailable during all retries and only become available after all retries have been exhausted. And if yt-dlp spends too much time waiting for missing fragments it will miss the remainder of the stream.

Not possible: yt-dlp –download-sections “-5:00-inf” $URL*

This does not currently supported for streams on YouTube. Why is this not possible? Who knows… This would be helpful if a fragment failed and it is known approximately where, in time, that fragment is.

–fixup never

This option has to be mentioned because of its usefulness though it does not help downloading fragments directly. When a download has ended yt-dlp spends a lot of time (and space) to fix the stream. I do not know what this actually does but it certainly takes a lot of time and space. The entire stream is written to disk again meaning yt-dlp requires double the stream’s size for the stream. –fixup never can be used to quickly stop after the download and to quickly restart it. Since it is a post-processing of the video it is fine to skip it for the moment. I wish the documentation was more clear of what it does and how it can be replicated after the download has finished.

What I Want

I want to tell yt-dlp to just download the stream! Why is it so hard!? When downloading a stream could you please try repeatedly to download the fragments that have failed but without preventing the download of the remainder of the stream? Stop creating incomplete and corrupt files by default!

There is no good solution to this problem where fragments are downloaded sequentially. The solution must be able to fetch fragments non-sequentially and these fragments are retried infinitely without stopping to download the new fragments. This has a simple solution though. Just put the failed fragments in an array and retry them every now and then.

yt-dlp would need to change their insane default behaviour and policy that it is better to have an corrupt final file than no file at all. This is a false dichotomy. Once the file has been finalized it is much harder to fix it. This is made even worse when yt-dlp corrupts the file by default often without the knowledge of the user.

Some features that would help.

Non-sequential fragment downloader that keeps trying to download missing fragments in the background while it still downloads the ongoing stream.
An option to not combine fragments when using –keep-fragments. I can combine them myself afterwords.
An option for downloading specific fragments either by number, by range, by list, etc. E.g. download fragment number 4321 or fragments 100-200.
Downloading time ranges in streams as one would expect –download-sections to work.
Not creating broken videos by default.

Other users have been complaining for years for similar reasons.

My Workaround

To workaround these problems I have implemented a simple orchestrator that starts and maintains multiple processes of yt-dlp that redundantly download the stream. These processes are started using –abort-on-unavailable and, if a yt-dlp process exits, the orchestrator will waitpid(2) on it and start a new worker. As it automatically restart workers the user is no longer required to babysit the download. Through careful selection of flags it can even start download a new video (with different id) should the streamer stop the stream and restart it on their end.

For more information about the orchestrator see its separate article: Livestream - README.

Braider

Wed, 31 Dec 2025 00:00:00 +0000

An old a new screenshot of the prototype.

Problem: I want to browse videos from several YouTube channels in chronological order within a given date range. You would think that YouTube would make this easy, but it is not. It is especially hard if the channels in question publish a lot of videos and the time in question is several years ago.

Using my prototype, Braider, you can browse thousands of videos from multiple channels, including sorting videos on different values such as the number of views or length. If you tried to browse YouTube in your browser like this you would need multiple tabs each requiring 400+ MiB of memory.

The prototype features

a simple text search that matches titles and descriptions.
the ability to filter channels using mute and solo-pattern adapted from remedybg and DAWs.
an interactive histogram that shows the number of videos released in year-month buckets.
centering on videos. Use the sidebar to filter the videos, highlight one of the shown videos and then reset the filter. The selected video is now shown in the context of the full braid.
a JavaScript test suite using my own implementation of the describe-it-should test pattern.
an incredibly fast interface compared to YouTube.

There are some limitations in the prototype.

The Braids are currently hardcoded and is currently not user friendly to generate new ones.
The web client does not support varying resolutions. It will not look or function good on mobile.
YouTube might not like websites that embed many videos and the iframes sometimes produces errors.

Examples of what questions you could answer using Braider.

What is ThePrimeagen’s most popular video in the first quarter of 2025?
Limited Life was a subseries that occurred during H9. How many videos were published in that series? Which video is the most viewed?
Simon biked the Marcher Castles Way on GCN in 2025. Find the GCN Tech followup video that was published the day after.

This is mostly a proof of concept/early prototype. I wanted the user to be able to input a list of channels and it would then fetch all videos (with client-side JavaScript) and then display them. But it seems like that is not possible(?), or it is at least not straight forward due to security concerns.

It is written in JavaScript (yuck) but I tried to make it better and more pleasant using

pure JavaScript without using any dependencies or npm-hell.
ample use of JSDoc for type hints.
an LSP (ts_ls) which catches errors and type checks.
the Chrome debugger.

Check out YouTube Braider Prototype multiple hardcoded braids.

A Study in (Linux) Timers

Tue, 18 Feb 2025 00:21:58 +0200

I watched ThePrimeagen do a LeetCode problem where he was asked to implement debounce. I realised that I have no idea on how to do this in C. Therefore I set out to figure it out and this is what I came up with.

The Problem

📜

2627. Debounce

Given a function fn and a time in milliseconds t, return a debounced version of that function.

A debounced function is a function whose execution is delayed by t milliseconds and whose execution is cancelled if it is called again within that window of time. The debounced function should also receive the passed parameters.

For example, lets say t = 50ms, and the function was called at 30ms, 60ms, and 100ms.

The first 2 function calls would be cancelled, and the 3rd function call would be executed at 150ms.

If instead t = 35ms. The 1st function call would be cancelled, the 2nd would be executed at 95ms, and the 3rd would be executed at 135ms.

From LeetCode: https://leetcode.com/problems/debounce/description/

via ThePrimeagen: https://youtu.be/nO7J6pBEkJw?t=4793

Attempt One: `debounce_itimer.h`

The first thing I investigated was getitimer(2) and setitimer(2) which sets an interval timer. When the timer expires a signal is raised and the debounced function can be executed from the signal handler. I managed to get it to work in debounce_itimer.c but it has several limitations.

There is only one interval timer which means that only one function can be debounced at a time.
My solution relies on a lot of global data and boilerplate that is necessary for each function that you want to debounce. This global data reinforces the problem with 1: only one function can be debounced at a time.
Higher order functions in C is a mess. The ideal interface that I wanted is the one suggested in LeetCode code skeleton: a function dx = debounce(x) where x is a function and dx is a debounced version of x, but this is hard to accomplish in C.

An example usage of debounce_itimer is

    void (*dadd)(int, int) = debounce_add(50);

    millisleep(30);
    (*dadd)(1, 2);
    millisleep(30);
    (*dadd)(1, 3);
    millisleep(40);
    (*dadd)(1, 4);

This is the only solution where I managed to have a function return a callable function, but as you will notice it is not called debounce. debounce_add is a manually created boilerplate function that must be created for every function that you want to debounce.

You could argue that C is not meant to have inner functions or overloaded functions and that wrestling with it is counterproductive and whatever solution you end up with is not pythonic (but for C).

Here is the boilerplate required.

    int global_a = 0;
    int global_b = 0;
    int global_add_t = 0;

    // This function is run when the timer expires
    void add_alarm_handler(int signum) {
        add(global_a, global_b);
    }

    // This is the function that gets executed when the user calls the
    // debounced add. It sets global data, sighandler, and (re)starts the
    // timer.
    void debouncer_add(int a, int b) {
        global_a = a;
        global_b = b;
        set_signalhandler_and_clock(global_add_t, add_alarm_handler);
    }

    void (*debounce_add(int t))(int, int) {
        global_add_t = t;
        return debouncer_add;
    }

Attempt Two: `timeout.h`

After a careful reading of setitimer(2) it notes that setitimer has been obsoleted in favour of timer_settime(2) and related functions.

When ThePrimeagen solved this problem he did it using setTimeout in Javascript. Therefore, instead of going directly to implement debounce, I set out to first implement setTimeout using timer_settime(2).

Here is how to use it.

    timer_t id;
    struct AddArgs args;
    args.a = 1;
    args.b = 2;
    id = setTimeout(timeout_add, 50, &args);

This interface requires a struct AddArgs that stores the parameters for the timeouted function. It also requires a function timeout_add which is executed after the timer has expired and unpacks the struct and calls the function add.

    struct AddArgs {
        int a;
        int b;
    };

    void timeout_add(void *p) {
        struct AddArgs *a = (struct AddArgs*) p;
        add(a->a, a->b);
    }

Different this time is that when the timer expires it is possible to have the timer start a new thread and execute the timeouted function in parallel instead of concurrently. Or it is possible to continue using signals. Threads is probably the superior option.

The following code creates the timer with either a thread handler or a signal handler.

    /**
     * Sets up a timer according to `USE_THREAD`.
     */
    int setup_timer(
            void *data,
            void (*thread_handler)(union sigval),
            void (*signal_handler)(int, siginfo_t*, void*),
            timer_t *timerid) {

        struct sigevent sev = {};
        sev.sigev_value.sival_ptr = data;  // The data to be passed

    #if USE_THREADS
        assert(thread_handler);
        sev.sigev_notify = SIGEV_THREAD;
        sev.sigev_notify_function = thread_handler;
    #else
        assert(signal_handler);
        // Establish signal handler for timer signal
        struct sigaction sa;
        sa.sa_flags = SA_SIGINFO;
        sa.sa_sigaction = signal_handler;
        sigemptyset(&sa.sa_mask);
        if (sigaction(SIGTIMER, &sa, NULL) == -1) {
            perror("sigaction");
            return -1;
        }

        sev.sigev_notify = SIGEV_SIGNAL;
        sev.sigev_signo = SIGTIMER;
    #endif

        // Create the timer
        if (timer_create(CLOCKID, &sev, timerid) == -1) {
            perror("timer_create");
            return -1;
        }
        return 0;
    }

Attempt Three: `debounce.h`

I was now ready to implement debounce using timeout.h, but as I started doing this I realised that I did not want to use it. The code I had written to implement timeout.h was easily reused though to set up a timer and handlers.

    struct Debounce dadd = {};
    dadd.ms = ts[i];
    dadd.fn = add_h;
    debounce(&dadd);
    struct AddArgs aa = { .a = 0, .b = 0 };
    struct AddArgs bb = { .a = 1, .b = 1 };
    struct AddArgs cc = { .a = 2, .b = 2 };

    millisleep(30);
    debounce_call(&dadd, &aa);
    millisleep(30);
    debounce_call(&dadd, &bb);
    millisleep(40);
    debounce_call(&dadd, &cc);

This solution requires the user to fill in a struct Debounce with the function that is to be executed when the timer expires and the time until expiration. The function expects to be of the format void (*)(void*) and a struct per debounced is still necessary for the parameters.

    struct AddArgs {
        int a;
        int b;
    };

    void add_h(void *p) {
        struct AddArgs *a = (struct AddArgs*) p;
        add(a->a, a->b);
    }

Different this time is mutexes. The Debounce struct contains a mutex since it would be bad if the timer expires and the main thread changes the parameters while the debounced function is executing.

Attempt Four: `debounce2.h`

This time I wanted to do the crazy macro. I don’t know how to feel about these macros. They make the interface better for the caller but is it worth the cost. In this case I think it might. The macros that I made are quite readable.

    debounce(dadd, add, 50);

    millisleep(30);
    debounce_add(&dadd, 0, 0);
    millisleep(30);
    debounce_add(&dadd, 1, 1);
    millisleep(40);
    debounce_add(&dadd, 2, 2);

The boilerplate necessary for this is

    DEBOUNCEABLE2(add, int, int);

Where DEBOUNCABLE2 is a big macro that defines 3 things:

the struct add_args.
the handler function add_h.
and the debounce function debounce_add.

The 2 in DEBOUNCABLE2 refers to the number of parameters the function takes. This means that a new macro has to be made for every function that requires a new number of parameters. This is not as bad as it might seem. The macros copy paste pretty easily and there is only a few lines that need editing. But it would be nice if C supported better macros, with say recursion or nested definitions.

debounce is also a macro and it looks like this:

    #define debounce(Debounced, Function, Milliseconds)                         \
        struct Function ## _args Debounced ## _args = {};                       \
        struct Debounce Debounced = {};                                         \
        Debounced.ms = Milliseconds;                                            \
        Debounced.fn = Function ## _h;                                          \
        Debounced.args = & Debounced ## _args;                                  \
        debounce(&Debounced);

It declares and fills in automatically the struct Debounce from debounce.h. It also creates a struct add_args where the parameters are stored.

Unfortunately I don’t think it is possible to get the Javascript syntax of dx = debounce(fn, t) because you would need to return a function from debounce which depends on the given function fn. Perhaps if debounce returned a variadic function but that might be problematic if the user then accidentally calls it with the wrong number of parameters.

Testing

When print and add are executed they write an output string into a simple buffer which contains the time since the testcase began. This made it easy to then afterwards just parse the lines of this buffer and extract the timing to check if it was correct. This was also protected by a mutex.

    // Print also writes the string into `buffer` and advances `bindex`.
    // This is so that the test-functions can easily assert on the timing
    // and argument `s`.
    int length = snprintf(
        &buffer[bindex], BUFFER_SIZE-bindex,
        "[%lf ms] Hello '%s'\n", ms, s
    );
    bindex += length;

The assert reads this buffer with sscanf(3).

    matches = sscanf(
        &buffer[index], "[%f ms] Hello '%60[^]']'\n%n",
        &ms, string, &bytes
    );

What has not been easy to test is the mutexes and other race conditions.

Conclusion

Should you use this? Probably not.
If only I had Jai… Would be interesting with better metaprogramming.
Download code code with git clone https://mvidell.se/timers.git.

cmpfiles

Wed, 15 Mar 2023 00:00:00 +0000

I have a lot of old files and weird old copies that sometimes have similar files but in slightly different file tree structures. As I wanted to remove the duplicate files I wanted to make sure that the files actually were duplicates and were safe to delete.

$ tree
.
├── dir1
│   ├── a.txt
│   ├── b.txt
│   └── c.txt
└── dir2
    ├── dir3
    │   └── c.txt
    └── dir4
        ├── a.txt
        └── b.txt

5 directories, 6 files

The above examples shows that two different directories with different file tree structures. Comparing manually whether it is safe to delete dir2 (i.e if all files in dir1 is in dir2 and vice versa) can be time consuming. cmpfiles recursively traverses the file tree, hashes all files, and then compares whether the tree produced the same hashes by sorting the list of hashes.

$ cmpfiles dir1/ dir2/
'dir1/c.txt' does not exist in 'dir2'

'dir2/dir3/c.txt' does not exist in 'dir1'

'dir1/a.txt' <=> 'dir2/dir4/a.txt'
'dir1/b.txt' <=> 'dir2/dir4/b.txt'

In this example cmpfiles tells us that dir1/c.txt and dir2/dir3/c.txt are not found in the other directory (because they have different hashes) and we know that we should not remove these files before investigating this discrepancy. One file could be an older version of the other file.

Help

Usage: cmpfiles [OPTION]... DIR1 DIR2
Compares the files in DIR1 to the files in DIR2

Compares the files (but not the file structure) by hashing each file in DIR1
and in DIR2, and then comparing them. It does not dereference symbolic links.

With no options, it produces output in three sections. The first section contains
the files that are in DIR1 and not in DIR2. The second section contains the
files that are in DIR2 and not in DIR1. The third section contains the files
that are in both DIR1 and DIR2.

If the program encounters duplicate files inside DIR1 (or inside DIR2) it
ignores these and outputs warnings to stderr.

  -1     Suppress first section  (files in DIR1 but not in DIR2)
  -2     Suppress second section (files in DIR2 but not in DIR1)
  -3     Suppress third section  (files in both DIR1 and DIR2)

  --md5          Use MD5 as hash function instead of SHA256
  -h, --help     Display this help and exit

Examples:
  cmpfiles -12 dir1 dir2  Print only files in both dir1 and dir2.
  cmpfiles -3  dir1 dir2  Print files in dir1 and not in dir2, and vice versa.

Somewhat inspired by comm(1).

The code for the project can be downloaded with git clone https://mvidell.se/cmpfiles.git.

pdftoc

Sun, 25 Sep 2022 00:00:00 +0000

Printing and editing the contents of K&R.

A quick draft written about the project.

Problem: I have a lot of PDFs that I read, many of them do not have a proper Table of Contents (called Outline in the PDF specification). I wrote a Recursive Descent Parser (in C/C++) that parses the PDF file to find the outline.

You still have to do the hard work of writing the Table of Contents to a file. The goal was that you should be able to just copy over the Table of Contents inside the book (if it exists). Some care is needed to set the ranges correctly. Many books start off with a roman numeral section before page 1, e.g. in K&R (see demo) Contents starts on page v and Introduction on page 1 (15th page in the PDF).

The code for the project can be downloaded with git clone https://mvidell.se/pdftoc.git

Sigmar's Garden Solver

Thu, 18 Feb 2021 00:21:58 +0200

Figure 1: The first image shows a starting game board. The in-game conversation introduces the game. The second image shows a computed solution outputted by drawing lines and numbers on the input image. The third image shows an information panel containing the axes in the grid in green and blue, the coordinates for each hexagon, the extracted features, and the predicted type of each marble. The screenshots were taken from a Youtube video by Dexbonus.

Video 1: A recording of the program solving 3 puzzles.

INTRODUCTION

Sigmar’s Garden is a minigame in Opus Magnum, the puzzle game developed by Zachtronics. Sigmar’s Garden presents a hexagonal board with different types of marbles and the goal is to match two marbles with each other to remove them and clear the board. Each marble represents an alchemical element and there are rules on how you can match them. Importantly, only free marbles, marbles that have 3 or more connected empty neighbours, can be used when matching. The full ruleset as explained in the game can be seen in Fig 1.

The game constantly teases you to win more games. First it requests you to win a single game, then win 10 games, then win 25 games and so on. This drove me crazy and I got the idea to make a program that solves them. Using OpenCV and X11 libraries, the solver screenshots the game, classifies the marbles, finds a solution, and performs the solution using mouse movements and clicks.

TECHNICAL DETAILS

The project consists of a single cpp-file that is divided into several sections: a hashset, the solver, the OpenCV section, the X11 section, and a main section. I started writing the solver in C but after OpenCV was added the code turned more into C++ and is now a weird mix between them.

The hashset and the solver work together. The solver is a simple recursive exhaustive search that identifies possible moves and performs a depth first search over them. Each board state that is explored is inserted into the hashset as memoization to prevent exploring previously explored states. This works really well because it is easy to reach the same state multiple times.

OpenCV is used in two different ways. It is used for manipulation of images and for its Support Vector Machine (SVM) that I used to classify the marble types. At first I thought that perhaps it is enough to just use simple template matching in order to classify the marbles but a closer look shows that the marbles all have slightly different lighting effects depending on their position in the board. Having used OpenCVs SVM previously I just chose to use this again.

The SVM is trained on screenshots of the game from Jonathan Blow and Dexbonus. One problem that I encountered is that the marbles have different sprites when they are free or closed. This forced me to add screenshots of games in progress so that there are samples of metals that are in the free state when training the classifier.

The hexagonal grid is created by hardcoding the pixel values for the center and hexagonal width at 720p for reference. Using the hardcoded values a hexagonal coordinate system can be built using two axes. A limitation this causes is that all image inputs are expected to be in 16:9 aspect ratio or the grid might not be properly aligned. Red Blob Games has an excellent resource on hexagonal grids.

Features are extracted by cropping a 24x24 pixel square from each hexagon. These are preprocessed by converting the images to black and white, adding a bit of gaussian blur and then an adaptive threshold filter.

After classifying and solving a game the solver can output the solution in text, by drawing on the input image as shown in Fig 1, or in the form of mouse movements and clicks using X11, the common windowing system for Unix-like operating systems. The idea is that you bind the solver to a global hotkey that when pressed executes the solver. The solver will then take control of the mouse and perform movements and clicks inside the focused window (presumably the game itself, but for testing purposes YouTube or images work too, assuming they are correctly sized). The solver will press New Game before it exits.

RESULTS

The solver managed to solve 100 games of Sigmar’s Garden with only minor issues. Most games where solved is less than a second after invoking the solver. The time to perform the solution is not included in this. Performing the solution takes several seconds.

A few games took slightly longer than a second to solve, and only one game took more than 10 seconds to solve. That game also broke the hardcoded memory limit which was increased by 4x and never reached again.

During the trial there were no errors with classifying the different marbles. It should be pointed out that all solved games were solved from the beginning, and I expect that the biggest problem with the classifier is classifying the free metal marbles in the later game. Another limitation is in the resolution of the game. The solver could not manage 900p resolution of game, but 1080p worked. This is because the resolution option in the game is not a simple scaling.

FUTURE WORK

In the source code there are several comments regarding future and possible TODO such as some suggestions on improving the hashset, the solving algorithm, and improving the structure around labelled data for the SVM.

A large improvement would be to remove OpenCV. OpenCV is a huge 100+ MiB dependency and a massive overkill for this project. The image loading and manipulating can probably be replaced by stb_image, lodepng, or similar, and it is possible to write a new classifier relatively simply.

BUILD AND USAGE

Download code with git clone https://mvidell.se/sigmar.git and build sigmars_garden using the included build.sh script. On first start it will train the classifier and save it to sigmars_garden.svm. Try it using --images on the included training/test data!

$ ./sigmars_garden --help
Usage:
sigmars_garden [options]
Starts interactive mode, screenshots focused window and performs a solution
with mouse movements and clicks. Recommended to either bind sigmars_garden to
global hotkey or use sleep to be able to focus a window before it screenshots
and performs a solution.

sigmars_garden [options] --images image...
Transcribes the marbles and solves the garden, outputs a solution to stdout
and shows an image of a solution. Will run out of memory when run with a large
amount of images.

sigmars_garden [options] --text
Reads 91 marbles from stdin separated by whitespace in format 'QUICKSILVER
VITAE EARTH SALT EMPTY' etc (case insensitive) starting in top left corner,
reading left to right. Just copy a transcribed image and pipe it in!

Options:
--new-svm      Force generation of a new SVM
--old-rules    Solves the game with old rules (salt cannot match salt)
--verbose      Prints extra info
-h, --help     Shows this help and exits

REFERENCES

Ultimate Tic-Tac-Toe

Sat, 25 Jul 2020 00:00:00 +0000

Screenshots of the three different implementations. From left to right: AngularJS, Vue.js, Svelte.

Ultimate Tic Tac Toe is a more advanced variation on the ordinary tic-tac-toe. The game is divided into 3-by-3 boxes and each box contains an ordinary game of tic-tac-toe.

The goal of the game is to win three small games of tic-tac-toe so that the winning boards create three-in-a-row in the larger game. But there is a catch.

After a player has made their move by placing their piece in a cell in a board, the following player can only make a move in the board corresponding to the cell that the previous person played in.

Implementing Ultimate Tic-Tac-Toe has for me become a sort intermediate test of web development frameworks and libraries. I have made three different implementations of Ultimate Tic Tac Toe. The first was made in AngularJS when it starting to get popular (2013?). The second implementation was made with Vue.js in 2017, and the third one was made with Svelte in 2020.

Of the different framworks Svelte is definitely my favorite.

Kattis Submissions

Wed, 25 Jul 2018 00:00:00 +0000

This is an incomplete list of problems that I have solved on kth.kattis.com. Kattis is a service used for evaluating solutions to programming problems and was frequently used in my education. Most of the problems listed here have been solved using Python or C++.

Most problems were solved as part of the ‘Problem Solving and Programming Under Pressure’ course i took in 2015 (popup15). In the course we implemented many algorithms and data structures such as graph algorithms (e.g. shortest path, maximum flow), number theory (e.g. Chinese remainder, prime sieve), and dynamic programming. The problems marked with (*) were classified as harder problems in that course.

Since many problems are still used in education my solutions are not publicly available. Kattis still lists me on the high score lists on some of the problems (in Metadata > Statistics).

PS: Some links have been changed and you are now (sometimes) required to click on a course (e.g. popup15). The course you click on does not seem to matter though, they seem to lead to the same problem text and statistics.

Full list of problems for popup15

Quadratic Sieve

Sat, 14 Oct 2017 00:00:00 +0000

To be written…

Duck Hunt

Tue, 25 Jul 2017 00:00:00 +0000

This was an assignment in our Artificial Intelligence course. The goal was to use hidden markov models to classify different species of birds and predict their next movements in a simulated version of the game Duck Hunt.

Solutions were tested in several rounds in different environments. Each round is 100 time slices long and during each slice you could choose to fire at a bird by predicting its next move. If you missed you lost a point, and if you hit you gained a point. However, if you shoot the Black Stork you lose all points in that environment.

After each round you can guess the species of each bird. Guessing correct gains you a point and incorrect loses a point. After guessing you receive the correct answer allowing you to learn each species behavior. Guessing the species is where most points you earn come from.

In this assignment we implemented Hidden Markov Models including the forward, backwards, and viterbi-algorithms, and correctly applied them to the problem. My highscore is 399. Since this problem is still used in education I have not released my solution.

C Command Shell

Sun, 13 Dec 2015 00:00:00 +0000

To be written…

Playhouse

Mon, 27 Oct 2014 00:00:00 +0000

Press clippings from the project.

Playhouse was a large group project (9 people) to use Philips Hue lightbulbs to play animations and interactive games in the windows of buildings. The project was part of our education in a software development course and was made in collaboration with artist Håkan Lidbo.

Philips Hue lightbulbs are capable changing colours in RGB spectrum and can be controlled with an API by sending web requests to the included wireless bridges. The goal was to create animations and simple games such as mastermind and tic-tac-toe that can be played by users in their mobile browser.

We created two web servers written in Python using Tornado. The lamp server controls the lamps directly by sending web requests to the bridges. The second server handles the communication between users and the lamp server. It serves the webpages, communicates actions using websockets, handles game logic and sends commands to the lamp server.

In the fall of 2014 the project continued and we installed it in the newly built NOD building in Kista, as part of its inauguration.

DCPU Game of Life

Sun, 27 May 2012 00:00:00 +0000

Runnning acorn pattern in DCPU-IDE.

After his success with Minecraft, Markus ‘Notch’ Persson announced a new game called 0x10c. It was designed to be a science fiction sandbox that included a virtual computer and virtual hardware peripherals that Notch himself had designed the specifications for.

I thought the concept was interesting and decided to implement Conway’s Game of Life written in DCPU, the assembly-like language created for the virtual computer and output the result on the virtual monitor called LEM1802.

The monitor is divided into 32x12 characters and each character is divided into 4x8 pixels. A program can print characters using a program-defined alphabet of 128 characters. The characters are all monochrome and each character is defined by setting each pixel to be either foreground or background colour.

One way to implement Game of Life would be to have one Game of Life cell per character for a total of 32x12 = 384 cells, but I wanted to divide each character into 2x4 cells (each cell would be 2x2 pixels) for a total of 64x48 = 3072 cells.

However, 8 cells per character means 2^8 = 256 different combinations, double the available alphabet. I managed to solve this problem by utilising that each character can be printed in normal or inverted colours, effectively doubling the available alphabet if it is generated correctly. I used the last cell as a sort of sign bit (inspired by Two’s complement used to store signed integers) to determine whether the character should be printed in normal or in inverted colours.

The all-in-one.s source file is a concatenation of all source files, to simplify copying and pasting into the online emulator I used when creating this.

Includes two premade patterns, a glider and acorn. To change to acorn edit the line JSR write_glider to JSR write_acorn.

Square Dino

Fri, 27 Apr 2012 00:00:00 +0000

Screenshot of Square Dino.

Square Dino is a program that runs Conway’s Game of Life. It was created with a friend of mine in the end of spring 2012 as the last project in our introductory course in programming. The course completely focused on Java and object oriented programming.

A reqirement in the project was using Swing, the GUI widget toolkit. This was no problem for us as we had previous experience in it and decided to go far beyond the requirements.

We added many features including import and exporting to file, changing ruleset, and a backend using QuadTrees.

Features

Modular interface using Swing Toolbox.
Importing and exporting of patterns using a common format.
Change the rules of when cells are born or die.
Copy and paste selected areas.
Infinite board.
Zoom and pan.
Graph of population.

Instructions

Build with javac -d bin/ -cp src/ src/org/bitbucket/zetro/squaredino/SquareDino.java

Execute with java -cp bin/ org.bitbucket.zetro.squaredino.SquareDino

You should also be able to import it as a project in Eclipse and run it from there.

mvidell.se

Offsetof

Parsing Values

Demo

Possible Extensions

Conclusion

Livestream - README

My Workaround

Implementation Details

Maintaining workers

Starting Workers

Blocking, unblocking, and polling for signals

Download

Help

Livestream - Save the Frames

Some working assumptions

Lets look at some scenarios

Just use yt-dlp $URL to download the stream

Download the stream from the beginning with yt-dlp –live-from-start $URL

Add –keep-fragments option

–abort-on-unavailable-fragments

–abort-on-unavailable-fragments and –keep-fragments

Use yt-dlp –fragment-retries infinite $URL

Use yt-dlp –fragment-retries $X $URL where $X < infinite

Not possible: yt-dlp –download-sections “*-5:00-inf” $URL

–fixup never

What I Want

My Workaround

Braider

A Study in (Linux) Timers

The Problem

Attempt One: debounce_itimer.h

Attempt Two: timeout.h

Attempt Three: debounce.h

Attempt Four: debounce2.h

Testing

Conclusion

cmpfiles

Help

pdftoc

Sigmar's Garden Solver

INTRODUCTION

TECHNICAL DETAILS

RESULTS

FUTURE WORK

BUILD AND USAGE

REFERENCES

Ultimate Tic-Tac-Toe

Links

Kattis Submissions

Quadratic Sieve

Duck Hunt

Links

C Command Shell

Playhouse

Links

DCPU Game of Life

Links

Square Dino

Features

Instructions

Links

Not possible: yt-dlp –download-sections “-5:00-inf” $URL*

Attempt One: `debounce_itimer.h`

Attempt Two: `timeout.h`

Attempt Three: `debounce.h`

Attempt Four: `debounce2.h`