HUMBUG»Blog

November update:

Hello Everyone:

TL;DR: I have been hard at work over the past few weekends trying to get HUMBUG to a releasable state. Skip to the last paragraph for a brief summary of accomplishments. Hopefully I’ll be done with user interface stuff by the time I do the next update… well see :)

Long version:

Most all of the open complaints that I have from testers are issues with the the user interface. This is primarily because I didn't spend a lot of time trying to make a good user interface, but instead I just wanted to make something that was just better than using GDB in the terminal.

I had been using DearIMGui since the beginning but decided that it had some inherent limitations that would prevent it from being my long term solution. So I embarked on making a new UI toolkit that would be tailored to my needs. But before that I figured I'd take a quick side quest and try to get rid of SDL while I was at it.

I had wanted to get rid of SDL for a number of reasons, in brief: Runtime link dependancies on SDL(and Xlib), also also their runtime overhead; and also to have lower level control of event dispatching (i.e. just wanted to throw everything my application would wait on into a single epoll_wait call and dispatch on that).

Before I started on this journey I didn't know much about the internal workings of the X11 protocol or the state of Xlib or XCB. I did know that it was a client/server api and that XCB was a newer effort to try to make a lower level interface to X11. Fun fact: Xlib is now built on top of XCB, even though XCB is newer than Xlib.

I decided that for my goals it would be best to use XCB. It already was a very thin layer over the socket protocol. There were some things that I did not like about it, (mainly that it basically does malloc(s) for nearly every bit of traffic). Even with these issues I thought that I could just grab the source and begin to semantically inline the small bits of XCB that I was using and over time convert its implementation to something I would like better.

That was very foolish thought and...

Beware, there be dragons here.

Things started fairly well, and after a couple hours, I had a basic event loop where I could get mouse and keyboard input events. I got all this to work by following a tutorial on www.x.org. Things started to get hairy when I wanted to convert keyboard events into UTF-8 input. In the tutorial there is a line that simply reads: "TODO: Talk about getting the ASCII code from the key code."

I suspect that tutorial has not been updated for a very long time. This should have been a sign that I should have just given up. But I told myself, "But I've gotten this far fairly quickly, how hard could it be? Worst case, I'll just look at what Xlib does and reverse engineer it..." Also a second bad sign was that in the XCB tutorial, everything is given in terms of Xlib's api, again I simply brushed this off by thinking "This is probably just for people converting from Xlib, and surely there exists real reference and complete documentation.”

I won't go into all the little details but I'll briefly explain the process for which character input works in Linux. When you press a key on your keyboard the X11 server will send a packet over a socket to your X11 client and this packet will have a field called "detail" that has the keycode for the key that was pressed. This keycode (which is just a number) has no simple pre-defined relationship to any idea of what the key actually was (ie the KeySym, like this is the A key) or the character that key should generate if it were pressed (like "A").

There is a protocol that describes how one would convert those keycodes into characters. The process starts off by you telling the X server that you want to enable an extension called "XKB". Yes that's right, modern keyboard input is an "extension"... This extension was made in 1995. Also fun fact: Part of setting up XKB is the X11 server telling you what ID numbers the events for this extension start on, because for some reason we can't just have a header file with some defines in it to tell us this. After doing this, you then need to ask it for the keymap which will give you all the information needed to do the conversions. Also, you have to tell it that you want to receive events for situations where that keymap will change so you will know when to ask for it again.

Side note: You'd think at this point we might have considered just having the Xserver just give us reliable KeySyms and their related characters... Also another related fun fact: XKB is also the keyboard protocol for Wayland.

Luckily, I eventually found a library called xkbcommon, which can work on top of XCB and does most of this work for you. It even has relatively decent documentation to get you started. I found this merely by chance after lots of googling trying to get all this stuff to work.

A quick side note: getting an OpenGL context with XCB is tricky, if at all possible. At this point I had settled on just using GLX via XLib to get a GL Context. I had originally had hopes of getting that to work in the same spirit as the XKB stuff, and also was thinking about trying to get EGL to work. In the XCB code there is glimmers of hope that GLX can be achieved, but I was not brave enough to continue down that path. Morale was getting low for me at this point.

I am a very stubborn person, and I am not usually willing to admit defeat.

I had one last piece to get working before I could say that I had a platform layer for HUMBUG: Copy and Paste. This is its own flavor of insanity. This involves the XServer essentially serving as a generic Key-Value store, and you have to send messages between the windows in X11 to negotiate the Key and the format for the Value and also who "owns" things that are called "selections" which isn't strictly standardized, but upon popular consensus everyone uses a selection called "PRIMARY". Fun fact, when you want to specify things as strings (like "PRIMARY") in X11, you can’t just pass a string, you have to register that string as an "ATOM" and then use the ATOM's id as the string.

I had nearly everything written that I would need for a platform layer, but I still needed to clean some stuff up. For example, one specific thing I had to fix was that if you copied more than 4096 characters then I would silently truncate it. I’m probably missing some subtle detail here, but I was struggling to divine the correct procedure to get X11 to let me know the length of string ahead of time. And I was struggling to care.

This was my breaking point.

I was playing with my test code and I realized the cold hard truth: After spending almost two weekends working on this, I was not any closer to my goal. SDL and Xlib have their technical issues, but their actual "real world" impact on user experience is negligible and I had no more will left to continue working on this, and especially not to repeat this process on each platform.

I think this is a real clear case of underestimation and pre-mature optimization and its folly.

I still do one day want to revisit my XCB based platform code, but not any time soon.

After abandoning the platform code, I was able to get my lower level event code (via epoll) working the way I wanted by just working around the SDL code. Also I was able to build a static lib that had most of the dependancies in it. I got all that working how I wanted in just a couple hours. And then I moved on.

The rest of the month was much more productive. I spent a day making a simple build system that replaces a bunch of common code that I have in my build scripts. Some time I'll probably do a write up on it, it’s neat and makes it really easy for me to start quick one-off projects. The build system stuff was more of a quality of life improvement for me and my other projects than it really was for HUMBUG, but it was fun to work on and it got me going again. I also finally broke down and made a string library. Also learned how to use modern OpenGL (3.3). And I’ve made a decent amount of progress in getting the foundation for my GUI library. I’ve got some high hopes for what I can do with my own GUI that I could not do before.






Jeremiah Goerdt,
Despite all the set backs, that's a hugely productive month imo. I messed around a bit with Xlib before, but never to this extent. I gave up and went with SDL well before you did ;-)

As for Dear Imgui, I think it's fantastic for stuff like in-game tools and getting things started quickly, but for standalone, robust applications, you're probably better off using something else.

Good stuff!
Neo Ar,
This was a great read! I had no idea Xlib was built on top of XCB these days, that's really interesting. I dived right into XCB myself and I feel your pain regarding the lack of documentation, I spent countless hours scrounging docs, those old unfinished tutorials, and ultimately searching for source code from other programs using XCB to learn from. It's been a while but I think the reason I didn't bother with XKB was because I figured out I could get keysyms without it and liked having one less dependency.

Here are some snippets from pcalc showing how I approached the problem:
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
//on window creation
    win->keysyms = xcb_key_symbols_alloc(win->conn);
//in event handler
        case XCB_KEY_PRESS:
        {
                xcb_key_press_event_t * key = (xcb_key_press_event_t *)e;
                xcb_keysym_t sym = xcb_key_press_lookup_keysym(win->keysyms,
                        key, 0);
                switch (sym)
                {
                    case XK_Escape:
                        plt_e.sig = PLT_SIG_KEY;
                        plt_e.key_type = PLT_KEY_TYPE_SPECIAL;
                        plt_e.key.special = PLT_KEY_ESC;
                        break;
                    case XK_BackSpace:
                        plt_e.sig = PLT_SIG_KEY;
                        plt_e.key_type = PLT_KEY_TYPE_SPECIAL;
                        plt_e.key.special = PLT_KEY_BACKSPACE;
                        break;
                    default:
                    {
                            xcb_keysym_t sym1 = (key->state & XCB_MOD_MASK_SHIFT)
                                ? xcb_key_press_lookup_keysym(win->keysyms,
                                        key, 1)
                                : sym;
                            if (sym1 >= 0x20 && sym1 <= 0x7E)
                            {
                                plt_e.sig = PLT_SIG_KEY;
                                plt_e.key_type = PLT_KEY_TYPE_SYM;
                                plt_e.key.sym = (char)sym1;
                            }
                    } break;
                }
        } break;
        case XCB_MAPPING_NOTIFY:
            xcb_refresh_keyboard_mapping(win->keysyms,
                    (xcb_mapping_notify_event_t *)e);


This stuff is provided in xcb-util-keysyms. I remember that robust symbol lookup in more complicated than what I am doing though and there are a number of different symbol levels rather than just the sym1 I grab when shift is held. For a simple calculator I just need escape, backspace, and basic ASCII symbols which is why I approach it the way that I do.

I haven't looked into using epoll directly myself but I believe that is what XCB does under the hood (don't quote me on that), I'll need to do that I imagine when I get to event handling in the X11 backend but I'm still slogging through window creation...

Here are some other fun bits of xcb knowledge from my pcalc xcb backend:

To handle window deletion you actually need to request a delete window reply atom and change a property with it:
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
    xcb_intern_atom_cookie_t wm_protocols_cookie = xcb_intern_atom(
            win->conn,
            1,
            (u16)strlen(wm_protocols),
            wm_protocols);
    xcb_intern_atom_cookie_t wm_delete_win_cookie = xcb_intern_atom(
            win->conn,
            0,
            (u16)strlen(wm_delete_win),
            wm_delete_win);
    xcb_intern_atom_reply_t * wm_protocols_reply = xcb_intern_atom_reply(
            win->conn,
            wm_protocols_cookie,
            0);
    win->del_win_reply = xcb_intern_atom_reply(
            win->conn,
            wm_delete_win_cookie,
            0);
    xcb_change_property(
            win->conn,
            XCB_PROP_MODE_REPLACE,
            win->id,
            wm_protocols_reply->atom,
            4,
            32,
            1,
            &win->del_win_reply->atom);


I keep the del win reply around so that I can handle the actual event like this:
1
2
3
4
5
        case XCB_CLIENT_MESSAGE:
            if(((xcb_client_message_event_t *)e)->data.data32[0]
                    == win->del_win_reply->atom)
                plt_e.sig = PLT_SIG_QUIT;
            break;


Another thing I wanted to do was have a fixed size floating window, it turns out this actually relies on an additional protocol, the icccm protocol or the newer ewmh protocol. I don't remember the details but I went with icccm for my use case:
1
2
3
4
5
6
7
8
9
    xcb_icccm_size_hints_set_min_size(&wm_hints,
            (i32)win->width, (i32)win->height);
    xcb_icccm_size_hints_set_max_size(&wm_hints,
            (i32)win->width, (i32)win->height);
    xcb_icccm_set_wm_size_hints(
            win->conn,
            win->id,
            XCB_ATOM_WM_NORMAL_HINTS,
            &wm_hints);


And setting the window title involves changing a couple properties:
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
    xcb_change_property(
            win->conn,
            XCB_PROP_MODE_REPLACE,
            win->id,
            XCB_ATOM_WM_NAME,
            XCB_ATOM_STRING,
            8,
            (u32)strlen(title),
            title);
    xcb_change_property(
            win->conn,
            XCB_PROP_MODE_REPLACE,
            win->id,
            XCB_ATOM_WM_ICON_NAME,
            XCB_ATOM_STRING,
            8,
            (u32)strlen(title),
            title);


Regarding build systems, in pcalc I use a unity build and started the project with a simple shell script at the top of main.c:
1
2
3
4
5
6
7
8
#if 0
set -euo pipefail
musl-clang -Weverything -Wno-reserved-id-macro -Wno-documentation -Wno-documentation-unknown-command -Wno-vla -Wno-missing-prototypes -O3 -fno-unwind-tables -fno-asynchronous-unwind-tables -flto -static-libgcc $0 -o pcalc -I../3rdparty/include -L../3rdparty/lib -lXau -lxcb -lxcb-icccm -lxcb-keysyms
strip pcalc
ls -lh
./pcalc
exit
#endif


I just recently switched to a proper build script as well and wrote it in D (modified from the one I wrote for my BareD project)

Really cool to hear about you doing similar stuff in your project, keep up the great work!
David Butler,
Thanks for the support guys, and thanks for the code snippets :)
Ray,
Hey David!

Very interesting read! It was a big learning process! That's amazing!

Keep up the good work! :D