More on the UI

Duh, silly me, in my previous post I forgot that my routines in C were actually not in use anymore and was all pure python.

Then, I had some spare time to play around and get some hard data on my experiment, first let's describe a bit how that UI was done: basically win32 api's, it uses a LayeredWindow and AlphaBlend calls, this is because I wanted to see how far I could push it with plain old but well supported api's, also I wanted to have full per pixel alpha support on the root window (not just a global alpha value)
Another thing to consider, everything you saw in the video is a window (not a native Window, but my own defined window, again in python, the only real hwnd is of the root window): the title bar, each button, the background, even the scrollbars are composed of windows: the arrow buttons and the sliding block, all this was intentional as to exercise child windows and not a plain image with trickery.

In my notebook (i7 with a Radeon) the window that appears in the video is able to archive about 60 fps when resizing (just a small change in it to force a full redraw) with a close size to 1366x768, this is with no alpha pixels in the background while there are ones in the scrollbar.
Performance gets tricky here, if most of the background has alpha value (so you can see beneath it) then performance varies depending on how many pixels, it went as low as 45 fps.

Then, I made another 'driver' to suppress the alpha blending at all, so everything is opaque, while still paints over a LayeredWindow all the image ops are FillRect and BitBlt, also the call to UpdateLayeredWindow using ULW_OPAQUE, at this point the frame rate went all the way up to 120 fps, quite a speed boost.

Yet, I should try (whenever I can find the time) to use one driven by directx and possibly opengl too, i'm not too anxious to try the opengl one but I may do so.

Having said all that, it is important to remember that 60 fps is a lot for a commercial application (not for a game) and that 99.9% of the time for an application the redraw is usually a minimal part of the window.

Well then, I'm not sure if / when I'll continue with this experiment, i'm itchy about knowing the potential FPS with directx but no promises, I may or may not try it out

Hope it is of any use to some adventurous soul

-Mat

The (sorry) state of UI and some hope for the future

The user interface, the face we show to the users, our presentation card, nobody would argue that it is important right?
However, if we pay close attention to the state of it we can easily see lots of flaws, over time (in my case since the '80) many of us have even become blind to it's defects.

So, before I continue I should warn you: if you keep reading you may actually end up seeing (or I should rather say 'be more aware') many defects and imperfections in the software you use everyday, so don't blame me later if you feel displeased with your everyday interaction with your computer :)

And also, please keep in mind this applies mostly to Windows, Linux and to some extent Mac, ios and android suffers much less this issues.

Let's start our journey then, just open notepad, yes, you know what i'm talking about, everybody has it (or use gedit in linux), then resize let's say half the size of the screen should suffice.
Now, grab the top-left corner and resize it, moving it constantly to the top left corner of the screen and then to the bottom right corner, the purpose is to continuously resize the window and see how it is drawn, dont release the mouse button, just keep doing the movement and focus your view into the lower right corner of the notepad window.

If you're in windows, you can see the right scrollbar histerically jumping as we resize
If you're in linux (somewhat old versions of gnome) you'll see the same behavior, the drawing quite lagging and at a quite different possition from the mouse
If you're in linux latest gnome, it gets a bit better, yet you can easily see 'tearing' and lagging on the paint and mouse

Now this is not a notepad thing, it happens with all applications I've used, there are even worst offenders than the simple notepad, while one could argue that users are not all the time resizing their windows (or even doing it from the top left corner) it is an easy way to expose a drawing deficiency that affects in other more subtle ways.

If you do the same with the lower right corner you can see the same problem, but it is harder to see because your eyes have to track a moving target, in any case the problem is still there and still visible.

Another relatively easy flickering to spot: put some content in notepad (copy a somewhat long text into notepad), maximize notepad, then open the search dialog (ctrl + f) and close it with the escape key, do that repeatedly and you can see after a few tries that the text in notepad flashes (internally it clears the window and then paint the text back on top of it)

Another easy one? open system properties from control panel, then switch constantly between the tabs of the dialog, sometimes the painting produces a flickering.

More advanced ones? I tried yesterday with google chrome, resizing it from the lower left corner (just grab the lower left corner and bounce it around for a few seconds) and it was freezing the UI for a second or two (if there was some page loading or so)

Ok ok, there are many more presentation / drawing / transitioning glitches around but you get the idea right? so what?

Well one could argue that Mac are better at it, while they're better at UI and usability they still have their pile of UI glitches and they're bad at everything else, while Windows is better at many other things there's clearly a lot of user interface debt they have to pay, on the linux land... well, my experience has been quite bad and varied but there are tons of different windowing engines / managers / drivers and so all in different maturity, havent found yet one that does justice to today machine power capabilities

With all this I started my journey into what can be done about it, focusing mostly on windows I spent a horrible amount of time poking around ways to reduce this visual artifacts with mixed but all unsatisfactory degree of success, the problems in windows lies deep within the operating system, no matter what trick I tried.
Among other things I tried plain old win 32 apis, mfc, windows forms, WPF, java, even as far as to also use QT and GTK, none of them worked as I expected on this regard, it let me wonder why after all this years why Microsoft didn't-doesn't care / fixed (I checked this issues all the way back to windows 95, and yes, even checked it on windows 10)
After all, will you buy a car with the paint scratched or with some rust? or would you rather buy the car next to it with better / shinier paint free of rust? surely the power given by their dominance affect where they spent their efforts but man... I cannot praise or justify sloppy work on the 'presentation'.

Anyway after my long journey (countless hours of testing, prototyping and googling) I ended up going to the very basics: let's see how a modern windowing system could do this better, and did that.

My experiment consisted into relying only in a core, flat window and then do everything else by myself, effectively creating child windows, graphics primitives and all that within  my library, very similar to what Java did internally, it ended up performing like this: demo
Be warned, the quality of the filming is rather poor :)

So what was that? well, a basic windowing system that supports from the very core 32 bits per pixel alpha channel windows with flicker free and tear free drawing - effortlessly, meaning the developer doesn't need to do anything to get it 'right', displaying, moving, re-sizing, panel switching and so are truly rendered like if it is a game, it is of course modeled after what a modern computer is capable of.

Most surprising (for some of you) it is entirely coded in python 2.7 (except for 2 functions coded in c: fill and fill_rect), given the poor performance python has and the high speed archived in the demo you can easily see how feasible is to do this in a high performance language like c / c++, as my purpose was to demonstrate the feasibility it was just a great language to prototype it.

I'll be closing this post as it is getting quite long already, got some interesting insights into how a modern windowing library could be approached and I'd like to write and discuss about some of the aspects of it but that may happen in a future post. for now i'm signing off

-Mat

The poor man's garbage collector

While c++ is not my everyday thing, every time I take on it I miss some high level features like a garbage collector. Indeed it is a very polarizing topic for c++.
Smart pointers do come to give some help but they annoy me as they've their own rules to follow and they re not that simple or straightforward to use as a full-fledge gc

This weekend I had a bit of inspiration on it and come up with something to try, but be warned: I am still validating the concept, looking at different angles and frankly not entirely sure it works ... yet :)

So let's start with the basic: reference counters and the cycle problem, I wont go into details, there's plenty of reading around about how the reference counting work and the cycle problem. In my experiment I take them as the starting point, let's call my smart pointer GcPtr (which is as you guessed a c++ template, very much likely the existing ones in the c++ libraries).

It works as you may expect, so in order to break cycles I introduce the concept of 'Context', what is it? well just the function body where you are, in code a function looks like:

void do_something_interesting()
{
    Context context;
    GcPtr<Node> node(new Node());
        //a reference to itself is the most basic case of a cycle
    node->set_parent(node);
}


Well, that's not enough for showing much, but we see that starting a function with a context object allocated in the stack gives me the possibility to execute something in the destructor of the context, just when the function is going to go out of scope.
The idea here is that the GcPtr gets attached to the active context, hence if the context is destroyed everything attached to it becomes unreachable, if it wasn't claimed due to its reference count getting to 0 then it can still be reclaimed because it goes out of the context where it was allocated.
Yes, that is part of the story but not all, because a pointer can be 'passed down' to another function / level / context, let's see a more complex example:

void create_child(GcPtr<Node>& parent)
    Context context;
    GcPtr<Node> child(new Node());
    child->set_parent(parent);
}

void do_something_interesting()
{
    Context context;
    GcPtr<Node> node(new Node());
        create_child(node);
}
 
Now we got a parent-child circular reference, child was created in context 2, while node was created in context 1.
What happens internally is that inside the node class we have:

void Node::set_parent(GcPtr<Node>& parent)
{
    m_parent = parent
    parent->set_child(GcPtr<Node>(this))
    ...
 
void Node::set_child(GcPtr<Node>& child)
{
    m_child = child;
    ...

What happens here is subtle, m_child was attached to context 1 (it was created in do_something_interesting) so when it is assigned something attached to context 2 it gets promoted to context 1.
Again, child was created in context 2, but at the point we assign something that is reachable from context 1, it gets promoted, meaning context 2 does not know of it anymore and is added to context 1
Because of that, after returning from create_child, the child would still be alive (there is one reference in the node instance that references it)
Finally, after context 1 is destroyed, even as the reference count is 1 for node and 1 for child, it is destroyed as it leaves the context is attached too.

What would be the rules for using this smart pointer? pretty simple:

• keep references inside of GcPtr
• If you're creating objects in a function / method, start the function with a context creation as above
• dont touch GcPtr's in your destructors (as a rule of thumbs, I need to think it more deeply)

Neat? maybe, I did a proof of concept and works like a charm, on the positive side it is very deterministic, it does not rely on background threads neither stop-the-world syndromes as some complex GC do, there's nothing complicated like generations and so, the extra overhead is quite small.
It is still possible if desired to offload the deallocation to a background thread but that is in itself a tradeoff that will benefit some cases while punish others so it is not a win-win thing.
Another beauty is that you can go selectively using it, it is not an all-or-nothing approach.

Yet, as I mentioned before, I still have to look at it from different edges and angles, I'm not personally convinced it will handle all the cases, what if for example in the above example childs was a vector? how would the smart pointer realize the right context? not sure yet, and an elegant solution needs more poking around...

I'll publish the example code I created later on, it is just quite messy at the moment and needs heavy massaging but so far is about 50 lines of code or so

If you happen to have something to point out please do! I'm sure I'm missing stuff out.

-Mat


UPDATE:

Heh, too good to be true, not long after I wrote I realized that a class that holds a GcPtr would have to have the lowest context of any of it's reachable fields, so it still needs to climb up and traverse a chain of references, while it does not invalidate the whole concept it does put more complexity, it starts to look like a class will need a reserved field _base_context or so.
Well then, as I was suspected it was too simple to be fully working, surely I'm still missing something else.

After bashing the bash...

I've spend quite some time playing around with the usability issues of the console interface, both in windows and linux and embarked on a journey to see what could I improve.

So far the results of my experiments are nice, a simple command line 'readline' in python capable of basic editing like text selection, copy / paste without the mouse, select word, next / prev, etc etc
Thing is, done with a MVC pattern was easy peasy and got done in no time, even adapting it for a windowing environment would be rather trivial but the parsing thing lead me to implement an EBNF reader in python which indeed took a bit of an effort.
The ebnf reader was done with a non-recursive algorithm for performance and limitations related to python for recursive calls (it does support recursive calls but then they're limited and has some issues), in short the core parser is a one-function non-recursive state machine in the order of 200+ lines of python code.

I may release it if there's interest, it's usage is simple: you feed a ebnf text and returns a grammar, you call parse passing a string and returns an AST (abstract syntax tree) so there's no intermediate steps like traditional yacc / lex / bison and so.

Still, on the usability front, played with few ideas, the one i'm currently after is implementing autocomplete and 'in place' editor, what's the in place editor?
Imagine you're typing something, say:

mycommand.py "my string arg is \"1\""

The oddity here is to have to escape the string, i've been wondering about having some sort of 'enter editor mode' key that would let me write the string normally and will escape it for me. for example once the cursor is inside the quotes I could press say ctrl+enter or so and the line switches to something like:

mycommand.py "my string arg is "1""

then just ctrl+enter to exit the 'string edit mode' or just leave the section moving with cursors or shortcuts so as to have it escaped for me.
The idea is simple, while not terribly usable but explores the idea of 'in place specialized editor' that can be used for other things besides auto escaped strings

So next things next, need to plug the ebnf with the autocomplete and do a proof of concept of the in-place subeditor, surely it will take some iterations to get it easily usable.
Then who knows, i'll be poking around with the ebnf and user inputs, maybe something iteresting arises out of it.

-Mat

Bashing the bash, usability sucks

Oh, not really, I don't meant to rant about Bash but about usability in general.

It has been quite some days that I started thinking about usability from the bottom up, if I would have to start an operating system from scratch, what would I do different, what would I change?

By operating system, I mean the user interface, be it command line or windowed environment, while operating systems had advanced a great deal in terms of handling the new hardware, task scheduling, memory management and all that low level stuff It feels like the user has been abandoned in many ways.

Let's start ranting :)

The command line sucks big time, there is no denying that it is extremely powerful, you can accomplish incredible things that are near impossible in a windowed environment, however, in any linux or windows distribution nowadays can you actually do something as simple as copy paste WITHOUT using the mouse? I'm no linux expert but I wasn't able to find a way to do it, yes you can ctrl + shift + c but how do you actually select text without the mouse?
Even if there is a way or a special console or terminal it is entirely up to you to find, install and configure it, face it, both windows command line and linux console / terminal did a poor job in modernizing some stuff.
Windows console is by any means extremely primitive, not even worth ranting at it so let's not even talk about it, what about linux?
Well, we start with a terminal (not the same as the console, however usability wise they're not that far apart), the terminals starts with an emulator of an old VT100, old? according to wikipedia "It was introduced in August 1978" oh my god... while most terminal emulators starts from there isn't it time to 'upgrade' from 1978?
Does it gets the job done? yes, can we do better for the end user? sure! please! at least a way to select text without using the mouse!

So here is some 'idea' (and i'm being sarcastic here...) why don't we do so that we can select text by pressing shift and the arrow keys? following up that line, lets also make use of the backspace key in a sane way (console and terminal hackers know what I mean), home, end and all the other basic things, while we're at it, copy paste would be appreciated (Thanks Larry Tesler, who's credited as the inventor of it somewhere between 1973-1976), or is it so that "real men don't copy/paste, they make holes in cards"?

With this thoughts I embarked on a new adventure and started coding some experiments that go further than that, only to open a can of worms... linux configuration files for the editors and terminals...
Before continuing, my disclaimer: "I am not an advanced linux user, I poked around a bit and am in the process of learning more"

So here's what I found out: it sucks, vi has it's keyboard bindings, emacs it's own way, the terminal another one, and while we're at it, the console is another story.
Oh my... here's another 'idea', a central configuration for the user preferences, for example:
ACTION_SELECT_LINE: KEY_CONTROL + KEY_L (for example)
Ideally nano, vi /vim, emacs and the usual suspects could use system provided default action bindings while allowing local customizations for maximum flexibility. (there is some attempt in some config files for termscaps and so, but is far from working). When I think about the students getting themselves for the first time with these tools I cant help but feel sorry for having to deal with such usability mess.
You know, I don't deny the power of emacs or vi, but come on... the power wont go away by being friendly towards me...

Where did my observations lead me? well I started a small experiment, while I'm not sure where I'm heading it is surely illustrative and entertaining to a degree, this is what I'm up to:

A system level text input mechanism
Must support pluggable parsers with cascade capabilities (nesting languages)
Must provide letter, word, paragraph and document 'pluggable services'
Must use global configuration for key bindings to actions with possibility of local override configuration
Must work in windowed, console environments (web pages is a plus)

Things I'd like to achieve:

enter a command in the command line, strings are escaped for me (parser integrated into the text input, or think of it as a subeditor you can easily enter / exit 'in place')
autosuggest / text completion 'bash style' everywhere (as long as you set the right parser for the input)
context dictionary suggestions
nearly ready for use in code editors, given of course the parser and lexers necessary for it
somehow standard way of a program to exhibit it's command line parameters for autocomplete in the command line, to some degree bash does this (I think it has a database of programs with their parameters, need to dig deeper into how it does it)

Why? just because!

-Mat