Basti's Scratchpad on the Internet
24 Mar 2024

ðŸŠĶ Emacs 2011-2023

For the last dozen years, I have used Emacs as my text editor and development environment. But this era ended. In this post, I outline how I went from using Emacs as a cornerstone of my digital life, to abandoning it.

In an ironic twist of history, it was Visual Studio that drove me to Emacs in the first place, and what ultimately pulled me away from it: In 2011, I was working on the firmware of a digital mixing console. This was edited in Visual Studio, compiled with an embedded compiler software, and source-controlled with command-line Git. It was ultimately Emacs that allowed me to tie this hodgepodge of idiosyncratic C+1, Git, and the proprietary compiler into a somewhat sane development environment.

Over the years, my Emacs config grew, I learned Elisp, I published my own Emacs packages, and developed my own Emacs theme. I went back to university, did my PhD, worked both OSS and commercially, and almost all of this was done in Emacs. As particular standouts beyond traditional text editing, I used Emacs' Git-client Magit every single day, and my own org-journal was absolutely vital as my research/work journal.

My monochrome Emacs theme
My custom Emacs theme, all monochrome, with varying fonts instead of colors

In 2023, however, I started a new job, once again with a Visual Studio codebase. This time, however, the code base and build system was tightly woven into the Visual Studio IDE, and only really navigable and editable therein. It thus made no sense to edit this code in Emacs, so I didn't. Perhaps I also needed a break.

And as my Emacs usage waned, so its ancient keyboard shortcuts started to become a liability. I started mis-typing Emacs things in Visual Studio, and hitting Windows shortcuts in Emacs. Friction began to arise. At the same time, I started noticing how poorly Emacs runs on Windows. Startup takes many seconds, it does not integrate well into the task bar2, it doesn't handle resolution changes gracefully, and it's best I don't start talking about its horrendously broken mouse scrolling. And of course it can't scroll point out of the window3.

My last use-case for Emacs was org-journal. I ended up porting a basic version of it to Visual Studio Code. Having thus written a text editor plugin for both editors, I have to be blunt: both, the anachronistic bone-headedness of Elisp, and the utter insanity of TypeScript's node APIs, are terrible environments for writing plugins. A few years ago I did the same exercise in Sublime Text Python, which was a beautiful, simple, quick affair. But I do enjoy a programming puzzle, so here we are.

The final nail in Emacs' coffin came from an unexpected corner: For all my professional life, I was a solo coder. My Emacs was proudly black-and-white (different fonts instead of different colors!), and my keyboard shortcuts were idiosyncratically my own. I did not merely use Emacs. I had built MY OWN Emacs. I like to think this built character, and API design experience. But it was of course a complete non-starter for pair programming. After having tasted Visual Studio (Âą Code) Live Sharing, there was simply no going back.

And thus, I am saddened to see that I haven't started Emacs in several weeks. I guess this is goodbye. This blog is still rendered by Emacs, and I still maintain various Emacs modules. My journal is still written in org-mode. But it is now edited in Visual Studio Code.

Footnotes:

1

An eclectic subset of C++, intersected with the limitations of the embedded compiler. This was decidedly pre-"modern" C++, and probably less than the sum of its parts.

2

Usually, the program's taskbar button starts the program, and represents it while running. Emacs spawns a new button instead.

3

This is emacs-speak for "it can't scroll the cursor outside the viewport"

Tags: emacs

Two Years with Legacy Code

From January 2021 to the beginning of 2023, I worked on a legacy code base at Fraunhofer IDMT in Oldenburg. My task was the maintenance and development of a DNN-based speech recognition engine that had become terra incognita when its original developer had left the company a year before I started. The code had all the hallmarks of severe technical debt, with layers of half-used abstractions, many unused branches of unknown utility, and the handwriting of several concurrent programmers at odds with each other.

The code had evidently been written in a mad dash to bring the product to market. And not to discredit its developers, had been in production for several years, with a core of robust algorithms surrounded by helper scripts that had allowed the company to build upon, even after the original developers had left.

It was my job to clean it up. Having spent six years on my PhD recently, I welcomed the calmer waters of 'just' programming for a bit. This blog post is a summary of the sorts of challenges I faced during this time, and what kinds of techniques helped me overcome them.

The lay of the land

I approached the task from the outside, sorting through the build scripts first. Evidently, at least three authors were involved: One old-school Unix geek that wrote an outdated dialect of CMake, one high-level Python scripter, and one shell scripter that deeply believed in abstraction-by-information-hiding. The result of this was… interesting.

For a good few weeks I "disassembled" these scripts by tracing their execution manually through their many layers, and writing down the necessary steps that were actually executed. My favorite piece of code was a Makefile that called a shell script that ran a Python program, which instantiated a few classes and data structures, which ultimately executed "configure; make; make install" on another underying Makefile. I derived great satisfaction from cutting out all of these middle-men, and consolidating several directories of scripts into a single Makefile.

Similar simplifactions were implemented at the same time across several code bases by my colleagues. In due time, this concerted effort enabled us to implement continuous integration, automated benchmarking, and automated builds, but more on that later.

Data refactoring

The speech recognition software implemented a sort of interpreter for the DNN layers, originally encoded as a custom binary blob. Apparently, a custom binary approach had been taken to avoid dependencies on external parsing libraries. Yet the data had become so convoluted that both its compilation and its parsing were now considered unchangeable black boxes that impeded further development.

Again, I traced through the execution of the compiling code, noted down the pieces of data it recorded, and rewrote the compiler to produce a MsgPack file. On the parsing side, I wrote a custom MsgPack parser in C. Looking back, every job I've had involved writing at least a couple of data dumpers/parsers, yet many developers seem intimidated by such tasks. But why write such a thing yourself instead of using an off-the-shelf solution? In an unrelated code review later in the year one colleague used the cJSON library for parsing JSON; in the event, cJSON was several magnitudes bigger and more complex than the code base it was serving, which is clearly absurd. Our job as developers is to manage complexity, including that of our dependencies. In cases such as these, I often find a simple, fit-for-purpose solution preferable to more generalized external libraries.

A part of the DNN data came from the output of a training program. This output however was eternally unstable, often breaking unpredictably between version, and requiring complex workarounds to accommodate different versions of the program. The previous solution to this was a deeply nested decision tree for the various permutations the data could take. I simplified this code tremendously by calling directly into the other program's libraries, instead of trying to make sense of its output. This is another technique I had to rely on several times, hooking into C/C++ libraries from various Python scripts to bridge between data in a polyglot environment.

Doing these deep dives into data structures often revealed unintended entanglements. In order to assemble one data structure, you had to grab pieces of multiple different source data. Interestingly, once data structures were cleaned up to no longer have such entanglements, algorithms seemed to fall into place effortlessly. However, this was not a one-step process, but instead an ongoing struggle to keep data structures minimal and orthogonal. While algorithms and functions often feel easier to refactor than data structures, I have learned from this that it is often the changes to data structures that have the greatest effect, and should therefore receive the greatest scrutiny.

Code refactoring

My predecessor had left me a few screen casts by way of documentation. While the core program was reasonably well-structured, it was embedded in an architectural curiosity that told the tale of a frustrated high-level programmer forced to do low-level gruntwork. There were poor-man's-classes implemented as C structs with function pointers, there were do-while-with-goto-loops for exception handling, there were sort-of-dynamically-typed data containers, accompanied by angry comments decrying the stupidity of C.

Now I like my high-level-programming as much as the next guy, but forcing C to be something it isn't, is not my idea of fun. So over a few months I slowly removed most of these abstractions. Somewhat to my surprise, most of them turned out pure overhead that could simply be removed. Where a replacement was needed, I reverted to native C constructs. Tagged unions instead of casting, variable-length-arrays instead of dynamic arrays. Treating structs as values instead of references. This, alone, reduced the entire code base by a good 10%. The harder part was sorting out the jumble of headers and dependencies that had evidentally built up over time. Together with the removal of dead code paths, the overall code base shrank by almost half. There are few things more satisfying than excising and deleting unnecessary code.

I stumbled upon one particularly interesting problem when trying to integrate another code base into ours. Within our own software, build times were small enough to make logging and printf-debugging easier than an interactive debugger such as GDB. The other code base however was too complex to recompile on a whim, and a different solution had to be found. Now I am a weird person who likes to touch the raw command line instead of an IDE. And in this case this turned out to be a huge blessing, as I found that GDB can not only be used interactively, but can also be scripted! So instead of putting logging into the other library, I wrote GDB scripts that augmented break points with a little call printf(...) or print/d X. These could get suprisingly complicated, where one breakpoint might enable or disable other breakpoints conditionally, and break point conditions could call functions on their own. It took some learning, but these debugging scripts were incredibly powerful, and a technique I will definitely refer to in the future.

When adding new features to the software, I often found it impossible to work the required data flow into the existing program code without snowballing complexity. I usually took these situations as code smells that called for a refactoring. Invariably, each cleaning up of program flow or data structures inched the program closer and closer to allow my feature addition. After a while, this became an established modus operandi: independently clean the code until feature additions become easy and obvious, then do the obvious thing. Thus every task I finished also left the surrounding code in a better state. In the end, about 80% of the code base had gotten this treatment, and I strongly believe that this has left the project in a much better state than it was before. To say nothing of the added documentation and tests, of course.

More velocity makes bigger craters

As I slowly shifted from cleanup work to new features, change management became a pressing issue. New features had to be evaluated, existing features had to be tested, and changes had to be documented and downstreamed. Fascinatingly, the continuous integration and evaluation tools we built for this purpose, soon unearthed a number of hidden problems in other parts of the product that we had not been aware of (including that the main task I had been hired to do was less worthwhile than thaught, LOL). That taught us all a valuable lesson about testing, and proving our assertions. That said, I never found bottom-level unit tests all that useful for our purposes; the truly useful tests invariably were higher-level integration tests.

Eventually, my feature additions led to downstream changes by several other developers. While I took great care to present a stable API, and documenting all changes and behavior appropriately, at the end of the day my changes still amounted to a sizeable chunk of work for others. This was a particularly stark contrast to the previous years of perfect stagnation while nobody had maintained the library. My main objective at this point was to avoid the mess I had started out with, where changes had evidentally piled on changes until the whole lot had become unmaintainable.

Thus a balance had to be struck between moving fast (and breaking things), and projecting stability and dependability. One crucial tool for this job turned out to be code reviews. By involving team members directly with the code in question, they could be made more aware of its constraints and edge cases. It took a few months to truly establish the practice, but by the end of a year everyone had clearly found great value in code reviews as a tool for communication.

Conclusions

There is a lot more to be said about my time at Fraunhofer. The deep dive into the world of DNN engines was truly fascinating, as were the varied challenges of implementing these things on diverse platforms such as high-performance CPU servers, Laptops, Raspberry Pis, and embedded DSPs. I learned to value automation of developer tasks, and of interface stability and documentation for developer productivity.

But most of all, I learned to appreciate legacy code. It would have been easy to call it a "mess", and advocate to rewrite it from scratch. But I found it much more interesting to try to understand the code's heritage, and tease out the algorithmic core from the abstractions and architectural supports. There were many gems to be found this way, and a lot to be learned from the programmers before you. I often felt a strange connection to my predecessor, as if we were talking to each other through this code base. And no doubt my successor feels the same way about my code now.

Tags: programming

OS Customization and MacOS

I was an Apple fanboy some years ago. Back then, whenever something was odd on my computer, I was surely just using it wrong. Nowadays, I see things the other way around: We're not "holding it wrong", the computer is just defective. Computers should do our bidding, not vice versa. So here's a bunch of things that I do to my computers to mold them to my way of working.

Keyboard Layout

I switch constantly between a German and English keyboard layout, and regularly between various machines. My physical keyboards are German, and my fingers are used to the Windows-default German and (international) US keyboard layout. These are available by default on Windows and Linux, but MacOS goes its own way.

However, keyboard layouts on MacOS are saved in relatively simple text files, and can be modified with relative ease. The process goes like this: Download Ukelele (free) to create a new keyboard layout bundle for your base layout1. Inside that bundle, there's a *.keylayout file, which is an XML file that defines the characters that each key-modifier combination produces. I changed that to create a Windows-like US keyboard layout. And I replaced the keyboard icon with something sane (not "A") by creating a 256x256 pixel PNG, opening it in Preview, holding alt while saving to select the ICNS format. Save the keyboard bundle to ~/Library/Keyboard Layouts and reboot. Then I remove the unremovable default ("A") German layout by selecting another one, then plutil -convert xml1 ~/Library/Preferences/com.apple.HIToolbox.plist, and delete the entry from AppleEnabledInputSources. Now reboot. Almost easy. Almost.

One the one hand, this was quite the ordeal. On the other, I have tried to do this sort of thing on Windows and Linux before, and for the life of me could not do it. So I actually think this is great!

Keyboard Shortcuts

My main text editor is Emacs, and I am very used to its keyboard shortcuts. Of particular note are CTRL-A/E for going to the beginning/end of a line, and Alt-B/F for navigating forward/backwards by word. I have long wanted to use these shortcuts not just in Emacs and readline-enabled terminal applications, but everywhere else, too. And with MacOS, this is finally possible: Install BetterTouchTool ($22), and create keyboard shortcuts that maps, e.g. Alt-B/F to Alt-←/→. Ideally, put this in a new activation group that excludes Emacs. It may be necessary to remove the keyboard character for Alt-B/F from your keyboard layout before this works. I've spent an embarrassing number of hours trying to get this to work on Windows and Linux, and really got nowhere2. Actually, however, most readline shortcuts such as Ctrl-A/E/B/F/K/Y already work out of the box on MacOS!

Mouse Configuration

I generally use a trackpad, and occasionally a traditional mouse for image editing, and have used a trackball. I find that any one specific device will lead to wrist pain if used constantly, so I switch it up every now and then. The trackpad and trackball, however, need configuration to be usable.

After experimenting with many a trackpad device, I have found Apple touch pads the best trackpads on the market3. On MacOS, they lacks a middle mouse click. So I created a trackpad shortcut in the aforementioned BetterTouchTool ($22) to map the middle click on a three-finger tap (can also be had for free with MiddleClick (OSS)). For Windows, Magic Utilities ($17/y) provides a wonderful third-party driver for Apple devices that also supports the three-finger tap. I have not gotten the Apple touch pad to pair reliably on Linux, and have generally found their touch pad driver libinput a bit lacking.

My trackball is a Kensington SlimBlade. To scroll, you rotate the ball around its vertical axis. This is tedious for longer scroll distances, however. But there's an alternative scrolling method called "button scrolling", where you hold one button on the trackball, and move the ball to scroll. You need to install the Kensington driver to enable this on Windows and MacOS. Button scrolling is available on Linux as well using xinput4, but I haven't gotten the setting to stick across reboots or sleep cycles. So I wrote a background task that checks xinput every five seconds, which does the trick.

Window Management

Frankly, Windows does window management correctly. Win-←/→ moves windows to the left and right edge of the screen, as does dragging the window to the screen border. Further Win-←/→ then moves the window to the next half-screen in that direction, even across display boundaries. KDE does this correctly out of the box as well, Gnome does not do the latter, and it drives me mad. MacOS doesn't do any of these things. But Rectangle (OSS) does. Easy fix. (BetterTouchTool can do it, too, but Rectangle is prettier)

Furthermore, I want Alt-Tab to switch between windows. Again, MacOS is the odd one out, which uses CMD-Tab to switch between apps, now windows. And then there's another shortcut for switching between windows of the same app, but the shortcut really doesn't work at all on a German keyboard. Who came up with this nonsense? Anyway, Witch ($14) implements window switching properly.

Application Management

In Windows and Linux, I hit the Windows key and start typing to select and start an app. In MacOS, this is usually Cmd-Space, but BetterTouchTool can map it to a single short Cmd, if you prefer.

More annoying are the various docks and task bars. I always shove the dock off to the right edge of the screen, where it stays out of the way. Windows 10 had a sane dock, but then 11 came and forced it to the bottom of the screen. Dear OS makers, every modern screen has plenty of horizontal space. But vertical space is somewhat limited. So why on earth would you make a rarely used menu such as the dock consume that precious vertical space by default? And Microsoft, specifically, why not make it movable? Thankfully, there's StartAllBack ($5), which replaces the Windows task bar with something sensible, and additionally cleans up the start menu if you so desire. On KDE, I fractionally prefer Latte (OSS) over KDE's native dock. The MacOS dock is uniquely dumb, offering no start menu, and allowing no window selections. But it's unobtrusive and can be moved to the right edge, so it's not much of a bother.

File Management

One of the most crucial tasks in computer work in general is file management. I am not satisfied with most file managers. Dolphin on KDE works pretty well, it has tabs, can bulk-rename files, can display large directories without crashing, and updates in real time when new files are added to the current directory. Gnome Nautilus is so bad it is the main reason I switched to KDE on my Linux machines. Finder on MacOS is passable, I suppose, although the left sidebar is unnecessarily restrictive (why can't I add a shortcut to a network drive?). Windows Explorer is really rather terrible, lacking a bulk-rename tool, and crucially, tabs. In Windows 10, these can be added with Groupy ($12) (set it to only apply to explorer.exe). Windows 11 has very recently added native tabs, which work OK, but can't be detached from the window.

The sad thing is that there are plenty of very good file manager replacements out there, but none of the OSs have a mechanism for replacing their native file manager in a consistent way, so we're mostly stuck with the defaults.

Oh, and I always remove the iCloud/OneDrive sidebar entries, which is surprisingly tedious on Windows.

Hardware Control

On laptops, you can control screen brightness from your keyboard. On desktops, you can not. However, some clever hackers have put together BetterDisplay (OSS for screen brightness), which adds this capability to MacOS. That's actually a capability I have wanted for quite a while, and apparently it is only available in MacOS. Great stuff!

Less great is that MacOS does not allow volume control on external sound cards. SoundSource ($47) adds this rather crucial functionality back, once you go through the unnecessarily excruciating process of enabling custom kernel extensions. Windows and Linux of course natively support this.

Another necessary functionality for me is access to a non-sucky (i.e. no FAT) cross-platform file system. At the moment, the most portable file system seems to be NTFS, of all things. Regrettably, MacOS only supports reading NTFS, but no writing. Paragon NTFS (₮20) adds this with another kernel extension, and promptly kernel-panicked my computer. Oh joy. At least it's only panicking for file transfers initiated by DigiKam, which I can work around. Paragon Support says they're working on it. I'm not holding my breath. Windows and Linux of course natively support NTFS.

System Management

I have learned from experience not to trust graphical backup programs. TimeMachine in particular has eaten my backups many times already, and can not be trusted. But I have used Borg (OSS) for years, and it has so far performed flawlessly. Even more impressive, my Borg backups have a continuous history despite moving operating systems several times. It truly is wonderful software!

On Windows, I run Borg inside the WSL, and schedule its backups with the Windows Task Scheduler. On Linux, I schedule them with systemd units. On MacOS, I install Borg with Homebrew (OSS) and schedule the backups with launchd tasks. It's all pretty equivalent. One nice thing about launchd, however, is how the OS immediately pops up a notification if there's a new task file added, and adds the task to the graphical system settings.

I have to emphasize what a game-changer the WSL is on Windows. Where previously, such simple automations where a pain in the neck to do reliably, they're now the same simple shell scripts as on other OSes. And it perfectly integrates with Windows programs as well, including passing pipes between Linux and Windows programs. It's truly amazing! At the moment, I'd rate Windows a better Unix system than MacOS for this reason. Homebrew is a passable package manager on MacOS, but the way it's ill-integrated into the main system (not in system PATH) is a bit off-putting.

App Compatibility

I generally use my computer for three tasks: General document stuff, photo editing, and video games.

One major downside of Apple computers is that video games aren't available. This has become less of a problem to me since I bought a Steam Deck, which has taken over gaming duties from my main PC. Absolutely astonishingly, the Steam Deck runs Windows games on Linux through emulation, which works almost flawlessly, making video games no longer a Windows-only proposition.

What doesn't work well on Linux are commercial applications. Wine generally does not play well with them, and frustratingly for my photo editing, neither VMWare Workstation Player (free) nor VirtualBox (OSS) support hardware-accelerated VMs on up-to-date Linux5. So where MacOS lacks games, Linux lacks Photoshop. Desktop applications in general tend to be unnecessarily cumbersome to manage and update on Linux. Flatpak is helping in this regard, by installing user-facing applications outside of the OS package managers, but it remains more work than on Windows or MacOS. The occasional scanner driver or camera interface app can also be troublesome on Linux, but that's easily handled with a VirtualBox VM (with the proprietary Extension Pack for USB2 support), and hasn't really bothered me too much.

Luckily for me, my most-used apps are generally OSS tools such as Darktable (OSS) and DigiKam (OSS), or cross-platform programs like Fish (OSS), Git (OSS), and Emacs (OSS). This is however, where Windows has a bit of a sore spot, as these programs tend to perform noticeably worse on Windows than on other platforms. Emacs and git in particular are just terribly slow on Windows, taking several seconds for routine operations that are instant on other platforms. That's probably due to Windows' rather slow file system and malware scanner for the many-small-files style of file management that these Unix tools implement. It is very annoying.

Conclusions

So there's just no perfect solution. MacOS can't do games, Linux can't run commercial applications, and Windows is annoyingly slow for OSS applications. Regardless, I regularly use all three systems productively. My job is mostly done on Windows, my home computer runs MacOS, and my Steam Deck and automations run Linux.

Overall, I currently prefer MacOS as my desktop OS. It is surprisingly flexible, and more scriptable than I thought, and in some ways is actually more functional than Linux or Windows. The integrated calendar and contacts apps are nice, too, and not nearly as terrible as their Windows/Linux counterparts. To say nothing of the amazing M1 hardware with its minuscule power draw and total silence, while maintaining astonishing performance.

Linux is where I prefer to program, due to its sane command line and tremendously good compiler/debugger/library infrastructure. As a desktop OS, it does have some rough edges, however, especially for its lack of access to commercial applications. While Linux should be the most customizable of these three, I find things tend to break too easily, and customizations are often scattered widely between many different subsystems, making them very hard to get right.

Theoretically, Windows is the most capable OS, supporting apps, OSS, and games. But it also feels the most user-hostile of these three, and the least performant. And then there's the intrusive ads everywhere, its spying on my every move, and at work there's inevitably a heavy-handed administrator security setup that gets in the way of productivity. It's honestly fine on my home computer, at least since they introduced the WSL. But using it for work every day is quite enough, so I don't want to use it at home, too.

Footnotes:

1

if there's an easier way to create a keyboard layout bundle, please let me know. I didn't use ukelele for anything but the bundle creation.

2

It occurs to me that it might actually be possible to do something like on Windows this with AutoHotkey (free). I'll have to try that!

3

The Logitech T650 is actually not bad, but the drivers are a travesty. Some Wacom tablets include touch, too, but they palm rejection is abysmal and they don't support gestures.

4

xinput set-prop "Kensington Slimblade Trackball" "libinput Scroll Method Enabled" 0, 0, 1 # button scrolling
xinput set-prop "Kensington Slimblade Trackball" "libinput Button Scrolling Button" 8 # top-right button

5

VirtualBox does not have a good accelerated driver, and VMWare does not support recent kernels. Qemu should be able to solve this problem, but I couldn't get it to work reliably.

Tags: computers linux macos windows

Media of 2022

Around this time of the year, I usually write a blog post about my favorite books of the last year. But this time, not enough of them really stood out. So instead, here's my favorite pieces of media I consumed in the last year:

Firepower

book cover for Firepower

This is the story of how gunpowder changed the world. We get to see the familiar history of central Europe, but told unusually, from the bottom up: how seemingly small inventions change the course of peoples and nations. The book is in essence a history of warfare in the last few hundred years, but this time the movers and shakers are not Great Men, but chemistry and engineering. A fascinating perspective, to say the least.

Along the way, the book unraveled countless tangents and quirks I had previously stumbled upon, yet never understood. How castles went from tall walled structures to flat earthworks in the 18th century due to the disruptive invention of cannons. How wooden galleons of 1850 were obsoleted by turreted iron warships practically overnight. How rifling and shells bled dry the coffers of Central Europe and made conflict inevitable. This book contextualized many a story like this in the most riveting manner!

Steam Deck

steam deck controller

Ever since we had our second child, I didn't really find the time for video games any more, much to my regret. So when the Steam Deck was announced, essentially a portable gaming computer, it didn't feel like a worthwhile investment.

But boy, was I wrong about that. The genius of the Steam Deck is how it's instantly-on, instantly-off like a video games console, allowing me to play in short bursts that would not otherwise be available to gaming. But in contrast to a console, the deck allows me to play games without blocking the living room, and away from the computer screen I'm working at all day anyway.

It has reignited my video gaming, and surprisingly not just for newer titles, but thanks to EmuDeck, emulated retro games as well. I had a blast playing Banjo-Kazooie with my daughter, and Chorus, Ace Combat, Guardians of the Galaxy and Death Stranding on my own. To me, this is a revolutionary device, and I haven't touched my gaming PC or any of my video game consoles since!

Tags: books media

RAW Developer Comparison 2

It's that time of the year again when all image editing programs come out with new versions. This comes at an inopportune time, as I feel restless of late. So, naturally, I had to try them all. Or maybe I just wanted a justification for buying DxO PhotoLab, because people on the internet speak so well of it 🙄.

For the following comparison I downloaded trial versions of ACDSee Photo Studio Ultimate 2023 (₮155), Capture One 22 (₮350 or ₮220/a), Darktable 4.0 (free), DxO Photolab 6 (₮220 + ₮140 for FilmPack 6 + ₮99 for ViewPoint), Adobe Lightroom Classic 11.5 (₮142/a), ON1 Photo Raw 2023 (₮126), RawTherapee 5.8 (free), Silkypix Developer Studio Pro 11 (₮155), Exposure X7 (₮165), and Zoner Photo Studio X Fall 2022 (₮60/a). I also installed Luminar Neo (₮120 or ₮80/a) and Radiant Photo (₮140) but I disliked them so immediately and viscerally that I didn't include them in the comparison below.

In order to put them through their paces, I took a random smattering of images from the last few years that I found difficult to work with for one reason or another, and checked how each of the programs dealt with them. Of course I am no expert in any of them except Darktable and Capture One, so my results are probably flawed. I tried to inject some objectivity by limiting my edits to the most obvious sliders wherever possible, especially in the programs I know better. Regardless, my comparison is probably less scientific than last time, because my brain sort of broke after staring at too many renderings of the same images for too long. Don't try this at home, folks!

Test Case One: Fire

Fire is notoriously difficult to deal with, because it covers a large dynamic range, and we have strong associations of certain colors with heat. Thus a more yellow rendering usually suits fire very well, whereas a more neutral rendering quickly looks unnatural. In particular, we seem to expect highlights to twist to yellow, and midtones to remain orange.

My editing goals for this picture are simple: raise shadows or blacks until the pan becomes faintly visible, and reign in highlights if necessary to prevent excessive clipping.


Fujifilm X-Pro2 with XF35mmF1.4 R 1/1000s f/2.2 ISO400
📂 DSCF9359.RAF (23.0 MB) Creative Commons License

There's a difficult tradeoff to be made here: Fire highlights physically don't change hue much compared to the fire body, but we visibly expect "hotter" fire to twist yellow. So a balance has to be struck between merely desaturating highlights, and twisting them yellow. You can see Darktable v6, DxO PhotoLab, RawTherapee, Exposure, and Zoner leaning towards desaturation, while the others render yellow color twists of some form or another. ACDSee and ON1 probably take the yellow a bit too far.

However, there's a flip side to this: Every program that renders yellow highlights in fire, does the same for overexposed skin, twists overexposed skies cyan, and red flowers magenta. It's a difficult tradeoff to get right. This is especially tricky since color shifts in highlights are often baked deeply into the programs' color science, and hard to get rid off where they're unwanted.

Anyway, to my eyes, the only fire-like renderings here come from Capture One, Darktable v5, PhotoLab, and Lightroom.

A Hazy Mountain

This lovely scene is a deep crop into an image taken at long distance on a hazy day. The challenge is therefore dehazing the mountains, chiseling out the contours with clarity and texture and contrast, and removing any extraneous noise.

This is a difficult task, as emphasizing local contrast tends to produce halos near the horizon, dehazing tends to introduce unpleasant color shifts, and the deep crop necessitates a subtle balance between denoising and detail retention.


Fujifilm X-T2 with XF70-300 @ 231mm 1/400s f/5.7 ISO200
📂 DSF4442.RAF (23.0 MB) Creative Commons License

Overall, most of these renderings turned out fine. But there are pretty substantial differences in denoising, haloing, and detail retention of the shadows specifically. Halos especially are a personal bug-bear of mine, where the sky above the horizon line brightens and (worse) the mountains below the horizon line darken. This is especially visible in ACDSee, ON1, and Zoner, and to a lesser extent in DxO, Lightroom, and RawTherapee.

There were also significant differences in noise removal. Especially ACDSee, Exposure, and Zoner had weirdly ineffective denoising, and were unusually difficult to balance against smearing and worms in Lightroom, ON1, and Silkypix. Colors were a bit difficult to control in ACDSee, ON1, and RawTherapee.

The most pleasant renderings to my eye are DxO, Exposure, and Darktable in this round, although many of the others are very usable as well.

High Key Rendering

Contrary to the popular style at moment, I sometimes like my highlights a little blown. I especially like a smooth, desaturated, film-like drop-off in my highlights, which seems strangely difficult to replicate in digital photography.

So instead of turning the following picture into an HDR hellscape, I want to compress the bright background into the highlights, while expanding the dark foreground to fill the midtones. The capture actually has all highlight information intact, so I don't want to blow anything out completely, just gracefully fade it into pastels.


Fujifilm X-Pro2 with XF16-80mm @ 16mm 1/400s f/9 ISO400
(no raw, for privacy reasons)

The different renderings of this image are more of a matter of preference than any of the previous ones. To my eyes, I like the versions drawn by ACDSee, Capture One, Zoner Photo, and DxO PhotoLab best, and I recon that many of the other versions could have been improved with a little more manual tuning. The only problematic versions of this image were the strange HDR-like ON1 render, and the Silkypix variant with its lost black point.

Summary

Overall, all of these programs seem reasonable tools for image editing. Most of their differences can probably be overcome if you learn them better. That said, this comparison has left me with a few clear favorites. To be clear, the above comparison only showed the most decisive images of a much larger set which informs my opinion. It should also be noted that my tastes in image editing do not focus on detail recovery, detailed masking, nontrivial retouching, or scene-changing edits such as wholesale sky swapping.

One area I am particularly interested in, however, is the inherent complexity of the editing tools: For example, I like my saturation slider to only change saturation and leave lightness alone. Similarly, contrast adjustments should not affect saturation. Another interesting part is the behavior of highlights adjustments. Ideally, I'd like highlights to be able to counteract exposure adjustments so I can balance them against one another. Better yet if the same can be done with the tone curve.

Name Editing is Export takes Saturation changes Contrast changes Highlights rescues Tonecurve rescues
      lightness? saturation? overexposure? overexposure?
ACDSee Photo Studio realtime 25s yes yes yes no
Capture On realtime 15s a little no yes no
Darktable delayed 15s no no yes yes
DxO PhotoLab delayed 45s yes a little yes a little
Lightroom Classic realtime 15s a little a little yes a little
ON1 Photo RAW realtime 30s yes a little no no
RawTherapee wait and see 30s a little yes no no
Silkypix Developer lo-res wait/see 80s no a little no no
Exposure X7 realtime 30s yes strongly yes a little
Zoner Photo Studio lo-res realtime 30s yes yes yes a little

If saturation changes lightness and contrast changes saturation, editing can become rather more difficult, as the effect tends to be hard to counteract without complex luminosity masking. This is a reason for me to dislike my experience with ACDSee, ON1, Exposure, and Zoner particularly. The highlights slider issue also has a big influence on how you edit images. If highlights can be recovered after exposure adjustments, you can use exposure mostly for image brightness, and recover highlights later if needed. On the other hand, if highlights can't be recovered, then the exposure slider must instead be used to protect highlights, and image brightness has to be relegated to the tone curve or shadows/midtones sliders. This feels weirdly backwards to me, and is a reason for me to disregard ON1, RawTherapee, and Silkypix.

The gold standard here is of course Darktable, where saturation ("chroma") is completely decoupled from lightness. And, as a particular point of pride, any module whatsoever can edit highlights without any risk of them blowing irretrievably. In practice, this actually simplifies development noticeably. The former is a bit of a double-edged sword, though, as chroma of bright colors can only be pushed so far without going out of gamut; Darktable therefore provides a second saturation control that additionally lowers brightness to keep bright colors in gamut, much like other tools.

You may also have noticed that the three examples pictures above were taken with Fujifilm cameras. These cameras are highly acclaimed for their film simulations. Of the tested software, Lightroom Classic, Silkypix Developer, Capture One, and ON1 Photo RAW natively support these film simulations. DxO PhotoLab can add them for an additional ₮139 FilmPack. And ACDSee Photo Studio, RawTherapee, Darktable, and Exposure X7 at least support third-party LUTs which can retrofit film simulations. Funny how programs either charge money for native film simulations, or support generic LUTs. What a coincidence! (Only ON1 Photo RAW supports both film simulations and (ICC) LUTs, and only Zoner Photo Studio supports neither).

So, after spending a few evenings with these programs, what is my verdict?

ACDSee Photo Studio Ultimate 2023 ★★★☆☆

Overall a rather good package. Fantastic organizational features, too. Even sports an API for extending its functionality! And pixel editing, a mobile app, and just a ton more.

However, it does not suit my tastes. Something about the UI and some tools seems a bit unpolished. Particularly, clarity and dehaze produce too strong halos for me, and the denoising is unpleasantly ineffective and smeary. Panning sometimes breaks the image into a pixelated mess, and there's no Fuji film sim support. Still, when it works, it produced some of my favorite renders in this comparison!

As another weird point, it's Windows only, and behaves oddly windowsy, too. For example, the library view by default shows all files and folders, not just images, and you actually need to reinstall the entire software if you want to change its language.

Capture One 22 ★★★★★

A product I know well, and own a license for. This comparison has reinforced that choice. Capture One's image editing tools are somewhat basic, but they seem refined and flexible. There's a strong focus on workflow efficiency, too, with its speed edit shortcuts and style brushes.

If there is a criticism to be leveled at Capture One, it's the high price and the somewhat slow pace of development. Many a competitors' feature is only included in Capture One years after they have become widespread. And many new features focus on the needs of working professional photographer instead of amateurs like me.

Regardless, Capture One will remain my one-of-two raw developer of choice. And it runs on my rather slow Surface tablet for emergency vacation edits!

Darktable 4.0 ★★★★☆

Darktable is the other raw developer I know intimately. In a sense, it is the polar opposite of Capture One: all the algorithms, parameters, and details are laid bare; nothing is hidden or automated. Its editing tools are also by far the most unusual of this bunch, which must no doubt be bewildering to newcomers. But if you're interested in deep control and alternative editing workflows, there's just nothing like it. Personally, I have scripted it and molded it extensively, which has made my Darktable similarly efficient and fast as Capture One. Such flexibility is actually rather rare in image editing software.

But I seriously hope they fix that horrendous highlights rendering of filmic v6 in the next revision. That's currently a constant pain point to work around.

Darktable will remain my one-of-two raw developer of choice. And it runs on Linux!

DxO PhotoLab 6 ★★★★☆

This program really is what prompted this entire ordeal. I heard so many good things about DxO PhotoLab, and was planning on replacing Capture One with it after this comparison.

There truly is a lot to like about DxO PhotoLab. Its tools seem robust, its default rendering is often very close to a finished image, and its denoising is rather remarkable. However, the program just felt clunky to use. Things are organized inefficiently, some operations take annoyingly long to process, and some effects are only visible in a tiny preview window or indeed the exported file. The local adjustments also seemed unnecessarily cumbersome to use, with that weird radial menu and those awkwardly imprecise "Control Points".

And what's with the weirdly branded sliders all over the program? Why is it "DxO Smart Lighting" instead of shadows, "DxO ClearView Plus" instead of dehaze, and "Microcontrast" instead of clarity, and "DxO DeepPrime XD" instead of denoising?

Truthfully, I might still have bought a license for this program as it is powerful and fun to use, despite my complaints. But ₮220 for the main program plus ₮140 for the FilmPack (for Fujifilm film simulations, but also basics such as a vignette tool) plus ₮100 for ViewPoint (for cool distortion stuff, but also the keystone tool) is just a bit too much, thank you very much.

Lightroom Classic 11.5 ★★★★☆

People like this program, and for good reason. Robust tools, a pleasant rendering, and just a staggering amount of tutorials and help online due to its popularity.

Nevertheless, Lightroom does not appeal to me. I don't like how Lightroom seems to takes undue possession of my images on import, I am repelled by the weird old-fashioned UI with its bonkers conventions (hold Alt to show masks, crop moves image instead of rectangle, etc.). I don't like its yearly-subscription-only pricing structure, either, although the price and terms are actually rather reasonable. And I don't like that Adobe Creative Cloud mothership that's necessary to install and maintain Lightroom.

But I do realize that this is actually good software. It's just not my favorite.

ON1 Photo RAW 2023 ★★☆☆☆

The new AI denoising and sharpening produced only artifacts, the new AI masking completely missed my subjects, there were algorithm artifacts everywhere, such as halos, hue shifts, and clipping. Perhaps something about my version was broken, being a very recent release. Additionally, one time I wanted to do a 1:2 crop, which you have to create a new crop preset for. However, the preset will not be saved with a ":" in the name. It took me a few tries to figure out that that's what prevented me from cropping. It doesn't help that the UI is surprisingly slow, often taking a second or so to redraw a tool change. And the installer is 2.7 GB, three times the size of any other tool in this list!

On the other hand, there are some cool tools in the effects panel, and some renders actually didn't look half bad. Perhaps it just needs a few bugfixes or more polish. But as it stands, I can not recommend this software.

RawTherapee 5.8 ★★★☆☆

I suppose RawTherapee should be regarded like Darktable, a killer program that requires deep study to wield well. I did not wield it well, but that probably says more about me than RawTherapee.

Still, I don't like the somewhat busy program layout, and how some operations take a long time to process. The export workflow is also strangely unusual, but I'm sure that that's something I could get used to. And I hear the next version will come with local adjustments.

Perhaps it is better-suited for a detail-oriented user than me. There's a lot to like here, but it's not what I'm looking for.

Silkypix Developer Studio Pro 11 ★☆☆☆☆

Something about this program is endearing to me. But in this comparison, it was just more cumbersome than useful. Many of its tools simply were not up to the task (can't raise shadows far enough, denoising produces only artifacts). And at some point, it slowed down to the point where it would take several seconds to see the effects of a single slider movement. This program did not work for me.

Exposure X7 ★★☆☆☆

Another strangely unpolished program on this list. Somehow, fonts everywhere are huge for no apparent reason, and sliders uncomfortably short. And I struggled preventing it from blowing highlights and clipping shadows. I don't see the appeal of this program.

Zoner Photo Studio X Fall 2022 ★★★★☆

As of the most recent version, Zoner Photo finally added native support for some Fujifilm cameras. Not all of my cameras are included yet, but to their credit, Zoner Photo can still open the missing files through ACR, albeit a bit more slowly.

Really, there is a lot to like about this program. Most tools work as advertised, with few issues (no local white balance, somewhat ineffective denoising), and a strong automatic mode. I also enjoy how it is unapologetically Windows-only, and uniquely feels native to Windows in a pleasant way.

It's a somewhat basic program compared to some of these others, but it's appropriately affordable, and fast. Not exactly what I'm looking for, but highly recommended for what it is!