bastibe.de

Two Years with Legacy Code

Sun, 15 Oct 2023 15:01:00 +0200

From January 2021 to the beginning of 2023, I worked on a legacy code base at Fraunhofer IDMT in Oldenburg. My task was the maintenance and development of a DNN-based speech recognition engine that had become terra incognita when its original developer had left the company a year before I started. The code had all the hallmarks of severe technical debt, with layers of half-used abstractions, many unused branches of unknown utility, and the handwriting of several concurrent programmers at odds with each other.

The code had evidently been written in a mad dash to bring the product to market. And not to discredit its developers, had been in production for several years, with a core of robust algorithms surrounded by helper scripts that had allowed the company to build upon, even after the original developers had left.

It was my job to clean it up. Having spent six years on my PhD recently, I welcomed the calmer waters of 'just' programming for a bit. This blog post is a summary of the sorts of challenges I faced during this time, and what kinds of techniques helped me overcome them.

The lay of the land

I approached the task from the outside, sorting through the build scripts first. Evidently, at least three authors were involved: One old-school Unix geek that wrote an outdated dialect of CMake, one high-level Python scripter, and one shell scripter that deeply believed in abstraction-by-information-hiding. The result of this was… interesting.

For a good few weeks I "disassembled" these scripts by tracing their execution manually through their many layers, and writing down the necessary steps that were actually executed. My favorite piece of code was a Makefile that called a shell script that ran a Python program, which instantiated a few classes and data structures, which ultimately executed "configure; make; make install" on another underying Makefile. I derived great satisfaction from cutting out all of these middle-men, and consolidating several directories of scripts into a single Makefile.

Similar simplifactions were implemented at the same time across several code bases by my colleagues. In due time, this concerted effort enabled us to implement continuous integration, automated benchmarking, and automated builds, but more on that later.

Data refactoring

The speech recognition software implemented a sort of interpreter for the DNN layers, originally encoded as a custom binary blob. Apparently, a custom binary approach had been taken to avoid dependencies on external parsing libraries. Yet the data had become so convoluted that both its compilation and its parsing were now considered unchangeable black boxes that impeded further development.

Again, I traced through the execution of the compiling code, noted down the pieces of data it recorded, and rewrote the compiler to produce a MsgPack file. On the parsing side, I wrote a custom MsgPack parser in C. Looking back, every job I've had involved writing at least a couple of data dumpers/parsers, yet many developers seem intimidated by such tasks. But why write such a thing yourself instead of using an off-the-shelf solution? In an unrelated code review later in the year one colleague used the cJSON library for parsing JSON; in the event, cJSON was several magnitudes bigger and more complex than the code base it was serving, which is clearly absurd. Our job as developers is to manage complexity, including that of our dependencies. In cases such as these, I often find a simple, fit-for-purpose solution preferable to more generalized external libraries.

A part of the DNN data came from the output of a training program. This output however was eternally unstable, often breaking unpredictably between version, and requiring complex workarounds to accommodate different versions of the program. The previous solution to this was a deeply nested decision tree for the various permutations the data could take. I simplified this code tremendously by calling directly into the other program's libraries, instead of trying to make sense of its output. This is another technique I had to rely on several times, hooking into C/C++ libraries from various Python scripts to bridge between data in a polyglot environment.

Doing these deep dives into data structures often revealed unintended entanglements. In order to assemble one data structure, you had to grab pieces of multiple different source data. Interestingly, once data structures were cleaned up to no longer have such entanglements, algorithms seemed to fall into place effortlessly. However, this was not a one-step process, but instead an ongoing struggle to keep data structures minimal and orthogonal. While algorithms and functions often feel easier to refactor than data structures, I have learned from this that it is often the changes to data structures that have the greatest effect, and should therefore receive the greatest scrutiny.

Code refactoring

My predecessor had left me a few screen casts by way of documentation. While the core program was reasonably well-structured, it was embedded in an architectural curiosity that told the tale of a frustrated high-level programmer forced to do low-level gruntwork. There were poor-man's-classes implemented as C structs with function pointers, there were do-while-with-goto-loops for exception handling, there were sort-of-dynamically-typed data containers, accompanied by angry comments decrying the stupidity of C.

Now I like my high-level-programming as much as the next guy, but forcing C to be something it isn't, is not my idea of fun. So over a few months I slowly removed most of these abstractions. Somewhat to my surprise, most of them turned out pure overhead that could simply be removed. Where a replacement was needed, I reverted to native C constructs. Tagged unions instead of casting, variable-length-arrays instead of dynamic arrays. Treating structs as values instead of references. This, alone, reduced the entire code base by a good 10%. The harder part was sorting out the jumble of headers and dependencies that had evidentally built up over time. Together with the removal of dead code paths, the overall code base shrank by almost half. There are few things more satisfying than excising and deleting unnecessary code.

I stumbled upon one particularly interesting problem when trying to integrate another code base into ours. Within our own software, build times were small enough to make logging and printf-debugging easier than an interactive debugger such as GDB. The other code base however was too complex to recompile on a whim, and a different solution had to be found. Now I am a weird person who likes to touch the raw command line instead of an IDE. And in this case this turned out to be a huge blessing, as I found that GDB can not only be used interactively, but can also be scripted! So instead of putting logging into the other library, I wrote GDB scripts that augmented break points with a little call printf(...) or print/d X. These could get suprisingly complicated, where one breakpoint might enable or disable other breakpoints conditionally, and break point conditions could call functions on their own. It took some learning, but these debugging scripts were incredibly powerful, and a technique I will definitely refer to in the future.

When adding new features to the software, I often found it impossible to work the required data flow into the existing program code without snowballing complexity. I usually took these situations as code smells that called for a refactoring. Invariably, each cleaning up of program flow or data structures inched the program closer and closer to allow my feature addition. After a while, this became an established modus operandi: independently clean the code until feature additions become easy and obvious, then do the obvious thing. Thus every task I finished also left the surrounding code in a better state. In the end, about 80% of the code base had gotten this treatment, and I strongly believe that this has left the project in a much better state than it was before. To say nothing of the added documentation and tests, of course.

More velocity makes bigger craters

As I slowly shifted from cleanup work to new features, change management became a pressing issue. New features had to be evaluated, existing features had to be tested, and changes had to be documented and downstreamed. Fascinatingly, the continuous integration and evaluation tools we built for this purpose, soon unearthed a number of hidden problems in other parts of the product that we had not been aware of (including that the main task I had been hired to do was less worthwhile than thaught, LOL). That taught us all a valuable lesson about testing, and proving our assertions. That said, I never found bottom-level unit tests all that useful for our purposes; the truly useful tests invariably were higher-level integration tests.

Eventually, my feature additions led to downstream changes by several other developers. While I took great care to present a stable API, and documenting all changes and behavior appropriately, at the end of the day my changes still amounted to a sizeable chunk of work for others. This was a particularly stark contrast to the previous years of perfect stagnation while nobody had maintained the library. My main objective at this point was to avoid the mess I had started out with, where changes had evidentally piled on changes until the whole lot had become unmaintainable.

Thus a balance had to be struck between moving fast (and breaking things), and projecting stability and dependability. One crucial tool for this job turned out to be code reviews. By involving team members directly with the code in question, they could be made more aware of its constraints and edge cases. It took a few months to truly establish the practice, but by the end of a year everyone had clearly found great value in code reviews as a tool for communication.

Conclusions

There is a lot more to be said about my time at Fraunhofer. The deep dive into the world of DNN engines was truly fascinating, as were the varied challenges of implementing these things on diverse platforms such as high-performance CPU servers, Laptops, Raspberry Pis, and embedded DSPs. I learned to value automation of developer tasks, and of interface stability and documentation for developer productivity.

But most of all, I learned to appreciate legacy code. It would have been easy to call it a "mess", and advocate to rewrite it from scratch. But I found it much more interesting to try to understand the code's heritage, and tease out the algorithmic core from the abstractions and architectural supports. There were many gems to be found this way, and a lot to be learned from the programmers before you. I often felt a strange connection to my predecessor, as if we were talking to each other through this code base. And no doubt my successor feels the same way about my code now.

OS Customization and MacOS

Fri, 21 Apr 2023 10:19:00 +0200

I was an Apple fanboy some years ago. Back then, whenever something was odd on my computer, I was surely just using it wrong. Nowadays, I see things the other way around: We're not "holding it wrong", the computer is just defective. Computers should do our bidding, not vice versa. So here's a bunch of things that I do to my computers to mold them to my way of working.

Keyboard Layout

I switch constantly between a German and English keyboard layout, and regularly between various machines. My physical keyboards are German, and my fingers are used to the Windows-default German and (international) US keyboard layout. These are available by default on Windows and Linux, but MacOS goes its own way.

However, keyboard layouts on MacOS are saved in relatively simple text files, and can be modified with relative ease. The process goes like this: Download Ukelele (free) to create a new keyboard layout bundle for your base layout¹. Inside that bundle, there's a *.keylayout file, which is an XML file that defines the characters that each key-modifier combination produces. I changed that to create a Windows-like US keyboard layout. And I replaced the keyboard icon with something sane (not "A") by creating a 256x256 pixel PNG, opening it in Preview, holding alt while saving to select the ICNS format. Save the keyboard bundle to ~/Library/Keyboard Layouts and reboot. Then I remove the unremovable default ("A") German layout by selecting another one, then plutil -convert xml1 ~/Library/Preferences/com.apple.HIToolbox.plist, and delete the entry from AppleEnabledInputSources. Now reboot. Almost easy. Almost.

One the one hand, this was quite the ordeal. On the other, I have tried to do this sort of thing on Windows and Linux before, and for the life of me could not do it. So I actually think this is great!

Keyboard Shortcuts

My main text editor is Emacs, and I am very used to its keyboard shortcuts. Of particular note are CTRL-A/E for going to the beginning/end of a line, and Alt-B/F for navigating forward/backwards by word. I have long wanted to use these shortcuts not just in Emacs and readline-enabled terminal applications, but everywhere else, too. And with MacOS, this is finally possible: Install BetterTouchTool ($22), and create keyboard shortcuts that maps, e.g. Alt-B/F to Alt-←/→. Ideally, put this in a new activation group that excludes Emacs. It may be necessary to remove the keyboard character for Alt-B/F from your keyboard layout before this works. I've spent an embarrassing number of hours trying to get this to work on Windows and Linux, and really got nowhere². Actually, however, most readline shortcuts such as Ctrl-A/E/B/F/K/Y already work out of the box on MacOS!

Mouse Configuration

I generally use a trackpad, and occasionally a traditional mouse for image editing, and have used a trackball. I find that any one specific device will lead to wrist pain if used constantly, so I switch it up every now and then. The trackpad and trackball, however, need configuration to be usable.

After experimenting with many a trackpad device, I have found Apple touch pads the best trackpads on the market³. On MacOS, they lacks a middle mouse click. So I created a trackpad shortcut in the aforementioned BetterTouchTool ($22) to map the middle click on a three-finger tap (can also be had for free with MiddleClick (OSS)). For Windows, Magic Utilities ($17/y) provides a wonderful third-party driver for Apple devices that also supports the three-finger tap. I have not gotten the Apple touch pad to pair reliably on Linux, and have generally found their touch pad driver libinput a bit lacking.

My trackball is a Kensington SlimBlade. To scroll, you rotate the ball around its vertical axis. This is tedious for longer scroll distances, however. But there's an alternative scrolling method called "button scrolling", where you hold one button on the trackball, and move the ball to scroll. You need to install the Kensington driver to enable this on Windows and MacOS. Button scrolling is available on Linux as well using xinput⁴, but I haven't gotten the setting to stick across reboots or sleep cycles. So I wrote a background task that checks xinput every five seconds, which does the trick.

Window Management

Frankly, Windows does window management correctly. Win-←/→ moves windows to the left and right edge of the screen, as does dragging the window to the screen border. Further Win-←/→ then moves the window to the next half-screen in that direction, even across display boundaries. KDE does this correctly out of the box as well, Gnome does not do the latter, and it drives me mad. MacOS doesn't do any of these things. But Rectangle (OSS) does. Easy fix. (BetterTouchTool can do it, too, but Rectangle is prettier)

Furthermore, I want Alt-Tab to switch between windows. Again, MacOS is the odd one out, which uses CMD-Tab to switch between apps, now windows. And then there's another shortcut for switching between windows of the same app, but the shortcut really doesn't work at all on a German keyboard. Who came up with this nonsense? Anyway, Witch ($14) implements window switching properly.

Application Management

In Windows and Linux, I hit the Windows key and start typing to select and start an app. In MacOS, this is usually Cmd-Space, but BetterTouchTool can map it to a single short Cmd, if you prefer.

More annoying are the various docks and task bars. I always shove the dock off to the right edge of the screen, where it stays out of the way. Windows 10 had a sane dock, but then 11 came and forced it to the bottom of the screen. Dear OS makers, every modern screen has plenty of horizontal space. But vertical space is somewhat limited. So why on earth would you make a rarely used menu such as the dock consume that precious vertical space by default? And Microsoft, specifically, why not make it movable? Thankfully, there's StartAllBack ($5), which replaces the Windows task bar with something sensible, and additionally cleans up the start menu if you so desire. On KDE, I fractionally prefer Latte (OSS) over KDE's native dock. The MacOS dock is uniquely dumb, offering no start menu, and allowing no window selections. But it's unobtrusive and can be moved to the right edge, so it's not much of a bother.

File Management

One of the most crucial tasks in computer work in general is file management. I am not satisfied with most file managers. Dolphin on KDE works pretty well, it has tabs, can bulk-rename files, can display large directories without crashing, and updates in real time when new files are added to the current directory. Gnome Nautilus is so bad it is the main reason I switched to KDE on my Linux machines. Finder on MacOS is passable, I suppose, although the left sidebar is unnecessarily restrictive (why can't I add a shortcut to a network drive?). Windows Explorer is really rather terrible, lacking a bulk-rename tool, and crucially, tabs. In Windows 10, these can be added with Groupy ($12) (set it to only apply to explorer.exe). Windows 11 has very recently added native tabs, which work OK, but can't be detached from the window.

The sad thing is that there are plenty of very good file manager replacements out there, but none of the OSs have a mechanism for replacing their native file manager in a consistent way, so we're mostly stuck with the defaults.

Oh, and I always remove the iCloud/OneDrive sidebar entries, which is surprisingly tedious on Windows.

Hardware Control

On laptops, you can control screen brightness from your keyboard. On desktops, you can not. However, some clever hackers have put together BetterDisplay (OSS for screen brightness), which adds this capability to MacOS. That's actually a capability I have wanted for quite a while, and apparently it is only available in MacOS. Great stuff!

Less great is that MacOS does not allow volume control on external sound cards. SoundSource ($47) adds this rather crucial functionality back, once you go through the unnecessarily excruciating process of enabling custom kernel extensions. Windows and Linux of course natively support this.

Another necessary functionality for me is access to a non-sucky (i.e. no FAT) cross-platform file system. At the moment, the most portable file system seems to be NTFS, of all things. Regrettably, MacOS only supports reading NTFS, but no writing. Paragon NTFS (€20) adds this with another kernel extension, and promptly kernel-panicked my computer. Oh joy. At least it's only panicking for file transfers initiated by DigiKam, which I can work around. Paragon Support says they're working on it. I'm not holding my breath. Windows and Linux of course natively support NTFS.

System Management

I have learned from experience not to trust graphical backup programs. TimeMachine in particular has eaten my backups many times already, and can not be trusted. But I have used Borg (OSS) for years, and it has so far performed flawlessly. Even more impressive, my Borg backups have a continuous history despite moving operating systems several times. It truly is wonderful software!

On Windows, I run Borg inside the WSL, and schedule its backups with the Windows Task Scheduler. On Linux, I schedule them with systemd units. On MacOS, I install Borg with Homebrew (OSS) and schedule the backups with launchd tasks. It's all pretty equivalent. One nice thing about launchd, however, is how the OS immediately pops up a notification if there's a new task file added, and adds the task to the graphical system settings.

I have to emphasize what a game-changer the WSL is on Windows. Where previously, such simple automations where a pain in the neck to do reliably, they're now the same simple shell scripts as on other OSes. And it perfectly integrates with Windows programs as well, including passing pipes between Linux and Windows programs. It's truly amazing! At the moment, I'd rate Windows a better Unix system than MacOS for this reason. Homebrew is a passable package manager on MacOS, but the way it's ill-integrated into the main system (not in system PATH) is a bit off-putting.

App Compatibility

I generally use my computer for three tasks: General document stuff, photo editing, and video games.

One major downside of Apple computers is that video games aren't available. This has become less of a problem to me since I bought a Steam Deck, which has taken over gaming duties from my main PC. Absolutely astonishingly, the Steam Deck runs Windows games on Linux through emulation, which works almost flawlessly, making video games no longer a Windows-only proposition.

What doesn't work well on Linux are commercial applications. Wine generally does not play well with them, and frustratingly for my photo editing, neither VMWare Workstation Player (free) nor VirtualBox (OSS) support hardware-accelerated VMs on up-to-date Linux⁵. So where MacOS lacks games, Linux lacks Photoshop. Desktop applications in general tend to be unnecessarily cumbersome to manage and update on Linux. Flatpak is helping in this regard, by installing user-facing applications outside of the OS package managers, but it remains more work than on Windows or MacOS. The occasional scanner driver or camera interface app can also be troublesome on Linux, but that's easily handled with a VirtualBox VM (with the proprietary Extension Pack for USB2 support), and hasn't really bothered me too much.

Luckily for me, my most-used apps are generally OSS tools such as Darktable (OSS) and DigiKam (OSS), or cross-platform programs like Fish (OSS), Git (OSS), and Emacs (OSS). This is however, where Windows has a bit of a sore spot, as these programs tend to perform noticeably worse on Windows than on other platforms. Emacs and git in particular are just terribly slow on Windows, taking several seconds for routine operations that are instant on other platforms. That's probably due to Windows' rather slow file system and malware scanner for the many-small-files style of file management that these Unix tools implement. It is very annoying.

Conclusions

So there's just no perfect solution. MacOS can't do games, Linux can't run commercial applications, and Windows is annoyingly slow for OSS applications. Regardless, I regularly use all three systems productively. My job is mostly done on Windows, my home computer runs MacOS, and my Steam Deck and automations run Linux.

Overall, I currently prefer MacOS as my desktop OS. It is surprisingly flexible, and more scriptable than I thought, and in some ways is actually more functional than Linux or Windows. The integrated calendar and contacts apps are nice, too, and not nearly as terrible as their Windows/Linux counterparts. To say nothing of the amazing M1 hardware with its minuscule power draw and total silence, while maintaining astonishing performance.

Linux is where I prefer to program, due to its sane command line and tremendously good compiler/debugger/library infrastructure. As a desktop OS, it does have some rough edges, however, especially for its lack of access to commercial applications. While Linux should be the most customizable of these three, I find things tend to break too easily, and customizations are often scattered widely between many different subsystems, making them very hard to get right.

Theoretically, Windows is the most capable OS, supporting apps, OSS, and games. But it also feels the most user-hostile of these three, and the least performant. And then there's the intrusive ads everywhere, its spying on my every move, and at work there's inevitably a heavy-handed administrator security setup that gets in the way of productivity. It's honestly fine on my home computer, at least since they introduced the WSL. But using it for work every day is quite enough, so I don't want to use it at home, too.

Footnotes:

if there's an easier way to create a keyboard layout bundle, please let me know. I didn't use ukelele for anything but the bundle creation.

It occurs to me that it might actually be possible to do something like on Windows this with AutoHotkey (free). I'll have to try that!

The Logitech T650 is actually not bad, but the drivers are a travesty. Some Wacom tablets include touch, too, but they palm rejection is abysmal and they don't support gestures.

⁴

xinput set-prop "Kensington Slimblade Trackball" "libinput Scroll Method Enabled" 0, 0, 1 # button scrolling
xinput set-prop "Kensington Slimblade Trackball" "libinput Button Scrolling Button" 8 # top-right button

⁵

VirtualBox does not have a good accelerated driver, and VMWare does not support recent kernels. Qemu should be able to solve this problem, but I couldn't get it to work reliably.

Media of 2022

Sun, 12 Feb 2023 15:02:00 +0100

Around this time of the year, I usually write a blog post about my favorite books of the last year. But this time, not enough of them really stood out. So instead, here's my favorite pieces of media I consumed in the last year:

Firepower

This is the story of how gunpowder changed the world. We get to see the familiar history of central Europe, but told unusually, from the bottom up: how seemingly small inventions change the course of peoples and nations. The book is in essence a history of warfare in the last few hundred years, but this time the movers and shakers are not Great Men, but chemistry and engineering. A fascinating perspective, to say the least.

Along the way, the book unraveled countless tangents and quirks I had previously stumbled upon, yet never understood. How castles went from tall walled structures to flat earthworks in the 18th century due to the disruptive invention of cannons. How wooden galleons of 1850 were obsoleted by turreted iron warships practically overnight. How rifling and shells bled dry the coffers of Central Europe and made conflict inevitable. This book contextualized many a story like this in the most riveting manner!

Steam Deck

Ever since we had our second child, I didn't really find the time for video games any more, much to my regret. So when the Steam Deck was announced, essentially a portable gaming computer, it didn't feel like a worthwhile investment.

But boy, was I wrong about that. The genius of the Steam Deck is how it's instantly-on, instantly-off like a video games console, allowing me to play in short bursts that would not otherwise be available to gaming. But in contrast to a console, the deck allows me to play games without blocking the living room, and away from the computer screen I'm working at all day anyway.

It has reignited my video gaming, and surprisingly not just for newer titles, but thanks to EmuDeck, emulated retro games as well. I had a blast playing Banjo-Kazooie with my daughter, and Chorus, Ace Combat, Guardians of the Galaxy and Death Stranding on my own. To me, this is a revolutionary device, and I haven't touched my gaming PC or any of my video game consoles since!

RAW Developer Comparison 2

Tue, 11 Oct 2022 19:15:00 +0200

It's that time of the year again when all image editing programs come out with new versions. This comes at an inopportune time, as I feel restless of late. So, naturally, I had to try them all. Or maybe I just wanted a justification for buying DxO PhotoLab, because people on the internet speak so well of it 🙄.

For the following comparison I downloaded trial versions of ACDSee Photo Studio Ultimate 2023 (€155), Capture One 22 (€350 or €220/a), Darktable 4.0 (free), DxO Photolab 6 (€220 + €140 for FilmPack 6 + €99 for ViewPoint), Adobe Lightroom Classic 11.5 (€142/a), ON1 Photo Raw 2023 (€126), RawTherapee 5.8 (free), Silkypix Developer Studio Pro 11 (€155), Exposure X7 (€165), and Zoner Photo Studio X Fall 2022 (€60/a). I also installed Luminar Neo (€120 or €80/a) and Radiant Photo (€140) but I disliked them so immediately and viscerally that I didn't include them in the comparison below.

In order to put them through their paces, I took a random smattering of images from the last few years that I found difficult to work with for one reason or another, and checked how each of the programs dealt with them. Of course I am no expert in any of them except Darktable and Capture One, so my results are probably flawed. I tried to inject some objectivity by limiting my edits to the most obvious sliders wherever possible, especially in the programs I know better. Regardless, my comparison is probably less scientific than last time, because my brain sort of broke after staring at too many renderings of the same images for too long. Don't try this at home, folks!

Test Case One: Fire

Fire is notoriously difficult to deal with, because it covers a large dynamic range, and we have strong associations of certain colors with heat. Thus a more yellow rendering usually suits fire very well, whereas a more neutral rendering quickly looks unnatural. In particular, we seem to expect highlights to twist to yellow, and midtones to remain orange.

My editing goals for this picture are simple: raise shadows or blacks until the pan becomes faintly visible, and reign in highlights if necessary to prevent excessive clipping.

Fujifilm X-Pro2 with XF35mmF1.4 R 1/1000s f/2.2 ISO400
📂 DSCF9359.RAF (23.0 MB)

There's a difficult tradeoff to be made here: Fire highlights physically don't change hue much compared to the fire body, but we visibly expect "hotter" fire to twist yellow. So a balance has to be struck between merely desaturating highlights, and twisting them yellow. You can see Darktable v6, DxO PhotoLab, RawTherapee, Exposure, and Zoner leaning towards desaturation, while the others render yellow color twists of some form or another. ACDSee and ON1 probably take the yellow a bit too far.

However, there's a flip side to this: Every program that renders yellow highlights in fire, does the same for overexposed skin, twists overexposed skies cyan, and red flowers magenta. It's a difficult tradeoff to get right. This is especially tricky since color shifts in highlights are often baked deeply into the programs' color science, and hard to get rid off where they're unwanted.

Anyway, to my eyes, the only fire-like renderings here come from Capture One, Darktable v5, PhotoLab, and Lightroom.

A Hazy Mountain

This lovely scene is a deep crop into an image taken at long distance on a hazy day. The challenge is therefore dehazing the mountains, chiseling out the contours with clarity and texture and contrast, and removing any extraneous noise.

This is a difficult task, as emphasizing local contrast tends to produce halos near the horizon, dehazing tends to introduce unpleasant color shifts, and the deep crop necessitates a subtle balance between denoising and detail retention.

Fujifilm X-T2 with XF70-300 @ 231mm 1/400s f/5.7 ISO200
📂 DSF4442.RAF (23.0 MB)

Overall, most of these renderings turned out fine. But there are pretty substantial differences in denoising, haloing, and detail retention of the shadows specifically. Halos especially are a personal bug-bear of mine, where the sky above the horizon line brightens and (worse) the mountains below the horizon line darken. This is especially visible in ACDSee, ON1, and Zoner, and to a lesser extent in DxO, Lightroom, and RawTherapee.

There were also significant differences in noise removal. Especially ACDSee, Exposure, and Zoner had weirdly ineffective denoising, and were unusually difficult to balance against smearing and worms in Lightroom, ON1, and Silkypix. Colors were a bit difficult to control in ACDSee, ON1, and RawTherapee.

The most pleasant renderings to my eye are DxO, Exposure, and Darktable in this round, although many of the others are very usable as well.

High Key Rendering

Contrary to the popular style at moment, I sometimes like my highlights a little blown. I especially like a smooth, desaturated, film-like drop-off in my highlights, which seems strangely difficult to replicate in digital photography.

So instead of turning the following picture into an HDR hellscape, I want to compress the bright background into the highlights, while expanding the dark foreground to fill the midtones. The capture actually has all highlight information intact, so I don't want to blow anything out completely, just gracefully fade it into pastels.

Fujifilm X-Pro2 with XF16-80mm @ 16mm 1/400s f/9 ISO400
(no raw, for privacy reasons)

The different renderings of this image are more of a matter of preference than any of the previous ones. To my eyes, I like the versions drawn by ACDSee, Capture One, Zoner Photo, and DxO PhotoLab best, and I recon that many of the other versions could have been improved with a little more manual tuning. The only problematic versions of this image were the strange HDR-like ON1 render, and the Silkypix variant with its lost black point.

Summary

Overall, all of these programs seem reasonable tools for image editing. Most of their differences can probably be overcome if you learn them better. That said, this comparison has left me with a few clear favorites. To be clear, the above comparison only showed the most decisive images of a much larger set which informs my opinion. It should also be noted that my tastes in image editing do not focus on detail recovery, detailed masking, nontrivial retouching, or scene-changing edits such as wholesale sky swapping.

One area I am particularly interested in, however, is the inherent complexity of the editing tools: For example, I like my saturation slider to only change saturation and leave lightness alone. Similarly, contrast adjustments should not affect saturation. Another interesting part is the behavior of highlights adjustments. Ideally, I'd like highlights to be able to counteract exposure adjustments so I can balance them against one another. Better yet if the same can be done with the tone curve.

Name	Editing is	Export takes	Saturation changes	Contrast changes	Highlights rescues	Tonecurve rescues
			lightness?	saturation?	overexposure?	overexposure?
ACDSee Photo Studio	realtime	25s	yes	yes	yes	no
Capture On	realtime	15s	a little	no	yes	no
Darktable	delayed	15s	no	no	yes	yes
DxO PhotoLab	delayed	45s	yes	a little	yes	a little
Lightroom Classic	realtime	15s	a little	a little	yes	a little
ON1 Photo RAW	realtime	30s	yes	a little	no	no
RawTherapee	wait and see	30s	a little	yes	no	no
Silkypix Developer	lo-res wait/see	80s	no	a little	no	no
Exposure X7	realtime	30s	yes	strongly	yes	a little
Zoner Photo Studio	lo-res realtime	30s	yes	yes	yes	a little

If saturation changes lightness and contrast changes saturation, editing can become rather more difficult, as the effect tends to be hard to counteract without complex luminosity masking. This is a reason for me to dislike my experience with ACDSee, ON1, Exposure, and Zoner particularly. The highlights slider issue also has a big influence on how you edit images. If highlights can be recovered after exposure adjustments, you can use exposure mostly for image brightness, and recover highlights later if needed. On the other hand, if highlights can't be recovered, then the exposure slider must instead be used to protect highlights, and image brightness has to be relegated to the tone curve or shadows/midtones sliders. This feels weirdly backwards to me, and is a reason for me to disregard ON1, RawTherapee, and Silkypix.

The gold standard here is of course Darktable, where saturation ("chroma") is completely decoupled from lightness. And, as a particular point of pride, any module whatsoever can edit highlights without any risk of them blowing irretrievably. In practice, this actually simplifies development noticeably. The former is a bit of a double-edged sword, though, as chroma of bright colors can only be pushed so far without going out of gamut; Darktable therefore provides a second saturation control that additionally lowers brightness to keep bright colors in gamut, much like other tools.

You may also have noticed that the three examples pictures above were taken with Fujifilm cameras. These cameras are highly acclaimed for their film simulations. Of the tested software, Lightroom Classic, Silkypix Developer, Capture One, and ON1 Photo RAW natively support these film simulations. DxO PhotoLab can add them for an additional €139 FilmPack. And ACDSee Photo Studio, RawTherapee, Darktable, and Exposure X7 at least support third-party LUTs which can retrofit film simulations. Funny how programs either charge money for native film simulations, or support generic LUTs. What a coincidence! (Only ON1 Photo RAW supports both film simulations and (ICC) LUTs, and only Zoner Photo Studio supports neither).

So, after spending a few evenings with these programs, what is my verdict?

ACDSee Photo Studio Ultimate 2023 ★★★☆☆

Overall a rather good package. Fantastic organizational features, too. Even sports an API for extending its functionality! And pixel editing, a mobile app, and just a ton more.

However, it does not suit my tastes. Something about the UI and some tools seems a bit unpolished. Particularly, clarity and dehaze produce too strong halos for me, and the denoising is unpleasantly ineffective and smeary. Panning sometimes breaks the image into a pixelated mess, and there's no Fuji film sim support. Still, when it works, it produced some of my favorite renders in this comparison!

As another weird point, it's Windows only, and behaves oddly windowsy, too. For example, the library view by default shows all files and folders, not just images, and you actually need to reinstall the entire software if you want to change its language.

Capture One 22 ★★★★★

A product I know well, and own a license for. This comparison has reinforced that choice. Capture One's image editing tools are somewhat basic, but they seem refined and flexible. There's a strong focus on workflow efficiency, too, with its speed edit shortcuts and style brushes.

If there is a criticism to be leveled at Capture One, it's the high price and the somewhat slow pace of development. Many a competitors' feature is only included in Capture One years after they have become widespread. And many new features focus on the needs of working professional photographer instead of amateurs like me.

Regardless, Capture One will remain my one-of-two raw developer of choice. And it runs on my rather slow Surface tablet for emergency vacation edits!

Darktable 4.0 ★★★★☆

Darktable is the other raw developer I know intimately. In a sense, it is the polar opposite of Capture One: all the algorithms, parameters, and details are laid bare; nothing is hidden or automated. Its editing tools are also by far the most unusual of this bunch, which must no doubt be bewildering to newcomers. But if you're interested in deep control and alternative editing workflows, there's just nothing like it. Personally, I have scripted it and molded it extensively, which has made my Darktable similarly efficient and fast as Capture One. Such flexibility is actually rather rare in image editing software.

But I seriously hope they fix that horrendous highlights rendering of filmic v6 in the next revision. That's currently a constant pain point to work around.

Darktable will remain my one-of-two raw developer of choice. And it runs on Linux!

DxO PhotoLab 6 ★★★★☆

This program really is what prompted this entire ordeal. I heard so many good things about DxO PhotoLab, and was planning on replacing Capture One with it after this comparison.

There truly is a lot to like about DxO PhotoLab. Its tools seem robust, its default rendering is often very close to a finished image, and its denoising is rather remarkable. However, the program just felt clunky to use. Things are organized inefficiently, some operations take annoyingly long to process, and some effects are only visible in a tiny preview window or indeed the exported file. The local adjustments also seemed unnecessarily cumbersome to use, with that weird radial menu and those awkwardly imprecise "Control Points".

And what's with the weirdly branded sliders all over the program? Why is it "DxO Smart Lighting" instead of shadows, "DxO ClearView Plus" instead of dehaze, and "Microcontrast" instead of clarity, and "DxO DeepPrime XD" instead of denoising?

Truthfully, I might still have bought a license for this program as it is powerful and fun to use, despite my complaints. But €220 for the main program plus €140 for the FilmPack (for Fujifilm film simulations, but also basics such as a vignette tool) plus €100 for ViewPoint (for cool distortion stuff, but also the keystone tool) is just a bit too much, thank you very much.

Lightroom Classic 11.5 ★★★★☆

People like this program, and for good reason. Robust tools, a pleasant rendering, and just a staggering amount of tutorials and help online due to its popularity.

Nevertheless, Lightroom does not appeal to me. I don't like how Lightroom seems to takes undue possession of my images on import, I am repelled by the weird old-fashioned UI with its bonkers conventions (hold Alt to show masks, crop moves image instead of rectangle, etc.). I don't like its yearly-subscription-only pricing structure, either, although the price and terms are actually rather reasonable. And I don't like that Adobe Creative Cloud mothership that's necessary to install and maintain Lightroom.

But I do realize that this is actually good software. It's just not my favorite.

ON1 Photo RAW 2023 ★★☆☆☆

The new AI denoising and sharpening produced only artifacts, the new AI masking completely missed my subjects, there were algorithm artifacts everywhere, such as halos, hue shifts, and clipping. Perhaps something about my version was broken, being a very recent release. Additionally, one time I wanted to do a 1:2 crop, which you have to create a new crop preset for. However, the preset will not be saved with a ":" in the name. It took me a few tries to figure out that that's what prevented me from cropping. It doesn't help that the UI is surprisingly slow, often taking a second or so to redraw a tool change. And the installer is 2.7 GB, three times the size of any other tool in this list!

On the other hand, there are some cool tools in the effects panel, and some renders actually didn't look half bad. Perhaps it just needs a few bugfixes or more polish. But as it stands, I can not recommend this software.

RawTherapee 5.8 ★★★☆☆

I suppose RawTherapee should be regarded like Darktable, a killer program that requires deep study to wield well. I did not wield it well, but that probably says more about me than RawTherapee.

Still, I don't like the somewhat busy program layout, and how some operations take a long time to process. The export workflow is also strangely unusual, but I'm sure that that's something I could get used to. And I hear the next version will come with local adjustments.

Perhaps it is better-suited for a detail-oriented user than me. There's a lot to like here, but it's not what I'm looking for.

Silkypix Developer Studio Pro 11 ★☆☆☆☆

Something about this program is endearing to me. But in this comparison, it was just more cumbersome than useful. Many of its tools simply were not up to the task (can't raise shadows far enough, denoising produces only artifacts). And at some point, it slowed down to the point where it would take several seconds to see the effects of a single slider movement. This program did not work for me.

Exposure X7 ★★☆☆☆

Another strangely unpolished program on this list. Somehow, fonts everywhere are huge for no apparent reason, and sliders uncomfortably short. And I struggled preventing it from blowing highlights and clipping shadows. I don't see the appeal of this program.

Zoner Photo Studio X Fall 2022 ★★★★☆

As of the most recent version, Zoner Photo finally added native support for some Fujifilm cameras. Not all of my cameras are included yet, but to their credit, Zoner Photo can still open the missing files through ACR, albeit a bit more slowly.

Really, there is a lot to like about this program. Most tools work as advertised, with few issues (no local white balance, somewhat ineffective denoising), and a strong automatic mode. I also enjoy how it is unapologetically Windows-only, and uniquely feels native to Windows in a pleasant way.

It's a somewhat basic program compared to some of these others, but it's appropriately affordable, and fast. Not exactly what I'm looking for, but highly recommended for what it is!

Darktable for Fujifilm Cameras

Wed, 04 May 2022 20:41:00 +0200

You know what I like to see when I import photos from my Fujifilm camera into Darktable?

Each RAF file has tags with the aspect ratio, DR mode, and film simulation, is exposed correctly, cropped correctly, and has the correct film simulation applied.

However, that is not the default. Darktable, like most raw developers, is camera-agnostic.

agnostic

ăg-nŏs′tĭk
noun
[...]

One who is doubtful or noncommittal about something.

Which means that Darktable does not know about any Fujifilm-specific raw file metadata, such as crop, dynamic range modes, or film simulations. Thus what you'd normally see in Darktable is more like this:

Default darktable, DR200/DR400 files are underexposed, no crops are applied, colors don't quite match, no tags.

Notice how all the DR200/DR400 images are underexposed by one and two stops, how the first JPG is a square crop, but the RAF is 3:2, how the color of the grass and the train are subtly different in RAF and JPG.

But thankfully, Darktable has a scripting interface for automating things. And what I've done here is a little script that uses exiftool to read the missing metadata from the RAF file and apply appropriate styles to get Darktable's default rendering close to the JPG.

Here's the lua script in its entirety:

fujifilm ̲auto ̲settings.lua

--[[ fujifilm_auto_settings-0.2

Apply Fujifilm film simulations, in-camera crop mode, and dynamic range.

Copyright (C) 2022 Bastian Bechtold 

This program is free software; you can redistribute it and/or modify
it under the terms of the GNU General Public License as published by
the Free Software Foundation; either version 2 of the License, or
(at your option) any later version.

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

You should have received a copy of the GNU General Public License along
with this program; if not, write to the Free Software Foundation, Inc.,
51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
--]]

--[[About this Plugin
Automatically applies styles that load Fujifilm film simulation LUTs,
copy crop ratios from the JPG, and correct exposure according to the
chosen dynamic range setting in camera.

Dependencies:
- exiftool (https://www.sno.phy.queensu.ca/~phil/exiftool/)
- Fuji LUTs (https://blog.sowerby.me/fuji-film-simulation-profiles/)

Based on fujifim_dynamic_range by Dan Torop.

  Film Simulations
  ----------------

Fujifilm cameras are famous for their film simulations, such as Provia
or Velvia or Classic Chrome. Indeed it is my experience that they rely
on these film simulations for accurate colors.

Darktable however does not know about or implement these film
simulations. But they are available to download from Stuart Sowerby as
3DL LUTs. (PNG LUTs are also available, but they show a strange
posterization artifact when loaded in Darktable, which the 3DLs do
not).

In order to use this plugin, you must prepare a number of styles:
- provia
- astia
- velvia
- classic_chrome
- pro_neg_standard
- pro_neg_high
- eterna
- acros_green
- acros_red
- acros_yellow
- acros
- mono_green
- mono_red
- mono_yellow
- mono
- sepia

These styles should apply the according film simulation in a method of
your choosing.

This plugin checks the image's "Film Mode" exif parameter, and applies
the appropriate style. If no matching style exists, no action is taken
and no harm is done.

  Crop Factor
  -----------

Fujifilm cameras allow in-camera cropping to one of three aspect
ratios: 2:3 (default), 16:9, and 1:1.

This plugin checks the image's "Raw Image Aspect Ratio" exif
parameter, and applies the appropriate style.

To use, prepare another four styles:
- square_crop_portrait
- square_crop_landscape
- sixteen_by_nine_crop_portrait
- sixteen_by_nine_crop_landscape

These styles should apply a square crop and a 16:9 crop to
portrait/landscape images. If no matching style exists, no action is
taken and no harm is done.

  Dynamic Range
  -------------

Fujifilm cameras have a built-in dynamic range compensation, which
(optionally automatically) reduce exposure by one or two stops, and
compensate by raising the tone curve by one or two stops. These modes
are called DR200 and DR400, respectively.

The plugin reads the raw file's "Auto Dynamic Range" or "Development
Dynamic Range" parameter, and applies one of two styles:
- DR200
- DR400

These styles should raise exposure by one and two stops, respectively,
and expand highlight latitude to make room for additional highlights.
I like to implement them with the tone equalizer in eigf mode, raising
exposure by one/two stops over the lower half of the sliders, then
ramping to zero at 0 EV. If no matching styles exist, no action is
taken and no harm is done.

These tags have been checked on a Fujifilm X-T3 and X-Pro2. Other
cameras may behave in other ways.

--]]

local dt = require "darktable"
local du = require "lib/dtutils"
local df = require "lib/dtutils.file"

du.check_min_api_version("7.0.0", "fujifilm_auto_settings")

-- return data structure for script_manager

local script_data = {}

script_data.destroy = nil -- function to destory the script
script_data.destroy_method = nil -- set to hide for libs since we can't destroy them completely yet, otherwise leave as nil
script_data.restart = nil -- how to restart the (lib) script after it's been hidden - i.e. make it visible again

local function exiftool_get(exiftool_command, RAF_filename, flag)
    local command = exiftool_command .. " " .. flag .. " -t " .. RAF_filename
    dt.print_log(command)
    local output = io.popen(command)
    local exiftool_result = output:read("*all")
    output:close()
    if #exiftool_result == 0 then
        dt.print_error("[fujifilm_auto_settings] no output returned by exiftool")
        return
    end
    local exiftool_result = string.match(exiftool_result, "\t(.*)")
    if not exiftool_result then
        dt.print_error("[fujifilm_auto_settings] could not parse exiftool output")
        return
    end
    exiftool_result = exiftool_result:match("^%s*(.-)%s*$") -- strip whitespace
    return exiftool_result
end

local function apply_style(image, style_name)
    for _, s in ipairs(dt.styles) do
        if s.name == style_name then
            dt.styles.apply(s, image)
            return
        end
    end
    dt.print_error("[fujifilm_auto_settings] could not find style " .. style_name)
end

local function apply_tag(image, tag_name)
    local tagnum = dt.tags.find(tag_name)
    if tagnum == nil then
        -- create tag if it doesn't exist
        tagnum = dt.tags.create(tag_name)
        dt.print_log("[fujifilm_auto_settings] creating tag " .. tag_name)
    end
    dt.tags.attach(tagnum, image)
end


local function detect_auto_settings(event, image)
    if image.exif_maker ~= "FUJIFILM" then
        dt.print_log("[fujifilm_auto_settings] ignoring non-Fujifilm image")
        return
    end
    -- it would be nice to check image.is_raw but this appears to not yet be set
    if not string.match(image.filename, "%.RAF$") then
        dt.print_log("[fujifilm_auto_settings] ignoring non-raw image")
        return
    end
    local exiftool_command = df.check_if_bin_exists("exiftool")
    if not exiftool_command then
        dt.print_error("[fujifilm_auto_settings] exiftool not found")
        return
    end
    local RAF_filename = df.sanitize_filename(tostring(image))

    -- dynamic range mode
    -- if in DR Auto, the value is saved to Auto Dynamic Range, with a % suffix:
    local auto_dynamic_range = exiftool_get(exiftool_command, RAF_filename, "-AutoDynamicRange")
    -- if manually chosen DR, the value is saved to Development Dynamic Range:
    if auto_dynamic_range == nil then
        auto_dynamic_range = exiftool_get(exiftool_command, RAF_filename, "-DevelopmentDynamicRange") .. '%'
    end
    if auto_dynamic_range == "100%" then
        apply_tag(image, "DR100")
        -- default; no need to change style
    elseif auto_dynamic_range == "200%" then
        apply_style(image, "DR200")
        apply_tag(image, "DR200")
        dt.print_log("[fujifilm_auto_settings] DR200")
    elseif auto_dynamic_range == "400%" then
        apply_style(image, "DR400")
        apply_tag(image, "DR400")
        dt.print_log("[fujifilm_auto_settings] DR400")
    end

    -- cropmode
    local raw_aspect_ratio = exiftool_get(exiftool_command, RAF_filename, "-RawImageAspectRatio")
    if raw_aspect_ratio == "3:2" then
        apply_tag(image, "3:2")
        -- default; no need to apply style
    elseif raw_aspect_ratio == "1:1" then
        if image.width > image.height then
            apply_style(image, "square_crop_landscape")
        else
            apply_style(image, "square_crop_portrait")
        end
        apply_tag(image, "1:1")
        dt.print_log("[fujifilm_auto_settings] square crop")
    elseif raw_aspect_ratio == "16:9" then
        if image.width > image.height then
            apply_style(image, "sixteen_by_nine_crop_landscape")
        else
            apply_style(image, "sixteen_by_nine_crop_portrait")
        end
        apply_tag(image, "16:9")
        dt.print_log("[fujifilm_auto_settings] 16:9 crop")
    end

    -- filmmode
    local raw_filmmode = exiftool_get(exiftool_command, RAF_filename, "-FilmMode")
    local style_map = {
        ["Provia"] = "provia",
        ["Astia"] = "astia",
        ["Classic Chrome"] = "classic_chrome",
        ["Eterna"] = "eterna",
        ["Acros+G"] = "acros_green",
        ["Acros+R"] = "acros_red",
        ["Acros+Ye"] = "acros_yellow",
        ["Acros"] = "acros",
        ["Mono+G"] = "mono_green",
        ["Mono+R"] = "mono_red",
        ["Mono+Ye"] = "mono_yellow",
        ["Mono"] = "mono",
        ["Pro Neg Hi"] = "pro_neg_high",
        ["Pro Neg Std"] = "pro_neg_standard",
        ["Sepia"] = "sepia",
        ["Velvia"] = "velvia",
    }
    for key, value in pairs(style_map) do
        if string.find(raw_filmmode, key) then
            apply_style(image, value)
            apply_tag(image, key)
            dt.print_log("[fujifilm_auto_settings] film simulation " .. key)
        end
    end
end

local function detect_auto_settings_multi(event, shortcut)
    local images = dt.gui.selection()
    if #images == 0 then
        dt.print(_("Please select an image"))
    else
        for _, image in ipairs(images) do
            detect_auto_settings(event, image)
        end
    end
end

local function destroy()
    dt.destroy_event("fujifilm_auto_settings", "post-import-image")
    dt.destroy_event("fujifilm_auto_settings", "shortcut")
end

if not df.check_if_bin_exists("exiftool") then
    dt.print_log("Please install exiftool to use fujifilm_auto_settings")
    error "[fujifilm_auto_settings] exiftool not found"
end

dt.register_event("fujifilm_auto_settings", "post-import-image", detect_auto_settings)

dt.register_event("fujifilm_auto_settings", "shortcut", detect_auto_settings_multi, "fujifilm_auto_settings")

dt.print_log("[fujifilm_auto_settings] loaded")

script_data.destroy = destroy

return script_data

However, there's a catch: Scripts in Darktable can not modify darkroom state directly. But they can load styles. So to make the script work, we need to define a number of styles that do the heavy lifting here:

Two styles DR200 and DR400 for the dynamic range modes that brighten the image by one and two stops, respectively (I like to use the Tone Equalizer like this).
Four styles square ̲crop_landscape and square ̲crop_portrait and sixteen ̲by ̲nine ̲crop_landscape and sixteen ̲by ̲nine ̲crop_portrait that crop landscape/portrait images to 1:1 and 16:9 ratio.
One style for each film simulation you use: provia, astia, velvia, classic ̲chrome, pro ̲neg ̲standard, pro ̲neg ̲high, eterna, acros, acros ̲green, acros ̲red, acros ̲yellow, mono, mono ̲green, mono ̲red, mono ̲yellow, sepia. The linked styles use Stuart Sowerby's Film Simulation LUTs as film simulations, which must be installed in $LUTs/Fujifilm XTrans III/$lut.3dl.

Download a zip file with all the above styles here, and appropriately-renamed LUTs here. (This section will be revised once I finish building my own set of LUTs).

Then copy the lua script to ~/.config/darktable/lua/contrib/, activate it in the script manager (bottom left in the lighttable), and it should automatically run whenever you import new Fujifilm raf files! (Start Darktable with darktable -d opencl to see debug messages, and bind a keyboard shortcut to lua scripts/fujifilm_auto_settings to trigger the script manually.)

Converting Capture one Presets to LUTs

Sun, 09 Jan 2022 14:37:00 +0100

A while ago, I bought an RNI film pack for Capture One. That's a set of presets that makes your digital photos look similar to analog film scans. However, since then my other image editor, Darktable just released a new version, I'm now back to using Darktable instead of Capture One, thus without access to those presets.

Here's how to export Capture One presets to LUTs, to make them accessable to other programs.

The fun thing is, LUTs are just PNG files that contain a table of colors. You know, a "Look Up Table", of sorts. So, in order convert a preset to a LUT, all we need to do is apply the preset to a pristine "identity" LUT, and export it as a new PNG.

Get yourself an identity LUT.
For example, the one included in Stuart Sowerby's Fuji Film Simulation Profiles. Choose the sRGB PNG LUTs, for RawTherapee and Affinity Photo.
Open the LUT PNG in Capture One.
Apply the preset you want.
Optionally lower saturation by -15, see below.
Export as PNG.
Make sure the color space is sRGB, just like the original file.

As easy as that.

A few more adjustments: many Capture One presets expect to be working on raw data, which is less saturated than Darktable's default. So I export with -15 saturation. Also, many presets include spacial adjustments such as Highlights or Shadows that are bound to not play well with the LUT PNG. To disable them, delete the offending lines from the *.costyle files¹, or compensate the values with opposite slider movements.

When applying the LUTs in Darktable's lut 3D module, there are a few more things you can do to fit them into your workflow. For example, you can lower the opacity of the lut 3D module to vary their effect. Or you can choose chromaticity as blend more to only apply their color transformation, but keep Darktable's tonal rendering. In normal blend mode, some LUTs prefer a flat rendering as their input, so lower contrast in filmic rgb to zero and use the auto-pickers to set the image black and white point.

Footnotes:

don't do this for RNI LUTs, it's forbidden by the User License Agreement that is hidden quite well in dark-grey-on-black at the very bottom of their website

Books of 2021

Tue, 04 Jan 2022 09:40:00 +0100

Hologrammatica

In the last years, I have almost exclusively read books in English. Science fiction, in particular, seems to happen exclusively in English (and perhaps Chinese). So much so, that I have come to associate German only with bad translations and personal communications, whereas English was the language of science, engineering, and fiction. In 2021 however, to my surprise, I stumbled upon Tom Hillenbrand's Hologrammatica, an science fiction thriller in German. It was a peculiar experience, reading the familiar tropes of the genre in a different language. Somehow it made the story feel more immediate and approachable to me. Strange, what effect language can have.

In the book, humanity has decided to hide reality behind holograms. Clothes and hairstyles can be altered on the fly, street lighting is replaced with projected advertisements. The twist is that in this semi-dystopian world, everyone knows that the holograms hide the truth, which is a collapsed society and dilapidated infrastructure. It is a smart, very current backdrop for a detective story with mind uploads, space elevators, and all the modern trappings of science fiction. A thoroughly enjoyable read!

Sci-Fi honorable mentions: Andy Weir's pop sci-fi Project Hail Mary and Martha Well's first novel-length murderbot entry Fugitive Telemetry.

Not Much of an Engineer

I have always been fascinated with aviation, and the technology of warfare. But most of the non-fiction I have read about these topics focuses on the stories of pilots and companies and soldiers, not engineers. In these stories some of the most important plot points came from advances in technology, yet they didn't describe how those changes came about. In 2021, I finally found a good history of aviation technology: Sir Stanley Hooker's Not Much of an Engineer describes the work of the author as the principal engineer at Bristol and Rolls-Royce from the Merlin-era second world war engines to modern turbofans. I guess you need to be a bit of an aviation/engineering geek to enjoy this, but to me this book was an important missing link that I had always looked for.

Non-fiction honorable mentions: David Goodsell's The Machinery of Life wonderfully illustrates the inner workings of cells.

Crafting Interpreters

How do computers work? This question is surprisingly hard to answer. For me, the answer came in three books: Charles Petzold's Code: The Hidden Language of Computer Hardware and Software explained to me how processors work. The Arpaci-Dusseau's Operating Systems: Three Easy Pieces describes the infrastructure of operating systems that make processors and storage and network available to programs. And now, finally, Robert Nystrom's Crafting Interpreters filled in the last step, how a programming language is built.

The book describes two implementations of a simple programming language. The first one is a high-level introductory scripting language implemented in Java, the second a high-performance reimplementation in C. Fascinatingly, the entire source code for these implementations is included in the book's text, such that you can entirely follow along (or program along) and have a working programming language at all times. A truly eye-opening glimpse into the innards of the tools we are using every day. The book requires familiarity with Java and C to enjoy.

Fixing AMD OpenCL on Windows

Sat, 04 Dec 2021 09:19:00 +0100

When I recently installed Windows 11 on my desktop, my photo editor Darktable suddenly got much slower than it used to be. When I looked into its preferences, I noticed OpenCL was no longer available.

As it turns out, some versions of the AMD graphics driver apparently no longer ship with OpenCL support on Windows. However, they do ship with the necessary libraries, it's just that these libraries are not registered any longer.

To register them, open the Registry Editor (aka regedit.exe), navigate to the key HKEY_LOCAL_MACHINE\SOFTWARE\Khronos\OpenCL\Vendors (if the key does not exist, create it), and create a new DWORD of Value 0. Now rename the DWORD to be the path to your amdocl64.dll. Mine is C:\Windows\System32\DriverStore\FileRepository\u0344717.inf_amd64_d38cec78c83eee99\B343886\amdocl64.dll. Search C:\Windows\System32\ for amdocl64.dll to find the correct path on your computer.

With that, OpenCL was once again recognized by Darktable and the rest of my photo editing programs.

(source)

Comparing the Fujifilm XF 16‑80 with the XF 18‑135 Travel Zoom Lenses

Sat, 09 Oct 2021 15:16:00 +0200

When I bought into the Fuji system, I selected the XF 18‑135 f/3.5‑5.6 R LM OIS WR as my main zoom lens. This is a lens with a very wide focal range, that is commonly called a “travel zoom” because you could travel the world with just this one lens. And indeed I happily did. In 2019, Fujifilm released a second travel zoom lens, the XF 16‑80 f/4 R OIS WR. Ever since, I have wondered how this new 16‑80 compares to my 18‑135. But given that these lenses are somewhat similar, few people on the internet were ever able to compare them side by side. This blog post will change that.

The 18‑135 has served me very well indeed. Of course it is not the world's brightest lens, nor sharpest, nor smallest. But so long as I can get the shot, these limitations don't bother me. Except for two things: The transition between in-focus and out-of-focus can be a bit rough, and my lens extends on its own when carried on a sling. My hope is that the 16‑80 has a nicer rendering, especially for people photos with out-of-focus backgrounds, and a stiffer zoom ring that doesn't creep.

Instead of smoothly blurring, the near-focus areas of the 18‑135 have this strange, hazy look

Physical

Physically, the 16‑80 is a slight bit smaller (89 mm vs 98 mm) and ligher (440 g vs 490 g) than the 18‑135, but the difference is negligible on camera. If anything, the 16‑80 actually feels a bit bigger to me due to its larger front element. Some people claim that the 16‑80 feels better built than the 18‑135. To that end, the rings on my 16‑80 turn more smoothly than my 18‑135's, but then that is comparing a new-ish lens to a well-used one. Autofocus speeds are also reputably different, but they feel similar to me.

The 16‑80 has a numbered aperture ring that allows for adjustments while the camera is turned off. On the other hand, the 18‑135 can switch to and from auto-aperture without losing its preset aperture, which is useful as well. The ideal lens for me would have both the auto-aperture switch and the numbered aperture ring. Oh well.

Size comparison of the two lenses. The 18‑135 is a longer, and the 16‑80 wider. Note the marked aperture ring on the 16‑80, and the Auto-Aperture/OIS switches on the 18‑135.

The 16‑80 has a 72 mm filter thread, while the 18‑135 uses 67 mm filters. This is somewhat annoying for me, as my filters are all 67 mm, which also fit on my 70‑300. I'd imagine users of the 72 mm 10‑24 see this differently. Anyway, using a thin polarizer with a 72‑to‑67 step-down ring works without issue on the 16‑80. My inch-thick macro filter however does vignette heavily until 23 mm.

Rendering

In terms of rendering, I find the 16‑80 to have a gentler transition from in-focus to out-of-focus, and indeed render out-of-focus backgrounds more smoothly than the 18‑135. At the long end, the 16-80 is actually a useful portrait lens, which is an unexpected but welcome feature for my photography.

Rendering of out-of-focus elements. Especially the points of light in the top right have an asymmetric crescent shape for the 18‑135, but blur smoothly for the 16‑80.

At the wide end, the 16‑80 exhibits some significant distortion. While the camera or post processing programs can easily correct this, it leaves the image corners stretched, and renders the 16‑80's nice round bokeh balls as ugly ovals. Thus large-aperture shots at the wide end can be somewhat problematic.

Resolution

Next, let's compare the resolution of the two lenses. To do that, I printed a test chart, and took some test images. All images were taken in identical illumination, on a tripod, with the chart covering half the sensor height/width. The following graphic shows a resolution scale near the center of the frame and near the left edge of the frame. And just for fun, I've thrown in a similar analysis from two prime lenses as well. The resolution scales are in 200x line widths per picture height. On my 6000×4000-pixel sensor, the theoretical maximum resolution would therefore be a resolution number of ⁴⁰⁰⁰∕₂₀₀ = 20. All test images are straight crops from original images, reproduced at their original resolution.

Comparison of lens resolution with a test chart. Each image is a direct crop at the X‑T2's native resolution. Resolution numbers in 200x line widths per image height.

Center images were focused in the center, and side images were focused on the side. A two-second timer was used to eliminate camera shake. Due to the geometry of my room and the size of my printer, all pictures were taken near the close-focusing distance of the lenses. A red line indicates the limit of resolution of these lines, as judged by my eyes. The corresponding resolution numbers are generally consistent with the ones published by opticallimits and lenstip¹, albeit I judged them a bit more conservatively.

The results of this test are surprising to me: the 16‑80 is actually my sharpest lens. It not only bests the 18‑135 at all settings, but also the 35 f/1.4. Only the 60 f/2.4 can reach similar resolution at the same f‑numbers. Generally, the 16‑80 is perfectly sharp right from f/4, while the 18‑135 has to be stopped down to f/8 to reach a similar resolution, especially on the image edges. Only at the long end does the 16‑80 benefit from stopping down.

That said, take these resolution measurements with a grain of salt. For instance, a “great” result of 15 (3000 LW/PH) translates to a blur radius of 1.3 pixels, while a “mediocre” result of 10 (2000 LW/PH) is instead 2 pixels. This is scientifically significant, but not at all relevant to (my) photography.

Additionally, the fact that the 60 f/2.4 macro lens scores so highly but the 35 f/1.4 does not is an indication that these measurements might be biased by being taken near the close focusing distance of the lenses. Thus the next set of images compares these lenses at more natural distances.

Comparison of lens resolution with natural images. Each image is a 500x500 pixel crop from original photographs. The subject is ca. 100 m from the camera.

At this farther distance, and with a more natural subject, the differences are no longer as easily visible. What differences there are this time favor the 18‑135 instead of the 16‑80. Interestingly, I didn't see any significant differences between these pictures when looking at them “merely” side-by-side in Capture One. Only when I actually assembled these here graphics and looked at them at 200% did the differences become apparant.

Nevertheless, it remains curious that there would be such a difference between the two lenses. Then, someone mentioned that the 16‑80 might suffer from shutter shock, where the camera's mechanical shutter jolts the camera enough to upset the image stabilization system and induce a slight bit of motion blur. An issue such as this might explain the 16‑80's slightly reduced resolution in my test shots. So I created another series of images, but this time both, with the mechanical shutter, and with electronic shutter. In electronic shutter mode, nothing moves in the camera and there can be no shutter shock.

Comparison of lens resolution and shutter shock. Each image is a 500x500 pixel crop from original images. The subject is ca. 100 m from the camera.

From this comparison, I can see no evidence of shutter shock. It might have been an issue on earlier firmware versions of the 16‑80, but my camera (an X-T2) and lens (at firmware 1.05) does not does not exhibit shutter shock. Furthermore, this series of pictures shows the 16‑80 and 18‑135 essentially matched in image resolution.

All of that said, I must add that all of these comparisons used extremely tight crops of high-contrast geometrical features. Most of the differences here are all but invisible in actual photographs. From these resolution experiments, I see no reason to prefer one lens over the other. Both of them are perfectly sharp in everyday use.

Conclusion

So, how to choose between the Fujifilm XF 16‑80 f/4 R OIS WR and the XF 18‑135 f/3.5‑5.6 R LM OIS WR? My 16‑80 has tighter aperture and zoom rings, does not creep, and has a smoother rendering of out-of-focus background. On the other hand, I do find the increased telephoto of the 18‑135 very useful, and it doesn't suffer from wide-angle distortion as much as the 16‑80.

In terms of resolution, I did not find fault with either lens. Both are very sharp across their entire focal range and the entire frame. That said, the 18‑135 does benefit from stopping down for optimum resolution, while the 16‑80 is sharp right from f/4, and the 16‑80 might be sharper for closer subjects.

My tentative conclusion from these experiments is therefore that I would slightly prefer the 16‑80 for people pictures, where the close-focus sharpness and nicer background rendering are advantageous, and the larger aperture at 80 mm might make a difference. And I would prefer the 18-135 for landscapes, where stopping down is usually easy and the longer focal length comes in handy.

That said, the differences in rendering and resolution are really very minor, and the choice most importantly comes down to the focal range. Which is as it should be with modern lenses. And both lenses are of course very well-built, weather sealed, and have fantastic image stabilization. But you probably knew that already.

Footnotes:

multiply lenstip numbers by 2×16.7 mm to convert from lpmm (lines per millimeter) to LW/PH (line width per picture height)

A Review of Darktable

Sat, 04 Sep 2021 00:00:00 +0200

With version 3.0, my favorite photo editing software Darktable started the journey towards a scene-referred editing pipeline. Which means that most edits are no longer bounded between a fixed black zero, and a white one, but can range between zero and infinity, like light itself. This is the norm in video editing and video games, but unique in photo editors at the moment. The scene-referred pipeline has brought changes to pretty much all parts of the editing workflow. I have been frustrated with it, too, even opting to use Capture One for a while. But with version 3.6, Darktable seems to have found a new stride. This article is about my favorite features in Darktable 3.6, contrasting it with Capture One's more traditional toolset.

A camera sensor records light. When there is very little light, we call that "black", but there is really no similar label for when there is a lot of it. It's just very bright. A very bright blue light is still blue, and could always be brighter. However, screens and print and analog film can not get arbitrarily bright. Instead, they have a limited maximum brightness, and the brightest color is always white. Which is why pushing things ever brighter in photo editing eventually turns them white, because that is the brightest thing possible on screen or print.

But this is a limitation of the display, not the editor or the recording. Thus it makes sense to edit photos with a physical understanding of light, where making things brighter does not make them whiter. Consider sharpening, or color grading, or contrast adjustments: All of these can push pixels brighter or darker, but really shouldn't suddenly whiten them at some arbitrary brightness. In the new scene-referred Darktable, colors are squeezed into a viewable or printable form only at the very end of the processing, retaining their physical properties for as long as possible. In most other tools, the squeezing is done early on, and edits hobbled thereafter.

In practice, this means Darktable can change tones with any number of tools without losing color information: the tone equalizer, or the rgb curve, or levels rgb, or the contrast equalizer, or sharpening, or color balance all manipulate tones without fear of losing highlight information. Only filmic rgb at the very end determines which bright pixels to turn white. In a way, the scene-referred part of the pipeline feels like raw editing, while the rest feels like editing a JPEG. Contrast this with Capture One, where only the sliders in the Exposure and High Dynamic Range modules can recover bright pixels from white, but e.g. Clarity, Levels, and Curve can not. After getting used to the scene-referred way of working, this seems like an arbitrary limitation, and really makes Levels and Curve less useful than they should be.

Darktable's main development screen with the tone equalizer and exposure module

Interestingly, this leads to a somewhat different editing workflow in Darktable compared to Capture One. Since the scene-referred pipeline is mostly nondestructive, I can play with the overall tonality with abandon, safe in the knowledge that I will be able to recover any lost detail later. Thus I commonly use the exposure slider to expose for my mid-tones, and then reign in my burning highlights with the tone equalizer¹ or filmc rgb white relative exposure². In Capture One, digital overexposure is much more difficult to recover than underexposure, so I usually use the high dynamic range tools to underexpose slightly, and then recover mid-tone exposure with the high dynamic range shadows slider or tone curve. While these two workflows are largely equivalent, I find it interesting that Darktable can go either way, while Capture One can not.

A related area of rapid development in Darktable has been the color balance rgb module, and color calibration. Both of these allow for very fine-grained control over various aspects of color, handily parametrized by hue or tone or channel. There is some subtlety in the various controls for "colorfulness", "vibrance", "chroma", "brilliance", and "saturation" that took some learning. But it also taught me to see color in a more nuanced way, which is a good thing. In mixed lighting, I can now use several instances of color calibration modules³ for different image areas, producing more natural results that I am capable of in Capture One.

A particular pain point in Darktable 3.4 and Capture One was the rendering of colored highlights. In Darktable 3.4, bright colors often turned weirdly radiant, and then aggressively desaturated, with a sharp and unnatural edge in between. I don't quite know why, but this problem seems much reduced in 3.6. Both issues are now dealt with quickly and cleanly with a quick tug on the filmic rgb white relative exposure slider. A real boon that slider is. Now I just wish it didn't affect contrast quite so much and didn't also influence shadows, despite its name. Oh well, a man can dream. Capture One had the opposite problem, and sometimes wisted colored highlights into bright primary colors before desaturating, leading to cyan skies and yellow skin at the fringes of overexposure. This is still one of my major complaints about Capture One.

My last point of struggle with Darktable has long been the rendering of colors of my Fujifilm camera. Something about the colors always seemed off, in a way they weren't in Capture One. Greens too blue, browns too magenta, and skin too pink. After quite some experimentation, I found that this was simply due to Fujifilm depending on a bit of proprietary color magic for appropriate rendering. I now apply one of Stuart Sowerby's LUTs in chromaticity blending mode after filmic rgb. This leaves tonal adjustments fully to Darktable, but fixes the awkward colors perfectly. And it even allows me to play with the film simulations Fuji is famous for.

A LUT is applied in chromaticity blend mode after filmic rgb

Actually, this highlights a unique strength of Darktable: the order of modules in the image editing pipeline is customizable, and most modules can be applied with masks and blending modes. Only since I used Capture One for a few months did I understand how powerful this concept really is. The key difference is that "layers" in Capture One (et al) merely apply the image editing pipeline with different parameters in different places⁴, while Darktable's modules actually apply sequentially. This enables true incremental editing, such as fixing white balance correctly for one part of the image, then applying another level of white balance just for the blue parts. Or fixing colors in a LUT after all the tonal edits, without fear of destroying highlights or shadows in the process.

So overall, I am very happy with Darktable in its latest iteration. As always, there are plenty of nitpicks I could bring forward, such as being a bit sluggish in operation, and the UI could be streamlined a bit⁵. But it has once again displaced Capture One for my own work, and it makes me happy.

Footnotes:

similar to the high dynamic range module, but more fine-grained

essentially a tone curve adjustment

the new way of doing white balance

⁴

i.e. you can fully desaturate a picture to black-and-white, then "recover" saturation in a layer

⁵

the new quick access panel in 3.6 has already helped a lot!

Camera Resolution

Thu, 27 May 2021 16:45:00 +0200

Some of my photographic lenses are reknown for their outstanding sharpness, others are said to be mediocre. But somehow I never quite saw a big difference in sharpness between them. Do older lenses have less resolution than the newer ones? Can a dedicated prime lens resolve more detail than a multi-purpose zoom lens? Let's find out.

These questions recently came to a head for me, in the choice between two compact cameras: A Fujifilm X100T, 16 MP, with a 23 mm lens, and a Ricoh GR III, 24 MP, with an 18 mm lens. In the last few months, I grew to like the X100T's 23 mm viewpoint, but did not enjoy how bulky the camera is in comparison to the GR III. So I wondered if I could replace the X100T with the Ricoh GR III and simply crop to 23 mm.

Being a scientist and all that, I set up an experiment: I printed out a resolution chart, set my cameras on a tripod, and took pictures with all my cameras such that the chart filled a similar portion of each image. Base ISO, two-second timer, processed in Capture One. White balance and contrast were equalized.

Red bars indicate the point where some lines are no longer distinguishable, i.e. the limit of the system's resolution.

Each row in the above grid is one camera/lens/focal length combination, taken at the brightest aperture in the left column, and at the sharpest aperture in the right column. The text on the left shows photographic parameters, and the text on the right image parameters. The red line is the spot on the chart where some lines start to blend together, the limit of the image's resolution, as judged by my eyeballs. Depending on the exact framing, DPIs vary somewhat between the images, but generally at a factor of around √(̅2̅4̅/̅1̅6̅)̅ = 1.22 between the 24 MP images and the 16 MP images. The bottom row shows a synthetic image at simulated 24 MP / 380 DPI and 16 MP / 280 DPI.

From these images, the biggest contribution to resolution seems to be sensor megapixels. All 24 MP images resolve closer line pairs than any 16 MP image. This could also be a hint that lenses seem to generally outresolve their sensors.

Looking closely at the images, another hint becomes visible: aliasing is visible in all images. This is only possible if lens resolution is higher than sensor resolution.

Frankly, this is a rather big surprise to me. Reading discussions on the internet had predisposed me to believe that some lenses should be much higher resolution than others, and my big travel-zoom in particular should be terribly soft. From my measurements however, I see no evidence of this claim. All of these lenses outresolve their sensors, and differences between lenses is rather miniscule in general.

Another common wisdom is that lenses wide open are less sharp than stopped down. In this test sample, this seems to be true for the X100T's, and to a lesser extent for the LX100, and the Fujifilm XF 60. All other lenses do not show significant (!) differences between apertures. And even for the affected lenses, the difference in resolution is not dramatic.

However, resolution is not sharpness. While line pairs may remain resolvable, their edges do lose definition at wide-open apertures on the Fujifilm XF 18-135 at some focal lengths, and the X100T. These lenses definitely need at least additional sharpening at wider apertures. Contrast is also visibly different, but I chose to edit out contrast differences to keep the graphs legible. Generally, the Ricoh GRs' lenses (particularly the GR I) were by far the most contrasty of these lenses.

Another interesting comparison can be drawn between the synthetic images in the bottom row and the camera exposures above: None of the 24 MP images reach the resolution of the synthetic image, perhaps due to demosaicing artifacts. However, the 16 MP exposures are very close to the synthetic image. I don't yet know how to interpret this.

Speaking of demosaicing however, there does not seem to be a difference in resolution between the X-Trans Fujifilm sensors and the Bayer Ricoh/Panasonic sensors. I also checked whether a different raw developer would make a difference, but did not find anything noteworthy.

Overall, this experiment has taught me that all my lenses are high resolution, and that lens resolution in general is probably not a topic worth fussing over. Subpixel patterns are equally unimportant. Instead, lenses do differentiate themselves in contrast and sharpness, and camera sensors in megapixels. And finally, the tiny GR III in crop mode can indeed outresolve the X100T, and out-contrast and out-sharpness it as well.

Books of 2020

Wed, 05 May 2021 17:14:00 +0200

Chickenhawk

Most first hand accounts of wars I have read are written by a somewhat amateur author. Not this one. Robert Mason infused his terrifying tale with plenty of drama and humanity, and managed to write one of the best personal military history I have ever read.

Part of it is surely the subject matter, as I am a huge fan of aviation and its lore. And part of it is that I haven't yet read a lot about the Vietnam War.

But either way, I could not put this book down. Every mission seems to be more dramatic than the last, and you can see Mason's flying and survival skills just barely keeping up with the challenge. You can also viscerally feel the surviver bias at work, with plenty of close calls and dead friends. It's terrifying and thrilling at the same time.

And towards the end, the book truly surprised me with a very frank account of PTSD and the life of a veteran, which I hadn't read anywhere else in this level of clarity.

I cannot recommend this book enough. Perhaps a bit depressing at times, but an utterly compelling story, and without a doubt one of the best books I have ever read.

Mountain Light

Quite simply, the best photography book I have ever read. From the era of color film, that short time between the black-and-white darkroom and photoshop, where pictures really couldn't be edited. It all came down to the skill of the photographer, which makes a book from this era much more useful than its darkroom-edited predecessors or photoshopped successors.

Besides that, the story of an avid adventurer who took part in several expeditions to climb the most difficult mountains in the world for the first time, and document life in the most remote places on earth before man could ever touch them.

All in all, a fantastic book of outdoorsmanship, adventures, and photography, three things I love dearly.

Two Years Before the Mast

The story of the young author joining a trader ship to the US west coast, in 1834, when it was still largely uninhabited and wilderness. They traveled to many a well-known place, such as San Francisco or Santa Barbara, but these places were still merely trading outposts and missions.

At the same time, the author chronicles a sailor's life on board the last generation of sailing vessels, just before they were replaced by steamers. He joined the ship on a two-year-or-so contract around the American continent, on an entirely self-sufficient ship with only the occasional sighting of a fellow ship or the lighting of cargo in a trading depot. Which is an utterly fascinating historical perspective considering that the story happens more-or-less at the same time as the comparatively modern-seeming Sherlock Holmes.

But a true eye-opener happens at the very end, where the author re-visits San Francisco later in his life. The city by then has grown to a bustling trade town with regular ferry routes and rail connections, churches and pubs, and all the amenities of modern life. It is hard to believe how such industry would grow from these humble beginnings in just a few decades.

Bobiverse: We Are Legion / For We Are Many / All These Worlds

Like good candy, I just couldn't stop reading. The story of a regular guy, uploaded into a colony ship as an AI, roaming the galaxy, and exploring the boundaries of what it means to be human. Which sounds much more philosophical than it actually is. The book is written in a lighthearted and fast-paced style that I could hardly put down. I read the first book in one week, and immediately followed through with book two and three.

And it actually comes to a satisfying end in the third book, instead of methastesizing into a franchise, which I can't praise highly enough. Perhaps a little bit superficial and handwavy at times, but a thoroughly good time for a science fiction fan.

A Review of the RAW Photo Editor Capture One

Tue, 04 May 2021 13:24:00 +0200

About half a year ago, my second child was born, and all that precious free time I had used for photo editing evaporated. So I started looking for faster photo editor, as a temporary replacement for Darktable. Based on my research last time, I chose Capture One. During the last six months, I got to know the program well, and want to share my thoughts on it, and how it compares to Darktable.

Let's start with a bit of background about my photography, to put my views into context. I learned photo editing on Linux, with Darktable, mostly for documenting my travels and family. I gather that a photographer who does not know Lightroom is a bit of an oddity. But it should provide an interesting perspective for the purposes of this review. Furthermore, my interest in post processing is not purely artistic. Being a signal processing scientist, I can't help but analyze the underlying image processing algorithms, and am perhaps more annoyed by certain editing artifacts than most people.

In terms of photography, I tend towards quick reactionary shooting of the fleeting moments of my family's life. Mostly because I prefer to spend my time playing with my kids instead of pointing a camera at them. I am therefore perhaps a bit more tolerant of slight imperfections in terms of focus and exposure than is usual. This also means I value my camera/editor's capability for recovering misjudged exposures or imperfect lighting very highly.

Most review articles of photo editing software are conducted from only a short glance at the contesting programs, since spending the months necessary to get to know each program is usually neither practical nor affordable. But due to my particular circumstances, the present text actually comes from a few months of more-or-less exclusive use of Capture One, on about 30 thousand pictures, of which I edited about 2000 (all new, not just re-edits of old ones, and implicitly the incentive to recreate another program’s rendering).

Capture One's main develop screen

So, Capture One. It is very fast. You move a slider, you see the results of that change immediately, even on my lowly Surface 7 Pro tablet. But this focus on speed actually goes deeper than just slider movements: I discovered that many parts of the application seemed actively optimized for quick operation, which allows me to process pictures much more quickly. This was frankly a revelation in the time-starved months after my second baby’s birth.

As an example of how much quicker Capture One's default workflow is than Darktable's, here's my usual editing flow in Darktable: I start with fixing exposure, in the Exposure module. Then I adjust shadow density and recover highlight color with the black and white sliders in the Filmic RGB module. Then I adjust white balance to taste in the Color Calibration module. Then I crop in Crop and Rotate, then add additional adjustments such as Color Zones, Color Balance, Tone Equalizer, Denoise (Profiled), or Contrast Equalizer. Last come local adjustments if necessary. The thing is, more or less every image needs at least Exposure adjustments, some Filmic RGB tweaking, and cropping. All of these items are situated in different modules in Darktable, each requiring multiple clicks to access the module and then change the value. This is supposed to be improved in the next version of Darktable, which will let you put all your favorite sliders in a custom module. I am very much looking forward to that.

Darktable's basic controls

In contrast, a similar workflow in Capture One happens entirely on one screen, just by going from one slider to the next:

Capture One's basic controls

Furthermore, common tools such as cropping, rotating, and the white balance picker are accessible at all times with highly memorable keyboard shortcuts (C, R, W, respectively). Taken as a whole, this allows me to positively blaze through images compared to my Darktable workflow. It took me a while to appreciate how significant this difference was in my use: I now sometimes do a few days’ edits in a spare half hour, which used to be an all-evening affair in my usual Darktable workflow.

Which is not entirely Darktable's fault. Assigning custom keyboard shortcuts in Darktable can speed things up, and of course Darktable’s module system is infinitely more powerful, so there's a reason for its complexity. But the above example highlights a bit of a philosophical difference between Darktable’s unflinching priority on user control, and Capture One’s compromise between power and speed. There are upsides and downsides to both, but at this moment in my life, I begrudgingly value speed over power.

However, there were also a number of occasions where I missed Darktable’s deep control. Most notably, Capture One’s High Dynamic Range sliders and Clarity controls feel a bit restrictive and oversimplified: It can be hard to control which parts of the image are affected, and the tools are prone to produce artifacts such as halos if not managed carefully. In contrast, Darktable's continuous Tone Equalizer mask gives very precise and adjustable control over the affected area. Similarly, Darktable's Contrast Equalizer can control local contrast at arbitrary wavelet sizes, not just the very small ("Structure") or very big ("Clarity") ones, for example for specifically highlighting tree trunks or bird feathers.

Highlighting feather detail with darktable's contrast equalizer

Another annoyance with Capture One can be found in colors at the edges of the tone curve: When pushing exposure, Capture One tends to turn brightly colored highlight towards primary colors such as cyan/magenta/yellow before desaturating them into white. Similarly, shadow recovery sometimes pushes brightness and saturation a bit more strongly for brightly colored shadows than for dull ones, leading to an unnatural glowy effect. These cases are usually easy enough to fix with a quick tug on the shadows/highlights slider or by reigning in the offending color's lightness in the Color Editor, but they smell a bit of a runaway algorithm, which bothers me. Although in direct comparison they bother me less than the various rocks and hard places in Darktable's Filmic RGB chrominance preservation modes.

On the topic of bothersome details, Capture One's repair layer seemingly has a mind of its own when deciding whether to create a new control point, or adding to an existing one. The Highlight and White slider in the High Dynamic Range module sometimes can't quite decide whether to move white point, and sometimes lead to lightness reversals around bright objects. More on the algorithmic side, it annoys me that only the Exposure and High Dynamic Range modules have access to the full dynamic range of the raw file, while all the other tools apparently come after the screen transform, and therefore can't reach beyond the black point and white point. This makes Levels and Curve a bit less useful than I'm used to. Demosaicing is also not quite as detailed as Darktable's, although only by a tiny margin.

One surprising limitation of Capture One is that its support for cameras and lenses is a bit hit-and-miss. Film simulations are only supported for my Fuji X-T2, but not for the older Fuji X100T, nor the Ricohs or Panasonic. Some of their lens profiles are also laughably bad, and leave obvious reverse-vignetting or barrel distortion when enabled. Frankly, Darktable does better in terms of lens support, despite relying solely on volunteer support.

In terms of UI, Capture One is generally well-organized and easily configurable. But there is a constant stumbling block in the Layers module that I find very annoying: many of Capture One's tools automatically create new layers, but deselecting the tool does not deselect the layer. As delightfully easy it is to press B for the brush and paint in some Clarity, as unnecessarily laborious it is to then spend three clicks to slide open the Layers module and select the background layer again to resume editing. Or alternatively, wonder why your edits don't work while you don't have the background layer selected. It's a true pain.

With a layer selected, edits only affect the layer

As the last point of the nit picking, I was disappointed by the number of minor technical bugs I encountered in my use of Capture One: Half the time, Capture One starts half zoomed-in, and leaves me scratching my head at what weird compositions I chose for a few seconds. And the main, zoomed-out view is weirdly blurry if you use two displays with different DPI. Both of these issues are well-documented on their forums for several releases, but have not been fixed. For the exorbitant price that Capture One commands, such issues and customer communication are frankly unaccetable.

Finally, a few words about the library module and file organization. At first glance, I hated Capture One’s library. You have to import every single directory manually (I organize my images in daily directories), all the edits go into a central catalogue and nowhere else, and the sidecar files contain no editing information. But then someone told me a much better way: Instead of using Capture One’s Catalogue, create a Session, but ignore all those pre-built input and output directories, as well as the import button, and instead simply navigate to any old directory on your computer with the sidebar file browser. This is clearly not how sessions are meant to be used. But it actually works reasonably well, and puts editing information in a subdirectory next to the raw file.

On the whole, I grew to quite like Capture One, despite its flaws, mostly for its streamlined user interface and speed of operation. In terms of image quality, I honestly didn’t see much difference between Darktable and Capture One. But perhaps I am not the most discerning of users, either, as my focus is not on crazy detail recovery or the more technical arts of macro or astro. When it comes to control, I occasionally felt restricted and, dare I say, patronized by Capture One. But the speed of operation and general good-enough-ness of the image quality are still hard to argue with. I just wish they fixed their UI bugs, and improved their algorithms a bit.

A review of iOS, from an Android user's perspective

Sun, 27 Dec 2020 14:32:00 +0100

My Pixel 2 phone received its final software update the other day, and its battery took the occasion to really show its age. So it was time for a new phone. Which is not a decision to make lightly, considering that I spend multiple hours per day using it. And this time, I was inclined to give the iPhone another go, mostly because of Google's gross lack of human decency in the last few years. So, after years of using various Android devices, I bought a used, red, iPhone SE (2020). I made this choice with some trepidations, as my last experience with iOS was an iPad 3 from 2012. These are my experiences.

As a seasoned Android user, my first impressions of iOS were a bit of a mess. There are things that swipe up from the bottom, swipe down from the top, to the left of the home screens, to the right of the home screens, and at various permutations of pressing/holding the home button. Everything animates, takes a long time, and moves the viewport around. Particularly annoying is the placement of the back button in the top left corner, i.e. the most-inconvenient place on the entire screen for the arguably most-important gesture of the entire UI¹. A close second are the context menus that slide out from a long-pressed item, with the menu thus in a different position on the screen every time you long press anything. These things may be less flashy on Android, but are seriously simpler.

I also immediately missed Android's customizeable home screen, with freely-positionable app icons and a plethora of useful widgets. iOS is very restrictive in this regard, and seemingly for no good reason. Why is the list of all apps (right-of-homescreen) sorted arbitrarily into nonsensical folders instead of a plain list? Why are app widgets allowed, but only on that weird left-of-the-home-screen screen? Why can't I have a currently-playing widget for my podcast player, a weather radar, or my groceries list? Apparently, iOS 14 has a brand new API that does now allow Android-like widgets in iOS, but at the moment they were only available for Apple's own (useless) apps.

My second big stumbling block was the iOS on-screen keyboard. At first glance, I thought it a terribly clunky thing. Actions as simple as inserting a comma require multiple taps, and positioning the cursor seemed almost comically difficult. But then I discovered that the "123" button in the bottom left can be swiped instead of tapped, which makes commas and periods and hyphens available to a quick swipe. That is seriously cool, if slightly hampered by my accidentally activating Control Center instead of swiping from "123" a bit too often. And precise cursor positioning is hidden behind a long-press of the spacebar. Very cool indeed. With these gestures, the iOS on-screen keyboard is actually not bad at all.

Autocorrect seems capable as well, and multi-language aware (hear that, Android?). And, mind-blowingly, a triple-swipe on the content area engages undo and redo in the current text area. Albeit a nigh-undiscoverable gesture, this is miles better than Androids undo/redo system (there isn't one). Actually, Android text fields do support undo and redo if you press Ctrl-Z on the keyboard, it's just that no on-screen keyboard has a Ctrl key. This is my number one grievance with Android at the moment (although there are workarounds).

As a summary to the keyboard situation, I came to respect the iOS keyboard. I still miss a German "ß" key, more control about the autocorrect system, and a more modern design. But it works reasonably well. Better than Android's stock keyboard for sure.

My second big hurdle with iOS was the camera. Just like my Pixel's camera, the iPhone takes perfectly acceptible photos. But everything else is just plain worse than on Android. It starts with the way you get access to the camera, either by swiping right on the lock screen, or by tapping the camera app on the home screen. On Android, you double-click the lock button to start the camera, then hit the volume button to take a photo. Not only is this significantly faster, it also requires no ungloved finger, and no look onto the screen. And it doesn't make that abominable shutter sound, either, that the iPhone requires (unless the entire phone is silenced or you take video).

When I take a picture with my phone, it is because I don't have a dedicated camera at hand. Considering the number of cameras I own, this is most likely to happen when I need a camera fast. Thus the speed from pants to picture is extremely important to me, and this is a clear victory for Android. And additionally, Pixel phones have supported stacked RAWs since the original Pixel², and offer quite a comprehensive image editing suite right in the stock camera app. And I prefer the Pixel's relatively neutral rendering over the iPhone's oversharpened, over-denoised, waxy images. But that might be just me.

At this point, I had more or less given up on iOS, and bought a Pixel 4a, which is probably Android's closest competitor to the iPhone SE.

Beyond the camera, there were a number of annoyances with iOS that I found hard to get used to. Like the fingerprint reader on the SE being very unreliable for me (50% miss rate), and awkwardly requiring a distinct push to activate the home button, where the Pixel's is quicker and in a more convenient location. The home button is in fact not a button, but an immovable touch pad that fakes a button-like behavior with a little rumble effect when pressed uncomfortably hard. Pressing a hard surface really hard did not get comfortable to me, especially not when invoking the multi-tasking switcher with a double-press. And it's awkwardly placed at the very bottom of the device, where my thumb does not rest with any kind of force in normal usage. Another real problem is the lack of third-party browsers³, which just leaves you in the cold should a website not work in Safari, or should you dislike Safari's rendering or its very limited ad-blockers. I was particularly annoyed by Safari's context menu for links, which slowly slides out from the link's location, often extending far outside of the screen. Thus opening links in new tabs is just awkwardly slow and annoying in Safari, where Android's more utilitarian menu gets the job done much quicker. Although a phone with a bigger screen might mitigate this problem somewhat.

On the topic of animations in general, one of my favorite features in Android is the animation speed setting in the Developer settings. If you want to feel like you got a new, faster phone, just set the animation speed to 0.5x, and marvel at the newfound snappiness of everything.

Apps in general feel annoyingly restrictive on iOS. Where is a true Firefox, with addons? Where is a Youtube player that can play a YouTube video in the background while the screen is locked, or block ads (NewPipe is the bee's knees!). Or a file synchronization tool that can actually synchronize useful data outside of its own sandox? In fact, my original hopes for iOS were driven by my memory of the fabled higher quality apps available only on the App Store. Looking at some of my old favorites on the SE however, I was forced to take off those rose-tinted glasses. These apps might have been radical back in the day, but the world has moved on. I don't see a pervasive difference in app quality between iOS and Android any longer. Of course iOS apps do still cost a more money on average, and often can't be test-driven without paying. That two-hour free return policy on the Play Store is seriously genius.

Additionally, there were a number of odd problems with the iPhone SE that I found hard to make sense of. For example, apps were very frequently getting booted out of memory. So much so in fact, that often my RSS feeds would not refresh in the background, and podcasts would not download. Even though the iPhone SE sure does have a lot less memory than the Pixel 4a, I hadn't expected this to be an actual problem⁴. And the iPhone would frequently misunderstand a vertical swipe for a horizontal one, or initiate an edge swipe when my finger was still clearly on the screen; My bluetooth headphones sounded noticeably worse, with a strange clipping artifact that should not be there; Significantly worse cell reception and voice quality; Low contrast and tiny text on lock screen notifications; And only barely adequate battery life. And let's not even talk about the stupid Lightning cable, the (comparatively) laggy and tiny TFT screen, the lack of a podcast client I like, and my horrible experiences during the setup process.

So, in summary, the iPhone SE was not for me. Don't get me wrong, it's a nice enough phone, and probably enough of a smartphone for most people. But there were a number of issues with both its hardware and its software, where I found Android and the Pixel 4a plainly more productive. Which is surprising, as my mind still had iOS pegged as the premium alternative, that I would totally buy if I wasn't too cheap for it. But in actual use I was annoyed at its preference of form over function, with many a distracting animation and a number of glaring ergonomic mistakes. Well, another lesson learned. The only truly significant advantage of iOS remains its vastly superior software support story: The SE will probably receive software updates until 2025 or 2027, where the best-in-class Pixel 4a will definitely expire in 2023.

Footnotes:

in some places, a right swipe can be used instead of the back button. But the behavior is too inconsistent to be useful

what Apple calls "ProRAW", and only makes available in the iPhone 12. iOS does support ordinary RAW, but only in third-party camera apps, and anyway non-stacked RAW files are rather useless from a tiny phone sensor

there are different browsers, but they are all required to use Safari's web renderer.

⁴

I hear this might have been a bug in iOs 14.3? What this says about the software quality of iOS in general is even more troubling, however.

Differences Between Camera Sensors

Tue, 27 Oct 2020 10:00:00 +0100

Most cameras have the option to capture raw images, i.e. un-processed image data right from the image sensor. In theory, these images are pure physical measurements of light, and should therefore be very comparable between cameras. But are they? To investigate, I took a raw picture of a color target with each of my five cameras, and compared their output.

Capturing accurate colors is a surprisingly intricate matter, as the color depends on the spectrum of the illumination, the reflective spectrum of the colored object, and the color filters in the camera. I normalized the illumination by taking pictures on an overcast day outside¹, and used a standard color target with 24 colored patches of a certified color.

The resulting colored spectra are captured on the camera sensor by photon counters behind three color filters “red”, “green”, and “blue”², which differ from camera to camera in their spectral sensitivity. The recorded colors then get projected on a computer screen with another set of “red”, “green”, and “blue” LEDs of some different spectral makeup, or printed in three or more inks on paper. And this is then seen with eyes of yet another set of “red”, “green”, and “blue” retinal cells. Let's leave it at color is complicated.

At any rate, I took pictures at base ISO in raw, white-balanced and exposed for the third grey patch, and adjusted in saturation on the red patch. Here are the colors on the color checker and the recorded colors of my cameras:

The image shows the 24 colored patches of the color target³. Within each patch, five rectangles show the recorded color of (in reading order) the Pentax Q7, the Ricoh GR, the Panasonic LX100, the Fujifilm X-T2, and the Google Pixel 2.

Due to my exposure and white balance calibration, the third grey patch is completely equalized and shows no difference between the cameras and the target. The “Light Skin” patch (second on the first row) is also very close to equal, which is an important sanity check for my measurement procedure, as skin tones are optimized by all cameras. Seeing that the skin color patch is one of the most similar hopefully indicates that I did not do anything gravely wrong.

The last row shows greys, which are very similar across cameras (excluding the white patch for now). This indicates that RAW values indeed correspond to photon counts, with little additional processing. I have no idea why the white patch is off. Perhaps due to specular reflections on the white patch? Color is complicated.

As for the other colors, the Panasonic LX100 (center) and the Fujifilm X-T2 (bottom left) seem to record somewhat more accurate colors than the other three cameras. The Pentax Q7 (top left) and Ricoh GR (top right) seem comparatively weak across the board. The Panasonic (center) is great in darker blues and reds and greens, but somewhat weaker in yellows and pastels. The Fujifilm (bottom left) is very good in almost all colors, with pastels somewhat weaker than darker colors. The Google Pixel 2 (bottom right) is better in reds and greens and yellows than in blues and oranges. Pastel colors in general seem weaker than darker colors, which might be due to the same process that affected the white patch as well.

I think I learned something from this experiment: There are indeed color differences between cameras even when shooting RAW, but they are relatively minor. Whenever I see larger differences in my pictures, they are likely (easily correctible) white balance differences, and not sensor differences. When in doubt however, my Panasonic LX100 and Fujifilm X-T2 do capture more lifelike colors than my other cameras.

Footnotes:

any kind of artificial light is comparatively terrible for color reproduction!

in quotes, because colors are three-channel information, and we're talking about single channels here.

encoded as sRGB, which might or might not be displayed correctly on your screen, depending on your OS, your browser, your settings, your room's illumination, and your screen. Color is complicated.

Lenses Matter

Fri, 16 Oct 2020 12:14:00 +0200

A common trope in discussions about cameras on the internet is: “sensor size trumps all”, and as a corollary, “smartphone cameras suck”. Which is obviously false to anyone who has ever taken a good picture on a smartphone (with its tiny sensor).

The argument, however, goes like this: Bigger pixels¹ can capture a greater dynamic range, and bigger sensors are made from bigger pixels. Therefore, and here is the dangerous leap, cameras with bigger sensors take better pictures.

But I currently have a bit of time on my hands, and way too many cameras, so let's try it out, shall we?

My five contestants are:

A Google Pixel 2, with a 12.2 MP 5.75 × 4.32 mm “1/2.55 in” sensor from 2017, and a 3.8 mm f/1.8 lens (27 mm f/13 equivalent, crop 7.14),
A Pentax Q7, with a 12 MP 7.44 × 5.58 mm “1/1.7 in” sensor from 2013, and a 5-15 mm f/2.8-4.5 lens (24-70 mm f/13-22 equivalent, crop 4.76),
A Panasonic LX100, with a 13 MP, cropped 17.3 × 13 mm “micro four thirds” sensor from 2014, and a 11-34 mm f/1.7-2.8 lens (24-75 mm f/3.8-6 equivalent, crop 2.2),
A Ricoh GR, with a 16 MP 23.7 × 15.7 mm “APS-C” sensor from 2013, and a 28 mm f/2.8 lens (27 mm f/4.2 equivalent, crop 1.5),
A Fujifilm X-T2, with a 24 MP 23.6 × 15.6 mm “APS-C” sensor from 2016, and a 18-135 mm f/3.5-5.6 lens (27-200 mm f/5.25-8.4 equivalent, crop 1.5).

So I went for a walk around the neighborhood, and took a few pictures with a bag full of cameras. In particular, I was looking for a high dynamic range scene, in the form of a sunset. All pictures were taken at a low ISO² and underexposed by two stops to account for the backlighting. All images were taken at 27 mm (equivalent³) focal length.

Measurements predict that the dynamic range and noise levels of these cameras should get better as the sensor size increases.

The first scene was taken essentially in auto mode, and adjusted in Darktable to match roughly in tonality (exposure adjustments of ±0.25EV), but not color. In general, all cameras were able to capture the entire dynamic range of this scene, from the brightest almost-white area around the sun, to the darkest patches in the shade near the waterline.

This, right off the bat, is the most important result of this experiment: Most scenes do not have enough dynamic range to matter.

Differences appear in the noise levels, which are fairly constant across all cameras, with only the X-T2 being noticeably less noisy than the rest. However, this advantage of the X-T2 is not one of sensor size (the Ricoh GR has a same-size sensor), but simply of being two-generations newer.

The largest difference between these photos is instead in flare: The Pixel 2, Pentax Q7 and Ricoh GR are flaring across the entire image, while the LX100 and X-T2 are far more controlled. This is probably because the Pixel 2 and Ricoh GR have collected dust and scratches from my pockets, and the Pentax Q7 lens being somewhat mediocre. The LX100 and X-T2 are in a better condition, and a better design, too. All lenses also show some amount of colorful lens flares to the bottom-right of the sun, particularly in the LX100 and Pentax Q7, and strange, colored rings around the edge of the Ricoh GR's frame.

To look into these differences between lenses in more detail, the second scene was taken from a slightly different spot, with the lenses stopped down (if possible). This image includes two bright sun spots, one from the sun itself, and one in the reflection on the water.

Notably, the Pixel 2, like most smartphones, can not change its aperture. And it produces a bright red flare towards the left edge of the image. The Pentax Q7 again flares dramatically and colorful, and my Ricoh GR's lens is probably beyond cleaning at this point, judging from the massive flare it produces. Again, the standout pictures here are the X-T2 and LX100, with well-defined sun stars and comparatively little flaring. (Note the red dots in the X-T2: these are the phase-detection pixels on the X-T2's sensor reflecting light back into the lens and back on the image.)

Here is an enlargement of the dark patch at the right edge of the second scene, to judge noise levels and detail in the shadow. Perhaps surprisingly, the differences between these images are not big at all. The X-T2 clearly wins in terms of both shadow detail and noise levels, but by a rather small margin, and only really visible when enlarged. The Pentax Q7 resolves almost no detail in the shadow, but neither does the Ricoh GR. So sensor size does not seem to affect this round.

This third scene was shot one day later, with settings optimized for optimal image quality: Base ISO, aperture one stop closed from wide open, focused on a high-contrast part of the scene. I processed these images with white balance from a grey card, then edited exposure, contrast, and saturation to match across images. Chromatic aberrations and vignetting were corrected using a profile or by hand. No sharpening was applied, but the highest-quality deinterlacing was selected (AMaZe for Bayer sensors, Markensteijn 3-pass for X-Trans sensors).

At this magnification, even my 24in/4K screen only barely hints at differences in sharpness. Which implies that prints up to A3 should be possible from all of these cameras without much difference in perceived sharpness.

If we crop into these images, differences in detail resolution indeed become apparent. The Pentax Q7 is definitely the least detailed, on account of its soft lens. The Pixel 2 and LX100 show a very similar amount of details, which is a testament to how good modern smart phone cameras truly have become. While the two cameras with the biggest sensors, the Ricoh GR and the X-T2, do resolve visibly more detail, the differences are not dramatic.

So does sensor size trump all? From these experiments, I wouldn't say so. Essentially, all of these cameras took decent images. The X-T2 clearly was better than the rest, but not hugely so, while being much bigger and more expensive and newer.

While the measured dynamic range differences surely exist, they did not manifest even in a sunrise, so I'm inclined to discount their importance. This is, frankly, not what I expected. The miniscule differences in resolution were equally unexpected.

What did however make a big difference was lens quality. Flare control, aperture range, and focal range play a huge role in the resulting image quality. Zooming in (and stopping down) are still the most effective way of extracting more detail from a scene. And having a camera with a secure grip that is quick to operate helps, too. I think I am growing out of sensor size snobbery, is what I'm saying.

Footnotes:

technically, “sensor elements”, not “picture elements”, i.e. “sensels”

lower ISO → less noise.

the focal length a lens on a 35 mm film camera would need to cover the same angle of view.

Dear Computer, We Need to Talk

Thu, 17 Sep 2020 10:25:00 +0200

After years of using Linux on my desktop, I decided to install Windows on my computer, to get access to a few commercial photo editing applications. I'll go into my grievances with Linux later, but for now:

I tried to install Windows, you won't believe what happened next

Like I have done many times with Linux, I download a Windows image from my university, and write it to a USB drive, then reboot into the USB drive. The USB drive can't be booted. A quick internet search leads me to a Microsoft Support page on how to Install Windows from a USB Flash Drive, which says that

Instead, one has to format the stick as FAT32, make it active, then copy the files from the image to it. So I follow the instructions and open the Disk Management program. It does not offer an option of FAT32, nor for making the partition active. I settle on (inactive) exFAT instead. It doesn't boot.

I switch over to Linux, where I can indeed make a FAT32 partition, and I can mark it as bootable, which I take as the equivalent of active. But Linux can not open the Windows image to copy the files onto the USB stick. So back to Windows, for copying the files. Except they can't be copied, because some of them are larger than 4Gb, which can't be written to a FAT32 partition. What now?

While researching how to download a different version of Windows 10, I stumble upon the Media Creation Tool, which automatically downloads Windows 10 and writes it to the USB stick correctly. Why was this not pointed out in the article above? Who knows. At any rate, it works. I can finally install Windows.

The installation process requires the usual dozen-or-so refusals of tracking, ads, privacy intrusions, and voice assistants. I wish I could simply reject them all at once. And then the install hangs, while "polishing up a few things". Pressing the helpful back button, and then immediately the forward button unhangs it, and the installation completes.

Next up are drivers. It feels anachronistic to have to install drivers manually in this day and age, but oh well. The new GPU driver to make screen tearing go away, a driver for my trackball to recognize the third mouse button, a wacom driver, ten or so Intel drivers of unknown utility. The trackball driver is not signed. I install it anyway. The GPU driver does not recognize my GPU and can't be installed. A quick Internet search reveals that my particular AMD/Intel GPU/CPU was discontinued from support by both AMD and Intel, and does not have a current driver. But fora suggest that up to version 20.2.1 of the AMD driver work fine. They don't, the driver crashes when I open images in my photo editor. An even older version published by Intel does work correctly. So now I am running an AMD GPU with an Intel driver from 2018.

Installing and setting up Firefox and my photo editors works without issue, thank goodness. Emacs has a Windows installer now, which is greatly appreciated. OpenCL and network shares just work. This is why I'm installing Windows next to my Linux.

But Windows is still not activated. I copy my university's product key in the appropriate text box, but hesitate: That's for Windows Enterprise, and I'd be just fine with Home. So I cancel the activation without activating. A helpful link in the activation systems sends me to the Microsoft Store to get my very own version of Windows Home for €145, which normally retails for around €95, so that's a no-go. Whatever, I'll go with my university's Enterprise edition. Except the activation box now says my product key is invalid. And the Store now literally says "We don't know how you got here but you shouldn't be here" instead of selling me Windows. After a restart it installs and activates Windows Enterprise, even though I never actually completed the activation.

I install Git, but in order to access my Github I need to copy over my SSH key from the Linux install. Which I can't boot at the moment, because installing Windows overwrites the boot loader. This is normal. So I download Ubuntu, write it to the USB stick, boot into it, recover the bootloader, boot into my old install, reformat the stick, copy the files to the stick, boot back into Windows, and the files aren't on the stick. Tough. Boot back into Linux, copy the files onto the stick, eject the stick, boot back into Windows, copy the files to the computer. Great user experience.

Now that I have my SSH key, I open a Git Bash to download a project. It says my credentials are incorrect. I execute the same commands in a regular CMD instead of Git Bash, and now my credentials are correct. Obviously.

There are several programs that claim to be able to read Linux file systems from Windows. They do not work. But Microsoft has just announced that you will be able to mount Linux file systems from WSL in a few weeks or months. So maybe that will work!

I set my lock screen to a slideshow of my pictures. Except my pictures do not show up, and I get to see Window's default pictures instead. An internet search reveals that this is a wide-spread problem. Many "solutions" are offered in the support fora. What works for me is to first set the lock screen to "Windows Spotlight", then to "Slideshow". Only in that order will my pictures be shown.

I will stop here. I could probably go on ad infinitum if I wanted to. This was my experience of using Windows for one day. I consider these problems relatively benign, in that all of them had solutions, if non-obvious ones.

Why install Windows in the first place?

Part of the reason for installing Windows was my growing frustration with Linux. I have been a happy user of KDE of various flavors for about seven years now. But ever since I got into photo editing, things began to become problematic:

My photo editor requires OpenCL, but the graphics driver situation on Linux is problematic, to say the least. I generally managed to get RocM running most of the time, but kernel updates frequently broke it, or required down- or upgrading RocM. It was a constant struggle.

I wanted to work with some of my data on a network share, but KDE's implementation of network shares does not simply mount them for applications to use, but instead requires each application to be able to open network locations on their own. Needless to say, this worked almost never, requiring many unnecessary file copies. Perhaps Gnome handles network shares better, but let's not open that can of worms.

Printing photos simply never worked right for me. The colors were off, photo papers were not supported, the networked printer was rarely recognized. Both for a Samsung printer and an Epson and a Canon. One time a commercial printer driver for Linux printed with so much ink it dripped off the paper afterwards. Neither Darktable nor Gimp nor Digikam have a robust printing mode. I generally resorted to Windows for printing.

I ran that Windows in a virtual machine. With Virtualbox, the virtual machine would be extremely slow, to the point where it had a delay of several seconds between typing and seeing letters on the screen. VMWare did better, but would suddenly freeze and hang for minutes at a time. Disabling hugepages helped sometimes, for a short while. The virtual machine network was extremely unreliable. Some of these issues were probably related to my using a 4K screen.

Speaking of screens, I have two screens, one HighDPI 4k and one normal 1440p. Using X, the system can be either in HighDPI mode, or in normal mode. But it can't drive the two displays in different modes. Thus the second monitor was almost useless and I generally worked only on the 4k screen. With Wayland I would have been able to use both screens in different modes, but not be able to color-calibrate them or record screen casts. Which is completely unacceptable. So I stuck with one screen and X. In Windows, I can use both screens and calibrate them.

Additionally, Linux hardware support is still a bit spotty. My SD card reader couldn't read some SD cards because of driver issues. It would sometimes corrupt the SD card's file systems. USB-connected cameras were generally not accessible. The web cam did not work reliably. The CPU fan ran too hot most of the time.

So there had been numerous grievances in Linux that had no solutions. Still I stuck with it because so many more smaller issues were actually fixable if I put in the work. In fact I had accumulated quite a number of small hacks and scripts for various issues. I feared that Windows would leave me without recourse in these situations. And it doesn't. But at least the bigger features generally work as advertised.

Where do we go from here?

Just for completion's sake, I should really find an Apple computer and run it through its paces. From my experience of occasionally using a Macbook for teaching over the last few years, I am confident that it fares no better than Linux or Windows.

Were things always this broken? How are normal people expected to deal with these things? No wonder every sane person now prefers a smartphone or tablet to their computers. Limited as they may be, at least they generally work.

There is no joy in technology any more.

How to Write a Dissertation

Wed, 27 May 2020 18:11:00 +0200

Assembling scientific documents is a complex task. My documents are a combination of graphs, data, and text, written in LaTeX. This post is about combining these elements, keeping them up to date, while not losing your mind. My techniques work on any Unix system on Linux, macOS, or the WSL.

Text

For engineering or science work, my deliverables are PDFs, typically rendered from LaTeX. But writing LaTeX is not the most pleasant of writing environments. So I've tried my hand at org-mode and Markdown, compiled them to LaTeX, and then to PDF. In general, this worked well, but there always came a point where the abstraction broke, and the LaTeX leaked up the stack into my document. At which point I'd essentially write LaTeX anyway, just with a different syntax. After a few years of this, I decided to cut the middle-man, bite the bullet, and just write LaTeX.

That said, modern LaTeX is not so bad any more: XeLaTeX supports normal OpenType fonts, mixed languages, proper unicode, and natively renders to PDF. It also renders pretty quickly. My entire dissertation renders in less than three seconds, which is plenty fast enough for me.

To render, I run a simple makefile in an infinite loop that recompiles my PDF whenever the TeX source changes, giving live feedback while writing:

diss.pdf: diss.tex makefile $(graph_pdfs)
	xelatex -interaction nonstopmode diss.tex

We'll get back to $(graph_pdfs) in a second.

Graphs

A major challenge in writing a technical document is keeping all the source data in sync with the document. To make sure that all graphs are up to date, I plug them into the same makefile as above, but with a twist: All my graphs are created from Python scripts of the same name in the graphs directory.

But you don't want to simply execute all the scripts in graphs, as some of them might be shared dependencies that do not produce PDFs. So instead, I only execute scripts that start with a chapter number, which conveniently sorts them by chapter in the file manager, as well.

Thus all graphs render into the main PDF and update automatically, just like the main document:

graph_sources = $(shell find graphs -regex "graphs/[0-9]-.*\.py")
graph_pdfs = $(patsubst %.py,%.pdf,$(graph_sources))

graphs/%.pdf: graphs/%.py
	cd graphs; .venv/bin/python $(notdir $<)

The first two lines build a list of all graph scripts in the graphs directory, and their matching PDFs. The last two lines are a makefile recipy that compiles any graph script into a PDF, using the virtualenv in graphs/.venv/. How elegant these makefiles are, with recipe definitions independent of targets.

This system is surprisingly flexible, and absolutely trivial to debug. For example, I sometimes use those graph scripts as glorified shell scripts, for converting an SVG to PDF with Inkscape or some similar task. Or I compile some intermediate data before actually building the graph, and cache them for later use. Just make sure to set an appropriate exit code in the graph script, to signal to the makefile whether the graph was successfully created. An additional makefile target graphs: $(graph_pdfs) can also come in handy if you want ignore the TeX side of things for a bit.

Data

All of the graph scripts and TeX are of course backed by a Git repository. But my dissertation also contains a number of databases that are far too big for Git. Instead, I rely on git-annex to synchronize data across machines from a simple webdav host.

To set up a new writing environment from scratch, all I need is the following series of commands:

git clone git://mygitserver/dissertation.git dissertation
cd dissertation
git annex init
env WEBDAV_USERNAME=xxx WEBDAV_PASSWORD=yyy git annex enableremote mywebdavserver
git annex copy --from mywebdavserver
(cd graphs; pipenv install)
make all

This will download my graphs and text from mygitserver, download my databases from mywebdavserver, build my Python environment with pipenv, recreate all the graph PDFs, and compile the TeX. A process that can take a few hours, but is completely automated and reliable.

And that is truly the key part; The last thing you want to do while writing is being distracted by technical issues such as "where did I put that database again?", "didn't that graph show something different the other day?", or "I forgot to my database file at work and now I'm stuck at home during the pandemic and can't progress". Not that any of those would have ever happened to me, of course.

How I Record Screen Casts

Tue, 19 May 2020 16:33:00 +0200

This semester is weird. Instead of holding my "Applied Programming" lecture as I normally would, live-coding in front of the students and narrating my foibles, this time it all had to be done online, thanks to the ongoing pandemic. Which meant I had to record videos. I had no idea how to record videos. This is a writeup of what I did, in case I have to do more of it. You can see the results of my efforts in my Qt for Python video tutorials and my file parsing with Python video tutorials. Through some strange coincidences, wired.com wrote an article about my use of OBS.

Working on Linux, I used the Open Broadcaster Software, or OBS for short, as my recording program. OBS can do much more than record screencasts, but I only use it for two things: Recording a portion of my screen, and switching between different portions.

To this end, I divide my screen into four quadrants. The top left is OBS, for monitoring my recording and mic levels. The bottom left is a text editor with my speaker notes. The top right and bottom right are my two recording scenes, usually a terminal or browser in the top right, and a text editor in the bottom right. The screenshot shows the Editor scene, which has a filter applied to its source to record only the bottom right quadrant. On a 4K screen, each quadrant is exactly full HD.

In OBS's settings, I set hotkeys to switch scenes: I use F1 and F2 to select the Browser and Editor scenes, and F6 for starting and stopping recordings. For more compatible video files, I enable "Automatically remux to mp4" in OBS' advanced settings.

The second ingredient to my recording setup is KDE, where I assign F3 and F4 to activate the browser or editor window (right click any window → More Actions → Assign Window Shortcut). And to make my recordings look clean, I disable window shadows for the duration of the recording.

With these shortcuts, I hit F1 and F3 to switch focus and scene to the browser, or F2 and F4 for the text editor. To make this work smoothly, I disabled these shortcuts within my terminal, browser, and text editor. But always be weary of accidentally getting those out of sync. I don't know how often I accidentally recorded the wrong part of the screen and had to redo a recording.

Anyway, with this setup, I can record screen casts with very minimal effort. The last ingredient however is editing; and I loathe video editing. I'd much rather record a few more takes than spend the same time in a video editor. Instead, I record short snippets of a few minutes each, and simply concatenate them with FFmpeg:

Create a file concatenate.txt, that lists all the files to be concatenated:

file part-one.mp4
file part-two.mp4
file part-three.mp4

then run ffmpeg -f concat -i concatenate.txt -c copy output.mp4 to concatenate them into a new file output.mp4.

The great thing about this method is that it uses the copy codec, which does not re-encode the file. I.e. it only takes a fraction of a second, and does not degrade quality.

In summary, this setup works very well for me. It is simple and efficient, and does not require any video editing. The ability to switch scenes is cool and powerful. Still, recording videos is a lot of work. All in all, the 18 videos in the file parsing tutorials took 250 takes, according to my trash directory.

File Parsing with Python Video Tutorial

Thu, 14 May 2020 00:00:00 +0200

This video series was produced in the spring of 2020, during the COVID19-pandemic, when all lectures had to be held electronically, without physical attendance. It is a tutorial, in German, for parsing text files, and basic unit testing.

If the videos are too slow, feel free to speed them up by right-clicking, and adjusting play speed (Firefox only, as far as I know).

You may also download the videos and share them with your friends. Please do not upload them to social media or YouTube, but link to this website instead. If you want to modify them or create derivative works, please contact me.

The Qt for Python Video Tutorial by Bastian Bechtold is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License.

1 Intro

Prerequisites: A basic understanding of Python, and a working installation of python ≥3.6.

An overview over the topics discussed in the rest of the videos, and installation of pytest.

2 INI: First Steps

Basic setup, and our first test.

Code: inifile_1.py and inifile_test_1.py

3 INI: Sections

Parsing INI sections.

Code: inifile_2.py and inifile_test_2.py

4 INI: Variables

Parsing INI variable assignments.

Code: inifile_3.py and inifile_test_3.py

5 INI: Bugfixes and Integration Tests

Parsing difficult values, and comments.

Code: inifile_4.py and inifile_test_4.py

6 INI: Test the Tests

Tests can be wrong, too.

7 CSV: First Prototype

A simple parser for values without quotes.

Code: csvfile_1.py and csvfile_test_1.py

8 CSV: Quotes

Parsing quoted values makes everything harder.

Code: csvfile_2.py and csvfile_test_2.py

9 CSV: A Few More Features

Comments and a choice of separators.

Code: csvfile_3.py and csvfile_test_3.py

10 JSON: Keyword Parser

Parsing the simplest of JSON expressions.

Code: jsonfile_1.py and jsonfile_test_1.py

11 JSON: Strings

Parsing JSON strings is not as simple as it seems.

Code: jsonfile_2.py and jsonfile_test_2.py

12 JSON: Numbers

Numbers in JSON.

Code: jsonfile_3.py and jsonfile_test_3.py

13 JSON: Data Structures

The rest of JSON: Objects and Arrays.

Code: jsonfile_4.py and jsonfile_test_4.py

14 Regular Expressions 1

How to parse parts of INI files with regular expressions.

Code: inifile_regex.py

15 Regular Expressions 2

How to parse parts of JSON files with regular expressions.

Code: jsonfile_regex.py

16 Wrapup

A summary of the topics discussed.

RAW Developer Comparison

Fri, 01 May 2020 19:15:00 +0200

It bit me again: I got software envy. What if I could develop my pictures faster with a different RAW developer? What if they looked better than they do now? Questions like these keep me up at night.

The Problem

Choice. There are so many RAW developers out there. And they all have rabid fan bases, and apparently unique rendering. How to choose?

Here are my house rules:

Must run on Windows or Linux
Must run acceptably on my Surface tablet
Must run acceptably with files on a network share
Must support my past and present cameras (Fuji X-E3, Ricoh GR, Pentax Q7, Nikon D7000)

In contrast to most other comparisons on the 'net¹, I won't concern myself too much with sharpness and noise reduction and demosaicing. I have yet to see a photograph that was ruined by them, and most RAW developers seem to do a sufficient job at them.

I would prefer a file-based workflow with edits stored alongside the RAW files², and I would prefer a perpetual license instead of a rental contract, but I'm willing to compromize on both if it's worth it³.

Obviously, I am a lot more proficient in my current tool, Darktable, than in any of the others. But for this test, I'm explicitly not doing anything particularly artistic; merely some highlight recovery, shadow recovery, and local white balance adjustments. By limiting myself to these edits, I hope to get an unbiased idea of the various RAW developer's implementations, without needing to ask the endless “what if” of what else I could have done. That said, I will leave all other adjustments on their default settings, to still get an impression of the general look of the programs.

The Contestants

Darktable 3.0.2 (Free)
Free, works on Linux, very familiar to me. Allegedly, not particularly fast, with a confusing user interface.
Adobe Lightroom Classic CC 7.5 ($10/month)
Probably the most widespread tool. Somehow unappealing to me. Currently only available at a subscription price, but there is a “free” version available through my university.
Capture One 20.0.4 ($29/month or $350)
Enormously expensive, even with the educational discount. Allegedly the best default color rendition, particularly for Fuji cameras.
RawTherapee 5.8 (Free)
Free, works on Linux. Allegedly extremely high image quality, but no local adjustments whatsoever.
Luminar 4.2.0 ($90)
A rather new developer, with fancy AI features such as automatic sky replacements. Not exactly what I'm looking for, but we'll see about its “normal” RAW development chops.
ON1 Photo RAW 2020.1 ($100)
Another highly regarded developer with rather traditional tools. This one works straight on files in the file system, though, which is highly attractive to me.
ACDSee Photo Studio Ultimate 2020 13.0 ($9/month or $150)
Wasn't this a fancy image viewer a few years ago? Apparently it's now a RAW developer.
Exposure X5 5.2.1.211 ($120)
Allegedly super fast, with great sharpening and noise removal.
Photo Ninja 1.3.7a ($130)
Another new developer, borne out of a dedicated noise reduction tool, and with an emphasis on “intelligent” tools.
Silkypix Developer Studio Pro 10.0.3.0 ($200)
A Japanese RAW developer. I wasn't aware of it until a comment brought it up, thank you for that! And I mention its country of origin, as the English translation is a bit rough sometimes. Quite unusual in its feature set.
Zoner Photo Studio X 19.2103.2.317 ($50/year)
A somewhat unknown RAW developer. So much so that I only stumbled upon it one year after the original article. A true Windows-only product that feels at home in Windows like none of the other programs in this list.
DXO Photolab ($220)
For the longest time, did not support Fuji X-Trans files. But this has changed in version 5, and I am eager to try it out!

Not considered:

Aftershot ($80)
Works on Linux, but doesn't support my Fuji camera because it hasn't been updated in ages.
Iridient Developer, Raw Power, Aperture
Mac only
Affinity Photo ($55)
Cheap, awesome, but not non-destructive. I'll probably buy this regardless in a sale, just because it's so affordable.

In terms of price, it is hard to argue with free. But considering the price of my other photographic equipment, most prices in this list are pretty adequate. Except for Capture One (and maybe Silkypix). $350 ($200) is a hard price to swallow.

The Test

I prepared 18 RAW images for this test, and made a plan of what exactly I would do with every one of them. Then I developed them all in each of the RAW editors. I will only show excerpts here, both to keep private the pictures I don't want to share, and to keep this already long post from exploding.

Also, I mostly show difficult files here that have some obvious challenge. This is because I am usually quite satisfied with my cameras' JPEGs in easy cases, and don't bother with RAW development.

Highlight Recovery

📂 DSCF3861.RAF (23.0 MB)

A shot of the sunset in Greece, with both the sun and its reflection in the water blowing out. I want to lower the highlights, and boost the shadows a bit. The transition from sky to sun should be smooth without lightness reversals or rings. The transition from water to reflection should have no color cast. The hills in the background should not show any halos.

Capture One, Lightroom, and DxO show the smallest sun without artifacts. RawTherapee, Darktable, ACDSee, Silkypix and Zoner produce a smooth transition, but a bigger sun. In Luminar, Exposure, Photo Ninja, and ON1 the sun is smaller, but has a distinct ring around it that looks wrong. In RawTherapee the sun is big and slightly ringed. Actually, Capture One and Silkypix also have a ring, but so faint that it wouldn't matter to me.

The reflections in the water are artifact-free in Darktable, Lightroom, Exposure, RawTherapee, and Zoner. The other developers show magenta artifacts to varying degrees, and DxO has some false-color pixels (their X-Trans support is still in beta). In terms of detail, Capture One, Lightroom, DxO, and Exposure recover a bit more wave details in the blown-out reflections.

The hills in the background show distracting halos in Capture One, Lightroom, and Exposure.

In the following list, the RAW developer name links to the sidecar file, if there is one:

Camera: Dynamic Range 400
ACDSee: Highlights 100, Fill Light 25
Darktable: My Defaults, Filmic RGB to shift dynamic range to include highlights, Highlight Reconstruction LCh and lower until magenta halo disappears
Capture One: Highlight and White -100, Shadow +20
Exposure: Highlights -100, Whites -50, Shadows +50 (less Whites desaturates)
Lightroom: Highlight -100, Shadow +50
Luminar: Highlights -100, Whites -50, Shadows +25
ON1: Highlights -50, Shadows +25 (More Highlights produce lightness reversals)
Photo Ninja: Illumination 27, Exposure offset -1.62, Highlights -0.50 (all chosen automatically)
RawTherapee: Highlight Compression 250, Highlights 100, Shadows 25
Silkypix: Highlight Dynamic Range +3, Hue 100
Zoner: Lights -50, Shadows 25
DxO: Highlights -100, Shadows 25

While a bit of a pathological image, there are clear differences in how these RAW developers handle it. Really, only Darktable, DxO and Lightroom produce a truly pleasing image for me, with second place to Capture One, ON1, and Silkypix. Surprisingly, the camera's own JPEG is amongst the best renditions as well.

Silkypix deserves a special mention, though, as its highlight control tool has a fantastic Hue slider, which trades off higher saturation against more accurate hue. Which is exactly the tradeoff that underlies all the rings and magenta artifacts in all the other programs.

On a side note, I have never quite understood why nobody seems to complain about the obvious haloing in Lightroom. I see it in almost every high dynamic range landscape shot on the internet, and I do not enjoy the look. But apparently I'm alone with this.

Dynamic Range Reduction

📂 DSCF6535.RAF (21.6 MB)

A shot of a very contrasty forest scene at Mt. Washington, with highlights slightly blowing out, and shadows close to drowning. I want to lower highlights and raise shadows, without it looking crushed or unrealistic.

The most important thing in this picture is to maintain a realistic progression of tones, even though the dynamic range is crushed beyond reason. To my eyes, Lightroom really stands out here, with a three-dimensional look that no other developer can match. ACDSee, Darktable, ON1, Photo Ninja, and RawTherapee come second, with a believable progression. Exposure, Luminar, and Capture One seemingly applied some kind of local contrast compression that destroys the balance between highlights and shadows and flattens the image.

All developers show magenta artifacts on the bright forest floor to some degrees. They are particularly unpleasant in Capture One, Darktable, ACDSee, and Exposure.

ACDSee: Highlights 100, Fill Light 25
Capture One: Highlights -50, Shadows +25, Black +50
Darktable: My Defaults, Filmic RGB to expand dynamic range
Exposure: Highlights -100, Shadows +50, Blacks +25 (Blacks and Shadows interact weirdly)
Lightroom: Highlights -75, Shadows +50, Blacks +50
Luminar: Highlights -100, Whites -50, Shadows +50, Blacks +50
ON1: Highlights -75, Shadows +50 (disable Recover Highlight Hue to prevent color fringes)
Photo Ninja: Illumination 25, Exposure offset -1.47, Highlights -0.50 (all chosen automatically)
RawTherapee: Highlights 50, Shadows 25, Dynamic Range Compression 50
Silkypix: HDR 50, Exposure -2/3
Zoner: Lights -75, Shadows 25, HDR Lights 25%, HDR Shadows 10%
DxO: Highlights -75, Shadows 25, Blacks 10

In terms of tools, I like the explicit dynamic range slider in Darktable, RawTherapee, and Silkypix better than the shadows and highlights sliders in the other tools. But if calibrated well, both methods can result in a pleasing image.

To my eyes, Lightroom, DxO, RawTherapee, Photo Ninja, and Zoner take the crown in this shot. But I expect that the tone progression could be improved in the other tools as well if I strayed beyond the default tools.

Local White Balance

📂 DSCF8214.RAF (22.1 MB)

A shot of myself, underexposed, in front of Space Shuttle Enterprise. I want to brighten myself and adjust the white balance on my body so it matches the rest of the room. (I have better examples than this, but they showed people other than me, which I don't share.)

Photo Ninja, and RawTherapee fail this test, as they lack local adjustment tools. Exposure for some reason shows terrible color bleeding, where my arm's color is leaking out onto the Space Shuttle in the background. Truly noteworthy is ACDSee with its intelligent brush, much like the intelligent selection tools in pixel editors. Darktable als stands out for being able to combine a drawn mask with a luminosity and hue mask.

Capture One strangely did something terrible to my skin, with weird gradients where there should be none. The Shuttle in the background lost details in the highlights in ACDSee and Exposure and Zoner. DxO has various ways of drawing masks. In this case I used control points, but none of them worked particularly well. I expect that they would need some practice to use well. Silkypix by default insisted on crazy noise reduction that turned the picture into a watercolor. Thankfully that is easy to turn down.

ACDSee: Fill Light 50, Develop Brush with WB -50 (no picker)
Capture One: Shadows +50, Black +75, Drawn Layer with White Balance picker on Backpack
Darktable: My Defaults, Filmic RGB to shift dynamic range to include shadows, Luminosity and Painted mask with Color Balance picker
Exposure: Shadows +100, Blacks +25, Layer with Color Temperature lowered (no picker)
Lightroom: Shadows +100, Local Adjustment with WB -14 (no picker)
Luminar: Shadows +50, Local Adjustments with WB -46, Tint -6, Shadows 100
ON1: Shadows +50, Local Adjustment with WB -18 and Tint +4 (no picker)
Photo Ninja: Illumination 25, Exposure offset -1.61, Highlights -0.50 (all chosen automatically), Shadows +0.50, No local adjustments available
RawTherapee: Shadows 50, No local adjustments
Silkypix: Dodge HDR 50, Noise Reduction Smoothness 25, Partial Correction with Hue 130, Saturation 0.37
Zoner: Shadows 50, Black 10, Local Adjustments with Shadows 30, Saturation -10, and Blue/Green pushed in Tone Curve (no picker)
DxO: Shadows 25, Black 25, Local Adjustments with lowered WB, Shadows 25

I find local color adjustments my main use for localized edits. Having a color picker for that is very useful, but only available in Capture One, Darktable, and Luminar. In the other tools, I had to either eyeball it, or manually adjust tones until the RGB values read grey.

Thus, it is Lightroom, ON1, and Darktable that pass this test.

Out of Gamut Colors

📂 DSCF0034.RAF (15.7 MB)

A shot of the Congress building in Leipzig, with a bright purple light that blows out the red color channel, which is wildly out of gamut of any reasonable color space. I want to see how the RAW developers deal with out-of-gamut colors. I raise Exposure by 1 EV, then push shadows until the clouds become faintly visible.

ACDSee, ON1, Photo Ninja, and RawTherapee fail this task, with obvious magenta or blue artifacts on the illuminated water jet. The other developers use various methods of inpainting, which look particularly convincing in Capture One, Lightroom, Silkypix, Luminar, DxO, and Zoner. Exposure and Darktable look less realistic, but acceptable in a pinch. Again, Silkypix' hue slider is very handy.

ACDSee: Fill Light 50, Exposure +1
Capture One: Black +75, Exposure +1
Darktable: My Defaults, Filmic RGB
Exposure: Blacks +50, Exposure +1
Lightroom: Shadows +100, Exposure +1
Luminar: Shadows +25, Exposure +1
ON1: Shadows +50, Exposure +1
Photo Ninja: Illumination 9, Highlights -0.50 (all chosen automatically), Exposure offset 0.0
RawTherapee: Shadows 50, Exposure +1 (Highlight Reconstruction: Blend)
Silkypix: Dodge HDR 100, Noise reduction Smoothness 25, Highlight Hue 100
Zoner: Exposure 1.0, HDR Shadows 25%
DxO: Exposure 1.0, Shadows 75, Midtones -100

I know the Darktable devs are actively working on improving this. In truth, Darktable would have failed this task just a few months ago. Issues like these also often happen with deep-blue flowers, which turn purple in the failing developers but maintain hue in the better ones.

Color Rendition and Detail

📂 DSCF9670.RAF (25.7 MB)

A shot of a field and forest. I want to see how the RAW developers render these details and colors. Zero out noise reduction, use default sharpening, JPEG 100%.

In terms of detail, Lightroom, Capture One, Exposure, Silkypix, Zoner, DxO, and Darktable seem to retain the most fine details, particularly in the little trees and the forest floor. ACDSee, Luminar, RawTherapee, Photo Ninja, and ON1 look comparatively soft or lose detail in the shadows. Silkypix, however, has a strange, painterly look to the grass details that I wasn't able to get rid of.

In terms of overall color, Exposure, Photo Ninja, RawTherapee, and Capture One clearly tend towards the most saturated look, with a clear distinction between a green and a yellow part in the field. I suspect that these try to approximate the punchy look of Fuji's colors. These color transitions are much more subtle in ACDSee, Darktable, Exposure, Lightroom, Luminar, Silkypix, Zoner, DxO, and ON1. The sky is distinctly blue in Darktable, Photo Ninja, DxO, and Luminar, more cyan in ACDSee, Lightroom, and Silkypix, and weirdly purple in Capture One and RawTherapee.

ACDSee: Amount 25
Capture One: Amount 140
Darktable: My Defaults, Sharpen 2
Exposure: Amount 50
Lightroom: Amount 40
Luminar: Details Enhancer, Sharpen 50
ON1: Sharpening 50
Photo Ninja: Sharpening strength 50
RawTherapee: Sharpening 20
Silkypix: Zero Noise reduction, Outline emphasis 30, Ringing artifact control 15 (defaults)
Zoner: Sharpening strength 150 (Smart Sharpen)
DxO: Lens Sharpness Global 1.0

I would not put too much emphasis on the colors, saturation, and contrast here, as these are easily and typically adjusted manually. I am a bit surprised about the differences in detail retention, however.

The Result

I went into this expecting to find Lightroom and Capture One to be vastly faster in use than Darktable, particularly on my Surface tablet. I also expected better out-of-the-box image beauty, large differences in user interfaces, and for most tools to have very few graphical artifacts. Surprisingly, however, almost every tool showed obvious artifacts of one kind or another, and few tools were actually faster than Darktable. In terms of tools, I found most tools look very similar, yet function vastly differently.

Simple saturation and contrast adjustments, a bit of local contrast, and rarely some dodging and burning or local color adjustments are apparently all I do most of the time, and this generally works well and similarly in all of these tools. However, that is not to say that the individual sliders do remotely the same thing in different tools. So confused was I by this that I measured the response curves of several tools, and they indeed did entirely different things. In one tool, Highlights pushes the upper half of the tone curve. In another, even the darkest shadows are affected a little bit. In yet another, Highlights burns out to the upper quarter of the tone curve if pushed all the way. Sometimes the white point stays white, sometimes it moves. Sometimes it only moves if the slider is pushed past half-way. And that's not even taking into account their different blending behavior and value scales; these sliders may look the same, but there hides complexity beyond measure.

And I did find a surprising amount of graphical artifacts in these programs, particularly the color bleeding in Exposure, and the highlight recovery problems in Luminar and ON1, as well as a number of smaller issues. The one program that truly stands out here is Lightroom, which is more robust to artifacts than any other tool in this list, seemingly due to some significant image-adaptive intelligence under the hood.

I have strong mixed feelings about Capture One. On the one hand, it has one of the most attractive user interfaces of all these tools. On the other, its color renditions are very opinionated, and not my favorite. I love how it reads and applies Fuji color profiles as shot, but then it doesn't apply the Fuji shadow/highlight adjustments and crushes the shadows unnecessarily. And while its color tools sure look nice, their functionality is not that much different than the other developers', and they are spread out needlessly across several tabs. And that price.

Playing around with Luminar was deeply impressive. There are a ton of magic and automatic features in there. But as cool as AI sky replacement is, it simply has no place in my toolbelt, and the general speed of the UI are a big minus.

I like ON1. It's relatively affordable, works with simple JSON sidecar files instead of a library, has reasonable tools, and impressive effects. It can even mimic the look of the embedded JPEG and supports Fuji film simulations. Not quite on the graphical level as Capture One or Lightroom, but very close. And it even runs acceptably fast on my Surface tablet.

Exposure is another program I could like a lot, but the color bleeding and graphical artifacts are just not up to snuff. In one example, it entirely failed to guess colors from an underexposed bar scene (not shown). In another it bled colors out onto adjacent objects for no reason. And white balance sometimes changed lightness as well as colors. I read that this might be a graphics driver issue, but regardless, it shook my confidence in Exposure.

ACDSee was a real surprise to me. I seem to remember it as a fast image viewer, but apparently it is (now?) an impressive RAW editor as well. There is a lot to like about this tool. The magic brush for local adjustments is a particularly noteworthy touch, as well as very robust healing tools. Alas, I found the UI rather slow, and it failed on exporting a few files. I'll try again in a year or so.

I was only made aware of Silkypix through a comment after the post had already been published (thank you!). And what I read on the website made me quite excited! Its tools stray somewhat from the Lightroom-inspired norm, which is a very good thing in some cases, such as the hue-priority highlight recovery. It also works on plain files, and seems to have outstanding Fuji film simulations. Alas, it was very slow to use, and not suitable for my Surface tablet.

I had tried RawTherapee a few times in the past, and was always frustrated by its lack of local adjustments, and the need to view things at 100% to see some adjustments. On the other hand, it can match the embedded JPEG tones, and has quite a number of impressive algorithms. Still, it does not appeal to me. But it's still an amazing achievement and a pretty inspiring community as well.

Photo Ninja is a curious program. Certainly not because of its ease of use, or speed of operation, or image quality. But because it did most things almost correctly automatically. That's not what I am looking for, but it is truly impressive.

I only discovered Zoner Photo about a year after this post was originally published. But I was impressed by what I found: It is a well-designed program with a solid array of tools, even including a pixel editor. And rather affordable, too. Curiously, Zoner Photo does not officially support some of my cameras, but can utilize the free Adobe DNG converter to make them readable on the fly, without creating a separate copy. Smart!

DxO PhotoLabs was another latecomer, which only started supporting Fuji X-Trans files in late 2021. I downloaded it on the day it became available, and was duly impressed. Its tools are less streamlined than in other programs, but seem to be of great algorithmic quality; there was hardly a lightness reversal or halo to be found, the bane of many a cheaper product. I am deeply intrigued.

And Lightroom. As I said, I somehow do not like Lightroom. Maybe because I like to be “different”, or because I associate Adobe too much with bloated software. But I have to say, Lightroom surprised me. While its tools are sometimes in weird locations, it is highly streamlined for a very fast workflow, and it deserves my highest praise for being outstandingly robust against artifacts. But I still don't like it.

Which leaves Darktable. This is a tool I am deeply familiar with, and have used for several years. Yet until this day, I never realized just how strange its tools are compared to the other programs. How weird Filmic RGB must feel if you are used to shadows and highlights sliders, and how alien the graph-based color zones and tone equalizer and contrast equalizer must seem.

Yet, in direct comparison, I find Darktable's tools equally efficient at solving problems, even if the solutions are sometimes a bit different from its Lightroom-inspired peers. One tool in particular I want to emphasize: Color Zones. At first glance, it looks like your standard HSL tool that allows brightness, saturation, and hue changes by color (albeit as a graph instead of sliders). But then you discover the “select-by” switch, and realize that you can modify colors by lightness and saturation, as well as hue. I use this frequently to saturate shadows, which is a great effect I haven't seen in any other program.

No doubt other programs have cool features, too, but Darktable (and RawTherapee) seem uniquely open about their inner workings. And this brings joy to me, on a level the closed, artistic programs can't match. I like graphs, and maths. I'm weird like that.

But what really prompted this whole comparison blog post was my frustration with Darktable's speed. Particularly on my 4K screen, it is not the fastest program out there. And the AMD/OpenCL situation on Linux is still a travesty, which doesn't help. But I learned a thing during this experiment: You must work bottom-up through the rendering chain, if you want Darktable to be fast⁴. Which, in my case, usually means working through Lens Correction → Crop and Rotate → Exposure → Tone Equalizer → Contrast Equalizer → Color Balance → Filmic RGB → Color Zones. As long as I (mostly) edit things in this order, Darktable is fast enough, even on my Surface tablet.

Lastly, I have to say a few words about file management. Most programs here work on some kind of local library that stores all edits. The downside of this is that these libraries are hard to sync between computers, are hard to back up, and need to manually be kept in sync when file locations change. Notable exceptions here are Lightroom, Darktable, Exposure⁵, Silkypix⁵, RawTherapee, ON1, DxO, and Zoner, which keep their edits alongside the RAW files in little text “sidecar” files⁶. Thus even if their library goes out of sync or is lost, at least the edits are still there. To be honest, this is quite an important factor for me.

As for library management, my requirements are small: I want to filter by date, rating, and maybe camera or lens. These needs are met by all programs except possibly Luminar and Photo Ninja. I do most of my file management in external programs on camera import, and with the exported JPEGs, so this area of the RAW developers is not very important to me.

However, this comparison also highlighted just how useful sidecar files are to photo management. I might choose different RAW developers over the years, and my photo management solutions might change over time as well. But as long as all edits are stored in simple text files next to the RAW file, I can rest safe in the knowledge that my edits will never be lost. This is a serious downside to ACDSee, Capture One, Luminar, and Photo Ninja, who keep edits secretly⁷ in their opaque databases.

I also timed my work with every one of these programs. Quite surprisingly, I couldn't find any significant differences between programs. Darktable's workflows, for example, are sometimes entirely different from other tools; but if you know what you're doing the path from identifying a problem to fixing it is still similarly straight-forward and fast. And there is no less experimentation until I arrive at a look I like.

Thus, I am left with Darktable, Lightroom, and ON1. And theoretically Capture One, but that price is just too high for me. If Capture One were $100 instead of $350, I would probably switch to it. Even educational pricing is only available for rentals. I'll have to decline that. And despite all my praise for Lightroom, I still don't like it.

I'll probably buy ON1 at its current, discounted price ($50), and see how I like it in actual daily use⁸. But at the same time. I'll also stick with Darktable on my Linux machine, at least for more complicated edits. I now know that Darktable can dance with the best of them, which is mighty impressive for a piece of free software.

With all that said and done, I have learned a lot about RAW development during this experiment. Regardless of which tool I end up sticking with, this has been a fascinating comparison. We'll see how long I can resist the urge to compare this time.

Addendum: Customer Support

So far, I have been in contact with the Customer Support people of ON1, Capture One, and Zoner. And I must say, ON1 and Zoner were incredibly pleasant and quick and helpful, while Capture One seemed almost reluctant to help. This is actually a big plus for ON1 and Zoner, and kind of big turnoff for Capture One.

Update: Silkypix

I added Silkypix to the comparison, thanks to kind comments underneath this post.

Update: Luminar Local Adjustments

I was made aware that Luminar does indeed have local adjustments, they were merely in such an uncommon place that I hadn't found the initially.

Update: Zoner Photo

One year after the original post, I added Zoner Photo to the comparison.

Update: DxO PhotoLab

One-and-a-half years after the original post, DxO Photo Labs added support for Fuji files, so I added them to the comparison.

Footnotes:

There are surprisingly few non-superficial comparisons; most are just feature matrix comparisons. The best ones I could find are a fantastic, in-depth comparison on nomadlens, A Fuji-centric discussion of detail extraction on Fuji vs. Fuji, one by Andy Bell that might be sponsored/biased by Luminar, a pretty good one on PetaPixel, an older one on DPReview, and a Nikon-centered one on WY Pictures.

Makes it much easier to share edits between computers, and back them up

I don't like rental software, as it locks away all past edits once I stop paying.

⁴

because lower stages in the pipeline are cached, but higher stages need to be recalculated after every edit.

⁵

in an unnecessary subdirectory

⁶

Capture One saves sidecar files, too, but only for metadata, not edits

⁷

They would say, "securely"

⁸

not really daily, but you know what I mean

The Value Proposition of Open Source Software

Fri, 27 Mar 2020 08:17:00 +0100

To quote Wikipedia:

Open Source Open-source software (OSS) is a type of computer software in which source code is released under a license in which the copyright holder grants users the rights to study, change, and distribute the software to anyone and for any purpose.

In practice, this generally means software developed by hobbyists in their free time, as opposed to professionals at a company.

But why should such software be preferable to commercial products? I shall ignore price, for the moment. While open source software generally does not cost money, I have no qualms about paying money for software, and can easily afford to, as well. So if it's not price, what then?

Richard Stallman makes an argument that it's all about Freedom. But I have a suspicion that he really wants the code to be free, not its users¹. He argues that "free" licenses make the program's source code available to users. Presumably, to read it and change it. However, I don't do that, generally. And neither do I redistribute software, which is another "freedom" granted by Stallman-style free software. Also, you don't need access to the source code to change a software. But I still prefer Open Source Software to commercial software, in most cases.

And the reason is, that in my experience, Open Source Software is generally better software. And I believe the reason for this is incentives:

Open Source Software is generally created to scratch an itch. One single developer was sufficiently disgruntled to throw all caution into the wind, and solve the problem him/herself. Which means that at least one person was dissatisfied with the commercial offerings at the time.

And there are plenty of reasons to be dissatisfied with commercial software. Commercial software needs to make money. And to justify recurring sales, it must keep adding features, often beyond the usefulness to users. Thus, big changes and new features are incentivized, while continued refinement and bug fixing are merely cost centers.

Contrast this with Open Source Software, which is quite content to simply work. No new features need to be added if the software is complete as it is, and no new effort needs to be invested if there aren't any bugs remaining. Such software is a joy to use.

In the commercial world, such software is not stable, but stagnant. A sign of death, not maturity. And when commercial software dies, it is buried, never to be used again. And all the work its users have created with it over the years, becomes inaccessible and obsolete with it. In the same way, today's online-only subscription-based software distributions are essentially protection rackets that require a monthly ransom just to keep access to your existing work. The moment you stop paying, all your previous work becomes inaccessible. Such moves are indeed Stallman-worthy invasions of freedom that are unacceptable to me².

Of course, Open Source Software allows users to contribute to the product. But In my own projects, I have found this to vary greatly between cultures. Some communities seem to be naturally tinker-friendly, such as Emacs or Darktable. Others, such as Python's or Matlab's, are strangely reluctant to help one another, with more users creating bug reports than contributors creating pull requests.

Funny enough, price does not enter this equation. Because what you need to use Open Source Software is often not money, but time. And only rich people have that. If you're poor, and paying for software is a problem, you're far more likely to pirate "professional" software with a known value proposition, than to spend time on an unknown quantity. Open Source Software is a toy for the rich, and paradoxically unattractive to people in need³.

So in summary, I have generally found Open Source Software to be better than commercial software in most circumstances. Perhaps as a next step, we should try to figure out how to get payed for creating it, and how to keep monetary incentives from ruining products.

Footnotes:

See this link for an exhaustive discussion of that topic. It's an amazing article!

Except for video games, which I only consume once, and which result in experiences, not artifacts. Strange, how this distinction is surprisingly hard to justify.

User-facing software, that is. Linux web servers are for everyone

Qt for Python Video Tutorial

Fri, 20 Mar 2020 00:00:00 +0100

If the videos are too slow, feel free to speed them up by right-clicking, and adjusting play speed (Firefox only, as far as I know).

The Qt for Python Video Tutorial by Bastian Bechtold is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License.

Update: As of April 2022, all code examples have been updated to use PySide6. In particular, this changes the imports, replaces app.exec_() with app.exec(), and replaces mouseEvent.pos() with mouseEvent.position().toPoint() (see note in map11.py:).

1 Intro

Prerequisites: A basic understanding of Python, and a working installation of python ≥3.4.

An overview over the topics discussed in the rest of the videos, and installation of Qt for Python and Pandas.

2 Hello World

Our first GUI program, a window with a text label.

Code: map2.py

3 Main Window

Create a QMainWindow, and build some structure for later episodes.

Code: map3.py

4 Layouts

Position a QLabel and a QPushButton side by side, using layouts.

Code: map4.py

5 Signals and Slots

Make the button change the label's text if clicked.

Code: map5.py

6 Loading Data

Load the data required to draw a map.

Code: map6.py

Data: countries_110m.json

7 Drawing the Map

Draw a world map into a QGraphicsScene.

Code: map7.py

8 Pens and Brushes

Make the map pretty, using QPens and QBrushes.

Code: map8.py

9 Resize Event

Resize the map when the window size changes, by overloading resizeEvent.

Code: map9.py

10 Mouse Tracking

Highlight the country under the mouse.

Code: map10.py

11 Custom Signal

Respond to clicks of a country.

Code: map11.py

12 Addendum

Improve the code by cutting out a middle man.

Code: map12.py

13 Pandas

A quick introduction to Pandas.

Data: covid19.csv

14 Pandas Integration

Load the COVID19 dataset and print some stats.

Code: map14.py

15 Model View Tables

Display the COVID19 dataset in a QTableView.

Code: map15.py

16 Table Header Data

Fill in the table headers from the dataset.

Code: map16.py

17 Country Selection

Show only a subset of the dataset when a country is clicked.

Code: map17.py

18 Cleanup

Summary, and a few finishing touches.

Code: map18.py

Books of 2019

Wed, 11 Mar 2020 17:08:00 +0100

Sapiens

More than any other book this year, Sapiens changed my view of the world. Or at least that's what it felt like while reading. It seemed as if every page contained a new and mind-blowing shift in perspective, such as casting money, democracy, and capitalism as "religions", and questioning the value of civilization in terms of individual human experience. My mind was reeling from the implications of these viewpoints, in a very good way.

On the other hand, my memory of the book is marred by the author's second book, Homo Deus, which highlights just how little the author's knowledge of history is worth when extrapolating into the future. It is, essentially, boring and shallow science fiction, with seemingly no knowledge decades of previous writing on the topic.

Come to think about it, this might reveal more about me than the author, for my own expertise is clearly more in science fiction than history; I might simply be much easier to please in the latter than in the former.

The Boat Who Wouldn't Float

This is the partly-fictional account of the author, Mowat Farley, and his boat. That leeks. And stinks. And doesn't sail well. And that is still entirely endearing.

Or maybe it is really about the humor with which the author recounts his misadventures on the Canadian shore. I stuck with this book even though my terrible digital library loaner had whole corrupted pages every chapter or two. But I could neither put it down, nor stop laughing. This might be the funniest book I have read in a decade. I must read more Mowat Farley!

Ninefox Gambit / Raven Stratagem / Revenant Gun

I did not like the first book in this series. It had a certain spark, but there was so much jargon and confusion, I just couldn't keep up. Yet. I kept coming back to it. That spark.

And in 2019, I bit the bullet, and picked up the sequel. And as with many things in life, it was better the second time around. Even though I still didn't quite get this particular sciency-fantasy-space-opera thing; I now felt at home in it, and could enjoy the story unraveling before me. At which point I couldn't put it down any longer. The two weeks I had to wait after finishing the second book before the third one released were harrowing. A masterpiece!

The Murderbot Diaries

Sometimes I don't want to think. Sometimes I like a simple bit of escapism, like chocolate. Sometimes, I need to read about the Murderbot.

I love how the Murderbot Diaries cast humanity as uncaring and somewhat dim animals, with the most humane characters invariably being artificial intelligences. And the character of Murderbot itself is just so plain delightful, how it simultaneously wants to get away from, yet deeply cares for, us silly, childish humans.

Brains vs. Bullets

This is an odd one. It's about soldiers, and how they fight. Or rather, what keeps them from fighting. Because every one has a breaking point, and soldiers are often quite close to it when they fight.

And as it turns out, humans are by and large pretty bad at killing each other. Even professional soldiers apparently are often unable to do so, despite years of training and propaganda. Which is a heartening message indeed, for a book on warfare.

Editing Fuji Photos on Linux

Sun, 09 Feb 2020 12:54:00 +0100

Darktable is my favorite RAW editor. It's a program for developing digital negatives ("RAW files") to JPEGs. But, I've long struggled with matching the quality of the out-of-camera JPEGs of my Fuji camera. Let me explain.

Today's cameras capture an astounding amount of detail, far more than monitors can display or printers can print. And then they crush it down to a printable and viewable JPEG file. But that crushing operation is idiomatic for each camera, irreversible, and not always appropriate to the image. And at those inappropriate times, when you want a wider dynamic range or different colors, a RAW editor like Darktable can take a digital sensor dump ("RAW file"), and render it differently. The challenge is that the camera's own JPEGs are already very, very good, and it's a fine line to walk between fixing a particular flaw with the image, while retaining as much detail as possible.

Fuji cameras produce notoriously difficult RAW files, as their internal processing is (said to be) particularly elaborate, and they use an unusual image sensor. This post is about matching the quality of Fuji's JPEGs in Darktable, while maintaining the fidelity and malleability of the RAW files.

Here are a few renditions of a photograph I took, the first created with my revised process, the second is Fuji's JPEG, and the third is Darktable's default rendition. First the entire picture, the a zoomed view so you can see each pixel:

While contrast and saturation don't match 100% in my version and Fuji's, the colors are close enough for further processing. Darktable's default version does not match those colors at all. The zoomed-in version highlights even worse problems with sharpness and detail retention. After working with Fuji files in Darktable for about a year, and editing about 3500 Fuji files, I finally found a workflow that reliably produces results on par with Fuji's JPEGs:

Use the Iridient X-Transformer to convert Fuji's RAF RAW-files to DNG RAW-files, and have the X-Transformer do the "More Detailed" demosaicing, and the full lens correction.
Use Stuart Sowerby's LUTs instead of Darktable's Base Curve or Filmic RGB.

My first gripe with Darktable's rendering of Fuji files is that the demosaicing and lens corrections are not particularly great. Pictures simply come out softer than with other tools, and chromatic aberrations remain an issue. The Iridient X-Transformer completely fixes this issue for me. The X-Transformer can also do sharpening and denoising, but I find those are better relegated to Darktable where needed. "But… that's proprietary Windows software!!!". Yes, it is. And it works beautifully in Wine.

My second issue is colors. Darktable needs a lot of massaging to produce Fuji-like colors. The Base Curve module (shown above) really does not do Fuji files justice most of the time. And while Filmic RGB is much better, it requires a lot of tedious adjustments even in simple cases. So instead, I use LUTs extracted with a color chart and some fancy math to replicate Fuji's colors. At the moment (v3.0), Darktable understands only .cube and .png LUTs. But Stuart Sowerby's website has them in .3dl. So for now, you need to install a 3D LUT converter, and convert them to .cube manually. On Windows.

With these two steps you can develop your Fuji RAFs better than your camera, and with relatively little fuss. I'll leave you with a finished render of the image above:

I Miss My Mac

Tue, 21 Jan 2020 16:00:00 +0100

There was a time, maybe from 2009 to 2011, when Apple computers were glorious. They were elegant devices, beautiful to me, and powerful. Never before, or since, have I loved a machine as I loved that MacBook Pro early 2011.

I miss that, being able to love the elegance and power a machine.

Nowadays I mostly use KDE Linux, which I can tolerate. Things occasionally break, stuff sometimes doesn't work as intended, but I can generally work around these quirks, so they don't bother me too much. Today, off the top of my head, Firefox crashed, Thunderbird couldn't display one folder, Zotero had to be re-installed, one monitor had to be power-cycled when waking the computer, and the laptop battery died way too quickly again.

I also use a Windows Surface tablet, which is a wonderful little device. It currently shows an error message every time it boots that I can't get rid of, has to be rebooted every few days to stay fast, and sometimes just won't react until restarted. And it always wants to install some update, and just refuses to do it overnight as intended. But generally it works pretty well.

And yesterday, in a bout of nostalgia, I opened my macOS laptop again and endeavored to write a bit on it. There were no updates, then there were updates but they couldn't be installed, then they could be installed, then the download stalled, and hat to be un-paused for some reason. Then my project files randomly couldn't be downloaded to the laptop, and the window management in macOS is a horrid mess. Also the update took a full hour, and didn't change anything noteworthy. I didn't end up actually getting to write on it.

In the evening, my camera corrupted its file system, as it sometimes does if I forget to format the SD card as FAT instead of its default exFAT. The recovery process was tedious but worked. I've done it a few times now and I know what to do.

My phone, meanwhile, can't use its fingerprint reader any longer ever since I "up"graded to Android 10. It's a Pixel 2, as google as Google can be. And apps are sometimes unresponsive and have to be restarted to become usable again. And it still can't undo text edits in the year 2020, and probably spies on my every move.

My amplifier nominally supports Bluetooth, but it only works some of the time. Currently it doesn't, so we use a portable Bluetooth speaker instead. Its Spotify integration has never worked even once. But at least it now reliably sends an image to the projector ever since we vowed to only ever touch the one reliable HDMI port on the front of it.

And as much as I love my ebook reader, it sometimes crashes when there's too much stuff on the page. One time I even had to completely reset it to get it to work again. And every time I install an update, the visual layout changes. Most of the time, mostly for the better. But it just erodes trust so much, to be at the mercy of the whims of an obviously demented software designer somewhere.

As I said, I miss my Mac, back when technology was magical, and just worked. And any error could be clearly explained as user error instead of terrible programming. When I was young and foolish, in other words.

I should go play with my child now, and run around in the yard, or maybe read a paper book. There's no joy in technology any more.

Efficiency is not Intuitive

Sun, 29 Dec 2019 12:06:00 +0100

A few days ago, a new version of Darktable was released. Darktable is an Open Source RAW image processor, and one of my favorite pieces of software of all time. It may be THE most powerful RAW processor on the market, far exceeding its proprietary brothers. But in the social media discussions following the release, it got dismissed for being unintuitive.

Which is silly, ironic, sad; why should Intuition be a measure of quality for professional software? Because in contrast to a transient app on your phone, RAW processing is a complex task that requires months of study to do well. Like most nontrivial things, it is fundamentally not an intuitive process. In fact, most of my favorite tools are like this: Efficient, but not Intuitive; like Darktable, Scientific Python, the Unix Command Line, Emacs.

These tools empower me to do my job quickly and directly. They provide sharp tools for intelligent users, which can be composed easily and infinitely. Their power derives from clean metaphors and clear semantics, instead of "intuitive", "smart", magics. They require learning to use well, but reward that effort with efficiency. They are professional, and treat their users as adults.

In my career as a software developer, I know these ideas as traits of good APIs: Orthogonal, minimal, composable. APIs are great not when they are as expansive and convenient as possible, but when they are most constrained and recombinable. In other words, "don't optimize for stupid". That's what I understand as good design in both programming, and in professional UIs.

Which is quite the opposite of Intuitive. Quite the opposite of "easy to grasp without reading the manual". That's a description of a video game, not a professional tool. As professionals, we should seek and build empowering, efficient tools for adults, not intuitive games maskerading as professional software.

Travel Cameras

Sun, 27 Oct 2019 20:22:00 +0100

We went on a once-in-a-lifetime vacation this year. Six weeks in the USA, roving New England in an RV, and going on a helicopter tour around New York City. I have spent months thinking about the perfect camera for going on this trip. I went on several vacations with various permutations of cameras and lenses as a test-run. I data-mined my existing photographs to find the perfect setup through science. And I scoured the internet for untold hours to get it right.

Here's what I took with me:

Figure 1: A Fuji X-E3 with an XF 18-135 lens, and a Ricoh GR

That's two cameras! Most people would take ~~their smartphone~~ one camera with multiple lenses. I take multiple cameras with one lens each. Why? Because they're both awesome, at different things:

The Fujifilm X-E3 with an XF 18-135 f/3.5-5.6 lens

The X-E3 is a mirrorless camera, which means it's small. It is a very recent camera with a modern sensor. It's a rangefinder-style camera with the viewfinder in the top left. So it doesn't obscure my whole face when I lift it to my eye.

Compared to a DSLR, this is a small and light camera¹. But paired with the lens, it is neither. That lens, though: a massive focal range from 18 mm to 135 mm is suitable for wide landscapes, as well as tight tele shots of wildlife. Cramming it all in one lens must come with a loss in sharpness and aperture, but the 18-135 is a best of its breed, and still plenty sharp enough for me. And it comes with an image stabilization so sophisticated that I can hand-hold half-second exposures. Which is magic.

Figure 2: The X-E3 with the lens extended. Small, it is not.

This is a great camera for deliberately framing a shot; it feels great to hold, with a neat and efficient control layout for quickly playing with apertures and shutter speeds and focal lengths with a minimum amount of fuss. It is a dependable tool, an amazing piece of engineering, and an artistic means of expression. It is truly a great camera.

If it has an Achille's heel, it's that it takes a while to get ready. Not because it is slow to start up, but because there's a delay before the viewfinder activates when raised to the face. Which just means you will stare into the void for half a second, while seeing the critical moment pass in front of you. Half a second of missing the shot.

And the white balance can struggle a bit with forests and lakes. Forests are green, and to compensate, the X-E3 turns everything else magenta. Easy to fix in post, but annoying. These are minor complaints, and most of the time the X-E3 is as perfect a camera as I've ever seen one.

Here are a few pictures I took with the X-E3 on my last vacation²:

Image quality is simply excellent. Not once has the X-E3 let me down. Autofocus is accurate, and instant. The lens is amazingly sharp all through its focal range. Dynamic range is unreal, at times. As you can see in the pictures, I've shot everything from landscapes to people, to wildlife, long exposures, and even some astro. All with one lens, on one camera.

The Ricoh GR

So, if the X-E3 is my "professional" camera, the Ricoh GR is my fun camera. In 2019, the GR old technology. It was released in 2013, it has a big but aging 16 megapixel APS-C sensor, and a fixed 18 mm f/2.8 lens with no zoom. It shouldn't be able to compete with the Fuji. But there is one thing the GR does better than any other camera: Speed.

Figure 3: The Ricoh GR in all its glory (before the vacation)

The GR is in my pants pocket. It wouldn't be my pants without a GR in it. I have it with me everywhere, all the time. And at the right moment, I will pull it out, and take a picture, before you even notice it's there. That's the GR's mission: The fastest possible time from pants to picture!

Seriously, the GR goes from off to taking the first shot before even its screen turns on³. I can whip out my GR and take a picture long before you have unlocked your phone, never mind started the camera app, selected a focus point, or aimed it at your prey. In the car, I had the GR in the center console, and could take a picture of the roadside before it whizzed past. And without taking my eyes off the road, naturally.

Since the GR has a fixed and wide focal length, I don't have to look at its screen to see what I'm taking a picture of. I can do it by feel. Because all the controls are on the right, I can adjust everything with my right hand only, without taking my hands off the wheel, or stroller, or wife. And then there's snap focus, where a full-press of the shutter will take a shot at a pre-defined distance, without waiting for autofocus. I wish every camera had this feature!

However, all of this took some getting used to at first. I missed focus a lot in the beginning. But once I figured it out, a new world opened up. A world of whimsy and spontaneity that is simply not possible with a bigger or slower camera. A good half of my favorite pictures were only possible thanks to the GR.

Figure 4: The Ricoh GR after the vacation: scarred, but stronger for it.

Again, there are downsides. It's an old sensor. Dynamic range is limited. Autofocus could be faster. There's a noticeable shutter delay. There is no viewfinder or image stabilization. The white balance is not great. It can struggle with a full memory card⁴.

But none of that matters. I love its colors. I love its sharpness. It has taken quite the beating on this vacation, but I wouldn't have it any other way. The GR is magnificent. There is nothing else quite like it⁵. And so many memories would have been lost if not for the GR:

Many of these shots were taken in the spur of the moment. In situations where I didn't take the big camera with me. Where a big camera would have been inappropriate. But the GR is so tiny, so non-threatening, it can become part my daily life. And capture the fleeting moments that make life worth living. That is a photographic super power if there ever was one.

Footnotes:

The body weighs 337g, which is a feather. The lens weighs 490g.

No personal pictures, though. I don't share those publicly.

The magic is "snap focus". If you depress the shutter fully, without waiting for autofocus, the GR immediately takes the shot at a pre-set distance. Works in any mode, and even when just powering on: Press the power button, aim, press the shutter, and before even the screen turns on you will have taken a picture. Lightning fast.

⁴

As the card fills up, getting into playback mode takes longer and longer. Minutes, for a full 64 Gb card.

⁵

Forget about the Fuji XF10. I tried it. It's a newer sensor, but so clumsy, so slow, so awkward to use. It's no comparison.

Analyzing Speech Signals in Time and Frequency

Fri, 20 Sep 2019 09:40:00 +0200

Our first stop in speech analysis is usually the short-time Fourier transform, or STFT¹:

Figure 1: Top: an STFT of speech (it's me, saying: "Es war einmal ein Mann"). Bottom: the signal's waveform.

As we can see, this speech signal has a strong fundamental frequency track around 120 Hz, and harmonics at integer multiples of the fundamental frequency. We perceive the frequency of the fundamental as the speech's pitch. The magnitude of the harmonics varies over time, which we perceive as the sounds of different vowels.

All of this is clearly visible in this STFT. To calculate the STFT, we chop up the audio signal into short, overlapping blocks, apply a window function ² to each of these blocks, and Fourier-transform each windowed block into the frequency domain:

Figure 2: The STFT is a two-dimensional graph of short-time spectra. Window functions and corresponding spectra of some blocks are highlighted.

We zoomed in a bit to make this visible. At this zoom level, an old friend becomes visible in the waveform: The waveform of our harmonic spectra is clearly periodic. We also see that each of our blocks contain multiple periods of the speech signal. So what happens if we make our blocks shorter?

Figure 3: STFT with shorter blocks (graph shows a shorter time, too). Each window now covers roughly one period of the signal.

As we make our blocks shorter, such that they only contain a single period of the signal, a funny thing happens: the harmonic structure vanishes. Why? Because the harmonic structure is caused by the interference between neighboring clicks; As soon as each block is restricted to a single period, there is no interference any longer, and no harmonic structure.

But instead of a harmonic structure, we see the periodicity of the waveform emerge in the STFT! To look at this phenomenon in a bit more detail, we will have to leave the classical STFT domain, and look at a close sibling, the Constant Q Transform, or CQT. The CQT is simply an STFT with different block sizes for different frequencies:

Figure 4: The CQT uses shorter block sizes for higher frequencies and longer block sizes for lower frequencies.

The periodic structure of the signal is now perfectly apparent. Each period of the signal gives rise to a skewed click-like spectrum that repeats at the fundamental frequency. Interestingly, however, the fundamental itself shows no periodicity. To see why this is so, and to finally get a grasp of how the short-time Fourier transforms interprets signals, we have to turn to our last graph, a gamma-tone filterbank:

Figure 5: A gamma-tone filterbank is a series of band-pass filters. Each horizontal line is now a frequency-constrained waveform.

The fundamental frequency is now clearly a visible as a single sinusoid, rotating its merry way at 120 Hz. Each harmonic is its own sinusoid at an integer multiple of the fundamental frequency, but changing in amplitude in lockstep with the findamental. Thus in this graph, we see both the harmonicity and the periodicity at the same time, and can finally appreciate the complex interplay of magnitudes and phases that give rise to the beautiful and simple sound of the human voice³.

Footnotes:

also known as "spectrogram"

a Hann window, in this case

in the magnitude domain only; Phases, and different window functions, go beyond the scope of this article.

Periodicity and Harmonicity

Thu, 19 Sep 2019 13:51:00 +0200

A periodic signal has a harmonic spectrum. In the extreme case, a click train signal has a comb spectrum:

Figure 1: A click train (left), and its corresponding comb spectrum (right).

But why? After all, a solitary click has a uniform spectrum. So why should the sum of multiple clicks have a non-uniform spectrum?

Figure 2: A click (left), and its corresponding spectrum (right).

The answer exposes a lot of detail of how spectra work, and gives us a glimpse into the inner workings of spectral phases. Let's start with that click: In that spectrum, we see that we can decompose a click into a sum of sine waves:

Figure 3: A click (left), and its Fourier series (right). For simplicity, not all frequencies of the Fourier series are shown.

As we have seen in the spectrum of the click earlier, it is composed of all frequencies (uniform spectrum). But we see here that the individual sine waves are delayed just so, that they all add constructively at exactly one time, and form the click. At all other times, they cancel each other out. This per-sine delay is called the phase of that frequency.

But what happens if we have more than one click? How does that change things?

Figure 4: A click train (left), and its Fourier series (right). For simplicity, not all frequencies of the Fourier series are shown. The contributions of each click are shown in different colors.

In a click train, odd frequencies in the Fourier series of the individual clicks cancel each other out (red/blue), and only harmonic frequencies (brown) remain. And this is exactly what our first spectrum showed: A periodic click train results in a harmonic spectrum.

Even though each click has a uniform spectrum, adding multiple clicks together cancels out all non-harmonic parts, and only a harmonic comb spectrum remains.

How we perceive Speech

Wed, 11 Sep 2019 16:35:00 +0200

Now that I am officially a failed scientist, I might as well talk about my research in public. I spent the last few years analyzing speech recordings. Particularly, voiced speech, where vibrations in the vocal folds excite resonances in the vocal tract, and a tonal sound leaves our mouths and noses.

As humans, we are particularly tuned to recognizing these kinds of sounds. Even in loud background noise, even with dozens of people talking at the same time, we can clearly identify the sound of a human voice (even if we might not be able to understand the words).

Figure 1: A spectrogram of speech (it's me, saying: "Es war einmal ein Mann")

Looking at these sounds from a physical point of view, we can see that it is made up of a fundamental frequency at the voice's pitch, and harmonics at integer multiples of the fundamental. And even though the sound is clearly composed of multiple harmonics, we perceive it as a single sound with a single pitch. Even more perplexing, we attribute all of these harmonics to a single voice, even if they criss-cross with tonal sounds from different sources.

Yet, speech recognition systems regularly struggle with such tasks, unless we feed them unholy amounts of data and processing power. In other words, there has to be more to speech than the simple figure above indicates.

One area is definitely time resolution. Obviously, when the vocal folds open to admit a puff of air into the vocal tract, phases align, and loudness is higher than when the vocal folds have closed again and phases go out of sync. This happens several hundreds of times per second, at the frequency of the fundamental. Yet, this phase coherence is invisible in most of our visualizations, such as the spectrogram above, or the MFCCs usually used in speech recognition, as they are too coarse for such short-time detail.

An even more interesting detail emerges from fMRI scans of people who are speaking, and people who are listening to speech: their activation patterns are strikingly similar. As in, motor groups are activating when listening, just as if actual speech muscles were moved. To me, this indicates that when we listen to speech, we simulate speaking. And I find it highly likely that we understand speech mostly in terms of the movements we would have to make to imitate it. In other words, we do not internalize speech as an audio signal, but as muscle movements.

This matches another observation from a different area: When learning a foreign language, we can not hear what we can not produce. If you didn't learn how to speak an Ö and Ü (two German umlauts) as a child, you will have a hard time hearing the difference as an adult. Yet they sound completely distinct to me. In a production model, this makes a lot of sense, as we wouldn't know how to simulate a sound we can not produce.

Bringing this back to the science of signal processing, I believe that most speech analysis algorithms are currently lacking a production model of speech. Speech can not be fully understood as an audio signal. It needs to be understood in terms of the variables and limitations of a human vocal tract. I believe that if we integrated such a physiological production model into our machine learning models, we wouldn't need to feed them such vast amounts of data and electricity, and might even get by without them.

Publish or Perish

Tue, 09 Jul 2019 13:48:00 +0200

As part of my PhD, I am supposed to publish three papers. So far, I have been unable to do so. But this is not about me, I will survive regardless. This is about the systems behind our papers' rejections. Because they are… bad. Political. Un-scientific.

Our first manuscript was submitted for publications, and got a middling review. If we wanted our work to be published, we were to expand on our introduction to mention the reviewers' favorite publications, and broaden our comparison to include their work. This is considered normal. In the second round of reviews we then got rejected, because our introduction was now too long, and our comparison too broad.

The reviews additionally claimed that "novelty cannot be claimed for something that is not validated and compared to state of the art" and that "[our work lacks] formal statistical evaluation of the estimation performance". Which is certainly true, but is also true of every other work published on the same topic in the last five years (I checked). We showed this evidence to the reviewers, but it was not even deemed worthy of a comment.

In hindsight however, we realized that we had included at least one reviewer's own algorithms in our comparison, and found it lacking. Their work had only ever been tested, publicly, with a single example recording, where it worked well. Our comparison did the same with twenty thousand recordings, which highlighted some issues. So our paper was rejected. Of course we can't be sure that this was ultimately true, as the reviewers' names are not disclosed to the reviewees (but certainly vice-versa).

Our next submission was to a different journal. This time, we had learned from our mistakes, and kept the scope of our investigation more minimal. There would be only a very small comparison, and we would be very careful not to step on anyone else's toes. The review was, again, negative.

This time, the grounds for rejection were lack of comparison to state of the art (not a winning move, see above), and our too high false negative rate. Additionally, it contained wonderful verbiage like:

The are many methods that are very similar to the presented method in the sense of being feature extraction methods operating in the STFT domain.

…which is just patently ridiculous. If being a "feature extraction method in the STFT domain" was grounds for rejection, there would be no publications in our area of research. And let's ignore for a minute that our publication was not, in fact, such a method.

Again, hindsight showed the real culprit: Our manuscript reported a high false negative rate of roughly 50%. Had we just not mentioned this, no one would have noticed. That is what everyone else is doing. More importantly however, reporting on false positive/negative rates in our evaluation called into question every other publication that hadn't. And we can't have that.

Another submission was liked because no one had done anything similar before, and was found to provide value to many researchers, but rejected because it still somehow "lacked novelty".

So, in summary, our first submission was rejected because it made one of the reviewers look bad, and the second because we not only wanted to report on our method's advantages, but also its shortcomings. Worse, in following the evidence where it lead, we had created new error measures that could potentially find flaws in existing publications, which could potentially make a whole lot of researchers look bad.

After five years of dealing with this, I am thoroughly disheartened. Instead of a system for disseminating knowledge, I have found the scientific publishing system a political outlet for the famous, and a lever for keeping uncomfortable knowledge suppressed. This is not the scientific world I want to live in, and apparently, it doesn't want me to live in it, either.

Fuji Zoom Lenses

Sat, 11 May 2019 13:18:00 +0200

So I bought a new camera. Now I need new lenses. In this post, I am looking for a standard zoom lens, i.e. something that covers a bit of wide-angle, all the way through the normal range, up to a bit of telephoto. In Fuji's lineup these needs are met by

~~the XC 15‑45 mm f/3.5‑5.6 OIS PZ (€ 150, 136 g, 4.4 cm)~~
the XC 16‑50 mm f/3.5‑5.6 OIS II (€ 150, 195 g, 6.5 cm)
the XF 18‑55 mm f/2.8‑4 R LM OIS (€ 250, 310 g, 7.0 cm)
the XF 18‑135 mm f/3.5‑5.6 R LM OIS WR (€ 500, 490g, 9.8 cm)
~~the XF 16‑55 mm f/2.8 R LM WR (€ 650, 655 g, 10.6 cm)~~
~~a bag of primes (€ inf, many g, lots of cm)~~
the XF 27 mm f/2.8 (€ 150, 78 g, 2.3 cm)

The XC 15‑45 is out, because the zoom ring is not turn-to-zoom like any other zoom lens. I tried it; I couldn't stand it. The XF 16‑55 is out because it is just too expensive and big for me. A bag of primes is not what I want, but I included the XF 27, because that's what I happened to have at hand. All of the above prices are used prices as of early 2019 in Germany.

Figure 1: From left to right, the XC 16‑50, XF 27, XF 18‑135, XF 18‑55

Before we get started, all of these lenses are perfectly sharp to my eyes. At least in the center-ish area of the image, every pixel shows different information, which is not something I could have said about some of the Nikon lenses I used to own. Because of that, I will not compare sharpness.

The word of mouth is that the XF 18‑55 is a stellar kit lens, the XC 16‑50 is a bit cheap, and the XF 18‑135 a bit of a compromise. If internet forums are to be believed, these differences are massive, and the XF 18‑55 is really the only acceptable non-prime lens any self-respecting Fuji fanboy can buy. But then again, that's what internet forums would say, right?

With that out of the way, let's have a look at these lenses! I'll use crops from a terribly boring shot linked here for most of my examples. Why this shot? Because a), that's what was available, and b), because it contains areas that nicely showcase these lenses' qualities. All shots were taken at f/8 at 27 mm, ISO 400 and a shutter speed around 1/300 s.

Micro Contrast

I often read that micro contrast really tells lenses apart. To illustrate this point, here's an area with very little overall contrast, particularly between the fence and the orange paint, and the fence and the gray stairs:

Figure 2: 100% crops, mouse pointers near the critical areas. (Click to view bigger)

If you look very closely, you might find the fence slightly less visible on the XC 16‑50 than on the XF 18‑135 and XF 27, and ever so slightly more visible on the 18‑55. But it should also be obvious that these differences are incredibly tiny, and not worth fussing over.

Corner Sharpness and Chromatic Aberrations

Another common point is corner sharpness, which is typically said to strongly favor primes. This time, the images are cropped from the bottom-right corner of the image that contains some detail, but most importantly a bright white warning sign:

Figure 3: 100% crops of the image corner. (Click to view bigger)

And indeed, the XC 16‑50 is noticeably blurry this time, with the other three lenses similarly sharp. The warning sign also highlights color fringes on the transitions from the bright white sign to the dark background. These chromatic aberrations are almost invisible on the XF 18‑55, and mild on the XF 18‑135 and XF 27.

Bear in mind, however, that these are 100% crops in the very furthest corners of a high-contrast image. In normal pictures, none of these issues will be noticeable unless you really zoom in on fine details at the edges of your frame. The chromatic aberrations seen here were already treated in software with the lens correction module in Darktable, but it might be possible to improve on these results with more dedicated processing.

Update 1: Even more comparison pictures

After publishing the blog post, I still wasn't satisfied: What if the results I got were only true at f/8? What if image quality got worse at a longer focal length? How does my XF 18 stack up?

To answer this, I took another set of pictures and prepared another composite of an crops near the image center, and another one near the lower right corner.

Figure 4: 100% crops of the image center. (Click to view bigger)

Figure 5: 100% crops of the image corner. (Click to view bigger)

To be perfectly honest, I can not see any significant differences between any of these pictures. At this point, I am starting to question the entire concept of sharpness and micro contrast for evaluating lenses. But at least I learned a lot about how to use the Gimp.

As a sanity check, I repeated the experiment with my old Nikon 18-200, and this was in fact noticeably less sharp. And slightly overexposed. And slightly off-color. That's why I switched to Fuji. But as I said, this was a sanity check, not a fair comparison, as the Nikon D7000 body is much older than my Fuji X-E3, and the lens has surely seen better days as well.

Update 1: Ergonomics and Balance

The XC 16‑50 and XF 27 operate their aperture with the control wheel on your right thumb. The XC 18, XF 18‑55, and XF 18‑135 have a dedicated aperture ring on the lens instead. Thus, the former two lenses can be controlled with the right hand alone, while the latter two require the left hand on the lens barrel. Zoom is always controlled on the barrel, though.

This preference for one-handed or two-handed operation is supported by the lenses' weight, as well: My camera, the X-E3, weighs about 330 g. With the 200 g XC 16‑50, the weight is mostly in the camera body, and can easily be held and operated with one hand. The 250 g XF 18‑55 and the 500 g XF 18‑135 are more lens-heavy, which makes a two-handed grip necessary, which is a better fit for the aperture ring on the lens.

Personally, I actually prefer the aperture on the thumb wheel over the unmarked aperture rings on the XF 18‑55 and the XF 18‑135. It just feels more natural in my hands. On the other hand, I like the marked aperture ring on the XF 18, particularly for resetting the aperture without looking through the viewfinder, or when the camera is turned off. In fact, I find the ability to operate the camera while turned off to be very useful in general. It is one of the major reasons why I like Fuji cameras.

Update 2: Image Stabilization

In order to assess the image stabilization systems built into these lenses, I took a series pictures of a static subject at 18 mm, 27 mm, and 50 mm, for shutter speeds of 1/30 s, 1/15 s, 1/8 s, 1/4 s, and 1/2 s. I then looked at five images for every combination of lens, focal length, and shutter speed, and labeled them either sharp if there was no visible blur at all, or usable if there was micro-shake only visible at 100 %, or miss if the shot was too blurry.

The XC 16‑50 had perfect sharpness at 1/30 s, was at least ok between 1/15 s and 1/8 s, and even 1/4 s still had a few usable shots. 1/2 s or longer was unusable. There was no significant difference between the focal lengths. That last bit is really interesting, as I would have expected shorter focal lengths to be easier to hand-hold than longer ones.

The XF 18‑55 stayed perfectly sharp one stop longer until 1/15 s, but otherwise performed exactly the same as the XC 16‑50. I would guess that the small difference in stability between these two lenses is mostly due to their weight difference, but that the image stabilization system is identical.

The XF 18‑135, however, was another matter: All shots up until 1/8 s were perfectly sharp, and remained at least usable until 1/2 s! Only at 1 s of shutter speed did I see significant numbers of missed shots! Again, there was no significant difference across focal lengths.

With disabled image stabilization, I could hand-hold most shots for at most 1/focal length, but missed or fudged a few shots even there.

In summary, I found the XC 16‑50 and XF 18‑55 image stabilization good for about two stops, and astonishingly, the XF 18‑135 stable for a full four stops over my personal hand-holding skills. Some of that stability is no doubt due to the increased weight of the XF 18‑135, but nevertheless, I find these results astonishing!

Close Focus Distance and Magnification

And now, the darling of all photographers: out-of-focus backgrounds. Common wisdom is that the bigger the aperture, the more the background is thrown out of focus. But that's only part of the truth, and honestly, not the most interesting part for these kinds of limited-aperture lenses. Much more powerful is getting closer to your subject: The closer you focus, and the farther away your background, the more the background will be out of focus. This effect gets even stronger when you zoom in.

Figure 6: Widest (top) and longest (bottom) shots, each cropped vertically but not horizontally. All shots at f/5.6. (Click to view bigger)

The XC 16‑50 focuses much more closely than any other lens in this list, at 12 and 30 cm (Fuji says 15 cm). You can get really nice background separation with this lens, and great magnification in your macro shots. The XF 18‑55 focuses at 25 and 35 cm (Fuji: 40 cm), which is not particularly impressive. The XF 18‑135 focuses even farther, at 33 and 43 cm (Fuji: 45 cm), but gains magnification through its long tele zoom. The XF 27 is not optimized for this kind of thing at all, at 29 cm (Fuji: 34 cm).

Conclusions

To me, the XC 16‑50 is the winner for a small/light zoom kit. It might be the least great option optically, but the differences are not dramatic at all, and it is the cheapest, smallest, and lightest lens with the most useful wide end and the closest focusing. But it lacks a dedicated aperture ring and is a plastic construction instead of a metal one, which does detract from the haptic joy somewhat.

The XF 18‑55 is optically the strongest lens. It might even beat the XF 27 prime lens on its own turf! But the optical differences to the cheaper XC 16‑50 and the more versatile XF 18‑135 are quite small, and are not be worth the price/weight/inconvenience to me.

The XF 18‑135 is really surprisingly good. The much longer focal range necessarily comes with compromises in optical quality and bulk, but it seems no significant corners where cut in this case. And the image stabilization is a significant step above the other two lenses. Considering that this lens usually replaces at least two other lenses, I even find the price reasonable. This is my first choice as a do-everything zoom kit.

The XF 27 is not very strong in any particular way, except size. And that size trumps all. If I just want to throw a camera in my bag without any particular photographic intentions, the XF 27 is my first choice. And possibly the XF 18, if I still have room in my bag.

As some small buying advice, the XC 16‑50 was refreshed in 2015 with the OIS II version, which introduced that nice close focusing distance (highly recommended). The XF 18‑135 was apparently built in two batches, the original made in China version that seemed to have horrible QA issues, and a second made in Philippines version in 2017 without.

What I didn't mention

Aperture. The XF 18‑55 and XF 27 have a wider maximum aperture than the XC 16‑50 or XF 18‑135, by about two thirds of a stop. Shooting at bigger apertures makes brighter pictures with stronger background blur, and some loss in sharpness. I don't find the optical performance wide-open particularly interesting, because most of the time I'd use large apertures to blur the background, making sharpness and distortion mostly irrelevant. And as I said above, getting closer is usually more effective for background blur than maximum aperture, anyway.

Image stabilization. The three zooms offer optical image stabilization systems. From what I can tell, the XF 18‑135 is significantly more effective in this regard than the XC 16‑50 or the XF 18‑55. Hand-held shots with up to about 1/10th of a second seem easily achievable with the XF 18‑135, whereas the unstabilized XF 27 becomes blurry at 1/40th. Videos are noticeably smoother with the XF 18‑135 as well.

Weather sealing. The XF 18‑135 is weather sealed, the other lenses are not. My camera is not, so I don't care.

Distortion and Vignetting. Is fixed in post. No need obsessing over it.

Autofocus speed. Is good. No need obsessing over it.

On camera sensor sizes

Mon, 25 Mar 2019 20:20:00 +0100

A common internet wisdom about photography is that bigger camera sensors capture more light. So if you want to work in low light, you need a full frame camera, and a bigger sensor always produces better image quality. I have struggled with this a lot. It just doesn't make sense: Lenses can focus light on any surface, so why should the surface size matter?

The answer turns out to be… disappointing. Big-sensor cameras allow for larger (practical) apertures, and lower base ISO. But less noise for the same picture is simply impossible with the same sensor technology. Because that's not how physics works. Let me explain.

At this point, I had previously launched into a long-winded explanation on equivalence, and reached a slightly misguided conclusion. So here's a better one:

It is possible to build equivalent lenses for differently sized sensors, which capture the same amount of light, with the same depth of field, and the same field of view, but simply project this light on a differently sized sensor. Or you can use a "speed booster", which adapts a "bigger" lens to a "smaller" sensor. Both of these will give you identical images, and identical amounts of noise (if the same sensor technology is used).

As an example, a micro four thirds 23mm f/1.4 lens produces identical images to an APS-C 35mm f/2 lens or a full frame 50mm f/2.8 lens. You might notice that $\frac{23}{1.4} = \frac{35}{2} = \frac{50}{2.8}$, which means each of these lenses will have the same physical aperture size, and therefore admit the same amount of light. And be the same size and weight. Since the smaller sensor collects the same light on a smaller area, the image will be brighter, and will need to use a lower ISO for the same exposure. And since lower ISO produces less noise, the "one stop advantage" of bigger sensors is bunk, if equivalent lenses are used.

The one remaining difference between sensor sizes is that bigger photosites carry more charge, which means more dynamic range at base ISO on a bigger sensor. This is only a factor if you indeed shoot at base ISO.

But, that's not the interesting part. Camera manufacturers cleverly designed their product lines such that small-sensor cameras are physically smaller, and get smaller lenses, while large-sensor cameras are bigger, with bigger lenses. Thus, if you want to get access to the most fancy glass, you will have to use a bigger, more expensive camera body as well. And inversely, if you want to use the most compact glass, it will only be available on smaller, less expensive camera bodies. And that's not pure marketing, either, as bigger lenses require bigger bodies to hold them comfortably, while smaller lenses balance better on a smaller body.

So in the end, it comes down to a compromize: If you want/need the best glass, it will only be available on the bigger bodies with the bigger sensors. If, on the other hand, you want to go small and light, you will need to sacrifice big apertures, but get access to smaller bodies with smaller sensors. But it's really all about the glass, not the sensor sizes.

Learning about Photography: Sunstars

Wed, 13 Mar 2019 20:17:00 +0100

Normally, when you take a picture of something too bright, you get bloom: An all-consuming brightness that plunges everything around it into pure whiteness. Ugly.

But if the light source is reeeally tiny, and your aperture is teeeensy as well, you get something else: sunstars

This particular sunstar has fourteen corners, and therefore comes from a seven-bladed aperture (in my Fuji XC 16-50). It happens because tiny apertures are not perfectly circular any longer, but instead, in my case, septagonal, and therefore bloom more in some directions than in others. The effect is kind of beautiful.

In this picture, the sun was just barely peeking into the edge between the tree and the building, and my aperture was set to its smallest setting, f22. I actually wanted to capture the raindrops on the branches, which I largely failed at. In the end, the picture didn't turn out very pretty, but at least I got some fine sunstars!

What I learned about Amateur Photography

Sun, 24 Feb 2019 10:50:00 +0100

I am an amateur, as in "lover of", photography. I love cameras as tactile devices, I love how photography makes me consider the world as art, how that little viewfinder can reveal unknown beauty in well-known places or people. And I love looking at my photos, and remembering vacations and meaningful moments. For me, photography is about finding beauty, and capturing memories.

However, most of the writing on photography seems to be focused not on my needs, but the needs of professional photographers: A super competitive field of visual artists who compete on image quality and novelty, and use crazy and expensive gear. I have found many of their lessons not applicable to my amateur needs, or even actively detrimental:

Embrace the noise

Many pros limit their ISO numbers, because high-ISO noise is ugly. Which it is. But you know what is even worse for an amateur? Not having that picture of my baby, because it was too dark.

So I set my ISO to unlimited, reduce my shutter speed and aperture so I actually have a chance of capturing my fast-moving toddler. And embrace the ensuing noise. Some of my favorite pictures look unbearably noisy on my 4k screen, but look just fine when printed, or on a smartphone (the two most important mediums in the world). Because of this, I find noise reduction rarely worth the effort. Color heavy noise reduction works ok, but anything else looks worse than the problem. I vastly prefer a sharp, noisy shot to a mushy denoised shot with no detail.

Step it down

Another common Pro argument: Wider apertures are better. Which they are, at capturing light, and blurring the background. But as an amateur, a wide-aperture super-shallow depth of field just makes me miss shots. At f1.8, the area in focus is barely a few centimetres deep. I missed too many shots because I accidentally focused on the nose instead of the eye. So, in the absence of studio lighting, and arbitrarily many retries, I prefer to step it down and live with the noise, if need be.

As a fun corollary, all those fancy prime lenses with crazy-wide apertures, they are simply wasted on me. Anything beyond, say, f2.8, is not something I need to spend money on. Also, lenses are noticeably sharper when stepped down! I have been disappointed with the sharpness of a number of shots because I forgot to step it down. Nowadays, I typically shoot at f5.6 or f8, and only go wider if I actually need to, because of lack of light, or if I specifically want a blurry background.

Wide-ish lenses are easier

I wondered, for a long time, why pros seem to like long-ish lenses for portraits: The answer is, because longer lenses have a shallower the depth of field (for the same f-number), and pro photographers love their blurry backgrounds. But as I said before, that is not for me.

Instead, I prefer wide-ish lenses. If something is too small on a wide-ish lens, it is usually no problem to get a bit closer or to crop afterwards. If something is too big on a long lens though, backing off is often not possible, and you miss your shot¹. Plus, wide-ish angles don't blur as much from shaky hands, and are more compact. They often focus more closely, too, which is a huge bonus if you want to take pictures of a toddler.

I have tried, unsuccessfully, a 50mm and 35mm prime (APS-C). Now I own a 27mm pancake prime, which I find perfect: long enough to get nice portraits, but still wide enough to capture a landscape.

Be careful with super-wide angle lenses, though. Even though they are a ton of fun, I have found anything below 16mm to be very difficult to use effectively. It's just too easy to get that fisheye-distorted look, especially near the sides. That distorted look, by the way, is caused by being too close, not by lens distortion. The same thing happens if you take a longer lens or a cell phone and get too close. Just don't do that.

Gear

Pros use the biggest sensor they can get, to get the best image quality possible. But that also makes everything else much more cumbersome: Bigger sensors mean bigger and heavier bodies. And bigger and heavier lenses. And shallower depth of field (see above). And smaller focal ranges, hence more lens changing. And, not least of all, much, much, much higher prices. It is not for me.

To some extent, the same goes for different quality levels: Personally, I have found entry-level APS-C mirrorless interchangeable lens cameras a good compromise. These entry-level plastic lenses and cameras are usually smaller, lighter, and cheaper than their higher-end brethren, but compromise on robustness and aperture sizes (e.g. Fuji X-E2 375g/€250 + Fuji XC 16-50mm, 195g/€150 vs. Fuji X-T1 450g/€250 + Fuji XF 18-55mm, 300g/€250 vs. my old Nikon gear). And for my everyday camera, I' take a smaller, pocketable camera over a "better", bigger one any day.

Haptics are important, too. I have seen great cameras and lenses that just didn't feel good in my hand. Which meant I wouldn't ever take them with me, and wouldn't take any pictures with them. I now go try stuff in the store before I buy anything. This has talked me out of a number of unnecessary purchases, internet consensus notwithstanding.

And finally, I buy used gear. Cameras and lenses depreciate about 70% within the first two years, without losing any quality. A great camera from two years ago is still a great camera, but costs a third of the original price, and can be resold without loss. And is better for the environment. Win-win-win.

TL;DR

I have found many "common" rules about photography useless for my amateur needs. I have found cheap, plastic, used gear more useful than pro gear. I have found noisy, small-aperture pictures to be better at capturing important memories than clean, "professional" ones. I have found haptics, size and weight to be much more important than ultimate image quality (within reason).

The funny thing is, you don't find this kind of information on the internet, since most review websites seem to focus on the professional viewpoint, even for gear that is clearly meant for amateurs like me.

Footnotes:

You need to back off much farther on a long lens than you have to move closer on a wide lens.

Books of 2018

Sun, 06 Jan 2019 10:54:00 +0100

There is no other media more immersive to me than reading. When I read, I dive deep into another world, and seize to perceive the reality around me. It is my escape from the everyday stresses of living in the modern world. Reading is truly sacred to me. So here are my favorite pieces of writing of 2018:

Shadow Divers

This is a book about deep wreck diving. I am not interested in wrecks, or diving. Nevertheless, no book gripped me as hard last year as Shadow Divers. At first, I was a bit thrown off by the writing style, and thought it dramatized things too much. But I just couldn't put this book down. There were times when I surfaced from reading with a distinct feeling of having been there, of having touched that menacing shipwreck at the bottom of the ocean.

By the end of it, I was entirely engrossed in the story and its characters and the history of the mystery wreck. And then, in the epilogue, I learned that what I had thought embellished at first was actually meticulously researched and not overdramatic at all.

And best of all, my favorite podcast, Omega Tau had an interview with the main protagonist of the book, which is just fascinating!

The Shortest History of Germany

This book is a strange choice to put on my list of books, because I didn't like it too much. I found it offputting which parts of history the author emphasized, and which parts he skipped. Apparently, the two world wars did not bear mentioning. The book paints a strangely specific west-vs-east map of Germany, which is not without merit, but surely not the whole story. And worst of all, I really didn't care for the writing style at all.

But when I look back on 2018, this book has stuck with me. Never mind its shortcomings, it has given me a map of Germany's history that I can fit other pieces of knowledge into. And that makes it valuable to me.

And then there were (N-One)

I love short stories. And this one is just brilliant. It is by Sarah Pinsker, about Sarah Pinsker, and all Sarah Pinskers from neighboring parallel universes. Every character in this book is a Sarah Pinsker. And all of them agree, Sarah is no murderer. Yet, one of them has been found murdered.

I won't spoil more of it, just head over to the Uncanny Magazine and read it for yourself!

Graphic Novels

For years now, I have struggled to find graphic novels that speak to me. I usually find the drawing style off-putting, and would have preferred a more fleshed out novel instead of a shallow graphic novel. But this year, I found two brilliant graphic novels that I loved dearly:

Maus

The half-auto-biographical narrative of a survivor of one the Nazi concentration camps. The novel juxtaposes the author's struggle to interview his father about his past, and thereby shows both the cruelty of the father's experience, and the man that this experience created. Both of these perspectives add weight to the unspeakable nastiness of the Holocaust.

But beyond that, it is also a story of humanity, struggle, and the small moments of joy amidst all the terror. Truly a masterpiece of a book.

Nimona

In sharp contrast to the previous book, Nimona is just plain fun! Nimona is a teenage girl, and a shark, and the sidekick of Balister Blackheart, the biggest name in Supervillainy. And if you haven't guessed it from this description, it is wacky, and funny, and uplifting, and just… fun!

You can read the first three chapters online, and then you'll want to read the rest, too. It's just too much fun!

Cool Python Libraries: TQDM and Resampy

Thu, 08 Nov 2018 14:52:00 +0100

In my recent post about appreciation for open source software, I mentioned that we should praise our open source heros more often. So here are two lesser-known libraries that I use daily, and which are unabashedly awesome:

TQDM

TQDM draws text progress bars for long-running processes, simply by wrapping your iterator in tqdm(iterator). And this, alone, would be awesome. But, TQDM is one of those libraries that aren't just a good idea, but then go the extra mile, and add fantastic documentation, contingencies for all kinds of weird use cases, and integration with notebooks and GUIs.

I use TQDM all the time, for running my scientific experiments and data analysis, and it just works. For long-running tasks, I recommend using tqdm(iterator, smoothing=0, desc='calculating'), which adds a meaningful description to the progress bar, and an accurate runtime estimate.

Resampy

Resampy resamples numpy signals. Resample your data with resample(signal, old_samplerate, new_samplerate). Just like with TQDM, this simple interface hides a lot of complexity and flexibility under the hood, yet remains conceptually simple and easy to use.

But beyond simplicity, resampy uses a clever implementation that is a far cry better than scipy.signal.resample, while still being easy to install and fast. For a more thorough comparison of resampling algorithms, visit Joachim Thiemann's blog.

Dealing with Unreliable Software

Fri, 26 Oct 2018 09:45:00 +0200

About a year ago, I started working on a big comparison study between a bunch of scientific algorithms. Many of these have open-source software available, and I wanted to evaluate them with a large variety of input signals. The problem is, this is scientific code, i.e. the worst code imaginable.

Things this code has done to my computer:

it crashed, throwing an error, and shutting down nicely
it crashed with a segfault, taking the owning process with it
it crashed, leaving temporary files lying around
it crashed, leaving zombie processes lying around
it spin-locked, and never returned
it spin-locked, and forked uncontrollably until all process decriptors were exhausted
it spin-locked, and ate memory uncontrollably until all memory was consumed
it crashed other, unrelated programs (no idea how it managed that)
it corrupted the root file system (no idea how it managed that)

Note that the code did not do any of this intentionally. It was merely code written by non-expert programmers, the problems often a side effect of performance optimizations. The code mostly works fine if called only once or twice. My problems only become apparent if I ran it, say, a few hundred thousand times, with dozens of processes in parallel.

So, how do you deal with this? Multi-threading is not an option, since a segfault would kill the whole program. So it has to be multi-processing. But all the multi-processing frameworks I know will lose all progress if one of the more sinister scenarios from the above list hard-crashed one or more of its processes. I needed a more robust solution.

Basically, the only hope of survival at this point is the kernel. Only the kernel has enough power to rein in rogue processes, and deal with hard crashes. So in my purpose-built multi-processing framework, every task runs in its own process, with inputs and outputs written to unique files. And crucially, if any task does not finish within a set amount of time, it and all of its children are killed.

It took me quite a while to figure out how to do this, so here's the deal:

# start your process with a new process group:
process = Popen(..., start_new_session=True)

# after a timeout, kill the whole process group:
process_group_id = os.getpgid(process.pid)
os.killpg(process_group_id, signal.SIGKILL)

This is the nuclear option. I tried SIGTERM and SIGHUP instead, but programs would happily ignore it. I tried killing or terminating only the process, but that would leave zombie children. Sending SIGKILL to the process group does not take prisoners. The processes do not get a chance to respond or clean up after themselves. But you know what, after months of dealing with this stuff, this is the first time that my experiments actually run reliably for a few days without crashing or exhausting some resource. If that's what it takes, so be it.

Appreciation for Open Source and Commercial Software

Sun, 14 Oct 2018 10:20:00 +0200

I recently released my first-ever piece of commercial software, a plugin for the X-Plane flight simulator. I wrote this primarily to scratch my own itch, but thought other users might like it, too, so I put it up on the store. What struck me however, were the stark difference between the kinds of responses I got to this, as compared to my open source projects: They were astonishingly, resoundingly, positive!

You see, I have a bunch of open source projects, with a few thousand downloads per month, and a dozen or so issues on Github per week. Most of my interactions with my users are utilitarian, and efficient. Someone reports a bug or asks for help, I ask for clarification or a pull request, we iterate a few times until the issue is resolved. The process is mechanical and the tone of our conversation is equally unemotional. This is as it should be.

After having released my flight simulator plugin, however, people thanked me! They congratulated me! They extolled about the greatness of what I had built! And they did this despite the fact that the initial release had quite a few major bugs, and even flat-out did not work for some people. Yet even people who couldn't get it to work were grateful for my help in resolving their issue!

This blew my mind, in comparison with the drab "I found a bug", "Could you implement…" I was used to from my open source work. There, the feedback I got was mostly neutral (bug reports, feature requests), and sometimes even negative ("You broke something!"). So I release my software for free, as a gift, and get average-negative feedback. My commercial work, in contrast, costs money, and yet the feedback I get is resoundingly positive! I can not overstate how motivating it is to get praise, and love, from my users.

I think this is a huge problem for our Open Source community. I had my run-ins with burnout, when all the pull requests came to be too much, and I started dreading the little notification icon on Github. And I think the negativity inherent in bug reports and feature requests has a huge part to do with this. In the future, I will try to add more praise to my bug reports from now on, just to put things into perspective.

But I think we should go further than that. We should create tools for praising stuff, beyond the impersonal Stars on Github. We should be able to write reviews on Github, and recommendations, and blog posts about cool libraries we find.

I recently got my first github issue that was just a thank-you note. I loved it! We need more positivity like that.

Syncing Org-Journal with your Calendar

Sun, 03 Jun 2018 10:28:00 +0200

A month ago, org-journal learned to deal with future journal entries. I use future journal entries for appointments or not-yet-actionable tasks that I don't want in my current TODO list just yet. This works really well while I am at my computer, and really does not work at all when I am not (Orgzly does not work with my 1k-file journal directory).

But, as I keep re-discovering, org-mode already has a solution for this: org-mode can export your agenda to an iCalendar file! Most calendar applications can then subscribe to that file, and show your future journal entries right in your calendar. And if you set it up right, this will even sync changes to your calendar!

First, you need to set up some kind of regular export job. I use a cron job that regularly runs an Emacs batch job emacs --batch --script ~/bin/calendar_init.el with the following code in calendar_init.el:

;; no init file is loaded, so provide everything here:
(add-to-list 'load-path "~/etc/org-journal/")
(setq org-journal-dir "~/journal/"            ; where my journal files are
      org-journal-file-format "%Y-%m-%d.org"  ; their file names
      org-journal-enable-agenda-integration t ; so entries are on the agenda
      org-icalendar-store-UID t               ; so changes sync correctly
      org-icalendar-include-todo "all"        ; include TODOs and DONEs
      org-icalendar-combined-agenda-file "~/calendar/org-journal.ics")

(require 'org-journal)
(org-journal-update-org-agenda-files) ; put future entries on the agenda
(org-icalendar-combine-agenda-files)  ; export the ICS file
(save-buffers-kill-emacs t)           ; save all modified files and exit

It is important to set org-icalendar-store-UID, as otherwise every change to a future entry would result in a duplicated calendar entry. It will clutter up your journal entries with an UID property, though.

I do this on my web server, with my journal files syncthinged from my other computers. With that, I can subscribe to the calendar file from any internet-connected computer or mobile phone (using ICSdroid). But you could just as well sync only the ICS file, or just subscribe to the local file, if you don't want to upload your complete yournal to a web server.

(Incidentally, I first implemented my own ICS export, before realizing that this functionality already existed in org-mode. It was a fun little project, and I learned a lot about org-mode's internal data structures and the weirdness that are iCalendar files.)

GDPR Compliance

Fri, 25 May 2018 17:21:00 +0200

Personal email is dead. The signal-to-noise ratio of my personal email account has been deeply negative for years. But the last few days have been especially riveting, with a torrent of GDPR-compliance emails from just about every company that has ever gotten their hands on my email address. Anecdotally, if spam makes up about 90% of all email traffic, and the last few days have seen a ten-fold increase in traffic due to GDPR emails, we might even have "defeated spam" for a few days! Yay internet!

But sadly, I can't add to the signal-to-noise ratio, because this website does not collect any email addresses. And even more sadly, it doesn't even collect IP addresses, or use any kind of analytics at all. Sorry about that. I honestly do not know how many people read my stuff, and I can not rank my blog posts by their popularity. And I like it that way. It prevents me from getting on any kind of treadmill to please any kind of imagined audience.

That is… with one exception: Comments on this website are powered by Disqus, and I'm sure Disqus collects all kinds of data about all kinds of things. That's why comments are now hidden behind a "Load Disqus Comments" button. I promise you that no foreign Javascript is executed unless you press that button¹. And honestly, GDPR didn't even have anything to do with that. I just didn't want any foreign JavaScript.

Footnotes:

Unless katex.min.js somehow is loading external javascript…

Appending to Matlab Arrays

Tue, 10 Apr 2018 13:48:00 +0200

The variable $var appears to change size on every loop iteration. Consider preallocating for speed.

So sayeth Matlab.

Let's try it:

x_prealloc = cell(10000, 1);
x_end = {};
x_append = {};
for n=1:10000
    % variant 1: preallocate
    x_prealloc(n) = {42};
    % variant 2: end+1
    x_end(end+1) = {42};
    % variant 3: append
    x_append = [x_append {42}];
end

Which variant do you think is fastest?

Unsurprisingly, preallocation is indeed faster than growing an array. What is surprising is that it is faster by a constant factor of about 2 instead of scaling with the array length. Only appending by x = [x {42}] actually becomes slower for larger arrays. (The same thing happens for numerical arrays, struct arrays, and object arrays.)

TL;DR: Do not use x = [x $something], ever. Instead, use x(end+1) = $something. Preallocation is generally overrated.

Defeating News Addiction

Sat, 07 Apr 2018 10:02:00 +0200

I want to consume the news, both because it is genuinely relevant for my work, and because conversations about news are part of my social life. But I do not want to be consumed by news, and end up scanning news websites over and over for new content, even though you know that the likelihood of finding anything interesting is small.

Over the last few months, I have tried hard to find all instances of this repeated-scanning behavior, and eliminate it. The key is to automate the scanning such that I am only ever presented with new content, but do not get hooked on the addictive variable-reward cycle of checking websites for changes over and over again.

And it all works thank to the magic of RSS:

News Sources: I read several blogs, newspapers, and webcomics. All of them have RSS feeds. Easy.
Hacker News: The brilliant service hnrss.org provides RSS feeds for Hacker News, and filters them to for example only include posts that made it to the front page, and have accumulated at least 100 points.
Reddit: Every subreddit has its own feed, at reddit.com/r/subreddit.rss. Sadly there is no way to filter for a minimum number of upvotes.
YouTube: Again, every YouTube channel has its own RSS feed, but Google is trying very hard to make it as cumbersome as possible to get at those feeds. You need to go to your Subscription Manager, then scroll all the way down, and "Export Subscriptions". The resulting file helpfully does not have a file extension, which you will have to add before you can import it into your RSS reader. I honestly can't reconstruct how I found that subscription manager, either, but presumably there is some series of clicks that would take you there.

With all this settled, I have a veritable firehose of news every day. I estimate that only 1 % of this is actually worth reading. So in the next step, I filter this list for spam. For this purpose, I use Feedbin, which aggegates all these feeds, and remembers whether I have read an article. The remaining ham I either read immediately, or forward it to Pinboard for later consumption.

With this system, I never miss anything, but once I consume all the news in my feed reader, I know I am done, and there is no point in checking and re-checking various websites over and over again.

Scheduling Future TODOs in org-journal

Mon, 02 Apr 2018 11:38:00 +0200

I keep a simple journal in org-journal: One text file per day, in org-mode. But over the years, org-journal has grown somewhat beyond this simple use case. About three years ago, a gentleman named Vladimir Kazanov implemented a very fast text search. Thus, my journal became an information archive. About two years ago, org-journal learned to carry over TODO items to the current day if you hadn't completed them on the previous day. So it to become a to-do list. And today, org-journal gained the ability to work with future journal entries, thus becoming a calendar.

Despite all of these features however, org-journal remains one org file per day, with fancy functions to do fancy things if you want them—or ignore them, if the journal is all you need.

Back to scheduling: This work was prompted by my colleague, who organizes everything in org-mode, but is not a user of org-journal. He even eschews the use of a traditional calendar, and instead uses a few org files and the magic of org-agenda to give him a nice overview like this for the coming week[1]:

Week-agenda (W14):
Monday     2 April 2018 W14
  2018-04-02: Easter Monday 
Tuesday    3 April 2018
Wednesday  4 April 2018
Thursday   5 April 2018
  2018-04-05: Scheduled:  Give Lecture 4 on Applied Programming             :BB:
Friday     6 April 2018
  2018-04-06: Scheduled:  Release of new Eels record
Saturday   7 April 2018
Sunday     8 April 2018
  2018-04-08: Scheduled:  TODO Celebrate Sunday

And lo and behold, this now works in org-journal as well! Just create a new journal entry in the future, either by pressing i j in M-x calendar or by calling org-journal-new-scheduled-entry, and org-journal will create an entry with a SCHEDULED property of the appropriate date (prefix to suppress TODO). When the current day reaches that entry, it will incorporate it into the daily journal.

Future journal entries are highlighted in M-x calendar, and you can get an overview of them with org-journal-schedule-view, or, if you enable org-journal-enable-agenda-integration, through the ordinary org-agenda, as shown above. The agenda integration does not include past journal entries in the agenda, since agenda searches tend to become very slow if they have to traverse the hundreds of files in my journal.

[1]: This is of course not his calendar, but mine.

Blogging with org-static-blog (Updated)

Thu, 22 Mar 2018 08:37:00 +0100

A while ago, I scratched an old itch and wrote my own static site generator, called org-static-blog. It is a simple thing: You hand it a directory full of *.org files with a #+title: and #+date:, and it assembles a bunch of HTML pages and an RSS feed from them. There are no external dependencies beyond Emacs.

Today, I released version 1.1.0 of org-static-blog, which introduces two new features: speed, and tags. You can now—optionally—add #+filetags: to your *.org files, (setq org-static-blog-enable-tags t), and org-static-blog will add tag links to every blog post, create tag indices, and add tags to the RSS feed.

Update:

As Kaushal Modi pointed out in the comments, org-mode uses #+filetags: to set tags for entire files. My first implementation used #+tags:. This is now deprecated. #+tags: still works, but issues a warning, and will be removed in the future. Sorry for the inconvenience.

Speeding up org-static-blog

Sat, 17 Mar 2018 00:00:00 +0100

Three years ago, I had enough of all the static site generators out there. Over the life of this blog, I had used Octopress, then Pelican, then Coleslaw, then org-mode, and then wrote my own static site generator, org-static-blog. Above all, org-static-blog is simple. It iterates over all *.org files in org-static-blog-posts-directory, and then exports all of these files to HTML. Simple is good. Simple is reliable. Simple means I can fix things.

However, simple can also mean inefficient. Most glaringly, org-static-blog exports every single blog post three times every time you publish: Once to render the HTML, then once to render the RSS feed, then once to render the Index and Archive pages.

Today, I finally tackled this problem: Now, org-static-blog only exports each post once, when the *.org file changes. The RSS feed, the Index page, and the Archive page simply read the already-rendered HTML instead of exporting again.

Thus, a full rebuild of this blog and all of its 85 posts used to take 2:12 min, and now takes 42 s. More importantly, if only one org file changed, the rebuild used to take 1:08 min, and now takes 1.5 s. Things like this are hugely satisfying to me!

Files and Processes

Wed, 24 Jan 2018 00:00:00 +0100

In the last few months, I have created three notable programming projects: RunForrest saves function call graphs to disk and runs them in parallel processes; TimeUp creates backups using rsync and keeps different numbers of hourly, daily, and weekly backups; and JBOF, which organizes large collections of data and metadata as structured, on-disk datasets.

These projects have one thing in common: They use Python to interact with external things, such as files, libraries, or processes. It surprised me that none of these projects were particularly hard to build, even though they accomplish "hard" tasks. This prompted some soul-searching about why I thought these tasks to be hard, and I have come up with two observations:

Files and processes are considered "not part of the language", and are therefore not taught. Most programming classes and programming tutorials I have seen focus on the internals of a programming language, i.e. its data structures, and built-in functions and libraries. Files and processes are not part of this, and are often only mentioned in passing, as a thing that a particularly library can do. Worse, many curricula never formally explain files or processes, or their use in building programs.

I now believe that this is unfortunate and misguided, since the interaction with the computer's resources is the central benefit of programming, and you can not make much use of those resources without a thorough understanding of files and processes. In a way, the "inside" of a programming language is a mere sandbox, a safe place for toying with imaginary castles. But it is the "outside", that is, external programs, libraries, and files, that unlocks the true power of making the computer do work. And in Python in particular, using these building blocks to build useful programs is surprisingly simple.
However, such small and simple programs are much less popular than bloated behemoths. I built RunForrest explicitly because Dask was too confusing and unpredictable for the job. I build JBOF because h5py was too complex and slow. And this is surprising, since these tools certainly are vastly more mature than anything I can whip up. But they were developed by large organizations to solve large-organization problems. But I am not a large organization, and my needs are different as well.

I now believe that such small-scale solutions are often preferable to high-profile tools, but they lack the visibility and publicity of tools such as Dask and HDF. And even worse, seeing these big tools solve such mundane problems, we form the belief that these problems must be incredibly complex, and beyond our abilities. Thus forms a vicious cycle. Of course, this is not to say that these big tools do not serve a purpose. Dask and HDF were built to solve particular problems, but we should be aware that most big tools were built for big problems, and our own problems are often not big enough to warrant their use.

In summary, we should teach people about files and processes, and empower them to tackle "hard" tasks without resorting to monolithic libraries. Not only is this incredibly satisfying, it also leads to better programs and better programmers.

How to set up rsnapshot instead of Time Machine

Fri, 29 Dec 2017 00:00:00 +0100

(This blog post was changed since my initial strategy of disabling the lockfile didn't work. Turns out, the lockfile is required, and backups have to be stacked.)

Yesterday, I wrote about how Time Machine has failed me. Time Machine keeps regular backups, going back as far as your hard drive space permits. In theory. In practice, every year or so it messes up somehow and has to start over, thereby deleting all your older backups. A backup that is not reliable is not a backup.

Luckily, there are alternatives. Probably the easiest is rsync[1], a very cool tool that copies files and directories from one place to another. You could simply run this once a day, and have a new backup every day. You can even configure rsync so it doesn't need to copy unchanged files, and instead hard-links them from an older backup. rsnapshot automates this process to keep a number of tiered copies, for example ten hourly backups, seven daily backups, four weekly backups, and a hundred monthly backups. Each backup is then simply a directory that contains your files. No fancy starfield-GUI, but utterly reliable and trivial to understand [2].

Setting up rsnapshot on macOS is not quite as straight-forward as I'd like, and I couldn't find a great guide online. So, without further ado, here's how to configure rsnapshot on macOS:

Install rsnapshot
```
brew install rsnapshot
```

Write the config file

You can copy a template from homebrew:

cp /usr/local/Cellar/rsnapshot/1.4.2/etc/rsnapshot.conf.default /usr/local/etc/rsnapshot.conf

And then configure the new configuration file to your liking (preserve the tabs!):

  config_version	1.2 # default
  verbose		2   # default
  loglevel	3   # default

  # this is where your backups are stored:
  snapshot_root	/Volumes/BBackup/Backups.rsnapshot/ # make sure this is writeable
  # prevent accidental backup corruption:
  lockfile	/Users/bb/.rsnapshot.pid
  # use this if you back up to an external drive:
  no_create_root	1   # don't back up if the external drive is not connected

  # configure how many tiers of backups are created:
  retain	hourly	10
  retain	daily	7   # dailies will only be created once 10 hourlies exist
  retain	weekly	4   # weeklies will only be created once 7 dailies exist
  retain	monthly	100 # monthlies will only be created once 4 weeklies exist

  # the list of directories you want to back up:
  backup	/Users/bb/Documents		localhost/
  backup	/Users/bb/eBooks		localhost/
  backup	/Users/bb/Movies		localhost/
  backup	/Users/bb/Music		localhost/
  backup	/Users/bb/Pictures		localhost/
  backup	/Users/bb/Projects		localhost/
  backup	/Users/bb/Projects-Archive		localhost/

Instead of localhost, you can use remote machines as well. Check man rsync for details

Make sure it works and create initial backup
```
rsnapshot -c /usr/local/etc/rsnapshot.conf hourly
```
The first backup will take a while, but subsequent backups will be fast. A normal backup on my machine takes about two minutes and runs unnoticeably in the background.
Write launchd Agent

Next, we have to tell macOS to run the backups in regular intervals. Conceptually, you do this by writing a launchd agent script[3], which tells launchd when and how to run your backups. In my case, I create four files in /Users/bb/Library/LaunchAgents/, called rsnapshot.{hourly,daily,weekly,monthly}.plist. Apple's documentation for these files is only mildly useful (as usual), but man launchd.plist and man plist should give you an idea how this works.

Here is my hourly launchd agent (I'll explain the bash/sleep thing later):
```
  
  
  
  
      Label
      rsnapshot.hourly
      ProgramArguments
      
          /bin/bash
          -c
          sleep 0 && /usr/local/bin/rsnapshot -c /usr/local/etc/rsnapshot.conf hourly
      
      StartCalendarInterval
      
          Minute
          0
      
  
  
```
For the other four scripts, change the two occurrences of hourly to {daily,weekly,monthly} and change the portion at the end to
- daily:
```
    Minute
    0
    Hour
    0
```
- weekly:
```
    Minute
    0
    Hour
    0
    Weekday
    1
```
- monthly:
```
    Minute
    0
    Hour
    0
    Day
    1
```
However, rsnapshot can only ever run one backup at a time without stepping on its own toes. This is a problem when the computer wakes up, and more than one backup was scheduled during its sleep, since launchd will then happily launch all missed backups at the same time. But only one of them will succeed.

To fix this, I delay the later backup tiers using the sleep 0 directive. I use sleep 900 (15 minutes later) for daily, sleep 1800 (30 minutes), and sleep 2700 (45 minutes) for the lower tiers[4]. It seems that there should be a more elegant solution than this, but I haven't found one.

From the documentation, you might think that Program would be more succinct than supplying the binary as the first argument of ProgramArguments, but this apparently uses a different syntax and does not in fact work as expected.

Load launchd agents

launchctl load ~/Library/LaunchAgents/rsnapshot.*

Test launchd agent
```
launchctl start rsnapshot.hourly
```
If it doesn't work, Console.app might show a relevant error message.
Remove backup directory from Spotlight

Go to System Preferences → Spotlight → Privacy → Add your snapshot_root directory from earlier
Disable TimeMachine and delete your existing backup (if you want)

Start Time Machine, right-click any directory you want to delete, and select "delete all backups of $dir"

[1] rsync is one of those reliable tools I talked about. It is rock solid, incredibly versatile, and unapologetically single-minded. A true gem!

[2] This works great for local backups. If you need encrypted backups or compressed backups (maybe on an untrusted remote machine), this post recommends Borg instead of rsnapshot, but you will lose the simplicity of simple directories.

[3] I use launchd instead of cron since launchd will re-schedule missed backups if the computer was asleep.

[4] This will fail if the hourly backup takes longer than 15 minutes. This is rather unlikely, though, or at least should not happen often enough to be of concern.

Caveats

The configuration file of rsnapshot says that you might experience data corruption if you run several copies of rsnapshot at the same time (and you can use the lockfile to prevent this). This is a problem if your computer is asleep while rsnapshot is scheduled to run, since launchd will then re-schedule all missed tasks at once when the computer wakes up. If you enable the lockfile, only one of them will run.

On the other hand, only the hourly task will actually create a new backup. All higher-level backup tiers merely copy existing backups around, so in theory, they shouldn't step on each other's toes when run concurrently. I have opened an issue asking about this.

There are other possible solutions: ① You could modify the launchd entry such that backups only trigger after a few minutes or, better yet, only once all other instances of rsnapshot have finished. I am not sure if launchd supports this, though. ② You could schedule the hourly task using cron instead of launchd, since cron will not reschedule missed tasks. This would only work for two tiers of backups, though. ③ You could just ignore the issue and hope for the best. After all, if a daily or hourly backup gets corrupted every now and then, you still have enough working backups…

Dropbox deleted my pictures and Time Machine didn't backup

Thu, 28 Dec 2017 00:00:00 +0100

Dropbox deleted some of my favorite photos. Have you looked at all your old pictures lately and checked if they are still there? I have, and they were not. Of course Dropbox denies it is their fault, but no other program routinely accessed my pictures. I am not alone with this problem. It must have happened some time between the summer of 2015, when I put my pictures on Dropbox, and the summer of 2016, when Time Machine last corrupted its backups and had to start over, thereby deleting my last chance of recovering my pictures. The pictures are gone for good.

So, what have I learned? Dropbox loses your data, and Time Machine can't restore it. These programs are obviously no good for backups. Let me repeat this: Dropbox and Time Machine are not a backup! A true backup needs to be reliable, keep an infinite history, and never, never, never accidentally delete files.

From now on, I will use rsnapshot for backups. Here's a tutorial on how to set it up on a Mac. I have used rsnapshot for years at work, and it has never let me down. For syncronizing things between computers, I now use syncthing. Both of these programs are not as user-friendly as Dropbox or Time Machine, but that is a small price to pay for a working backup.

A few years ago, I had high hopes that Apple and Dropbox and Google and Amazon would lead us to a bright future of computers that "just work", and could solve our daily chores ever more conveniently and reliably. But I was proven wrong. So. Many. Times. It seems that for-profit software inevitably becomes less dependable as it adds ever more features to attract ever more users. In contrast, free software can focus on incremental improvements and steadily increasing reliability.

Books of 2017

Wed, 20 Dec 2017 00:00:00 +0100

In late 2016, I took a short ferry flight to a small island in the area, and rekindled my love for aviation. Shortly afterwards, I started training for a pilot's license, and reading about aviation. From a literary perspective, aviation exists in the perfect goldilocks time frame of being just old enough to be thoroughly romanticized, but young enough for first-hand reports and thorough documentation to be available. What is more, powered flight has provided human observers with an unprecedented view of our world and our struggles, and is often as philosophical as it is exhilarating.

Aviation Reading List

Out of a long list of fascinating books on aviation I have read over the last two years, my favorites are:

Fate is the Hunter by Ernest K. Gann, a gripping memoir of the early days of aviation. It has been only a little more than a hundred years since humans first took to the skies in Kitty Hawk in 1905, yet today aviation feels as mundane as horse-drawn carriages must have felt to the Wright Brothers. Gann lived through these early days, and tells his tales from a time when aviation was still young, dangerous, and perhaps more interesting. If you want to read more like this, Flight of Passage by Rinker Buck, and The Spirit of St. Louis by Charles Lindbergh are easy recommendations as well.

Carrying the Fire by Michael Collins, one of the few first-hand accounts of an Apollo astronaut's voyage to the Moon. Most astronauts have published books later on, but most had them ghost-written, and none are as visceral and engaging as Michael Collins' journey on Apollo 8. It is humbling that Apollo's achievements have not been surpassed, despite our much more advanced technology and science. Other accounts worth reading are The Last Man on the Moon by Gene Cernan on Apollo 17, and How Apollo Flew to the Moon by W. David Woods, for a more technical view.

Skyfaring by Mark Vanhoenacker is a more modern, and more philosophical account of how aviation has changed our perception of the world. If you yearn to fly like I do, then this book is a balm for the soul. The almost spiritual feeling of cutting your bonds with the ground is what this book is about, despite being written in today's un-romantic days of routine commercial airliners. I love it dearly. A more grounded account of aviation's history is Turbulent Skies by T. A. Heppenheimer, and maybe Slide Rule by Nevil Chute for some history on airships.

Fiction Reading List

But as much as I love aviation, my first love is still Science Fiction. We live in strange times of unprecedented prosperity, and yet we are strangely unsatisfied, as if the future didn't turn out to be the utopia it was meant to be. Or is this just a reflection of ourselves, how we do not live up to the future we built?

A Closed and Common Orbit by Becky Chambers describes a more distant future, when space travel is as mundane as airliners are today, and humanity is just one of several alien species. Yet, with all its technological marvels, we still yearn for meaning and love, regardless of what strange world we live in. A Closed and Common Orbit is the second book in the series, and the first book I read. I think I prefer it in this order.

The Laundry Files Series by Charles Stross is closer to home. Did you ever notice how computer programming is eerily similar to the arcane incantations we use to describe magic in fiction? Where is the difference between invoking a function that effects a robot, and invoking a magic spell that affects a demon? According to Charles Stross, this difference is really only playing with semantics, and we should be very careful with our incantations, lest the ghost in the machine really does have our demise in mind. These books have made me laugh out loud so many times, like when a weaponized PowerPoint turned people into zombies, or when a structured cabling project turned out to create an inadvertent summoning grid for an elder horror.

PyEnv is the new Conda

Mon, 20 Nov 2017 00:00:00 +0100

How to install Python? If your platform has a package manager, you might be tempted to use that to install Python. But I don't like that. That version is often outdated, and you risk messing with an integral part of your operating system. Instead, I like to install a separate Python in my home directory. I used to use Anaconda (or WinPython, or EPD) to do this. But now there is a better way: PyEnv

The thing is: PyEnv installs (any version of) Python. That's all it does.

So why would I choose PyEnv over the more popular Anaconda? Because Anaconda is a Python distribution, a package manager, an environment manager, and a platform for paid packages. In the past, they once did break pip because they wanted to promote conda instead. Some features of conda require a login, some require a paid subscription. When you install packages through conda, you get binaries and source code from anaconda's servers, not the official packages from PyPi, which might or might not be up-to-date and feature-complete. For every package you install, you have to make a choice of using pip or conda, and the same goes for specifying your dependencies.

As an aside, many of these complaints are just as true for package-manager-provided Python packages (which often break pip, too!). Just like Anaconda, package managers want to be the true and only source of packages, and don't like to interact with Python's own package manager.

In contrast, with PyEnv, you install a Python. This can be a version of CPython, PyPy, IronPython, Jython, Pyston, stackless, miniconda, or even Anaconda. It downloads the sources from the official repos, and compiles them on your machine [1]. Plus, it provides an easy and transparent way of switching between installed versions (including any system-installed versions). After that, you use Python's own venv and pip.

I find this process much simpler, and easier to manage, because it relies on small, orthogonal tools (pyenv, venv, pip) instead of one integrated conda that kind of does everything. I also like to use these official tools and packages instead of conda's parallel universe of mostly-open, mostly-free, mostly-standard replacements.

Mind you, conda solved real problems back in the day (binary package distributions, Python version management, and environment management), and arguably still does (MKL et al, paid packages). But ever since wheels became ubiquitous and painless, and virtualenv was integrated into Python, and the development of PyEnv, these issues now have better solutions, and conda is no longer needed for my applications.

[1] the downside of compilation is: no Windows support.

Installing Emacs on Windows

Sun, 19 Nov 2017 00:00:00 +0100

The official website states that you need to download Emacs from a nearby GNU mirror. However, this does not install gnutls, which is required for installing packages from melpa or marmalade. The documentation says that this can be obtained from ezwinports.

However, I have found that this does not work any more: As of Emacs 25, Emacs is available in 64 bit, but ezwinport only supplies 32 bit binaries. I had to search a bit, but the (in retrospect, obvious) solution is to download the required binaries from GnuTLS's website, directly. Then unpack all *.dll from the bin directory to your Emacs's bin directory, and you are good to go.

This situation is really a bit sad. Installing Emacs should not be this hard. But there is talk about providing a Windows installer for Emacs in one of the next versions, which will hopefully fix these issues.

The Long Game

Tue, 24 Oct 2017 00:00:00 +0200

Here's a thing I do: I write open source libraries that solve my problems. Bad libraries. And then, over the course of a year or two, I slowly improve them, until they are not so bad any more.

I have gone through this pattern a few times now. SoundFile started out buggy and slow. Transplant crashed and froze very often. Org-Journal deleted journal entries every now and then. But then users found errors, and errors were fixed. And slowly, over time, these libraries transformed from buggy prototypes into dependable tools. After a year or so, I saw Github Issues gradually drying up, and feedback changing from "X is broken" to "How can I do X?". It is an oddly gratifying process.

Interestingly though, I rarely hear people talking about this, that building a thing is only the beginning, and the meat of open source development is sticking with it for a long time, and gradually, slowly, weeding out the bugs. All great open source libraries I know evolved this way, yet it is rarely written about.

Playing the long game means anticipating and managing the amount of maintenance I am going to do on a project. When releasing a new library, the first few months and users will shake out bugs, and create work for the maintainers. This can be a dangerous source of stress and anxiety. I've had my brush with burnout in the past, and my only defense to this is to be honest about my availability. Often, this means stating outright that I probably won't have the time to work on a thing.

However, I have also learned that it is OK to defer issues to the future. There have been many problems that I didn't have the time to address when they first came up, but did address a few months later, probably in response to a separate bug report. Important problems have a way of getting fixed eventually. Other times, a solution just takes a long time to mature. There have been multiple issues that lay dormant for months until I stumbled upon a good solution.

And good solutions are important for open source software: If a bug fix adds too much complexity, it will inevitably lead to more bugs. This can easily snowball into a situation where each hour I invest into a project creates more hours of future maintenance burden. A situation like this feels hopeless and terrible. It feels like work. It is therefore important for my sanity to control the complexity of my projects, even if that means not adding features or not addressing problems. Open source development is not beholden to the dollar sign, and gives you the freedom to do it right.

This is not universally understood by programmers or users. And it doesn't just involve technical questions: Some users have expected me to provide commercial-grade support, and got surprisingly unpleasant when I didn't. But it is crucial to realize that publishing open source software does not imply an obligation to fix every issue right now. Quite on the contrary, many open source developers necessarily work on their own schedule, and part of the enjoyment of open source development is derived from this exact freedom!

At this point, my Github lists a bunch of open source projects. Some of them have gotten attention, which lead to bug reports, and, over time, let them grow to maturity. Some turned out to be useful to me, but not to others. Some are useful to others, but not to me. Some turned out to be not useful at all, and are now largely abandoned. And some of them, like RunForrest and SoundCard, are still new and raw and terrible, but will mature over time.

That's a thing I do: I write open source libraries. Good libraries, but as a wise man once said "When you feel how depressingly / slowly you climb, / it's well to remember that / Things Take Time". Working on open source grants you this freedom, and is joyful for it.

Multi-Font Themes in Emacs

Tue, 19 Sep 2017 00:00:00 +0200

Traditionally, text editor themes are all about colors, right? In programming, we use color to tell variables from type declarations, comments, or strings. However, any other text document uses typography instead of color to distinguish between headlines, list items, and keywords. I think that our current approach to highlighting code is misguided.

I think that color themes are an accident of history. Terminal text editors are unable to switch fonts. All they can do is switch colors, so colors is all we use. But in todays graphical world, we are no longer bound by the shackles of terminals, and Emacs can switch fonts just as easily as colors¹.

It all started about a year ago, when I fell in love with a font called PragmataPro. One of the coolest features of Pragmata was its native bold and italic letters, and its wide support for unicode symbols. I had to find a use for these features! And so, down the rabbit hole I went. Keywords should be bold! # Comments should be italic! And while we're at it, why not add some underlines? And so on.

The logical next step was then to get rid of colors altogether. At first, as an experiment. Do I really need colors in code? The very pretty eink theme seemed to claim otherwise. After a few months of this lunacy, I realized that while I didn't strictly need colors, the stylistic variations of just one font aren't quite sufficient for source code. In particular, it wasn't always easy to distinguish between italic and roman type in PragmataPro, which lead to some confusion.

But then, inspiration hit me: Who says that I could only use one single font? No one!

The tricky bit is to find fonts that work well together. In this example, I'm using PragmataPro for all regular code, Iosevka Slab² for strings, and oblique³ Iosevka for comments. Ubuntu Mono and InputCompressed work well, too. You can find my current theme on Github. The only downside is that while these fonts share the character width, the heights differ slightly, which sometimes leads to uneven line heights.

Still, I love the look of this! (Of course it won't work in a terminal, or most other text editors.)

Footnotes:

: Can other text editors do this? I honestly have no idea.

: I just love the look of the slab-serif characters in Iosevka! Look at the beautiful "}" in the screenshot!

: not to be confused with italic, which changes glyphs in addition to slanting.

EuroScipy 2017

Fri, 01 Sep 2017 00:00:00 +0200

The second conference I attended this year was the EuroScipy 2017 in Erlangen. I gave a Talk about Audio in Python and a Lightning Talk about my Python/Matlab bridge.

My most striking impression of EuroScipy is that every person I talked to was working on something interesting, and could talk about his/her topic clearly and with enthusiasm. This mirrors my feelings from last year's Chaos Communication Congress, where the short scientific section stood out for its clarity and passion. I also enjoyed the fact that attendees were international and diverse, and exuded a heart-warming sense of community.

Even though each scientific discipline has their own data sets, features, and models, everyone seemed to use a common set of methods (statistics, signal processing, machine learning) for working with that data. And, absolutely everybody used Jupyter Notebooks for tutorials and teaching, and almost all data analyses were done in Pandas. This is particularly heartening since these technologies are geared towards reproducible research and open data.

The hot topic of the conference clearly was machine learning and neural networks. However, the current confusion of competing frameworks and network architectures does not seem to be a good long-term solution. I hope that this ecosystem will eventually reach its NumPy moment, and collapse into a single, unified package. Then, neural networks might find their place as just another machine learning method with a few reusable parametrized prototype implementations in scikit-learn. Tools like Keras look like good steps towards this goal.

Finally, there was a lot of talk about “the reproducibility crisis” in science, and possible steps to improve the scientific process. In particular, I learned that it is absolutely necessary to not look at your test data before the final evaluation, to avoid overfitting your brain. You need to split your data into a development set for training, a validation set for parameter tuning, and a totally separate evaluation set for the final evaluation. In a similar way, it is important to state your hypotheses in writing before you test them, to avoid “HARKing” (Hypothesis After the Results are Known; aka “Noise Mining”, “P-Hacking”, or “Procedural Overfitting”). I dearly hope that Registered Reports will catch on, and absolve us from these all too human biases.

In conclusion, EuroScipy 2017 was a ton of fun, and educational in many ways that I did not expect. If you are a scientific programmer, or if you maintain a scientific Python module, or if you are plain interested in scientific Python, I can highly recommend going to EuroScipy next year.

Web Audio Conference and JavaScript

Sun, 27 Aug 2017 00:00:00 +0200

I am not much of a web programmer. I have written the odd website, I have supervised a few student projects, but I have never worked on any nontrivial JavaScript code base. Nevertheless, last week, I attended the Web Audio Conference 2017 in London. To put it succinctly: The web is home to fascinating people, but the technology is full of problems.

Those people sure were amazing, though. I talked to a musician/programmer, who spent the last few years writing his own sequencer web app, and did an amazing live performance in his web browser! I witnessed another guy live coding a synthesizer in a web app on stage as a music piece. We attended a gastronomical concert with smartphone-synchronized distributed olfactory and audible experiences. And we played the piano, with each participant controlling one key from his smartphone. It was all truly inspiring!

All of this was built on JavaScript, however. One musical performance reached its climax when the musician queued too many samples and synthesizers and JavaScript crashed. The crowd cheered, but this still highlights how JavaScript, garbage collection pauses, and heap exhaustion are real problems. And there was not a single talk that did not long for dedicated Audio Workers to fix some of these problems.

But more than that, I learned that most of these apps have to pile frameworks upon frameworks just to get a simple demo app going. This is inescapable, since JavaScript is just not usable as an application platform without these frameworks. I take this as a warning: One should build a project from scratch every now and then, just to keep a realistic feel of how deep one's technology stack really is.

All of these people are using JavaScript as an application platform. I would wager that an application platform is basically something like ①: layout, ②: widgets, ③: data bindings, and ④: event handling. If you think about it, JavaScript without libraries currently covers at most ① (CSS) and maybe ② (using HTML Custom Elements), though not at all at the same level as true GUI frameworks such as Qt or Cocoa. For ③ and ④, you have to add something like Ember, React, or Angular. It is strange to me that the JavaScript ecosystem does not seem to gravitate towards integrated frameworks but seems to prefer this wild west of codependent libraries.

Then again, limitations breed genius, and the limits of the platform no doubt fueled the amazing feats these web audio people pulled off at WAC. It is truly inspiring what miracles dedicated people can pull off if they set their mind on them. And let's not forget that the Web Audio API itself is the work of a single person (per browser) as well.

So, in summary, I learned a lot at WAC. I learned that web technologies are not ideal as an application platform. I learned that deep framework stacks are not desirable, yet often necessary. But above all, I learned that all this does not stop people from writing astonishing applications with those technologies. More power to the crazy ones!

Audio APIs, Part 3: WASAPI / Windows

Mon, 10 Jul 2017 14:12:00 +0200

This is part three of a three-part series on the native audio APIs for Windows, Linux, and macOS. This third part is about WASAPI on Windows.

It has long been a major frustration for my work that Python does not have a great package for playing and recording audio. My first step to improve this situation was a small contribution to PyAudio, a CPython extension that exposes the C library PortAudio to Python. However, I soon realized that PyAudio mirrors PortAudio's C API a bit too closely for comfort. Thus, I set out to write PySoundCard, which is a higher-level wrapper for PortAudio that tries to be more pythonic and uses NumPy arrays instead of untyped bytes buffers for audio data. However, I then realized that PortAudio itself had some inherent problems that a wrapper would not be able to solve, and a truly great solution would need to do it the hard way:

Instead of relying on PortAudio, I would have to use the native audio APIs of the three major platforms directly, and implement a simple, cross-platform, high-level, NumPy-aware Python API myself. This effort resulted in PythonAudio, a new pure-Python package that uses CFFI to talk to PulseAudio on Linux, Core Audio on macOS, and WASAPI[1] on Windows.

This series of blog posts summarizes my experiences with these three APIs and outlines the basic structure of how to use them. For reference, the singular use case in PythonAudio is block-wise playing/recording of float data at arbitrary sampling rates and block sizes. All available sound cards should be listable and selectable, with correct detection of the system default sound cards (a feature that is very unreliable in PortAudio).

[1]: WASAPI is part of the Windows Core Audio APIs. To avoid confusion with the macOS API of the same name, I will always to refer to it as WASAPI.

WASAPI

WASAPI is one of several native audio libraries in Windows. PortAudio actually supports five of them: Windows Multimedia (MME), the first built-in audio API for Windows 3.1x; DirectSound, the audio subsystem of DirectX for Windows 95; Windows Driver Model / Kernel Streaming (WDM/KS), the improved audio system for Windows 98; ASIO, a third-party API developed by Steinberg to make pro audio possible on Windows; and finally, Windows Audio Session API (WASAPI), introduced in Windows Vista to bring a modern audio API to Windows.

In other words, audio on Windows has a long and troubled history, and has had a lot of opportunity for experimentation. It should then be no surprise that WASAPI is a clean and well-documented audio API that avoids many of the pitfalls of its predecessors and brethren. After having experienced the audio APIs of Windows, Linux, and macOS, I am beginning to understand why some programmers love Windows.

But let's take a step back, and give an overview over the API. First of all, this is a cross-language API that is meant to be used from C#, with a solid bridge for C++, and a somewhat funky bridge for C. This is crucial to understand. The whole API is designed for a high-level, object-oriented runtime, but I am accessing it from a low-level language that has no concept of objects, methods, or exceptions.

Objects are implemented as pointers to opaque structs, with an associated list of function pointers to methods. Every method accepts the object pointer as its first argument, and returns an error value if an exception occurred. Both inputs and outputs are function arguments, with outputs being implemented as pointer-to-pointer values. While this looks convoluted to a C programmer, it is actually a very clean mapping of object oriented concepts to C that never gave me any headaches.

However, there are a few edge cases that did take me a while to understand: Since the C API is inherently not polymorphic, you sometimes have to manually specify types as cryptic UUID structs. Figuring out how to convert the UUID strings from the header files to such structs was not easy. Similarly, it took me a while to reverse-engineer that strings in Windows are actually uint16, despite being declared char. But issues such as these are to be expected in a cross-language API.

In general, I did not find a good overview on how to interpret high-level C#-concepts in C. For example, it took a long time until I learned that objects in C# are reference counted, and that I would have to manage reference counts manually. Similarly, I had one rather thorny issue with memory allocations: in rare occasions (PROPVARIANT), C# is expected to re-allocate memory of an object if the object does not have enough memory when passed into a method. This does not work as intended if you don't use C#'s memory allocator to create the memory. This was really painful to figure out.

Another result of the API's cross-language heritage are its headers: There are hundreds. And they all contain both the C API and the C++ API, separated by the occasional #ifdef __cplusplus and extern C. Worse yet, pretty much every data type and declaration is wrapped in multiple levels of preprocessor macros and typedef. There are no doubt good reasons and a rich history for this, but it took me many hours to assemble all the necessary symbols from dozens of header files to even begin to call WASAPI functions.

Nevertheless, once these hurdles are overcome, the actual WASAPI API itself is well-structured and reasonably simple. You acquire an IMMDeviceEnumerator, which returns IMMDeviceCollections for microphones and speakers. These contain IMMDevices, which represent sound cards and their properties. You activate an IMMDevice with a desired data format to get an IAudioClient, which in turns produces an IAudioRenderClient or IAudioCaptureClient for playback or recording, respectively. Playback and recording themselves are done by requesting a buffer, and reading or writing raw data to that buffer. This is about as straight-forward as APIs get.

The documentation deserves even more praise: I have rarely seen such a well-documented API. There are high-level overview articles, there is commented example code, every object is described abstractly, and every method is described in detail and in reference to related methods and example code. There is no corner case that is left undescribed, and no error code without a detailed explanation. Truly, this is exceptional documentation that is a joy to work with!

In conclusion, WASAPI leaves me in a situation I am very unfamiliar with: praising Windows. There is a non-trivial impedance mismatch between C and C# that has to be overcome to use WASAPI from C. But once I understood this, the API itself and its documentation were easy to use and understand. Impressive!

Audio APIs, Part 2: Pulseaudio / Linux

Tue, 27 Jun 2017 14:12:00 +0200

This is part two of a three-part series on the native audio APIs for Windows, Linux, and macOS. This second part is about PulseAudio on Linux.

It has long been a major frustration for my work that Python does not have a great package for playing and recording audio. My first step to improve this situation was a small contribution to PyAudio, a CPython extension that exposes the C library PortAudio to Python. However, I soon realized that PyAudio mirrors PortAudio a bit too closely for comfort. Thus, I set out to write PySoundCard, which is a higher-level wrapper for PortAudio that tries to be more pythonic and uses NumPy arrays instead of untyped bytes buffers for audio data. However, I then realized that PortAudio itself had some inherent problems that a wrapper would not be able to solve, and a truly great solution would need to do it the hard way:

[1]: WASAPI is part of the Windows Core Audio APIs. To avoid confusion with the macOS API of the same name, I will always to refer to it as WASAPI.

PulseAudio

PulseAudio is not the only audio API on Linux. There is the grandfather OSS, the more modern ALSA, the more pro-focused JACK, and the user-focused PulseAudio. Under the hood, PulseAudio uses ALSA for its actual audio input/output, but presents the user and applications with a much nicer API and UI.

The very nice thing about PulseAudio is that it is a native C API. It provides several levels of abstraction, the highest of which takes only a handful of lines of C to get audio playing. For the purposes of PythonAudio however, I had to look at the more in-depth asynchronous API. Still, the API itself is relatively simple, and compactly defined in one simple header file.

It all starts with a mainloop and an associated context. While the mainloop is running, you can query the context for sources and sinks (aka microphones and speakers). The context can also create a stream that can be read or written (aka recorded or played). From a high level, this is all there is to it.

Most PulseAudio functions are asynchronous: Function calls return immediately, and call user-provided callback functions when they are ready to return results. While this may be a good structure for high-performance multithreaded C-code, it is somewhat cumbersome in Python. For PythonAudio, I wrapped this structure in regular Python functions that wait for the callback and return its data as normal return values.

Doing this shows just how old Python really is. Python is old-school in that it still thinks that concurrency is better solved with multiple, communicating processes, than with shared-memory threads. With such a mind set, there is a certain impedance mismatch to overcome when using PulseAudio. Every function call has to lock the main loop, and block while waiting for the callback to be called. After that, clean up by decrementing a reference count. This procedure is cumbersome, but not difficult.

What is difficult however, is the documentation. The API documentation is fine, as far as it goes. It could go into more detail with regards to edge cases and error conditions; But it truly lacks high-level overviews and examples. It took an unnecessarily long time to figure out the code path for audio playback and recording, simply because there is no document anywhere that details the sequence of events needed to get there. In the end, I followed some marginally-related example on the internet to get to that point, because the two examples provided by PulseAudio don't even use the asynchronous API.

Perhaps I am missing something, but it strikes me as strange that an API meant for audio recording and playback would not include an example that plays back and records audio.

On an application level, it can be problematic that PulseAudio seems to only value block sizes and latency requirements approximately. In particular, if computing resources become scarce, PulseAudio would rather increase latency/block sizes in the background than risk skipping. This might be convenient for a desktop application, but it is not ideal for signal processing, where latency can be crucial. It seems that I can work around these issues to an extent, but this is an inconvenience nontheless.

In general, I found PulseAudio reasonably easy to use, though. The documentation could use some work, and I don't particularly like the asynchronous programming style, but the API is simple and functional. Out of the three APIs of WASAPI/Windows, Core Audio/macOS, and PulseAudio/Linux, this one was probably the easiest to get working.

Audio APIs, Part 1: Core Audio / macOS

Sat, 17 Jun 2017 14:45:00 +0200

This is part one of a three-part series on the native audio APIs for Windows, Linux, and macOS. This first part is about Core Audio on macOS.

It has long been a major frustration for my work that Python does not have a great package for playing and recording audio. My first step to improve this situation were a small contribution to PyAudio, a CPython extension that exposes the C library PortAudio to Python. However, I soon realized that PyAudio mirrors PortAudio a bit too closely for comfort. Thus, I set out to write PySoundCard, which is a higher-level wrapper for PortAudio that tries to be more pythonic and uses NumPy arrays instead of untyped bytes buffers for audio data. However, I then realized that PortAudio itself had some inherent problems that a wrapper would not be able to solve, and a truly great solution would need to do it the hard way:

This series of blog posts summarizes my experiences with these three APIs and outlines the basic structure of how to use them. For reference, the singular use case in PythonAudio is playing/recording of short blocks of float data at arbitrary sampling rates and block sizes. All connected sound cards should be listable and selectable, with correct detection of the system default sound card (a feature that is very unreliable in PortAudio).

[1]: WASAPI is part of the Windows Core Audio APIs. To avoid confusion with the macOS API of the same name, I will always to refer to it as WASAPI.

CoreAudio, or the Mac's best kept secret

CoreAudio is the native audio library for macOS. It is known for its high performance, low latency, and horrible documentation. After having used the native audio APIs on all three platforms, CoreAudio was by far the hardest one to use. The main problem is lack of documentation and lack of feedback, and plain missing or broken features. Let's get started.

The basic unit of any CoreAudio program is the audio unit. An audio unit can be a source (aka microphone), a sink (aka speaker) or an audio processor (both sink and source). Each audio unit can have several input buses, and several output buses, each of which can have several channels. The meaning of these buses varies wildly and is often underdocumented. Furthermore, every audio unit has several properties, such as a sample rate, block sizes, and a data format, and parameters, which are like properties, but presumably different in some undocumented way.

In order to use an audio unit, you create an AudioComponentDescription that describes whether you want a source or sink unit, or an effect unit, and what kind of effect you want (AudioComponent is an alternative name for audio unit). With the description, you can create an AudioComponentInstance, which is then an opaque struct pointer to your newly created audio unit. So far so good.

The next step is then to configure the audio unit using AudioUnitGetProperty and AudioUnitSetProperty. This is surprisingly hard, since every property can be configured for every bus (sometimes called element) of every input or output of every unit, and the documentation is extremely terse on which of these combinations are valid. Some invalid combinations return error codes, while others only lead to errors during playback/recording. Furthermore, the definition of what constitutes an input or output is interpreted quite differently in different places: One place calls a microphone an input, since it records audio; another place will call it an output, since it outputs audio data to the system. In one crazy example, you have to configure a microphone unit by disabling its output bus 0, and enabling its input bus 1, but then read audio data from its ostensibly disabled output bus 0.

The property interface is untyped, meaning that every property has to be given an identifier, a void pointer that points to a matching data structure, and the size of that data structure. Sometimes the setter allocates additional memory, in which case the documentation does not contain any information on who should free this memory. Most objects are passed around as opaque struct pointers with dedicated constructor and destructor functions. All of this does not strike me as particularly C-like, even though CoreAudio is supposedly a native C library.

Once your audio unit is configured, you set a render callback function, and start the audio unit. All important interaction now happens within that callback function. In a strange reversal of typical control flow, input data to the callback function needs to be fetched by calling AudioUnitRender (evoked on the unit itself) from within the callback, while output is written to memory provided as callback function arguments. Many times during development, AudioUnitRender would return error codes because of an invalid property setting during initialization. Of course, it won't tell which property is actually at fault, just that it can't fulfill the render request at the moment.

Error codes in general are a difficult topic in CoreAudio. Most functions return an error code as an OSStatus value (aka uint32), and the header files usually contain a definition of some, but not all, possible error codes. Sometimes these error codes are descriptive and nice, but often they are way too general. My favorite is the frequent kAudioUnitErr_CannotDoInCurrentContext, which is just about as useless an error description as possible. Worse, some error codes are not defined as numeric constants, but as int err = 'abcd', which makes them un-searchable in the source file. Luckily, this madness can be averted using https://osstatus.com/, which is a dedicated database for OSStatus error codes.

By far the worst part of the CoreAudio API is that some properties are silently ignored. For example, you can set the sample rate or priming information on a microphone unit, and it will accept that property change and it will report that property as changed, but it will still use its default value when recording (aka "rendering" in CoreAudio). A speaker unit, in contrast, will honor the sample rate property, and resample as necessary. If you still need to resample your microphone recordings, you have to use a separate AudioConverter unit, which is its own bag of fun (and only documented in a remark in one overview document).

Lastly, all the online documentation is written for Swift and Objective-C, while the implementation is C. Worse, the C headers contain vastly more information than the online documentation, and the online documentation often does not even reference the C header file name. Of course header files are spread into the CoreAudio framework, the AudioToolkit framework, and the AudioUnit framework, which makes even grepping a joy.

All of that said, once you know what to do and how to do it, the resulting code is relatively compact and readable. The API does contain inconsistencies and questionable design choices, but the real problem is the documentation. I spent way too much time reading the header files over and over again, and searching through (often outdated or misleading) example projects and vague high-level overviews for clues on how to interpret error messages and API documentation. I had somewhat better luck with a few blog posts on the subject, but the general consensus seems to be that the main concepts of CoreAudio are woefully under-explained, and documentation about edge cases is almost nonexistent. Needless to say, I did not enjoy my experience with CoreAudio.

Scientific Code

Sun, 14 May 2017 13:05:00 +0200

A short while ago, I spent a few weeks collecting and evaluating various implementations of speech analysis algorithms for my current work. TL;DR: the general quality of scientific software is bad, and needs to improve.

Let me state up front that I am explicitly not talking about the quality of the science itself. This blog post is exclusively focused on the published software. In general, the problems were bad programming, old and unmaintained code, and lack of documentation. While none of these things mean bad science, they are a real challenge to reproducibility. Follow-up work can not quote algorithms they can't run.

Most scientists are not programmers, and use programming as a tool for doing science. Scientists typically don't conceptualize algorithms as code, and don't reason about their implementation in terms of code. Consequently, executable implementations tend to not read like code, either. And frankly, this is as it should be. In my area of research, programming abstractions are not a great medium for expressing scientific ideas, and programming transformations are weak in comparison to mathematical operations.

Still, the result is bad code. And bad code is a problem for reproducibility and comparability. Scientists therefore need to produce better code that is readable and portable. Straight-up letter-by-letter translations of the published math is just not enough, with all its single-letter variable names and complex nested expressions. We have to do better.

Include an Example Script

I have seen many instances of code that clearly worked at one point on the original author's machine, but doesn't on my computer. Maybe that is because I am running a different version of the programming environment, maybe my data is subtly different, maybe the author forgot to document a dependency. All of these happened to me, and all of these are exactly what you would expect from old, unmaintained code from non-expert programmers.

The least we can do to improve this situation, is to document how the code was supposed to work. A simple example script with example data and example results lets me verify that the code does what it is supposed to be doing. Without this, I can not diagnose errors. Worse, if no example results are included, I might conclude that the algorithm was bad, when in reality I was simply using an incorrect version of a dependency.

Document Dependencies

It is important to document all toolboxes, libraries, and programming languages used, including their exact version and operating system. At one point, I spent several hours debugging some Java code that worked in Java 5, but didn't on any recent Java version. If this had been documented, I could have fixed that error much more quickly. In another case, one piece of C code contained a subtle error if compiled with a 64 bit compiler. Again, this took a long time to track down, and would have been much easier if the original operating system and compiler version had been documented.

That same piece of C code could only be run by compiling several library dependency from old versions of their source code. In that case, the author helpfully included copies of the original source code for these libraries with his source distribution. Without that, I would never have gotten that piece of code to run!

Also, if at all possible, published code should use as few dependencies as possible. Not only might I not have access to that fancy Matlab Toolbox, but every step in the installation instructions is another thing that can go wrong. The more contained the code, the more likely it is that it will still be executable many years later.

Higher Level Languages are better than Lower Level Languages

In general, I had far less problems with non-compiled, high-level code for Matlab, Python, or R, than with low-level C or Java code. On the one hand, this is a technological problem: It is easier to keep an interpreter compatible with outdated code than it is to keep a more complex tool chains with compilers, linkers, and runtime libraries compatible. On the other hand though, this is a human problem as well: I happen to know Python, Matlab, and C pretty well, but I don't know much R or Java. Still, it is much easier to reason about the runtime behavior of R code (because I can inspect variables like in any other dynamic programming language) than to debug some unknown build tool interaction in Java (because compilation tool chains are typically much more complex and variable than REPLs).

Keep it Simple

A particular problem are compiled modules for an interpreted language. Such Mex files and CPython extensions are almost guaranteed to break when programming language versions change, and are often hard to upgrade, since the C APIs often change with the language version as well. Often, these compiled modules are provided for performance reasons. But what was painfully slow on the original author's work laptop a few years ago might not be a problem at all on a future compute cluster. If the code absolutely has to be provided a Mex file or a CPython extension, we should at least provide an interpreted version as well.

And speaking of compute clusters, I have had a lot of trouble with clever tricks such as massively parallelized code or GPU-optimized code. Not only does this age poorly, it also wreaks havoc when trying to run this code on a compute cluster. Such low-level performance optimizations are rarely a good idea for published scientific code, and should at least be optional and well-documented.

Document the Code

Code is never self-documenting. Given enough time, entropy increases inexorably, and times change. Conventions change, and what seemed obvious a few years ago might look like gibberish today. What is perfectly readable to a domain expert is inscrutable even to scientists of closely related areas of research. It is therefore necessary to always document published code, even if the code looks perfectly obvious to the author. In the same vein, we should refrain from using jargon in our code, and clearly declare our variables and invariants (they are bound to be different for different people).

Importantly, this includes documenting all file formats and data structures. Institutions often have in-house standards for how to represent certain kinds of data, and practitioners don't realize that these conventions are not followed universally. I had to ignore a few apparently high-quality algorithms simply because I could not figure out how to supply data to their code. For many other algorithms, I had to write custom data exporters and data importers. Again, an algorithm won't get quoted if it can't be run.

Make it Automatable

Another thing I see frequently are very fancy graphical user interfaces for scientific code. The only thing that breaks faster than compiled language extensions are GUI programs. GUIs are necessarily complex, and therefore hard to debug. And worse yet, if the code can only be run in a GUI, it can't be automated, and I won't be able to compare its performance in a big experiment with hundreds of runs. In effect, GUIs make an algorithm non-reproducible. If a GUI needs to be included in the code, it should at least be made optional.

Do Release Code

However, more importantly than anything I said so far: Do release source code for your algorithm. Nothing is more wasteful than reading about an amazing algorithm that I can't try out. If no source code is released, it is not reproducible, it can't be compared to other algorithms in the field, and it might as well not have been published.

Matlab Metaprogramming

Wed, 12 Apr 2017 00:00:00 +0200

Why is it, that I find Matlab to be a fine teaching tool and a fine tool for solving engineering problems, but at the same time, extremely cumbersome for my own work? Recently, the answer struck me: metaprogramming in Matlab sucks.

Matlab is marketed as a tool for engineers to solve engineering problems. There are convenient data structures for numerical data (arrays, tables), less convenient data structures for non-numeric data (cells, structs, chars), and a host of expensive but powerful functions and methods for working with this kind of data. This is the happy path.

But don't stray too far from the happy path; horrors lurk where The Mathworks don't dare going. Basic stuff like talking to sockets or interacting with other programs is very cumbersome in Matlab, and sometimes even downright impossible for the lack of threads, pipes, and similar infrastructure. But this is common knowledge, and consistent with Matlab's goals as an engineering tool, not a general purpose programming language. These are first-order problems, and they are rarely insurmountable.

The more insidious problem is metaprogramming, i.e. when the objects of your code are code objects themselves. The first order use of programming is to solve real-world problems. If these problems are numeric in nature, Matlab has got you covered. But as every programmer discovers at some point, the second order use of programming is to solve programming problems. And by golly, will Matlab let you down when you try that!

As soon as you climb that ladder of abstraction, and the objects of your code become code objects themselves, you will enter weird country. You think exist will tell you whether a variable name is taken? Try calling it on a method. You think nargout will always give you a number? Again, methods will enlighten you. Quick, how do you capture all output arguments of a function call in a variable? x = cell(nargout(fun), 1); [x{:}] = fun(...), obviously (this sometimes fails). And don't even think of trying to overload subsref to create something generic. Those subsref semantics are crazy talk!

I could go on. The real power of code is abstraction. We use programming to repeatedly and reliably solve similar problems. The logical next step is to use those same programming tools to solve similar kinds of problems. This happens to all of us, and Matlab makes it extremely hard to deal with. Thus, it puts a ceiling on the level of abstraction that is reasonably achievable, and limits engineers to first-order solutions. And after a few years of acclimatization, it will put that same ceiling on those engineers' thinking, because you can't reason about what you can't express.

Pip is great

Thu, 24 Nov 2016 00:00:00 +0100

Installing Python packages used to be a pain. Very surprisingly, this is no longer the case. Nowadays, pip install whatever will reliably install pretty much anything without any trouble.

To recap, the problem is that many Python packages rely on C code, which needs to be compiled before installation. In the past, this burden was mostly on the user. Depending on the user's knowledge of C and compilers, and the user's operating system, this could become almost arbitrarily hairy.

This problem was solved, to some extent, using pre-compiled packages, first as binary installers on the package websites, then pre-packaged Python distributions, and later through third-party package managers such as conda.

This worked well, but it fractured the ecosystem into several different mostly-compatible package sources. This was no big problem, but users had to decide on a package-by-package basis whether to download an installer, use conda, or use pip.

But I am happy to announce that these dark days are behind us now. Pip now automatically installs binary Python packages called wheels, and most package developers do now provide wheels on PyPI. This is so obviously superior to the past that many high-profile packages don't even provide binary installers any more.

For me, this means I don't have to rely on conda any longer. I don't have to keep a mental list of which packages to install though conda and which packages to install through pip any longer. I can go back to using virtualenv instead of conda-envs¹ again. I don't have to tell students to download pre-packaged Python distributions any longer. Pip is great!

Footnotes:

Not because I dislike conda, just to simplify my development environment.

Soma

Tue, 22 Nov 2016 00:00:00 +0100

For a long time, I was afraid of picking up Soma, since it came from the developers who did Amnesia. Amnesia was the first game that scared me so thoroughly that I just couldn't bring myself to pick it up again after the first session. I admired it for its amazing world design, interactivity, and story, but it was too scary for me. I don't enjoy being scared.

I was thus of two minds when I heard about Soma: Soma was supposed to be less scary, and SciFi. But still, let me get this out of the way; This game scared me whitless. I had to use an online game guide to warn me of monsters and tell me how to evade them. I was too scared to figure this stuff out by myself, all alone in this creepy, nasty, underwater space station.

But at the same time, this game delivered one of the best stories I have ever seen in a video game. The story left me shaken and thoughtful for days, and the story was what dragged me back into the game even though I was really struggling with the monster sections. Even the delivery through conversations, environmental clues, and audio logs, was truly outstanding. I just love grimy Alien-style SciFi, and this is one of the very few video game stories that could fill more than a paragraph or two in a well-written book. I love it.

Apparently, if you like horror games, this is a very mild one. Boring even, if you can believe my favorite online critics. But believe me, it was right on the edge of what I was able to take. I still can't quite decide whether I would have preferred this game without the monsters, or whether the monsters were a necessary tool for evoking this intense sense of dread on the lonely SciFi ocean floor. Maybe the monsters lent some much-needed life and interactivity to the walk-em-up formula.

At any rate, I thoroughly enjoyed this experience, and can't wait for what this developer will create next. I wholeheartedly recommend this, as one of the best SciFi stories of this year, in any medium. ★★★★★

MATLAB Syntax

Wed, 07 Sep 2016 00:00:00 +0200

In a recent project, I tried to parse MATLAB code. During this trying exercise, I stumbled upon a few… unique design decisions of the MATLAB language:

Use of apostrophes (`'`)

Apostrophes can mean one of two things: If applied as a unary postfix operator, it means transpose. If used as a unary prefix operator, it marks the start of a string. While not a big problem for human readers, this makes code surprisingly hard to parse. The interesting bit about this, though, is the fact that there would have been a much easier way to do this: Why not use double quotation marks for strings, and apostrophes for transpose? The double quotation mark is never used in MATLAB, so this would have been a very easy choice.

Use of parens (`()`)

If Parens follow a variable name, they can mean one of two things: If the variable is a function, the parens denote a function call. If the variable is anything else, this is an indexing operation. This can actually be very confusing to readers, since it makes it entirely unclear what kind of operation foo(5) will execute without knowledge about foo (which might not be available until runtime). Again, this could have been easily solved by using brackets ([]) for indexing, and parens (()) for function calls.

Use of braces (`{}`) and cell arrays

Cell arrays are multi-dimensional, ordered, heterogeneous collections of things. But in contrast to every other collection (structs, objects, maps, tables, matrices), they are not indexed using parens, but braces. Why? I don't know. In fact, you can index cell arrays using parens, but this only yields a new cell array with only one value. Why would this ever be useful? I have no explanation. This constantly leads to errors, and for the life of me I can not think of a reason for this behavior.

Use of line breaks

In MATLAB, your line can end on one of three characters: A newline character, a semicolon (;), and a comma (,). As we all know, the semicolon suppresses program output, while the newline character does not. The comma ends the logical line, but does not suppress program output. This is a relatively little-known feature, so I thought it would be useful to share it. Except, the meaning of ; and , changes in literals (like [1, 2; 3, 4] or {'a', 'b'; 3, 4}). Here, commas separate values on the same row and are optional, and semicolons end the current row. Interestingly, literals also change the meaning of the newline character: Inside a literal, a newline acts just like a semicolon, overrides a preceding comma, and you don't have to use ellipsis (...) for line continuations.

Syntax rules for commands

Commands are function calls without the parenthesis, like help disp, which is syntactically equivalent to help('disp'). You see, if you just specify a function name (can't be a compound expression or a function handle), and don't use parenthesis, all following words will be interpreted as strings, and passed to the function. This is actually kind of a neat feature. However, how do you differentiate between variable_name + 5 and help + 5? The answer is: Commands are actually a bit more complex. A command starts with a function name, followed by a space, which is not followed by an operator and a space. Thus, help +5 + 4 is a command, while help + 5 + 4 is an addition. Tricky!

The more-than-one-value value

If you want to save more than one value in a variable, you can use a collection (structs, matrices, maps, tables, cell arrays). In addition though, MATLAB knows another way of handling more than one values at once: The thing you get when you index a cell array with {:} or assign a function call with more than one result. In that case, you get something that is assignable to several variables, but that is not itself a collection. Just another quirk of MATLAB's indexing logic. However, you can capture these values into matrices or cell arrays using brackets or braces, like this: {x{:}} or [x{:}]. Note that this also works in assignments in a confusing way: [z{:}] = x{:} (if both x and z have the same length). Incidentally, this is often a neat way of converting between different kinds of collections (but utterly unreadable, because type information is hopelessly lost).

P⋅O⋅L⋅L⋅E⋅N and Walk 'em ups

Sun, 26 Jun 2016 00:00:00 +0200

I like boring games. I like games that give me time to think. Like flight simulators, truck simulators, history simulators, and (don't call them walking simulators) walk 'em ups. And this weekend, this year's Steam Summer Sale started, and thus it was time to get some gaming done!

First, I played Everybody's Gone to the Rapture, a walk 'em up that got good reviews, and was especially praised for it's story. And it dutifully enraptured me, with its British landscapes, and personal story lines. But somehow it didn't quite connect. Maybe I'm not British enough, and I certainly didn't get all of the story. Still, this is well worth picking up, and visiting a digital Shropshire after just having visited the real one earlier this year was a real treat!

Second, I played P⋅O⋅L⋅L⋅E⋅N. It did not start well: The download took ages, I had to unplug all that fancy flight simulation gear to get it to recognize my controller, and performance was rather lackluster. Oh, and reviews were rather mediocre, too. (Also, here's a free tip for you if you games developers: putting fancy Unicode characters into your game title does not improve SEO).

But, I entirely fell in love with P⋅O⋅L⋅L⋅E⋅N! You arrive on Titan in your dinky near-future space capsule, take a short walk on the surface of the moon, and then explore the local space station. Of course, something went wrong, and the space station is deserted. Oh, and there is something about space bees.

What really worked for me here was the environmental story telling. This is a space station much in the tradition of 2001: A Space Odyssey and Alien. It's that perfect utilitarian and inhumanely sterile work place that human corporation always strive for, and human employees always mess up. The stark industrial bleakness of these space corridors is the perfect backdrop to make those little human irregularities really stand out.

There is no conflict, only light puzzles, and the story is developed only through voice diaries and environmental clutter. But in contrast to Rapture, it all makes sense. The futurist setting, the deep humanity of the people dealing with it. The creepiness of machines without operators. The struggle of the scientists and engineers against the forces of nature and greedy corporations. The nods —nay— genuflections to everything that has ever been great in Scifi. Even that Kubrick ending. I loved every inch of this! ★★★★★

Transplant, revisited

Tue, 21 Jun 2016 00:00:00 +0200

A few months ago, I talked about the performance of calling Matlab from Python. Since then, I implemented a few optimizations that make working with Transplant a lot faster:

The workload consisted of generating a bunch of random numbers (not included in the times), and sending them to Matlab for computation. This task is entirely dominated by the time it takes to transfer the data to Matlab (see table at the end for intra-language benchmarks of the same task).

As you can see, the new Transplant is significantly faster ~~for small workloads, and still a factor of two faster for larger amounts of data~~. It is now almost always a faster solution than the Matlab Engine for Python (MEfP) and Oct2Py. ~~For very large datasets, Oct2Py might be preferable, though.~~

This improvement comes from three major changes: Matlab functions are now returned as callable objects instead of ad-hoc functions, Transplant now uses MsgPack instead of JSON, and loadlibrary instead of a Mex file to call into libzmq. All of these changes are entirely under the hood, though, and the public API remains unchanged.

The callable object thing is the big one for small workloads. The advantage is that the objects will only fetch documentation if __doc__ is actually asked for. As it turns out, running help('funcname') for every function call is kind of a big overhead.

Bigger workloads however are dominated by the time it takes Matlab to decode the data. String parsing is very slow in Matlab, which is a bad thing indeed if you're planning to read a couple hundred megabytes of JSON. Thus, I replaced JSON with MsgPack, which eliminates the parsing overhead almost entirely. JSON messaging is still available, though, if you pass msgformat='json' to the constructor. Edit: Additionally, binary data is no longer encoded as base64 strings, but passed directly through MsgPack. This yields about a ten-fold performance improvement, especially for larger data sets.

Lastly, I rewrote the ZeroMQ interaction to use loadlibrary instead of a Mex file. This has no impact on processing speed at all, but you don't have to worry about compiling that C code any more.

Oh, and Transplant now works on Windows!

Here is the above data again in tabular form:

Task	New Transplant	Old Transplant	Oct2Py	MEfP	Matlab	Numpy	Octave
startup	4.8 s	5.8 s	11 ms	4.6 s
sum(randn(1,1))	3.36 ms	34.2 ms	29.6 ms	1.8 ms	9.6 μs	1.8 μs	6 μs
sum(randn(1,10))	3.71 ms	35.8 ms	30.5 ms	1.8 ms	1.8 μs	1.8 μs	9 μs
sum(randn(1,100))	3.27 ms	33.9 ms	29.5 ms	2.06 ms	2.2 μs	1.8 μs	9 μs
sum(randn(1,1000))	4.26 ms	32.7 ms	30.6 ms	9.1 ms	4.1 μs	2.3 μs	12 μs
sum(randn(1,1e4))	4.35 ms	34.5 ms	30 ms	72.2 ms	25 μs	5.8 μs	38 μs
sum(randn(1,1e5))	5.45 ms	86.1 ms	31.2 ms	712 ms	55 μs	38.6 μs	280 μs
sum(randn(1,1e6))	44.1 ms	874 ms	45.7 ms	7.21 s	430 μs	355 μs	2.2 ms
sum(randn(1,1e7))	285 ms	10.6 s	643 ms	72 s	3.5 ms	5.04 ms	22 ms

Teaching with Matlab Live Scripts

Wed, 15 Jun 2016 00:00:00 +0200

For a few years now, I have been teaching programming courses using notebooks. A notebook is an interactive document that can contain code, results, graphs, math, and prose. It is the perfect teaching tool:

You can combine introductory resources with application examples, assignments, and results. And after the lecture, students can refer to these notebooks at their leisure, and re-run example code, or try different approaches with known data.

The first time I saw this was with the Jupyter notebook (née IPython notebook). I immediately used it for teaching an introductory programming course in Python.

Later, I took over a Matlab course, but Matlab lacked a notebook. So for the next two years of teaching Matlab, I hacked up a small IPython extension that allowed me to run Matlab code in an Jupyter notebook as a cell magic.

Now, with 2016a, Matlab introduced Live Scripts, which is Mathworkian for notebook. This blog post is about how Live Scripts compare to Jupyter notebooks.

First off, Live Scripts work. The basic functionality is there: Code, prose, figures, and math can be saved in one document; The notebook can be exported as PDF and HTML, and Students can download the notebook and play with it. This latter part was not possible with my homegrown solution earlier.

However, Live Scripts are new, and still contain a number of bugs. You can't customize figure sizes, formatting options are very basic, image rendering is terrible, and math rendering using LaTeX is of poor quality and limited. Also, using Live Scripts on a retina Mac is borderline impossible: Matlab crashes on screen resolution changes (i.e. connecting a projector), Live Scripts render REALLY slowly (type a word, watch the characters crawl onto the screen one by one), and all figures export in twice their intended size (fixed in 2016b). You can work around some of these issues by starting Matlab in Low Resolution Mode.

No doubt some of these issues are going to get addressed in future releases. 2016b added script-local functions, which I read mostly as "you can now write functions in Live Scripts", and autocorrection-like text replacements that convert Markdown formatting into formatted text. This is highly appreciated.

Additionally though, here are a few features I would love to see:

Nested lists, and lists entries that contain newlines (i.e. differentiate between line breaks and paragraph breaks).
Indented text, for quoting things, or to work around the lack of multi-line list entries.
More headline levels.
Magics. This is probably a long shot, but line/cell magics in Jupyter notebooks are really useful.

Still, all griping aside, I want to reiterate that Live Scripts work. They aren't quite as nice as Jupyter notebooks, but they serve their purpose, and are a tremendously useful teaching tool.

Matlab has an FFI and it is not Mex

Tue, 31 May 2016 11:00:00 +0200

Sometimes, you just have to use C code. There's no way around it. C is the lingua franca and bedrock of our computational world. Even in Matlab, sometimes, you just have to call into a C library.

So, you grab your towel, you bite the bullet, you strap into your K&R, and get down to it: You start writing a Mex file. And you curse, and you cry, because writing C is hard, and Mex doesn't exactly make it any better. But you know what? There is a better way!

Because, unbeknownst to many, Matlab includes a Foreign Function Interface. The technique was probably pioneered by Common Lisp, and has since been widely adopted everywhere: calling functions in a C library without writing any C code and without invoking a compiler!

Mind you, there remains a large and essential impedance mismatch between C's statically typed calling conventions and the vagaries of a dynamically typed language such as Matlab, and even the nicest FFI can't completely hide that fact. But anything is better than the abomination that is Mex.

So here goes, a very simple C library that adds two arrays:

// test.c:
#include test.h

void add_arrays(float *out, float* in, size_t length) {
    for (size_t n=0; n
void add_arrays(float *out, float *in, size_t length);

Let's compile it! gcc -shared -std=c99 -o test.so test.c will do the trick.

Now, let's load that library into Matlab with loadlibrary:

if not(libisloaded('test'))
    [notfound, warnings] = loadlibrary('test.so', 'test.h');
    assert(isempty(notfound), 'could not load test library')
end

Note that loadlibrary can't parse many things you would commonly find in header files, so you will likely have to strip them down to the bare essentials. Additionally, loadlibrary doesn't throw errors if it can't load a library, so we always have to check the notfound output argument to see if the library was actually loaded successfully.

With that, we can call functions in that library using calllib. But we can't just pass in Matlab vectors, that would be too easy. We first have to convert them to something C can understand: Pointers

vector1 = [1 2 3 4 5];
vector2 = [9 8 7 6 5];

vector1ptr = libpointer('singlePointer', vector1);
vector2ptr = libpointer('singlePointer', vector2);

What is nice about this is that this automatically converts the vectors from double to float. What is less nice is that it uses its weird singlePtr notation instead of the more canonical float* that you would expect from a self-respecting C header.

Then, finally, let's call our function:

calllib('test', 'add_arrays', vector1ptr, vector2ptr, length(vector1));

If you see no errors, everything went smoothly, and you will now have changed the content of vector1ptr, which we can have a look at like this:

added_vectors = vector1ptr.Value;

Note that this didn't change the contents of vector1, only of the newly created pointer. So there will always be some memory overhead to this technique in comparison to Mex files. However, runtime overhead seems pretty fine:

timeit(@() calllib('test', 'add_arrays', vector1ptr, vector2ptr, length(vector1)))
%   ans = 1.9155e-05
timeit(@() the_same_thing_but_as_a_mex_file(single(vector1), single(vector2)))
%   ans = 4.6262e-05
timeit(@() the_same_thing_plus_argument_conversion(vector1, vector2))
%   ans = 1.2326e-04

So as you can see, the calllib is plenty fast. However, if you add the Matlab code for converting the double arrays to pointers and extracting the summed data afterwards, the FFI is noticeably slower than a Mex file.

However, If I ask myself whether I would sacrifice 0.00007 seconds of computational overhead for hours of my life not spent with Mex, there really is no competition. I will choose Matlab's FFI over writing Mex files every time.

Updating the Matplotlib Font Cache

Mon, 30 May 2016 16:30:00 +0200

When publishing papers or articles, I want my plots to integrate with the text surrounding them. I want them to use the correct font size, and the correct font.

This is easy to do with Matplotlib:

import matplotlib
matplotlib.rcParams['font.size'] = 12
matplotlib.rcParams['font.family'] = 'Calibri'

However, sometimes, Matplotlib won't find the correct, even though it is clearly installed. This happens when Matplotlib's internal font cache is out of date.

To refresh the font cache, use

matplotlib.font_manager._rebuild()

Happy Plotting!

The Style of Scientific Code

Sun, 13 Mar 2016 09:00:00 +0100

What does quality code look like? One common school of thought focuses on small, descriptive functions that take few arguments. To quote from Clean Code: "The first rule of functions is that they should be small", on the order of less than ten lines. "Functions should not be large enough to hold nested structures". "The ideal number of arguments for a function is zero [, one, or two]. Three [or more] arguments should be avoided where possible".

A few years ago, when I was working mostly on user interaction and data management, all of this made sense to me. What is the overhead of a few function calls and class lookups here and there if it makes the code more readable? In other words: Readability counts, and is usually more important than performance.

But lately, I have come to struggle with these rules. I am now writing a lot of scientific code, where algorithms are intrinsically complex beyond the syntactic complexity of the code. How do you "Express yourself in code [instead of comments]", when that code only consists of linear algebra and matrix multiplications?

def rectwin_spectrum(angular_frequency, specsize):
    """The spectrum of a rectangular window. [...]"""
    # In case of angular_frequency == 0, this will calculate NaN. Since
    # this will be corrected later, suppress the warning.
    with np.errstate(invalid='ignore'):
        spectrum = ( np.exp(-1j*angular_frequency*(specsize-1)/2) *
                     np.sin(specsize*angular_frequency/2) /
                     np.sin(angular_frequency/2) )
    # since sin(x) == x for small x, the above expression
    # evaluates to specsize for angular_frequency == 0.
    spectrum[angular_frequency == 0.0] = specsize
    return spectrum

A lot of my scientific code ends up quite compact like that. Maybe a hundred lines of dense numeric expressions, plus a few hundred lines of explanations and documentation. The point is, scientific code often does not decompose into easily understood extractable functions.

On a related issue, how do you avoid long argument lists in heavily parametrized equations? As Clean Code states, "when a function seems to need more than two or three arguments, it is likely that some of those arguments ought to be wrapped in a class of their own". However, in Matlab in particular, it is quite unusual to create small one-trick classes to encapsulate a few function arguments:

classdef SignalBlocks < handle
    properties
        data
        samplerate
        blocksize
        hopsize
    end
    properties (Dependent)
        duration
    end
    methods
        function obj = SignalBlock(data, samplerate, blocksize, hopsize)
            % blocksize and hopsize are optional. What a mess.
            narginchk(2, 4);
            obj.data = data;
            obj.samplerate = samplerate;
            if nargin >= 3
                obj.blocksize = blocksize;
            else
                obj.blocksize = 2048;
            end
            if nargin == 4
                obj.hopsize = hopsize;
            else
                obj.hopsize = 1024;
            end
        end
        function time = get.duration(obj)
            time = length(obj.data)/obj.samplerate;
        end
    end
end

This is not just cumbersome to write and maintain, it is also slower than passing data, samplerate, blocksize, and hopsize to each function call individually (although the overhead has gotten considerably smaller in newer versions of Matlab). Additionally, there is often a large performance benefit of not extracting every function and not keeping intermediate values in variables. Thus, it's not just readability that is hard to maintain in scientific code. Performance is a problem, too.

The sad thing is, I don't know the answer to these questions. There have been a lot of discussions about coding style and code quality in our department lately, with the clear objective to clean up our code. But common code quality criteria don't seem to apply to scientific code all that well.

Do you have any idea how to progress from here?

Toren

Sat, 23 Jan 2016 00:00:00 +0100

I've been playing a lot of indie games lately. One of them has not been talked about much: Toren. Toren is a platformer about a girl that has to climb a tower to defeat a dragon and revive her world. This is probably the least polished game I have played in a long time. Animations are janky, controls are imprecise and clunky, and there are loads of little glitches. Yet, I really enjoyed this.

There is something about this world that feels honest to me: As you climb the tower, the child grows from a toddler to an adolescent, and is gradually introduced to more and more mature concepts. I didn't understand much of the iconography of this game, but it felt oddly cathartic to climb this tower of life, and overcome it's challenges.

I particularly liked how death played such an integral role in this story and some of the puzzles. The tower is a monument to a dead people, and yet the story and game mechanics are as much about dying as they are about rebirth and not giving up. This is underlined by the wonderful art style of this game, which contrasts vivid colors with brooding, dark architecture.

At just about two hours, Toren is not a long game. Instead of exploring one particular game mechanic, it mixes it's game up every few minutes. Every sequence looks different and beautiful, and yet it manages to tell a cohesive and effective story. ★★★★☆

Books of 2015

Wed, 06 Jan 2016 00:00:00 +0100

Among Others, by Jo Walton

I don't usually enjoy fantasy novels and their romantic escapism. I much prefer fascinating sci-fi thought experiments. But this book won all the most important awards, so I gave it a shot. What if random chance could be bent a little with creativity, the power of believing in something, and some mysticism? You end up with a world that is richer, more meaningful, and altogether more alive, if you just cared to observe and to appreciate it's beauty. Reading this book left me enchanted and more observant long after I put it down. What a wonderful book!

A Son of the Circus, by John Irving

This is one of those books that was on my to-read list for months. It starts out as quirky and likeable as you would expect from John Irving. This time, we follow the tale of a Canadian/Indian doctor throughout his life, and his summer vacation in India. But this would not be John Irving if there weren't plenty of colourful characters, astute observations of human strangeness, and a meticulously crafted story. There is no scene in this book that does not serve a purpose, and so many moving parts my mind just boggles at the construction of it all. Yet at the same time, I was regularly laughing out loud. I loved every minute of this!

The Windup Girl, by Paolo Bacigalupi

The whole world changed in the near future, when gasoline is a rare luxury, sea levels have risen and swallowed all the coastal cities, and man-made scourges have devastated most crops. And now it's not just humans that populate our urbanized world, but so too are our inventions, artificial humans called "windups" for their stutter-stop movements. But at the core, both humans and windups struggle for the same security, prosperity as ever. Such an inventive world, so much vivid creativity, social commentary, in this human struggle to not destroy ourselves.

Zen and the art of motorcycle maintenance, by Robert Pirsig

This is about equal parts a motorcycle journey of a father and his son across the US, and a dive into another man's discoveries of philosophy. To be honest, I liked this book more for it's character descriptions and travelling adventures than it's philosophy. I am really conflicted about putting this book on this list at all, but I kept thinking about this long after I finished reading it, so I guess this had a bigger influence on me than I realized.

Nexus/Crux/Apex, Rad/Blue/Green Mars, The Martian, The Three-Body Problem

What happens when you take today's world, and add nanotech brain upgrades (Nexus/Crux/Apex), or strand a lone scientist on Mars (The Martian), or send a large number of people to found a new colony on Mars (Red/Blue/Green Mars), or suddenly make contact with Aliens (The Three-Body Problem)? This what-if is what Science Fiction does best: Take this little what-if, and spin a gripping yarn from that. These books inspired me, made me think, were incredibly thrilling, but they did not have a lasting impact. Still well worth a read if you like Science Fiction, though.

Calling Matlab from Python

Tue, 03 Nov 2015 15:05:00 +0100

For my latest experiments, I needed to run both Python functions and Matlab functions as part of the same program. As I noted earlier, Matlab includes the Matlab Engine for Python (MEfP), which can call Matlab functions from Python. Before I knew about this, I created Transplant, which does the very same thing. So, how do they compare?

Usage

As it's name suggests, Matlab is a matrix laboratory, and matrices are the most important data type in Matlab. Since matrices don't exist in plain Python, the MEfP implements it's own as matlab.double et al., and you have to convert any data you want to pass to Matlab into one of those. In contrast, Transplant recognizes the fact that Python does in fact know a really good matrix engine called Numpy, and just uses that instead.

       Matlab Engine for Python        |              Transplant
---------------------------------------|---------------------------------------
import numpy                           | import numpy
import matlab                          | import transplant
import matlab.engine                   |
                                       |
eng = matlab.engine.start_matlab()     | eng = transplant.Matlab()
numpy_data = numpy.random.randn(100)   | numpy_data = numpy.random.randn(100)
list_data = numpy_data.tolist()        |
matlab_data = matlab.double(list_data) |
data_sum = eng.sum(matlab_data)        | data_sum = eng.sum(numpy_data)

Aside from this difference, both libraries work almost identical. Even the handling of the number of output arguments is (accidentally) almost the same:

       Matlab Engine for Python        |              Transplant
---------------------------------------|---------------------------------------
eng.max(matlab_data)                   | eng.max(numpy_data)
>>> 4.533                              | >>> [4.533 537635]
eng.max(matlab_data, nargout=1)        | eng.max(numpy_data, nargout=1)
>>> 4.533                              | >>> 4.533
eng.max(matlab_data, nargout=2)        | eng.max(numpy_data, nargout=2)
>>> (4.533, 537635.0)                  | >>> [4.533 537635]

Similarly, both libraries can interact with Matlab objects in Python, although the MEfP can't access object properties:

       Matlab Engine for Python        |              Transplant
---------------------------------------|---------------------------------------
f = eng.figure()                       | f = eng.figure()
eng.get(f, 'Position')                 | eng.get(f, 'Position')
>>> matlab.double([[ ... ]])           | >>> array([[ ... ]])
f.Position                             | f.Position
>>> AttributeError                     | >>> array([[ ... ]])

There are a few small differences, though:

Function documentation in the MEfP is only available as eng.help('funcname'). Transplant will populate a function's __doc__, and thus documentation tools like IPython's ? operator just work.
Transplant converts empty matrices to None, whereas the MEfP represents them as matlab.double([]).
Transplant represents dict as containers.Map, while the MEfP uses struct (the former is more correct, the latter arguable more useful).
If the MEfP does not know nargout, it assumes nargout=1. Transplant uses nargout(func) or returns whatever the function writes into ans.
The MEfP can't return non-scalar structs, such as the return value of whos. Transplant can do this.
The MEfP can't return anonymous functions, such as eng.eval('@(x, y) x>y'). Transplant can do this.

Performance

The time to start a Matlab instance is shorter in MEfP (3.8 s) than in Transplant (6.1 s). But since you're doing this relatively seldomly, the difference typically doesn't matter too much.

More interesting is the time it takes to call a Matlab function from Python. Have a look:

This is running sum(randn(n,1)) from Transplant, the MEfP, and in Matlab itself. As you can see, the MEfP is a constant factor of about 1000 slower than Matlab. Transplant is a constant factor of about 100 slower than Matlab, but always takes at least 0.05 s.

There is a gap of about a factor of 10 between Transplant and the MEfP. In practice, this gap is highly significant! In my particular use case, I have a function that takes about one second of computation time for an audio signal of ten seconds (half a million values). When I call this function with Transplant, it takes about 1.3 seconds. With MEfP, it takes 4.5 seconds.

Transplant spends its time serializing the arguments to JSON, sending that JSON over ZeroMQ to Matlab, and parsing the JSON there. Well, to be honest, only the parsing part takes any significant time, overall. While it might seem onerous to serialize everything to JSON, this architecture allows Transplant to run over a network connection.

It is a bit baffling to me that MEfP manages to be slower than that, despite being written in C. Looking at the number of function calls in the profiler, the MEfP calls 25 functions (!) on each value (!!) of the input data. This is a shockingly inefficient way of doing things.

TL;DR

It used to be very difficult to work in a mixed-language environment, particularly with one of those languages being Matlab. Nowadays, this has thankfully gotten much easier. Even Mathworks themselves have stepped up their game, and can interact with Python, C, Java, and FORTRAN. But their interface to Python does leave something to be desired, and there are better alternatives available.

If you want to try Transplant, just head over to Github and use it. If you find any bugs, feature requests, or improvements, please let me know in the Github issues.

Massive Memory Leak in the Matlab Engine for Python

Thu, 29 Oct 2015 12:18:00 +0100

As of Matlab 2014b, Matlab includes a Python module for calling Matlab code from Python. This is how you use it:

import numpy
import matlab
import matlab.engine

eng = matlab.engine.start_matlab()
random_data = numpy.random.randn(100)
# convert Numpy data to Matlab:
matlab_data = matlab.double(random_data.tolist())
data_sum = eng.sum(matlab_data)

You can call any Matlab function on eng, and you can access any Matlab workspace variable in eng.workspace. As you can see, the Matlab Engine is not Numpy-aware, and you have to convert all your Numpy data to Matlab double before you can call Matlab functions with it. Still, it works pretty well.

Recently, I ran a rather large experiment set, where I had a set of four functions, two in Matlab, two in Python, and called each of these functions a few thousand times with a bunch of different data to see how they performed.

While doing that I noticed that my Python processes were growing larger and larger, until they consumed all my memory and a sizeable chunk of my swap as well. I couldn't find any reason for this. None of my Python code cached anything, and the sum total of all global variables did not amount to anything substantial.

Enter Pympler, a memory analyzer for Python. Pympler is an amazing library for introspecting your program's memory. Among its many features, it can list the biggest objects in your running program:

from pympler import muppy, summary
summary.print_(summary.summarize(muppy.get_objects()))

                                      types |   # objects |   total size
=========================================== | =========== | ============
                        


Now that is interesting. Apparently, I was lugging around close to three gigabytes worth of bare-Python array.array. And these are clearly not Numpy arrays, since those would show up as numpy.ndarray. But I couldn't find any of these objects in my workspace.



So let's get a reference to one of these objects, and see who they belong to. This can also be done with Pympler, but I prefer the way objgraph does it:



import array
# get a list of all objects known to Python:
all_objects = muppy.get_objects()
# sort out only `array.array` instances:
all_arrays = [obj for obj in all_objects if isinstance(obj, array.array)]

import objgraph
objgraph.show_backrefs(all_arrays[0], filename='array.png')










It seems that the array.array object is part of a matlab.double instance which is not referenced from anywhere but all_objects. A memory leak.



After a bit of experimentation, I found the culprit. To illustrate, here's an example: The function leak passes some data to Matlab, and calculates a float. Since the variables are not used outside of leak, and the function does not return anything, all variables within the function should get deallocated when leak returns.



def leak():
    test_data = numpy.zeros(1024*1024)
    matlab_data = matlab.double(test_data.tolist())
    eng.sum(matlab_data)




Pympler has another great feature that can track allocations. The SummaryTracker will track and display any allocations between calls to print_diff(). This is very useful to see how much memory was used during the call to leak:



from pympler import tracker
tr = tracker.SummaryTracker()
tr.print_diff()
leak()
tr.print_diff()



                     types |   # objects |   total size
========================== | =========== | ============
       


And there you have it. Note that this leak is not the Numpy array test_data and it is not the matlab array matlab_data. Both of these are garbage collected correctly. But the Matlab Engine for Python will leak any data you pass to a Matlab function.



This data is not referenced from anywhere within Python, and is counted as leaked by objgraph. In other words, the C code inside the Matlab Engine for Python copies all passed data into it's internal memory, but never frees that memory. Not even if you quit the Matlab Engine, or del all Python references to it. Your only option is to restart Python.



Postscriptum



I since posted a bug report on Mathworks, and received a patch that fixes the problem. Additionally, Mathworks said that the problem only occurs on Linux.

Tags: matlab

OS X Finder Woes

Fri, 16 Oct 2015 09:51:00 +0200

The Mac. It used to be the most streamlined, thought-through general computing device on the market.

Even it's file management used to be top-notch. There were many cool little touches. One particularly useful feature was the Proxy Icon–if a window displayed a file's content, that file's icon would show up in the window's title. And you could drag that icon directly onto a thumb drive or email, without having to use the Finder. But the Finder, too, had many neat little features. I loved the fact that when you renamed a file in an alphabetically sorted file list, Finder would not immediately re-shuffle it to its new location, but would wait half a second before doing so. When renaming multiple files, this was really useful, since you could go through them one by one and rename them, simply by pressing arrow keys and return.

But as you might have guessed from my use of the past tense, these golden days are gone. The Finder used to know a JPEG from a ZIP regardless of file extension. Now it doesn't any more. The Proxy Icon is still draggable, but it will create an alias instead of a copy–perfectly useless on a thumb drive or in an email.

And with the newest version of OS X, El Capitan, they finally blew it for me. Before, even though the Finder inexplicably never had the ability to cut and paste files, you could always install programs like TotalFinder to fix that. Not so with El Capitan. The Finder now is holy land, and can not be touched any more by third parties. So no more cut and paste, no more un-hiding system files. No more side-by-side Finder tabs. And brand new with El Capitan as well: No more waiting after renaming. Now, when you rename a file, it is immediately re-sorted to its new position, thus making renaming multiple files terribly inconvenient.

So, good bye OS X. I updated my work laptop first, and I regret it. I never regretted an OS X update before. My home machine is not going to get the update. It is honestly sad to see my once-beloved Mac platform becoming worse and worse and worse with every new release.

Changing File Creation Dates in OSX

Sat, 03 Oct 2015 00:00:00 +0200

On my last vacation, I have taken a bunch of pictures, and a bunch of video. The problem is, I hadn't used the video camera in a long time, and it believed that all it's videos were taken on the first of January 2012. So in order for the pictures to show up correctly in my picture library, I wanted to correct that.

For images, this is relatively easy: Most picture libraries support some kind of bulk date changes, and there are a bunch of command line utilities that can do it, too. But none of these tools work for video (exiftool claims be able to do that, but I couldn't get it to work).

So instead, I went about to change the file creation date of the actual video files. And it turns out, this is surprisingly hard! The thing is, most Unix systems (a Mac is technically a Unix system) don't even know the concept of a file creation date. Thus, most Unix utilities, including most programming languages, don't know how to deal with that, either.

If you have XCode installed, this will come with SetFile, a command line utility that can change file creation dates. Note that SetFile can change either the file creation date, or the file modification date, but not both at the same time, as any normal Unix utility would. Also note that SetFile expects dates in American notation, which is about as nonsensical as date formats come.

Anyway, here's a small Python script that changes the file creation date (but not the time) of a bunch of video files:

import os.path
import os
import datetime
# I want to change the dates on the files GOPR0246.MP4-GOPR0264.MP4
for index in range(426, 465):
    filename = 'GOPR0{}.MP4'.format(index)
    # extract old date:
    date = datetime.datetime.fromtimestamp(os.path.getctime(filename))
    # create a new date with the same time, but on 2015-08-22
    new_date = datetime.datetime(2015,  8, 22, date.hour, date.minute, date.second)
    # set the file creation date with the "-d" switch, which presumably stands for "dodification"
    os.system('SetFile -d "{}" {}'.format(new_date.strftime('%m/%d/%Y %H:%M:%S'), filename))
    # set the file modification date with the "-m" switch
    os.system('SetFile -m "{}" {}'.format(new_date.strftime('%m/%d/%Y %H:%M:%S'), filename))

Numpy Broadcasting Rules

Tue, 29 Sep 2015 00:00:00 +0200

They say that all arithmetic operations in Numpy behave like their element-wise cousins in Matlab. This is wrong, and seriously tripped me up last week.

In particular, this is what happens when you multiply an array with a matrix[1] in Numpy:

     [[  1],           [[1, 2, 3],       [[ 1,    2,   3],
      [ 10],       *    [4, 5, 6],   =    [ 40,  50,  60],
      [100]]            [7, 8, 9]]        [700, 800, 900]]

 [  1,  10, 100]       [[1, 2, 3],       [[  1,  20, 300],
        OR         *    [4, 5, 6],   =    [  4,  50, 600],
[[  1,  10, 100]]       [7, 8, 9]]        [  7,  80, 900]]

They behave as if each row was evaluated separately, and singular dimensions are repeated where necessary. It helps to think about them as row-wise, instead of element-wise. This is particularly important in the second example, where the whole 1d-array is multiplied with every row of the 2d-array.

Note that this is not equivalent to multiplying every element as in [a[n]*b[n] for n in range(len(a))]. I guess that's why this is called broadcasting, and not element-wise.

[1] "matrix" here refers to a 2-d numpy.array. There is also a numpy.matrix, where multiplication is matrix multiplication, but this is not what I'm talking about.

Python Numeric Performance

Mon, 28 Sep 2015 00:00:00 +0200

Recently, I was working on a dynamic programming algorithm that involves a lot of number crunching in nested loops. The algorithm looks like this:

def y_change_probability_python(oct_per_sec):
    """ ... """
    b = 1.781
    mu = -0.301
    return 1/(2*b)*math.exp(-abs(oct_per_sec-mu)/b)

def y_idx_range_python(y_idx, max_y_factor, height):
    """ ... """
    y = (y_idx/height)*(max_y-min_y)+min_y
    y_lo = max(y/max_y_factor, min_y)
    y_hi = min(y*max_y_factor, max_y)
    y_lo_idx = int((y_lo-min_y)/(max_y-min_y)*height)
    y_hi_idx = int((y_hi-min_y)/(max_y-min_y)*height)
    return y_lo_idx, y_hi_idx

def find_tracks_python(cost_matrix, delta_x):
    """ ... """
    tracks = np.zeros(correlogram.shape, dtype=np.int64)
    cum_cost = np.zeros(correlogram.shape)

    max_y_factor = (2**5)**(delta_x)
    height = correlogram.shape[1]

    probabilities = np.empty((height, height))
    for y_idx in range(height):
        y = (y_idx/height)*(max_y-min_y)+min_y
        for y_pre_idx in range(*y_idx_range_numba(y_idx, max_y_factor, height)):
            y_pre = (y_pre_idx/height)*(max_y-min_y)+min_y
            doubles_per_x = math.log2((y/y_pre)**(1/delta_x))
            probabilities[y_idx, y_pre_idx] = y_change_probability_numba(doubles_per_x)

    for x_idx, cost_column in enumerate(cost_matrix):
        if x_idx == 0:
            cum_cost[x_idx] = cost_column
            continue
        for y_idx, cost in enumerate(cost_column):
            for y_pre_idx in range(*y_idx_range_numba(y_idx, max_y_factor, height)):
                weighted_cum_cost = cum_cost[x_idx-1, y_pre_idx] + cost*probabilities[y_idx, y_pre_idx]
                if weighted_cum_cost > cum_cost[x_idx, y_idx]:
                    cum_cost[x_idx, y_idx] = weighted_cum_cost
                    tracks[x_idx, y_idx] = y_pre_idx
            cum_cost[x_idx, y_idx] = cum_cost[x_idx-1, tracks[x_idx, y_idx]] + cost

    return tracks, cum_cost

I'm not going into the details of what this algorithm does, but note that it iterates over every column and row of the matrix cost_matrix, and then iterates over another range previous_y_range for each of the values in cost_matrix. On the way, it does a lot of basic arithmetic and some algebra.

The problem is, this is very slow. For a $90 \times 200$ cost_matrix, this takes about 260 ms. Lots of loops? Lots of simple mathematics? Slow? That sounds like a perfect match for Numpy!

If you can express your code in terms of linear algebra, Numpy will execute them in highly-optimized C code. The problem is, translating loops into linear algebra is not always easy. In this case, it took some effort:

def y_change_probability_numpy(doubles_per_x):
    """ ... """
    b = 1.781
    mu = -0.301
    return 1/(2*b)*np.exp(-np.abs(doubles_per_x-mu)/b)

def find_frequency_tracks_numpy(cost_matrix, delta_x):
    """ ... """
    tracks = np.zeros(cost_matrix.shape, dtype=np.int)
    cum_cost = np.zeros(cost_matrix.shape)

    max_y_factor = (2**5)**(delta_t) # allow at most 5 octaves per second (3 sigma)
    height = cost_matrix.shape[1]

    # pre-allocate probabilities matrix as minus infinity. This matrix
    # will be sparsely filled with positive probability values, and
    # empty values will have minus infinite probability.
    probabilities = -np.ones((height, height))*np.inf
    for y_idx in range(probabilities.shape[0]):
        y = (y_idx/height)*(max_y-min_y)+min_y
        y_pre_idx = np.arange(int((max(y/max_y_factor, min_y)-min_y)/(max_y-min_y)*height),
                              int((min(y*max_y_factor, max_y)-min_y)/(max_y-min_y)*height))
        y_pre = (y_pre_idx/height)*(max_y-min_y)+min_y
        doubles_per_x = np.log2((y/y_pre)**(1/delta_x))
        probabilities[y_idx, y_pre_idx] = y_change_probability(doubles_per_x)

    cum_cost[0] = cost_matrix[0]
    for x_idx in range(1, len(cost_matrix)):
        cost_column = cost_matrix[x_idx:x_idx+1] # extract cost_column as 2d-vector!
        weighted_cum_cost = cum_cost[x_idx-1] + cost_column.T*probabilities
        tracks[x_idx] = np.argmax(weighted_cum_cost, axis=1)
        cum_cost[x_idx] = cum_cost[x_idx-1, tracks[x_idx]] + cost_column

    return tracks, cum_corrs

This code does not look much like the original, but calculates exactly the same thing. This takes about 15 ms for a $90 \times 200$ cost_matrix, which is about 17 times faster than the original code! Yay Numpy! And furthermore, this code is arguably more readable than the original, since it is written at a higher level of abstraction.

Another avenue for performance optimization is Numba. Numba applies dark and powerful magic to compile humble Python functions into blazingly fast machine code. It is proper magic, if you ask me. Simply add an innocuous little decorator to your functions, and let Numba do it's thing. If all goes well, your code will work just as before, except with unheard-of performance:

@jit(numba.float64(numba.float64), nopython=True)
def y_change_probability_numba(doubles_per_x):
    ...

@jit((numba.int64, numba.float64, numba.int64), nopython=True)
def y_idx_range_numba(y_idx, max_y_factor, height):
    ...

@jit((numba.float64[:,:], numba.float64), nopython=True)
def find_tracks_numba(cost_matrix, delta_t):
    ...

However, Numba is no silver bullet, and does not support all of Python yet. In the present case, it is missing support for enumerate for Numpy matrices. Thus, I had to rewrite the first two loops like this:

# python version
for x_idx, cost_column in enumerate(cost_matrix):
    ...

# numba version
for x_idx in range(len(cost_matrix)):
    cost_column = cost_matrix[x_idx]
    ...

Another area that proved problematic is N-D slice writing. Instead of using expressions like m1[x,y:y+3] = m2, you have to write for idx in range(3): m1[x,y+idx] = m2[idx]. Not a difficult transformation, but it basically forced me to unroll all the nice vectorized code of the Numpy version back to their original pure-Python form. That said, Numba is getting better and better, and many constructs that used to be uncompilable (yield) are not a problem any more.

Anyway, with that done, the above code went down from 260 ms to 2.2 ms. This is a 120-fold increase in performance, and still seven times faster than Numpy, with minimal code changes. This is proper magic!

So why wouldn't you just always use Numba? After all, when it comes down to raw performance, Numba is the clear winner. The big difference between performance optimization using Numpy and Numba is that properly vectorizing your code for Numpy often reveals simplifications and abstractions that make it easier to reason about your code. Numpy forces you to think in terms of vectors, matrices, and linear algebra, and this often makes your code more beautiful. Numba on the other hand often requires you to make your code less beautiful to conform to it's subset of compilable Python.

Matlab and Audio Files

Wed, 22 Apr 2015 00:00:00 +0200

So I wanted to work with audio files in Matlab. In the past, Matlab could only do this with auread and wavread, which can read *.au and *.wav files. With 2012b, Matlab introduced audioread, which claims to support *.wav, *.ogg, *.flac, *.au, *.mp3, and *.mp4, and simultaneously deprecated auread and wavread.

Of these file formats, only *.au is capable of storing more than 4 Gb of audio data. But the documentation is actually wrong: audioread can actually read more data formats than documented: it reads *.w64, *.rf64, and *.caf no problem. And these can store more than 4 Gb as well.

It's just that, while audioread supports all of these nice file formats, audiowrite is more limited, and only supports *.wav, *.ogg, *.flac, and *.mp4. And it does not support any undocumented formats, either. So it seems that there is no way of writing files larger than 4 Gb. But for the time being, auwrite is still available, even though deprecated. I tried it, though, and it didn't finish writing 4.8 Gb in half an hour.

In other words, Matlab is incapable of writing audio files larger than 4 Gb. It just can't do it.

Unicode and Matlab on the command line

Wed, 15 Apr 2015 00:00:00 +0200

As per the latest Stackoverflow Developer Survey, Matlab is one of the most dreaded tools out there. I run into Matlab-related trouble daily. In all honesty, I have never seen a programming language as user-hostile and as badly designed as this.

So here is today's problem: When run from the command line, Matlab does not render unicode characters (on OSX).

I say "(on OSX)", because on Windows, it does not print a damn thing. Nope, no disp output for Windows users.

More analysis: It's not that Matlab does not render unicode characters at all when run from the command line. Instead, it renders them as 0x1a aka SUB aka substitute character. In other words, it tries to render unicode as ASCII (which doesn't work), and then replaces all non-ASCII characters with SUB. This is actually reasonable if Matlab were running on a machine that can't handle unicode. This is not a correct assessment of post-90s Macs, though.

To see why Matlab would do such a dastardly deed, you can use feature('locale') to get information about the encoding Matlab uses. On Windows and OS X, this defaults to either ISO-8859-1 (when your locale is pure de_DE or en_US) or US-ASCII, if it is something impure. In my case, German dates but English text. Because US-ASCII is obviously the most all-encompassing choice for such mixed-languages environments.

But luckily, there is help. Matlab has a widely documented (not) and easily discoverable (not) configuration option to change this: To change Matlab's encoding settings, edit %MATLABROOT%/bin/lcdata.xml, and look for the entry for your locale. For me, this is one of

 ...
 ...

In order to make Matlab's encoding default to UTF-8, change the entry for your locale to

 ...
 ...

With that, Matlab will print UTF-8 to the terminal.

You still can't type unicode characters to the command prompt, of course. But who would want that anyway, I dare ask. Of course, what with Matlab being basically free, and frequently updated, we can forgive such foibles easily…

Decisions in Pillars of Eternity

Sun, 12 Apr 2015 00:00:00 +0200

Early on in Pillars of Eternity, you are tasked to kill one of two characters: Either you kill King Rethoric, who executed many innocent people, or you kill Rolsc, the leader of the rebellion. This is an interesting moral choice, but it is also profoundly sad that the game presents killing either of them as the only resolution to this conflict.

It got me thinking. In the real world, I would not consider killing to be an option, ever. But in the game, you have to play by the game's rules. And furthermore, you can't just walk away and have the two characters duke it out amongst themselves: The whole game is built around the player, and without the player's interaction, the game world doesn't evolve.

Thus, this conflict won't resolve itself, and the player is forced to kill. I wish there were a diplomatic option, or a way of fixing the underlying problem so the two characters are not at odds any more. I fear how these choices in video games might influence our perceptions of everyday choices, and crave for video games that offer interesting choices that do not revolve around murder.

From what I hear, Planescape Torment did offer such choices. As did many of my favorite games of all times.

Julia First Impressions

Mon, 12 Jan 2015 00:00:00 +0100

Julia is a high-level dynamic programming language designed to address the requirements of high-performance numerical and scientific computing while also being effective for general purpose programming. —Wikipedia

In other words, it is supposed to be as fast as C, as practical as Python, and as scientific as Matlab. The next step in mainstream scientific computing.

However, Julia is still very young, and still evolving rapidly. But if any of the above is true, I am very interested!

So, over the last few days, I re-implemented an algorithm in Julia. The previous version of the algorithm was written in Python. The algorithm spends most of it's time in FFTs, thus I didn't expect big performance gains.

It is actually nice to have a language that is built for scientific computation. Coming from Python, it is refreshing to have array literals, ranges and mathematics available without importing anything. Much like Python, Julia also has a proper module system, comprehensions, and more than one function per file.

On the other hand, there are a few questionable design decisions as well. Julia uses 1-based, inclusive indexing (range[1:3] = [1 2 3] as opposed to Python's range[:3] = [0, 1, 2]). In practice, my experience is that I rarely need to add ±1 when indexing in Python, but I frequently need it in Julia/Matlab.

Also, Julia has no docstrings, which makes me sad. The whole documentation story is sad, really: Documentation is often incomplete, or missing altogether. Unit testing is not widespread at all, and still crude. 0.4 will apparently add docstrings using macros. This is ugly and doesn't work for one's own code, but it's certainly a step in the right direction. Those error messages would need some work, too.

All of that is hopefully just a symptom of Julia being young, and will improve over time. Similarly, signal processing functions are missing entirely, and I had to re-implement some. The community is growing rapidly, though, and a lot of missing functionality can be installed through the package manager already.

When it comes to writing code, there is a lot to like about Julia. Julia's type system does impose a bit of overhead, but it also grants immediate benefits: Many of Python's runtime errors happen at evaluation time and functions can easily document their arguments' types. Also, thinking strictly about types actually improved performance by a good 20% in my algorithm.

And finally, Performance is surprisingly good! Although my algorithm spends almost all of its time doing FFTs, Julia performed about twice as fast as Python. This is pretty much exactly twice as much as I had expected! It also exposed more of the features of the underlying libraries, which could be used for another speed-up of some 10%.

At the end of the day, Julia clearly isn't mature yet, but very promising. Documentation and libraries will no doubt grow, and performance is already excellent. I'll definitely keep an eye on it, and will experiment further when the opportunity presents itself.

The 31. Chaos Communication Congress

Thu, 01 Jan 2015 00:00:00 +0100

Every year, between Christmas and New Year's Eve, a very special convention is held: The Chaos Computer Club hosts the Chaos Communication Congress. Among my peers, this is just "The Congress", the real highlight at the end of our year.

Don't be fooled by the name though, The Congress is not just some conference, with talks and meetings and business cards. The Congress is where every internet citizen, technologist, hacker, DIYer, net-politician, programmer, computer scientist, or geek convenes to exchange ideas. It is a place of levity, amazement and enlightening.

The Congress lasts four days, and every day is packed full of exciting talks. My favorites:

Ich sehe, also bin ich … Du (German) As it turns out, you don't need special hardware to defeat defeat biometric scanners or passwords. A digital camera and some ingenuity is often enough.
From Computation to Consciousness (English) This really resonated with me: What is consciousness? (Why) are animals or computers conscious or not? What does it all mean? Really, this is a computationally motivated introduction to constructivism. Highly recommended.
Traue keinem Scan, den du nicht selbst gefälscht hast (German) The story of an entertaining person discovering that a scanned PDF does not necessarily contain the same text that was scanned if your scanner was built by Xerox.

All of these talks and many more are also available online, and I highly recommend you watch some of them. But The Congress is not just talks; this is where all the German hackspaces and user groups meet and present their projects. Walking through the halls, there are laser shows, quadrocopters, 3D printers, stitching machines, automatic cocktail mixers, mechanical and human musicians, and so much more.

At its core though, this is a place where a bunch of like-minded people meet and take over a convention center. The Congress happens in the CCH, the Hamburg Convention Center. Over the course of the four days of The Congress, this building transforms from a sterile business location to a colorful geek-party. Within a day, all the lights in the building are somehow turned colorful, a pneumatic tube network sends messages across the building, there are lounges and coffee rooms, and strange and beautiful sculptures of light and electronics show up all over the place. Just walking across this cornucopia of light and sound is a treat for the senses.

And this is just the natural thing that happens if you let a bunch of creative people do their thing. It is really a crowd like no other: Nowhere else have I seen such an open-minded and approachable crowd. There are no unpleasant drunk people, no fights, no harassment. All over the place, people talk to strangers, adults and children build things together, and men and women interact as equals.

The Congress is really unique. I can't recommend it enough. If you can free up those four days between Christmas and New Year's Eve at all, go there. You won't regret it!

Books of 2014

Tue, 30 Dec 2014 00:00:00 +0100

Ancillary Justice / Ancillary Sword, by Ann Leckie

I have been reading a lot of science fiction in the last few years. These books are part one and two of the best space opera I have ever read. The story is written from the perspective of an AI, sometimes inhabiting a lot of bodies, sometimes only one. Interestingly, this makes for a very introspective viewpoint, where politics and actions are expressed as consequences of nuanced human behavior and astute observation, as opposed to arbitrary human decisions. This perspective is extremely compelling, and makes for an extremely human story in a world that is a clever amalgamation of ancient Roman society and post-modern egalitarianism. There is no book I look forward to more fiercely than the conclusion of this series.

Scott Pilgrim, by Bryan Lee O'Malley

Scott Pilgrim is a twenty-something in the nineties. Scott Pilgrim doesn't have a job, and likes to play video games. Scott Pilgrim is fighting epic Anime battles against his girlfriends' exes. Scott Pilgrim is everyone, living in the real world. Scott Pilgrim is awesome, especially the new color-versions. The last part of the series will be released next year, and I really can't wait! (There's also a movie that is supposedly very good, but I won't watch it until I read all the comic books).

Doomsday Book, by Connie Willis

This is about time travel from near-future Britain to the fifteenth century. But then, an epidemic breaks loose: Britain is in quarantine and civil order starts breaking. At the same time in the fifteenth century, the plague hits. Disease, and the people trying to deal with it, is a human tragedy beyond belief. Connie Willis manages to write about these things with empathy, and astounding humane love and humor. This story moved me more deeply than I care to acknowledge.

A Digital Thank You

Sat, 20 Dec 2014 00:00:00 +0100

This is the time of the year when we reflect on our lives, and be thankful. We write Christmas cards to people we like, and celebrate with our loved ones. At my job, I am sitting in front of a screen all day, and I interacted not only with people, but also their software. So this is a column where I want to thank people I don't know for their delightful software:

Org Mode

Thank you, Carsten Dominik, Bastien Guerry, and everyone else, for this amazing piece of software! I have used Org mode for my research journal, this blog, writing my thesis, general note-taking, and all-around planning tool. Thank you for making my life so much easier to manage!

Matthias Geier

Thank you, Matthias, for your many contributions to PySoundFile and PySoundCard, and our many enlightening discussions. You have brought these two projects far further than I would have ever gone, and I learned a lot in the process! Also, a quick thank you to Github, which made our collaboration effortless and enjoyable.

fish

Thank you, Ridiculous Fish, for bringing sane scripting, glorious VGA color, and general awesomeness to the command line! Finally, a shell that does not drown you in messy configuration, crazy syntax, and archaic conventions. Finally, a shell with beautiful documentation, and sane defaults! Thanks for all the fish!

BitTorrent Sync

Dropbox is awesome, no doubt, but it still feels awkward to upload all my documents to a faceless corporation. Synchronizing data between computers is still hard–or rather, used to be hard. Because this year I discovered btsync. I now have a real off-site backup and file synchronization system fully under my own control, and it took all of ten minutes to set up. Thank you!

X-Plane

This year, I got back into an old hobby of mine: Flight Simulation. In particular, this year I re-discovered X-Plane, and the marvelous and free sceneries by SimHeaven and Andras Fabian. Sadly, there doesn't seem to be any pilot school around where I can complete my real-world pilot's license, so flight simulation will have to do. But with this simulator, I am enjoying flight simulation more than ever!

Choosing a German Email Provider

Tue, 09 Dec 2014 00:00:00 +0100

Why not just use Gmail¹? Because if you're not paying for the service, you are the service. Google gives you Gmail for free, and in return they analyze and market your data. That's a fair deal, as far as I am concerned. But not a deal I want to make.

For one thing, I just don't like the idea of someone else reading through my email. Furthermore, I'd like my emails to be hosted in the same country I live in², so that all the laws that protect the privacy of my letters are just as valid for my emails. So a German email provider it is.

First, I tried mail.de. They show ads, even if you pay them. Also, their web interface is surprisingly ugly. No thank you.

Then, I found posteo.de, and they seemed to be doing everything right! Hosted in Germany, a buck a month, with a focus on security and privacy, and seemingly developed by friendly people. They even try to be environmentally friendly. In summary: perfect!

After half a year of using Posteo, though, I have found a few niggles. The web interface can't search mailboxes with a lot of emails–I have an archive directory with 14k emails and the search wouldn't work. Their support says that's because there are too many messages in that directory. Also, they have had a bit too much down time for my taste: I experienced three outages of about two hours each in the last half year. Not a deal breaker, but it doesn't exactly instill confidence in their infrastructure.

Enter mailbox.org. Also German, also a buck a month, also friendly and safe and eco-conscious. But with a much nicer web interface, support for custom domains, and a working search box. They even support ActiveSync, which you'll like if you use Outlook or Windows Phone. In other words, Posteo done right. I'm sold.

Footnotes:

Or outlook.com, which is the same deal from Microsoft. Use this if you don't like Google's "priority" inbox and wonky IMAP support.

For the record, Microsoft has made a point of having servers in each user's country and abiding by that country's law.

Org Mode Selective Section Numbering

Wed, 03 Dec 2014 00:00:00 +0100

This is the third revision of a post about selective headline numbering in Org mode. On its own, Org mode can either number all headlines, or none. For scientific writing, this is a non-starter. In a scientific paper, the abstract should not be numbered, the main body should be numbered, and appendices should not be numbered.

In LaTeX, this is easy to do: \section{} creates a numbered headline, while \section*{} creates an unnumbered section. Org mode does not have any facility to control this on a per-headline basis, but it can be taught:

(defun headline-numbering-filter (data backend info)
  "No numbering in headlines that have a property :numbers: no"
  (let* ((beg (next-property-change 0 data))
         (headline (if beg (get-text-property beg :parent data))))
    (if (and (eq backend 'latex)
         (string= (org-element-property :NUMBERS headline) "no"))
        (replace-regexp-in-string
         "\\(part\\|chapter\\|\\(?:sub\\)*section\\|\\(?:sub\\)?paragraph\\)"
         "\\1*" data nil nil 1)
      data)))

(setq org-export-filter-headline-functions '(headline-numbering-filter))

This creates a filter (an Org mode convention similar to a hook), which appends the asterisk to LaTeX headlines if the headline has a property :NUMBERS: no. If all you do is export to LaTeX, this works well.

If you need to export to HTML as well, things get more complicated. Since HTML does not have native numbering support, Org is forced to manually create section numbers. But times have changed, and with CSS3, HTML now indeed does support native numbering!

Here is some CSS that uses CSS3 counters to number all headlines and hide Org's numbers:

/* hide Org-mode's section numbers */
span.section-number-2 { display: none; }
span.section-number-3 { display: none; }
span.section-number-4 { display: none; }
span.section-number-5 { display: none; }
span.section-number-6 { display: none; }

/* define counters for the different headline levels */
h1 { counter-reset: section; }
h2 { counter-reset: subsection; }
h3 { counter-reset: subsubsection; }
h4 { counter-reset: paragraph; }
h5 { counter-reset: subparagraph; }

/* prepend section numbers before headlines */
h2::before {
    content: counter(section) " ";
    counter-increment: section;
}
h3::before {
    content: counter(section) "." counter(subsection) " ";
    counter-increment: subsection;
}
h4::before {
    content: counter(section) "." counter(subsection) "." counter(subsubsection) " ";
    counter-increment: subsubsection;
}
h5::before {
    content: counter(section) "." counter(subsection) "." counter(subsubsection) "." counter(paragraph) " ";
    counter-increment: paragraph;
}
h6::before {
    content: counter(section) "." counter(subsection) "." counter(subsubsection) "." counter(paragraph) "." counter(subparagraph) " ";
    counter-increment: subparagraph;
}

/* suppress numbering for headlines with class="nonumber" */
.nonumber::before { content: none; }

With this in place, we can extend the previous filter to work for HTML as well as LaTeX:

(defun headline-numbering-filter (data backend info)
  "No numbering in headlines that have a property :numbers: no"
  (let* ((beg (next-property-change 0 data))
         (headline (if beg (get-text-property beg :parent data))))
    (if (string= (org-element-property :NUMBERS headline) "no")
        (cond ((eq backend 'latex)
               (replace-regexp-in-string
                "\\(part\\|chapter\\|\\(?:sub\\)*section\\|\\(?:sub\\)?paragraph\\)"
                "\\1*" data nil nil 1))
              ((eq backend 'html)
               (replace-regexp-in-string
                "\\(]*>\\)"
                "\\1 class=\"nonumber\"\\2" data nil nil)))
      data)))

(setq org-export-filter-headline-functions '(headline-numbering-filter))

Previously, I implemented this in Org mode only (no CSS). While that worked as well, it required the modification of some fairly low-level Org functions. The CSS-based solution is much simpler, and should be much easier to maintain and adapt.

Writing a Thesis in Org Mode

Wed, 19 Nov 2014 00:00:00 +0100

Most of my peers write all their scientific documents in LaTeX. Being a true believer in the power of Emacs, I opted for writing my master's thesis in Org Mode instead. Here's my thoughts on this process and how it compares to the usual LaTeX work flow.

In my area of study, a thesis is a document of about 60 pages that contains numerous figures, math, citations, and the occasional table or source code snippet. Figures are usually graphs that are generated in some programming environment and creating those graphis is a substantial part of writing the thesis.

Org mode was a huge help in this regard, since it combines the document text and the executable pieces of code. Instead of having a bunch of scripts that generate graphs, and a bunch of LaTeX files that include those graphs, I had one big Org file that included both the thesis text and the graphing code.

As for the thesis text, I used Org's export functionality to convert the Org source to LaTeX, and compiled a PDF from there. This really works very well: It is very nice to use Org headlines instead of \section{...}, and clickable Org links instead of \ref{...}. While this is nice, it is just a change of syntax. I still had to enter the very same things and saving a few characters is not particularly impressive. For example, figures still require a caption, an ID, and a size:

#+CAPTION: Modulation tracks of a clarinet recording with and without white noise. The modulation tracks are not normalized.
#+ATTR_LATEX: :width 6in :height 2.5in :float multicolumn
#+NAME: fig:summary_tracks
[[file:images/summary_tracks.pdf]]

In LaTeX, this would be

\begin{figure*}
\centering
\includegraphics[width=6in,height=2.5in]{images/summary_tracks.pdf}
\caption{\label{fig:summary_tracks}Modulation tracks of a clarinet recording with and without white noise. The modulation tracks are not normalized.}
\end{figure*}

As you can see, there really is not that much of a difference between these two, and you might even consider the LaTeX example more readable. In some other areas, Org mode is simply lacking features: Org does not have any syntax for page formatting, and thus can't create a perfectly formatted title page. Similarly, it can't do un-numbered sections, and it can't do numbered equations. For all of those, I had to fall back to writing LaTeX. This is not a big deal, but it breaks the abstraction.

A bigger problem is that Org documents include all the chapters in one big file. While Org can deal with large files no problem, it means that LaTeX compiles take a while. In LaTeX, I would have split my document into a number of smaller files that could be separately compiled in order to keep compilation time down. This is confounded by Org's default behavior of deleting intermediate LaTeX files, which forces a full triple-recompile on each export. At the end of my thesis, a full export took about 15 seconds. Not a deal-breaker, but annoying.

The one thing where Org really shines, though, is the inclusion of code fragments: Most of my figures were created in Python, and Org mode allowed me to include that Python code right in my document. Hit C-c C-c on any code fragment, and Org ran that code and created a new image file that is automatically included as a figure. This was really tremendously useful!

At the end of the day, I am not sure whether Org mode is the right tool for writing a thesis. It worked fine, but there were a lot of edge cases and workarounds, which made the whole process a bit uncomfortable. The only really strong argument in favor of Org is the way it can include both code and prose in the same document. But maybe a similar thing could be implemented with LaTeX and some literate programming tool.

Org Mode Citation Links

Tue, 23 Sep 2014 00:00:00 +0200

I am writing my master's thesis in Org Mode, and export to LaTeX for publishing. For the most part, this works incredibly well. Using Org Mode instead of plain LaTeX means no more fiddly \backslash{curly brace} all over the place. No more scattering code fragments and markup across hundreds of files. And on top of that, deep integration with my research notes and task tracking system.

But not everything is perfect. For one thing, citations do not work well. Sure, you can always write \cite{cohen93}, but then you are writing LaTeX again. Also, all the other references and footnotes are clickable, highlighted Org Mode links, but \cite{cohen93} is just inline LaTeX.

But luckily, this is Emacs, and Emacs is programmable. And better yet, Org Mode has just the tool for the job:

(org-add-link-type "cite"
     (defun follow-cite (name)
       "Open bibliography and jump to appropriate entry.
        The document must contain \bibliography{filename} somewhere
        for this to work"
       (find-file-other-window
        (save-excursion
          (beginning-of-buffer)
          (save-match-data
            (re-search-forward "\\\\bibliography{\\([^}]+\\)}")
            (concat (match-string 1) ".bib"))))
       (beginning-of-buffer)
       (search-forward name))
     (defun export-cite (path desc format)
       "Export [[cite:cohen93]] as \cite{cohen93} in LaTeX."
       (if (eq format 'latex)
           (if (or (not desc) (equal 0 (search "cite:" desc)))
               (format "\\cite{%s}" path)
             (format "\\cite[%s]{%s}" desc path)))))

This registers a new link type in Org Mode: [[cite:cohen93]], which will jump to the appropriate bibliography entry when clicked, and get exported as \cite{cohen93} in LaTeX. Awesome!

Transplant

Tue, 19 Aug 2014 00:00:00 +0200

In academia, a lot of programming is done in Matlab. Many very interesting algorithms are only available in Matlab. Personally, I prefer to use tools that are more widely applicable, and less proprietary, than Matlab. My weapon of choice at the moment is Python.

But I still need to use Matlab code. There are a few ways of interacting with Matlab out there already. Most of them focus on being able to eval strings in Matlab. Boring. The most interesting one is mlab, a full-fledget bridge between Python and Matlab! Had I found this earlier, I would probably not have written my own.

But write my own I did: Transplant. Transplant is a very simple bridge for calling Matlab functions from Python. Here is how you start Matlab from Python:

import transplant
matlab = transplant.Matlab()

This matlab object starts a Matlab interpreter in the background and connects to it. You can call Matlab functions on it!

matlab.eye(3)
>>> array([[ 1.,  0.,  0.],
           [ 0.,  1.,  0.],
           [ 0.,  0.,  1.]])

As you can see, Matlab matrices are converted to Numpy matrices. In contrast to most other Python/Matlab bridges, matrix types are preserved¹:

matlab.randi(255, 1, 4, 'uint8')
>>> array([[246,    2, 198, 209]], dtype=uint8)

All matrix data is actually transferred in binary, so both Matlab and Python work on bit-identical data. This is very important if you are working with precise data! Most other bridges do some amount of type conversion at this point.

This alone accounts for a large percentage of Matlab code out there. But not every Matlab function can be called this easily from Python: Matlab functions behave differently depending the number of output arguments! To emulate this in Python, every function has a keyword argument nargout ². For example, the Matlab function max by default returns both the maximum value and the index of that value. If given nargout=1 it will only return the maximum value:

data = matlab.randn(1, 4)
matlab.max(data)
>>> [1.5326, 3] # Matlab: x, n = max(...)
matlab.max(data, nargout=1)
>>> 1.5326      # Matlab: x = max(...)

If no nargout is given, functions behave according to nargout(@function). If even that fails, they return the content of ans after their execution.

Calling Matlab functions is the most important feature of Transplant. But there is a more:

You can save/retrieve variables in the global workspace:

  matlab.value = 5 # Matlab: value = 5
  x = matlab.value # Matlab: x = value

You van eval some code:

  matlab.eval('class(value)')
  >>> ans =
  >>>
  >>> double
  >>>

The help text for functions is automatically assigned as docstring. In IPython, this means that matlab.magic? displays the same thing help magic would display in Matlab.

Under the hood, Transplant is using a very simple messaging protocol based on 0MQ, JSON, and some base64-encoded binary data. Sadly, Matlab can deal with none of these technologies by itself. Transplant therefore contains a full-featured JSON parser/serializer and base64 encoder/decoder in pure Matlab. It also contains a minimal mex-file for interfacing with 0MQ.

There are a few JSON parsers available for Matlab, but virtually all of them try parse JSON arrays as matrices. This means that these parsers have no way of differentiating between a list of vectors and a matrix (want to call a function with three vectors or a matrix? No can do). Transplant's JSON parser parses JSON arrays as cell arrays and JSON objects as structs. While somewhat less convenient in general, this is a much better fit for transferring data structures between programming languages.

Similarly, there are a few base64 encoders available. Most of them actually use Matlab's built-in Java interface to encode/decode base64 strings. I tried this, but it has two downsides: Firstly, it is pretty slow for short strings since the data has to be copied over to the Java side and then back. Secondly, it is limited by the Java heap space. I was not able to reliably encode/decode more than about 64 Mb using this³. My base64 encoder/decoder is written in pure Matlab, and works for arbitrarily large data.

All of this has been about Matlab, but my actual goal is bigger: I want transplant to become a library for interacting between more than just Python and Matlab. In particular, Julia and PyPy would be very interesting targets. Also, it would be useful to reverse roles and call Python from Matlab as well! But that will be in the future.

For now, head over to Github.com/bastibe/transplant and have fun! Also, if you find any bugs or have any suggestions, please open an issue on Github!

Footnotes:

Except for integer complex numbers, since those are not supported by Numpy.

Like the Matlab function nargout

At 192 Mb of Java heap space. And even those 64 Mb were pretty unreliable if I didn't call java.lang.Runtime.getRuntime.gc all the time.

Working and Learning

Sun, 20 Jul 2014 00:00:00 +0200

At the university, I have a big advantage: I can program. So many of my fellow students are programming as their main means of doing science, yet clearly never learned how to program efficiently. It is saddening to see them "fight Matlab" for days, for things that would take a programmer hours.

So how did I get to this point? After all, I went to the same university and studied the same topics they did. My first introduction to programming was our Introduction to Programming in the first semester. We learned how to write simple text-based programs in C.

My own blogging engine

At the time, my fellow students and I wanted to organize our lecture notes, copied exams, and assignments on our own website. Not knowing any better, I picked up PHP and set out to write a little website for this. It turned into a little CMS, all hand-written in PHP, HTML and (almost no) CSS.

This happened about four weeks into the introductory programming course, so I only knew a few bits of C and didn't appreciate the differences between programming languages yet. Many bad things have been said about PHP, but it allowed me to hack together a blog, file browser, gallery, and calendar with knowing little more about programming than branches and loops.

It scares me to look at the ease with which I picked up PHP at the time. With more experience, I seem to become more reluctant to try out new things. This might be a very bad thing.

C programming at the university

In my third semester, a professor offered me a job as an undergraduate research assistant. As my first assignment, he wanted me to program a MIDI interface for Matlab. The idea was to use the Matlab-C interface Mex to connect portmidi to Matlab. At this point, I had had two programming courses (C and C++/Matlab), and had read The Pragmatic Programmer.

I remember the professor telling me to stop obsessing about that piece of code. He said "You are an engineer, 95% is good enough for engineers". Yet, reading through this code now, it is some fine C code. Everything is well-commented, the implementation is clearly split into one Matlab-related part and one portmidi-related part, and there even is an (informal) test suite! To my mind, those extra 5% make an incredible difference! It is astonishing how much my early career has been influenced by The Pragmatic Programmer.

Another project concerned extending a C program that simulated small-room acoustics. It took me eight months to admit defeat. Every week, I would spend ten hours staring at my screen, trying to understand that program. Every week, I would fail in frustration. After eight months, I told the professor that I couldn't do it. A few months later, I told the professor that I would have another go at the project. This time, I read the research paper associated with the program first. After that, the project was completed in one afternoon. Sometimes, just reading the code is not enough. (This is probably more true in academia than anywhere else).

Qt and Cocoa and OpenGL

I wrote my bachelor's thesis for a small company in southern Germany. At my university, the thesis was supposed to take half a year, and should be written at some company, so students would get some first-hand experience of the real world outside the university.

The basic algorithm for the thesis was working after a few weeks. Since my boss was more interested in a commercial result than in more research, he proposed that I write a desktop application for it. This is how I was introduced to Qt. Qt is an incredibly complex framework. Luckily for me, it is also an incredibly well-documented framework! As a newcomer to programming, API docs can be a very daunting thing, filled with jargon and implementation detail. This was the first time I learned something mostly from reading the API docs, and I am grateful that I happened to pick Qt for that.

After finishing my thesis, I worked remotely for the same company, writing another GUI application. This time, the program was to be written in Objective-C/Cocoa. In contrast to Qt, I needed the book to learn Cocoa. Working through the book was a very different experience than learning Qt from API docs. The book not only described the API, but also things like best practices and programming patterns. As a result, my final program was much easier to understand and extend than the Qt program I wrote earlier.

Cocoa and Qt show two very different styles of documentation. The Qt documentation is very complete, and very well-written. It is a rare feat for a framework this complex to be learnable from the documentation alone! Doing the same thing with the Cocoa documentation instead of the book would have been painful. The book really went much further than pure API documentation can reasonably go, and my experience was better for it! (By the way, I also learned and used OpenGL during that time. The less said about the OpenGL documentation, the better).

Why Software is paid for

In the meantime, the company had been bought by a foreign investor. While this meant that my program would never see actual users, it also meant that they could offer me a proper job. And like every good engineering student, I needed Matlab, Photoshop, and Microsoft Office. And like every cliché foreign investor, they replied with "This is too expensive, here is a link to the Pirate Bay".

This did not sit well with me. After a bit of soul-searching, and my most interesting and obviously soon-deleted Stack Overflow question, I realized that I could not pirate software any longer. My own livelihood depended on people paying for the code I wrote. There was no way I would use other people's code without paying for it.

And thus we used Inkscape instead of Illustrator, and Python instead of Matlab. Mere weeks later, I discovered that Inkscape produces (mostly) standard-compatible SVG files that could be opened with any regular text editor, and manipulated with any regular XML parser! It soon became apparent that the open nature of this file format enabled us to use Inkscape for so much more than mere vector graphics! While in retrospect, it was not such a bright idea to use a vector graphics program as a GUI layout editor, it really drove home the value of open file formats and reusability! A lot of the later work on the project would have been impossible had we used Illustrator and Matlab.

Automation

We were working with a British company on a new digital mixing console at the time. Our team was mostly responsible for the software side of the project, while the British company was mostly concerned with the hardware. One big issue was that in order to get a testable system going, one had to compile some software, run some converter scripts on some files, zip some other files, set up the prototype hardware correctly, then send all the files to the prototype in the right order. Forget one step, or take an outdated version of something, and the system would not work.

It was a disaster. We would lose days debugging nonexistent issues, only because we had forgotten to update such-and-such library, or renaming some debugging file. It would be easy to blame this on my colleagues. But the reality is, no-one had ever done a project this large before, and our tools were utterly incapable of build automation of this kind.

In the end, I wrote some crazy Rube Goldberg Machine that integrated GNU make with Visual Studio, and delegated all the packaging and converting to makefiles. It would even download a large set of Unix tools and a full installation of Ruby if need be. I can't say I'm proud of this wild contraption, but anything is better than wasting days debugging non-issues. To its credit, there have been zero issues with wrongly packaged files with this system in place. I can not tell you how much stress and conflict this simple act of automation relieved. Never have a human do a machine's work.

Lua and DSLs

When I started on the job, a colleague of mine handed me a copy of "Programming for Windows 95", and told me to read it since he had modeled the internal GUI library after it. This was 2010. I was very unhappy about this. In the following years, I would rework many a subsystem within this library. But the more I changed, the more I had to take responsibility for the library. Before long, I had taken official ownership of the library, and I had to answer to questions and feature requests.

This turned out to be both a blessing and a curse. On the one hand, it gave me a great deal of freedom and authority in my own little world. On the other hand, I didn't really care for responsibility for this much legacy code in an application domain I was not particularly interested in. Thus being motivated to change things had its upsides though, and I learned a lot when implementing a font rendering engine, a bitmap caching and memory allocation system, and various configuration mechanisms on an embedded platform.

But, at the end of the day, there is only so much you can do with a bad code base in a bad subset of C++ (largely due to compilers, not people). In another slow-going week when GUI work was not particularly important, I investigated implementing a scripting layer for our framework. We were not very optimistic about this, since the scripting engine had to run on a terribly slow embedded processor that was already running almost at capacity.

We chose the scripting language Lua for the job, since it was tiny, and easy to embed (in both meanings of the word). Lua turned out to be a stellar choice! As scripting systems often go, the Lua code took over most of the frontend work in the application. Before long, all the GUI layout was done in a Lua DSL instead of XML. Imagine creating 200 buttons in a two-line for loop instead of 200 lines of XML. Also, I consider the book Programming in Lua one of the pivotal books in my programming career!

All the GUI and hardware interaction was done in Lua. The mixing console had some 40000 parameters, and a terrifying number of hardware states. I daresay that it would have been all but impossible to implement the complex interplay between all of these states in a less dynamic environment than Lua. Years later, one of the later maintainers of the product told me how this system had saved his sanity many times. This was one of the proudest moments in my career!

I vividly remember the feeling of liberation when I transitioned from C++ to Lua. I don't think we would have managed to ship the mixing console in time without Lua. In fact, there was one feature from the old analogue mixing consoles that they never managed to implement in the newer digital consoles, because it was just too hard. With Lua, it was a giant headache, but it worked. Never underestimate the power of a different language when problems seem impossibly hard.

The role of boredom in my job

The Lua experiment started in a time when work was slow, and idle thoughts had the time to mature into ideas. The system automation was started in a similar time. I was lucky to have had a few of those weeks. Some of them amounted to cool projects in the company, others I spent on improving myself.

I always had a bit of a fetish for text editing. I just love the act of feeding thoughts to the computer through a keyboard. To me, it is a much more satisfying experience than using a pencil and a sheet of paper. At the university, I used Vim on Linux, then Textmate on OS X, then XCode. On the job, I was then forced to use Visual Studio, which still holds a special place in my heart, as one of the most miserable editing experience I ever had (though Lotus Notes and Microsoft Word only rank lower because I used them less).

It should come as no surprise then, that I was overjoyed when I discovered ViEmu. It really transformed my work at the time – what was previously a chore was now made enjoyable by the feeling of power conveyed through the Vim key bindings in Visual Studio! And this improved even further when I used another spare week to finally learn how to properly touch-type. These days, I am typing in Emacs, but enough has been written about that already.

I had one colleague who only used his two index fingers for typing. Seeing him type was maddening. But the worst thing was not his typing, but what he was not typing. Naturally, variable names were short, documentation was sparse, and code was optimized for brevity. He even resorted to some graphical code editing monstrosity, just to save himself some typing. I have written a Visual Studio tool that automatically filtered out some of the junk this tool produced, and wrote wrappers around his libraries to make them usable for other people. Seriously, don't be that guy. Typing is a core competency for any developer.

Open Source

Besides all of the GUI work I did for the company, I was actually hired for audio algorithm development. Since we didn't get a license for Matlab, I quickly grew to love Python instead. At the time, Python was right in the middle of the transition from Python 2 to Python 3, and one of the libraries I needed was Python 2 only. In another one of those fateful slow weeks, I set out to translate it to Python 3.

I didn't know much Python at the time, so the result was not exactly perfect. The maintainer of the library however was really nice about this, and helped me figuring out the problems with my code. This was the first time I ever talked to any programmer outside my company! And even better, this programmer seemed to be a professor at MIT, or something, and likely a lot more experienced and intelligent than I was! I was incredibly lucky that this first contact with the open source world was such a kind and positive one.

Not too long after that, I started writing my own open source libraries, and publishing them on the web. And before too long, people began using those projects! And then they started contributing to them as well! In a way, one of my main griefs with working for a company has always been that there are so few people with which you can talk about the things you do all day. And now, suddenly, random people from all over the world are showing interest and help for the things I do in my spare time! I really can't emphasize enough how much this involvement with the open source community and other people has improved my view of the world, and my understanding of the work I do!

My next adventure

This has been a summary of the things I did so far. It has been an incredible journey, and one that never stopped to surprise me. Now I am finishing my master's thesis, and getting ready for a doctorate after that. All the work I did and do is based on the incredible work of people before me. At least for the time being, I want this to be the goal of my further work: To advance the sum total knowledge of the world, if only by a tiny bit.

For this, my most important tool is still programming. Learning how to program is an immensely valuable skill, and doubly so if your job title is not "developer" or "programmer". Programming is not just a tool to talk to the computer and earn a living. We should not forget that programming is also a rich thinking tool for trying out new ideas, and sharing them with other people.

For the moment, I have no desire for being beholden to some company dictating my goals and hiding my achievements. Writing this up has proven to be a very liberating and insightful experience for myself, just the way my research journal is for my day-to-day work. Putting ideas and algorithms in writing is an incredibly useful tool for finding one's place in the world and contributing to its betterment!

Speeding Up Org Mode Publishing

Wed, 07 May 2014 00:00:00 +0200

I use org-mode to write my blog, and org-publish as my static site generator. While this system works great, I have found it to be really really slow. At this point, my blog has 39 posts, and org-publish will take upwards of a minute to re-generate all of them. To make matters worse, my workflow usually involves several re-generations per post. This gets old pretty quickly.

Since I am on a long train ride today, I decided to have a go at this problem. By the way, train rides and hacking on Emacs are a perfect match: Internet connectivity on trains is usually terrible, but Emacs is self-documenting, so internet access doesn't matter as much. It is sobering to work without an internet connection every once in a while, and Emacs is a perfect target for this kind of work.

One of the many things I learned on train rides is that Emacs in fact contains its own profiler! So, I ran (progn (profiler-start 'cpu) (org-publish "blog") (profiler-report)) to get a hierarchical list of where org-publish was spending its time. Turns out, most of its total run time was spent in functions relating to version control (starting with vc-).

Some package in my configuration set up vc-find-file-hook as part of find-file-hook. This means that every time org-publish opens a file, Emacs will look for the containing git repository and query its status. This takes forever! Worse yet, I don't even use vc-git at all. All my git interaction is done through magit.

But Emacs wouldn't be Emacs if this could not be fixed with a line or two of elisp. (remove-hook 'find-file-hooks 'vc-find-file-hook) will do the trick. This brought the runtime of org-publish down to 15 seconds. Yay for profiling and yay for Emacs!

Snow Crash

Sat, 03 May 2014 00:00:00 +0200

This is probably the one book that made me who I am. I bought this book in the English section of a Norwegian bookstore while on a bike trip there. I was just out of school and I had not decided yet what to do with my life. My English was just barely good enough to understand Snow Crash.

At its essence, Snow Crash introduced me to the idea of hacking. That is, hacking in the abstract sense of using knowledge about the inner workings of things to influence the world. At the time, I didn't know anything about software yet. Even though the main hero of the book is a master hacker and part of the book plays out in cyber space, it never really talks about the act of writing code. I would learn about that later.

Snow Crash was my red pill. Re-reading it today, after having been a professional software engineer for a few years, I can't overstate how influential this book has been in my life. It instilled in me the desire to understand the deep structures of things. And with that, the ability to influence and create the world around me. It set me on the path to becoming who I am today.

Snow Crash also served as my introduction to modern utopian cyberpunk. Even though I spent my youth playing video games and reading bad sci-fi, I still remember Snow Crash as my formal introduction to cyber space, virtual avatars, megacorporations, and light cycle races. And it told of the awesome power of reason.

If you haven't already, go read this book!

Sound in Python

Wed, 27 Nov 2013 00:00:00 +0100

Have you ever wanted to work with audio data in Python? I know I do. I want to record from the microphone, I want to play sounds. I want to read and write audio files. If you ever tried this in Python, you know it is kind of a pain.

It's not for a lack of libraries though. You can read sound files using wave, SciPy provides scipy.io.wavfile, and there is a SciKit called scikits.audiolab. And except for scikits.audiolab, these return the data as raw bytes. Like, they parse the WAVE header and that is great and all, but you still have to decode your audio data yourself.

The same thing goes for playing/recording audio: PyAudio provides nifty bindings to portaudio, but you still have to decode your raw bytes by hand.

But really, what I want is something different: When I record from the microphone, I want to get a NumPy array, not bytes. You know, something I can work with! And then I want to throw that array into a sound file, or play it on a different sound card, or do some calculations on it!

So one fateful day, I was sufficiently frustrated with the state of things that I set out to create just that. Really, I only wanted to play around with cffi, but that is beside the point.

So, lets read some audio data, shall we?

import soundfile
data = soundfile.read('sad_song.wav')

done. All the audio data is now available as a NumPy array in data. Just like that.

Awesome, isn't it?

OK, that was easy. So let's read only the first and last 100 frames!

import soundfile
first = soundfile.read('long_song.flac', stop=100)
last = soundfile.read(start=-100)

This really only read the first and last bit. Not everything in between!

Note that at no point I did explicitly open or close a file! This is Python! We can do that! When the SoundFile object is created, it opens the file. When it goes out of scope, it closes the file. It's as simple as that. Or just use SoundFile in a context manager. That works as well.

Oh, but I want to use the sound card as well! I want to record audio to a file!

from pysoundcard import Stream
from pysoundfile import SoundFile, ogg_file, write_mode
with Stream() as s: # opens your default audio device
    # This is supposed to be a new file, so specify it completely
    f = SoundFile('happy_song.ogg', sample_rate=s.sample_rate,
                  channels=s.channels, format=ogg_file,
                  mode=write_mode)
    f.write(s.read(s.sample_rate)) # one second

Read from the stream, write to a file. It works the other way round, too!

And that's really all there is to it. Working with audio data in Python is easy now!

Of course, there is much more you could do. You could create a callback function and be called every four[1] frames with new audio data to process. You could request your audio data as int16, because that would be totally awesome! You could use many different sound cards at the same time, and route stuff to and fro to your hearts desire! And you can run all this on Linux using ALSA or Jack, or on Windows using DirectSound or ASIO, or on Mac using CoreAudio[2]. And you already saw that you can read Wave files, OGG, FLAC or MAT-files[3].

You can download these libraries from PyPi, or use the binary Windows installers on Github. Or you can look at the source on Github (PySoundFile, PySoundCard), because Open Source is awesome like that! Also, you might find some bugs, because I haven't found them all yet. Then, I would like you to open an issue on Github. Or if have a great idea of how to improve things, please let me know as well.

UPDATE: It used to be that you could use indexing on SoundFile objects. For various political reasons, this is no longer the case. I updated the examples above accordingly.

[1] You can use any block size you want. Less than 4 frames per block can be really taxing for your CPU though, so be careful or you start dropping frames. [2] More precisely: Everything that portaudio supports. [3] More precisely: Everything that libsndfile supports.

Blogging with Emacs

Wed, 13 Nov 2013 00:00:00 +0100

When I first started blogging, it was on blogger.com (on the now-abandoned domain daskrachen.com). On blogger, writing new posts (back then) involved typing raw HTML into a web form. Not what I would call ideal. This improved somewhat when they introduced a fancy rich text editor that would automatically transform beautiful text into a horrible formatting mess.

Thus I switched. Getting my blog posts out of blogger was… Let's just say that I lost anything I didn't have a plain-text backup of. And I switched to Pelican, a static site generator written in Python. It worked beautifully, until I updated something, at which point it resorted to just throwing errors. Now I don't have anything in particular against Python stack traces, but these particular traces traced deep into stuff that was (then?) too complex for me to understand.

Thus I switched again. This time to C()λ∈slaw■, a static site generator written in Common Lisp. Mainly because I was interested in Common Lisp at the time. It worked really well. However, this was supposed to give me a chance to delve into Common Lisp, and I failed to understand C()λ∈slaw■'s code. Realistically though, this is probably not C()λ∈slaw■'s fault. My knowledge of Common Lisp is far from perfect.

Thus it was time to switch again. Having been enamored with Emacs for the last few years, it made sense to blog with Emacs as well. Besides, I am kind of fed up with the many conflicting flavors of Markdown out there and have switched my personal note-taking to Org mode long ago. So let's set up Emacs and Org as a blogging platform!

Before we start though, a short disclaimer: This will be a very bare bones blogging engine. It will consist of some articles, a front page, an archive page, and an RSS feed. And you will have to manage the front page and RSS feed semi-manually. No tags, no fancy history. Just what you see here.

On the plus side, this will be implemented entirely within Emacs and very simple to understand. Writing a new blog post will be as simple as writing an Org file and hitting a key combination! And you will get all of Org's fancy syntax highlighting and export magic for free!

Getting the pages to work is rather simple: You have to create a "publishing project" that specifies a base-directory where your Org files live and a publishing-directory, where the HTML files are going to be stored. Since this is Emacs, you could make your publishing directory any TRAMP path you like and insta-publish your workings!

(BTW, I am using Org 8.2.2 and I believe you need at least 8.0 for these examples to work)

(require 'ox-html)
(require 'ox-rss)
(require 'ox-publish)
(setq org-publish-project-alist
      '(("blog-content"
         :base-directory "~/Projects/blog/posts"
         :html-extension "html"
         :base-extension "org"
         :publishing-directory "~/Projects/blog/publish"
         :publishing-function (org-html-publish-to-html)
         :recursive t          ; descend into sub-folders?
         :section-numbers nil  ; don't create numbered sections
         :with-toc nil         ; don't create a table of contents
         :with-latex t         ; do use MathJax for awesome formulas!
         :html-head-extra ""   ; extra  entries go here
         :html-preamble ""     ; this stuff is put before your post
         :html-postamble ""    ; this stuff is put after your post
)))

Now hit M-x org-publish, type in blog-content, and you have a blog! Awesome! We are done here.

Well, how about an archive page that lists all your previous blog entries?

Emacs can auto-generate this for you. Simply add these lines to blog-content:

         :auto-sitemap t
         :sitemap-filename "archive.org"
         :sitemap-title "Archive"
         :sitemap-sort-files anti-chronologically
         :sitemap-style list
         :makeindex t

Also, you can put something like

Other posts

into your :html-postamble to make every page link to this. You can also add your Disqus snippet there to enable comments.

Adding a front page is simple, too. My front page is simply a normal page called index.org, which contains links and slugs for every article I want to have on the front page. For example:

 #+TITLE: RECENT POSTS

 * [[file:2013-05-30-speeding-up-matplotlib.org][Speeding up Matplotlib]]
 #+include: "~/Projects/blog/posts/2013-05-30-speeding-up-matplotlib.org" :lines "4-9"
 [[file:2013-05-30-speeding-up-matplotlib.org][read more...]]

But a blog is more than just text. There are images and CSS, too. I keep all that stuff in a separate directory and use a separate publishing project to copy it over to the publishing directory. Just add to your publishing-alist:

("blog-static"
 :base-directory "~/Projects/blog/static"
 :base-extension "png\\|jpg\\|css"
 :publishing-directory "~/Projects/blog/publish/static"
 :recursive t
 :publishing-function org-publish-attachment)

Setting up the RSS feed works similarly. The RSS feed is created from a single Org file. Create a new publishing project and put it into your publishing-alist

("blog-rss"
 :base-directory "~/Projects/blog/posts"
 :base-extension "org"
 :publishing-directory "~/Projects/blog/publish"
 :publishing-function (org-rss-publish-to-rss)
 :html-link-home "http://bastibe.de/"
 :html-link-use-abs-url t
 :exclude ".*"
 :include ("rss.org")
 :with-toc nil
 :section-numbers nil
 :title "Bastis Scratchpad on the Internet")

Make sure to exclude this rss.org from the blog-content project by adding it's name to the :exclude variable though. This rss.org file should contain headlines for every blog post. Every headline needs a publishing date and a permalink as property and the body of the post as content:

 * Speeding up Matplotlib
 :PROPERTIES:
 :RSS_PERMALINK: "http://bastibe.de/2013-05-30-speeding-up-matplotlib.html"
 :PUBDATE: <2013-05-30>
 :END:
 #+include: "~/Projects/blog/posts/2013-05-30-speeding-up-matplotlib.org" :lines "4-"

I exclude the first three lines, since they only contain #+title, #+date, and #+tags. You should at least exclude the #+title line. Otherwise, ox-rss will get confused about which title to choose for the feed.

You can even create a meta publishing project that executes all three projects in one fell swoop!

("blog"
 :components ("blog-content" "blog-static" "blog-rss"))

There is one more thing that is kind of fiddly though: As I said, I use Disqus for comments, but I don't want to have comment boxes on the front page or the archive. Thankfully though, ox-html allows you to set :html-preamble and :html-postamble to a function, in which case that function can decide what pre/postamble to draw! The function can take an optional argument that contains a plist of article metadata. In this case, I decide on the :title metadata whether to print the archive link and Disqus, only the archive link, or neither:

:html-postamble
(lambda (info)
  "Do not show disqus for Archive and Recent Posts"
  (cond ((string= (car (plist-get info :title)) "Archive")
         "")
        ((string= (car (plist-get info :title)) "Recent Posts")
         "Other posts")
        (t
    "Other posts
     
     
              
              comments powered by Disqus")))
         :exclude "rss.org\\|archive.org\\|theindex.org")
        ("blog-rss"
         :base-directory "~/Projects/blog/posts"
         :base-extension "org"
         :publishing-directory "~/Projects/blog/publish"
         :publishing-function (org-rss-publish-to-rss)
         :html-link-home "http://bastibe.de/"
         :html-link-use-abs-url t
         :exclude ".*"
         :include ("rss.org")
         :with-toc nil
         :section-numbers nil
         :title "Bastis Scratchpad on the Internet")
        ("blog-static"
         :base-directory "~/Projects/blog/static"
         :base-extension "png\\|jpg\\|css"
         :publishing-directory "~/Projects/blog/publish/static"
         :recursive t
         :publishing-function org-publish-attachment)))

All other sources, including the source code to all blog posts, can be found on Github (the master branch contains HTML, the source branch contains Org).

Addendum: I have since discovered that org-rss-publish-to-rss only handles top-level headlines, but disregards second-level or higher-level headlines. Thus, if you have a post with nested headlines, your RSS feed will only include the text of the top-level one. To fix this, I advised org-rss-publish-to-rss to use org-html-headline for non-top-level headlines like this:

(defadvice org-rss-headline
  (around my-rss-headline (headline contents info) activate)
  "only use org-rss-headline for top level headlines"
  (if (< (org-export-get-relative-level headline info) 2)
      ad-do-it
    (setq ad-return-value (org-html-headline headline contents info))))

Now, the RSS feed includes the full text of all articles.

Speeding up Matplotlib

Thu, 30 May 2013 10:50:00 +0200

For the record, Matplotlib is awesome! Its output looks amazing, it is extremely configurable and very easy to use. What more could you want?

Well… speed. If there is one thing I could criticize about Matplotlib, it is its relative slowness. To measure that, lets make a very simple line plot and draw some random numbers as quickly as possible:

import matplotlib.pyplot as plt
import numpy as np
import time

fig, ax = plt.subplots()

tstart = time.time()
num_plots = 0
while time.time()-tstart < 1:
    ax.clear()
    ax.plot(np.random.randn(100))
    plt.pause(0.001)
    num_plots += 1
print(num_plots)

On my machine, I get about 11 plots per second. I am using pause() here to update the plot without blocking. The correct way to do this is to use draw() instead, but due to a bug in the Qt4Agg backend, you can't use it there. If you are not using the Qt4Agg backend, draw() is supposedly the correct choice.

For a single plot, ten plots per second is not terrible. But then, this is really the simplest case possible, so ten frames per second in the simplest case probably means bad things for not so simple cases.

One thing that really takes time here is creating all the axes and text labels over and over again. So let's not do that.

Instead of calling clear() and then plot(), thus effectively deleting everything about the plot, then re-creating it for every frame, we can keep an existing plot and only modify its data:

fig, ax = plt.subplots()
line, = ax.plot(np.random.randn(100))

tstart = time.time()
num_plots = 0
while time.time()-tstart < 1:
    line.set_ydata(np.random.randn(100))
    plt.pause(0.001)
    num_plots += 1
print(num_plots)

which yields about 26 plots per second. Not bad for a simple change like this. The downside is that the axes are not re-scaled any longer when the data changes. Thus, they won't change their limits based on the data any more.

Profiling this yields some interesting results:

ncalls	tottime	percall	cumtime	percall	filename:lineno(function)
15	0.167	0.011	0.167	0.011	{built-in method sleep)

The one function that uses the biggest chunk of runtime is sleep(), of all things. Clearly, this is not what we want. Delving deeper into the profiler shows that this is indeed happening in the call do pause(). Then again, I was wondering if using pause really was a great idea for performance…

As it turns out, pause() internally calls fig.canvas.draw(), then plt.show(), then fig.canvas.start_event_loop(). The default implementation of fig.canvas.start_event_loop() then calls fig.canvas.flush_events(), then sleeps for the requested time. To add insult to injury, it even insists on sleeping at least one hundredth of a second, which actually explains the profiler output of 0.167 seconds of sleep() for 15 calls very well.

Putting this all together now yields:

fig, ax = plt.subplots()
line, = ax.plot(np.random.randn(100))

tstart = time.time()
num_plots = 0
while time.time()-tstart < 1:
    line.set_ydata(np.random.randn(100))
    fig.canvas.draw()
    fig.canvas.flush_events()
    num_plots += 1
print(num_plots)

which now plots about 40 frames per second. Note that the call to show() mentioned earlier can be omitted since the figure is already on screen. flush_events() just runs the Qt event loop, so there is probably nothing to optimize there.

The only thing left to optimize now is thus fig.canvas.draw(). What this really is doing is drawing all the artists contained in the ax. Those artists can be accessed using ax.get_children(). For a simple plot like this, the artists are:

the background ax.patch
the line, as returned from the plot() function
the spines ax.spines
the axes ax.xaxis and ax.yaxis

What we can do here is to selectively draw only the parts that are actually changing. That is, at least the background and the line. To only redraw these, the code now looks like this:

fig, ax = plt.subplots()
line, = ax.plot(np.random.randn(100))
plt.show(block=False)

tstart = time.time()
num_plots = 0
while time.time()-tstart < 5:
    line.set_ydata(np.random.randn(100))
    ax.draw_artist(ax.patch)
    ax.draw_artist(line)
    fig.canvas.update()
    fig.canvas.flush_events()
    num_plots += 1
print(num_plots/5)

Note that you have to add fig.canvas.update() to copy the newly rendered lines to the drawing backend.

This now plots about 500 frames per second. Five hundred times per second! Frankly, this is quite amazing!

Note that since we are only redrawing the background and the line, some detail in the axes will be overwritten. To also draw the spines, use for spine in ax.spines.values(): ax.draw_artist(spine). To draw the axes, use ax.draw_artist(ax.xaxis) and ax.draw_artist(ax.yaxis). If you draw all of them, you get roughly the same performance as fig.canvas.draw(). The axes in particular are quite expensive.

There is also a way of drawing the complete figure once and copying the complete but empty background, then reinstating that and only plotting a new line on top of it. This is equally fast as the code above without any visual artifacts, but breaks if you resize the plot.

In conclusion, I am quite impressed with the flexibility of Matplotlib. Matplotlib by default values quality over performance. But if you really need the performance at some point, it is flexible and hackable enough to let you tweak it to your hearts content. Really, an amazing piece of technology!

EDIT: As it turns out, fig.canvas.blit(ax.bbox) is a bad idea since it leaks memory like crazy. What you should use instead is fig.canvas.update(), which is equally fast but does not leak memory.

A Python Primer for Matlab Users

Sun, 20 Jan 2013 10:55:00 +0100

Why would you want to use Python over Matlab?

Because Python is free and Matlab is not.
Because Python is a general purpose programming language and Matlab is not.

Let me qualify that a bit. Matlab is a very useful programming environment for numerical problems. For a very particular set of problems, Matlab is an awesome tool. For many other problems however, it is just about unusable. For example, you would not write a complex GUI program in Matlab, you would not write your blogging engine in Matlab and you would not write a web service in Matlab. You can do all that and more in Python.

Python as a Matlab replacement

The biggest strength of Matlab is its matrix engine. Most of the data you work with in Matlab are matrices and there is a host of functions available to manipulate and visualize those matrices. Python, by itself, does not have a convenient matrix engine. However, there are three packages (think Matlab Toolboxes) out there that will add this capability to Python:

Numpy (the matrix engine)
Scipy (matrix manipulation)
Matplotlib (plotting)

You can either grab the individual installers for Python, Numpy, Scipy and Matplotlib from their respective websites, or get them pre-packaged from pythonxy() or EPD.

A 30,000 foot overview

Like Matlab, Python is interpreted, that is, there is no need for a compiler and code can be executed at any time as long as Python is installed on the machine. Also, code can be copied from one machine to another and will run without change.

Like Matlab, Python is dynamically typed, that is, every variable can hold data of any type, as in:

    # Python
    a = 5         # a number
    a = [1, 2, 3] # a list
    a = 'text'    # a string

Contrast this with C, where you can not assign different data types to the same variable:

    // C
    int a = 5;
    float b[3] = {1.0, 2.0, 3.0};
    char c[] = "text";

Unlike Matlab, Python is strongly typed, that is, you can not add a number to a string. In Matlab, adding a single number to a string will convert that string into an array of numbers, then add the single number to each of the numbers in the array. Python will simply throw an error.

    % Matlab
    a = 'text'
    b = a + 5 % [121 106 125 121]

    # Python
    a = 'text'
    b = a + 5 # TypeError: Can't convert 'int' object to str implicitly

Unlike Matlab, every Python file can contain as many functions as you like. Basically, you can organize your code in as many files as you want. To access functions from other files, use import filename.

Unlike Matlab, Python is very quick to start. In fact, most operating systems automatically start a new Python process whenever you run a Python program and quit that process once the program has finished. Thus, every Python program behaves as if it indeed were an independent program. There is no need to wait for that big Matlab mother ship to start before writing or executing code.

Unlike Matlab, the source code of Python is readily available. Every detail of Python's inner workings is available to everyone. It is thus feasible and encouraged to actively participate in the development of Python itself or some add-on package. Furthermore, there is no dependence on some company deciding where to go next with Python.

Reading Python

When you start up Python, it is a rather empty environment. In order to do anything useful, you first have to import some functionality into your workspace. Thus, you will see a few lines of import statements at the top of every Python file. Moreover, Python has namespaces, so if you import numpy, you will have to prefix every feature of Numpy with its name, like this:

    import numpy
    a = numpy.zeros(10, 1)

This is clearly cumbersome if you are planning to use Numpy all the time. So instead, you can import all of Numpy into the global environment like this:

    from numpy import *
    a = ones(30, 1)

Better yet, there is a pre-packaged namespace that contains the whole Numpy-Scipy-Matplotlib stack in one piece:

    from pylab import *
    a = randn(100, 1)
    plot(a)
    show()

Note that Python does not plot immediately when you type plot(). Instead, it will collect all plotting information and only show it on the screen once you type show().

So far, the code you have seen should look pretty familiar. A few differences:

No semicolons at the end of lines; In order to print stuff to the console, use the print() function instead.
No end anywhere. In Python, blocks of code are identified by indentation and they always start with a colon like so:

    sum = 0
    for n in [1, 2, 3, 4, 5]:
        sum = sum + n
    print(sum)

Function definitions are different. They use the def keyword instead of function. You don't have to name the output variable names in the definition and instead use return().

    # Python
    def abs(number):
        if number > 0:
            return number
        else:
            return -number

    % Matlab
    function [out] = abs(number)
        if number > 0
            out = number
        else
            out = -number
        end
    end

There is no easy way to write out a list or matrix. Since Python only gains a matrix engine by importing Numpy, it does not have a convenient way of writing arrays or matrices. This sounds more inconvenient than it actually is, since you are probably using mostly functions like zeros() or randn() anyway and those work just fine. Also, many places accept Python lists (like this [1, 2, 3]) instead of Numpy arrays, so this rarely is a problem. Note that you must use commas to separate items and can not use semicolons to separate lines.

    # create a numpy matrix:
    m = array([[1, 2, 3],
               [4, 5, 6],
               [7, 8, 9]])
    # create a Python list:
    l = [1 2 3]

Arrays access uses brackets and is numbered from 0. Thus, ranges exclude the last number (see below). Mostly, this just means that array access does not need any +1 or -1 when indexing arrays anymore.

    a = linspace(1, 10, 10)
    one = a[0]
    two = a[1]

    # "6:8" is a range of two elements:
    a[6:8] = [70, 80] # <-- a Python list!

Common traps

Array slicing does not copy.

    a = array([1 2 3 4 5])
    b = a[1:4] # [2 3 4]
    b[1] = rand() # this will change a and b!
    # make a copy like this:
    c = array(a[1:4], copy=True) # copy=True can be omitted
    c[1] = rand() # changes only c

Arrays retain their data type. You can slice them, you can dice them, you can do math on them, but a 16 bit integer array will never lose its data type. Use new = array(old, dtype=double) to convert an array of any data type to the default double type (like in Matlab).

    # pretend this came from a wave file:
    a = array([1000, 2000, 3000, 4000, 5000], dtype=int16)
    a = a * 10 # int16 only goes to 32768!
    # a is now [10000, 20000, 30000, -25536, -15536]

Going further

Now you should be able to read Python code reasonably well. Numpy, Scipy and Matplotlib are actually modeled after Matlab in many ways, so many functions will have a very similar name and functionality. A lot of the numerical code you write in Python will look very similar to the equivalent code in Matlab. For a more in-depth comparison of Matlab and Python syntax, head over to the Numpy documentation for Matlab users.

However, since Python is a general purpose programming language, it offers some more tools. To begin with, there are a few more data types like associative arrays, tuples (unchangeable lists), proper strings and a full-featured object system. Then, there is a plethora of add-on packages, most of which actually come with your standard installation of Python. For example, there are internet protocols, GUI programming frameworks, real-time audio interfaces, web frameworks and game development libraries. Even this very blog is created using a Python static site generator.

Lastly, Python has a great online documentation site including a tutorial, there are many books on Python and there is a helpful Wiki on Python. There is also a tutorial and documentation for Numpy, Scipy and Matplotlib.

A great way to get to know any programming language is to solve the first few problems on project euler.

Real Time Signal Processing in Python

Fri, 02 Nov 2012 10:53:00 +0100

Wouldn't it be nice if you could do real time audio processing in a convenient programming language? Matlab comes to mind as a convenient language for signal processing. But while Matlab is pretty fast, it is really only fast for algorithms that can be vectorized. In audio however, we have many algorithms that need knowledge about the previous sample to calculate the next one, so they can't be vectorized.

But this is not going to be about Matlab. This is going to be about Python. Combine Python with Numpy (and Scipy and Matplotlib) and you have a signal processing system very comparable to Matlab. Additionally, you can do real-time audio input/output using PyAudio. PyAudio is a wrapper around PortAudio and provides cross platform audio recording/playback in a nice, pythonic way. (Real time capabilities were added in 0.2.6 with the help of yours truly).

However, this does not solve the problem with vectorization. Just like Matlab, Python/Numpy is only fast for vectorizable algorithms. So as an example, let's define an iterative algorithm that is not vectorizable:

A Simple Limiter

A limiter is an audio effect that controls the system gain so that it does not exceed a certain threshold level. One could do this by simply cutting off any signal peaks above that level, but that sounds awful. So instead, the whole system gain is reduced smoothly if the signal gets too loud and is amplified back to its original gain again when it does not exceed the threshold any more. The important part is that the gain change is done smoothly, since otherwise it would introduce a lot of distortion.

If a signal peak is detected, the limiter will thus need a certain amount of time to reduce the gain accordingly. If you still want to prevent all peaks, the limiter will have to know of the peaks in advance, which is of course impossible in a real time system. Instead, the signal is delayed by a short time to give the limiter time to adjust the system gain before the peak is actually played. To keep this delay as short as possible, this "attack" phase where the gain is decreased should be very short, too. "Releasing" the gain back up to its original value can be done more slowly, thus introducing less distortion.

With that out of the way, let me present you a simple implementation of such a limiter. First, lets define a signal envelope $e[n]$ that catches all the peaks and smoothly decays after them:

\[ e[n] = \max( |s[n]|, e[n-1] \cdot f_r ) \]

where $s[n]$ is the current signal and $0 < f_r < 1$ is a release factor.

If this is applied to a signal, it will create an envelope like this:

Based on that envelope, and assuming that the signal ranges from -1 to 1, the target gain $g_t[n]$ can be calculated using

\[ g_t[n] = \begin{cases} 1 & e[n] < t \\ 1 + t - e[n] & e[n] > t \end{cases} \]

Now, the output gain $g[n]$ can smoothly move towards that target gain using

\[ g[n] = g[n-1] \cdot f_a + g_t[n] \cdot (1-f_a) \]

where $0 < f_a \ll f_r$ is the attack factor.

Here you can see how that would look in practice:

Zooming in on one of the limited section reveals that the gain is actually moving smoothly.

This gain can now be multiplied on the delayed input signal and will safely keep that below the threshold.

In Python, this might look like this:

    class Limiter:
        def __init__(self, attack_coeff, release_coeff, delay, dtype=float32):
            self.delay_index = 0
            self.envelope = 0
            self.gain = 1
            self.delay = delay
            self.delay_line = zeros(delay, dtype=dtype)
            self.release_coeff = release_coeff
            self.attack_coeff = attack_coeff

        def limit(self, signal, threshold):
            for i in arange(len(signal)):
                self.delay_line[self.delay_index] = signal[i]
                self.delay_index = (self.delay_index + 1) % self.delay

                # calculate an envelope of the signal
                self.envelope *= self.release_coeff
                self.envelope  = max(abs(signal[i]), self.envelope)

                # have self.gain go towards a desired limiter gain
                if self.envelope > threshold:
                    target_gain = (1+threshold-self.envelope)
                else:
                    target_gain = 1.0
                self.gain = ( self.gain*self.attack_coeff +
                              target_gain*(1-self.attack_coeff) )

                # limit the delayed signal
                signal[i] = self.delay_line[self.delay_index] * self.gain

Note that this limiter does not actually clip all peaks completely, since the envelope for a single peak will have decayed a bit before the target gain will have reached it. Thus, the output gain will actually be slightly higher than what would be necessary to limit the output to the threshold. Since the attack factor is supposed to be significantly smaller than the release factor, this does not matter much though.

Also, it would probably be more useful to define the factors $f_a$ and $f_r$ in terms of the time they take to reach their target and the threshold $t$ in dB FS.

Implementing audio processing in Python

A real-time audio processing framework using PyAudio would look like this:

(callback is a function that will be defined shortly)

    from pyaudio import PyAudio, paFloat32

    pa = PyAudio()

    stream = pa.open(format = paFloat32,
                     channels = 1,
                     rate = 44100,
                     output = True,
                     frames_per_buffer = 1024,
                     stream_callback = callback)

    while stream.is_active():
        sleep(0.1)

    stream.close()
    pa.terminate()

This will open a stream, which is a PyAudio construct that manages input and output to/from one sound device. In this case, it is configured to use float values, only open one channel, play audio at a sample rate of 44100 Hz, have that one channel be output only and call the function callback every 1024 samples.

Since the callback will be executed on a different thread, control flow will continue immediately after pa.open(). In order to analyze the resulting signal, the while stream.is_active() loop waits until the signal has been processed completely.

Every time the callback is called, it will have to return 1024 samples of audio data. Using the class Limiter above, a sample counter counter and an audio signal signal, this can be implemented like this:

    limiter = Limiter(attack_coeff, release_coeff, delay, dtype)

    def callback(in_data, frame_count, time_info, flag):
        if flag:
            print("Playback Error: %i" % flag)
        played_frames = counter
        counter += frame_count
        limiter.limit(signal[played_frames:counter], threshold)
        return signal[played_frames:counter], paContinue

The paContinue at the end is a flag signifying that the audio processing is not done yet and the callback wants to be called again. Returning paComplete or an insufficient number of samples instead would stop audio processing after the current block and thus invalidate stream.is_active() and resume control flow in the snippet above.

Now this will run the limiter and play back the result. Sadly however, Python is just a bit too slow to make this work reliably. Even with a long block size of 1024 samples, this will result in occasional hickups and discontinuities. (Which the callback will display in the print(...) statement).

Speeding up execution using Cython

The limiter defined above could be rewritten in C like this:

    // this corresponds to the Python Limiter class.
    typedef struct limiter_state_t {
        int delay_index;
        int delay_length;
        float envelope;
        float current_gain;
        float attack_coeff;
        float release_coeff;
    } limiter_state;

    #define MAX(x,y) ((x)>(y)?(x):(y))

    // this corresponds to the Python __init__ function.
    limiter_state init_limiter(float attack_coeff, float release_coeff, int delay_len) {
        limiter_state state;
        state.attack_coeff = attack_coeff;
        state.release_coeff = release_coeff;
        state.delay_index = 0;
        state.envelope = 0;
        state.current_gain = 1;
        state.delay_length = delay_len;
        return state;
    }

    void limit(float *signal, int block_length, float threshold,
               float *delay_line, limiter_state *state) {
        for(int i=0; idelay_index] = signal[i];
            state->delay_index = (state->delay_index + 1) % state->delay_length;

            // calculate an envelope of the signal
            state->envelope *= state->release_coeff;
            state->envelope = MAX(fabs(signal[i]), state->envelope);

            // have current_gain go towards a desired limiter target_gain
            float target_gain;
            if (state->envelope > threshold)
                target_gain = (1+threshold-state->envelope);
            else
                target_gain = 1.0;
            state->current_gain = state->current_gain*state->attack_coeff +
                target_gain*(1-state->attack_coeff);

            // limit the delayed signal
            signal[i] = delay_line[state->delay_index] * state->current_gain;
        }
    }

In contrast to the Python version, the delay line will be passed to the limit function. This is advantageous because now all audio buffers can be managed by Python instead of manually allocating and deallocating them in C.

Now in order to plug this code into Python I will use Cython. First of all, a "Cython header" file has to be created that declares all exported types and functions to Cython:

    cdef extern from "limiter.h":
        ctypedef struct limiter_state:
            int delay_index
            int delay_length
            float envelope
            float current_gain
            float attack_coeff
            float release_coeff

        limiter_state init_limiter(float attack_factor, float release_factor, int delay_len)
        void limit(float *signal, int block_length, float threshold,
                   float *delay_line, limiter_state *state)

This is very similar to the C header file of the limiter:

    typedef struct limiter_state_t {
        int delay_index;
        int delay_length;
        float envelope;
        float current_gain;
        float attack_coeff;
        float release_coeff;
    } limiter_state;

    limiter_state init_limiter(float attack_factor, float release_factor, int delay_len);
    void limit(float *signal, int block_length, float threshold,
               float *delay_line, limiter_state *state);

With that squared away, the C functions are accessible for Cython. Now, we only need a small Python wrapper around this code so it becomes usable from Python:

    import numpy as np
    cimport numpy as np
    cimport limiter

    DTYPE = np.float32
    ctypedef np.float32_t DTYPE_t

    cdef class Limiter:
        cdef limiter.limiter_state state
        cdef np.ndarray delay_line
        def __init__(self, float attack_coeff, float release_coeff,
                     int delay_length):
            self.state = limiter.init_limiter(attack_coeff, release_coeff, delay_length)
            self.delay_line = np.zeros(delay_length, dtype=DTYPE)

        def limit(self, np.ndarray[DTYPE_t,ndim=1] signal, float threshold):
            limiter.limit(np.PyArray_DATA(signal),
                       len(signal), threshold,
                       np.PyArray_DATA(self.delay_line),
                       &self.state)

The first two lines set this file up to access Numpy arrays both from the Python domain and the C domain, thus bridging the gap. The cimport limiter imports the C functions and types from above. The DTYPE stuff is advertising the Numpy float32 type to C.

The class is defined using cdef as a C data structure for speed. Also, Cython would naturally translate every C struct into a Python dict and vice versa, but we need to pass the struct to limit and have limit modify it. Thus, cdef limiter.limiter_state state makes Cython treat it as a C struct only. Finally, the np.PyArray_DATA() expressions expose the C arrays underlying the Numpy vectors. This is really handy since we don't have to copy any data around in order to modify the vectors from C.

As can be seen, the Cython implementation behaves nearly identically to the initial Python implementation (except for passing the dtype to the constructor) and can be used as a plug-in replacement (with the aforementioned caveat).

Finally, we need to build the whole contraption. The easiest way to do this is to use a setup file like this:

    from distutils.core import setup
    from distutils.extension import Extension
    from Cython.Distutils import build_ext
    from numpy import get_include

    ext_modules = [Extension("cython_limiter",
                             sources=["cython_limiter.pyx",
                                      "limiter.c"],
                             include_dirs=['.', get_include()])]

    setup(
        name = "cython_limiter",
        cmdclass = {'build_ext': build_ext},
        ext_modules = ext_modules
        )

With that saved as setup.py, python setup.py build_ext --inplace will convert the Cython code to C, and then compile both the converted Cython code and C code into a binary Python module.

Conclusion

In this article, I developed a simple limiter and how to implement it in both C and Python. Then, I showed how to use the C implementation from Python. Where the Python implementation is struggling to keep a steady frame rate going even at large block sizes, the Cython version runs smoothly down to 2-4 samples per block on a 2 Ghz Core i7. Thus, real-time audio processing is clearly feasable using Python, Cython, Numpy and PyAudio.

You can find all the source code in this article at https://github.com/bastibe/simple-cython-limiter

Disclaimer

I invented this limiter myself. I could invent a better sounding limiter, but this article is more about how to combine Python, Numpy, PyAudio and Cython for real-time signal processing than about limiter design.
I recently worked on something similar at my day job. They agreed that I could write about it so long as I don't divulge any company secrets. This limiter is not a descendant of any code I worked on.
Whoever wants to use any piece of code here, feel free to do so. I am hereby placing it in the public domain. Feel free to contact me if you have questions.

An error occurred loading this content. Try again later.

Thu, 01 Nov 2012 18:26:00 +0100

Since we moved, our Apple TV has been giving us trouble: we would rent a movie on the Apple TV, and instead of playing it, the Apple TV would just display

An error occurred loading this content. Try again later.

Rebooting, then trying again, would sometimes solve the issue. Trying to play a trailer before trying to play the movie seemed to increase chances of the Apple TV actually downloading the movie, too.

Finally, some forum or other pointed me to the correct solution: For some reason, the Apple TV did not play nice with our ISP DNS server. Simply changing the DNS server in the Apple TV settings to one of the Google DNS servers (8.8.8.8 and 8.8.4.4) solved the issue. Any other DNS will probably work, too.

Names

Sun, 28 Oct 2012 22:08:00 +0100

Names are everywhere in software. We name our variables, our functions, our arguments, classes, and packages. We name our source files and the directories that contain them. We name our jar files and war files and ear files. We name and name and name. Because we do so much of it, we'd better do it well.

– from the Introduction to chapter 2 "Meaningful Names" of "Clean Code" by Robert C. Martin.

Indeed we name a lot of things in software. As The Structure and Interpretation of Computer Programs points out, the primary purpose of a function (lambda) is to provide names for its arguments that are independent of names elsewhere. A function provides a closure in which stuff has defined names. The closure itself can then be embedded into other closures to form composite structures. Take any complex program structure and decompose its names through all its lambdas and you will only find more names right until you reach turtles.

At its heart, programming is about naming things. If I squint my eyes a little, I can nearly convince myself that naming is really all there is. All the rest is just playing games with syntax.

It's situations like this that I realize that The Structure and Interpretation of Computer Programs really changed how I view programming.

If you like programming at all, I implore you to read it, or watch it, or buy it. It really blew my mind.

Matlab, Mex, Homebrew and OS X 10.8 Mountain Lion

Sat, 20 Oct 2012 10:12:00 +0200

Now that I am a student again, I have to use Matlab again. Among the many joys of Matlab is the compilation of mex files.

Because it does not work. So angry.

Basically, mex does not work because it assumes that you have OS X 10.6 installed. In OS X 10.6 you had gcc-4.2 and your system SDK was stored in \/Developer\/SDKs\/MacOSX10.6.sdk. However, as of 10.7 (I think), the \/Developer directory has been deprecated in favor of distributing the whole development environment within the App package of XCode. Also, gcc has been deprecated in favor of clang. While a gcc binary is still provided, gcc-4.2 is not. Of course, that is what mex relies on. Lastly, mex of course completely disregards common system paths such as, say, \/usr\/local\/bin, so compiling against some homebrew library won't work.

At least these things are rather easy to fix, since all these settings are saved in a file called mexopts.sh, which is saved to ~\.matlab\/R2012a\// by default. The relevant section on 64-bit OS X begins after maci64) and should look like this: (changes are marked by comments)

    #----------------------------------------------------------------------------
                # StorageVersion: 1.0
                # CkeyName: GNU C
                # CkeyManufacturer: GNU
                # CkeyLanguage: C
                # CkeyVersion:
                CC='gcc' # used to be 'gcc-4.2'

                # used to be '/Developer/SDKs/MacOSX10.6.sdk'
                SDKROOT='/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.8.sdk'
                MACOSX_DEPLOYMENT_TARGET='10.8' # used to be '10.5'
                ARCHS='x86_64'
                CFLAGS="-fno-common -no-cpp-precomp -arch $ARCHS -isysroot $SDKROOT -mmacosx-version-min=$MACOSX_DEPLOYMENT_TARGET"
                CFLAGS="$CFLAGS  -fexceptions"
                CFLAGS="$CFLAGS -I/usr/local/include" # Homebrew include path
                CLIBS="$MLIBS"
                COPTIMFLAGS='-O2 -DNDEBUG'
                CDEBUGFLAGS='-g'
    #
                CLIBS="$CLIBS -lstdc++"
                # C++keyName: GNU C++
                # C++keyManufacturer: GNU
                # C++keyLanguage: C++
                # C++keyVersion:
                CXX=g++ # used to be 'g++-4.2'
                CXXFLAGS="-fno-common -no-cpp-precomp -fexceptions -arch $ARCHS -isysroot $SDKROOT -mmacosx-version-min=$MACOSX_DEPLOYMENT_TARGET"
                CXXLIBS="$MLIBS -lstdc++"
                CXXOPTIMFLAGS='-O2 -DNDEBUG'
                CXXDEBUGFLAGS='-g'
    #
                # FortrankeyName: GNU Fortran
                # FortrankeyManufacturer: GNU
                # FortrankeyLanguage: Fortran
                # FortrankeyVersion:
                FC='gfortran'
                FFLAGS='-fexceptions -m64 -fbackslash'
                FC_LIBDIR=`$FC -print-file-name=libgfortran.dylib 2>&1 | sed -n '1s/\/*libgfortran\.dylib//p'`
                FC_LIBDIR2=`$FC -print-file-name=libgfortranbegin.a 2>&1 | sed -n '1s/\/*libgfortranbegin\.a//p'`
                FLIBS="$MLIBS -L$FC_LIBDIR -lgfortran -L$FC_LIBDIR2 -lgfortranbegin"
                FOPTIMFLAGS='-O'
                FDEBUGFLAGS='-g'
    #
                LD="$CC"
                LDEXTENSION='.mexmaci64'
                LDFLAGS="-Wl,-twolevel_namespace -undefined error -arch $ARCHS -Wl,-syslibroot,$SDKROOT -mmacosx-version-min=$MACOSX_DEPLOYMENT_TARGET"
                LDFLAGS="$LDFLAGS -bundle -Wl,-exported_symbols_list,$TMW_ROOT/extern/lib/$Arch/$MAPFILE"
                LDFLAGS="$LDFLAGS -L/usr/local/lib" # Homebrew library path
                LDOPTIMFLAGS='-O'
                LDDEBUGFLAGS='-g'
    #
                POSTLINK_CMDS=':'
    #----------------------------------------------------------------------------

To summarize:

changed gcc-4.2 to gcc
changed /Developer/SDKs/MacOSX10.6.sdk to /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.8.sdk
changed 10.5 to 10.8
added CFLAGS="$CFLAGS -I/usr/local/include"
changed g++-4.2 to g++
added LDFLAGS="$LDFLAGS -L/usr/local/lib"

With those settings, the mex compiler should work and it should pick up any libraries installed by homebrew.

My Emacs customizations

Sun, 14 Oct 2012 09:14:00 +0200

I don't host my .emacs in a repository. I tried it for a while, but it did not work for me. I think repos are great for managing multiple divergent versions of the same source tree. However, my dotfiles should never diverge, they should just be kept in sync. This is what Dropbox is great at. So I use Dropbox instead of git.

One downside of that is that it is not as easy to provide a public link to my dotfiles. Or maybe it is. Here goes

My .emacs

Now on to the meat of this post: Some customizations I made that I think are cool.

When Emacs starts up, make it fit the left half of the screen

First, I need a function to set the size of a frame in terms of pixels. Emacs only provides set-frame-size, but that works on characters, not pixels.

(defun set-frame-pixel-size (frame width height)
  "Sets size of FRAME to WIDTH by HEIGHT, measured in pixels."
  (let ((pixels-per-char-width (/ (frame-pixel-width) (frame-width)))
		(pixels-per-char-height (/ (frame-pixel-height) (frame-height))))
	(set-frame-size frame
					(floor (/ width pixels-per-char-width))
					(floor (/ height pixels-per-char-height)))))

Next, a function that uses the above to set the frame size to exactly fit the left half of the screen. Note that the excess-... variables account for things like the menu bar, the dock or the task bar so you might have to adapt these values to your particular windowing environment. The values given here work for dockless OSX.

(defun use-left-half-screen ()
  (interactive)
  (let* ((excess-width 32)
		 (excess-height 48)
		 (half-screen-width (- (/ (x-display-pixel-width) 2) excess-width))
		 (screen-height (- (x-display-pixel-height) excess-height)))
	(set-frame-pixel-size (selected-frame) half-screen-width screen-height)))

Finally, when Emacs starts up, make it use half the screen. Just how I like it!

(if window-system
	(use-left-half-screen))

Make shell split windows auto-resize

Usually when you open a new split window in Emacs, it takes up half the frame. For some kinds of windows, I like them to be smaller. So here is a little snippet that shrinks some kinds of windows to 15 lines of height.

(add-hook 'window-configuration-change-hook
		  (lambda ()
			(when (or (string-equal (buffer-name) "*Python*")
					  (string-equal (buffer-name) "*eshell*")
					  (string-equal (buffer-name) "*tex-shell*"))
			  (if (not (eq (window-height) 15))
				  (enlarge-window (- 15 (window-height)))))))

Create a new line above/below the current one

There is one feature of Vim that I really missed: o~/~O, which 'opens' a new line above or below the current one without changing the current line.

(defun vi-open-line-above ()
  "Insert a newline above the current line and put point at beginning."
  (interactive)
  (unless (bolp)
    (beginning-of-line))
  (newline)
  (forward-line -1)
  (indent-according-to-mode))

(defun vi-open-line-below ()
  "Insert a newline below the current line and put point at beginning."
  (interactive)
  (unless (eolp)
    (end-of-line))
  (newline-and-indent))

(global-set-key (kbd "C-o") 'vi-open-line-below)
(global-set-key (kbd "M-o") 'vi-open-line-above)

When editing LaTeX, show PDF output in Emacs

Emacs provides an awesome mode for editing LaTeX files. But isn't it sad that you always have to leave Emacs for viewing the PDF? Well, not any more.

This function will open or refresh a split window with the generated PDF file in it. Thus, when I edit LaTeX, I will hit C-c C-c to compile, then C-c C-v to see the PDF. All that without having to leave the LaTeX file.

;; open/show pdf file within Emacs using doc-view-mode
(defun open-show-pdf ()
  (interactive)
  (let ((tex-buffer-name (buffer-name))
		(pdf-buffer-name (concat (TeX-master-file) ".pdf")))
	(if (get-buffer pdf-buffer-name)
		(switch-to-buffer-other-window pdf-buffer-name)
	  (find-file-other-window pdf-buffer-name))
	(if (not (eq major-mode 'doc-view-mode))
		(doc-view-mode))
	(doc-view-revert-buffer t t)
	(switch-to-buffer-other-window tex-buffer-name)))

(add-hook 'LaTeX-mode-hook
		  (lambda ()
			(define-key LaTeX-mode-map (kbd "C-c C-v") 'open-show-pdf)
			(visual-line-mode t)
			(turn-on-reftex))
		  t)

Text editing with confidence and Emacs

Sat, 13 Oct 2012 11:30:00 +0200

In college, I realized for the first time how a good text editor could save me serious time, when I dragged an image file into Textmate and it auto-inserted all the LaTeX boilerplate for a figure. A few years later, on my first job, I learned Vim key bindings for Visual Studio because I hated text editing in Visual Studio so much. This showed me another way how a good text editor could save me serious time. A year after that, I was bored and tried Emacs. With Emacs, I discovered the marvelous world of REPLs and outlining.

Note that this is not supposed to be a comparison between text editors. I have done that already. This will be a collection of some of the coolest things my text editor of choice is capable of. Stuff I love!

Text editing

In my admittedly short history of working with computers and text professionally I have realized a universal truth about the tools I like to use: A great tool is a tool I can trust. A tool I can use with confidence because I know that it will do what I want. Furthermore, a great tool is a tool that does what I want with a minimum amount of friction.

Here is one of my pet peeve with many text editors: Whenever I initiate a text search, a modal dialog box pops up where I have to enter the search phrase, then click a button to search for that word. In contrast great text editors allow searching for text without changing to a different window and show the results immediately while entering the search string. This is very fast and precise. In fact, it is so useful that searching is one of the most popular means of text navigation in text editors that have it.

This is a theme that goes through all these features: I look for stuff that is fast and precise, because this will enable more powerful applications of seemingly simple techniques.

Outlining and task management

Outlining is writing an hierarchical list of stuff that can be reordered and sub-trees can be hidden. I tried some graphical outliners before. I did not quite see the point of it. Instead, I always kept my todo lists in simple Markdown files. It worked well enough.

But with Emacs, I found out about org-mode, which is at its heart an outliner with a syntax somewhat like Markdown. It proved to be surprisingly powerful to be able to easily refile entries or whole sub-trees in my todo lists. Also, having trees be collapsible effectively enabled me to consolidate all my todo management into one file without that file becoming unmanageably big.

Add to that the more advanced task tracking features of org-mode, such as time tracking or advanced todo planning and this has become one of the most pivotal tools I use daily.

Shell interaction and operating system compatibility

Whenever my job required me to work on Windows, I often found it jarring to not have a command line available. Granted, there is Cygwin, but that does not play with the Windows directory structure too well. Also, many programs behave slightly differently on Linux, Cygwin and OSX. Basically, it drove me nuts.

Again, Emacs came to the rescue though: Eshell is a shell implemented in Emacs with no external dependencies. Thus, it comes installed on all my machines and works the same way on every operating system. Sadly though, Eshell is not quite full-featured and compatibility with some shell programs is kind of rough. In particular, it does not work well with less like scrolling buffers. However, some of those, like man or info are automatically opened in a special mode in Emacs itself, so this is not as bad as it sounds.

At more than one point, Eshell probably saved my sanity.

Git interaction

Source control is a crucial tool when working with source code. In college we used SVN. Even cooler is Git, though the initial learning curve is pretty bad. Personally, I learned it by mucking around on the command line and breaking stuff there, then using Tower to restore the repository to some sane state. Sadly though, Tower is OSX only, so I was kind of screwed on Windows and Linux.

Again, enter Emacs: Emacs has this magical mode called magit, which builds an interactive git interaction program within Emacs. With magit, all the major git commands are just a keystroke away and diffs or logs are easily accessible, too. It really is a major feat!

Grab bag

Emacs' integrated ediff is a joy to use.
REPLs are one honking great idea and come with most language modes for dynamic languages in Emacs.
Emacs' LaTeX mode is amazingly powerful. Combine that with Emacs' ability to display PDF files graphically in a buffer and you have an awesome LaTeX environment.
Emacs' documentation is wonderful. I learned many a trick just idly leafing through the built in Emacs documentation.
Built in package management for extensions is a great time saver. This is available in Vim and Sublime Text as an addon, too.
Lisp is indeed beautiful and elisp is quite joyful to program. I never really went beyond simple configuration in Vim. My .emacs file contains some quite sophisticated small programs though.

A Story about Schemes

Thu, 20 Sep 2012 20:21:00 +0200

Disclaimer: If you are a programmer, or wanting to be a programmer, or interested in programming, or, well, reading this, you should absolutely, positively watch the 1986 MIT lecture about The Structure and Interpretation of Computer Programs. (Most conveniently available here)

I tried reading the book numerous times, but it was too dry for my taste. The lecture however is juicy, brain-bending bliss. Especially if you don't know much functional programming yet. It certainly blew my mind. Frequently. Like, every ten minutes. Basically, I feel reborn as a programmer, with a new sense of what I am doing and how it should be done.

However, this is not about SICP, it is about Scheme, which happens to be the programming language that SICP uses to preach its sermons. Above all, Scheme beautiful. It is an astounding marriage of simplicity and power. It is also a mess in terms of implementations. There are dozens of implementations and they all implement a different, partly overlapping set of features that may or may not be part of the canonical Scheme–if there is such a thing.

So, how to select the correct Scheme? Naively, I first chose the one SICP uses, MIT-Scheme. However, I soon found out that it does not work well with Emacs (that is, it did not work at all with scheme-mode and run-scheme). Also, its prompt is weird: ]=>. Whatever.

So next, I tried Gambit Scheme, which works a lot better with Emacs. But then, I soon found out that it does not support functions like filter, which are kind of essential. I was quick to write my own version of that, but really, this stuff should be provided. Turns out, filter is part of a Scheme Request For Implementation, or SRFI. filter is part of SRFI1. There is a huge list of different Schemes and the SRFIs they support. Also, there is a list of Schemes sorted by performance. With plots, even. It's a mess.

So this left me profoundly confused. What Scheme would I use? Ideally, it should be fast, work with Emacs and implement a reasonable set of SRFIs. Homebrew lists these:

bigloo	chibi-scheme	chicken
gambit-scheme	gauche	guile
kawa	mit-scheme	plt-racket
scheme48	scsh	sisc-scheme
stklos	tinyscheme

Difficult decision. Frankly, I don't have an answer. That said, I found plt-racket to be a joy to work with. (help filter) will open your browser with the appropriate help page for filter. Amazing. Also, it implements a long list of SRFIs that will probably satisfy my simplistic needs. And the documentation is excellent. And then there is DrRacket, which is a nice REPL that comes right with Racket. I like it.

Kindle.app not starting on a case-sensitive file system

Thu, 13 Sep 2012 20:19:00 +0200

So Kindle.app was updated through the App Store and did not start any more. It just crashed and the crash reporter came up.

A quick look at Console.app turns up a link to the actual crash report. And the crash report starts with

    Dyld Error Message:
      Library not loaded: @executable_path/../Frameworks/libWEbCoreKRF.dylib
      Referenced from: /Applications/Kindle.app/Contents/Frameworks/libWebCoreViewer.dylib
      Reason: image not found

libWEbCoreKRF.dylib? With a capitalized E? That looks very much like a spelling error. So, point your terminal to /Applications/Kindle.app/Contents/Frameworks/, type

    sudo ln -s libWebCoreKRF.dylib libWEbCoreKRF.dylib

to create a symlink with the misspelled name. Done.

Kindle works again.

Spelling is hard, it seems.

On the Virtue of Contributing to and Using Open Source Software

Sat, 08 Sep 2012 08:44:00 +0200

I am a mostly self-taught programmer. Apart from a few programming side jobs at the university, I have been programming professionally for the last two and a half years.

About two years ago, we wanted to buy a Matlab license for our company. However, our investors declined for dubious reasons. So I started looking for alternatives. Inspired by a good friend (thank you, Marc), I looked into Python. Python has this brilliant environment for numpy scipy and matplotlib that, for my particular purposes, rivals Matlab.

However, just like Matlab, Python lacked a way of playing real time audio out of the box. At the university, we used Playrec for that purpose, which implements PortAudio bindings for Matlab. Luckily, there was a similar package available for Python, called PyAudio.

But, it had one crucial flaw: it only implemented blocking-mode audio I/O, so not much luck for my real time requirements. (Blocking mode means that whenever you want to play some audio, you have to wait until the piece of audio finishes playing before you can begin to play the next piece or do some computation. Clearly, you can not process audio while it is playing with such a scheme). After a lot of research, I figured that my best bet would be to dive in and add non blocking I/O to PyAudio.

At the time, I knew nothing of the Python C API (or Python, really), so those first attempts were riddled with subtle bugs and memory leaks. Nevertheless, it got the job done. I could play back audio in real time using Python!

This was a revelation to me: I could not only use a piece of software, I could also change it and mold it to my special needs!

Thus, I put up my changes on github and emailed the PyAudio maintainer. While the maintainer at the time did not have the time to incorporate my changes into the official PyAudio distribution, a few people discovered my version of it on github and contributed to it!

This, again, was a revelation to me: Not only could I change stuff other people did, wonderful people from around the world could collaborate and help each other. It always gave me a warm feeling of appreciation when some stranger on the internet chose to work with me!

Then, finally, I decided to take the plunge to Python 3. The main obstacle for this was that PyAudio did not support Python 3. So, again, I dove in and figured out how to make it compatible with Python 3. Again, I put the changes up on github and emailed the official PyAudio maintainer. This time, he took immediate notice and we started working on an official release of PyAudio including Python 3 support and non-blocking I/O.

And during those weeks, I had my third, and biggest, revelation about Open Source Software: The PyAudio maintainer is a brilliant mind, and I was humbled to find that I could learn a lot from this man.

By working on Open Source Software, you can work with really smart people, and learn from them. And the fruits of that labor can serve as instructions to yet more people to learn from.

Fixing Errors in Epydoc

Tue, 28 Aug 2012 12:14:00 +0200

I ran into this error twice now and wasted an hour both times, so it is time to put this on my universal scratchpad, i.e. this blog.

If you ever get this error when using epydoc:

    UNEXPECTED ERROR:
    'Text' object has no attribute 'data'

You are probably running a version of Python that is greater than the latest one that is supported by epydoc. This is because epydoc has not been updated since 2008 and Python 2.5.

Luckily, some fine folks on the internet have figured out how to fix these things.

Short answer: Find your site-packages directory:

    from distutils.sysconfig import get_python_lib
	print(get_python_lib())

Go there, navigate to the epydoc\/markup directory and change line 307 of the file restructuredtext.py from

	m = self._SUMMARY_RE.match(child.data)

	try:
		m = self._SUMMARY_RE.match(child.data)
	except AttributeError:
		m = None

This should fix that problem.

Using a Raspberry Pi as a Time Capsule for Mountain Lion

Sun, 29 Jul 2012 21:18:00 +0200

A while ago, I bought a Time Capsule to take care of my backups. I can't say it has been smooth sailing. Every now and then, the Time Capsule would claim that the backup did fail. Sometimes a reboot would help, sometimes not. Sometimes hdiutil would be able to salvage the backups, sometimes not. Sometimes, the backup disk image would simply be corrupted and the only option would be to delete it and start over.

This might be bad luck or it might be due to a defective Time Capsule or it might be due to my computer. I have no idea. But the thing is, if I have to hack on my backup system anyway, lets do it in style, at least. So here goes:

Ingredients: A Raspberry Pi, an external hard drive, some patience

Format an SD card as described in the wiki. I just installed the version of Debian that is provided on the official website. Now just boot up.

Next, I was stumped because I only have an Apple LED display and no convenient way of connecting the Raspberry Pi's HDMI output to the LED display's Mini Display Port. After some searching and a combination of three adapter cables, I finally got it connected and could see it boot. Really, I have no use whatsoever for the HDMI port on the Raspberry pi. So the first thing I did was to enable SSH, which luckily is available right there in the configuration utility that starts when you boot the thing for the first time.

After that, I disconnected the display and immediately was stumped because I now had no way of finding the Pi's IP address. Actually, I did not even have a network to connect it to. So I strung an ethernet cable from my laptop to the Pi and enabled Internet Sharing in order to (1) start the DHCP server and (2) give the Pi internet access. The IP address was then easily found using arp -a.

Setting up the hard drive

First off, I needed to format and mount my external hard drive to be usable as a Time Machine volume. ls /dev showed the hard drive as /dev/sda. Thus, I installed parted using sudo apt-get install parted and used it sudo parted. In parted, select /dev/sda sets it up to modify the external hard drive, rm 1 deleted its main partition, q to quit parted. Next, creating a new partition: sudo fdisk, then in there n with p and 1 to create a new primary partition, then w to apply the changes and exit. Lastly, I created the file system with sudo mkfs -t ext4 /dev/sda1 with the whole partition as its size. Now lastly, I created a mount point for it using mkdir ~/TimeMachine (don't use sudo!) and auto-mounted it by appending this to /etc/fstab

    /dev/sda1 /home/pi/TimeMachine ext4 rw,auto,user,exec,sync 0 0

Note: sync specifies that all file system changes have to be written to disk immediately, without caching. This might be bad for performance, but on the other hand, this behavior is probably a good idea for a backup system. I once read something somewhere that Apple is enforcing a similar behavior on their Time Capsules and that this is the reason why they won't allow any other network drive as Time Capsules.

Setting up the shared folder

First up, this requires netatalk, so I did sudo apt-get update and sudo apt-get install netatalk to install it. Next, netatalk has to be configured to actually share the drive on the network. This is accomplished by appending this line to /etc/netatalk/AppleVolumes.default:

    /home/pi/TimeMachine TimeMachine allow:pi cnidscheme:dbd options:upriv,usedot,tm

Also, the afp daemon should be configured to use the proper authentification schemes. Thus, add this to /etc/netatalk/afpd.conf:

    - -transall -uamlist uams_randnum.so,uams_dhx.so,uams_dhx2.so -nosavepassword -advertise_ssh

~~(maybe append mdns to the hosts in /etc/nsswitch.conf? Probably not necessary.)~~

~~I am also not quite sure whether I actually had to create a new file /etc/avahi/services/afpd.service and write into it:~~

    
    
    
		%h
		
			_afpovertcp._tcp
			548
		
		
			_device-info._tcp
			0
			model=Xserve

~~And maybe, you have to create an empty file that signifies the drive as Time Machine compatible using touch ~/TimeMachine/.com.apple.timemachine.supported.~~

Edit: Turns out, all these were not necessary. Thank you, reader Philipp, for trying them out!

I certainly did all that, but I am not quite sure which of these steps were strictly necessary. If you know, please let me, too.

Anyway, with all that done, restart both the netatalk and the Bonjour daemon using sudo /etc/init.d/netatalk restart and sudo /etc/init.d/avahi-daemon restart.

Setting up the Time Machine

~~Now, back to the Mac. In order to make Time Machine accept the new network share, run~~

~~defaults write com.apple.systempreferences TMShowUnsupportedNetworkVolumes 1~~

Edit: Turns out, this setting is not necessary. OSX just picks the Raspberry Pi as a usable Time Machine drive by default.

Finally, the TimeMachine folder on the Raspberry Pi was available as one of the backup drives. Halleluja!

Now transfer speeds for the initial backup are not exactly what I would call fast, but this might not be the Pi's fault. For one thing, the Pi is reporting to only run at half load. For another thing, the external hard drive and its USB connection is probably not very speedy. And lastly, I seem to remember that initial backups always were slow. But really, only time will tell how well this thing can do the job of a Time Capsule.

Further testing shows that transfer speeds are very comparable to the Time Capsule. Thus, I declare this a raging success!

This article heavily steals from these fine folks on the internet:

How Apple is Failing Me

Thu, 19 Jul 2012 19:49:00 +0200

I bought my first Apple computer in 2007 after a long time of gaming on Windows and a few years of Linux. In the beginning, I was just amazed at the consistency, practicality and sanity of the whole experience. I think I had a bad case of tinkeritis and neophilia in my Linux days, so this was probably to be expected.

Later, I marveled at the level of polish in the third party software ecosystem around Macs. Then I was just glad to have a slick GUI with actual Unix underpinnings. For reference, this happened in the time span from OS X 10.5 Leopard until 10.7 Lion. However, in that same time frame, I also noticed a gradual increase in problems. The computer seems to crash slightly more often, programs seemed to become more buggy, behavior less logical. At first, I marked this up as mostly a matter of my own increasing knowledge of the system, but as time went on, I became more and more convinced that it was actually the system getting worse, not myself becoming more sensitive.

So here are some of those problems:

Lack of Uninstallers

Computer people like to try new software. Some new software is crap. You should delete crap from your computer. Amazingly however, the Mac does not have any way of uninstalling some software.

At the moment, there are three ways of getting software on your Mac.

Install from the App Store. Installation and un-installation work through the App Store.
Install by dragging n *.app bundle onto your computer. Uninstall by deleting the bundle. Works fine unless the app installed some stuff outside the bundle.
Install by executing a *.pkg file. No way to uninstall whatsoever unless the developer provides a separate uninstall mechanism.

Really, this is beyond ridiculous. There is no way of uninstalling software that has been installed from a *.pkg. How the heck am I supposed to use my computer without uninstalling software? Seriously, Apple? Is this a joke?

(So as a rule, I will only install *.pkg if there is absolutely no other way. Most of the time, I use or abuse homebrew to mitigate this problem.)

Failing Time Machine

So the Mac comes with this awesome backup solution called Time Machine. Just plug in your external hard drive, and it will magically keep all your data save, at all times. Better yet, buy a Time Capsule and this will happen over the wireless network so you don't even need to plug in that hard drive any more. Magic!

Except, every few weeks the Time Machine icon in the menu bar will have a small exclamation point, which indicates that something went wrong. Usually, it means that Time Machine somehow lost track of what it was doing and silently corrupted all backups beyond repair. This happens across several computers and Time Machines. The only solution is to delete all backups and start anew.

So, backups that are not backing up. How amazing is that?

Living in the Cloud

OS X is this astonishingly modern operating system, where all your email, contacts, calendars and settings are synced in the cloud, accessible from anywhere and any device or service you use. In theory.

Synchronization is cool. I used to live within the Googleverse, and all my contacts, calendars and email accounts would sync through Google to all my Apple devices. Life was good! Until things started to go bad. At some point, Mail.app lost an email or two. Not a big deal. Then iCal would not connect any more, and would need to be restarted every other day. And finally, Address Book deleted all my contacts from Google. Actually, it did not delete the whole contacts, just all the information inside them, and leave empty husks with only a name and maybe a partial email address.

I wish I was making this up. I tested this quite thoroughly. Whenever Contacts was set up to sync with Google, after a few days or weeks it would delete all my contacts. Of course, this being cloud backed, it would delete them everywhere on all my devices, without any backup or anything. Thankfully, Address Book on iOS seems to not have this problem.

Things can go wrong, I understand that. But deleting all my contacts? Not cool. Not. Cool.

iPhoto

My grandfather has this G4 PowerBook. He stores all his photos on it. So at one point, we just could not stand it any more to support his software problems and set him up with a newer MacBook (2007) just so we could at least roughly recreate his problems on our machines.

But to do that, we had to convert his old library from iPhoto 1876 to iPhoto 2008. iPhoto itself claims to be able to do that, and lo and behold, it magically imported the old library, started an update program, then another, then another, then another, then another, and updated the old library through the ages up to the most recent version. And everything seemed to work fine, all the albums were there, all the thumbnails looked right, just perfect. Except if you opened any image, it would only show black. I mean, I can understand that an import of a library that old does not work. But at least tell me so. Don't silently corrupt everything!

And then there was my own library, started in late 2007 and handed through different revisions of iPhoto to today. It countains about 25 Gb worth of photos. Funny enough, its size on disk is actually 65 Gb. What. The. Heck?

At this point, a giant thank you to the iPhoto Library Manager by Fat Cat Software, which saved both my library and my sanity! Those guys rock!

OS X Lion with Auto Save and Versions

The newest and brightest in the land of Apple is OS X 10.7 Lion (at the time of writing). This amazing piece of software has a life-changing new feature: It friggin breaks one of the oldest tradition in computer history: Saving files.

Now normally, I would applaud this. Just save everything automatically and give me a sensible undo mechanism instead. Great!

However, in a stroke of genius, they also got rid of the possibility of save as…, which is just an easy way of duplicating a document. Say, because you received it in an email and want to file it somewhere else on your hard drive. Or because the document is saved on a thumb drive and you want to copy it over to your hard drive. No sir, you have to navigate to the document in the finder and copy it by hand from there. Thank you for your patience.

But the worst offender is Preview. Every program on the Mac has this very handy proxy icon in the window title. Whenever a window is representing a file on your computer, it will show that very file in the title, so you can drag and drop it somewhere else. Except preview won't give you the actual document. Instead, it will give you an alias to that document. Which is worth nothing at all if you drag it into an email, your Dropbox or a thumb drive.

Or try to enable File sharing on your local network, which works sometimes, but not all the time. Or Internet Sharing, which might share your internet connection, or break your DNS settings. Or take Bluetooth, which for the life of me, I can not make work for tethering with my iPhone, even though the same works just fine with the iPad and other Macs. Or take networking in general, which every once in a while just completely breaks down on my company's network and just refuses to send or receive any data over the Ethernet cable until I reboot my computer.

I could go on. Let's just say that Lion increased my WTF-per-minute rate significantly.

Performance, Memory and File System

Lets talk performance. I can't really complain here. In my MacBook Pro I have four cores, eight Gb of memory and a fast, Apple-sanctioned SSD. Life is good. Except that I happen to use virtual machines a whole lot and regardless of whether I use Linux (Ubuntu, Fedora, OpenSuse) or Windows (7, 8) on there, they feel faster.

I mean, you don't feel the pain of starting applications much on OS X, because in general, you simply keep your programs running all the time. But even so, Firefox starts up on Windows in, like, no time at all, but takes noticeable time on OS X (same configuration and plugins). I have an SSD, so I mostly don't care, but still.

Or take boot times. Even in a virtual machine, both Windows 7 and 8 and Ubuntu 12.04 and Fedora 17 boot up in less than half a minute. In comparison, OS X Lion takes significantly longer, sometimes minutes. Again, you usually don't feel that pain too much because you don't reboot your Mac all that often (unless you run Bootcamp for gaming regularly). Still, OS X boot and shutdown times are downright shameful these days.

Lets run a few virtual machines. I have eight Gb of memory. Lets boot up a Windows VM and give it two Gb. Do anything you want, run Eclipse and Visual Studio and Firefox with loads of tabs and Steam. Windows will do just fine. Those two Gb are plenty for anything except gaming and photo/video editing. Do the same thing with Linux, same result. In fact, Linux will work just fine with even less memory. But OS X? Don't even think of running it with less than two Gb. Four Gb will give you a workable system if you don't run too many memory intensive applications. But start up XCode and Eclipse and you are done for. This is a huge waste.

And lastly, let's talk HFS+. Did you know that HFS+ has a global lock that prevents any program from accessing the hard drive while another program is doing so? That explains why OS X is so friggin slow on non-SSD drives. And did you ever notice that HFS+ is slowly corrupting your data, even if there are no power failures or anything? I fire up Disk Utility every so often and it will always find some file system corruptions. It rarely breaks stuff, but really, the file system is the very last part of my system I want to see failing randomly.

So, what now?

With all this love, why am I still using a Mac? Well, for one thing, the hardware is seriously great. I would love to see some laptop manufacturer build something equally sturdy, light and good looking as a MacBook Pro. I would love to see someone build a great touch pad into a non-Apple device. I would love to see Retina displays everywhere¹. But I don't.

And then there is software. Where is Tower for Windows or Linux? Or Sparrow or Tweetbot or Reeder or Pixelmator or iPhoto? I'm not saying there are no alternatives, but I certainly have not been able to find any that were really up to the same level of polish.

That said, I have recently been playing around with Windows 8, Ubuntu 12.04 and Fedora 17 and I must say, I am quite taken with them. All of them. As for the next computer I am going to buy, I am doubtful if it will be an Apple computer again.

Footnotes:

As a matter of fact, I would love to see well enough to see Retina at all, but that is beside the point

Blogging with Pelican (and not Octopress)

Wed, 18 Jul 2012 21:11:00 +0200

For a while now, I have been moving more and more services I use off Google. The reasons for that are manyfold, and few of them have anything to do with Google being evil or not–just to get that out of the way.

One of the last holdouts has been my neglected Blogspot blog. And one of the reasons for it being neglected is that it was hosted on Blogspot. Now don't get me wrong here, Blogspot is a terriffic blogging platform. You have this very nice nearly-WYSIWYG text editor right in your browser, you can upload images, you can publish instantly to your blog… Basically everything is taken care of for you conveniently right there in your browser. Google Style.

Its just that I don't like to work that way. I like plain text. I like typing stuff into a plain text editor. I like to be in control. And Blogspot might be convenient, but it did not make me feel like I was in control. In fact, I lost at least one article to Blogspot for unknown reasons.

Enter static site generators. The idea is that instead of writing rich text into some website, you create you content however you want on your own computer, then use a static generator which converts it into a set of static HTML pages and upload those to your website. Now all of the creation process is happening on your computer. You are in control. Probably the most popular program to do that is Jekyll.

The second part of the equation is some kind of publishing platform. With these static site generators, really any web server does that trick. Just push your generated HTML files to the server and be done with it. Even cooler is Github. Using Github pages, you can use your existing Github account and infrastructure to publish your blog just by pushing the HTML to Github. This is seriously cool!

So, I set out and tried Octopress, which combines these two things into a nice blogging platform. Honestly though, I don't know much Ruby and all that rake workflow did not make me much more comfortable than pushing stuff into Blogspot.

Hence, I looked for alternatives. What I ended up with is Pelican, a very simple static site generator written in Python. Finally, this is a codebase that is easy enough for me to understand and modify. If there is any trouble with my blog, I will be (and have been) able to just look at the source code and figure out what is going wrong. I like this!

To publish a new blog post, I will start by writing the post in Markdown (a format I understand), process it using the very simple command line pelican -s my_config_file.py posts/, and push the result to GitHub. Easy as pie. And I feel like I am in control again!

Actually, if this is just a bit too technical for you, check out Calepin instead. It uses the very same Pelican engine, but instead of fiddling with Git, you just put your markdown files into your Dropbox, and–poof–you magically have a Blog!

If you want to see my blog as a repo on Github, just go have a look (the master branch contains the HTML, the source branch contains the configuration and Markdown).

How to get solarized syntax highlighting on your blog

Sun, 15 Jul 2012 14:33:00 +0200

Yesterday, I spent a few hours creating a solarized like CSS file for use with my new blogging tool. You could just scrape the settings from this very website, but for convenience, here they are:

/* Text */
.codehilite .t   { color: #586e75 }

/* Whitespace */
.codehilite .w   { color: #073642 }

/* Error */
.codehilite .err { color: #cb4b16; }

/* Keyword */
.codehilite .k   { color: #859900 }
.codehilite .kc  { color: #2aa198 } /* Keyword.Constant */
.codehilite .kd  { color: #268bd2 } /* Keyword.Declaration */
.codehilite .kn  { color: #b58900 } /* Keyword.Namespace */
.codehilite .kp  { color: #859900 } /* Keyword.Pseudo */
.codehilite .kr  { color: #073642 } /* Keyword.Reserved */
.codehilite .kt  { color: #b58900 } /* Keyword.Type */

/* Name */
.codehilite .n   { color: #586e75 }
.codehilite .na  { color: #2aa198 } /* Name.Attribute */
.codehilite .nb  { color: #268bd2 } /* Name.Builtin */
.codehilite .nc  { color: #268bd2 } /* Name.Class */
.codehilite .ne  { color: #cb4b16 } /* Name.Error */
.codehilite .no  { color: #2aa198 } /* Name.Constant */
.codehilite .nd  { color: #2aa198 } /* Name.Decorator */
.codehilite .ni  { color: #2aa198; font-weight: bold } /* Name.Entity */
.codehilite .nf  { color: #268bd2 } /* Name.Function */
.codehilite .nn  { color: #586e75; } /* Name.Namespace */
.codehilite .nt  { color: #2aa198; font-weight: bold } /* Name.Tag */
.codehilite .nv  { color: #cb4b16 } /* Name.Variable */

/* Builtin */
.codehilite .b   { color: #859900 }
.codehilite .bp  { color: #586e75 } /* Name.Builtin.Pseudo */

/* Variable */
.codehilite .v   { color: #586e75 }
.codehilite .vc  { color: #586e75 } /* Name.Variable.Class */
.codehilite .vg  { color: #268bd2 } /* Name.Variable.Global */
.codehilite .vi  { color: #268bd2 } /* Name.Variable.Instance */

/* Literal */

/* Literal.Number */
.codehilite .m { color: #268bd2 } /* Literal.Number */
.codehilite .mf { color: #268bd2 } /* Literal.Number.Float */
.codehilite .mh { color: #268bd2 } /* Literal.Number.Hex */
.codehilite .mi { color: #268bd2 } /* Literal.Number.Integer */
.codehilite .mo { color: #268bd2 } /* Literal.Number.Oct */

/* Literal.String */
.codehilite .s { color: #2aa198 }
.codehilite .sb { color: #2aa198 } /* Literal.String.Backtick */
.codehilite .sc { color: #2aa198 } /* Literal.String.Char */
.codehilite .sd { color: #2aa198 } /* Literal.String.Doc */
.codehilite .s2 { color: #2aa198 } /* Literal.String.Double */
.codehilite .se { color: #cb4b16 } /* Literal.String.Escape */
.codehilite .sh { color: #2aa198 } /* Literal.String.Heredoc */
.codehilite .si { color: #cb4b16 } /* Literal.String.Interpol */
.codehilite .sx { color: #2aa198 } /* Literal.String.Other */
.codehilite .sr { color: #cb4b16 } /* Literal.String.Regex */
.codehilite .s1 { color: #2aa198 } /* Literal.String.Single */
.codehilite .ss { color: #cb4b16 } /* Literal.String.Symbol */

/* Literal.Integer */
.codehilite .il { color: #268bd2 } /* Literal.Number.Integer.Long */

/* Operator */
.codehilite .o  { color: #586e75 }
.codehilite .ow { color: #859900 } /* Operator.Word */

/* Punctuation */
.codehilite .p  { color: #586e75 }

/* Comment */
.codehilite .c { color: #93a1a1; font-style: italic }
.codehilite .cm { color: #93a1a1; } /* Comment.Multiline */
.codehilite .cp { color: #93a1a1 } /* Comment.Preproc */
.codehilite .c1 { color: #93a1a1; } /* Comment.Single */
.codehilite .cs { color: #93a1a1; } /* Comment.Special */

.codehilite .hll { background-color: #dc322f }

/* Generic */
.codehilite .g { color: #586e75 }
.codehilite .gd { color: #586e75 } /* Generic.Deleted */
.codehilite .ge { font-style: italic } /* Generic.Emph */
.codehilite .gr { color: #586e75 } /* Generic.Error */
.codehilite .gh { color: #586e75; font-weight: bold } /* Generic.Heading */
.codehilite .gi { color: #586e75 } /* Generic.Inserted */
.codehilite .go { color: #586e75 } /* Generic.Output */
.codehilite .gp { color: #586e75 } /* Generic.Prompt */
.codehilite .gs { font-weight: 586e75 } /* Generic.Strong */
.codehilite .gu { color: #586e75; font-weight: bold } /* Generic.Subheading */
.codehilite .gt { color: #586e75 } /* Generic.Traceback */

code, pre {
	background: #fdf6e3;
	-webkit-box-shadow: inset 0 0 2px #000000;
	-moz-box-shadow: inset 0 0 2px #000000;
	box-shadow: inset 0 0 2px #000000;
	color: #586e75;
	margin-left: 0px;
	font-family: 'Droid Sans Mono', monospace;
	padding: 2px;
	-webkit-border-radius: 4px;
	-moz-border-radius: 4px;
	border-radius: 4px;
	-moz-background-clip: padding;
	-webkit-background-clip: padding-box;
	background-clip: padding-box;
}

Keep in mind though that I have no formal knowledge of CSS whatsoever beyond what I could gather from these very files.

How to gem install rlua on OSX/homebrew

Wed, 22 Feb 2012 11:43:00 +0100

If you want to use rlua, you need to have Lua installed. However, OSX does not come with Lua preinstalled, so you install it using

#+begin src sh brew install lua #end_src

Oh, would it be nice if everyone agreed on how to install stuff like this. Case in point, homebrew installs liblua (quite reasonable) in /usr/local/lib/liblua.[5.1[.4]].dylib

rlua however expects it to be called liblua5.1.dylib (notice the missing .).

Similarly, the headers are installed plainly into /usr/local/include, whereas rlua expects them to be in a folder called lua5.1.

Hence, here is how you get rlua to install:

    ln -s /usr/local/Cellar/lua/5.1.4/lib/liblua.5.1.4.dylib /usr/local/lib/liblua5.1.dylib
    ln -s /usr/local/Cellar/lua/5.1.4/include/ /usr/local/include/lua5.1

Not exactly a beautiful solution, but it works.

Text Editors

Fri, 03 Feb 2012 18:22:00 +0100

As far as I can tell, there has been a resurgence of development in the landscape of text editors of late. Only a few years ago, the text editing scene one the Mac seemed to be dominated by

Crazy unix farts using Vim or Emacs
BBEdit users
Textmate users

During a very short period of time though, a raft of new text editors became available. Sublime Text in particular has been getting a lot of attention and rave reviews by many people. To me, this warrants another look at what these new (and old) text editors are offering.

Vim

Vim is very old software. It is a more or less direct descendant of ed from the early 1970s. It grew up in text based terminals without any graphical windows or mouses. Hence, all of its functionality is really meant to be used from the keyboard only, even though mousing is supported these days.

The terminal inheritance limits its graphical capabilities somewhat. There are no graphical drawers or animations or pixel-precise scrolling. Everything is displayed in terms of rectangular characters, hence scrolling can only ever scroll one line at a time and there are no graphical images anywhere. Since Vim predates graphical displays altogether, it does not adhere to its standards much. In fact, in its normal mode it won't even type out the characters you hit on your keyboard.

This is at really Vim's greatest strength and weakness: In its 'normal mode', all key presses are interpreted like keyboard shortcuts in other programs. And what shortcuts there are! Pretty much every key on the keyboard has some special function, most keys even serve multiple functions depending on what mode you are in at the moment. Thus it should come as no surprise that you can do anything with these shortcuts at astounding speed. Watching a seasoned Vimmer do his keyboard dance is something to behold. Moving the cursor is especially powerful. Usually, any place in the current file is reachable using just a few key strokes. But of course, you are not limited to just viewing one file at a time either: Vim supports arbitrary split views and can even be used for efficient diffing or merging. And when it comes to changing that text, Vim is no slouch, either. There are various registers for saving text or locations, there is an increadibly powerful macro system, amazing searching capabilities, command line integration, there are specialized functions for programming… Really, when it comes to pure text editing chops, nothing beats Vim.

Vim is also incredibly customizable. Of course, there is no graphical preferences windows to do the customization in. You customize Vim by editing simple text files. Vim even includes its very own scripting language, VimScript, to enable users to extend it. There is a huge wealth of plugins available. These range from file browsers to support for new languages, or even some limited integration with compilers or source control tools. However, VimScript is not exactly a very pleasant language to code in and Vim does not actually like to interface with external tools much. These two issues limit the scope of what is possible with Vim somewhat–deep compiler integration and graphical debugging are fiddly at best, so you will probably just keep a terminal open and do them there. That said, the plugin ecosystem for Vim is probably still leaps and bounds beyond what is possible with most other text editor out there, but it does not quite reach the same breadth or integration as some IDEs or Emacs do. All that power comes at a price. Learning Vim is hard. If you start out with Vim, it will probably take at least a week or so until you can approach your old productivity again. Mastering Vim will take years. Even after months and months of diligent learning, you will find some new tricks and features to increase your speed.

At the end of the day though, Vim is an amazingly powerful tool, and it is certainly worth it to at least learn the basics of it. Actually, most people I know who tried it have actually stuck with it. Myself, I have some two years of Vimming under my belt, too, and it has proven to be a very important tool for me.

Emacs 24

The thing you have to realize about Emacs is that… Emacs is powerful. People have called it the thermonuclear text editor, and for a reason. Emacs can edit text, of course, but that is really only the most mundane of its features. Really, Emacs is a little world of its own. You can read and write your mail from within Emacs, you can do spreadsheets, calendaring, it can host terminals, debuggers, compilers, there is Tetris, hell, it even includes its very own psychiatrist! Speaking off the record here, I have a suspicion that Emacs might achieve sentience pretty soon.

Similar to Vim, Emacs is old software. It was invented in the late 1970s at MIT and has been growing ever since. Also like Vim, its terminal inheritance shows its teeth sometimes: mouseweel scrolling can be awkward, especially if you like your inertial scrolling, the menu bar seems to not get much love and keyboard shortcuts are not conformant with what you might be used to from other text editors. On the other hand, Emacs has some very modern features like mixing of proportional fonts and fixed-width fonts or inline image display.

The upside of being old is that Emacs is very mature software. There is a ginormous selection of extensions available for Emacs, most of it actually distributed right with Emacs itself and a lot of it is superbly documented. Besides that, there are several integrated repositories for additional tools that can be downloaded and installed from the internet.

The real power of Emacs is in that it is really not so much a text editor but a virtual machine for a programming language called eLisp. Really anything that can possibly be written in Lisp and remotely involves text editing is possible and probably already available in Emacs. As such, Emacs probably has the most diverse feature set of any text editor out there. Most relevant to programmers will be debugger integration, automatic syntax and spell checking, powerful and context-aware autocompletion, refactoring capabilities and much much more. Emacs is the only program in this list which can actually be used as a full fledged IDE on its own.

But Emacs is not limited to programming at all. As mentioned before, it includes an Email client, a great calendar with an agenda, several IM clients, RSS readers, an astoundingly powerful outliner and spreadsheet editor, it is frequently used for blogging, writing screen plays, books or really anything you can think of.

The only real downside to this is that Emacs is, well, old. There is great power there, but is is only accessible to those willing to learn the myriad key combinations to invoke it. Rellay, mastering Emacs is a task for years, not weeks. But of course, you are not forced to wield all the power Emacs has to offer at once. Getting up to speed with basic text editing in Emacs will only take a few minutes, and the built-in help system and tutorial will guide you further whenever you feel the need to explore.

I have been using Emacs for several months now and I am really enjoying it. It has a few shortcomings, but it is constantly being improved and getting more modern every day. I can see myself giving in to it and just live in Emacs all day long, but for now, I'm happy with it just being my primary text editor. However, I can't quite get beyond the fact that its pure text editing chops are nowhere near Vim. Then again, Emacs does support Vim key bindings, so this might turn out not to bother me in the long run.

At any rate, I would recommend anyone to give Emacs a shot at least for a short while. The power of Emacs can be an exhilarating experience, really.

Textmate (2)

For the longest time, all GUI text editors could generally be classified as either Vim-based, Emacs-based or shortcut-based. The prevalent crop of shortcut-based GUI text editors mapped all its advanced functions to certain modifier-letter combinations and/or menu bar items.

When Textmate was introduced, it introduced a new concept: snippets. Snippets are short pieces of text which, upon activating a certain trigger, would expand to arbitraryly complex constructs.

Thus, to define a class in some programming language, you would type class, then hit TAB, and it would expand to a complete class declaration with constructor, destructor and documentation. Further yet, the class name would be highlighted immediately, so you could start editing it to your liking. These edits would even automatically percolate to all the relevant places in the class declaration and thus automatically change all scope declarations and the like. Or, you could drag some image file into some LaTeX code and it would expand (dragging being the trigger here) to a whole \begin{figure} complete with \caption, \label, the correct path to the image and everything else you need.

This has proven to be such a popular feature that since Textmate's inception, implementations of snippets have been developed for pretty much every IDE or text editor out there.

Textmate also featured a very flexible and easily extensible regex based syntax parser, beautiful color schemes, a very elegant project management system and a vibrant community extending it in many directions. This was even more fascinating in that most of this functionality was implemented using a convenient shell scripting engine that could utilize any programming language your shell supported.

Sadly though, its main developer got stuck somewhere along the way and development all but dried up for five years. This primarily meant that some issues just would not be fixed and thus, got all the more jarring. Most prominently, Textmate lacked split views, regex incremental search and would only do single character undo/redos. Many people left Textmate because of this lack of progress.

While I was in college, I used Textmate extensively and it proved to be a veritable tool for many editing tasks. It is somewhat limited in its integration with programming tools, though, so don't expect any complex compiler or debugger integration. Ultimately, I left it behind for lack of cross platform compatibility and lack of development. There used to be a Windows program called E Text Editor, which wanted to become a fully compatible Textmate alternative for Windows and Linux, but development never even reached production quality.

Recently however, an early alpha version of Textmate 2 has been released that could reinvigorate the community and fix long standing issues. Whether that will actually happen will remain to be seen.

Vico

Vico is a very new application that has spung to live only in 2011 and is not finished yet. It aims to be a modern Vim, combining the virtue of the powerful mode-based editing system of Vim with a modern Cocoa interface. It even merges Vim's editing capabilities with Textmate-derived snippets and syntax highlighting.

Really, it tries to be an organic symbiosis of Vim and Textmate. And for most purposes, it very much succeeds in this. Syntax highlighting is very solid, there is a code browser, a nice file browser and full support for Textmate snippets. Vico even includes a powerful scripting environment that enables you to extend it in a language called Nu, which has the interesting aspect of being able to call into every object or method in the Cocoa libraries, thus opening the doors to a boundless world of wonders. Its Vim integration is well on its way, too. The most notable omission at this point are macros. The developer is working on it though and has promised to implement them in the near future. Some other areas are lacking, too, but if development continues I see no reason why it should not become a very nice text editor.

As it stands though, development of Vico is going slow and its community is not very large yet. Vico is a very nice tool, but at the end of the day, I miss the raw power of an actual Vim just as I miss the vibrant community around Textmate. If you are not spoiled by Vim yet or find Vim just a bit too ugly for your taste, you could give Vico a try though.

Personally, I like it quite a bit and I am hoping very much that it will not be forgotten as a failed attempt to modernize Vim. That said, what with Sublime Text 2 including a limited support for Vim key bindings, I can't really see Vico taking off.

Sublime Text 2

Sublime Text is a fairly recent development. It is a one-man project that has gained a lot of enthusiastic following in the last few months. In many ways, it feels like the next step in text editor development. Much like several other text editors out there, it has adopted Textmate snippets, color themes and syntax definitions as its core feature set. On top of that though, it has built a very powerful and flexible extension system that really sets it apart.

Do you remember CMD-T from Textmate? To open some file, you would hit CMD-T and start typing a file name. The name would be fuzzy matched to select from all available files. Thus, typing bcc would select BeaconController.cpp. This form of selection is increadibly intuitive and fast. It is also the basis for the extension system of Sublime Text. If there is no keyboard shortcut for a command, hit CMD-Shift-P and start typing to invoke the command.

In very much the same way as M-x in Emacs (though with fuzzy matching), this can invoke arbitrarily complex commands such as install bundle or using some refactoring library. Additionally, the same mechanism can be used to jump to method names or opening files. Really, these features are very efficient implementations of a code browser and file browser.

The second big thing about Sublime Text is just the ridiculous amount of polish it received. For example, if you have two files with the same name, it will prefix the tab titles with the folder they reside in. Simple, but so useful! If you jump around in a file, there is always a subtle scrolling animation. Even simple text selections have slightly rounded borders and just look amazingly beautiful.

Also of note is that Sublime Text supports multiple cursors. Want to change then name of all occurrences of a variable? Just select them all (!) and change them all at once. This is another amazingly useful feature. Its plugin system is based in Python, which is a refreshingly non-awkward choice for a text editor and spawned an astonishing amount of very interesting plugins already. Indeed the plugin system is flexible enough to support things like linters, source control integration and even something akin to Emacs's org-mode. And it also supports Vim key bindings. Not very complete, but easily enough to be useful. Oh, and it is available cross-platform on Mac, Windows and Linux, too.

I think there are three big families of text editors: Emacs, Vim, and shortcut-based text editors. Since I discovered Textmate however, I started believing that it represents a new branch in the big tree of text editors. Sublime Text seems to be the next step in the evolution of the Textmate branch.

Really, Sublime Text is an amazing achievement. Maybe not quite as hackable as Emacs and not quite as flexible as Vim, but easily beating both in terms of elegance and modernity. If you don't want to learn Emacs or Vim, Sublime Text is what you should use. In fact, it is the first text editor ever that has tempted me to leave Emacs and Vim behind. Nuff said.

What else is out there

Of course, that little list up there is by no means complete. Neither does it list all the amazing features these text editors have to offer, nor does it represent an exhaustive list of them. To the best of my knowledge, this is a short list of other text editors for the Mac platform. Note however, that I have never used any of them extensively and can only tell you stuff from heresay.

BBEdit

BBEdit is the big daddy of Mac text editors. It is currently available in version 10 and has a huge follwing predominantly amongst web developers. As far as I can tell, it includes amazing features for editing HTML. Maybe amongst the best out there. The remainder of its feature set seems rather standard crop though. There is some support for compilation, source code control, snippets, plugins… Though nothing on the level of Vim or Emacs really. Its most important disadvantage is probably its lack of cross-platform availability and extensibility.

TextWrangler

TextWrangler is the free smaller brother of BBEdit. Its feature set is somewhat pared down in comparison with BBEdit. In particular, it is missing BBEdit's famous HTML magic and some advanced external tool integration. Even for free, there are probably more capable candidates available, though maybe not at the same level of platform integration. That said, TextWrangler is not a bad choice and probably just fine for some casual text editing.

JEdit

I really don't know much about JEdit other than that it is written in Java, it has a sizeable following and it is available cross-platform. It seems like it could be about as useful as any shortcut based text editor can ever be, which is no small achievement. Also, it features a rich plugin system, of which I only heard good things. For all I know, this could be a very worthy alternative if you are on a budget (no Sublime Text) and don't want to learn Emacs or Vim.

Chocolat

Chocolat is yet another text editor that came into being in the post-Textmate void. It offers a good range of standard features, though notably missing advanced plugins apparently. Apart from that, it seems to be a solid shortcut based Mac text editor that is relatively cheap and actively developed.

Kod

Kod started out as an open source alternative to Textmate. This is quite uncommon for a post-Textmate text editor and worth supporting. After a good start however, the developer found a new job and development has pretty much stalled.

SubEthaEdit

The great thing about SubEthaEdit is collaborative editing. if you want to edit text collaboratively with several people, SubEthaEdit performs the task seamlessly and elegantly. Beyond that, it is a capable shortcut-based text editor. If you don't need the collaboration feature, you should probably look elsewhere though.

Smultron

Another venerable veteran on the Mac, Smultron used to be a free open source text editor that had a sizeable following. However, the developer ceased development at some point and later restarted the effort as a paid app in the Mac App Store. Smultron lost most of its following in that transition.

SlickEdit

SlickEdit is easily the most expensive text editor in this list. A single user license for one platform and one developer costs a scant 300 bucks. That is a lot of money for a text editor and is usually only shelled out only for business critical platform exclusive IDEs. It seems as if SlickEdit tries to be exactly that for general purpose text editing. It's feature list reads very well and checks all mayor boxes. It is available on pretty much any platform out there and is probably only rivalled by Emacs or Vim in that regard. It also supports emulation for Emacs or Vim key bindings. That said, 300-600 $ per developer is a pretty hefty price tag. Personally, I doubt that SlickEdit can live up to that price if you compare it to some of the other examples in this list. It is undoubtedly a well-maintained and powerful text editor though.

Komodo Edit

Komodo Edit is the free open source offspring of ActiveState's Komodo IDE. It's feature set is very complete and seems to be very worthy for many editing tasks. Being of IDE ancestry provides it with nice plugin support and very helpful deep language integration like sophisticated autocompletion and syntax checking. There are also quite good Vim key bindings and it is available on the three major platforms. However, its language support is limited to Perl, Python, Tcl, PHP, Ruby and Javascript. Probably not a bad choice if you can live with the language selection and are on a budget.

TextEdit

The built-in text editor in OSX. No syntax highlighting, project management or any programming support whatsoever make this a rather poor choice. There are plenty of free alternatives out there. That said, TextEdit does support rich text editing and might be of value for the odd letter to your grandma.

Text editors for web development

Strangely, all the major Mac text editors that cater specifically for web development are not cross platform. If web development is all you are ever doing though, these text editors might be well worth their money.

Coda

Combine the SubEthaEdit text editing engine including its collaborative tools with the great FTP program Transmit and you have CODA, the program for "one-window web development". It even includes a reference book to HTML in the package. For what it is, probably of great value and nice polish. As a general purpose text editor, there are better alternatives.

Espresso

Another text editor gearing specifically for web development. Thus, you get powerful HTML and CSS editing features and good support for typical web development languages such as PHP, Ruby or Markdoen, but no support for other languages. There is an extension system though that could improve language support. This might be a slightly less expensive alternative to Coda.

Taco HTML Editor

Another text editor geared exclusively towards web development. This time around though, there is no support for languages other than HTML, CSS and PHP, which makes this program a rather poor choice.

skEdit

skEdit supports a nice array of web development languages and offers a good range of features for a web development text editor. Also, it is pretty inexpensive in comparison to its brethren here. Probably a nice choice for web development if you are on a budget.

How to make Tagbar work with Objective-C

Sun, 04 Dec 2011 20:21:00 +0100

For the longest time, ctags did not support Objective-C. Without tags, many of the cool programming-related features of Vim simply didn't work. (There are ways to coax other programs to output tags, but this is usually painful)

Recently however, ctags finally gained support for Objective-C! Huzzah! As of today, Objective-C support is not in the stable branch yet, but thanks to the magic of homebrew, an Obj-C ready ctags is only a brew install ctags --HEAD away!

With that, tag navigation in Vim works for Objective-C. Awesome! However, Tagbar does not work yet. There used to be a workaround that apparently worked for some people, but it did not work for me. Things like this drive me nuts. I can see that tags are being generated, but Tagbar won't show them. What is going wrong?

Some investigation proved: There is simply no template in Tagbar for Objective-C! Luckily, that can be fixed quite easily: Simply put this code in your .vimrc:

    " add a definition for Objective-C to tagbar
    let g:tagbar_type_objc = {
        \ 'ctagstype' : 'ObjectiveC',
        \ 'kinds'     : [
            \ 'i:interface',
            \ 'I:implementation',
            \ 'p:Protocol',
            \ 'm:Object_method',
            \ 'c:Class_method',
            \ 'v:Global_variable',
            \ 'F:Object field',
            \ 'f:function',
            \ 'p:property',
            \ 't:type_alias',
            \ 's:type_structure',
            \ 'e:enumeration',
            \ 'M:preprocessor_macro',
        \ ],
        \ 'sro'        : ' ',
        \ 'kind2scope' : {
            \ 'i' : 'interface',
            \ 'I' : 'implementation',
            \ 'p' : 'Protocol',
            \ 's' : 'type_structure',
            \ 'e' : 'enumeration'
        \ },
        \ 'scope2kind' : {
            \ 'interface'      : 'i',
            \ 'implementation' : 'I',
            \ 'Protocol'       : 'p',
            \ 'type_structure' : 's',
            \ 'enumeration'    : 'e'
        \ }
    \ }

With that, Tagbar should work. Admittedly, it won't work perfectly yet. Class definitions will show up in addition to their contents and the contents of all categories will go into the same list. Also, you won't get much more than a flat list of functions with no way to distinguish between class methods and object methods. This can be confusing at times, but by and large, it does the job!

Happy Vimming!

Compiling Scipy and Matplotlib again

Thu, 13 Oct 2011 15:27:00 +0200

Well, it's compile time again. Once again, I need to install scipy and matplotlib using homebrew and pip on Lion. It seems things have improved since I tried to compile last time! Well, it still does not work out of the box, but at least now it can be done without compiling by hand: (remember to brew install pkg-config gfortran first)

pip install -e git+https://github.com/scipy/scipy#egg=scipy-dev
pip install -e git+https://github.com/matplotlib/matplotlib#egg=matplotlib

I must say, this is still a mess. But at least, it is getting less bad.

Adobe AIR Application Crashes

Sat, 03 Sep 2011 13:08:00 +0200

Adobe, oh Adobe. Why?

Your updater pops open. Adobe AIR 2.5, if I remember correctly. It runs, and after that, all AIR applications stop working. They boot, then crash and barf up a crash report.

Turns out, this is a problem with case sensitive file systems. If your file system is not case sensitive, you are experiencing some other problem and this fix is not for you.

See, Adobe has this library called WebKit.dylib. However, their code actually looks for Webkit.dylib (note the capitalisation of the K). Well, spelling is hard, I presume.

At least this is an error that can easily be fixed. Fire up your terminal, type

cd /Library/Frameworks/Adobe\ AIR.framework/Version/Current/Resources/
sudo ln -s WebKit.dylib Webkit.dylib

Then enter your password and your AIR applications should work again.

Dear Adobe, is it too much to ask to pretty please make your software work on a case sensitive file system? Spelling is primary school stuff. It is really not that hard!

Compiling Scipy and Matplotlib using pip on Lion

Mon, 01 Aug 2011 12:40:00 +0200

So I upgraded to Lion. Predictably, some things went wrong. This time, the main thing that bit me was that for some reason, pip stopped working. After a bit of messing around with brew, pip and easy_install, I found out it was almost entirely my own fault. I messed up my PATH.

In the meantime, I had uninstalled all of brew's Python, so I had to reinstall. For me, that entails Python, Numpy, Scipy and Matplotlib. Only this time, Scipy would not build. Some obscure error in some veclib_cabi_c.c would report errors. A quick round of googling reveals:

In order to get Scipy to compile, you need to insert #include in

    ./scipy/lib/blas/fblaswrap_veclib_c.c.src
    ./scipy/linalg/src/fblaswrap_veclib_c.c
    ./scipy/sparse/linalg/eigen/arpack/ARPACK/FWRAPPERS/veclib_cabi_c.c

That done, Scipy compiles perfectly fine.

But, that is not enough yet. As this blogpost outlines, Matplotlib is not currently compatible with libpng 1.5, which ships with Lion. Fortunately, this is already fixed in the most recent source on the Matplotlib repo, so you just have to checkout that:

    pip install -e git+https://github.com/matplotlib/matplotlib.git#egg=matplotlib

By doing that, Matplotlib should install just fine.

Seriously though, these PyPi repos are in a very sorry state. Every time I install one of these packages, I have to jump through hoops and spend hours debugging packages that really should work right out of the box. After all, brew, rvm and gem can do it just fine. Why is pip such a horrible mess?

Installing Pygame using Homebrew

Fri, 04 Mar 2011 14:16:00 +0100

So I want to do audio development on the Mac without using Matlab. An alternative to Matlab is Python, or rather, Numpy, Scipy and Matplotlib. They are awesome for working with audio data. What they don't do however is playing back audio. There are several packages out there that would afford audio playback. If you are serious about this though, you not only want audio playback, you want asynchronous audio playback. That is, you want to send some audio data to the sound card and continue with your program without waiting for the audio to finish playing. This allows continuous audio playback of computer-generated sound.

Pygame is one package that allows this. (I will submit a patch to Pyaudio soon that will enable it there, too). There are pre-built binaries on the Pygame website that you can install easily. But then there would be no easy way to uninstall them, so what I would rather want is to install Pygame using package managers that allow easy updating and uninstallation. My tool of choice on the Mac is of course Homebrew.

Note that although I am mostly interested in audio playback, this post will detail the installation of all modules of Pygame, not just pygame.mixer.

Homebrew won't install Pygame, but it will install all the prerequisites for Pygame. So, let's do that.

brew install sdl, sdl_mixer, sdl_ttf, libpng, jpeg, sdl_image, portmidi

This will install most packages for you. Note that libpng is also available as a system library, so it is installed keg_only, that is, without linking it in your path. We will need to compile against it though, so the next step is

brew link libpng

Now there is still one package missing, smpeg. Sadly, smpeg does not install its headers, so you can't compile against it. To fix that, type

brew edit smpeg

and add the following line just above the two end at the end of the file

include.install Dir["*.h"]

Then save the file. (I submitted a bug to have this fixed, so you might not need to do this when you read this). Now you can install smpeg with the usual

brew install smpeg

and you will get the headers, too. Isn't Homebrew great?

Now that all the prerequisites are met, lets look at Pygame itself. This is rather more difficult, as it will not build properly against Homebrew libraries on its own. First, download the source package of Pygame from the [official website](http://www.pygame.org/download.shtml). Unpack it to some directory.

Now open a terminal and navigate to that directory. Me, I like [iTerm](http://iterm.sourceforge.net/), but Terminal.app will do just fine, too. In there, run python config.py to create an initial setup file.

At this point, the setup file is mostly useless since config.py failed to find any homebrew-installed library. It is also strangely garbled, so there is some manual labor to do. Open the file Setup (no extension) in your favourite text editor. After the first comment block, you will see a line that looks like this

SDL = -I/NEED_INC_PATH_FIX -L/NEED_LIB_PATH_FIX -lSDL

Obviously, this is lacking the paths to the SDL library. If you installed Homebrew to its default directory, this will be in /usr/local…. Hence, change this line to

SDL = -I/usr/local/include/SDL -L/usr/local/lib -lSDL

The next lines are strangely garbled. They say, for example

FONT = -lS -lD -lL -l_ -lt -lt -lf

Where they actually should say

FONT = -lSDL_ttf

Instead of having one -l and then the library name SDL_ttf, they put -l in front of every single letter of the name. This is strange, and certainly wrong. So, correct it for FONT, IMAGE, MIXER and SMPEG.

Note that I did not tell you to do this for PORTTIME, too. Actually, PORTTIME is already correctly linked in PORTMIDI, so you don't need that at all any more. Just delete or comment the PORTTIME line.

Now that all the dependencies are corrected, lets enable the features. A few lines further down, there will be a block of lines, where most lines begin with a # except for the ones beginning with _numericsurfarray… and _camera…, These are the different features of Pygame: The ones with the # are disabled, the other two are enabled.

With all the stuff we installed earlier, you can now enable all features (remove the # in front of imageext…, font…, mixer…, mixer_music…, _minericsndarray…, movie…, scrap… and pypm…).

Remember we disabled PORTTIME a while ago? Right, so we have to remove that dependency: In the line starting with pypm…, delete the part that says $(PORTTIME). Great. That was easy, right? Now save that file and go back to the Terminal.

We are now going to compile and install Pygame. The nice thing is, even though we are installing it manually, it will go in the right directories and it will be registered with pip or easy_install, so you can just invoke them if you want to uninstall it later by typing pip uninstall pygame. This is something I love about Python!

Alright, now without further ado, install Pygame by typing

python setup.py install

Great! That's it! Everything should work now!

Installing Python/Numpy/Scipy/Matplotlib on OSX

Thu, 03 Feb 2011 16:23:00 +0100

For numerical analysis and signal processing prototyping, you would use Matlab. However, Matlab has some downsides that might make it unsuitable for your project. It might be too expensive. You might be a snobbish programmer that can't stand less-than-elegant programming languages. I certainly am.

So, you look for alternatives. You could take Octave, which is free, but that would not solve that ugly-code issue. You could take any scripting language you fancy, but Ruby, Perl and Python are too slow to do serious number crunching.

Then, you stumble upon that Python package called Numpy, which seems to be nearly as fast as Matlab when it comes to matrix processing and linear algebra. You then discover SciPy, which would add all that signal processing prowess of Matlab (do quick transformations, random numbers, statistics) to your toolbox. Last but not least, you need plotting. That would be Matplotlib then, which provides quick plotting facilities in Python.

And the best thing is, these three systems work really well together. They seem to be the perfect replacement for Matlab that could even be superiour to it in many regards.

Next up, you need to install all that stuff. If you are like me, you naturally want to do all that on a Mac. Also, you kind of dislike all these installer-thingies, which install stuff to unknown places and are nigh impossible to uninstall or update cleanly. Even though, you could of course just go to the individual websites, download Python, Numpy, SciPy and Matplotlib, run them installers, and be done. You would save yourself a lot of trouble that way.

But since you allegedly are like me, you instead fire up brew and try to install all that stuff using that. Again, you could use MacPorts or Fink instead, but you probably had some bad experiences with them and you generally love the hackishness of Homebrew, so this is your natural first try.

So you set about this, you believe in packet managers and trust them to take care of every obstacle that might be lying in your way. First of all, install the latest developer tools from developer.apple.com. You might need to register (for free) to get them. Also, you need to install Homebrew.

To cut this short, here is what you need to get that Python running:

    brew install python

This one should be obvious. At the time of writing, it will install Python 2.7.1. You could take Python 3, but matplotlib is not compatible to it, so you kind of have to stick with 2.7.1 instead.

You also need to put /usr/local/bin and /usr/local/sbin in the beginning of your path to make sure the new Python gets loaded instead of the pre-installed one. You do that by writing

    export PATH=/usr/local/bin:/usr/local/sbin:$PATH

in your \~/.bash_profile. (Create it if its not there–it is just a simple text file).

Now, if you type python --version, you should get Python 2.7.1 as a response.

Alright, next up, install the python package manager:

    brew install distribute
    brew install pip

This will come preconfigured for your newly installed Python. In an ideal world, this should be all. The world being as it is, the pip package of Matplotlib is severely broken and has one other unstated dependency:

    brew install pkg-config

Also, SciPy is using some FORTRAN sources, so you need a fortran compiler:

    brew install gfortran

Alright. That was enough. Now on to pip. With all these dependencies cleared, pip should be able to download Numpy and Scipy without trouble:

    pip install numpy
    pip install scipy

Matplotlib, on the other hand, is more difficult to install. You see, pip is looking at the Python package repository PyPi for each package. PyPi then provides a URL. Pip then scans that website for links to suitable package files. But, Sourceforge changed its links a while ago, so pip gets confused and will download an outdated version. Sourceforge says, its new links are way better and no way we will change them back; Pip says, well, if Sourceforge can't provide proper links, that's not our problem. Oh My. Silly children.

So we have to do this manually:

    pip install -f http://sourceforge.net/projects/matplotlib/files/matplotlib/matplotlib-1.0.1/matplotlib-1.0.1.tar.gz matplotlib

That URL comes straight from Sourceforge. Look for the latest version of Matplotlib, search for the download link to the source distribution (*.tar.gz), copy that link and strip any trailing '/download'.

UPDATE:

It seems the matplotlib package was updated in the meantime, so you can just run pip install matplotlib now.

This should now download and install matplotlib.

Thank you for reading.

The strangeness of widescreen displays in modern operating systems

Sat, 03 Jul 2010 12:08:00 +0200

So today, pretty much everyone has widescreen displays. That is, displays that are far more wide than high. This was not always so. In ye olden days, computers were mainly used for displaying text, hence displays tended to have a similar layout as books or magazines. Today, they are more like movies (which might be worrying in itself).

So todays displays are widescreen. To do all that modern stuff, like watch (widesreened) videos or multitask (display two windows side by side). However, this also means that vertical pixels are something of a scarcity. Especially on those small Laptop screens. In fact, the first Netbook screens were so tiny that many of Windows' own windows could not be used at all since the lower parts did not fit on the screen. Raise your hand if Word 2007 leaves barely ten lines of visible text between all its blue-tinted UI-splendour on your laptop screen.

This gets most straining when reading text. On the internet for example. There is practically no website at all that can be displayed in its entirity even on one of those full HD displays. Print-formatted documents are a similar matter. Actually, I find myself craving for pixels regularly. I even disable the bookmark bar in my browser to free those extra two lines of text. And I memorize keyboard shortcuts so I can hide toolbars. And I use Google Chrome instead of Firefox/Safari/Internet Explorer, not least of all since it has the smallest title bar and no bottom bar.

Why, then, do modern operating systems still waste so much vertical space with that Dock/Taskbar? This is something I really don't get. Vertical space is such a scarcity, yet virtually every operating system choses to waste at least three lines of text with something that could easily go on the side of the display. Well, at least on Windows 7 and OSX that is something you can easily change.

So if you are like me and appreciate every added line of text, do yourself a favour and put your Dock/Taskbar on the side. Really, this should be the default.

Gedanken zu User Interfaces

Sat, 25 Apr 2009 14:15:00 +0200

Jeder kennt dieses Fenster:

Dieses Fenster ist eine ziemlich schlechte Idee, denn jeder geübte Benutzer hat irgendwann gelernt, dass die Aktion "Dateien Löschen" aus drei Gesten besteht: (1) Dateien auswählen, (2) Die Löschaktion einleiten, (3) Den Löschdialog bestätigen. Leider hat man schon vor mehr als zwanzig Jahren nachgewiesen, dass solche zusammengehörigen Gesten von den Benutzern als einzelne "Datei-Löschen" Geste abstrahiert werden. Man kennt das vom Tippen: Man tippt Worte nicht, indem man einzelne Buchstaben aneinander hängt, sondern man tippt Worte fast immer am Stück; Hat man erst einmal angefangen, ein falsches Wort zu tippen, kann man damit nicht aufhören, bis das gesamte Wort getippt ist. Auf ähnliche Weise ist es dem Benutzer auch nicht möglich, auf den Löschdialog sinnvoll zu reagieren, da das Bestätigen der Sicherheitsfrage vollkommen automatisiert ist und daher nicht einmal verhindert werden könnte, wenn man wollte. Die Alternative ist ein alter Bekannter: Undo. Statt den Benutzer im Vorfeld zu fragen, ob er die Datei wirklich löschen möchte (was er durch Einleiten der Löschaktion bereits bejaht hat) gibt man ihm die Möglichkeit, die Aktion im Nachhinein wieder rückgängig zu machen. Das ist wesentlich effektiver und kommt ohne nerviger Dialogbox aus. Übrigens funktioniert Dateiaktion-Undo bereits heute in allen Betriebssystem außer Linux, jedoch ohne Menüicon und zumeist nur für die letzte Aktion. Wäre es nicht schön, wenn dies noch weiter ausgebaut würde?

Noch ein gefährlicher Dialog:

Das ist ebenfalls ein alter Bekannter, der immer dann erscheint, wenn man ein Programm schließen will, welches noch ungesichte Änderungen enthält: Abgesehen von dem offensichtlichen Problem, dass "Abbrechen" keinen wirklichen Sinn ergibt (Was abbrechen? Das Programm?) erfordert diese Frage jedes Mal das komplette Lesen der Meldung, bis man entschlüsseln kann, was "Ja" und "Nein" in diesem Kontext bedeuten. Um das noch einmal zu verdeutlichen, hier ein besonders schlimmes Beispiel:

Es leuchtet ein, dass hier ein eindeutigerer Dialog wesentlich sinnvoller wäre, bei dem sofort ersichtlich ist, was welcher Button tun wird: (Merke: Auf Buttons gehören immer Verben)

]]

Aber warum eigentlich überhaupt speichern? Warum muss ich mich persönlich darum kümmern, meine Arbeit zu speichern? Ich dachte, ich würde mit einer Datei arbeiten – aber wenn diese Datei nicht geändert wird, wenn ich nicht zuerst "Speichern" anklicke, habe ich wohl eigentlich doch nicht mit dieser Datei, sondern mit einer heimlichen Kopie gearbeitet. Wäre es nicht viel sinnvoller, immer automatisch zu speichern, und statt des "Speichern"-Buttons eine "auf Urzustand zurücksetzen"-Funktion bereitzustellen? Ich bin mir auf jeden Fall sicher, dass man den "Undo all changes"-Button wesentlich seltener bräuchte als den "Speichern"-Button, denn wenn der Benutzer nicht vorgehabt hätte, neuen Text in eine Datei zu schreiben, dann hätte er keinen neuen Text in die Datei geschrieben. Hat er dennoch "versehentlich" Text eingegeben, ist das ein klarer "Benutzerfehler", also der logische Einsatzzweck für Undo.

Das sind nur zwei Beispiele, wie man intelligente Undo-Mechanismen sinnvoll einsetzen könnte, um das Arbeiten am Computer angenehmer zu gestalten. Ich wünschte, mehr Leute würden sich mit diesen Dingen auseinander setzen…

Tagged File System

Sun, 19 Apr 2009 09:41:00 +0200

Ein großes Problem bei der Benutzung von Computern ist, dass unerfahrene Benutzer oftmals kein Verständnis für Ordnerstrukturen haben. Das ist im Grunde auch klar, denn die Ordner-Metapher legt nahe, dass sich Ordner auf dem Computer wie echte Aktenordner verhalten, also dass jeder Ordner mehrere Dateien, jedoch nicht andere Ordner enthalten kann. Ganz im Gegenteil dazu basiert aber eine normale Verzeichnisstruktur meist aus vielen, tief ineinander geschachtelten Ordnern. Von einem Usablitity-Standpunkt aus ist daher der Begriff "Ordner" wahrscheinlich schlecht gewählt. Vielleicht würde es schon reichen, den älteren Namen "Verzeichnis" wieder einzuführen. (Dann bräuchten wir nur noch ein passendes Piktogramm für "Verzeichnis"…)

Es wäre daher wünschenswert, eine einfacher zu verstehende Alternative zu Ordnerhierarchien zu haben. Die gibt es auch schon, in Form der bekannten Verzeichnisse "Meine Bilder", "Meine Dokumente", etc. Diese Ordner wollen den Benutzer mit einfachen Piktogrammen und klaren Namen dazu animieren, einen natürlichen Ort für seine Dateien zu wählen und so ein wenig Ordnung zu schaffen. Tatsächlich ist dieses Konzept einer Tag-Struktur schon relativ ähnlich, da auch hier nicht davon ausgegangen wird, dass sich der Benutzer selbst um eine tief geschachtelte Orderhierarchie kümmert, sondern nur wenige, einfach zu verstehende Markierungsmöglichkeiten ("Bilder", "Dokumente") geboten werden.

Ein echtes Tag-basiertes System könnte vollkommen ohne Verzeichnisse auskommen, wobei man dann eben beim Speichern einer Datei nicht mehr aus einer hierarchischen Liste von Ordnern den Speicherort auswählen würde, sondern von einer flachen Liste von Tags. Die Usability-Kosten davon wären vernachlässigbar.

Ähnlich würde das Finden von Dateien funktionieren: Statt eine hierarchische Liste von Ordnern nacheinander anzuklicken, würde man eine flache Liste von Tags nacheinander anklicken, jedoch mit dem Bonus, dass man die gesuchte Datei nicht erst bei Anklicken aller Tags, sondern mit großer Wahrscheinlichkeit schon nach ein oder zwei Tags gefunden hätte.

Ein Problem würde jedoch mit Projekten entstehen, die aus mehreren Dateien bestehen. Hier müsste man sicher stellen, dass sie im Dateisystem nur als einzelne große Projektdatei auftauchen und nicht jede einzelne Unterdatei gelistet wird. Das ließe sich zum Beispiel durch "Bundles" lösen, wie sie heute schon in OSX vorkommen (Ordner mit definierter Namesendung und Inhalt werden wie Dateien behandelt) oder einfach durch einen speziellen Tag, der die einzelnen Dateien vor der normalen Suche versteckt. Dieses Konzept wird übrigens heute schon vielfach verwendet, so ist etwa eine aktuelle Word-Datei nur eine ZIP-Datei, die eine definierte Verzeichnisstruktur mit allen Bildern, dem Text (als XML), einer Vorschau-Grafik etc. enthält.

Ich glaube, dass man mit solch einem Tag-basierten Dateisystem deutlich einfacher arbeiten könnte als mit den heute üblichen Verzeichnisstrukturen. Tja, jetzt fehlt nur noch eine innovative Firma, die sich um die Umsetzung kümmert…

Es geht um Tabs

Tue, 31 Mar 2009 20:09:00 +0200

Tabs sind kaputt. Tabs waren noch nie eine gute Idee. Tabs werden dafür benutzt, um in einem Programm mehrere Fensterflächen voll Inhalt in nur einem Fenster darzustellen. Speziell zu diesem Zweck entwarf man in grauer Vorzeit eine neue Klasse von Programmen namens Fenstermanager. Fenstermanager erlauben es, mehrere Fenster gleichzeitig darzustellen, gerne auch nebeneinander, hintereinander oder übereinander. Kennt jeder. Und Tabs sind eine Krücke, die genau dieses Prinzip unterlaufen, denn sie organisieren Fenster ineinander.

Aber es ist ja nicht nur das. Wir kennen Tabs schon lange in ganz verschiedenen Formen:

All diesen Formen gemein ist, dass sie wenigstens einen Rahmen bilden, der andeutet, welcher Teil des Fensters sich ändern wird, wenn man einen anderen Tab öffnet. Die Ausnahme: Browser.

Wo hat ein Browser seine Tabs? zwischen Adressleiste und Webseiteninhalt. Bei Firefox (besondere Perversion) sind die Tabs sogar so dargestellt, als würden sie nur die Adresszeile ändern (oben befestigte Reiter), ganz im Gegensatz dazu ändern sie aber sowohl die Adresszeile als auch -viel wichtiger- den Webseiteninhalt. Wenn überhaupt, dann sollten sich die Tabs also am oberen Bildschirmrand befinden, so dass man mit ihnen wenigstens konsistent den kompletten Fensterinhalt ändert. Immerhin hier sieht man bereits Licht: Google Chrome und Safari 4 funktionieren bereits so:

Die Lösung von Safari hat noch ein paar Probleme: Wo verschiebt man Tabs? (Ungeschickt: an dem kleinen Handle am rechten Tabgreifer-Rand), Wieso haben die Tabs keine Favicons? Aber nicht verzagen: Noch ist Safari 4 lediglich eine Beta. Gut hingegen fände ich es, wenn dieses Tabbed-Fenster-Interface nicht nur für Safari, sondern für jedes andere Programm auch verwendbar wäre, also als Feature des Fenstermanagers implementiert würde. Man müsste daran noch ein wenig feilen, aber es würde Einiges sehr viel übersichtlicher gestalten. Snow Leopard anyone?

Weitere gute Ideen rund um Tabs, sortiert nach Browser:

Microsoft Internet Explorer 8
- Tabs werden nach Farben sortiert, je nach dem von welchem Tab aus sie geöffnet wurden
- Im neuen Fenstermanager von Windows 7 werden alle Tabs als eigene Fenster aufgeführt
- Jeder Tab läuft in seinem eigenen Prozess. Bringt aber nix, weil wenn einer abstürzt nimmt er trotzdem den ganzen Browser mit. Ist aber auch noch nur Beta!
Firefox 3 Preview
- Tabs sollen über ein neuartiges Interface beim Wechsel als Miniaturen dargestellt werden, so dass man einfacher zwischen ihnen wechseln kann
- Viele andere nette Vorschläge rund um diese Idee gibts hier: Aza Raskins Blog
Safari 4
- Die Tableiste nimmt keinen eigenen Platz auf dem Bildschirm ein
- Tabs sind am richtigen Ort (oben)
Google Chrome
- Tabs sind am richtigen Ort (oben) aber es gibt dennoch eine klassische Titelleiste des Fensters (im Gegensatz zu Safari 4)
- Jeder Tab läuft in einem eigenen Prozess, daher können einzelne Tabs abstürzen ohne den Browser mitzunehmen.
- Jeder Tab läuft in seiner eigenen Sandbox, was es Angreifern wesentlich erschwert, den Browser hochzunehmen.

Fazit: Es gibt noch viel zu tun, aber anscheinend ergibt sich langsam aber sicher ein Konsens, dass etwas mit Tabs getan werden muss. Es bleibt spannend.

Mirrors Edge

Sun, 29 Mar 2009 22:22:00 +0200

Ich habe am Wochenende, wie in Twitter schon verkündet, Mirror’s Edge (durch-) gespielt. Ganz kurz: Mirror’s Edge ist ein Videospiel für PC/XBOX360/PS3, im Endeffekt ein Platformer aus der Ego-Perspektive, zwar mit der Möglichkeit, Waffen aufzunehmen, aber mit dem klaren Fokus auf Rennen, genauer: Parkours, dem modernen Hindernislauf, hier mit dem Extratwist: auf Hochhausdächern.

Es wurde schon viel gesagt über dieses Spiel. Müde Story, schlechte Charactermodelle, schlechte Shooter-Elemente, unübersichtliche Atmosphäre, kurze Spieldauer, um nur einige der üblichen Kritikpunkte zu nennen.

Ich sage: Alles Popauswurf. Was Mirror’s Edge versucht, ist die Direktheit und Intimität der Egoperspektive mit dem Flow und der Geschwindigkeit eines modernen Platformers zu verbinden. Faith, der Hauptcharakter ist eine kleine “Eurasierin” (Zitat der Entwickler), deren Eltern in einem Widerstandskampf… Bla, bla, bla. Es geht um Flow, es geht darum, mit blitzschnellen Reflexen und akrobatischem Geschick einen Weg über die Dächer einer Großstadt zu finden. Es geht darum, nicht stehen zu bleiben, nicht zu zögern. Es geht um Geschwindigkeit und Freiheit, über Straßenschluchten zu springen, zwischen Klimaanlagen und Baugerüsten zu sprinten und immer schneller, immer weiter, den Großstadtlärm hinter sich zu lassen. Nur begleitet von Faiths immer wieder ins Bild ragenden Armen und Beinen, ihrem schnellen Atem, dem Rhythmus ihrer Schritte. Und ja, man stirbt häufig, wenn man sich bei einem Sprung verschätzt oder es doch nicht schafft, den Jägern zu entkommen. Aber um so größer ist die Befriedigung einen komplizierten Run endlich zu schaffen und mit einem perfekten Sprung über eine Häuserschlucht die Polizei hinter sich zu lassen.

Mirror’s Edge perfektioniert den Flow, das Gefühl der Freiheit und der Geschwindigkeit. Klar, dass da die Ego-Shooter-Mechanik auf der Strecke bleibt, aber Faith ist auch keine Kämpferin, sondern selbsterklärter Runner. Sicher, die Story ist ein wenig lahm, aber wir reden hier über ein Computerspiel: Die Story hat allein das Ziel, die einzelnen Schauplätze miteinander zu verbinden, und das gelingt ihr. Charakterentwicklung kennt das Spiel nicht, allein der Spieler wird besser. Und auch das ist gut so, denn in den späteren Leveln ist der Schwierigkeitsgrad nicht von schlechten Eltern. Aber Checkpoints sind häufig und fair verteilt, so dass dies selten ein Problem wird.

Ach ja, und der Sound ist genial, selten habe ich mich so sehr in eine Spielfigur hineinversetzt gefühlt wie in Faith. Und die Grafik… Ich könnte hier viel schreiben; Ich glaube schlicht, dass die aktuelle Konsolengeneration im Grunde ein Niveau erreicht hat, auf dem weitere Verbesserungen keine große Rolle mehr spielen. Mirror’s Edge, mit seiner gleißend-Weiß-plus-Farbe Ästhetik zeigt hier eine Welt, wie man sie noch nie gesehen hat, in der Farbe nicht nur als Stilmittel, sondern auch als Wegweiser und tatsächlich Spielhilfe eingesetzt wird: Rot ist, wo es weiter geht. Diese Ästhetik ist schwer in Worte zu fassen und gibt dem Ganzen eine ganz eigene Freiheit, im starken Kontrast zum “realistisch”-Grau-Braun, dem sich so viele andere Spiele verschrieben haben.

Und was bleibt unterm Strich? Ich sage: Das Spiel ist es Wert. Es mag seine Mängel haben, aber allein der Mut, einmal etwas Neues zu machen und nicht dem Einheitsbrei ins Land der Realo-Shooter und der digitalen Nebenjobs zu folgen ist der Beachtung Wert. Einmal nicht schießen zu müssen und stumpfen Missionen zu folgen, sondern in einem Spiel Freiheit zu finden, dass ist eine außergewöhnliche Erfahrung. Wer als Spieler etwas Erfrischung sucht, sollte es probieren. Ich finds geil!

Synchronisieren von Google, Äpfeln, Fenstern und Telefonen

Sat, 28 Mar 2009 17:18:00 +0100

Ich hatte lange Zeit zwei Computer, einen Desktop und einen Laptop, jeweils mit verschiedenen Betriebssystemen und Datensätzen. Um dennoch immer mit den selben Daten arbeiten zu können, verwendete ich eine externe Festplatte. Obwohl sehr low-tech, funktionierte diese Lösung absolut tadellos: Meine Linux-Kisten mounteten die Festplatte automatisch in ihr jeweiliges home-Verzeichnis und so konnte ich auf verschiedenen Computern arbeiten, ohne mich um die Synchronizität der Daten kümmern zu müssen.

Fast-forward ein Jahr, tausche Linux gegen Apple und finde es jetzt doch sehr anstrengend, immer eine externe Festplatte mit mir herumzuschleppen – Apple-Snob, der ich bin. Sieht auch unelegant aus, dieses schwarze Kästchen an den hübschen Laptop zu klemmen. Es muss also eine andere Lösung her, um immer auf beiden Rechnern mit aktuellen Dateien arbeiten zu können. Es bietet sich an: MobileMe (damals noch .Mac), genauer, die iDisk, also ein Stück online-Speicher bei Apple, auf dem man von mehreren (Apple-) Rechnern aus arbeiten kann. Das Angebot ist verlockend, aber leider erfüllt MobileMe meine Erwartungen nicht, es gehen Daten verloren und ich ärgere mich, jemals Geld für diesen Dreck ausgegeben zu haben.

Eine Alternative finde ich in Dropbox, welches die Vision des immer synchronen Datenspeichers "in the cloud" endlich erfüllt, wenn auch als Ordner und nicht als Laufwerk. Inzwischen erbringt auch Syncplicity diese Leistung, wenn auch mit einem eigenen Set an Einschränkungen (Es ist aber noch Beta, also kein Grund zur Sorge).

Dennoch: ganz zufrieden bin ich nicht, einfach, weil zwei Rechner immer eine gewisse Menge "mental overhead" bedeuten. Allein, nicht immer am selben Gerät zu sitzen stellt einfach eine Irritation dar, die im Grunde nicht nötig ist. Na gut, und dieses neue "Unibody"-MacBook Pro ist einfach sexy. Also, tausche iMac + MacBook gegen MacBook Pro. Das löst -logisch- auch alle Synchronizitätsprobleme.

Aber ich wäre nicht der Sohn meines Vaters, wenn ich nicht immer noch mehr technischen Schnickschnack haben müsste, enter: the iPhone. Dank Apple und iTunes ist es natürlich kein Problem, Kalender, Email, Kontakte und Musik immer synchronisiert zu halten; Kabel reinstecken, iTunes machen lassen und fertig. Perfektioniert wird das alles aber erst durch Beihilfe von Google, welches durch ActiveSync (sprich: Exchange) nun auch alle meine Kontakte, Kalendereinträge, Emails und Dokumente auf allen Geräten zur Verfügung stellt. Dieses Setup ist nun endlich wirklich extrem zufriedenstellend. Es ist zwar ein Haufen Kleinkram, den man durcharbeiten muss, bis man das alles richtig konfiguriert hat, aber hat man das einmal getan funktioniert es wirklich tadellos! Und Syncplicity und Dropbox laufen auf dem Rechner einfach nur noch als Backup weiter.

E Text Editor goes Open Source

Tue, 24 Mar 2009 15:16:00 +0100

Als ich heute von der FH nach Hause kam, lag das hier in meinem Newsreader und ich war, in Ermangelung eines besseren Wortes, geschockt.

Nur, um es noch einmal zu wiederholen: E, der meiner Meinung nach beste Texteditor der Welt wird Open Source. Das kann man nicht oft genug sagen.

Warum ist E so fantastisch? E ist ein recht neuer, kleiner Texteditor für Windows. Nein, um ihm Genüge zu tun, muss man E in einem Satz mit Emacs, Vim und Textmate nennen, denn wie diese Drei Großen Texteditoren sind auch E keine Grenzen gesetzt, da all seine Funktionalität durch kleine, einfache Skripte entsteht, die von jedem Nutzer beliebig verbessert werden können. Genauer gesagt: E tritt in die direkten Fußstapfen von Textmate, einem Editor für den Mac und ist vollkommen kompatibel mit dessen Erweiterungen, so dass E schon bei seiner Markteinführung auf eine riesige Masse von Sprachen, Snippets und Programmen zurückgreifen konnte, mit der sich jede noch so obskure Textmanipulation bewältigen lässt. Anders als Emacs oder Vim folgt E dabei aber modernen Bedienungs- und Designrichtlinien und fühlt sich genauso flüssig und heimisch an, wie das für eine Windows-Anwendung nur möglich ist. Seine zweite große, und kaum weniger berauschende Stärke ist sein History-System. Jeder Benutzer eines Texteditors kennt die Undo-Taste (mit Ausnahme von Textmate. What a shame.). Undo macht die zuletzt gemachte Änderung rückgängig und ist damit der beste Freund von notirisch fehleranfälligen Menschen. E bringt Undo auf das nächste Level: Denn neben dem normalen Undo-Befehl gibt es noch eine komplette Übersicht aller jemals an einem Dokument gemachten Änderungen, komplett mit verschiedenen Pfaden, die zu verschiedenen Zeiten genommen wurden kompakt und übersichtlich verpackt in einem fantastischen Baumdiagramm. Dazu kommen ein riesiger Haufen Features, die eigentlich jedes Programm haben sollte, welche man aber nur viel zu selten findet:

Inkrementielle Suche mit sofortiger Ergebnisvorschau und Unterstützung von Regulären Ausdrücken!
Direkte Unterstützung von Cygwin für alle möglichen Skriptsprachen!
Syntax-Highlighting für so ziemlich jede denkbare Sprache!
Unterstützung für Snippets!
Multiline-Editing!
Kontinuierliche Weiterentwicklung des Editors!
…

Wer mehr wissen will, der surfe nach e-texteditor.com und schaue sich dort den Screencast und den Blog an, lade sich die Demo herunter oder suche im Internet nach Reviews zu E.

Selbst ich, als eingefleischter Textmate-Benutzer halte E als meinen heimlichen Lieblingseditor, denn schließlich kann er alles, was Textmate kann (und das ist schon enorm gut) und dazu noch eine gute Latte mehr. Wer, wie ich, eigentlich seinen Tag nur mit (a) surfen und (b) tippen verbringt, sollte wirklich darüber nachdenken, ein wenig Geld für einen guten Texteditor auszugeben. Es lohnt sich, und E ist ohne Frage einer der Besten. Und jetzt wird er Open Source. Ich bin begeistert!

Western Digital Festplatten-Umtausch mit Überraschungen

Thu, 19 Mar 2009 21:17:00 +0100

Ich hatte mir vor einer Weile eine externe Festplatte nur für TimeMachine gekauft. Denn TimeMachine ist super, die einzige Backup-Lösung die ganz bewusst nie in Erscheinung tritt es sei denn man braucht sie. TimeMachine läuft leise und unbeachtet im Hintergrund, und gibt mir dieses flauschige Gefühl von Sicherheit, quasi das unsicht- und spürbare Kondom der Computerwelt (bzw. Apple-Welt).

…Bis ich einmal den fatalen Fehler beging, meine externe Festplatte HOCHZUHEBEN. Nicht ruckartig, nicht gewaltsam, sondern tatsächlich sehr sanft, aber wohl gerade zu einem ungünstigen Zeitpunkt, denn die Festplatte gab ein leises Klick von sich und hörte auf zu funktionieren.

Es handelt sich hierbei um eine externe Festplatte der Marke Western Digital MyBook mit 500 Gigabytes, gekauft bei Norskit. Ein Anruf beim Verkäufer ergab, dass dies eine alte Bestellung (weltbewegende 9 Monate) sei und ich daher bei einer anderen Nummer anrufen sollte. Eine sehr freundliche Mitarbeiterin teilte mir dort mit, dass die Firma leider Insolvent sei und ich meine Supportanfrage daher an den Hersteller richten sollte. Und was soll ich sagen? EIN GLÜCK, dass sie das sagte!

Denn auf der Webseite von Western Digital gibt es nicht nur eine Support-Telefonnummer, sondern gleich ein komplettes austausch-Programm für Festplatten, welches vollkommen automatisiert abläuft: Man gibt die Modellnummer seiner Festplatte und -für Notfälle- seine Email-Adresse an und bekommt sofort eine neue Festplatte zugeschickt, mit der einzigen Auflage, die alte, defekte Platte innerhalb von 30 Tagen einzuschicken. Und tatsächlich wurde die neue Platte prompt am nächsten Tag verschickt und kam heute pünktlich per UPS bei mir an.

Faszinierend, so wünsche ich mir Kundenservice. Der einzige Wermutstropfen ist, dass Western Digital offenbar nicht sehr überzeugt von der Ausfallsicherheit seiner Festplatten ist, wenn sie so viel Infrastruktur für den einfachen Austausch bereitstellen… Na egal, ich will mal nicht meckern ;-)

Nachschlag: Es stellt sich heraus, die neue Festplatte unterscheidet sich ein wenig von der alten: Sie ist silber statt schwarz und hat zwei Firewire 800 Anschlüsse anstatt Firewire 400 – ein Glück, dass auch Kabel mitgeliefert wurden, sonst könnte ich sie jetzt nicht anschließen!

Debugging und GCC auf Windows

Sun, 15 Mar 2009 15:10:00 +0100

So, jetzt habe ich mein Mex-File zum Einlesen beliebiger Audiodateien endlich lauffähig auf Windows und Mac. Leider werde ich nicht dafür bezahlt, auch noch eine Linux-Version zu bauen, aber falls Interesse besteht, versuche ich mich vielleicht einmal daran.

The State of The Union: Kleine Dateien einlesen, kein Problem. Exotische Formate einlesen, kein Problem. Metadaten auslesen, kein Problem. Dateigröße, Bitrate und Samplerate auslesen, ein kleines Problem, da diese Parameter bei komprimierten Formaten nicht unbedingt fest stehen. Große Dateien einlösen, auf dem Mac kein Problem, auf Windows… nun ja, es dauert. Eine WAV-Datei von 5:30 min einzulesen, dauert mit Windows momentan ca. eine Stunde. Das kann nicht sein, in der Zeit habe ich die Datei dem Programm vorgelesen, wenn es sein muss.

Also, was ist da faul? Jetzt heißt es debuggen: GDB ist mein Freund, aber leider spreche ich seine Sprache nicht, also Oldschool-Debugging mit printf() (bzw. mexPrintf(); Aber da `#define printf mexPrintf` ist das das selbe). Blöd nur, dass Matlab selbst entscheidet, wann es meine Printfs auf den Bildschirm schreibt und es sich dazu entschlossen hat, dies immer erst nach dem Ausführen der Datei, also erst nachdem es bereits eine Stunde gearbeitet hat, zu tun. Einiges Hirnen später konnte ich Matlab endlich über eine Kombination aus Typecasts, sprintf und mexWarnMsgTxt dazu überreden, wenigstens sporadisch ein paar Informationen herauszugeben.

Das Ergebnis:

Die Datei funktioniert tadellos, ist nur ein wenig langsam (s.o.)
Wer ist schuld? Realloc ist schuld!

Das kam überraschend! Offenbar ist realloc auf dem Mac um mehrere Größenordnungen performanter als auf MinGW/Windows, denn die selbe Anwendung, die auf dem Mac ca. eine Sekunde braucht, braucht auf Windows eine Stunde! Und das allein wegen realloc! (Eigentlich: eine halbe Stunde wegen realloc, der Rest ist der Tatsache geschuldet, dass Windows in einer VM läuft)

Bei WAV-Dateien werden immer 2048 Samples an einem Stück ausgelesen. Danach verwende ich ein realloc, um meinen haupt-Speicherpuffer um diese Größe zu vergrößern und kopiere die neuen Daten dort hinein. Bei meinen 5:30 min macht das bei einer Samplerate von 44100 kHz und zwei Kanälen ca. 15000 Aufrufe von realloc. Komprimierte Datenformate haben üblicherweise kleinere Frames und damit noch einmal wesentlich mehr realloc-Aufrufe. Der Plan ist also, jetzt statt häufiger, kleiner realloc-Aufrufe, seltenere, größere Aufrufe zu machen. Zeit für ein paar Experimente:

realloc()-Größe	realloc()-Aufrufe	benötigte Zeit
2¹¹ = 2048	15000	~1 h
2¹⁶ = 65536	470	~2 min
2¹⁷ = 131072	240	~1 min
2¹⁸ = 262144	120	30 s
2¹⁹ = 524288	60	18 s
2²⁰ = 1048576	30	10.5 s
2²¹ = 2097152	15	7.3 s
2²² = 4194304	7	5.1 s
2²³ = 8388608	3	4.2 s

Das Spannende ist: Ich ändere durch meine Methodik praktisch nichts außer der Anzahl und Größe der realloc-Aufrufe, aber man erkennt einen eindeutigen Zusammenhang zwischen Performance und Anzahl der Aufrufe, ergo ist realloc der alleinige Schuldige für mein Performanceproblem auf Windows.

An dieser Stelle fiel mir ein, dass ich bereits an früherer Stelle einmal die gesamte Länge des Audio-Streams anhand der Metadaten geschätzt hatte. Durch eine somit vorgenommene Prä-Allokation des gesamten Speichers lässt sie die Laufzeit weiter auf 2.2 s drücken. Das ist immernoch nicht einmal halb so schnell wie auf OSX (0.9 s), aber das mag auch an der virtuellen Maschine liegen.

Mehr als diesen anecdotal Evidence kann ich nicht anbieten, aber ich bin mir sicher, dass ich ab jetzt die Finger von inkrementiellen Speichervergrößerungen auf MinGW/Windows lassen werde. Ist das in MSVC ähnlich schlimm, oder habe ich da etwa einen Bug entdeckt?

Get a Mac

Sat, 14 Mar 2009 10:24:00 +0100

Bei meinem üblichen, Samstagmorgendlichen, bettlägrigen Web-Rundgang habe ich heute drei sehr nette Artikel von David Alison gefunden, einem Windows-Programmierer und selbsternanntem Microsoft-Fanboy, der sich einen Mac zulegt und unverhofft glücklich damit wird. So glücklich sogar, dass er inzwischen mehr oder weniger ein Mac-only-User geworden ist.

Ziemlich genau wie ihm ging es mir auch.

Teil 1: A hardcore Windows guy gets a Mac
Teil 2: After two months of Mac, here's why I switched
Teil 3: Switching from Windows to Mac - One Year Later

Wens interessiert: Das ist eine sehr hübsche Einführung darin, warum Macs toll sind. Viel besser als alles, was ich dazu schreiben würde oder bereits getan habe.

Kompilieren auf Windows

Thu, 12 Mar 2009 17:14:00 +0100

Seit einigen Wochen arbeite ich an einem kleinen Projekt: Eine Matlab-Funktion, die, ähnlich wie die standard-Funktion wavread(), Audiodateien einlesen kann. Aber nicht irgendwelche Audiofiles, sondern ALLE MÖGLICHEN Audiofiles. Wie geht das? Jeder kennt VLC, den Video-Player, der so ziemlich jedes Video öffnen kann, das man ihm vorsetzt, selbst wenn man überhaupt keine Codecs installiert hat. VLC basiert auf FFmpeg, einem Open-Source Programm, welches Funktionen bereit stellt, um eben alle möglichen Mediendaten zu öffnen.

Und da FFmpeg freie Software ist, kann man sie auch für andere Dinge verwenden, etwa, um mit Matlab Audiodateien zu öffnen. Fehlt noch eine Verbindung zwischen Matlab und den FFmpeg-C-Bibliotheken, und die gibt es in Form von Mex, der C-Schnittstelle von Matlab. Feine Sache, zwar hat es eine Weile gedauert, bis ich mich in libavformat und libavcodec eingearbeitet hatte (die beiden wichtigsten FFmpeg-Bibliotheken), aber im Endeffekt lief das alles sehr schmerzfrei – und das, obwohl ich bisher Mex-Kompilieren mit Matlab immer als eine grausige Beschäftigung in Erinnerung hatte, gespickt von kryptischen Kompiler-Fehlern und hässlichen Notlösungen.

Bumms, Zack, kaum hatte ich mich versehen, hatte ich ein lauffähiges, tadellos funktionierendes Mex-File auf meinem Mac liegen. Damit hatte ich nicht gerechnet. Also sofort die momentane Euphorie ausnutzen und weiter zu Schritt 2, das Ganze nochmal auf Windows. Meine Probleme, Windows so einzurichten, dass ich endlich Kompilieren kann, hatte ich ja schon berichtet. Ich hatte also Visual Studio 2005 installiert, um Matlab zufrieden zu stellen und einen anständigen Kompiler auf dem System zu haben. Aber war ja klar, MSVC macht wieder sein eigenes Ding und nichts ist mit Standardkonformität und Trallalla: Keine C99-Unterstützung, also keine Variablendeklarationen mitten im Code und keine stdint.h oder inttype.h. Ein Glück, es gibt wieder ein wenig mehr Free Software, die wenigstens letztere Lücke schließt. Dennoch; Ich bekomme mein mex-File nicht zum Laufen. Es ist wie verflucht, kaum setze ich mich an eine Windows-Maschine zum Programmieren, fällt meine Produktivität auf das Niveau eines Backsteins.

Enter gnumex, noch ein weiteres Stück FOSS, das es ermöglicht, GCC als Mex-Kompiler zu verwenden, AUF WINDOWS. Um die Dinge zu vereinfachen, verwendete ich die MinGW-Variante und kaum war diese Hürde genommen… lief alles. Einfach so. Wahrscheinlich bin ich ein Dickschädel und habe einfach nicht die Geistesschärfe, mit Windows-Kompilern zu arbeiten, aber mir scheint, alles was ich diesbezüglich anfasse und das nicht GCC heißt ist zum Scheitern verurteilt. Ein Glück, dass es die vielen klugen Jungen und Mädchen gibt, die so wunderbare freie Software schreiben, die mir das Leben so viel einfacher macht!

Eine Fortsetzung kommt noch…

Vom Sinn und Unsinn, ein Cross-Platform-Compiling-Matlab-System aufzubauen

Thu, 05 Mar 2009 13:55:00 +0100

Mal wieder: Ich schreibe ein Stück Software für meinen Nebenjob bei meinem Signalverarbeitung-Prof. Diesmal geht es darum, beliebige Audio-Files in Matlab einlesen zu können. Perfekt geeignet ist dafür die selbe Bibliothek, die auch von VLC verwendet wird, libavcodec/libavformat. Das ist eine normale C-Bibliothek, es braucht also nur noch ein kleines mex-File, um ihre Funktionalität für Matlab zur Verfügung zu stellen. Klappt auch wunderbar. Auf dem Mac.

Schritt zwei ist dann, das Ganze auf Windows und Linux zum Laufen zu bringen. Eigentlich kein Problem, denn ich habe keine wilden Dinge getan und die Libraries selbst sind wunderbar Cross-Platform, es gibt sie sogar schon vorkompiliert für praktisch jedes denkbare Betriebssystem.

Also, was brauche ich? Zwei Dinge: Matlab und einen C-Compiler (der mitgelieferte LLC-Compiler macht mein Hirn bluten). Matlab zu installieren ist meiner Erfahrung nach schmerzhaft. Bigtime. Nicht, weil Matlab schwer zu installieren wäre, sondern, weil Mathworks nur zwei Installationen pro Schachtel erlaubt, was für meine drei Betriebssysteme zu gewissen Problemen führt. Außerdem müsste ich meine eine Lizenz erst für die Windows-installation umschreiben lassen, und… ach, Schmerzen. Offenbar habe ich die Jungs dort aber schon derart häufig mit Lizenzanfragen genervt, dass sie mich einfach als hoffnungslos aufgegeben haben, denn dieses Mal musste ich keine neue Lizenz erstellen lassen, sondern einfach installieren, Passwort eingeben, und los. Mein Account meldet jetzt, dass ich fünf gleichzeitige Installationen hätte (von zwei erlaubten). Mir solls Recht sein.

Außerdem: ein aktuelles Linux muss her. VMWare sei Dank, lauert im Linux-Installieren nicht mehr der Schrecken, im Zweifelsfall den kompletten Festplatteninhalt zu verlieren, sondern nur noch, an akuter Progressbar-itis zu ersticken. War ja klar, dass Autoupdate sich diesen Nachmittag aussucht, um meine Ubuntu-VM hoffnungslos zu zerstören. Also, neues Ubuntu heruntergeladen, neu installiert, neu Updates aufgespielt, zwei Stunden Lifetime verloren. Immerhin: es hat fehlerfrei funktioniert, das ist was Neues. Matlab hinterher, VMWare Tools dazu, fertig ist die Development-Kiste. Jetzt fehlt nur noch eine Verbindung zu meinem Development-Verzeichnis, damit ich auf meine Dateien zugreifen kann. Fehlanzeige. Dukommsthiernichrein. Na Toll.

Also weiter zu Windows. Frühere Versuche ergaben bereits, dass ich Matlab nicht dazu bewegen kann, (a) GCC als Compiler zu nehmen oder (b) das bereits installierte Microsoft Visual Studio C++ .Net Professional Directors Cut Special Edition 2008 Ultimate zu verwenden. Nähere Nachforschungen zeigen: Zu neu, Kennternich. Geht nur bis MSVC Jahrgang 2005. Also: Neues MSVC deinstallieren, Altes installieren. ich freue mich immer darauf, MSVC zu deinstallieren, denn es besteht lediglich aus kompakten 12 Programmen, die sich zwar alle auf einem Haufen Installieren- jedoch nicht De-Installieren lassen. Immerhin ist es dank MSDNAA-Membership nicht schwer, an die alten Versionen heranzukommen. Und klar, die Systemsteuerung lässt einen auch immer nur ein Programm auf einmal deinstallieren. Multitasking ist nicht. Dank Syncplicity kann Windows die Zwischenzeit immerhin dazu verwenden, all meine Development-Files auf den Rechner zu laden. Yay! Das Schöne an Fortschrittsbalken ist ja, sie zeigen Fortschritt. Damit haben sie einen klaren Vorteil gegenüber etwa Dachbalken oder den Bittewartenpunktpunktpunkt-Balken, die die Microsoft SQL Server 2008-Deinstallation stolz herumzeigt. Die fühlt sich wohl sehr wichtig, denn sie rödelt eine starke halbe Stunde im Bitte-Warten-Modus herum. So mag ich Deinstallationen.

To be continued…