Basti's Scratchpad on the Internet

Posts tagged "macos":

29 Dec 2017

How to set up rsnapshot instead of Time Machine

(This blog post was changed since my initial strategy of disabling the lockfile didn't work. Turns out, the lockfile is required, and backups have to be stacked.)

Yesterday, I wrote about how Time Machine has failed me. Time Machine keeps regular backups, going back as far as your hard drive space permits. In theory. In practice, every year or so it messes up somehow and has to start over, thereby deleting all your older backups. A backup that is not reliable is not a backup.

Luckily, there are alternatives. Probably the easiest is rsync[1], a very cool tool that copies files and directories from one place to another. You could simply run this once a day, and have a new backup every day. You can even configure rsync so it doesn't need to copy unchanged files, and instead hard-links them from an older backup. rsnapshot automates this process to keep a number of tiered copies, for example ten hourly backups, seven daily backups, four weekly backups, and a hundred monthly backups. Each backup is then simply a directory that contains your files. No fancy starfield-GUI, but utterly reliable and trivial to understand [2].

Setting up rsnapshot on macOS is not quite as straight-forward as I'd like, and I couldn't find a great guide online. So, without further ado, here's how to configure rsnapshot on macOS:

[1] rsync is one of those reliable tools I talked about. It is rock solid, incredibly versatile, and unapologetically single-minded. A true gem!

[2] This works great for local backups. If you need encrypted backups or compressed backups (maybe on an untrusted remote machine), this post recommends Borg instead of rsnapshot, but you will lose the simplicity of simple directories.

[3] I use launchd instead of cron since launchd will re-schedule missed backups if the computer was asleep.

[4] This will fail if the hourly backup takes longer than 15 minutes. This is rather unlikely, though, or at least should not happen often enough to be of concern.

Caveats

The configuration file of rsnapshot says that you might experience data corruption if you run several copies of rsnapshot at the same time (and you can use the lockfile to prevent this). This is a problem if your computer is asleep while rsnapshot is scheduled to run, since launchd will then re-schedule all missed tasks at once when the computer wakes up. If you enable the lockfile, only one of them will run.

On the other hand, only the hourly task will actually create a new backup. All higher-level backup tiers merely copy existing backups around, so in theory, they shouldn't step on each other's toes when run concurrently. I have opened an issue asking about this.

There are other possible solutions: ① You could modify the launchd entry such that backups only trigger after a few minutes or, better yet, only once all other instances of rsnapshot have finished. I am not sure if launchd supports this, though. ② You could schedule the hourly task using cron instead of launchd, since cron will not reschedule missed tasks. This would only work for two tiers of backups, though. ③ You could just ignore the issue and hope for the best. After all, if a daily or hourly backup gets corrupted every now and then, you still have enough working backups…

Tags: backup macos
17 Jun 2017

Audio APIs, Part 1: Core Audio / macOS

This is part one of a three-part series on the native audio APIs for Windows, Linux, and macOS. This first part is about Core Audio on macOS.

It has long been a major frustration for my work that Python does not have a great package for playing and recording audio. My first step to improve this situation were a small contribution to PyAudio, a CPython extension that exposes the C library PortAudio to Python. However, I soon realized that PyAudio mirrors PortAudio a bit too closely for comfort. Thus, I set out to write PySoundCard, which is a higher-level wrapper for PortAudio that tries to be more pythonic and uses NumPy arrays instead of untyped bytes buffers for audio data. However, I then realized that PortAudio itself had some inherent problems that a wrapper would not be able to solve, and a truly great solution would need to do it the hard way:

Instead of relying on PortAudio, I would have to use the native audio APIs of the three major platforms directly, and implement a simple, cross-platform, high-level, NumPy-aware Python API myself. This effort resulted in PythonAudio, a new pure-Python package that uses CFFI to talk to PulseAudio on Linux, Core Audio on macOS, and WASAPI[1] on Windows.

This series of blog posts summarizes my experiences with these three APIs and outlines the basic structure of how to use them. For reference, the singular use case in PythonAudio is playing/recording of short blocks of float data at arbitrary sampling rates and block sizes. All connected sound cards should be listable and selectable, with correct detection of the system default sound card (a feature that is very unreliable in PortAudio).

[1]: WASAPI is part of the Windows Core Audio APIs. To avoid confusion with the macOS API of the same name, I will always to refer to it as WASAPI.


CoreAudio, or the Mac's best kept secret

CoreAudio is the native audio library for macOS. It is known for its high performance, low latency, and horrible documentation. After having used the native audio APIs on all three platforms, CoreAudio was by far the hardest one to use. The main problem is lack of documentation and lack of feedback, and plain missing or broken features. Let's get started.

The basic unit of any CoreAudio program is the audio unit. An audio unit can be a source (aka microphone), a sink (aka speaker) or an audio processor (both sink and source). Each audio unit can have several input buses, and several output buses, each of which can have several channels. The meaning of these buses varies wildly and is often underdocumented. Furthermore, every audio unit has several properties, such as a sample rate, block sizes, and a data format, and parameters, which are like properties, but presumably different in some undocumented way.

In order to use an audio unit, you create an AudioComponentDescription that describes whether you want a source or sink unit, or an effect unit, and what kind of effect you want (AudioComponent is an alternative name for audio unit). With the description, you can create an AudioComponentInstance, which is then an opaque struct pointer to your newly created audio unit. So far so good.

The next step is then to configure the audio unit using AudioUnitGetProperty and AudioUnitSetProperty. This is surprisingly hard, since every property can be configured for every bus (sometimes called element) of every input or output of every unit, and the documentation is extremely terse on which of these combinations are valid. Some invalid combinations return error codes, while others only lead to errors during playback/recording. Furthermore, the definition of what constitutes an input or output is interpreted quite differently in different places: One place calls a microphone an input, since it records audio; another place will call it an output, since it outputs audio data to the system. In one crazy example, you have to configure a microphone unit by disabling its output bus 0, and enabling its input bus 1, but then read audio data from its ostensibly disabled output bus 0.

The property interface is untyped, meaning that every property has to be given an identifier, a void pointer that points to a matching data structure, and the size of that data structure. Sometimes the setter allocates additional memory, in which case the documentation does not contain any information on who should free this memory. Most objects are passed around as opaque struct pointers with dedicated constructor and destructor functions. All of this does not strike me as particularly C-like, even though CoreAudio is supposedly a native C library.

Once your audio unit is configured, you set a render callback function, and start the audio unit. All important interaction now happens within that callback function. In a strange reversal of typical control flow, input data to the callback function needs to be fetched by calling AudioUnitRender (evoked on the unit itself) from within the callback, while output is written to memory provided as callback function arguments. Many times during development, AudioUnitRender would return error codes because of an invalid property setting during initialization. Of course, it won't tell which property is actually at fault, just that it can't fulfill the render request at the moment.

Error codes in general are a difficult topic in CoreAudio. Most functions return an error code as an OSStatus value (aka uint32), and the header files usually contain a definition of some, but not all, possible error codes. Sometimes these error codes are descriptive and nice, but often they are way too general. My favorite is the frequent kAudioUnitErr_CannotDoInCurrentContext, which is just about as useless an error description as possible. Worse, some error codes are not defined as numeric constants, but as int err = 'abcd', which makes them un-searchable in the source file. Luckily, this madness can be averted using https://osstatus.com/, which is a dedicated database for OSStatus error codes.

By far the worst part of the CoreAudio API is that some properties are silently ignored. For example, you can set the sample rate or priming information on a microphone unit, and it will accept that property change and it will report that property as changed, but it will still use its default value when recording (aka "rendering" in CoreAudio). A speaker unit, in contrast, will honor the sample rate property, and resample as necessary. If you still need to resample your microphone recordings, you have to use a separate AudioConverter unit, which is its own bag of fun (and only documented in a remark in one overview document).

Lastly, all the online documentation is written for Swift and Objective-C, while the implementation is C. Worse, the C headers contain vastly more information than the online documentation, and the online documentation often does not even reference the C header file name. Of course header files are spread into the CoreAudio framework, the AudioToolkit framework, and the AudioUnit framework, which makes even grepping a joy.

All of that said, once you know what to do and how to do it, the resulting code is relatively compact and readable. The API does contain inconsistencies and questionable design choices, but the real problem is the documentation. I spent way too much time reading the header files over and over again, and searching through (often outdated or misleading) example projects and vague high-level overviews for clues on how to interpret error messages and API documentation. I had somewhat better luck with a few blog posts on the subject, but the general consensus seems to be that the main concepts of CoreAudio are woefully under-explained, and documentation about edge cases is almost nonexistent. Needless to say, I did not enjoy my experience with CoreAudio.

Tags: audio programming macos
16 Oct 2015

OS X Finder Woes

Mac.png

The Mac. It used to be the most streamlined, thought-through general computing device on the market.

Even it's file management used to be top-notch. There were many cool little touches. One particularly useful feature was the Proxy Icon–if a window displayed a file's content, that file's icon would show up in the window's title. And you could drag that icon directly onto a thumb drive or email, without having to use the Finder. But the Finder, too, had many neat little features. I loved the fact that when you renamed a file in an alphabetically sorted file list, Finder would not immediately re-shuffle it to its new location, but would wait half a second before doing so. When renaming multiple files, this was really useful, since you could go through them one by one and rename them, simply by pressing arrow keys and return.

But as you might have guessed from my use of the past tense, these golden days are gone. The Finder used to know a JPEG from a ZIP regardless of file extension. Now it doesn't any more. The Proxy Icon is still draggable, but it will create an alias instead of a copy–perfectly useless on a thumb drive or in an email.

And with the newest version of OS X, El Capitan, they finally blew it for me. Before, even though the Finder inexplicably never had the ability to cut and paste files, you could always install programs like TotalFinder to fix that. Not so with El Capitan. The Finder now is holy land, and can not be touched any more by third parties. So no more cut and paste, no more un-hiding system files. No more side-by-side Finder tabs. And brand new with El Capitan as well: No more waiting after renaming. Now, when you rename a file, it is immediately re-sorted to its new position, thus making renaming multiple files terribly inconvenient.

So, good bye OS X. I updated my work laptop first, and I regret it. I never regretted an OS X update before. My home machine is not going to get the update. It is honestly sad to see my once-beloved Mac platform becoming worse and worse and worse with every new release.

Tags: macos ui
03 Oct 2015

Changing File Creation Dates in OSX

On my last vacation, I have taken a bunch of pictures, and a bunch of video. The problem is, I hadn't used the video camera in a long time, and it believed that all it's videos were taken on the first of January 2012. So in order for the pictures to show up correctly in my picture library, I wanted to correct that.

For images, this is relatively easy: Most picture libraries support some kind of bulk date changes, and there are a bunch of command line utilities that can do it, too. But none of these tools work for video (exiftool claims be able to do that, but I couldn't get it to work).

So instead, I went about to change the file creation date of the actual video files. And it turns out, this is surprisingly hard! The thing is, most Unix systems (a Mac is technically a Unix system) don't even know the concept of a file creation date. Thus, most Unix utilities, including most programming languages, don't know how to deal with that, either.

If you have XCode installed, this will come with SetFile, a command line utility that can change file creation dates. Note that SetFile can change either the file creation date, or the file modification date, but not both at the same time, as any normal Unix utility would. Also note that SetFile expects dates in American notation, which is about as nonsensical as date formats come.

Anyway, here's a small Python script that changes the file creation date (but not the time) of a bunch of video files:

import os.path
import os
import datetime
# I want to change the dates on the files GOPR0246.MP4-GOPR0264.MP4
for index in range(426, 465):
    filename = 'GOPR0{}.MP4'.format(index)
    # extract old date:
    date = datetime.datetime.fromtimestamp(os.path.getctime(filename))
    # create a new date with the same time, but on 2015-08-22
    new_date = datetime.datetime(2015,  8, 22, date.hour, date.minute, date.second)
    # set the file creation date with the "-d" switch, which presumably stands for "dodification"
    os.system('SetFile -d "{}" {}'.format(new_date.strftime('%m/%d/%Y %H:%M:%S'), filename))
    # set the file modification date with the "-m" switch
    os.system('SetFile -m "{}" {}'.format(new_date.strftime('%m/%d/%Y %H:%M:%S'), filename))
Tags: programming macos
13 Sep 2012

Kindle.app not starting on a case-sensitive file system

So Kindle.app was updated through the App Store and did not start any more. It just crashed and the crash reporter came up.

A quick look at Console.app turns up a link to the actual crash report. And the crash report starts with

    Dyld Error Message:
      Library not loaded: @executable_path/../Frameworks/libWEbCoreKRF.dylib
      Referenced from: /Applications/Kindle.app/Contents/Frameworks/libWebCoreViewer.dylib
      Reason: image not found

libWEbCoreKRF.dylib? With a capitalized E? That looks very much like a spelling error. So, point your terminal to /Applications/Kindle.app/Contents/Frameworks/, type

    sudo ln -s libWebCoreKRF.dylib libWEbCoreKRF.dylib

to create a symlink with the misspelled name. Done.

Kindle works again.

Spelling is hard, it seems.

Tags: macos
29 Jul 2012

Using a Raspberry Pi as a Time Capsule for Mountain Lion

A while ago, I bought a Time Capsule to take care of my backups. I can't say it has been smooth sailing. Every now and then, the Time Capsule would claim that the backup did fail. Sometimes a reboot would help, sometimes not. Sometimes hdiutil would be able to salvage the backups, sometimes not. Sometimes, the backup disk image would simply be corrupted and the only option would be to delete it and start over.

This might be bad luck or it might be due to a defective Time Capsule or it might be due to my computer. I have no idea. But the thing is, if I have to hack on my backup system anyway, lets do it in style, at least. So here goes:

Ingredients: A Raspberry Pi, an external hard drive, some patience

Format an SD card as described in the wiki. I just installed the version of Debian that is provided on the official website. Now just boot up.

Next, I was stumped because I only have an Apple LED display and no convenient way of connecting the Raspberry Pi's HDMI output to the LED display's Mini Display Port. After some searching and a combination of three adapter cables, I finally got it connected and could see it boot. Really, I have no use whatsoever for the HDMI port on the Raspberry pi. So the first thing I did was to enable SSH, which luckily is available right there in the configuration utility that starts when you boot the thing for the first time.

After that, I disconnected the display and immediately was stumped because I now had no way of finding the Pi's IP address. Actually, I did not even have a network to connect it to. So I strung an ethernet cable from my laptop to the Pi and enabled Internet Sharing in order to (1) start the DHCP server and (2) give the Pi internet access. The IP address was then easily found using arp -a.

Setting up the hard drive

First off, I needed to format and mount my external hard drive to be usable as a Time Machine volume. ls /dev showed the hard drive as /dev/sda. Thus, I installed parted using sudo apt-get install parted and used it sudo parted. In parted, select /dev/sda sets it up to modify the external hard drive, rm 1 deleted its main partition, q to quit parted. Next, creating a new partition: sudo fdisk, then in there n with p and 1 to create a new primary partition, then w to apply the changes and exit. Lastly, I created the file system with sudo mkfs -t ext4 /dev/sda1 with the whole partition as its size. Now lastly, I created a mount point for it using mkdir ~/TimeMachine (don't use sudo!) and auto-mounted it by appending this to /etc/fstab

    /dev/sda1 /home/pi/TimeMachine ext4 rw,auto,user,exec,sync 0 0

Note: sync specifies that all file system changes have to be written to disk immediately, without caching. This might be bad for performance, but on the other hand, this behavior is probably a good idea for a backup system. I once read something somewhere that Apple is enforcing a similar behavior on their Time Capsules and that this is the reason why they won't allow any other network drive as Time Capsules.

Setting up the shared folder

First up, this requires netatalk, so I did sudo apt-get update and sudo apt-get install netatalk to install it. Next, netatalk has to be configured to actually share the drive on the network. This is accomplished by appending this line to /etc/netatalk/AppleVolumes.default:

    /home/pi/TimeMachine TimeMachine allow:pi cnidscheme:dbd options:upriv,usedot,tm

Also, the afp daemon should be configured to use the proper authentification schemes. Thus, add this to /etc/netatalk/afpd.conf:

    - -transall -uamlist uams_randnum.so,uams_dhx.so,uams_dhx2.so -nosavepassword -advertise_ssh

(maybe append mdns to the hosts in /etc/nsswitch.conf? Probably not necessary.)

I am also not quite sure whether I actually had to create a new file /etc/avahi/services/afpd.service and write into it:

    <?xml version="1.0" standalone='no'?><!--*-nxml-*-->
    <!DOCTYPE service-group SYSTEM "avahi-service.dtd">
    <service-group>
        <name replace-wildcards="yes">%h</name>
        <service>
            <type>_afpovertcp._tcp</type>
            <port>548</port>
        </service>
        <service>
            <type>_device-info._tcp</type>
            <port>0</port>
            <txt-record>model=Xserve</txt-record>
        </service>
    </service-group>

And maybe, you have to create an empty file that signifies the drive as Time Machine compatible using touch ~/TimeMachine/.com.apple.timemachine.supported.

Edit: Turns out, all these were not necessary. Thank you, reader Philipp, for trying them out!

I certainly did all that, but I am not quite sure which of these steps were strictly necessary. If you know, please let me, too.

Anyway, with all that done, restart both the netatalk and the Bonjour daemon using sudo /etc/init.d/netatalk restart and sudo /etc/init.d/avahi-daemon restart.

Setting up the Time Machine

Now, back to the Mac. In order to make Time Machine accept the new network share, run

defaults write com.apple.systempreferences TMShowUnsupportedNetworkVolumes 1

Edit: Turns out, this setting is not necessary. OSX just picks the Raspberry Pi as a usable Time Machine drive by default.

Finally, the TimeMachine folder on the Raspberry Pi was available as one of the backup drives. Halleluja!

Now transfer speeds for the initial backup are not exactly what I would call fast, but this might not be the Pi's fault. For one thing, the Pi is reporting to only run at half load. For another thing, the external hard drive and its USB connection is probably not very speedy. And lastly, I seem to remember that initial backups always were slow. But really, only time will tell how well this thing can do the job of a Time Capsule.

Further testing shows that transfer speeds are very comparable to the Time Capsule. Thus, I declare this a raging success!

This article heavily steals from these fine folks on the internet:

Tags: macos backup
19 Jul 2012

How Apple is Failing Me

I bought my first Apple computer in 2007 after a long time of gaming on Windows and a few years of Linux. In the beginning, I was just amazed at the consistency, practicality and sanity of the whole experience. I think I had a bad case of tinkeritis and neophilia in my Linux days, so this was probably to be expected.

Later, I marveled at the level of polish in the third party software ecosystem around Macs. Then I was just glad to have a slick GUI with actual Unix underpinnings. For reference, this happened in the time span from OS X 10.5 Leopard until 10.7 Lion. However, in that same time frame, I also noticed a gradual increase in problems. The computer seems to crash slightly more often, programs seemed to become more buggy, behavior less logical. At first, I marked this up as mostly a matter of my own increasing knowledge of the system, but as time went on, I became more and more convinced that it was actually the system getting worse, not myself becoming more sensitive.

So here are some of those problems:

Lack of Uninstallers

Computer people like to try new software. Some new software is crap. You should delete crap from your computer. Amazingly however, the Mac does not have any way of uninstalling some software.

At the moment, there are three ways of getting software on your Mac.

  1. Install from the App Store. Installation and un-installation work through the App Store.
  2. Install by dragging n *.app bundle onto your computer. Uninstall by deleting the bundle. Works fine unless the app installed some stuff outside the bundle.
  3. Install by executing a *.pkg file. No way to uninstall whatsoever unless the developer provides a separate uninstall mechanism.

Really, this is beyond ridiculous. There is no way of uninstalling software that has been installed from a *.pkg. How the heck am I supposed to use my computer without uninstalling software? Seriously, Apple? Is this a joke?

(So as a rule, I will only install *.pkg if there is absolutely no other way. Most of the time, I use or abuse homebrew to mitigate this problem.)

Failing Time Machine

So the Mac comes with this awesome backup solution called Time Machine. Just plug in your external hard drive, and it will magically keep all your data save, at all times. Better yet, buy a Time Capsule and this will happen over the wireless network so you don't even need to plug in that hard drive any more. Magic!

Except, every few weeks the Time Machine icon in the menu bar will have a small exclamation point, which indicates that something went wrong. Usually, it means that Time Machine somehow lost track of what it was doing and silently corrupted all backups beyond repair. This happens across several computers and Time Machines. The only solution is to delete all backups and start anew.

So, backups that are not backing up. How amazing is that?

Living in the Cloud

OS X is this astonishingly modern operating system, where all your email, contacts, calendars and settings are synced in the cloud, accessible from anywhere and any device or service you use. In theory.

Synchronization is cool. I used to live within the Googleverse, and all my contacts, calendars and email accounts would sync through Google to all my Apple devices. Life was good! Until things started to go bad. At some point, Mail.app lost an email or two. Not a big deal. Then iCal would not connect any more, and would need to be restarted every other day. And finally, Address Book deleted all my contacts from Google. Actually, it did not delete the whole contacts, just all the information inside them, and leave empty husks with only a name and maybe a partial email address.

I wish I was making this up. I tested this quite thoroughly. Whenever Contacts was set up to sync with Google, after a few days or weeks it would delete all my contacts. Of course, this being cloud backed, it would delete them everywhere on all my devices, without any backup or anything. Thankfully, Address Book on iOS seems to not have this problem.

Things can go wrong, I understand that. But deleting all my contacts? Not cool. Not. Cool.

iPhoto

My grandfather has this G4 PowerBook. He stores all his photos on it. So at one point, we just could not stand it any more to support his software problems and set him up with a newer MacBook (2007) just so we could at least roughly recreate his problems on our machines.

But to do that, we had to convert his old library from iPhoto 1876 to iPhoto 2008. iPhoto itself claims to be able to do that, and lo and behold, it magically imported the old library, started an update program, then another, then another, then another, then another, and updated the old library through the ages up to the most recent version. And everything seemed to work fine, all the albums were there, all the thumbnails looked right, just perfect. Except if you opened any image, it would only show black. I mean, I can understand that an import of a library that old does not work. But at least tell me so. Don't silently corrupt everything!

And then there was my own library, started in late 2007 and handed through different revisions of iPhoto to today. It countains about 25 Gb worth of photos. Funny enough, its size on disk is actually 65 Gb. What. The. Heck?

At this point, a giant thank you to the iPhoto Library Manager by Fat Cat Software, which saved both my library and my sanity! Those guys rock!

OS X Lion with Auto Save and Versions

The newest and brightest in the land of Apple is OS X 10.7 Lion (at the time of writing). This amazing piece of software has a life-changing new feature: It friggin breaks one of the oldest tradition in computer history: Saving files.

Now normally, I would applaud this. Just save everything automatically and give me a sensible undo mechanism instead. Great!

However, in a stroke of genius, they also got rid of the possibility of save as…, which is just an easy way of duplicating a document. Say, because you received it in an email and want to file it somewhere else on your hard drive. Or because the document is saved on a thumb drive and you want to copy it over to your hard drive. No sir, you have to navigate to the document in the finder and copy it by hand from there. Thank you for your patience.

But the worst offender is Preview. Every program on the Mac has this very handy proxy icon in the window title. Whenever a window is representing a file on your computer, it will show that very file in the title, so you can drag and drop it somewhere else. Except preview won't give you the actual document. Instead, it will give you an alias to that document. Which is worth nothing at all if you drag it into an email, your Dropbox or a thumb drive.

Or try to enable File sharing on your local network, which works sometimes, but not all the time. Or Internet Sharing, which might share your internet connection, or break your DNS settings. Or take Bluetooth, which for the life of me, I can not make work for tethering with my iPhone, even though the same works just fine with the iPad and other Macs. Or take networking in general, which every once in a while just completely breaks down on my company's network and just refuses to send or receive any data over the Ethernet cable until I reboot my computer.

I could go on. Let's just say that Lion increased my WTF-per-minute rate significantly.

Performance, Memory and File System

Lets talk performance. I can't really complain here. In my MacBook Pro I have four cores, eight Gb of memory and a fast, Apple-sanctioned SSD. Life is good. Except that I happen to use virtual machines a whole lot and regardless of whether I use Linux (Ubuntu, Fedora, OpenSuse) or Windows (7, 8) on there, they feel faster.

I mean, you don't feel the pain of starting applications much on OS X, because in general, you simply keep your programs running all the time. But even so, Firefox starts up on Windows in, like, no time at all, but takes noticeable time on OS X (same configuration and plugins). I have an SSD, so I mostly don't care, but still.

Or take boot times. Even in a virtual machine, both Windows 7 and 8 and Ubuntu 12.04 and Fedora 17 boot up in less than half a minute. In comparison, OS X Lion takes significantly longer, sometimes minutes. Again, you usually don't feel that pain too much because you don't reboot your Mac all that often (unless you run Bootcamp for gaming regularly). Still, OS X boot and shutdown times are downright shameful these days.

Lets run a few virtual machines. I have eight Gb of memory. Lets boot up a Windows VM and give it two Gb. Do anything you want, run Eclipse and Visual Studio and Firefox with loads of tabs and Steam. Windows will do just fine. Those two Gb are plenty for anything except gaming and photo/video editing. Do the same thing with Linux, same result. In fact, Linux will work just fine with even less memory. But OS X? Don't even think of running it with less than two Gb. Four Gb will give you a workable system if you don't run too many memory intensive applications. But start up XCode and Eclipse and you are done for. This is a huge waste.

And lastly, let's talk HFS+. Did you know that HFS+ has a global lock that prevents any program from accessing the hard drive while another program is doing so? That explains why OS X is so friggin slow on non-SSD drives. And did you ever notice that HFS+ is slowly corrupting your data, even if there are no power failures or anything? I fire up Disk Utility every so often and it will always find some file system corruptions. It rarely breaks stuff, but really, the file system is the very last part of my system I want to see failing randomly.

So, what now?

With all this love, why am I still using a Mac? Well, for one thing, the hardware is seriously great. I would love to see some laptop manufacturer build something equally sturdy, light and good looking as a MacBook Pro. I would love to see someone build a great touch pad into a non-Apple device. I would love to see Retina displays everywhere1. But I don't.

And then there is software. Where is Tower for Windows or Linux? Or Sparrow or Tweetbot or Reeder or Pixelmator or iPhoto? I'm not saying there are no alternatives, but I certainly have not been able to find any that were really up to the same level of polish.

That said, I have recently been playing around with Windows 8, Ubuntu 12.04 and Fedora 17 and I must say, I am quite taken with them. All of them. As for the next computer I am going to buy, I am doubtful if it will be an Apple computer again.

Footnotes:

1

As a matter of fact, I would love to see well enough to see Retina at all, but that is beside the point

Tags: macos
22 Feb 2012

How to gem install rlua on OSX/homebrew

If you want to use rlua, you need to have Lua installed. However, OSX does not come with Lua preinstalled, so you install it using

#+begin src sh brew install lua #endsrc

Oh, would it be nice if everyone agreed on how to install stuff like this. Case in point, homebrew installs liblua (quite reasonable) in /usr/local/lib/liblua.[5.1[.4]].dylib

rlua however expects it to be called liblua5.1.dylib (notice the missing .).

Similarly, the headers are installed plainly into /usr/local/include, whereas rlua expects them to be in a folder called lua5.1.

Hence, here is how you get rlua to install:

    ln -s /usr/local/Cellar/lua/5.1.4/lib/liblua.5.1.4.dylib /usr/local/lib/liblua5.1.dylib
    ln -s /usr/local/Cellar/lua/5.1.4/include/ /usr/local/include/lua5.1

Not exactly a beautiful solution, but it works.

Tags: compiling macos
03 Sep 2011

Adobe AIR Application Crashes

Adobe, oh Adobe. Why?

Your updater pops open. Adobe AIR 2.5, if I remember correctly. It runs, and after that, all AIR applications stop working. They boot, then crash and barf up a crash report.

Turns out, this is a problem with case sensitive file systems. If your file system is not case sensitive, you are experiencing some other problem and this fix is not for you.

See, Adobe has this library called WebKit.dylib. However, their code actually looks for Webkit.dylib (note the capitalisation of the K). Well, spelling is hard, I presume.

At least this is an error that can easily be fixed. Fire up your terminal, type

cd /Library/Frameworks/Adobe\ AIR.framework/Version/Current/Resources/
sudo ln -s WebKit.dylib Webkit.dylib

Then enter your password and your AIR applications should work again.

Dear Adobe, is it too much to ask to pretty please make your software work on a case sensitive file system? Spelling is primary school stuff. It is really not that hard!

Tags: macos
01 Aug 2011

Compiling Scipy and Matplotlib using pip on Lion

So I upgraded to Lion. Predictably, some things went wrong. This time, the main thing that bit me was that for some reason, pip stopped working. After a bit of messing around with brew, pip and easy_install, I found out it was almost entirely my own fault. I messed up my PATH.

In the meantime, I had uninstalled all of brew's Python, so I had to reinstall. For me, that entails Python, Numpy, Scipy and Matplotlib. Only this time, Scipy would not build. Some obscure error in some veclib_cabi_c.c would report errors. A quick round of googling reveals:

In order to get Scipy to compile, you need to insert #include <complex.h> in

    ./scipy/lib/blas/fblaswrap_veclib_c.c.src
    ./scipy/linalg/src/fblaswrap_veclib_c.c
    ./scipy/sparse/linalg/eigen/arpack/ARPACK/FWRAPPERS/veclib_cabi_c.c

That done, Scipy compiles perfectly fine.

But, that is not enough yet. As this blogpost outlines, Matplotlib is not currently compatible with libpng 1.5, which ships with Lion. Fortunately, this is already fixed in the most recent source on the Matplotlib repo, so you just have to checkout that:

    pip install -e git+https://github.com/matplotlib/matplotlib.git#egg=matplotlib

By doing that, Matplotlib should install just fine.

Seriously though, these PyPi repos are in a very sorry state. Every time I install one of these packages, I have to jump through hoops and spend hours debugging packages that really should work right out of the box. After all, brew, rvm and gem can do it just fine. Why is pip such a horrible mess?

Tags: compiling python macos
04 Mar 2011

Installing Pygame using Homebrew

So I want to do audio development on the Mac without using Matlab. An alternative to Matlab is Python, or rather, Numpy, Scipy and Matplotlib. They are awesome for working with audio data. What they don't do however is playing back audio. There are several packages out there that would afford audio playback. If you are serious about this though, you not only want audio playback, you want asynchronous audio playback. That is, you want to send some audio data to the sound card and continue with your program without waiting for the audio to finish playing. This allows continuous audio playback of computer-generated sound.

Pygame is one package that allows this. (I will submit a patch to Pyaudio soon that will enable it there, too). There are pre-built binaries on the Pygame website that you can install easily. But then there would be no easy way to uninstall them, so what I would rather want is to install Pygame using package managers that allow easy updating and uninstallation. My tool of choice on the Mac is of course Homebrew.

Note that although I am mostly interested in audio playback, this post will detail the installation of all modules of Pygame, not just pygame.mixer.

Homebrew won't install Pygame, but it will install all the prerequisites for Pygame. So, let's do that.

brew install sdl, sdl_mixer, sdl_ttf, libpng, jpeg, sdl_image, portmidi

This will install most packages for you. Note that libpng is also available as a system library, so it is installed keg_only, that is, without linking it in your path. We will need to compile against it though, so the next step is

brew link libpng

Now there is still one package missing, smpeg. Sadly, smpeg does not install its headers, so you can't compile against it. To fix that, type

brew edit smpeg

and add the following line just above the two end at the end of the file

include.install Dir["*.h"]

Then save the file. (I submitted a bug to have this fixed, so you might not need to do this when you read this). Now you can install smpeg with the usual

brew install smpeg

and you will get the headers, too. Isn't Homebrew great?

Now that all the prerequisites are met, lets look at Pygame itself. This is rather more difficult, as it will not build properly against Homebrew libraries on its own. First, download the source package of Pygame from the [official website](http://www.pygame.org/download.shtml). Unpack it to some directory.

Now open a terminal and navigate to that directory. Me, I like [iTerm](http://iterm.sourceforge.net/), but Terminal.app will do just fine, too. In there, run python config.py to create an initial setup file.

At this point, the setup file is mostly useless since config.py failed to find any homebrew-installed library. It is also strangely garbled, so there is some manual labor to do. Open the file Setup (no extension) in your favourite text editor. After the first comment block, you will see a line that looks like this

SDL = -I/NEED_INC_PATH_FIX -L/NEED_LIB_PATH_FIX -lSDL

Obviously, this is lacking the paths to the SDL library. If you installed Homebrew to its default directory, this will be in /usr/local…. Hence, change this line to

SDL = -I/usr/local/include/SDL -L/usr/local/lib -lSDL

The next lines are strangely garbled. They say, for example

FONT = -lS -lD -lL -l_ -lt -lt -lf

Where they actually should say

FONT = -lSDL_ttf

Instead of having one -l and then the library name SDL_ttf, they put -l in front of every single letter of the name. This is strange, and certainly wrong. So, correct it for FONT, IMAGE, MIXER and SMPEG.

Note that I did not tell you to do this for PORTTIME, too. Actually, PORTTIME is already correctly linked in PORTMIDI, so you don't need that at all any more. Just delete or comment the PORTTIME line.

Now that all the dependencies are corrected, lets enable the features. A few lines further down, there will be a block of lines, where most lines begin with a # except for the ones beginning with _numericsurfarray… and _camera…, These are the different features of Pygame: The ones with the # are disabled, the other two are enabled.

With all the stuff we installed earlier, you can now enable all features (remove the # in front of imageext…, font…, mixer…, mixer_music…, _minericsndarray…, movie…, scrap… and pypm…).

Remember we disabled PORTTIME a while ago? Right, so we have to remove that dependency: In the line starting with pypm…, delete the part that says $(PORTTIME). Great. That was easy, right? Now save that file and go back to the Terminal.

We are now going to compile and install Pygame. The nice thing is, even though we are installing it manually, it will go in the right directories and it will be registered with pip or easy_install, so you can just invoke them if you want to uninstall it later by typing pip uninstall pygame. This is something I love about Python!

Alright, now without further ado, install Pygame by typing

python setup.py install

Great! That's it! Everything should work now!

Tags: compiling python macos
03 Feb 2011

Installing Python/Numpy/Scipy/Matplotlib on OSX

For numerical analysis and signal processing prototyping, you would use Matlab. However, Matlab has some downsides that might make it unsuitable for your project. It might be too expensive. You might be a snobbish programmer that can't stand less-than-elegant programming languages. I certainly am.

So, you look for alternatives. You could take Octave, which is free, but that would not solve that ugly-code issue. You could take any scripting language you fancy, but Ruby, Perl and Python are too slow to do serious number crunching.

Then, you stumble upon that Python package called Numpy, which seems to be nearly as fast as Matlab when it comes to matrix processing and linear algebra. You then discover SciPy, which would add all that signal processing prowess of Matlab (do quick transformations, random numbers, statistics) to your toolbox. Last but not least, you need plotting. That would be Matplotlib then, which provides quick plotting facilities in Python.

And the best thing is, these three systems work really well together. They seem to be the perfect replacement for Matlab that could even be superiour to it in many regards.

Next up, you need to install all that stuff. If you are like me, you naturally want to do all that on a Mac. Also, you kind of dislike all these installer-thingies, which install stuff to unknown places and are nigh impossible to uninstall or update cleanly. Even though, you could of course just go to the individual websites, download Python, Numpy, SciPy and Matplotlib, run them installers, and be done. You would save yourself a lot of trouble that way.

But since you allegedly are like me, you instead fire up brew and try to install all that stuff using that. Again, you could use MacPorts or Fink instead, but you probably had some bad experiences with them and you generally love the hackishness of Homebrew, so this is your natural first try.

So you set about this, you believe in packet managers and trust them to take care of every obstacle that might be lying in your way. First of all, install the latest developer tools from developer.apple.com. You might need to register (for free) to get them. Also, you need to install Homebrew.

To cut this short, here is what you need to get that Python running:

    brew install python

This one should be obvious. At the time of writing, it will install Python 2.7.1. You could take Python 3, but matplotlib is not compatible to it, so you kind of have to stick with 2.7.1 instead.

You also need to put /usr/local/bin and /usr/local/sbin in the beginning of your path to make sure the new Python gets loaded instead of the pre-installed one. You do that by writing

    export PATH=/usr/local/bin:/usr/local/sbin:$PATH

in your \~/.bash_profile. (Create it if its not there–it is just a simple text file).

Now, if you type python --version, you should get Python 2.7.1 as a response.

Alright, next up, install the python package manager:

    brew install distribute
    brew install pip

This will come preconfigured for your newly installed Python. In an ideal world, this should be all. The world being as it is, the pip package of Matplotlib is severely broken and has one other unstated dependency:

    brew install pkg-config

Also, SciPy is using some FORTRAN sources, so you need a fortran compiler:

    brew install gfortran

Alright. That was enough. Now on to pip. With all these dependencies cleared, pip should be able to download Numpy and Scipy without trouble:

    pip install numpy
    pip install scipy

Matplotlib, on the other hand, is more difficult to install. You see, pip is looking at the Python package repository PyPi for each package. PyPi then provides a URL. Pip then scans that website for links to suitable package files. But, Sourceforge changed its links a while ago, so pip gets confused and will download an outdated version. Sourceforge says, its new links are way better and no way we will change them back; Pip says, well, if Sourceforge can't provide proper links, that's not our problem. Oh My. Silly children.

So we have to do this manually:

    pip install -f http://sourceforge.net/projects/matplotlib/files/matplotlib/matplotlib-1.0.1/matplotlib-1.0.1.tar.gz matplotlib

That URL comes straight from Sourceforge. Look for the latest version of Matplotlib, search for the download link to the source distribution (*.tar.gz), copy that link and strip any trailing '/download'.

UPDATE:

It seems the matplotlib package was updated in the meantime, so you can just run pip install matplotlib now.

This should now download and install matplotlib.

Thank you for reading.

Tags: compiling python macos
14 Mar 2009

Get a Mac

osx.png

Bei meinem üblichen, Samstagmorgendlichen, bettlägrigen Web-Rundgang habe ich heute drei sehr nette Artikel von David Alison gefunden, einem Windows-Programmierer und selbsternanntem Microsoft-Fanboy, der sich einen Mac zulegt und unverhofft glücklich damit wird. So glücklich sogar, dass er inzwischen mehr oder weniger ein Mac-only-User geworden ist.

Ziemlich genau wie ihm ging es mir auch.

Wens interessiert: Das ist eine sehr hübsche Einführung darin, warum Macs toll sind. Viel besser als alles, was ich dazu schreiben würde oder bereits getan habe.

Tags: macos
Other posts