09 Jul 2019

# Publish or Perish

As part of my PhD, I am supposed to publish three papers. So far, I have been unable to do so. But this is not about me, I will survive regardless. This is about the systems behind our papers' rejections. Because they are… bad. Political. Un-scientific.

Our first manuscript was submitted for publications, and got a middling review. If we wanted our work to be published, we were to expand on our introduction to mention the reviewers' favorite publications, and broaden our comparison to include their work. This is considered normal. In the second round of reviews we then got rejected, because our introduction was now too long, and our comparison too broad.

The reviews additionally claimed that "novelty cannot be claimed for something that is not validated and compared to state of the art" and that "[our work lacks] formal statistical evaluation of the estimation performance". Which is certainly true, but is also true of every other work published on the same topic in the last five years (I checked). We showed this evidence to the reviewers, but it was not even deemed worthy of a comment.

In hindsight however, we realized that we had included at least one reviewer's own algorithms in our comparison, and found it lacking. Their work had only ever been tested, publicly, with a single example recording, where it worked well. Our comparison did the same with twenty thousand recordings, which highlighted some issues. So our paper was rejected. Of course we can't be sure that this was ultimately true, as the reviewers' names are not disclosed to the reviewees (but certainly vice-versa).

Our next submission was to a different journal. This time, we had learned from our mistakes, and kept the scope of our investigation more minimal. There would be only a very small comparison, and we would be very careful not to step on anyone else's toes. The review was, again, negative.

This time, the grounds for rejection were lack of comparison to state of the art (not a winning move, see above), and our too high false negative rate. Additionally, it contained wonderful verbiage like:

The are many methods that are very similar to the presented method in the sense of being feature extraction methods operating in the STFT domain.

…which is just patently ridiculous. If being a "feature extraction method in the STFT domain" was grounds for rejection, there would be no publications in our area of research. And let's ignore for a minute that our publication was not, in fact, such a method.

Again, hindsight showed the real culprit: Our manuscript reported a high false negative rate of roughly 50%. Had we just not mentioned this, no one would have noticed. That is what everyone else is doing. More importantly however, reporting on false positive/negative rates in our evaluation called into question every other publication that hadn't. And we can't have that.

Another submission was liked because no one had done anything similar before, and was found to provide value to many researchers, but rejected because it still somehow "lacked novelty".

So, in summary, our first submission was rejected because it made one of the reviewers look bad, and the second because we not only wanted to report on our method's advantages, but also its shortcomings. Worse, in following the evidence where it lead, we had created new error measures that could potentially find flaws in existing publications, which could potentially make a whole lot of researchers look bad.

After five years of dealing with this, I am thoroughly disheartened. Instead of a system for disseminating knowledge, I have found the scientific publishing system a political outlet for the famous, and a lever for keeping uncomfortable knowledge suppressed. This is not the scientific world I want to live in, and apparently, it doesn't want me to live in it, either.

11 May 2019

# Fuji Zoom Lenses

So I bought a new camera. Now I need new lenses. In this post, I am looking for a standard zoom lens, i.e. something that covers a bit of wide-angle, all the way through the normal range, up to a bit of telephoto. In Fuji's lineup these needs are met by

• the XC 15‑45 mm f/3.5‑5.6 OIS PZ (€ 150, 136 g, 4.4 cm)
• the XC 16‑50 mm f/3.5‑5.6 OIS II (€ 150, 195 g, 6.5 cm)
• the XF 18‑55 mm f/2.8‑4 R LM OIS (€ 250, 310 g, 7.0 cm)
• the XF 18‑135 mm f/3.5‑5.6 R LM OIS WR (€ 500, 490g, 9.8 cm)
• the XF 16‑55 mm f/2.8 R LM WR (€ 650, 655 g, 10.6 cm)
• a bag of primes (€ inf, many g, lots of cm)
• the XF 27 mm f/2.8 (€ 150, 78 g, 2.3 cm)

The XC 15‑45 is out, because the zoom ring is not turn-to-zoom like any other zoom lens. I tried it; I couldn't stand it. The XF 16‑55 is out because it is just too expensive and big for me. A bag of primes is not what I want, but I included the XF 27, because that's what I happened to have at hand. All of the above prices are used prices as of early 2019 in Germany.

Before we get started, all of these lenses are perfectly sharp to my eyes. At least in the center-ish area of the image, every pixel shows different information, which is not something I could have said about some of the Nikon lenses I used to own. Because of that, I will not compare sharpness.

The word of mouth is that the XF 18‑55 is a stellar kit lens, the XC 16‑50 is a bit cheap, and the XF 18‑135 a bit of a compromise. If internet forums are to be believed, these differences are massive, and the XF 18‑55 is really the only acceptable non-prime lens any self-respecting Fuji fanboy can buy. But then again, that's what internet forums would say, right?

With that out of the way, let's have a look at these lenses! I'll use crops from a terribly boring shot linked here for most of my examples. Why this shot? Because a), that's what was available, and b), because it contains areas that nicely showcase these lenses' qualities. All shots were taken at f/8 at 27 mm, ISO 400 and a shutter speed around 1/300 s.

## Micro Contrast

I often read that micro contrast really tells lenses apart. To illustrate this point, here's an area with very little overall contrast, particularly between the fence and the orange paint, and the fence and the gray stairs:

If you look very closely, you might find the fence slightly less visible on the XC 16‑50 than on the XF 18‑135 and XF 27, and ever so slightly more visible on the 18‑55. But it should also be obvious that these differences are incredibly tiny, and not worth fussing over.

## Corner Sharpness and Chromatic Aberrations

Another common point is corner sharpness, which is typically said to strongly favor primes. This time, the images are cropped from the bottom-right corner of the image that contains some detail, but most importantly a bright white warning sign:

And indeed, the XC 16‑50 is noticeably blurry this time, with the other three lenses similarly sharp. The warning sign also highlights color fringes on the transitions from the bright white sign to the dark background. These chromatic aberrations are almost invisible on the XF 18‑55, and mild on the XF 18‑135 and XF 27.

Bear in mind, however, that these are 100% crops in the very furthest corners of a high-contrast image. In normal pictures, none of these issues will be noticeable unless you really zoom in on fine details at the edges of your frame. The chromatic aberrations seen here were already treated in software with the lens correction module in Darktable, but it might be possible to improve on these results with more dedicated processing.

## Update 1: Even more comparison pictures

After publishing the blog post, I still wasn't satisfied: What if the results I got were only true at f/8? What if image quality got worse at a longer focal length? How does my XF 18 stack up?

To answer this, I took another set of pictures and prepared another composite of an crops near the image center, and another one near the lower right corner.

To be perfectly honest, I can not see any significant differences between any of these pictures. At this point, I am starting to question the entire concept of sharpness and micro contrast for evaluating lenses. But at least I learned a lot about how to use the Gimp.

As a sanity check, I repeated the experiment with my old Nikon 18-200, and this was in fact noticeably less sharp. And slightly overexposed. And slightly off-color. That's why I switched to Fuji. But as I said, this was a sanity check, not a fair comparison, as the Nikon D7000 body is much older than my Fuji X-E3, and the lens has surely seen better days as well.

## Update 1: Ergonomics and Balance

The XC 16‑50 and XF 27 operate their aperture with the control wheel on your right thumb. The XC 18, XF 18‑55, and XF 18‑135 have a dedicated aperture ring on the lens instead. Thus, the former two lenses can be controlled with the right hand alone, while the latter two require the left hand on the lens barrel. Zoom is always controlled on the barrel, though.

This preference for one-handed or two-handed operation is supported by the lenses' weight, as well: My camera, the X-E3, weighs about 330 g. With the 200 g XC 16‑50, the weight is mostly in the camera body, and can easily be held and operated with one hand. The 250 g XF 18‑55 and the 500 g XF 18‑135 are more lens-heavy, which makes a two-handed grip necessary, which is a better fit for the aperture ring on the lens.

Personally, I actually prefer the aperture on the thumb wheel over the unmarked aperture rings on the XF 18‑55 and the XF 18‑135. It just feels more natural in my hands. On the other hand, I like the marked aperture ring on the XF 18, particularly for resetting the aperture without looking through the viewfinder, or when the camera is turned off. In fact, I find the ability to operate the camera while turned off to be very useful in general. It is one of the major reasons why I like Fuji cameras.

## Update 2: Image Stabilization

In order to assess the image stabilization systems built into these lenses, I took a series pictures of a static subject at 18 mm, 27 mm, and 50 mm, for shutter speeds of 1/30 s, 1/15 s, 1/8 s, 1/4 s, and 1/2 s. I then looked at five images for every combination of lens, focal length, and shutter speed, and labeled them either sharp if there was no visible blur at all, or usable if there was micro-shake only visible at 100 %, or miss if the shot was too blurry.

The XC 16‑50 had perfect sharpness at 1/30 s, was at least ok between 1/15 s and 1/8 s, and even 1/4 s still had a few usable shots. 1/2 s or longer was unusable. There was no significant difference between the focal lengths. That last bit is really interesting, as I would have expected shorter focal lengths to be easier to hand-hold than longer ones.

The XF 18‑55 stayed perfectly sharp one stop longer until 1/15 s, but otherwise performed exactly the same as the XC 16‑50. I would guess that the small difference in stability between these two lenses is mostly due to their weight difference, but that the image stabilization system is identical.

The XF 18‑135, however, was another matter: All shots up until 1/8 s were perfectly sharp, and remained at least usable until 1/2 s! Only at 1 s of shutter speed did I see significant numbers of missed shots! Again, there was no significant difference across focal lengths.

With disabled image stabilization, I could hand-hold most shots for at most 1/focal length, but missed or fudged a few shots even there.

In summary, I found the XC 16‑50 and XF 18‑55 image stabilization good for about two stops, and astonishingly, the XF 18‑135 stable for a full four stops over my personal hand-holding skills. Some of that stability is no doubt due to the increased weight of the XF 18‑135, but nevertheless, I find these results astonishing!

## Close Focus Distance and Magnification

And now, the darling of all photographers: out-of-focus backgrounds. Common wisdom is that the bigger the aperture, the more the background is thrown out of focus. But that's only part of the truth, and honestly, not the most interesting part for these kinds of limited-aperture lenses. Much more powerful is getting closer to your subject: The closer you focus, and the farther away your background, the more the background will be out of focus. This effect gets even stronger when you zoom in.

The XC 16‑50 focuses much more closely than any other lens in this list, at 12 and 30 cm (Fuji says 15 cm). You can get really nice background separation with this lens, and great magnification in your macro shots. The XF 18‑55 focuses at 25 and 35 cm (Fuji: 40 cm), which is not particularly impressive. The XF 18‑135 focuses even farther, at 33 and 43 cm (Fuji: 45 cm), but gains magnification through its long tele zoom. The XF 27 is not optimized for this kind of thing at all, at 29 cm (Fuji: 34 cm).

## Conclusions

To me, the XC 16‑50 is the winner for a small/light zoom kit. It might be the least great option optically, but the differences are not dramatic at all, and it is the cheapest, smallest, and lightest lens with the most useful wide end and the closest focusing. But it lacks a dedicated aperture ring and is a plastic construction instead of a metal one, which does detract from the haptic joy somewhat.

The XF 18‑55 is optically the strongest lens. It might even beat the XF 27 prime lens on its own turf! But the optical differences to the cheaper XC 16‑50 and the more versatile XF 18‑135 are quite small, and are not be worth the price/weight/inconvenience to me.

The XF 18‑135 is really surprisingly good. The much longer focal range necessarily comes with compromises in optical quality and bulk, but it seems no significant corners where cut in this case. And the image stabilization is a significant step above the other two lenses. Considering that this lens usually replaces at least two other lenses, I even find the price reasonable. This is my first choice as a do-everything zoom kit.

The XF 27 is not very strong in any particular way, except size. And that size trumps all. If I just want to throw a camera in my bag without any particular photographic intentions, the XF 27 is my first choice. And possibly the XF 18, if I still have room in my bag.

As some small buying advice, the XC 16‑50 was refreshed in 2015 with the OIS II version, which introduced that nice close focusing distance (highly recommended). The XF 18‑135 was apparently built in two batches, the original made in China version that seemed to have horrible QA issues, and a second made in Philippines version in 2017 without.

## What I didn't mention

Aperture. The XF 18‑55 and XF 27 have a wider maximum aperture than the XC 16‑50 or XF 18‑135, by about two thirds of a stop. Shooting at bigger apertures makes brighter pictures with stronger background blur, and some loss in sharpness. I don't find the optical performance wide-open particularly interesting, because most of the time I'd use large apertures to blur the background, making sharpness and distortion mostly irrelevant. And as I said above, getting closer is usually more effective for background blur than maximum aperture, anyway.

Image stabilization. The three zooms offer optical image stabilization systems. From what I can tell, the XF 18‑135 is significantly more effective in this regard than the XC 16‑50 or the XF 18‑55. Hand-held shots with up to about 1/10th of a second seem easily achievable with the XF 18‑135, whereas the unstabilized XF 27 becomes blurry at 1/40th. Videos are noticeably smoother with the XF 18‑135 as well.

Weather sealing. The XF 18‑135 is weather sealed, the other lenses are not. My camera is not, so I don't care.

Distortion and Vignetting. Is fixed in post. No need obsessing over it.

Autofocus speed. Is good. No need obsessing over it.

25 Mar 2019

# On camera sensor sizes

A common internet wisdom about photography is that bigger camera sensors capture more light. So if you want to work in low light, you need a full frame camera, and a bigger sensor always produces better image quality. I have struggled with this a lot. It just doesn't make sense: Lenses can focus light on any surface, so why should the surface size matter?

The answer turns out to be… disappointing. Big-sensor cameras allow for larger (practical) apertures, and lower base ISO. But less noise for the same picture is simply impossible with the same sensor technology. Because that's not how physics works. Let me explain.

At this point, I had previously launched into a long-winded explanation on equivalence, and reached a slightly misguided conclusion. So here's a better one:

It is possible to build equivalent lenses for differently sized sensors, which capture the same amount of light, with the same depth of field, and the same field of view, but simply project this light on a differently sized sensor. Or you can use a "speed booster", which adapts a "bigger" lens to a "smaller" sensor. Both of these will give you identical images, and identical amounts of noise (if the same sensor technology is used).

As an example, a micro four thirds 23mm f/1.4 lens produces identical images to an APS-C 35mm f/2 lens or a full frame 50mm f/2.8 lens. You might notice that $$\frac{23}{1.4} = \frac{35}{2} = \frac{50}{2.8}$$, which means each of these lenses will have the same physical aperture size, and therefore admit the same amount of light. And be the same size and weight. Since the smaller sensor collects the same light on a smaller area, the image will be brighter, and will need to use a lower ISO for the same exposure. And since lower ISO produces less noise, the "one stop advantage" of bigger sensors is bunk, if equivalent lenses are used.

The one remaining difference between sensor sizes is that bigger photosites carry more charge, which means more dynamic range at base ISO on a bigger sensor. This is only a factor if you indeed shoot at base ISO.

But, that's not the interesting part. Camera manufacturers cleverly designed their product lines such that small-sensor cameras are physically smaller, and get smaller lenses, while large-sensor cameras are bigger, with bigger lenses. Thus, if you want to get access to the most fancy glass, you will have to use a bigger, more expensive camera body as well. And inversely, if you want to use the most compact glass, it will only be available on smaller, less expensive camera bodies. And that's not pure marketing, either, as bigger lenses require bigger bodies to hold them comfortably, while smaller lenses balance better on a smaller body.

So in the end, it comes down to a compromize: If you want/need the best glass, it will only be available on the bigger bodies with the bigger sensors. If, on the other hand, you want to go small and light, you will need to sacrifice big apertures, but get access to smaller bodies with smaller sensors. But it's really all about the glass, not the sensor sizes.

13 Mar 2019

Normally, when you take a picture of something too bright, you get bloom: An all-consuming brightness that plunges everything around it into pure whiteness. Ugly.

But if the light source is reeeally tiny, and your aperture is teeeensy as well, you get something else: sunstars

This particular sunstar has fourteen corners, and therefore comes from a seven-bladed aperture (in my Fuji XC 16-50). It happens because tiny apertures are not perfectly circular any longer, but instead, in my case, septagonal, and therefore bloom more in some directions than in others. The effect is kind of beautiful.

In this picture, the sun was just barely peeking into the edge between the tree and the building, and my aperture was set to its smallest setting, f22. I actually wanted to capture the raindrops on the branches, which I largely failed at. In the end, the picture didn't turn out very pretty, but at least I got some fine sunstars!

24 Feb 2019

# What I learned about Amateur Photography

I am an amateur, as in "lover of", photography. I love cameras as tactile devices, I love how photography makes me consider the world as art, how that little viewfinder can reveal unknown beauty in well-known places or people. And I love looking at my photos, and remembering vacations and meaningful moments. For me, photography is about finding beauty, and capturing memories.

However, most of the writing on photography seems to be focused not on my needs, but the needs of professional photographers: A super competitive field of visual artists who compete on image quality and novelty, and use crazy and expensive gear. I have found many of their lessons not applicable to my amateur needs, or even actively detrimental:

## Embrace the noise

Many pros limit their ISO numbers, because high-ISO noise is ugly. Which it is. But you know what is even worse for an amateur? Not having that picture of my baby, because it was too dark.

So I set my ISO to unlimited, reduce my shutter speed and aperture so I actually have a chance of capturing my fast-moving toddler. And embrace the ensuing noise. Some of my favorite pictures look unbearably noisy on my 4k screen, but look just fine when printed, or on a smartphone (the two most important mediums in the world). Because of this, I find noise reduction rarely worth the effort. Color heavy noise reduction works ok, but anything else looks worse than the problem. I vastly prefer a sharp, noisy shot to a mushy denoised shot with no detail.

## Step it down

Another common Pro argument: Wider apertures are better. Which they are, at capturing light, and blurring the background. But as an amateur, a wide-aperture super-shallow depth of field just makes me miss shots. At f1.8, the area in focus is barely a few centimetres deep. I missed too many shots because I accidentally focused on the nose instead of the eye. So, in the absence of studio lighting, and arbitrarily many retries, I prefer to step it down and live with the noise, if need be.

As a fun corollary, all those fancy prime lenses with crazy-wide apertures, they are simply wasted on me. Anything beyond, say, f2.8, is not something I need to spend money on. Also, lenses are noticeably sharper when stepped down! I have been disappointed with the sharpness of a number of shots because I forgot to step it down. Nowadays, I typically shoot at f5.6 or f8, and only go wider if I actually need to, because of lack of light, or if I specifically want a blurry background.

## Wide-ish lenses are easier

I wondered, for a long time, why pros seem to like long-ish lenses for portraits: The answer is, because longer lenses have a shallower the depth of field (for the same f-number), and pro photographers love their blurry backgrounds. But as I said before, that is not for me.

Instead, I prefer wide-ish lenses. If something is too small on a wide-ish lens, it is usually no problem to get a bit closer or to crop afterwards. If something is too big on a long lens though, backing off is often not possible, and you miss your shot1. Plus, wide-ish angles don't blur as much from shaky hands, and are more compact. They often focus more closely, too, which is a huge bonus if you want to take pictures of a toddler.

I have tried, unsuccessfully, a 50mm and 35mm prime (APS-C). Now I own a 27mm pancake prime, which I find perfect: long enough to get nice portraits, but still wide enough to capture a landscape.

Be careful with super-wide angle lenses, though. Even though they are a ton of fun, I have found anything below 16mm to be very difficult to use effectively. It's just too easy to get that fisheye-distorted look, especially near the sides. That distorted look, by the way, is caused by being too close, not by lens distortion. The same thing happens if you take a longer lens or a cell phone and get too close. Just don't do that.

## Gear

Pros use the biggest sensor they can get, to get the best image quality possible. But that also makes everything else much more cumbersome: Bigger sensors mean bigger and heavier bodies. And bigger and heavier lenses. And shallower depth of field (see above). And smaller focal ranges, hence more lens changing. And, not least of all, much, much, much higher prices. It is not for me.

To some extent, the same goes for different quality levels: Personally, I have found entry-level APS-C mirrorless interchangeable lens cameras a good compromise. These entry-level plastic lenses and cameras are usually smaller, lighter, and cheaper than their higher-end brethren, but compromise on robustness and aperture sizes (e.g. Fuji X-E2 375g/€250 + Fuji XC 16-50mm, 195g/€150 vs. Fuji X-T1 450g/€250 + Fuji XF 18-55mm, 300g/€250 vs. my old Nikon gear). And for my everyday camera, I' take a smaller, pocketable camera over a "better", bigger one any day.

Haptics are important, too. I have seen great cameras and lenses that just didn't feel good in my hand. Which meant I wouldn't ever take them with me, and wouldn't take any pictures with them. I now go try stuff in the store before I buy anything. This has talked me out of a number of unnecessary purchases, internet consensus notwithstanding.

And finally, I buy used gear. Cameras and lenses depreciate about 70% within the first two years, without losing any quality. A great camera from two years ago is still a great camera, but costs a third of the original price, and can be resold without loss. And is better for the environment. Win-win-win.

## TL;DR

I have found many "common" rules about photography useless for my amateur needs. I have found cheap, plastic, used gear more useful than pro gear. I have found noisy, small-aperture pictures to be better at capturing important memories than clean, "professional" ones. I have found haptics, size and weight to be much more important than ultimate image quality (within reason).

The funny thing is, you don't find this kind of information on the internet, since most review websites seem to focus on the professional viewpoint, even for gear that is clearly meant for amateurs like me.

## Footnotes:

1

You need to back off much farther on a long lens than you have to move closer on a wide lens.