Benchmarking

owensd

If you are going to go out and claim that someone else is lying (see: http://www.splasmata.com/?p=2798), you really should back up your claims with provable and reviewable facts. In the case of benchmarks, you need to provide the following:

  1. The source code you used—this should be common sense. People should be able to peer review your work.
  2. Compiler and compiler flags used; without these, your results are meaningless.
  3. The average and standard deviation of your results.
  4. Test methodology used

This is the bare minimum. The author is missing all but #4.

For all of the following benchmarks, I used the Xcode 6 Beta build with the new XCTest performance feature. Tests were conducted on a Mid 2012 Retina MacBook Pro: 2.7GHz Intel Core i7 and 16 GB RAM. All tests were run under the standard Release mode settings.

So let’s look at the author’s claims:

Loop a million times

Swift: 0.0036s

Objective-C: .0021s (1.7x faster)

Since we have no code, we’ll have write our own.

Swift Timing (all three versions): 0.001s, 11 STDEV (gist)

ObjC Timing: 0.002s, 23% STDEV (gist)

Swift came out ahead here, but not by any statistically significant margin. Also, the author made the following claims:

  1. for _ in 0..999999 was “glacial”. The gist above shows this form to have identical performance to the other forms.
  2. ++ was really slow, yet the numbers say the same thing—identical performance to the other forms.

Seems like we’ve hit the basic problem: compiling with optimizations off—DEBUG builds.

Let’s check out the next one.

Increment

Swift: 0.024s

Objective-C: 0.0023s (10.4x faster)

The author claims that Swift has a performance problem with the ++ operator. Let’s check it out… oh wait, already did above. Yep, no issue in release builds. If we run the same tests under the debug configuration, we can see why the author is making the claims he made… oops!

We’ll take a look at one more:

Append native integer to native array

Swift: 6.51s

Objective-C: 0.023s (283x faster)

Ok, that looks like it hurts… let’s see if it is true.

Swift: 0.271s, 6% STDEV (gist)

ObjC: 0.039s, 38% STDEV (gist)

Looks like ObjC comes out on top here, but not by anywhere near the margins claim by the author.

Moral of the story, it’s best not to call people liars when your “facts” are poorly backed up. Also, benchmarks should always be peer-reviewable.

Swift has lots of potential issues with it, but performance over ObjC does not seem to be one of the big concerns.

Benchmarking