Parsing Package.swift

As I’m working on my Swift language server, one of the things I need to do is to parse the Package.swift file. There is a PackageDescription library, but that’s unable for use within your own program if you are actually using SwiftPM.


So what are we to do? HACK IT UP!

Basically, we need to run this command:

$ swift -I /Library/Developer/Toolchains/swift-latest.xctoolchain/usr/lib/swift/pm \
    -L /Library/Developer/Toolchains/swift-latest.xctoolchain/usr/lib/swift/pm \
    -lPackageDescription Package.swift -fileno 1

Not really a big deal, just annoying.

Here’s the snippet to get the output:

Curious note: when you use string literal interpolation, even if your variable is an implicitly unwrapped optional, it will get output as an optional. Hence you see: "\(projectPath!)/package.swift". Go figure.

Of course, the output is JSON. If you need a parser, you can use json-swift or any of the many others.


The shell function is just a helper that I add to my SwiftFixes.swift file:

Anyhow, if you too need to parse that pesky Package.swift file, here you go.

Parsing Package.swift

vscode-json swift v2.0 released

I just published version 2.0 of my JSON parser library for Swift: json-swift. Version 2 brings in a lot of changes, mostly around RFC-7159 compliance and performance. As of this writing, I am not aware of any issues with the compliance tests.

As for performance, well, it’s gotten markedly faster! One of the test files I have is large-dict.json. It’s about 26 megs and contains lot of various data in it. This took a really long time to parse, I believe around 14s. However, now we’re looking pretty good!

performance results: min: 0.427, max: 0.484, avg: 0.446

performance results: min: 0.966, max: 1.060, avg: 0.985

Freddy Results:
performance results: min: 0.875, max: 0.920, avg: 0.897


I put the comparison with Freddy in there too for good measure. There’s still a bit more I think can squeeze out, but there is a lot of retain/release overhead too that I need to figure out before I’ll be able to get down to beating NSJSONSerialization.

There are some breaking changes in this release as well, namely if you used the error and hasValue properties. Those are now gone as the entire API surface has been changed to use throws as the mechanism to promote errors. This cleaned up the code quite a bit and unified some of the concepts better.

This means that code like this: json["hello"][123]["would"] would normally require ? at each index. However, I provided Optional<JSValue> wrappers for all of those. This wasn’t possible in the early versions of Swift, so it was nice to clean up this design workaround.

Happy JSON parsing!

P.S. When Swift 4 becomes a bit more stabilized, I’ll be providing the Encoder and Decoder implementations to work with the new Codable protocol.

vscode-json swift v2.0 released

Looking at my JSON Parsing Performance

I’ve been working on my JSON parser lately in the hopes to fix two major issues:

  1. Correctness
  2. Performance

A lot of the design of the parser was influenced by early Swift limitations, so I was able to go through and get rid of a bunch of the weird boxing and internal backing stores I needed to use back then. Sadly, removing those didn’t really help performance much.

However, there is one piece that is hit by pretty much every part of the parser all of the time: my ReplayableGenerator type. The idea of this was that I’d simply call next() and replay() in the parsing code to remove the a lot of that logic out.

The current implantation requires a Sequence. This is fine except that the way I had things setup, I needed to turn the string into an array of UInt8. It turns out, that is relatively expensive. Even when creating the generator by using string.utf8 and using that iterator directly, performance was still 10x worse than JSONSerialization.


All was not lost though! Instead of using a Sequence.Iterator to back my ReplayableGenerator, I figured I’d just straight up use an UnsafeBufferPointer<UInt8>.


performance results: min: 0.0126, max: 0.0215, avg: 0.014

performance results: min: 0.0364, max: 0.050, avg: 0.0392

Yay! Getting there. There is still more work to be done and some correctness issues to work out, but getting happier with things now.

Just one more quick thing to note: one of the biggest perf gains was changing how I was getting the string content.

One thing I maybe should have tried, but forgot to, was getting a lazy string.utf8 back. That might have made some difference.

Looking at my JSON Parsing Performance

Making Mistakes: print()

I’m implementing a Swift version of the Language Server Protocol. The way that it integrates within Visual Studio Code (VS Code) is via stdin and stdout. That’s all fine and dandy. It also makes uses of a modified JSON-RPC message construct for its communication.

While testing out my server’s ability to handle commands coming in from stdin, I was simply using print() to output the response message.  Anyhow, input a message, and the output was working great.

However, when I went to test it within VS Code, I would get the initialize request, send a message back, and nothing. What I expected to have happen was for VS Code to start sending me more messages.

So what was the problem? I honestly had no idea.

Problem #1: From the LSP spec, it isn’t immediately obvious what the response messages should look like. Should it include the message header? Should it just have the JSON-RPC part? Is my message even formatted correctly? The spec calls for \r\n instead of \n, did I mess that up?

I go through and validate the message and output in all of the different permutations I can think of, but nothing. After spending some time digging around other LSP implementations, I come to the conclusion that I am indeed sending back the right message format, so what could it be?

Problem #2: Esoteric history and undocumented (or implied) behavior.

Ok, so if the message format is correct, maybe the output isn’t actually working as it looks like it is. So I run mkfifo output, mkfifo input, and tail output. Let’s see what is happening.

Running cat initialize.lsp > input (my saved message content for an initialize request) gets my language server to handle the message, but no output.


It turns out that Swift’s print() simply routes to the underlying stdio output. Which, if you don’t know, does buffered or unbuffered output depending on what its actually being output to. In the case of the console, it’s output immediately. In the case of a file descriptor, it’s buffered.

Solution (temporary):

setbuf(stdout, nil)

It’s temporary because I actually need to write the proper version of the output code to ensure that I’m writing the content correct and only the number of bytes that are specified in the response message.


Here’s the thing, I actually knew about how stdio buffers it’s output. However, when I looked at the print() documentation, I simply became complacent and assumed since it didn’t mention buffering that it indeed immediately wrote the content out. Later testing, of course, would prove otherwise.


The problem here is somewhat systemic of our programming culture. It came from a combination of unclear documentation (from two sources, nonetheless), and an assumption of knowledge that, even if the person knows, can forget to apply in certain contexts.

Hopefully this radar gets fixed:

Making Mistakes: print()

travis-ci + swift + os/linux config

Since I was having such a hard time getting this working, I figure I’ll put this up here for everyone else too.

I have a project that I want to build for both Linux and OS X using Swift on

First you’ll need a proper .travis.yml file:

This is what is working for me. The super important part is the disttrusty and sudo: required. Without those for the linux build, you won’t get Ubuntu 14.04 and thus won’t be able to run the Swift compiler.

The script looks like this:

It’s not a perfect setup, but it was enough to get me up and running. Hopefully it’s helpful to others in the same boat as me.

travis-ci + swift + os/linux config

Build Times

I’m finally getting back to fixing up a few things in my json-swift project, but this makes me sad:

9:53:02 › time swift build
Compile Swift Module 'JSONLib' (10 sources)
swift build 3.69s user 1.03s system 120% cpu 3.918 total

9:53:12 › wc -l Sources/JSONLib/*.swift
 55 Sources/JSONLib/Error.swift
 56 Sources/JSONLib/Functional.swift
 89 Sources/JSONLib/JSValue.Accessors.swift
 38 Sources/JSONLib/JSValue.Encodings.swift
 50 Sources/JSONLib/JSValue.ErrorHandling.swift
 68 Sources/JSONLib/JSValue.Indexers.swift
 74 Sources/JSONLib/JSValue.Literals.swift
 671 Sources/JSONLib/JSValue.Parsing.swift
 271 Sources/JSONLib/JSValue.swift
 67 Sources/JSONLib/ReplayableGenerator.swift
 1439 total

While it may not seem like 3.69s is not a long time… add in compiling and running the unit tests and we are at 7.66s. This is a trivially sized project and the pain of fast iteration loops stinks here.

Build Times

Bad APIs

I write a lot of C++ code at my day job, so often I don’t want to do that for my hobby projects. There’s a lot of complexity in C++, no doubt. However, I think many of troubles of C++ are simply bad API designs.

As just an example, let’s look at the std::map API. I mean, really?

OK… so there are some things we know about a map, but the most important is that the key values are unique. That is, if “name” exists, there will be one and only one of them.

So what the flip is up with the iterator stuff? And why on earth is there a count() function instead of just a contains() function?

I’m sure there is some explanation for it, but I’m not terribly interested in it. As a consumer of the API, I don’t really care why you make my life more challenging. I want to work with nice, clean APIs.

Swift vs. C++

Part of what make Swift look more appealing is the clean syntax and sometimes better (though, there are many examples where this isn’t the case today) API interfaces. Let’s take a look at a snippet of a Vapor project I’m toying around with:

That’s some pretty reasonable code. Maybe I’d change a few things, but there’s nothing that makes me go, “what the heck is wrong with you?”

Compare that to what it might look like in straight up C++:

I mean… it’s not that terrible, but there is a lot of WTF? in there. The biggest problem is that these APIs cause the code to really switch context from the get() handling to the details of parsing out data. That’s not cool.

Better APIs

We can fix this though and create something that is much more easier to follow and use:

This would be the same exact number of lines of code as the Swift version if I change the Swift version to not throw on an error trying to get the parameter. But more importantly, we didn’t lose anything in the C++ version. However, we did gain code that is much easier to read and maintain.

Build better APIs, your code and coworkers will thank you for it.

Bad APIs