Smoke testing APIs… at the type level [1/3]

Posted March 2017

Why smoke testing?

Smoke testing is a great way to quickly sanity-check entire systems and too often something teams don’t spend enough time on. I’ve seen realistic smoke tests catch show-stopper problems in CI / dev systems way before this gets to the wild… despite all unit tests passing.

But we have unit and integration tests…

Yes: especially in distributed systems, developers often (rightly) test their own microservices to death, and yet: bugs are still found by QA (or worse: users) in / nearer Production.

…so why are there still bugs?

  • Misunderstanding the contracts between systems. What was that String ID referring to again? Or no, you have to call this endpoint first…
  • Version compatibility problems: what’s actually deployed in the real world isn’t always master at the time you developed it. Backwards compatibility can be hard.
  • Other tests, or other people, are using the system under test – database tables aren’t empty, non-zero counts in message queues.
  • “Edge cases” in semi-typed content types: JSON’s “missing key” vs null vs "" – are these exactly the same everywhere?
  • Real-world data, with all the missing fields, awkward names etc. can exercise logic branches / “corner cases” not found in traditional testing.

Failure testing

More advanced smoke testing will also add failure testing (though this could live elsewhere too):

  • Network failures
  • Adding latency, out-of-order requests.
  • Naughty-but-realistic ignoring RFC specs (Accept headers, retrying, abusing HTTP caching)

At the extreme end you have Netflix’s Chaos Monkey, part of their Simian Army.

  • Testing entire systems requires an understanding greater than any one part – it’s rare when developers completely understand and have the full context of the systems at work.
  • To be useful it has to grow with the system from infancy – retrofitting it is an uphill battle (but doable).
  • It’s very hard to impose this in your CI systems. Just because an environment (ephemeral or not) is broken, doesn’t mean your commit in your changed component caused it. So it can’t really block the build, instead getting run as a nice to have afterwards… and then… ignored.
  • People get distracted by the green lights in their build & test tooling (NPM / setuptools / Maven / Gradle / Stack etc), or the Ops teams monitoring dashboards… and think that’s enough, look at all that win. These things are important but good smoke testing reduces the over-reliance on expensive, riskier reactive fixing.
  • Sometimes so much effort gets burned on the UI testing systems (Selenium, PhantomJS etc), pre-loading data & stubbing responses, or the style of the testing (BDD vs unit testing etc) that the real goal of the testing is neglected.

How can we make this easier?

More abstractions of course!

Recently I started a smoke-testing project from scratch, so had another chance to approach this problem at a more abstract / greenfield stage. Also - last year at Haskell Exchange 2016 I saw a great talk by Julian Arni.

He discussed using Servant, a Haskell library for defining APIs at the type level, and Quickcheck, the pioneering property testing framework together to fully test JSON/HTTP APIs written in any language using his servant-quickcheck.

Within a couple of days (and with not much Haskell experience) I was able to get this working and verifying a real system (spun up locally with Docker Compose, which is great for these sorts of tasks)

Example: Testing Stack Overflow

Here’s a simple GET example URL from Stack Exchange’s public API:

https://api.stackexchange.com/2.2/tags?site=stackoverflow

This will get some interesting tags around the search term testing. We’ve ignored optional query parameters, thus leaving the defaults.

Define the type-level API in Servant

If you haven’t used Servant before (or even much Haskell), don’t worry too much about the workings or even syntax of this for now. That said, know that:

  • An API is a single Haskell type typically created using combinators and smaller types.
  • The combinator :> roughly speaking combines features of an endpoint (path fragments, HTTP verbs, query parameters, return types etc)
  • …whereas <|> defines alternative endpoints (not used here yet)

Note we don’t have to define the complete data type returned by endpoints – this makes it much less brittle, and concise when testing more complex APIs.

Next up

In the next article, we’ll delve into what that all means, and how to actually get some testing results.

Further Reading