Smoke testing APIs… at the type level [2/3]

Where were we?

In part 1, we covered some of the higher level testing approaches, and introduced a minimal example of a Servant API. In this article, we’ll develop that into one that actually works for our StackExchange tags API use-case.

A few bugs

In the meantime, and as a result of writing the articles, I’d found a couple of bugs (this query string generation problem and a path delimiter problem, if you’re interested). After working with the maintainers those PRs are now merged – and has been a great experience for a newcomer.

So make sure you use servant-quickcheck-0.0.2.4 or above…

Building the code

…which brings us neatly to build tools. I won’t cover the ins and outs of Haskell builds here, but in short you’ll need Stack, with a newish resolver, e.g. lts-8.5.

The library itself isn’t currently on Stackage, plus when using the latest versions of Servant some dependency hell totally manageable versioning issues seem to arise, so a little dependency solving is necessary¹ – or if you’re interested just copy this stack.yaml.

Refining our API

Better textual types

In real-world Haskell, the String type is a little unloved now, in favour of more Unicode-friendly and time/space efficient text data types, so we’ll use the would-be standard Data.Text for our query string parameters, not least because this is used in the HTTP libraries we’re interfacing with.

Wrapping responses

If you look back to our original API definition, it was gloriously simple, but in reality didn’t quite match what StackExchange returns, which is a JSON object of {"items": [...]}.

So let’s define a TagResponse type:

data TagResponse = TagResponse {
  items :: [Tag]
} deriving (Eq, Show, Read, Generic, FromJSON)

data Tag = Tag {
  name :: Text,
  count :: Int,
} deriving (Eq, Show, Read, Generic, FromJSON)

Essentially the response is a JSON list of objects each of which has (at least) a name and a count (we’ve dropped the unused Bool for further brevity).

Note here we are deriving “the usuals” (Eq, Show, Read). For those not familiar with Haskell, think of this as a little like implementing interfaces, but without even needing to write any code. By using Aeson’s FromJSON and the magic of DeriveGeneric, we get free deserialisation from JSON too. Pretty clever stuff, but we’ll put that aside for now.

Stronger Parameter types

Being fans of strong and meaningful typing, we can encapsulate our domain better in the type system, by creating new wrapper types (at no performance cost) using… newtype; this will also make for neater Arbitrary generation – but more on this later.

-- Site
newtype SiteName = SiteName Text
instance ToHttpApiData SiteName
  where toUrlPiece (SiteName ss) = ss

-- Search Terms
newtype SearchTerm = SearchTerm Text
instance ToHttpApiData SearchTerm
  where toUrlPiece (SearchTerm st) = st

Note that by implementing ToHttpApiData (just by unwrapping) we can use these in generation of URLs.

If you’re really on the ball you may have noticed we could have used the newtype trick for our TagResponse as it features a constructor with just one field (HLint suggests this in fact). We’ll leave it for now - as the API actually returns more metadata that we might want to model, so we’d have to go back to data then anyway.

The Updated API

It’s also more readable than before now:

type StackExchangeAPI =
    "tags"
        :> QueryParam "site" SiteName
        :> QueryParam "inname" SearchTerm
        :> Get '[JSON] TagResponse

Some BDD tests

Using Hspec, a popular Haskell testing framework lovingly derived from Ruby’s RSpec, we can use behavioural-style tests in plain(ish) English directly in our code; this is possible largely due to Haskell’s friendliness for DSLs, if you’re prepared to overlook the odd do and $, and various combinators in our case.

The core test (Expectation in HSpec) in servant-quickcheck is serverSatisifies which as the name implies make sure that a server satisfies a predicate (or combination thereof). Being also the interface with QuickCheck, this allows you to test properties of a server hold true, across all sorts of actual values. It’s worth stressing this if you’re used to unit testing, which is more akin to examples of things that should work. It takes a few parameters – a BaseUrl, and QuickCheck config and of course, an API (proxy) to test, which is trivial to instantiate:

api :: Proxy StackExchangeAPI
api = Proxy

The server property test spec

The advice is to chain these properties (as Monoid-like Predicates instances operating on the server request and response) using the provided <%> operator, and a mempty instance. We’re going to use some generic predicates provided with the library:

spec = describe "The Stack Overflow API" $
    it "returns JSON objects without error" $
      serverSatisfies soApi baseUrl stdArgs
        (not500
         <%> onlyJsonObjects
         <%> mempty)

Fairly readable, I reckon. We’re asserting here that not500 and onlyJsonObjects hold true across all our tested values. What do these predicates do? Well, not500 asserts the server doesn’t return an error response, and onlyJsonObjects makes sure that any JSON response is an object e.g. {"numbers": [1,2,3]} rather than a raw array or value type – something most people recommend for APIs.

Generating values with QuickCheck

In order for QuickCheck to test properties across thousands of variants, it needs to be able to generate some – this is modelled by the Arbitrary typeclass. If you use standard datatypes (Int, String etc), you’ll get these for free.

For our custom types, we’ll need to define instances for the typeclass, defining the one method, arbitrary, that returns a Gen Monad instance.

Now SiteName should be pretty easy – it just needs to be hard-coded to stackoverflow for now:

instance Arbitrary SiteName
  where arbitrary = return $ SiteName "stackoverflow"

Note we have to wrap in our Gen Monad with return.

For SearchTerm, we’ll start with a very constrained arbitrary generation – just pick from one of a list of predefined values, using the oneof function:

instance Arbitrary SearchTerm
  where arbitrary = do
          term <- elements ["haskell", "quickcheck", "servant"]
          return (SearchTerm term)

The code in full

I’ve put it up on GitHub if you want to see in full. The main bits:

• The API definition
• The Test

Putting it together

Start a test HTTP server

As we want to see what’s happening, we’ll test against a local dummy server for now. Simple is better so normally I’d use python -m SimpleHTTPServer 8080, but here we’d prefer something that serves JSON here by default, so instead let’s use the popular Node module json-server (assuming you have npm installed).

So to install and set up some mock data:

$ npm i -g json-server
...
$ echo '{"tags": {"items": ["foo", "bar"]}}' > tags.json

and to serve it’s as simple as:

$ json-server -p 8080 tags.json

Running our tests

➜  servant-quickcheck-sandbox git:(master) stack test
servant-quickcheck-sandbox-0.1.1.0: test (suite: servant-quickcheck-sandbox-test)
Progress: 1/2
StackExchange
  The Stack Exchange API
    returns JSON objects without error

Finished in 0.0957 seconds
1 example, 0 failures

Meanwhile we should see appropriate requests successfully hitting our local server:

...
GET /tags?site=stackoverflow&inname=quickcheck 200 0.187 ms - 41
GET /tags?site=stackoverflow&inname=haskell 200 0.357 ms - 41
GET /tags?site=stackoverflow&inname=servant 200 0.344 ms - 41
GET /tags?site=stackoverflow&inname=servant 200 0.244 ms - 41
GET /tags?site=stackoverflow&inname=servant 200 0.262 ms - 41
GET /tags?site=stackoverflow&inname=quickcheck 200 0.276 ms - 41
GET /tags?site=stackoverflow&inname=haskell 200 0.216 ms - 41

Note how QuickCheck is repeatedly choosing randomly across the (few) values defined for inname – and each one having properties verified (i.e. it’s returning a JSON object and not returning HTTP 500). When we generalise this further next time, the benefits should become clearer.

Hopefully this is a hint at the power available combining property testing with type-level APIs – API style-checking / RFC compliance validation across entire APIs, in just a few lines of code!

Next Up

In the final part, we’ll extend the arbitrary generation, add some new predicates, and discuss a few ideas of where to go next.

¹ another reason I love Stack – Maven, Gradle, pip, npm – take note!