Smoke testing APIs… at the type level [3/3]

Posted May 2017

Where were we?

In part 2, we developed our test API to model the real responses, and started testing an API against a stubbed service. In this part, we’ll:

  • add a new endpoint
  • build an API client tool
  • extend the arbitrary generation (of QuickCheck types)
  • add some new predicates

…and discuss a few ideas of where to go next.

Recap: our original API

type StackExchangeAPI =
        :> QueryParam "site" SiteName
        :> QueryParam "inname" SearchTerm
        :> Get '[JSON] TagResponse

Level Up

That’s been great, but feels more like an endpoint than an API, so let’s add another one – the Questions endpoint, which behaves in a similar way to our tags one. Servant allows an idea of composing endpoint descriptions with its own magic combinator, :<|>. Haskell regulars will note the similarity to <|>, but everyone else can treat it as a “clever or” concept and operator (i.e. each request to our API can be /tags or /questions)

type StackExchangeAPI =
            :> QueryParam "site" SiteName
            :> QueryParam "inname" SearchTerm
            :> Get '[JSON] TagResponse
    :<|> "questions"
            :> QueryParam "site" SiteName
            :> QueryParam "tagged" TagName
            :> Get '[JSON] (Response Question)

Detour: building an API client

Generating an API client for the entire API is now a one-liner (excluding signatures)

-- Set up the API client with pattern matching
getTags :: Maybe SiteName -> Maybe SearchTerm -> ClientM (Response Tag)
getQuestions :: Maybe SiteName -> Maybe TagName -> ClientM (Response Question)
getTags :<|> getQuestions = client api

Look at the Servant Client tutorial for details on the magic of pattern matching with the client function. The gist of it is that we have our two endpoints accessible using the getTags and getQuestions Haskell functions which return a ClientM monad instance (to allow IO, and optional failures as per runClientM’s return type IO (Either ServantError a)). Haskell’s side-effect tracking via IO monad means: it’s just a function now. Of course, if you’ve got this far, you probably knew all that though :). So enough philosophy…

Building queries

To query some tags, all we need do is pass in a strongly typed search term, and we’ll have a ClientM [Tag], i.e. a client function to return a list of Tag values for that term.

queryTags :: SearchTerm -> ClientM [Tag]
queryTags searchTerm = do
  theTags <- getTags (Just $ SiteName "stackoverflow") (Just searchTerm)
  return $ items theTags

Another side note: you can refactor this to queryTags searchTerm = items <$> getTags ... if you prefer, but the above is probably clearer for most people (including me, though neatness is appealing…).

Running those queries

main :: IO ()
main = do
  args <- getArgs
  let search = case args of
       word : _ -> T.pack word
       []    -> ""
  manager <- newManager defaultManagerSettings {managerModifyRequest = logIt}
  tagRes <- runClientM (queryTags (SearchTerm search)) (ClientEnv manager baseUrl)
  case tagRes of
    Left err   -> fail (show err)
    Right theTags -> mapM_ (TIO.putStrLn . name) theTags

So we:

  • get some CLI args, which we’re definiing as a single term to search for (defaulting to an empty string if there are none)
  • set up our HTTP manager with the defaults (don’t worry about this)
  • build a query and run it in the monad, staying in IO
  • …from which we deconstruct the Either monad, and if we’ve succeeded (Right _), we’ll have some tags.

Style note: pattern matching is nice, but again so is concise too. Another way I did the extract and match (but reverted) is:

  search <- fmap (maybe "" (T.pack . listToMaybe)) getArgs

(listToMaybe is a total function version of head, so we provide a default via maybe instead).


It’s just a case of getting their names out and printing them – lots of ways of doing this. Here’s an earlier, less concise one I used (still a long way from mastering Haskell; maybe this can help you too):

TIO.putStrLn $ T.intercalate "\n" $ fmap name tags

For fmap you can use map, i.e. the list specialisation,as tags is of [Tag], but I prefer to stay (type)classy and go functors…

By using mapM_ (we don’t care about the result) and composing the name accessor with putStrLn (which will print one newline per invocation) we can save some typing.

Build the tool

So when we run it (I’ve named the executable check-tags for brevity):
servant-quickcheck-sandbox git:(master) stack exec check-tags haskell

A live request bringing back a strongly typed API request, in just a few lines of code!

Some extra touches

I find it’s very useful to see what’s happening on the wire in any API testing or debugging. We can add some request logging here by using Network.HTTP’s convenient managerModifyRequest hook. We’ll just dump the whole thing to our output stream, then return the request unchanged. Sounds like some monadic sequencing!

logIt :: Show a => a -> IO a
logIt req = print req >> return req

We’ll set this up using some record syntax:

  manager <- newManager defaultManagerSettings {managerModifyRequest = logIt}

Now when we run it we get a full request log:

Request {
  host                 = ""
  port                 = 80
  secure               = False
  requestHeaders       = [("Accept","application/json")]
  path                 = "/2.2/tags"
  queryString          = "?site=stackoverflow&inname=haskell"
  method               = "GET"
  proxy                = Nothing
  rawBody              = False
  redirectCount        = 10
  responseTimeout      = ResponseTimeoutDefault
  requestVersion       = HTTP/1.1

I’ve added a few more tweaks. See the latest source if you’re interested.

But… smoke testing?

Yes, good point. We were getting distracted with seeing live results, because that’s another great feature of Servant. But smoke testing’s what we’re really interested in, so let’s get back to our HSpec test suite.

More predicates

Here’s a new spec for our API, with more predicates!

spec = describe "The Stack Exchange API" $
    it "returns success with JSON objects and best practice headers" $
      serverSatisfies api baseUrl stdArgs {maxSuccess = 5}
         <%> onlyJsonObjects
         <%> honoursAcceptHeader
         <%> notLongerThan (300 * millisInNanos)
         <%> mempty)

The predicates honoursAcceptHeader (checks Content-Type matches Accept) and onlyJsonObjects, taken from Servant.QuickCheck.Internal.Predicates are useful smoke tests you can run… against your whole API.

  • A one line performance checker for your entire API? notLongerThan
  • Make sure your JSON / XML is being declared and handled correctly? honoursAcceptHeader
  • Make sure all JSON response are wrapped as an object type? onlyJsonObjects

Writing our own predicate

That not500 is gone in favour of a (simple) new one isSuccess, that makes sure 3xx and 4xx are failures too. It looks like this:

isSuccess :: ResponsePredicate
isSuccess = ResponsePredicate p
  where rs = responseStatus
        p resp
          | rs resp `between` (status200, status300) = return ()
          | otherwise = throw $ PredicateFailure "isSuccess" Nothing resp
        between x (bottom, top) = bottom <= x && x < top

I’ve avoided the lambda style of expressing this (isSuccess = ResponsePredicate \resp -> ...), as it’s harder to read, IMHO.

Helpful failure messages

Like most APIs, StackExchange is rate-limited. If you max out your rate limit, you’ll get a 400 (yes… naughty StackExchange, don’t they know about 429 Too Many Requests). Now that 4xx are no longer tests passes, you can see a nice debug-friendly predicate failure message, thanks to our throw PredicateFailure as defined above:

  1) StackExchange, The Stack Exchange API, returns success with JSON objects and best practice headers
       Just Predicate failed
            Predicate: isSuccess
                 Method: "GET"
                 Path: /tags
                 Status code: 400
                 Headers:  "Cache-Control": "private"
                           "Content-Type": "application/json; charset=utf-8"
                           "Content-Encoding": "gzip"
                           "Access-Control-Allow-Origin": "*"
                           "Access-Control-Allow-Methods": "GET, POST"
                           "Access-Control-Allow-Credentials": "false"
                           "X-Content-Type-Options": "nosniff"
                           "Date": "Fri, 07 Apr 2017 21:16:17 GMT"
                           "Content-Length": "127"
                 Body: {"error_id":502,"error_message":"too many requests from this IP, more requests available in 824 seconds","error_name":"throttle_violation"}

The full request and response makes it very helpful seeing what went wrong (though targeted diagnosis would be very cool).

Better QuickCheck Arbitraries

Previously we’ve cheated a little on the QuickCheck side of this experiment – either Arbitrary instances are hard-coded (e.g. SiteName) or from a simple list (SearchTerm). It would be a little more realistic to use some slight randomness:

chars = '-' : ['a'..'z']

instance Arbitrary TagName
  where arbitrary = do
            s <- replicateM 6 $ elements chars
            return $ TagName $ T.pack s

At first I found the duality of this a bit confusing, but remember that a standard String is a list of characters, or thereabouts, and each character will itself be arbitrary (using elements which picks from a list). We define a list of safe chars and then use replicateM across the monad.

Where next

It’s taken a few detours but hopefully you’ve got a good idea of the ideas and potential in this kind of type-level testing or for those new to Haskell (like me still!) some mind-benders to stare at.

I’ve definitely learnt a lot digging into both the Quickcheck and especially Servant projects, and not just on the Haskell side (for the original project I was testing entirely Java-based microservices). As we strive to automate more and more of the testing space around ever more distributed services, it would be great to see more smoke tests and even non-functional requirements (NFRs) as code too (I can see some uses in IoT space too, given the numbers). I certainly plan to come back to Servant Quickcheck when I next get the chance…


Try @ThirdDeclension on Twitter, or if it’s a correction / issue around the codebase, please try the Github project issues.