Smoke testing APIs… at the type level [3/3]
Where were we?
In part 2, we developed our test API to model the real responses, and started testing an API against a stubbed service. In this part, we’ll:
- add a new endpoint
- build an API client tool
- extend the arbitrary generation (of QuickCheck types)
- add some new predicates
…and discuss a few ideas of where to go next.
Recap: our original API
type StackExchangeAPI =
"tags"
:> QueryParam "site" SiteName
:> QueryParam "inname" SearchTerm
:> Get '[JSON] TagResponse
Level Up
That’s been great, but feels more like an endpoint than an API, so let’s add another one – the Questions endpoint, which behaves in a similar way to our tags one. Servant allows an idea of composing endpoint descriptions with its own magic combinator, :<|>
. Haskell regulars will note the similarity to <|>
, but everyone else can treat it as a “clever or” concept and operator (i.e. each request to our API can be /tags
or /questions
)
type StackExchangeAPI =
"tags"
:> QueryParam "site" SiteName
:> QueryParam "inname" SearchTerm
:> Get '[JSON] TagResponse
:<|> "questions"
:> QueryParam "site" SiteName
:> QueryParam "tagged" TagName
:> Get '[JSON] (Response Question)
Detour: building an API client
Generating an API client for the entire API is now a one-liner (excluding signatures)
-- Set up the API client with pattern matching
getTags :: Maybe SiteName -> Maybe SearchTerm -> ClientM (Response Tag)
getQuestions :: Maybe SiteName -> Maybe TagName -> ClientM (Response Question)
:<|> getQuestions = client api getTags
Look at the Servant Client tutorial for details on the magic of pattern matching with the client
function. The gist of it is that we have our two endpoints accessible using the getTags
and getQuestions
Haskell functions which return a ClientM
monad instance (to allow IO
, and optional failures as per runClientM
’s return type IO (Either ServantError a)
). Haskell’s side-effect tracking via IO
monad means: it’s just a function now. Of course, if you’ve got this far, you probably knew all that though :). So enough philosophy…
Building queries
To query some tags, all we need do is pass in a strongly typed search term, and we’ll have a ClientM [Tag]
, i.e. a client function to return a list of Tag
values for that term.
queryTags :: SearchTerm -> ClientM [Tag]
= do
queryTags searchTerm <- getTags (Just $ SiteName "stackoverflow") (Just searchTerm)
theTags return $ items theTags
Another side note: you can refactor this to queryTags searchTerm = items <$> getTags ...
if you prefer, but the above is probably clearer for most people (including me, though neatness is appealing…).
Running those queries
main :: IO ()
= do
main <- getArgs
args let search = case args of
: _ -> T.pack word
word -> ""
[] <- newManager defaultManagerSettings {managerModifyRequest = logIt}
manager <- runClientM (queryTags (SearchTerm search)) (ClientEnv manager baseUrl)
tagRes case tagRes of
Left err -> fail (show err)
Right theTags -> mapM_ (TIO.putStrLn . name) theTags
So we:
- get some CLI args, which we’re definiing as a single term to search for (defaulting to an empty string if there are none)
- set up our HTTP manager with the defaults (don’t worry about this)
- build a query and run it in the monad, staying in
IO
- …from which we deconstruct the
Either
monad, and if we’ve succeeded (Right _
), we’ll have some tags.
Style note: pattern matching is nice, but again so is concise too. Another way I did the extract and match (but reverted) is:
<- fmap (maybe "" (T.pack . listToMaybe)) getArgs search
(listToMaybe
is a total function version of head
, so we provide a default via maybe
instead).
Output
It’s just a case of getting their names out and printing them – lots of ways of doing this. Here’s an earlier, less concise one I used (still a long way from mastering Haskell; maybe this can help you too):
TIO.putStrLn $ T.intercalate "\n" $ fmap name tags
For fmap
you can use map
, i.e. the list specialisation,as tags is of [Tag]
, but I prefer to stay (type)classy and go functors…
By using mapM_
(we don’t care about the result) and composing the name
accessor with putStrLn
(which will print one newline per invocation) we can save some typing.
Build the tool
So when we run it (I’ve named the executablecheck-tags
for brevity):
➜ servant-quickcheck-sandbox git:(master) stack exec check-tags haskell
haskell
haskell-stack
template-haskell
haskell-platform
haskell-pipes
wxhaskell
...
A live request bringing back a strongly typed API request, in just a few lines of code!
Some extra touches
I find it’s very useful to see what’s happening on the wire in any API testing or debugging. We can add some request logging here by using Network.HTTP’s convenient managerModifyRequest
hook. We’ll just dump the whole thing to our output stream, then return the request unchanged. Sounds like some monadic sequencing!
logIt :: Show a => a -> IO a
= print req >> return req logIt req
We’ll set this up using some record syntax:
<- newManager defaultManagerSettings {managerModifyRequest = logIt} manager
Now when we run it we get a full request log:
Request {
host = "api.stackexchange.com"
port = 80
secure = False
requestHeaders = [("Accept","application/json")]
path = "/2.2/tags"
queryString = "?site=stackoverflow&inname=haskell"
method = "GET"
proxy = Nothing
rawBody = False
redirectCount = 10
responseTimeout = ResponseTimeoutDefault
requestVersion = HTTP/1.1
}
I’ve added a few more tweaks. See the latest source if you’re interested.
But… smoke testing?
Yes, good point. We were getting distracted with seeing live results, because that’s another great feature of Servant. But smoke testing’s what we’re really interested in, so let’s get back to our HSpec test suite.
More predicates
Here’s a new spec for our API, with more predicates!
= describe "The Stack Exchange API" $
spec "returns success with JSON objects and best practice headers" $
it = 5}
serverSatisfies api baseUrl stdArgs {maxSuccess
(isSuccess<%> onlyJsonObjects
<%> honoursAcceptHeader
<%> notLongerThan (300 * millisInNanos)
<%> mempty)
The predicates honoursAcceptHeader
(checks Content-Type
matches Accept
) and onlyJsonObjects
, taken from Servant.QuickCheck.Internal.Predicates are useful smoke tests you can run… against your whole API.
- A one line performance checker for your entire API?
notLongerThan
- Make sure your JSON / XML is being declared and handled correctly?
honoursAcceptHeader
- Make sure all JSON response are wrapped as an object type?
onlyJsonObjects
Writing our own predicate
That not500
is gone in favour of a (simple) new one isSuccess
, that makes sure 3xx
and 4xx
are failures too. It looks like this:
isSuccess :: ResponsePredicate
= ResponsePredicate p
isSuccess where rs = responseStatus
p resp| rs resp `between` (status200, status300) = return ()
| otherwise = throw $ PredicateFailure "isSuccess" Nothing resp
= bottom <= x && x < top between x (bottom, top)
I’ve avoided the lambda style of expressing this (isSuccess = ResponsePredicate \resp -> ...
), as it’s harder to read, IMHO.
Helpful failure messages
Like most APIs, StackExchange is rate-limited. If you max out your rate limit, you’ll get a 400
(yes… naughty StackExchange, don’t they know about 429 Too Many Requests
). Now that 4xx
are no longer tests passes, you can see a nice debug-friendly predicate failure message, thanks to our throw PredicateFailure
as defined above:
1) StackExchange, The Stack Exchange API, returns success with JSON objects and best practice headers
Failed:
Just Predicate failed
Predicate: isSuccess
Request:
Method: "GET"
Path: /tags
Headers:
Body:
Response:
Status code: 400
Headers: "Cache-Control": "private"
"Content-Type": "application/json; charset=utf-8"
"Content-Encoding": "gzip"
"Access-Control-Allow-Origin": "*"
"Access-Control-Allow-Methods": "GET, POST"
"Access-Control-Allow-Credentials": "false"
"X-Content-Type-Options": "nosniff"
"Date": "Fri, 07 Apr 2017 21:16:17 GMT"
"Content-Length": "127"
Body: {"error_id":502,"error_message":"too many requests from this IP, more requests available in 824 seconds","error_name":"throttle_violation"}
The full request and response makes it very helpful seeing what went wrong (though targeted diagnosis would be very cool).
Better QuickCheck Arbitraries
Previously we’ve cheated a little on the QuickCheck side of this experiment – either Arbitrary
instances are hard-coded (e.g. SiteName
) or from a simple list (SearchTerm
). It would be a little more realistic to use some slight randomness:
= '-' : ['a'..'z']
chars
instance Arbitrary TagName
where arbitrary = do
<- replicateM 6 $ elements chars
s return $ TagName $ T.pack s
At first I found the duality of this a bit confusing, but remember that a standard String is a list of characters, or thereabouts, and each character will itself be arbitrary (using elements
which picks from a list). We define a list of safe chars and then use replicateM
across the monad.
Where next
It’s taken a few detours but hopefully you’ve got a good idea of the ideas and potential in this kind of type-level testing or for those new to Haskell (like me still!) some mind-benders to stare at.
I’ve definitely learnt a lot digging into both the Quickcheck and especially Servant projects, and not just on the Haskell side (for the original project I was testing entirely Java-based microservices). As we strive to automate more and more of the testing space around ever more distributed services, it would be great to see more smoke tests and even non-functional requirements (NFRs) as code too (I can see some uses in IoT space too, given the numbers). I certainly plan to come back to Servant Quickcheck when I next get the chance…
Contact
Try @ThirdDeclension
on Twitter, or if it’s a correction / issue around the codebase, please try the Github project issues.