Tag Archives: bdd

Unit-tested, acceptance-tested, type-checked and yet still buggy

So I wrote an MQTT broker once (twice really, but I never really finished the second version). It’s now my go-to way of learning a new computer language. Once Rust finally makes it to version 1.0, I’ll write an MQTT broker in it as well. It’s a problem I know well now, it tackles the hairy problems of networking and concurrency, and it’s small enough to not become a huge time sink. So that’s how I decided to try and learn Haskell.

With the immense paradigm shift that came with learning Haskell came a much longer development time. I’m ok with that, especially given that the code is so much shorter. This time I decided to BDD the whole thing, using Cucumber to drive the acceptance tests and writing unit tests for the rest.

After much toiling and trying to wrap my head around monads (I finally understand them! It only took months of reading multiple different blog posts and using them…), I got a version that compiled and passed all tests. A working MQTT broker! So let’s run Jeff’s benchmark on it to see how it compares with the implementations in other languages and… oops.

The program hangs. I try again with 10 messages and 2 connections. It still hangs. How can this be? I did my due dilligence, didn’t I? Aren’t Haskell programs just supposed to work if they compile? I mean, I had to jump around hoops to write a function to convert character literals into Word8 values so write my tests! After a lot of “printf” debugging (the Haskell debugging experience is not ideal) I find what’s causing the bug, and as always, it seems obvious in hindsight.

Networking is dirty, ugly and variable. Not the kind of thing that lends itself well to being unit-tested. So I cheated a bit. The part that dealt with client connections and whether or not the server should disconnect went into the main loop and wasn’t unit tested at all. I felt bad about it at the time but let it go on one of those “it’s ok, I know what I’m doing” feelings. And as usual, I break my own rules at my own peril.

I had an acceptance test for it, and it passed. It just wasn’t comprehensive enough. The TCP traffic has to be just so to trigger the bug, and it’s actually hard to craft packets that look like real-world usage scenarios. Who wants to have a list of bytes 512 bytes long in their test code, much less several of them?

My conclusion of the error of my ways? Not enough unit-testing. Too much code outside the nice, warm and fuzzy pure core. The feeling I shook off about the dirty networking code not being unit tested? I’m never doing that again. The fix is going to be relatively simple: purify as much of the code as I can so it’s trivially unit-testable and have the “real” code be a thin wrapper over the pure code.

The worst is that was already my belief of how to develop robust software. I just didn’t follow my own advice.

Advertisements
Tagged , , ,

To learn BDD with Cucumber, you must first learn BDD with Cucumber.

So I read about Cucumber a while back and was intrigued, but never had time to properly play with it. While writing my MQTT broker, however, I kept getting annoyed at breaking functionality that wasn’t caught by unit tests. The reason being that the internals were fine, the problems I was creating had to do with the actual business of sending packets. But I was busy so I just dealt with it.

A few weeks ago I read a book about BDD with Cucumber and RSpec but for me it was a bit confusing. The reason being that since the step definitions, unit tests and implementation were all written in Ruby, it was hard for me to distinguish which part was what in the whole BDD/TDD concentric cycles. Even then, I went back to that MQTT project and wrote two Cucumber features (it needs a lot more but since it works I stopped there). These were easy enough to get going: essentially the step definitions run the broker in another process, connect to it over TCP and send packets to it, evaluating if the response was the expected one or not. Pretty cool stuff, and it works! It’s what I should have been doing all along.

So then I started thinking about learning BDD (after all, I wrote the features for MQTT afterwards) by using it on a D project. So I investigated how I could call D code from my step definitions. After spending the better part of an afternoon playing with Thrift and binding Ruby to D, I decided that the best way to go about this was to implement the Cucumber wire protocol. That way a server would listen to JSON requests from Cucumber, call D functions and everything would work. Brilliant.

I was in for a surprise though, me who’s used to implementing protocols after reading an RFC or two. Instead of a usual protocol definition all I had to go on was… Cucumber features! How meta. So I’d use Cucumber to know how to implement my Cucumber server. A word to anyone wanting to do this in another language: there’s hardly any documentation on how to implement the wire protocol. Whenever I got lost and/or confused I just looked at the C++ implementation for guidance. It was there that I found a git submodule with all of Cucumber’s features. Basically, you need to implement all of the “core” features first (therefore ensuring that step definitions actually work), and only then do you get to implement the protocol server itself.

So I wanted to be able to write Cucumber step definitions in D so I could learn and apply BDD to my next project. As it turned out, I learned BDD implementing the wire protocol itself. It took a while to get the hang of transitioning from writing a step definition to unit testing but I think I’m there now. There might be a lot more Cucumber in my future. I might also implement the entirety of Cucumber’s features in D as well, I’m not sure yet.

My implementation is here.

Tagged , , , , , , , , , ,