Tag Archives: rustlang

Implementing Rust’s std::sync::Mutex in D

TL;DR? https://github.com/atilaneves/fearless

The first time I encountered a mutex in C++ I was puzzled. It made no sense to me at all that I was locking one to protect some data and the only way to indicate what data was protected by a certain mutex was a naming convention. It seemed to me like a recipe for disaster, and of course, it is.

I’ve hardly written any code in Rust, in fact only one project to learn the basics of the language. But the time I spent with it was enough to marvel at std::sync::Mutex – at last it made sense! Access to the variable has to go through the mutex’s API, and no convention is needed. And, of course, in the Rust tradition said access is safe.

That unsurprisingly made me slightly jealous. In D, shared is a keyword and it protects the programmer from inadvertently accessing shared state in an unsafe manner (mostly). Atomic operations typically take a pointer to a shared T, but larger objects (i.e. user-defined structs) are usually dealt with by locking a mutex, casting away shared and then using the object as thread-local. While this works, it’s tedious, error-prone, and certainly not safe. Since imitation is the highest form of flattery, I decided to shamelessly copy, as much as possible, the idea behind Rust’s Mutex.

Rust makes the API safe via the borrow checker, but D doesn’t have that. It does, however, have the sort-of-still-experimental DIP1000, which is similar in what it tries to achieve. I set out to use the new functionality to try and devise a safe way to avoid the current practice of BYOM (Bring Your Own Mutex).

I started off by reading the concurrency part of The Rust Book, which was very helpful. It even explains implementation details such as the fact that .borrow returns a smart pointer instead of the wrapped type. This too I copied. I then started thinking of ways to use D’s scope to emulate Rust’s borrow checker. The idea wasn’t to have the same semantics but to enable safe usage and fail to compile unsafe code. Pragmatism was key.

I was initially confused about why std::sync::Mutex is nearly always used with std::sync::Arc – it took me writing a bug to realise that shared data is never allocated on the stack. Obvious in retrospect but I somehow failed to realise that. Since Rust doesn’t have a mark-and-sweep GC, the only real option is to use reference counting for the heap-allocated shared data. This realisation led to another: in D there’s a choice between reference counting and simply using GC-allocated memory, so the API reflects that. The RC version is even @nogc!

The API forces the initialisation of the user-defined type to happen by passing parameters to the constructor of that type. This is because passing an extant object isn’t safe – other references to it may exist in the program and data races could occur. Rust can do this and guarantee at compile-time that other mutable references don’t exist, but D can’t, hence the restriction. For types without mutable indirections the restriction is lifted, made possible by D’s world class static reflection capabilities. The API also enforces that the type is shared – there’s no point to using this library if the type isn’t, and even less of a point making the user type `shared T` all the time.

Although D has an actor model message passing library in std.concurrency, none of the functions are @safe. I also realised it would be trivial to write a deadlock by sending the shared data while a mutex is held to another thread. To fix both of these issues, the library I wrote has @safe versions of D’s concurrency primitives, and the send function checks to see if the mutex is locked before actually passing the compound (mutex, T) type (named Exclusive in the library) to another thread of execution.

DIP1000 itself was hard to understand. I ended up reading the proposal several times, and it didn’t help that the current implementation doesn’t match that document 100%. In the end, however, the result seems to work:

https://github.com/atilaneves/fearless

It’s possible that, due to bugs in DIP1000 or in fearless itself that a programmer can break safety, but to the extent of my knowledge this brings @safe concurrent code to D.

I’d love it if any concurrency experts could try and poke holes in the library so I can fix them.

Advertisement
Tagged , , , , , ,

Rust impressions from a C++/D programmer, part 2

Following up from my first post on Rust, I thought that after a week running a profiler and trying to optimise my code would have given me new insights into working with the language. That’s why I called it “part 1”, I was looking forward to uncovering more warts.

The thing is… I didn’t. It turns out that the 1st version I cranked out was already optimised. It’s not because I’m a leet coder: I’ve implemented an MQTT broker before and looked at profiler output. I know where the bottlenecks will be for the two benchmarks I use. So my first version was fast enough.

How anti-climatic. The only news-worthy thing that came out of benchmarking it is that the Rust/mio combo is really fast. You’ll have to wait for the official benchmarks comparing all the implementations to know how much though. I’m currently rewriting my problematic C++ version. I have to if I want to measure it: either I give it my best shot of making it fast or the reddit comments will be… interesting to say the least. It got nasty last time.

Tagged ,

Rust impressions from a C++/D programmer, part 1

Discussion on programming reddit

Discussion on Rust reddit

C++ and D aren’t the only languages I know, I labeled myself that way in the title because as far as learning Rust is concerned, I figured they would be the most relevant in terms of the audience knowing where I’m coming from.

Since two years ago, my go-to task for learning a new programming language is to implement an MQTT broker in it. It was actually my 3rd project in D, but my first in Haskell and now that I have some time on my hands, it’s what I’m using to learn Rust. I started last week and have worked on it for about 3 days. As expected, writing an MQTT broker is a great source of insight into how a language really is. You know, the post-lovey-dovey phase. It’s like moving in together straight away instead of the first-date-like “here’s how you write a Scheme interpreter”.

I haven’t finished the project yet, I’m probably somewhere around the 75% mark, which isn’t too shabby for 3 days of work. Here are my impressions so far:

The good

The borrow checker. Not surprising since this is basically the whole point of the language. It’s interesting how much insight it gave me in how broken the code I’m writing elsewhere might be.  This will be something I can use when I write in other systems languages, like how learning Haskell makes you wary of doing IO.

Cargo. Setting up, getting started, using someone’s code and unit testing what you write as you go along is painless and just works. Tests in parallel by default? Yes, please. I wonder where I’ve seen that before…

Traits. Is there another language other than D and Rust that make it this easy to use compile-time polymorphism? If there is, please let me know which one. Rust has an advantage here: as in Dylan (or so I read), the same trait can be used for runtime polymorphism.

Warnings. On by default, and I only had to install flycheck-rust in Emacs for syntax highlighting to just work. Good stuff.

Productivity. This was surprising, given the borrow checker’s infamy. It _does_ take a while to get my code to compile, but overall I’ve been able to get a good amound done with not that much time, given these are the first lines of Rust I’ve ever written.

Algebraic types and pattern matching. Even though I didn’t use the former.

Slices. Non-allocating views into data? Yes, please. Made the D programmer in me feel right at home.

Immutable by default. Need I say more?

Debugging. rust-gdb makes printing out values easy. I couldn’t figure out how to break on certain functions though, so I had to use the source file and line number instead.

No need to close a socket due to RAII. This was nice and even caught a bug for me. The reason being that I expected my socket to close because it was dropped, but my test failed. When I looked into it, the reference count was larger than 1 because I’d forgotten to remove the client’s subscriptions. The ref count was 0, the socket was dropped and closed, and the test passed. Nice.

No parens for match, if, for, …

The bad

The syntax. How many times can one write an ampersand in one’s source code? You’ll break new records. Speaking of which…

Explicit borrows. I really dislike the fact that I have to tell the compiler that I’m the function I’m calling is borrowing a parameter when the function signature itself only takes borrows. It won’t compile otherwise (which is good), but… since I can’t get it wrong what’s the point of having to express intent? In C++:

void fun(Widget& w);
auto w = Widget();
fun(w); //NOT fun(&w) as in Rust

In Rust:

fn fun(w: &mut Widget);
let w = Widget::new();
fun(&mut w); //fun(w) doesn't compile but I still need to spell out &mut. Sigh.

Display vs Debug. Printing out integers and strings with {} is fine, but try and do that with a Vec or HashMap and you have to use the weird {:?}. I kept getting the order of the two symbols wrong as well. It’s silly. Even the documentation for HashMap loops over each entry and prints them out individually. Ugh.

Having to rethink my code. More than once I had to find a different way to do the thing I wanted to do. 100% of the time it was because of the borrow checker. Maybe I couldn’t figure out the magical incantation that would get my code to compile, but in one case I went from “return a reference to an internal object, then call methods on it” to “find object and call method here right now”. Why? So I wouldn’t have to borrow it mutably twice. Because the compiler won’t let me. My code isn’t any safer and it was just annoying.

Rc<RefCell<T>> and Arc<Mutex<T>>. Besides the obvious “‘Nuff said”, why do I have to explicitly call .clone on Rc? It’s harder to use than std::shared_ptr.

Slices. Writing functions that slices and passing them vectors works well enough. I got tired of writing &var[..] though. Maybe I’m doing something wrong. Coming from D I wanted to avoid vectors and just slice arrays instead. Maybe that’s not Rusty. What about appending together some values to pass into a test? No Add impl for Vecs, so it’s massive pain. Sigh.

Statements vs Expressions. I haven’t yet made the mistake of forgetting/adding a semicolon, but I can see it happening.

No function overloading.

Serialization. There’s no way to do it well without reflection, and Rust is lacking here. I just did everything by hand, which was incredibly annoying. I’m spoiled though, in D I wrote what I think is a really good serialization library. Good in the lazy sense, I pretty much never have to write custom serialization code.

The ugly

Hashmaps. The language has operator overloading, but HashMap doesn’t use it. So it’s a very Java-like map.insert(key, value). If you want to create a HashMap with a literal… you can’t. There’s no equivalent macro to vec. You could write your own, but come on, this is a basic type from the standard library that will get used a lot. Even C++ does better!

Networking / concurrent IO. So I took a look at what my options were, and as far as my googling took me, it was to use native threads or a library called mio. mio’s API was… not the easiest to use so I punted and did what is the Rust standard library way of writing a server and used threads instead. I was sure I’d have performance problems down the road but it was something to worry about later. I went on writing my code, TDDed an implementation of a broker that wasn’t connected to the outside world and everything. At one point I realised that holding on to a mutable reference for subscribers wasn’t going to work so I used Rc<RefCell<Subscriber>> instead. It compiled, my tests passed, and all was good in the world. Then I tried actually using the broker from my threaded server. Since it’s not safe to use Rc<RefCell<>> in threads, this failed to compile. “Good!”, I thought, I changed Rc to Arc and RefCell to Mutex. Compile, run, …. deadlock. Oops. I had to learn mio after all. It wasn’t as bad as boost::asio but it wasn’t too far away either.

Comparing objects for identity. I just wanted to compare pointers. It was not fun. I had to write this:

fn is_same<T>(lhs: &T, rhs: &T) -> bool {
    lhs as *const T == rhs as *const T;
}
fn is_same_subscriber<T: Subscriber>(lhs: Rc<RefCell<T>>, rhs: Rc<RefCell<T>>) -> bool {
    is_same(&*lhs.borrow, &*rhs.borrow());
}

Yuck.

Summary

I thought I’d like Rust more than I actually do at this point. I’m glad I’m taking the time to learn it, but I’m not sure how likely I’ll choose to use it for any future project. Currently the only real advantage it has for me over D is that it has no runtime and could more easily be used on bare metal projects. But I’m unlikely to do any of those anytime soon.

I never thought I’d say this a few years back but…I like being able to fall back on a mark-and-sweep GC. I don’t have to use it in D either, so if it ever becomes a performance or latency problem I know how to deal with it. It seems to me to be far easier than getting the borrow checker to agree with me or having to change how I want to write my code.

We’ll see, I guess. Optimising the Rust implementation to be competitive with the D and Java ones is likely to be interesting.

Tagged , , , ,

Computer languages: ordering my favourites

This isn’t even remotely supposed to be based on facts, evidence, benchmarks or anything like that. You could even disagree with what are “scripting languages” or not. All of the below just reflect my personal preferences. In any case, here’s my list of favourite computer languages, divided into two categories: scripting and, err… I guess “not scripting”.

 

My favourite scripting languages, in order:

  1. Python
  2. Ruby
  3. Emacs Lisp
  4. Lua
  5. Powershell
  6. Perl
  7. bash/zsh
  8. m4
  9. Microsoft batch files
  10. Tcl

 

I haven’t written enough Ruby yet to really know. I suspect I’d like it more than Python but at the moment I just don’t have enough experience with it to know its warts. Even I’m surprised there’s something below Perl here but Tcl really is that bad. If you’re wondering where PHP is, well I don’t know because I’ve never written any but from what I’ve seen and heard I’d expect it to be (in my opinion of course) better than Tcl and worse than Perl. I’m surprised how high Perl is given my extreme dislike for it. When I started thinking about it I realised there’s far far worse.

 

My favourite non-scripting languages, in order:

  1. D
  2. C++
  3. Haskell
  4. Common Lisp
  5. Rust
  6. Java
  7. Go
  8. Objective C
  9. C
  10. Pascal
  11. Fortran
  12. Basic / Visual Basic

I’ve never used Scheme, if that explains where Common Lisp is. I’m still learning Haskell so not too sure there. As for Rust, I’ve never written a line of code in it and yet I think I can confidently place it in the list, especially with respect to Go. It might place higher than C++ but I don’t know yet.

 

Tagged , , , , , , , , , , , , ,