Tag Archives: D

2020/02/26
1 Comment
Programming

Seriously, just use D to call C from Python

Last week I suggested that if you want to call C code from Python that you should use D. I still think that 4 lines of code (two of which are header includes) and a build system is hard to beat if that indeed is your goal. The internet, of course, had different ideas.

I think most of the comments I got can be summarised as:

In all cases, the general sentiment was that these were easy options to get the task done. Either they didn’t read the blog post, or they’re nowhere near as lazy as I am.

I described how writing basically no code gave you access to nanomsg from Python, including macros. No glue code, no struct definitions, nothing. “I can haz C headers in Python?” and some boring build system things were the long and short of it. 4 lines of code that scale with O(1) is hard to beat.

Let’s look at the suggestions then, starting with ctypes. It can call symbols in loaded shared libraries, which is great if you’re calling int foo(int, int) but not so great for real code (like, you know, nanomsg). No enums, no macros, and you have to declare C structs yourself. I shudder to think of what I’d have to write to get anything done. Next.

Some acknowledged that ctypes wasn’t all that, and that cffi should be used instead. I just tried using it today, thinking that it’d be amazing if I could just give Python some C headers and magic happened. But I can’t. I can pass in C declarations as a Python string, which I can manually copy from a header. But not just tell it what header I want (or worse, two in the case of my nanomsg example). I tried reading the headers myself, concatenating the strings and giving that to cdef, but that doesn’t work,  because:

  • Real headers have pesky things like #include directives in themcausing ParseErrors to be raised. Oops. So much for valid C syntax.
  • Struct definitions usually have members that are structs defined in another header, which have members that are structs defined in another header, which…
  • Macros

I soon realised how much work calling nanomsg with cffi would be and gave up. I wanted to show how much more code it takes, but as previously mentioned I’m lazy so no. Just… no. I figured someone would have written a generator for cffi, and I was right. It requires installing something called Irkit. Turns out it’s a parser generator, so cffi-gen  doesn’t use a industrial grade C compiler. Good luck compiling C code in the wild with that.

I’ve used Cython before and it’s even more work than cffi. Cython is the reason that I wrote autowrap in the first place.

Both boost::python and pybind11 are to C++ what pyd is to D. And pyd is another reason I wrote autowrap, because it has the gall to require users to specify one by one all of the things (functions, classes, etc.) they want to make available. With D’s compile-time reflection, I find that requirement quite insulting.

Last week I presented a way of calling C from Python which I find incredibly easy.  I got told there were viable alternatives, and since I didn’t know any better at the time I stroked my beard and thought “iiiinteresting”.

This week I can confidently state my belief that the easiest way to call C code from Python is D. Disagree? Show me how you can call nanomsg from Python your way. Github or it didn’t happen.

Advertisement
Tagged , , , , , , ,
2020/02/19
17 Comments
Programming

Want to call C from Python? Use D!

In my last blog post I wrote about the power of D’s compile-time reflection and string mixins, showing how they could be used to call D from Python so easily it might as well be magic. As amazing as that may be for those of us who have D codebases we want to expose to Python users, this doesn’t help the vastly more numerous programmers who want to call pre-existing C code instead. If C had D’s metaprogramming abilities, imagine seamlessly calling into nanomsg with as much as ease as I showed in my previous blog post. Well… about that.

D can easily interoperate with C, with the only requirement being that the function and data structure declarations be translated into D syntax. But once the translation is done, those declarations are now D code that can be reflected on, fed to autowrap, and automagically wrapped for Python consumption. That would be a pretty powerful combo if not for the boring work of translating all needed declarations, macros included. It’s still a lot easier than talking to the Python C API itself of course, but maybe not quite killer feature material.

However, I wrote a little project called dpp because I’m lazy and don’t want to hand-translate C to D. Envious of C++’s and Objective C’s credible claim to be the only languages that can seamlessly interoperate with C (due to header inclusion and compatible syntax), I tried to replicate the experience in the D world. Using dpp, one can #include C headers in what would otherwise be D code and use it as one would in C++, even going to the point of supporting preprocessor macros. I wrote about the project in a different blog post.

Given this .dpp file:

// nanomsg.dpp
#include "nanomsg/nn.h"
#include "nanomsg/pipeline.h"

And this .d file:

import autowrap;
mixin(
    wrapDlang!(
        LibraryName("nanomsg"), // name of the .so
        Modules(Yes.alwaysExport, "nanomsg") // name of the D module
    )
);

When we build both of those files above into nanomsg.so, we get to write this Python code that actually sends packets:

from nanomsg import (nn_socket, nn_close, nn_bind, nn_connect,
                     nn_send, nn_recv, AF_SP, NN_PUSH, NN_PULL)
import time

uri = "inproc://test"

pull = nn_socket(AF_SP, NN_PULL)
nn_bind(pull, uri)
time.sleep(0.05)  # give it time to set up (awful I know, but meh)

push = nn_socket(AF_SP, NN_PUSH)
nn_connect(push, uri)
msg = b'abc'
nn_send(push, msg, len(msg), 0)

Python, welcome to C, via D, and without even having to write any code to do it. Did I mention that AF_SP, NN_PUSH, and NN_PULL are all C macros? And yet, look at Python importing and using them like a boss.

Want to try it yourself? It’s on github.

If you want to call C from Python, use D.

Tagged , , , , , ,
2019/04/24
4 Comments
Programming

Type inference debate: a C++ culture phenomenon?

I read two C++ subreddit threads today on using the auto keyword. They’re both questions: the first one asks why certain people seem to dislike using type inference, while the second asks about what commonly taught guidelines should be considered bad practice. A few replies there mention auto.

This confuses me for more than one reason. The first is that I’m a big proponent of type inference. Having to work to tell a computer what it already knows is one of my pet peeves. I also believe that wanting to know the exact type of a variable is a, for lack of a better term, “development smell”, especially in typed languages with generics. I think that the possible operations on a type are what matters, and figuring out the exact type if needed is a tooling problem that is solved almost everywhere. If you don’t at least have “jump to definition” in your development environment, you should.

The other main source of confusion for me is why this debate seems restricted to the C++ community, at least according to my anectodal evidence. As far as I’m aware of, type inference is taken to be a good thing almost universally in Go, D, Rust, Haskell, et al. I don’t have statistics on this at all, but from what I’ve seen so far it seems to be the case. Why is that?

One reason might be change. As I learned reading Peopleware, humans aren’t too fond of it. C++ lacked type inference until C++11, and whoever learned it then might not like using auto now; it’s “weird”, so it must be bad. I’ve heard C programmers who “like” looping over a collection using an integer index and didn’t understand when I commented on their Python code asking them to please not do that. “Like” here means “this is what I’m used to”. Berating people for disliking change is definitely not the way forward, however. One must take their feelings into account if one wants to effect change. Another lesson learned from Peopleware: the problem is hardly ever technical.

Otherwise, I’m not sure. I’m constantly surprised by what other people find to be more or less readable when it comes to code. There are vocal opponents to operator overloading, for instance. I have no idea why.

What do you think? Where do you stand on auto?

Tagged , , , , , , , , ,
2019/03/27
Leave a comment
Programming

My DConf2019 talk

I’ll be speaking at DConf this year. I haven’t yet decided how to structure the talk yet, but I’d like to take a page out of Dan Sak’s excellent CppCon 2016 talk that covered technical and psychological aspects of language adoption. In his case he was addressing a C++ audience on the difficulty of convincing C programmers to adopt C++ (an admirable goal!). In mine, it’s how do I out-C++ C++?

The genesis for my talk was my decision a few years ago to write tests for a legacy C codebase. I wanted to use D, but even as a fan of the language I chose to write the tests in C++ instead. And if even I wouldn’t have picked D for that task, who would?

I still think it was the right decision to make. The first step in”importing” the legacy project into a C++ test looks like this:

extern "C" {
    #include "legacy.h"
}

There is no second step. From here one #includes a test library such as catch2 and starts writing tests for the C functions in the API.

As far as I know, every language under the sun can interface with C. There’s usually (always?) some way to do FFI, and examples invariably show how to call a function that takes 2 integers, maybe even a const char* parameter. Unfortunately, that’s not what real codebases need to do. They need to call a function that takes a pointer to a structure that’s defined in a different header, which has 3 pointers to structures that are themselves defined in different headers, which…

You get the idea. Writing the definitions by hand error-prone, boring, and time-consuming. Even if correct, nobody seems to talk about the elephant in the C API room: macros. Not once have I encountered a non-trivial C API that doesn’t require the C preprocessor to use properly. C++ users are at an advantage here: just use the macros. Everyone else has to get by with ad-hoc solutions. And if the function-like macro implementations change, the code will break.

I think that the only way to make it so that C++ isn’t the obvious and/or only choice is that a challenger allows one to write `#include “legacy.h”` and have it work just as easily as it does natively. That’s why I started the dpp project, and I’m looking forward to talking about the technical challenges I’ve encountered on my journey so far.

See you at DConf!

Tagged , , , , , , ,
2019/03/13
Leave a comment
Programming

Issues DIP1000 can’t (yet) catch

D’s DIP1000 is its attempt to avoid memory safety issues in the language. It introduces the notion of scoped pointers and in principle (modulo bugs) prevents one from writing memory unsafe code.

I’ve been using it wherever I can since it was implemented, and it’s what makes it possible for me to have written a library allowing for fearless concurrency. I’ve also written a D version of C++’s std::vector that leverages it, which I thought was safe but turns out to have had a gaping hole.

I did wonder if Rust’s more complicated system would have advantages over DIP1000, and now it seems that the answer to that is yes. As the the linked github issue above shows,  there are ways of creating dangling pointers in the same stack frame:

void main() @safe {
    import automem.vector;

    auto vec1 = vector(1, 2, 3);
    int[] slice1 = vec1[];
    vec1.reserve(4096);  // invalidates slice1 here
}

Fortunately this is caught by ASAN if slice1 is used later. I’ve worked around the issue for now by returning a custom range object from the vector that has a pointer back to the container – that way it’s impossible to “invalidate iterators” in C++-speak. It probably costs more in performance though due to chasing an extra pointer. I haven’t measured to confirm the practical implications.

In Rust, the problem just can’t occur. This fails to compile (as it should):

use vec;

fn main() {
    let mut v = vec![1, 2, 3, 4, 5];
    let s = v.as_slice();
    println!("org v: {:?}\norg s: {:?}", v, s);
    v.resize(15, 42);
    println!("res: v: {:?}\norg s: {:?}", v, s);
}

I wonder if D can learn/borrow/steal an idea or two about only having one mutable borrow at a time. Food for thought.

Tagged , , , ,
2019/03/07
2 Comments
Programming

The joys of translating C++’s std::function to D

I wrote a program to translate C headers to D. Translating C was actually more challenging than I thought; I even got to learn things I didn’t know about the language even though I’ve known it for 24 years. The problems that I encountered were all minor though, and to the extent of my knowledge have all been resolved (modulo bugs).

C++ is a much larger language, so the effort should be considerably more. I didn’t expect it to be as hard as it’s been however, and in this blog I want to talk about how “interesting” it was to translate C++11’s std::function by hand.

The first issue for most languages would be that it relies on template specialisation:

template<typename>
class function;  // doesn't have a definition anywhere

template<typename R, typename... Args>
class function<R(Args...)> { /* ... */ }

This is a way of constraining the std::function template to only accept function types. Perhaps surprisingly to some, the C++ syntax for the type of a function that takes two ints and returns a double is double(int, int). I doubt most people see this outside of C++ standard library templates. If it’s still confusing, think of double(int, int) as the type that is obtained by deferencing a pointer of type double(*)(int, int).

D is, as far as I know, the only other language other than C++ to support partial template specialisation. There are however two immediate problems:

  • function is a keyword in D
  • There is no D syntax for a function type

I can mitigate the name issue by calling the symbol function_ instead; however, this will affect name mangling, meaning nothing will actually link. D does have pragma(mangle) to tell the compiler how to mangle symbols, but std::function is a template; it doesn’t have any mangling until it’s instantiated. Let’s worry about that later and call the template function_ for now.

The second issue can be worked around:

// C++: `using funPtr = double(*)(int, int);`
alias funPtr = double function(int, int);
// C++: `using funType = double(int, int);`
alias funType = typeof(*funPtr.init);

As in C++, the function type is the type one gets from deferencing a function pointer. Unlike C++, currently there’s no syntax to write it directly. First attempt:

// helper to automate getting an alias to a function type
template FunctionType(R, Args...) {
    alias ptr = R function(Args);
    alias FunctionType = typeof(*ptr.init);
}

struct function_(T);
struct function_(T: FunctionType!(R, Args), R, Args...) { }

This doesn’t work, probably due to this bug preventing the helper template FunctionType from working as intended. Let’s forget the template constraint:

extern(C++, "std") {
    struct function_(T) {
        import std.traits: ReturnType, Parameters;
        alias R = ReturnType!T;
        alias Args = Parameters!T;
        // In C++: `R operator()(Args) const`;
        R opCall(Args) const;
    }
}

void main() {
    alias funPtr = double function(double);
    alias funType = typeof(*funPtr.init);
    function_!funType f;
    double result = f(3.3);
}

This compiles but it doesn’t link: there’s an undefined reference to std::function_::operator()(double) const. Looking at the symbols in the object files using nm, we see that g++ emitted _ZNKSt8functionIFddEEclEd but dmd is trying to link to _ZNKSt9function_IFddEEclEd. As expected, name mangling issues related to renaming the symbol.

We could manually add a pragma(mangle) to tell D how to mangle the operator for the double(double) template instantiation, but that solution doesn’t scale. CTFE (constexpr if you speak C++ but not D) to the rescue!

// snip - as before
pragma(mangle, opCall.mangleof.fixMangling)
R opCall(Args) const;

// (elsewhere at file scope)
string fixMangling(string str) {
    import std.array: replace;
    return str.replace("9function_", "8function");
}

What’s going on here is an abuse of D’s compile-time power. The .mangleof property is a compile-time string that tells us how a symbol is going to be mangled. We pass this string to the fixMangling function which is evaluated at compile-time and fed back to the compiler telling it what symbol name to actually use. Notice that function_ is still a template, meaning .mangleof has a different value for each instantiation. It’s… almost magical. Hacky, but magical.

The final code compiles and links. Actually creating a valid std::function<double(double)> from D code is left as an exercise to the reader.

 

Tagged , , , , , , ,
2018/12/31
11 Comments
Programming

Comparing Pythagorean triples in C++, D, and Rust

EDIT: As pointed out in the Rust reddit thread, the Rust version can be modified to run faster due to a suble difference between ..=z and ..(z+1). I’ve updated the measurements with rustc0 being the former and rustc1 being the latter. I’ve also had to change some of the conclusions.

You may have recently encountered and/or read this blog post criticising a possible C++20 implementation of Pythagorean triples using ranges. In it the author benchmarks different implemetations of the problem, comparing readability, compile times, run times and binary sizes. My main language these days is D, and given that D also has ranges (and right now, as opposed to a future version of the language), I almost immediately reached for my keyboard. By that time there were already some D and Rust versions floating about as a result of the reddit thread, so fortunately for lazy me “all” I had to next was to benchmark the lot of them. All the code for what I’m about to talk about can be found on github.

As fancy/readable/extensible as ranges/coroutines/etc. may be, let’s get a baseline by comparing the simplest possible implementation using for loops and printf. I copied the “simplest.cpp” example from the blog post then translated it to D and Rust. To make sure I didn’t make any mistakes in the manual translation, I compared the output to the canonical C++ implementation (output.txt in the github repo). It’s easy to run fast if the output is wrong, after all. For those not familiar with D, dmd is the reference compiler (compiles fast, generates sub-optimal code compared to modern backends) while ldc is the LLVM based D compiler (same frontend, uses LLVM for the backend). Using ldc also makes for a more fair comparison with clang and rustc due to all three using the same backend (albeit possibly different versions of it).

All times reported will be in milliseconds as reported by running commands with time on my Arch Linux Dell XPS15. Compile times were measured by excluding the linker, the commands being clang++ -c src.cpp, dmd -c src.d, ldc -c src.d, rustc --emit=obj src.rs for each compiler (obviously for unoptimised builds). The original number of iterations was 100, but the runtimes were so short as to be practically useless for comparisons, so I increased that to 1000. The times presented are a result of running each command 10 times and taking the mininum, except for the clang unoptmised build of C++ ranges because, as a wise lady on the internet once said, ain’t nobody got time for that (you’ll see why soon). Optimised builds were done with -O2 for clang and ldc,  -O -inline for dmd and -C opt-level=2 for rustc. The compiler versions were clang 7.0.1, dmd 2.083.1, ldc 1.13.0 and rustc 1.31.1. The results for the simple, readable, non-extensible version of the problem (simple.cpp, simple.d, simple.rs in the repo):

Simple          CT (ms)  RT (ms)

clang debug      59       599
clang release    69       154
dmd debug        11       369
dmd release      11       153
ldc debug        31       599
ldc release      38       153
rustc0 debug    100      8445
rustc0 release  103       349
rustc1 debug             6650
rustc1 release            217

C++ run times are the same as D when optimised, with compile times being a lot shorter for D. Rust stands out here for both being extremely slow without optimisations, compiling even slower than C++, and generating code that takes around 50% longer to run even with optimisations turned on! I didn’t expect that at all.

The simple implementation couples the generation of the triples with what’s meant to be done with them. One option not discussed in the original blog to solve that is to pass in a lambda or other callable to the code instead of hardcoding the printf. To me this is the simplest way to solve this, but according to one redditor there are composability issues that may arise. This might or might not be important depending on the application though. One can also compose functions in a pipeline and pass that in, so I’m not sure what the problem is. In any case, I wrote 3 implementations and timed them (lambda.cpp, lambda.d, lambda.rs):

Lambda          CT (ms)  RT (ms)

clang debug      188       597
clang release    203       154
dmd debug         33       368
dmd release       37       154
ldc debug         59       580
ldc release       79       154
rustc0 debug     111      9252
rustc0 release   134       352
rustc1 debug              6811
rustc1 release             154

The first thing to notice is, except for Rust (which was slow anyway), compile times have risen by about a factor of 3: there’s a cost to being generic. Run times seem unaffected except that the unoptimised Rust build got slightly slower. I’m glad that my intuition seems to have been right on how to extend the original example: keep the for loops, pass a lambda, profit. Sure, the compile-times went up but in the context of a larger project this probably wouldn’t matter that much. Even for C++, 200ms is… ok. Performance-wise, it’s now a 3-way tie between the languages, and no runtime penalty compared to the non-generic version. Nice!

Next up, the code that motivated all of this: ranges. I didn’t write any of it: the D version was from this forum post, the C++ code is in the blog (using the “available now” ranges v3 library), and the Rust code was on reddit. Results:

Range           CT (ms)   RT (ms)

clang debug     4198     126230
clang release   4436        294
dmd debug         90      12734
dmd release      106       7755
ldc debug        158      15579
ldc release      324       1045
rustc0 debug     128      11018
rustc0 release   180        422
rustc1 debug               8469
rustc1 release              168

I have to hand it to rustc – whatever you throw at it the compile times seem to be flat. After modifying the code as mentioned in the edit at the beginning, it’s now the fastest out of the 3!

dmd compile times are still very fast, but the generated code isn’t. ldc does better at optimising, but the runtimes are pretty bad for both of them. I wonder what changes would have to be made to the frontend to generate the same code as for loops.

C++? I only ran the debug version once. I’m just not going to wait for that long, and besides, whatever systematic errors are in the measurement process don’t really matter when it takes over 2 minutes. Compile times are abysmal, the only solace being that optimising the code only takes 5% longer time than to not bother at all.

In my opinion, none of the versions are readable. Rust at least manages to be nearly as fast as the previous two versions, with C++ taking twice as long as it used to for the same task. D didn’t fare well at all where performance is concerned. It’s possible to get the ldc optimised version down to ~700ms by eliminating bounds checking, but even then it wouldn’t seem to be worth the bother.

My conclusion? Don’t bother with the range versions. Just… don’t. Nigh unreadable code that compiles and runs slowly.

Finally, I tried the D generator version on reddit. There’s a Rust version on Hacker News as well, but it involves using rust nightly and enabling features, and I can’t be bothered figuring out how to do any of that. If any Rustacean wants to submit something that works with a build script, please open a PR on the repo.

Generator      CT (ms)  RT (ms)

dmd debug      208      381
dmd release    222      220
ldc debug      261      603
ldc release    335      224

Compile times aren’t great (the worst so far for D), but the runtimes are quite acceptable. I had a sneaky suspicion that the runtimes here are slower due to startup code for std.concurrency so I increased N to 5000 and compared this version to the simplest possible one at the top of this post:

N=5k                    RT (ms)
dmd relase simple       8819
dmd release generator   8875

As expected, the difference vanished.

Final conclusions: for me,  passing a lambda is the way to go, or generators if you have them (C++ doesn’t have coroutines yet and the Rust version mentioned above needs features that are apparently only available in the nightly builds). I like ranges in both their C++ and D incarnations, but sometimes they just aren’t worth it. Lambdas/generators FTW!

Tagged , , , , ,
2018/12/11
4 Comments
Programming

Improvements I’d like to see in D

D, as any language that stands on the shoulder of giants, was conceived to not repeat the errors of the past, and I think it’s done an admirable job at that. However, and perfectly predictably, it made a few of its own. Sometimes, similar to the ones it avoided! In my opinion, some of them are:

No UFCS chain for templates.

UFCS is a great feature and was under discussion to be added to C++, but as of the C++17 standard it hasn’t yet. It’s syntatic sugar for treating a free function as a member function if the first parameter is of that type, i.e. allows one to write obj.func(3) instead of func(obj, 3). Why is that important? Algorithm chains. Consider:

range.map!fun.filter!gun.join

Instead of:

join(filter!gun(map!fun(range)));

It’s much more readable and flows more naturally. This is clearly a win, and yet I’m talking about it in a blog post about D’s mistakes. Why? Because templates have no such thing. There are compile-time “type-level” versions of both map and filter in D’s standard library called staticMap and Filter (the names could be more consistent). But they don’t chain:

alias memberNames = AliasSeq!(__traits(allMembers, T));
alias Member(string name) = Alias!(__traits(getMember, T, name));
alias members = staticMap!(Member, memberNames);
alias memberFunctions = Filter!(isSomeFunction, members);

One has to name all of these intermediate steps even though none of them deserve to be named, just to avoid writing a russian doll parenthesis unreadable nightmare. Imagine if instead it were:

alias memberFunctions = __traits(allMembers, T)
    .staticMap!Member
    .Filter!(isSomeFunction);

One can dream. Which leads me to:

No template lambdas.

In the hypothetical template UFCS chain above, I wrote staticMap!Member, where the Member definition is as in the example before it. However: why do I need to name it either? In “regular” code we can use lambdas to avoid naming functions. Why can’t I do the same thing for templates?

alias memberFunctions = __traits(allMembers, T)
    .staticMap!(name => Alias!(__traits(getMember, T, name)))
    .Filter!isSomeFunction

Eponymous templates

Bear with me: I think the idea behind eponymous templates is great, is just that the execution isn’t, and I’ll explain why by comparing it to something D got right: constructors and destructors. In C++ and Java, these special member functions take the name of the class, which makes refactoring quite a bit annoying. D did away with that by naming them this and ~this, which makes the class name irrelevant. The way to do eponymous templates right is to (obviously renaming the feature) follow D’s own lead here and use either this or This to refer to itself. I’ve lost count of how many times I’ve introduced a bug due to template renaming that was hard to find.

@property getters

What are these for given the optional parentheses for functions with no parameters?

inout

Template this can accomplish the same thing and is more useful anyway.

Returning a reference

It’s practically pointless. Variables can’t be ref, so assigning it to a function that returns ref copies the value (which is probably not what the user expected), so one might as well return a pointer. The exception would be UFCS chains, but before DIP1016 is accepted and implemented, that doesn’t work either.

Wrong defaults

I think there’s a general consensus that @safe, pure and immutable should be default. Which leads me to:

Attribute soup

I’ve lost count now of how many times I’ve had to write @safe @nogc pure nothrow const scope return. Really.

Tagged ,
2018/11/06
Leave a comment
Programming

Implementing Rust’s std::sync::Mutex in D

TL;DR? https://github.com/atilaneves/fearless

The first time I encountered a mutex in C++ I was puzzled. It made no sense to me at all that I was locking one to protect some data and the only way to indicate what data was protected by a certain mutex was a naming convention. It seemed to me like a recipe for disaster, and of course, it is.

I’ve hardly written any code in Rust, in fact only one project to learn the basics of the language. But the time I spent with it was enough to marvel at std::sync::Mutex – at last it made sense! Access to the variable has to go through the mutex’s API, and no convention is needed. And, of course, in the Rust tradition said access is safe.

That unsurprisingly made me slightly jealous. In D, shared is a keyword and it protects the programmer from inadvertently accessing shared state in an unsafe manner (mostly). Atomic operations typically take a pointer to a shared T, but larger objects (i.e. user-defined structs) are usually dealt with by locking a mutex, casting away shared and then using the object as thread-local. While this works, it’s tedious, error-prone, and certainly not safe. Since imitation is the highest form of flattery, I decided to shamelessly copy, as much as possible, the idea behind Rust’s Mutex.

Rust makes the API safe via the borrow checker, but D doesn’t have that. It does, however, have the sort-of-still-experimental DIP1000, which is similar in what it tries to achieve. I set out to use the new functionality to try and devise a safe way to avoid the current practice of BYOM (Bring Your Own Mutex).

I started off by reading the concurrency part of The Rust Book, which was very helpful. It even explains implementation details such as the fact that .borrow returns a smart pointer instead of the wrapped type. This too I copied. I then started thinking of ways to use D’s scope to emulate Rust’s borrow checker. The idea wasn’t to have the same semantics but to enable safe usage and fail to compile unsafe code. Pragmatism was key.

I was initially confused about why std::sync::Mutex is nearly always used with std::sync::Arc – it took me writing a bug to realise that shared data is never allocated on the stack. Obvious in retrospect but I somehow failed to realise that. Since Rust doesn’t have a mark-and-sweep GC, the only real option is to use reference counting for the heap-allocated shared data. This realisation led to another: in D there’s a choice between reference counting and simply using GC-allocated memory, so the API reflects that. The RC version is even @nogc!

The API forces the initialisation of the user-defined type to happen by passing parameters to the constructor of that type. This is because passing an extant object isn’t safe – other references to it may exist in the program and data races could occur. Rust can do this and guarantee at compile-time that other mutable references don’t exist, but D can’t, hence the restriction. For types without mutable indirections the restriction is lifted, made possible by D’s world class static reflection capabilities. The API also enforces that the type is shared – there’s no point to using this library if the type isn’t, and even less of a point making the user type `shared T` all the time.

Although D has an actor model message passing library in std.concurrency, none of the functions are @safe. I also realised it would be trivial to write a deadlock by sending the shared data while a mutex is held to another thread. To fix both of these issues, the library I wrote has @safe versions of D’s concurrency primitives, and the send function checks to see if the mutex is locked before actually passing the compound (mutex, T) type (named Exclusive in the library) to another thread of execution.

DIP1000 itself was hard to understand. I ended up reading the proposal several times, and it didn’t help that the current implementation doesn’t match that document 100%. In the end, however, the result seems to work:

https://github.com/atilaneves/fearless

It’s possible that, due to bugs in DIP1000 or in fearless itself that a programmer can break safety, but to the extent of my knowledge this brings @safe concurrent code to D.

I’d love it if any concurrency experts could try and poke holes in the library so I can fix them.

Tagged , , , , , ,
2018/04/09
7 Comments
Programming

#include C headers in D code

I’ll lead with a file:

// stdlib.dpp
#include <stdio.h>
#include <stdlib.h>

void main() {
    printf("Hello world\n".ptr);

    enum numInts = 4;
    auto ints = cast(int*) malloc(int.sizeof * numInts);
    scope(exit) free(ints);

    foreach(int i; 0 .. numInts) {
        ints[i] = i;
        printf("ints[%d]: %d ".ptr, i, ints[i]);
    }

    printf("\n".ptr);
}

The keen eye will notice that, except for the two include directives, the file is just plain D code. Let’s build and run it:

% d++ stdlib.dpp
% ./stdlib
Hello world
ints[0]: 0 ints[1]: 1 ints[2]: 2 ints[3]: 3

Wait, what just happened?

You just saw a D file directly #include two headers from the C standard library and call two functions from them, which was then compiled and run. And it worked!

Why? I mean, just… why?

I’ve argued before that #include is C++’s killer feature. Interfacing to existing C or C++ libraries is, for me, C++’s only remaining use case. You include the relevant headers, and off you go. No bindings, no nonsense, it just works. As a D fan, I envied that. So this is my attempt to eliminate that “last” (again, for me, reasonable people may disagree) use case where one would reach for C++ as the weapon of choice.

There’s a reason C++ became popular. Upgrading to it from C was a decision with essentially 0 risk.  I wanted that “just works” feature for D.

How?

d++ is a compiler wrapper. By default it uses dmd to compile the D code, but that’s configurable through the –compiler option. But dmd can’t compile code with #include directives in it (the lexer won’t even like it!), so what gives?

d++ will go through a .dpp file, and upon encountering an #include directive it expands it in-place, similarly to what would happen with a C or C++ compiler. Differently from clang or gcc however, the header file can’t just be inserted in, since the syntax of the declarations is in a different language. So it uses libclang to parse the header, and translates all of the declarations on the fly. This is trickier than it sounds since C and C++ allow for things that aren’t valid in D.

There’s one piece of the usability puzzle that’s missing from that story: the preprocessor. C header files have declarations but also macros, and some of those are necessary to use the library as it was intended. One can try and emulate this with CTFE functions in D, and sometimes it works. But I don’t want “sometimes”, I want guarantees, and the only way to do that is… to use the C preprocessor.

Blasphemy, I know. But since worse is better, d++ redefines all macros in the included header file so they’re available for use by the D program. It then runs the C preprocessor on the result of expanding all the #include directives, and the final result is a regular D file that can be compiled by dmd.

What next?

Bug fixing and C++ support. I won’t be happy until this works:

#include <vector>
void main() {
    auto v = std.vector!int();
    v.push_back(42);
}

Code or it didn’t happen

I almost forgot: https://github.com/atilaneves/dpp.

 

Tagged , ,
← Older posts