Monthly Archives: February 2020

2020/02/26
1 Comment
Programming

Seriously, just use D to call C from Python

Last week I suggested that if you want to call C code from Python that you should use D. I still think that 4 lines of code (two of which are header includes) and a build system is hard to beat if that indeed is your goal. The internet, of course, had different ideas.

I think most of the comments I got can be summarised as:

In all cases, the general sentiment was that these were easy options to get the task done. Either they didn’t read the blog post, or they’re nowhere near as lazy as I am.

I described how writing basically no code gave you access to nanomsg from Python, including macros. No glue code, no struct definitions, nothing. “I can haz C headers in Python?” and some boring build system things were the long and short of it. 4 lines of code that scale with O(1) is hard to beat.

Let’s look at the suggestions then, starting with ctypes. It can call symbols in loaded shared libraries, which is great if you’re calling int foo(int, int) but not so great for real code (like, you know, nanomsg). No enums, no macros, and you have to declare C structs yourself. I shudder to think of what I’d have to write to get anything done. Next.

Some acknowledged that ctypes wasn’t all that, and that cffi should be used instead. I just tried using it today, thinking that it’d be amazing if I could just give Python some C headers and magic happened. But I can’t. I can pass in C declarations as a Python string, which I can manually copy from a header. But not just tell it what header I want (or worse, two in the case of my nanomsg example). I tried reading the headers myself, concatenating the strings and giving that to cdef, but that doesn’t work,  because:

  • Real headers have pesky things like #include directives in themcausing ParseErrors to be raised. Oops. So much for valid C syntax.
  • Struct definitions usually have members that are structs defined in another header, which have members that are structs defined in another header, which…
  • Macros

I soon realised how much work calling nanomsg with cffi would be and gave up. I wanted to show how much more code it takes, but as previously mentioned I’m lazy so no. Just… no. I figured someone would have written a generator for cffi, and I was right. It requires installing something called Irkit. Turns out it’s a parser generator, so cffi-gen  doesn’t use a industrial grade C compiler. Good luck compiling C code in the wild with that.

I’ve used Cython before and it’s even more work than cffi. Cython is the reason that I wrote autowrap in the first place.

Both boost::python and pybind11 are to C++ what pyd is to D. And pyd is another reason I wrote autowrap, because it has the gall to require users to specify one by one all of the things (functions, classes, etc.) they want to make available. With D’s compile-time reflection, I find that requirement quite insulting.

Last week I presented a way of calling C from Python which I find incredibly easy.  I got told there were viable alternatives, and since I didn’t know any better at the time I stroked my beard and thought “iiiinteresting”.

This week I can confidently state my belief that the easiest way to call C code from Python is D. Disagree? Show me how you can call nanomsg from Python your way. Github or it didn’t happen.

Tagged , , , , , , ,
2020/02/19
17 Comments
Programming

Want to call C from Python? Use D!

In my last blog post I wrote about the power of D’s compile-time reflection and string mixins, showing how they could be used to call D from Python so easily it might as well be magic. As amazing as that may be for those of us who have D codebases we want to expose to Python users, this doesn’t help the vastly more numerous programmers who want to call pre-existing C code instead. If C had D’s metaprogramming abilities, imagine seamlessly calling into nanomsg with as much as ease as I showed in my previous blog post. Well… about that.

D can easily interoperate with C, with the only requirement being that the function and data structure declarations be translated into D syntax. But once the translation is done, those declarations are now D code that can be reflected on, fed to autowrap, and automagically wrapped for Python consumption. That would be a pretty powerful combo if not for the boring work of translating all needed declarations, macros included. It’s still a lot easier than talking to the Python C API itself of course, but maybe not quite killer feature material.

However, I wrote a little project called dpp because I’m lazy and don’t want to hand-translate C to D. Envious of C++’s and Objective C’s credible claim to be the only languages that can seamlessly interoperate with C (due to header inclusion and compatible syntax), I tried to replicate the experience in the D world. Using dpp, one can #include C headers in what would otherwise be D code and use it as one would in C++, even going to the point of supporting preprocessor macros. I wrote about the project in a different blog post.

Given this .dpp file:

// nanomsg.dpp
#include "nanomsg/nn.h"
#include "nanomsg/pipeline.h"

And this .d file:

import autowrap;
mixin(
    wrapDlang!(
        LibraryName("nanomsg"), // name of the .so
        Modules(Yes.alwaysExport, "nanomsg") // name of the D module
    )
);

When we build both of those files above into nanomsg.so, we get to write this Python code that actually sends packets:

from nanomsg import (nn_socket, nn_close, nn_bind, nn_connect,
                     nn_send, nn_recv, AF_SP, NN_PUSH, NN_PULL)
import time

uri = "inproc://test"

pull = nn_socket(AF_SP, NN_PULL)
nn_bind(pull, uri)
time.sleep(0.05)  # give it time to set up (awful I know, but meh)

push = nn_socket(AF_SP, NN_PUSH)
nn_connect(push, uri)
msg = b'abc'
nn_send(push, msg, len(msg), 0)

Python, welcome to C, via D, and without even having to write any code to do it. Did I mention that AF_SP, NN_PUSH, and NN_PULL are all C macros? And yet, look at Python importing and using them like a boss.

Want to try it yourself? It’s on github.

If you want to call C from Python, use D.

Tagged , , , , , ,