Tag Archives: reflection

2020/02/19
17 Comments
Programming

Want to call C from Python? Use D!

In my last blog post I wrote about the power of D’s compile-time reflection and string mixins, showing how they could be used to call D from Python so easily it might as well be magic. As amazing as that may be for those of us who have D codebases we want to expose to Python users, this doesn’t help the vastly more numerous programmers who want to call pre-existing C code instead. If C had D’s metaprogramming abilities, imagine seamlessly calling into nanomsg with as much as ease as I showed in my previous blog post. Well… about that.

D can easily interoperate with C, with the only requirement being that the function and data structure declarations be translated into D syntax. But once the translation is done, those declarations are now D code that can be reflected on, fed to autowrap, and automagically wrapped for Python consumption. That would be a pretty powerful combo if not for the boring work of translating all needed declarations, macros included. It’s still a lot easier than talking to the Python C API itself of course, but maybe not quite killer feature material.

However, I wrote a little project called dpp because I’m lazy and don’t want to hand-translate C to D. Envious of C++’s and Objective C’s credible claim to be the only languages that can seamlessly interoperate with C (due to header inclusion and compatible syntax), I tried to replicate the experience in the D world. Using dpp, one can #include C headers in what would otherwise be D code and use it as one would in C++, even going to the point of supporting preprocessor macros. I wrote about the project in a different blog post.

Given this .dpp file:

// nanomsg.dpp
#include "nanomsg/nn.h"
#include "nanomsg/pipeline.h"

And this .d file:

import autowrap;
mixin(
    wrapDlang!(
        LibraryName("nanomsg"), // name of the .so
        Modules(Yes.alwaysExport, "nanomsg") // name of the D module
    )
);

When we build both of those files above into nanomsg.so, we get to write this Python code that actually sends packets:

from nanomsg import (nn_socket, nn_close, nn_bind, nn_connect,
                     nn_send, nn_recv, AF_SP, NN_PUSH, NN_PULL)
import time

uri = "inproc://test"

pull = nn_socket(AF_SP, NN_PULL)
nn_bind(pull, uri)
time.sleep(0.05)  # give it time to set up (awful I know, but meh)

push = nn_socket(AF_SP, NN_PUSH)
nn_connect(push, uri)
msg = b'abc'
nn_send(push, msg, len(msg), 0)

Python, welcome to C, via D, and without even having to write any code to do it. Did I mention that AF_SP, NN_PUSH, and NN_PULL are all C macros? And yet, look at Python importing and using them like a boss.

Want to try it yourself? It’s on github.

If you want to call C from Python, use D.

Advertisement
Tagged , , , , , ,
2020/01/22
2 Comments
Programming

The power of reflection

When I was at CppCon 2016 I overheard someone ask “Everyone keeps talking about  reflection, but why do we actually need it?”. A few years before that, I also would have had difficulty understanding why it would be useful. After years of writing D, it’s hard to imagine life without it.

Serialisation is an obvious use-case and almost always the first one anyone comes up with if pressed for an example. But there’s a lot more, like Design by Instrospection. It allows one to write a mocking framework. You start seeing applications everywhere. My favourite way to use it is to make the compiler write code for me.

Let’s say one wants to write a Python extension in native code. Each top-level Python function must have a corresponding C function (well, C ABI at least) that looks something like this:

PyObject* myfunc(PyObject* self, PyObject* args, PyObject* kwargs) {
    // ...
    return result;
}

There are a lot of details to take care of. There’s error handling, managing ref counts, and in all likelihood conversion from and to Python types since in most cases one is usually interested in calling existing pre-written code and make it available to Python. It’s tedious, and I haven’t even shown all the boilerplate to initialise the Python module and register the functions. The code for two simple functions ends up looking like this. Just thinking of clicking that link makes me sigh. Imagine what making calls into a real codebase would look like. We can do better:

import autowrap;
mixin(
    wrapDlang!(
        LibraryName("mylib"),
        Modules(
            Module("mymodule"),
            Module("myothermodule"),
        )
    )
);

The code above, when compiled, will generate a Python extension (shared library) that exposes every D function marked as “export” in the modules “mymodule” and “myothermodule” as Python functions. It’ll even convert their names from camelCase to snake_case. Any D exceptions thrown will become Python exceptions. D structs and classes become Python classses. If the original D functions take a D string, you’ll be able to pass Python strings to them in user code. Modulo bugs, this… works! The code shown above is the only code that needs to be written. Setting up the build system takes more work!

“Only” two D features are used here: the ability to do reflection at compile-time (and therefore to know which functions are in those modules and what types they take and return), and being able to mix in strings at compile-time. All the boilerplate is written for the user and inserted inline as if written by hand, but it’s the compiler that’s doing the heavy lifting.

Imagine now that your boss, pleased with these results, now wants you to also make the same D code avaiable to Excel users. The code changes not one bit, those lines above also work for Excel (the trick is telling the build system to depend on the autowrap:excel dub package instead of autowrap:python). Instead of snake_case functions, one now gets PascalCase as per Excel convention.

Same API, same functionality, different implementation. And no code to write for the user. The curious can see how it’s done on github.

 

Tagged , , , ,