This is literally the reason I most recently wrote some C, but - do you really need linking? You can pretty easily use a Python library from Ruby by writing a little loop in Python that accepts JSON input and produces JSON output, and calling that as a Ruby subprocess.
It's fairly rare that you actually need to be in the same process. (The thing I wrote was a wrapper for unshare(), so it did strictly need to be in-process, but that's an unusual use case.) Even if you want to share large amounts of data between your process and the library, you can often use things like memory-mapped files.
And there are some benefits from splitting up the languages, including the ability to let each language's runtime handle parallelism (finding out that the C library you want to use isn't actually thread-safe is no fun!), letting each language do exception-handling on its own, increased testability, etc.
> You can pretty easily use a Python library from Ruby by writing a little loop in Python that accepts JSON input and produces JSON output, and calling that as a Ruby subprocess.
If you're a Ruby application, sure, you can use a Python library by forking off a subprocess.
Just make sure to document the installation requirements - in addition to having Ruby and the correct gems installed, you also need Python (which version?) and the correct pip libraries installed.
If you're a Ruby library, instead of an application, forking off a Python process is a nonstarter, unless you want to propagate those requirements out to every single application using your library.
Sure, but there's an equivalent problem in the C world - go back in time a couple years before we figured out how to do precompiled binary-compatible C extensions in Python and try running 'pip install numpy'.
I'm mostly thinking that if you already have a Python library you really like, chances are you already have a Python environment capable of running it. (The environment doesn't have to be related to your Ruby environment at all! If you want to get Ruby through your OS and Python through a Docker container, or whatever, that should work fine.)
Inter-linking is a superpower, as is having a very thin runtime that ships with most operating systems.
I realized recently that Python could have saved a decade by having a means to "interlink" modules allowing the use of python 2 modules in python 3. Much easier in the C world; there have been incompatible syntax changes and incompatible link changes BUT not both at the same time.
Embedding Python 2 inside of Python 3 (or vice-versa) is not very hard to do on Linux. Simply `dlmopen` the .so in a new linker namespace and write a little bit of bridging code to interface the two object layouts.
Python has a very rich runtime, so there are some tricky problems to solve if you want this to be perfect. e.g., circular references between the GCs. However, there are simplifying assumptions that can dramatically reduce the difficulty, and these assumptions might not significantly hinder the language-upgrade use-case.
We've known about this approach to embedded interpreters for a couple of years, but we've not found a sufficiently compelling use-case for anyone to develop it beyond a simple proof-of-concept.
In general, it seems like there most compelling use-case for inter-language interfaces is writing core libraries in something fast, (mostly) runtime-free, and portable, especially for codes that no one wants to write twice.
Though I spend most of my time writing codes in slow, rich runtime languages, I've been on the lookout for a better technology to replace C. There are quite a few interesting options, and some have even less of a runtime than (dynamically-linked) C!
> Embedding Python 2 inside of Python 3 (or vice-versa) is not very hard to do on Linux. Simply `dlmopen` the .so in a new linker namespace and write a little bit of bridging code to interface the two object layouts.
Where can I pip install this from?
Or is there a reason that none of the huge numbers of python users delaying their transition as much as possible until all their libraries were updated built it?
The above will launch ("embed") a Python 2 or Python 1.5 interpreter from within a Python 3 interpreter. If you use `dlmopen` and `LM_ID_NEWLM`, the guest interpreter will have its own linker namespace. In other words, the guest interpreter (and any DSOs it opens) will be totally isolated from the host interpreter.
The above shows the use of `PyRun_SimpleString`. Since `PyRun_String` accepts a `PyObject* globals` and `PyObject* locals`, you could build a very basic bridge in <30 lines of code by using serialisation to copy and convert. (For user-defined classes, you would need something better.)
I've given a many talks about this at various Python and PyData conferences. (I've mentioned it a number of times on HN, both in response to complaints about the Python 2→3 transition and in response to comments like yours suggesting this solution to that problem.)
Though presented as a joke, the approach could be made to work with some effort. I can't speculate on why no one has ever followed-up on it, and I can't speculate on why the Python 2→3 transition has been so difficult for some users. Perhaps in some places, financial or organisational arguments are more influential than technical arguments.
Indeed. C++ is relatively portable too (or at the very least, can be made to hide behind a C ABI in many cases) and it seems like Rust is as well. Whereas, if you have a library written in python that you want to leverage, you're forced to include a python interpreter. This sucks.
Portability. A C library can be trivially linked with any other language. But one will be hard pressed using a python library form Ruby, for example.