Backends as dynamic libraries

[alecandido]

This is an alternative to #12.

Summary

When the backend can be instantiated on the same machine, we can make it part of the same process if there is a common interface.

Pros & cons

This prevents the need of setting up a server/client pair as in #12 (though, to support both options, there may be a tiny server just replying "look for the extension here"), and any type of inter-process communication (IPC), which may simplify the handling of the results as well.

However, the server/client architecture would be a native qibo-client, and handling memory external to the process one is a useful paradigm for both cloud and accelerators, since it will make operations efficient and transparent to the user.
So, this implementation of this may result in pure overhead, with minimal gain.

In any case, a deeper analysis may be needed to investigate the possible benefits of keeping anything in a single process (possibly even on hardware), and it may be an interesting project on its own - though not a top-priority one.

Implementation details

The idea is to go along the Python way of supporting compile extensions. This is possible by establishing some interface for what a backend is (a module in Python) at the ABI level, and then loading it from a compiled dynamic library by means of dlopen and dlsym.
E.g. this is done in CPython in:
https://github.com/python/cpython/blob/main/Python/dynload_shlib.c

How would then look like an implementation of a backend?

C (and C++/Rust/...)

Any language which can be compiled to a dynamic library can just make use of some headers/definitions given in qibo-core itself, in order to create a function which will construct a Backend object with the required interface.

At that point, the user code will call methods of the qibo_core::Backend object through the bindings in whatever language, and the call will be passed down "to the Rust library" (to the compiled binary resulting from there) that will be exactly the object provided by the shared library loaded, since it is ABI-compatible.

Python (and Julia/...)

In case of an interpreted language, this can not be reduced to a compiled extension just making use of just that language, since an interpreter is needed to make sense of the source code (which is the only kind of distribution).
So, essentially the backend shared library should embed an interpreter, and receive the path to the source modules to load them.
There will be then a standard Backend object per-language, passing its calls to the interpreter, and extracting the results from it to return them to the caller.

Embedding is usually possible, e.g. with Python we should compile the backend dynamically linking with libpython. This should be done just once per language by qibo-core itself, while the actual developer implementing the backend should just distribute the source modules.

The full journey could be something like the following:

• a python@numpy backend is specified by the user to be set
• qibo-core interprets the initial python and looks for the qibo-python executable, which instructs loading the libqibo_python
  • qibo-python would be standard and distributed by qibo-core
  • unless we skip this part, and we just embed the language@ interpretation in qibo-core itself
  • libqibo_python will also be distributed by qibo-core, since standard support for Python as a viable language for backends implementation
• at this point, a standard Backend object is loaded from libqibo_python, implementing the qibo-core defined interface
• it gets initialized with numpy
• as the Python backend, it will have its own logic to look up for the source providing the numpy backend
  • e.g. by looking for an installed Python package named qibo-backend-numpy
• the inner Python-defined backend is loaded within the interpreter embedded in the Backend created by libqibo_python

After the backend is instantiated, a call would happen in the following way:

• when a Backend::execute_circuit() is called by the user, the Backend will propagate the function call to the Python object
• the result returned is obtained from the interpreter, and returned to the caller

Eventually, this would result in at most three layers:

flowchart TD
    user["Bindings"] <--> inner["qibo-core (aka native)"] <--> backend["Backend"]

Loading

and whenever the bindings or backend is compiled, the object will just be the native one unwrapped (so if the program is in Fortran, and the backend in C++, in principle there should be no need to wrap anything).
In principle, the bindings are always an interface, and never an actual wrapper. But this will depend both on the language and the framework used to bind.