Introduction to libprocess
In Apache Mesos we make heavy use of the Actor model implemented in the libprocess
library: the most recent implementation can be found under the 3rdparty
folder in the Mesos repository (here).
libprocess
was created originally at Berkley by Benjamin Hindman who is also the original creator of Mesos and implements “an actor style message-passing programming model (in C++).”
What this means, in practice, is that you can use it to implement a set of non-blocking processes which communicate with each other by sending “messages” (of an arbitrary nature – in Mesos, this means, for the most part Protocol Buffers) in a lock-free and race-free multi-threading model.
There is an in-depth explanation of libprocess
in this presentation and what I’d like to do here is to present a simple example of how to use it in practice.
For the impatient, this is the gist, the actual explanation of the code will be split in two parts.
Instantiating a simple process
In its simplest form, a libprocess
process is a class instance that derives from Process
and has its methods designed in a way to support deferred references – in other words, returning Future
objects:
class SimpleProcess : public process::Process<SimpleProcess> { public: Future<Nothing> doSomething(const std::string msg) { std::cout << "Wrapping message: " << msg << std::endl; return Nothing(); } Future<int> calc(int lhs, int rhs) { return Promise<int>(lhs + rhs).future(); } private: Promise<bool> shouldQuit; };
To invoke this process, one calls spawn()
(which is a low-cost call as it does not allocate and/or start a separate thread) and, when ready, dispatch()
es a message to one of its methods:
process::PID<SimpleProcess> pid = process::spawn(simpleProcess); process::dispatch( pid, &SimpleProcess::doSomething, "test test test"); Future<int> sum = process::dispatch(pid, &SimpleProcess::calc, 99, 101); sum.then([](int n) { cout << "99 + 101 = " << n << endl; return Nothing(); });
The libprocess
framework will ensure that the messages are correctly routed and that they are delivered in strict order (please refer to the slides above for more details).
HTTP-based RPC invocation
Calling an in-process class’s methods running in a separate thread, even with the guarantee of lock-free and sequential ordering of messages, is interesting, but hardly earth-shattering: the full power of libprocess
comes into its own when one considers that it enables seamless invocation of remote processes across a network (this is the process that is used in Mesos to enable the Master/Slave and Frameworks/Master interconnectivity).
In order to support a fully distributed system, libprocess
implements an HTTP protocol that enables any Process
-derived class to receive RPC calls over an HTTP socket: this is done transparently to the developer of the class, and can be exposed with the code presented in simpleserver.cpp.
In the virtual void initialize()
method we configure the routes that will be managed internally by libprocess
and will invoke the desired methods:
void SimpleProcess::initialize() { route( "/add", "Adds the two query arguments", [] (Request request) { // TODO: one should check here for correctness and existence int a = numify<int>(request.query["a"]).get(); int b = numify<int>(request.query["b"]).get(); std::ostringstream result; result << "{ "result": " << a + b << "}"; JSON::Value body = JSON::parse(result.str()).get(); return OK(body); }); route( "/quit", "Shuts the server down", [this] (Request request) { this->shutdown(); return OK("Shutting down server"); }); }
by assigning a “server name” to our SimpleProcess
class:
SimpleProcess() : ProcessBase("simple") {}
the full URL for the add
method will be something like:
http://localhost:51305/simple/add?a=33&b=9876
Without any further configuration (we’ll show how to do this, as well as to serialize and interpret Protocol Buffer paylodas in the next part) the system will bind the server to localhost
on an arbitrary spare port on the system.
In the next part, we will see how the use of the install()
method, alongside the serialization mechanism of Protocol Buffers, we can augment this mechanism to pass complex messages transparently across a network.
Running the examples
Unfortunately, the Github repo for libprocess is currently not up-to-date with the most recent changes that we have made in Mesos, so the best way to run the examples in the gist would be to download and install Mesos as described in the Starter Guide and then running:
make install
All the necessary header files will be in /usr/local/include
and the libmesos.so
to link against in /usr/local/lib
– alternatively, you can link against the built libprocess
in:
$MESOS_HOME/build/3rdparty/libprocess/.libs/libprocess.a
by using the -lprocess
flag with -L
pointing to the .libs
folder.
This is how I built the server in Eclipse:
08:48:17 **** Incremental Build of configuration Debug for project Playground ****
make all
Invoking: GCC C++ Compiler
g++ -std=c++0x -I/usr/local/include -O0 -g3 -Wall -c -fmessage-length=0 -MMD -MP \
-MF"src/play_process.d" -MT"src/play_process.d" -o "src/play_process.o" \
"src/play_process.cpp"
Finished building: ../src/play_process.cpp
Building target: SimpleServer
Invoking: MacOS X C++ Linker
g++ -L/usr/local/lib -o "SimpleServer" ./src/play_process.o ./src/playground.o -lgtest -lmesos
08:48:23 Build Finished (took 6s.422ms)
A simple run yields:
Running Server on http://127.0.0.1:53449/simple
Use /quit to terminate server...
Waiting for it to terminate...
are we done yet?
Shutting down server...
Done
if you hit the http://127.0.0.1:53449/simple/error endpoint, you will see a 500
error returned, and an error message logged to stderr
.
Next steps
Obviously, the whole point of libprocess
is far from implementing an HTTP server; it is about interprocess communication in a highly distributed system (such as Mesos): in the next part of the blog, we will see how to accomplish this.
Leave a Reply