Code Trips & Tips

Introduction to libprocess

In Apache Mesos we make heavy use of the Actor model implemented in the libprocess library: the most recent implementation can be found under the 3rdparty folder in the Mesos repository (here).

libprocess was created originally at Berkley by Benjamin Hindman who is also the original creator of Mesos and implements “an actor style message-passing programming model (in C++).”

What this means, in practice, is that you can use it to implement a set of non-blocking processes which communicate with each other by sending “messages” (of an arbitrary nature – in Mesos, this means, for the most part Protocol Buffers) in a lock-free and race-free multi-threading model.

There is an in-depth explanation of libprocess in this presentation and what I’d like to do here is to present a simple example of how to use it in practice.

For the impatient, this is the gist, the actual explanation of the code will be split in two parts.

Instantiating a simple process

In its simplest form, a libprocess process is a class instance that derives from Process and has its methods designed in a way to support deferred references – in other words, returning Future objects:

    class SimpleProcess : public process::Process<SimpleProcess> {
    public:

      Future<Nothing> doSomething(const std::string msg) {
        std::cout << "Wrapping message: " << msg << std::endl;
        return Nothing();
      }

      Future<int> calc(int lhs, int rhs) {
        return Promise<int>(lhs + rhs).future();
      }

    private:
      Promise<bool> shouldQuit;
    };

To invoke this process, one calls spawn() (which is a low-cost call as it does not allocate and/or start a separate thread) and, when ready, dispatch()es a message to one of its methods:

    process::PID<SimpleProcess> pid = process::spawn(simpleProcess);

    process::dispatch(
      pid, &SimpleProcess::doSomething, "test test test");

    Future<int> sum = process::dispatch(pid, &SimpleProcess::calc, 99, 101);
    sum.then([](int n) {
        cout << "99 + 101 = " << n << endl;
        return Nothing();
    });

The libprocess framework will ensure that the messages are correctly routed and that they are delivered in strict order (please refer to the slides above for more details).

HTTP-based RPC invocation

Calling an in-process class’s methods running in a separate thread, even with the guarantee of lock-free and sequential ordering of messages, is interesting, but hardly earth-shattering: the full power of libprocess comes into its own when one considers that it enables seamless invocation of remote processes across a network (this is the process that is used in Mesos to enable the Master/Slave and Frameworks/Master interconnectivity).

In order to support a fully distributed system, libprocess implements an HTTP protocol that enables any Process-derived class to receive RPC calls over an HTTP socket: this is done transparently to the developer of the class, and can be exposed with the code presented in simpleserver.cpp.

In the virtual void initialize() method we configure the routes that will be managed internally by libprocess and will invoke the desired methods:

    void SimpleProcess::initialize() {
      route(
          "/add",
          "Adds the two query arguments",
          [] (Request request) {
            // TODO: one should check here for correctness and existence
            int a = numify<int>(request.query["a"]).get();
            int b = numify<int>(request.query["b"]).get();
            std::ostringstream result;
            result << "{ "result": " << a + b << "}";
            JSON::Value body = JSON::parse(result.str()).get();
            return OK(body);
      });
      route(
          "/quit",
          "Shuts the server down",
          [this] (Request request) {
            this->shutdown();
            return OK("Shutting down server");
      });
    }

by assigning a “server name” to our SimpleProcess class:

    SimpleProcess() : ProcessBase("simple") {}

the full URL for the add method will be something like:

http://localhost:51305/simple/add?a=33&b=9876

Without any further configuration (we’ll show how to do this, as well as to serialize and interpret Protocol Buffer paylodas in the next part) the system will bind the server to localhost on an arbitrary spare port on the system.

In the next part, we will see how the use of the install() method, alongside the serialization mechanism of Protocol Buffers, we can augment this mechanism to pass complex messages transparently across a network.

Running the examples

Unfortunately, the Github repo for libprocess is currently not up-to-date with the most recent changes that we have made in Mesos, so the best way to run the examples in the gist would be to download and install Mesos as described in the Starter Guide and then running:

make install

All the necessary header files will be in /usr/local/include and the libmesos.so to link against in /usr/local/lib – alternatively, you can link against the built libprocess in:

$MESOS_HOME/build/3rdparty/libprocess/.libs/libprocess.a

by using the -lprocess flag with -L pointing to the .libs folder.

This is how I built the server in Eclipse:

08:48:17 **** Incremental Build of configuration Debug for project Playground ****

make all 
Invoking: GCC C++ Compiler
g++ -std=c++0x -I/usr/local/include -O0 -g3 -Wall -c -fmessage-length=0 -MMD -MP \
    -MF"src/play_process.d" -MT"src/play_process.d" -o "src/play_process.o" \
    "src/play_process.cpp"
Finished building: ../src/play_process.cpp

Building target: SimpleServer
Invoking: MacOS X C++ Linker
g++ -L/usr/local/lib -o "SimpleServer"  ./src/play_process.o ./src/playground.o   -lgtest -lmesos

08:48:23 Build Finished (took 6s.422ms)

A simple run yields:

Running Server on http://127.0.0.1:53449/simple
Use /quit to terminate server...
Waiting for it to terminate...
are we done yet? 
Shutting down server...
Done

if you hit the http://127.0.0.1:53449/simple/error endpoint, you will see a 500 error returned, and an error message logged to stderr.

Next steps

Obviously, the whole point of libprocess is far from implementing an HTTP server; it is about interprocess communication in a highly distributed system (such as Mesos): in the next part of the blog, we will see how to accomplish this.