Apache Mesos Leader Master Discovery using Zookeeper

Motivation

When deploying a Mesos cluster, made of several Master and Worker nodes, it may not be possible (or desirable) to know the IP address(es) of the Master nodes in advance; however, as they can be deployed using Zookeeper as backing storage, we can use the information stored there to retrieve the IP address of the Leader (and the other Masters too).

Background

When launched in HA mode, Mesos master nodes register with Zookeeper and then go through a simple “leader election” contest round: the first one to register (as decided by the lowest-numbered sequence znode) becomes the elected leader.

Hitting any of the masters’ HTTP endpoints will re-direct (302) to the elected leader’s URL – the information about the elected leader, its hostname and IP address are kept in a Protocol Buffer structure that is the data content of the sequential (ephemeral) znode created during the election round.

Approach

Given the above, our approach is simply to connect to the Zookeeper ensemble and, armed with the knowledge of the path to the Mesos masters’ znode (this must be provided when the masters are launched – see the Apache Mesos guide for more details) we can just retrieve the data from the sequential znodes there and deserialize it into Protocol Buffers.

The code in the zk-mesos repository is a Python implementation of the suggested approach.

See the README file there for instructions and more details about how to install it and the actual code.

Future

We are working on implementing a more friendly JSON data format for serializing the MasterInfo data to Zookeeper (see MESOS-2340) and I also have running prototype code that implements this (for release 0.24 – which is one-away from the next one: 0.23) as we need a compatibility transition path.

Until then, the approach provided here is simple and effective.

Advertisements

One thought on “Apache Mesos Leader Master Discovery using Zookeeper

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s