Getting Started on Hydra

Dependencies

Building

1. Checkout from Github
2. RabbitMQ

As described in the Hydra Readme.mdown,

Hydra uses rabbitmq for low volume commands and control message exchanges. On a modern Linux systems apt-get install rabbitmq-server and running with the default settings is adequate in most cases.

For Mac OS (my environment), I used Homebrew to install it. Super easy to setup Homebrew if you haven’t before, just open a Terminal and run:

ruby -e "$(curl -fsSL https://raw.github.com/Homebrew/homebrew/go/install)"

Following the instructions given by RabbitMQ, you double-check your brew is up-to-date ($ brew update), then install it with $ brew install rabbitmq.

Quick Note: RabbitMQ scripts are installed into your /usr/local/sbin directory. This is likely not on your PATH (check with echo $PATH), so to add it just edit the .bash_profile or .profilein your home directory to add or modify export PATH=$PATH:/usr/local/sbin.

The server can then be started with rabbitmq-server, but it will take over your window. You can also run rabbitmq-server -detached to have it in the background.

To stop the server, run rabbitmqctl stop. rabbitmqctl status will tell you if it is running.

3. FL-COW

For a single-node installation, FL-COW isn’t necessary and this step can be skipped.

fl-cow is used because hard links (think shortcuts) exist within the replicated data, and we don’t want to muck up the originals while mucking with a copy. So it performs a COW (Copy On Write). Installation instructions were tricky to find, so here they are:

  1. Set the FLCOW_PATH (the same way as for rabbitmq). This will control which paths are subject to COWing, so point it to your working data directory or directories (comma-separated): export FLCOW_PATH=/Users/ben/dev/hydra/data
  2. Also set export LD_PRELOAD=$LD_PRELOAD:/usr/lib/libflcow.so
  3. Run the following in your fl-cow directory to build the libflcow.so representing your FLCOW_PATH:
$ ./configure --prefix=/usr
$ make
$ make check
$ su
$ make install

To test that everything is ready, run $ LD_PRELOAD=./fl-cow/.libs/libflcow.so ./test/flcow-test. Or : 

$ make check
4. SSH

You’ll need to be able to ssh to your machine. See this separate tutorial if you need.

5. coreutils & wget

You will need both of these installed. Here I used Homebrew again:

$ brew install wget $ brew install coreutils
 
6. Build it with Maven

Using Maven, go to your $HYDRA_HOME in a terminal, and run:

$ mvn -Pbdbje package
7. Fire up a Local Stack

There is a script in hydra-uber/bin called local-stack.sh. This must be run from $HYDRA_HOME.

  • ./hydra-uber/bin/local-stack.sh start – start ZooKeeper
  • ./hydra-uber/bin/local-stack.sh start – start spawn, querymaster etc.
  • ./hydra-uber/bin/local-stack.sh seed – add some sample data
8. Test it Out

Now you can go to http://localhost:5052/ and see the Spawn interface.

When done ./hydra-uber/bin/local-stack.sh stop will stop everything except ZooKeeper, and running stop a second time will bring that process down as well.

More to come on jobs and querying!