CloudMan's features and uses
This page talks about CloudMan, its features, and potential uses. CloudMan is a cloud manager that orchestrates and manages a cloud infrastructure allowing one to simply use the underlying infrastructure. It is primarily being used in the context of Galaxy Cloud and CloudBioLinux but it can be used for any purpose where a cluster in a cloud is desired. Read on about descriptions of specific features.

In case you're interested in getting a jump start on working with Galaxy and CloudMan, the following is a set of links that you should inspect:

  1. The best place to start is the Galaxy CloudMan wiki, especially if you are only looking to use Galaxy or CloudMan: https://wiki.galaxyproject.org/CloudMan/
  2. If you are interested in learning how things actually operate behind the pretty interfaces, we have a few papers: http://usecloudman.org/publications
  3. If you're interested in deploying on a local/private cloud, CloudBioLinux provides most of the automation infrastructure to to get the necessary components built: https://github.com/chapmanb/cloudbiolinux/tree/master/contrib/flavor/cloudman

The above list should be considered a starting point in getting familiar with the overal system. Whenever you have a question, please shoot us an email. More about emailing the Galaxy community can be found at https://wiki.galaxyproject.org/MailingLists

 

CloudMan startup procedure

Added: 06 Mar 2013

When a CloudMan-enabled machine image is launched, the launched instance goes through a contextualization process. The end result of this process is start of the CloudMan application. The process proceeds as follows:

  1. cloudman upstart job runs (at launch level 2, 3, 4, or 5). The job definition is specified in /etc/init/cloudman.conf. There is no log output from running this script.
  2. ec2autorun.py script is executed next. This script is embeded in the image and is typically found in /usr/bin but some older imaged may have it  /opt/galaxy/pkg/bin/. The log output from this script is stored in the same directory as the script itself, called ec2autorun.log. This script processes the user data, downloads cm_boot.py script from either the cluster's bucket (if the bucket exists) or from the default bucket and runs the script.
  3. /tmp/cm/cm_boot.py runs next. This script prepares the instance for CloudMan, starts the web server, downloads CloudMan (from the cluster's bucket or the default bucket) and then launches CloudMan. The log output for this script is in /tmp/cm/cm_boot.py.log.
  4. Finally, CloudMan starts. CloudMan runs in /mnt/cm and its log is stored in /mnt/cm/paster.log.

And there it is, a functional cluster in the cloud. 

Ready to use machine images for using CloudMan cluster platforms exist on AWS and NeCTAR clouds (see BioCloudCentral.org about starting one). However, if you have a custom machine image and would like to use CloudMan with it, that is also possible. CloudMan can be installed on any (Ubuntu) machine image and thus turn an instance of that machine image into a cluster on the cloud instance. More so, the process of installing CloudMan on a custom machine image is automated and documented.

To get started, clone the set of mi-deployment scripts from https://bitbucket.org/afgane/mi-deployment/ to your local machine:

$ hg clone https://bitbucket.org/afgane/mi-deployment/
$ cd mi-deployment

Before actually running the scripts, make sure your system environment is properly setup: mi-deployment scripts require fabric and boto, so if you do not already have those installed on your local machine, do so with:

$ pip install fabric
$ pip install boto

Next, create the following file with your cloud credentials:

$ cat ~/.boto
[Credentials]
aws_access_key_id = <your cloud access key>
aws_secret_access_key = <your cloud secret key>

Lastly, define which applications you would like to have installed on your machine image by editing mi-deployment/conf_files/apps.yaml and (un)commenting any applications whose dependencies you want or do not want installed (this is primarily cloudman but others are also supported). 

The local environment is now properly setup, so start an instance of your machine image and simply start the mi-deployment configuration script. From your local machine, run:

$ fab -f mi_fabfile.py -i <key_file> -H <instance_IP> configure_MI

After the configuration step completes (typically, 6-8 minutes if CloudMan is the only application your're adding), bundle the instance into a new machine image by running the following command:

$ fab -f mi_fabfile.py -i <key_file> -H <instance_IP> create_image

And there it is. A new machine image with all of your customizations plus all the features of the CloudMan platform is ready to be used.