Development, Analysis And Research

Updated bash script 23/01/2010: The script was assumed to run from EC2 itself, however I have since modified this so its applicable to local environments and made a little more robust.

There are quite a number of new tools from EC2, each requiring some form of setup on the server. As a result I have created a bash script to install them automatically.

• AMI Tools
• API Tools
• Elastic Load Balancing Tools
• Cloud Watch Tools

The only prerequisite is to install the certificate and private key within ~/.ec2/pk.pem and ~/.ec2/cert.pem and Java if on Fedora.

I have tested this on Debian and Fedora, which uses either yum or apt to execute the install of some dependencies, although it does assume that you are running under root.

You can download it from here ec2.sh

#!/bin/bash

DEBUG=1;

if [ "$(id -u)" != "0" ]; then
  echo "This script must be run as root" 1>&2
  exit 1
fi

GREP="`which grep`";
if which apt-get >/dev/null; then
  PACKAGE_MANAGEMENT="`which apt-get` "
else
  PACKAGE_MANAGEMENT="`which yum`"
fi

if which dpkg >/dev/null; then
  PACKAGE_TEST="dpkg --get-selections | $GREP -q "
else
  PACKAGE_TEST="rpm -qa | $GREP -q "
fi

CURL_OPTS=" --silent --retry 1 --retry-delay 1 --retry-max-time 1 "

function log
{
  if [ "$DEBUG" -eq 1 ]; then
    echo $1;
  fi;
}

function bail
{
  echo -e $1;
  exit 1;
}

function check_env
{
  if [ -f ~/.bashrc ]; then
    . ~/.bashrc;
  fi

  # Tools already exist
  if [ -z `which ec2-describe-instances` ] || [ -z `which ec2-upload-bundle` ] || [ -z `which ec2-describe-instances` ] || [ -z `which elb-create-lb` ] || [ -z `which mon-get-stats` ] || [ -z `which as-create-auto-scaling-group` ]; then
    log "Amazon EC2 toolkit missing!"
    install_ec2;
  fi

  # EC2_HOME set
  if [ -z "$EC2_HOME" ]; then
    log "Amazon EC2 is not set-up correctly! EC2_HOME not set"
    if ! grep EC2_HOME ~/.bashrc; then
      echo "export EC2_HOME=/usr/local/ec2-api-tools/" >> ~/.bashrc
    fi;
    export EC2_HOME=/usr/local/ec2-api-tools/
    source ~/.bashrc
  fi

  # Java
  if [ -z "$JAVA_HOME" ]; then

    if grep -i yum "$PACKAGE_MANAGEMENT" > /dev/null; then
      bail "nPlease install java manually (do not use yum install java, it is incompatible)nsee JRE http://java.sun.com/javase/downloads/index.jspnDownload, run the bin file, place in /opt/ and update ~/.bashrc. Once complete run 'source ~/.bashrc;'";
    fi;

    $PACKAGE_MANAGEMENT install -y sun-java6-jdk
    JAVA_PATH=/usr/lib/jvm/java-6-sun/jre/;

    echo "export JAVA_HOME=$JAVA_PATH" >> ~/.bashrc
    export JAVA_HOME=$JAVA_PATH
    source ~/.bashrc
  fi

  # Keys
  EC2_HOME_DIR='.ec2';
  EC2_PRIVATE_KEY_FILE="$HOME/$EC2_HOME_DIR/pk.pem";
  EC2_CERT_FILE="$HOME/$EC2_HOME_DIR/cert.pem";

  if [ ! -d "$HOME/$EC2_HOME_DIR" ]; then
    mkdir -pv "$HOME/$EC2_HOME_DIR";
  fi

  install_ec2_env EC2_PRIVATE_KEY "$EC2_PRIVATE_KEY_FILE";
  install_ec2_env EC2_CERT        "$EC2_CERT_FILE";

  install_ec2_keys_files "$EC2_PRIVATE_KEY_FILE" "Private key";
  install_ec2_keys_files "$EC2_CERT_FILE" "Certificate";

  install_ec2_env AWS_AUTO_SCALING_HOME "/usr/local/ec2-as-tools/"
  install_ec2_env AWS_ELB_HOME          "/usr/local/ec2-elb-tools/"
  install_ec2_env AWS_CLOUDWATCH_HOME   "/usr/local/ec2-cw-tools/"

  get_region
  get_availability_zone

}

function install_ec2_env
{
  # Variable Variable for $1
  EC2_VARIABLE=${!1};
  EC2_VARIABLE_NAME=$1;
  EC2_FILE=$2;

  #log "VARIABLE: $EC2_VARIABLE_NAME=$EC2_VARIABLE";

  # Variable Variable
  if [ -z "$EC2_VARIABLE" ]; then
    log "Amazon $EC2_VARIABLE_NAME is not set-up correctly!";

    if ! grep -q "$EC2_VARIABLE_NAME" ~/.bashrc > /dev/null; then
      echo "export $EC2_VARIABLE_NAME=$EC2_FILE" >> ~/.bashrc;
    fi;
    export $EC2_VARIABLE_NAME=$EC2_FILE;
    source ~/.bashrc
  else
    if ! grep -q "$EC2_VARIABLE_NAME" ~/.bashrc > /dev/null; then
      echo "export $EC2_VARIABLE_NAME=$EC2_FILE" >> ~/.bashrc;
    else
      log "Amazon $EC2_VARIABLE_NAME var installed";
    fi;
  fi
}

function install_ec2_keys_files
{
  EC2_FILE=$1;
  EC2_DESCRIPTION=$2;
  EC2_CONTENTS='';

  if [ ! -f "$EC2_FILE" ]; then
    bail "Amazon $EC2_FILE does not exist, please copy your $EC2_DESCRIPTION to $EC2_FILE and re-run this script";
  else
    log "Amazon $EC2_FILE file installed";
  fi
}

function install_ec2
{

  for PACKAGE in curl wget tar bzip2 unzip zip symlinks unzip ruby; do
    if ! which "$PACKAGE" >/dev/null; then
      $PACKAGE_MANAGEMENT install -y $PACKAGE;
    fi
  done;

  # AMI Tools
  if [ -z "`which ec2-upload-bundle`" ]; then
    curl -o /tmp/ec2-ami-tools.zip $CURL_OPTS --max-time 30 http://s3.amazonaws.com/ec2-downloads/ec2-ami-tools.zip
    rm -rf /usr/local/ec2-ami-tools;
    cd /usr/local && unzip /tmp/ec2-ami-tools.zip
    ln -svf `find . -type d -name ec2-ami-tools*` ec2-ami-tools
    chmod -R go-rwsx ec2* && rm -rvf /tmp/ec2*
  fi

  # API Tools
  if [ -z "`which ec2-describe-instances`" ]; then
    log "Amazon EC2 API toolkit is not installed!"
    curl -o /tmp/ec2-api-tools.zip $CURL_OPTS --max-time 30 http://s3.amazonaws.com/ec2-downloads/ec2-api-tools.zip
    rm -rf /usr/local/ec2-api-tools;
    cd /usr/local && unzip /tmp/ec2-api-tools.zip
    ln -svf `find . -type d -name ec2-api-tools*` ec2-api-tools
    chmod -R go-rwsx ec2* && rm -rvf /tmp/ec2*
  fi

  # ELB Tools
  if [ -z "`which elb-create-lb`" ]; then
    curl -o /tmp/ec2-elb-tools.zip $CURL_OPTS --max-time 30 http://ec2-downloads.s3.amazonaws.com/ElasticLoadBalancing-2009-05-15.zip
    rm -rf /usr/local/ec2-elb-tools;
    cd /usr/local && unzip /tmp/ec2-elb-tools.zip
    mv ElasticLoadBalancing-1.0.3.4 ec2-elb-tools-1.0.3.4;
    ln -svf `find . -type d -name ec2-elb-tools*` ec2-elb-tools
    chmod -R go-rwsx ec2* && rm -rvf /tmp/ec2*
  fi

  # Cloud Watch Tools
  if [ -z "`which mon-get-stats`" ]; then
    curl -o /tmp/ec2-cw-tools.zip $CURL_OPTS --max-time 30 http://ec2-downloads.s3.amazonaws.com/CloudWatch-2009-05-15.zip
    rm -rf /usr/local/ec2-cw-tools;
    mv -v CloudWatch-1.0.2.3 ec2-cw-tools-1.0.2.3
    cd /usr/local && unzip /tmp/ec2-cw-tools.zip
    ln -svf `find . -type d -name ec2-cw-tools*` ec2-cw-tools
    chmod -R go-rwsx ec2* && rm -rvf /tmp/ec2*
  fi

  if [ -z "`which as-create-auto-scaling-group`" ]; then
    curl -o /tmp/ec2-as-tools.zip $CURL_OPTS --max-time 30 http://ec2-downloads.s3.amazonaws.com/AutoScaling-2009-05-15.zip
    rm -rf /usr/local/ec2-as-tools;
    mv -v AutoScaling-1.0.9.0 ec2-as-tools-1.0.9.0
    cd /usr/local && unzip /tmp/ec2-as-tools.zip
    ln -svf `find . -type d -name ec2-as-tools*` ec2-as-tools
    chmod -R go-rwsx ec2* && rm -rvf /tmp/ec2*
  fi

  ln -sf /usr/local/ec2-api-tools/bin/* /usr/bin/;
  ln -sf /usr/local/ec2-ami-tools/bin/* /usr/bin/;
  ln -sf /usr/local/ec2-elb-tools/bin/* /usr/bin/;
  ln -sf /usr/local/ec2-cw-tools/bin/* /usr/bin/;
  ln -sf /usr/local/ec2-as-tools/bin/* /usr/bin/;

  rm -f /usr/bin/ec2-*.cmd;

}

function get_availability_zone
{
  # Not reliable between availability zones using meta-data
  # export EC2_AVAILABILITY_ZONE="`curl $CURL_OPTS --max-time 2 http://169.254.169.254/2009-04-04/meta-data/placement/availability-zone`"

  get_instance_id;

  if [ ! -z "$EC2_INSTANCE_ID" ]; then
    EC2_AVAILABILITY_ZONE="`ec2-describe-instances | grep -q $EC2_INSTANCE_ID | awk '{print $11}'`"
    if [ -z "$EC2_AVAILABILITY_ZONE" ] && [ ! "$EC2_AVAILABILITY_ZONE"="" ]; then
      export EC2_AVAILABILITY_ZONE=$EC2_AVAILABILITY_ZONE;
      install_ec2_env EC2_AVAILABILITY_ZONE $EC2_AVAILABILITY_ZONE;
    fi;
  fi;
}

function get_region
{
  get_instance_id;
  if [ ! -z "$EC2_INSTANCE_ID" ]; then
    EC2_REGION="`ec2-describe-instances | grep $EC2_INSTANCE_ID | awk '{print $11}'`"
    if [ -z "$EC2_REGION" ]; then
      export EC2_REGION=$EC2_REGION;
      install_ec2_env EC2_REGION $EC2_REGION;
      install_ec2_env EC2_URL "https://ec2.$EC2_REGION.amazonaws.com" | sed 's/a.amazonaws.com/.amazonaws.com/'
    fi;
  fi;
}

function get_instance_id
{
  instanceId="`curl $CURL_OPTS --max-time 2 http://169.254.169.254/1.0/meta-data/instance-id`"
  if [ ! -z "$instanceId" ]; then
    export EC2_INSTANCE_ID="$instanceId";
  fi;
}

check_env

I recently came across a peculiar issue that meant dates and times were causing issues with a product we had developed within Australia. The issue being that within “Red Hat Enterprise Linux Server release 5 (Tikanga)” the date within PHP was being read as EST instead of AEST/AEDT, however running “date” from the terminal or running “SELECT NOW()” from MySQL displayed the correct time.

[user@server ~]$ date
Wed Oct 14 22:24:20 EST 2009

[user@server ~]$ php -r'var_dump(date("r"));'
string(51) "Wed, 14 Oct 2009 21:25:07 +1000 Australia/Melbourne"

[user@server ~]$ php -r'var_dump(date("r e"));var_dump(getenv("TZ"));var_dump(ini_get("date.timezone"));var_dump(date_default_timezone_get());';
string(51) "Wed, 14 Oct 2009 21:25:07 +1000 Australia/Melbourne"
bool(false)
string(0) ""
string(19) "Australia/Melbourne"

[user@server ~]$ mysql -uuser -ppassword -e 'SELECT NOW();'
+---------------------+
| NOW()               |
+---------------------+
| 2009-10-14 22:26:12 |
+---------------------+

As you can see php incorrectly gets the time, being an hour off. Running the above on debian worked perfectly fine and comparing the zoneinfo matched my local machine.


[user@server ~]$ md5sum /etc/localtime && md5sum /usr/share/zoneinfo/Australia/Sydney && md5sum /usr/share/zoneinfo/Australia/Melbourne
85285c5495cd5b8834ab62446d9110a9 /etc/localtime
85285c5495cd5b8834ab62446d9110a9 /usr/share/zoneinfo/Australia/Sydney
8a7f0f78d5a146db4bf865ca91cc1c42 /usr/share/zoneinfo/Australia/Melbourne


After a fair amount of digging I ended up coming across the following ticket @478566. Amazingly the ticket is marked as “CLOSED WONTFIX”.

There were a few interesting points from some of the conversations I read.

” Alphabetic time zone abbreviations should not be used as unique identifiers for UTC offsets as they are ambiguous in practice. For example, “EST” denotes 5 hours behind UTC in English-speaking North America, but it denotes 10 or 11 hours ahead of UTC in Australia; and French-speaking North Americans prefer “HNE” to “EST”. twinsun”

Due to different locations in Australia having various interpretations of summer time with start/end dates and clock shifts. As well as the operating system not having zoneinfo data for DEST, AEDT etc (unless you create these yourself) it means you cannot rely on the correct time from php on redhat.

So far I have resorted to the following

[user@server ~]$ php -r 'date_default_timezone_set("Etc/GMT-11"); var_dump(date("r"));';
string(31) "Wed, 14 Oct 2009 22:24:29 +1100"