Development, Analysis And Research

I just read “How to use locks in PHP cron jobs to avoid cron overlaps” and I thought I would elaborate on this and provide some more examples. In order for a lock to work correctly it must handle, Atomicity / Race Conditions, and Signaling.

I use the following bash script to create locks for crontabs and ensure single execution of scripts.

“The clever bit is to get a lock file test and creation (if needed) to be atomic, that is done without interruption. The set -C stops a redirection from over writing a file. The : > touches a file. In combination, the effect is, when the lock file exists, the redirection fails and exits with an error. If it does not exist, the redirection creates the lock file and exits without an error.The final part is to make sure that the lock file is cleaned up. To makes sure it is removed even if the script is terminated with a ctrl-c, a trap is used. Simply, when the script exits, the trap is run and the lock file is deleted.”, The Lab Book Pages

In addition it also checks the process list and tests whether the pid within the lock file is active.

#!/bin/bash

LOCK_FILE=/tmp/my.lock
CRON_CMD="php /var/www/..../fork.php -t17"

function check_lock {
    (set -C; : > $LOCK_FILE) 2> /dev/null
    if [ $? != "0" ]; then
        RUNNING_PID=$(cat $LOCK_FILE 2> /dev/null || echo "0");
        if [ "$RUNNING_PID" -gt 0 ]; then
            if [ `ps -p $RUNNING_PID -o comm= | wc -l` -eq 0 ]; then
                echo "`date +'%Y-%m-%d %H:%M:%S'` WARN [Cron wrapper] Lock File exists but no process running $RUNNING_PID, continuing";
            else
                echo "`date +'%Y-%m-%d %H:%M:%S'` INFO [Cron wrapper] Lock File exists and process running $RUNNING_PID - exiting";
                exit 1;
            fi
        else
            echo "`date +'%Y-%m-%d %H:%M:%S'` CRIT [Cron wrapper] Lock File exists with no PID, wtf?";
            exit 1;
        fi
    fi
    trap "rm $LOCK_FILE;" EXIT
}

check_lock;
echo "`date +'%Y-%m-%d %H:%M:%S'` INFO [Cron wrapper] Starting process";
$CRON_CMD &
CURRENT_PID=$!;
echo "$CURRENT_PID" > $LOCK_FILE;
trap "rm -f $LOCK_FILE 2> /dev/null ; kill -9 $CURRENT_PID 2> /dev/null;" EXIT;
echo "`date +'%Y-%m-%d %H:%M:%S'` INFO [Cron wrapper] Started ($CURRENT_PID)";
wait;
# remove the trap kill so it won't try to kill process which took place of the php one in mean time (paranoid)
trap "rm -f $LOCK_FILE 2> /dev/null" EXIT;
rm -f $LOCK_FILE 2> /dev/null;
echo "`date +'%Y-%m-%d %H:%M:%S'` INFO [Cron wrapper] Finished process";

With the implementation described in the post at abhinavsingh.com, it will fail if you put it as a background process as an example see below.

andrew@andrew-home:~/tmp.lock$ php x.php
==16169== Lock acquired, processing the job...
^C
andrew@andrew-home:~/tmp.lock$ php x.php
==16169== Previous job died abruptly...
==16170== Lock acquired, processing the job...
^C

andrew@andrew-home:~/tmp.lock$ php x.php
==16170== Previous job died abruptly...
==16187== Lock acquired, processing the job...
^Z
[1]+  Stopped                 php x.php
andrew@andrew-home:~/tmp.lock$ ps aux | grep php
andrew   16187  0.5  0.5  50148 10912 pts/2    T    09:53   0:00 php x.php
andrew   16192  0.0  0.0   3108   764 pts/2    R+   09:53   0:00 grep --color=auto php
andrew@andrew-home:~/tmp.lock$ php x.php
==16187== Already in progress...

You can use pcntl_signal to trap interruptions to the application and handle cleanup of the process. Here is a slightly modified implementation to handle cleanup. Just to highlight the register_shutdown_function will not help to cleanup on any signal/interruption.

<?php
class lockHelper {

	protected static $_pid;

	protected static $_lockDir = '/tmp/';

	protected static $_signals = array(
		// SIGKILL,
		SIGINT,
		SIGPIPE,
		SIGTSTP,
		SIGTERM,
		SIGHUP,
		SIGQUIT,
	);

	protected static $_signalHandlerSet = FALSE;

	const LOCK_SUFFIX = '.lock';

	protected static function isRunning() {
		$pids = explode(PHP_EOL, `ps -e | awk '{print $1}'`);
		return in_array(self::$_pid, $pids);
	}

	public static function lock() {
		self::setHandler();

		$lock_file = self::$_lockDir . $_SERVER['argv'][0] . self::LOCK_SUFFIX;
		if(file_exists($lock_file)) {
			self::$_pid = file_get_contents($lock_file);
			if(self::isrunning()) {
				error_log("==".self::$_pid."== Already in progress...");
				return FALSE;
			}
			else {
				error_log("==".self::$_pid."== Previous job died abruptly...");
			}
		}

		self::$_pid = getmypid();
		file_put_contents($lock_file, self::$_pid);
		error_log("==".self::$_pid."== Lock acquired, processing the job...");
		return self::$_pid;
	}

	public static function unlock() {
		$lock_file = self::$_lockDir . $_SERVER['argv'][0] . self::LOCK_SUFFIX;
		if(file_exists($lock_file)) {
			error_log("==".self::$_pid."== Releasing lock...");
			unlink($lock_file);
		}
		return TRUE;
	}

	protected static function setHandler() {
		if (!self::$_signalHandlerSet) {

			declare(ticks = 1);

			foreach(self::$_signals AS $signal) {
				if (!pcntl_signal($signal, array('lockHelper',"signal"))) {
					error_log("==".self::$_pid."== Failed assigning signal - '{$signal}'");
				}
			}
		}
		return TRUE;
	}

	protected static function signal($signo) {
		if (in_array($signo, self::$_signals)) {
			if(!self::isrunning()) {
				self::unlock();
			}
		}
		return FALSE;
	}
}

As an example:

andrew@andrew-home:~/tmp.lock$ php t.php
==16268== Lock acquired, processing the job...
^Z==16268== Releasing lock...

Whilst the implementation above simply uses files, it could be implemented with shared memory (SHM/APC), distributed caching (memcached), or a database. If over a network, factors such as packet loss, latency etc can cause race conditions and should be taken into account. Depending on the application it maybe better to implement as a daemon. If your looking to distribute tasks amongst servers, take a look at Gearman

• supervisord
• Create daemons in PHP

Last year I wrote an application to highlight media outlets and their reach (coverage of media outlets), selecting regions within the UK and highlighting aspects of a map. This had many issues where by hitting performance problems of rendering within browsers and also limitations of converting KML to tiles via google. A list of these limitations are:

1. Timeouts from google on large KML files.
2. Responsiveness of servers to deliver KML files to google.
3. Max KML size (Even when gzipped)
4. 500 Errors from google
5. Transparency within IE
6. ….

Some of these limits have since been increased by google and are documented.

Maximum fetched file size (raw KML, raw GeoRSS, or compressed KMZ)	3MB
Maximum uncompressed KML file size	10MB
Maximum number of Network Links	10
Maximum number of total document-wide features	1,000

In order to alleviate these issues I ended up with the following

• Caching KML files to avoid latency on a expensive database lookups/response.
• Chunking the response into 250 records and writing to individual static KML files. (Files would become very large and google would time out retrieving data sets).
• Proxying googles tiles after they had been converted from KML to images and caching them locally on our servers and then applying the overlays from our servers once merged

So depending on the depth (zoom) of the map and the area selected as well the volume of data, it would either use tiles or googles KML directly (Increased functionality).

In order to have greater control over the spatial data within our database we split this into areas, regions, and sub_regions, which held lookups to postcodes, towns and spatial data itself (There are a lot of discrepancies over outlines of maps).

Left hand menu:

<ul style="display: block;">
	<li id="East"><a href="#" onclick="loadTilesFromGeoXML('|1|'); return false;">East</a>
		<ul style="display: none;">
			<li><a href="#" onclick="loadTilesFromGeoXML('|1|6'); return false;">Bedfordshire</a></li>
			<li><a href="#" onclick="loadTilesFromGeoXML('|1|18'); return false;">Cambridgeshire</a></li>
			...
		</ul>
	</li>
</ul>

Javascript to locate tiles

  function loadTilesFromGeoXML(entity_id) {
    // Matches database record ids that are mapped to spatial data within MySQL
    mapTownsId = entity_id.toString().split('|')[0];
    mapRegionsId = entity_id.toString().split('|')[1];
    mapSubRegionsId = entity_id.toString().split('|')[2];
    locationUrl ='map_towns_id='+mapTownsId+'&map_regions_id='+mapRegionsId+'&map_sub_regions_id='+mapSubRegionsId;

    var cc = map.fromLatLngToDivPixel(map.getCenter());
    map.setZoom(1);

    // Request URL to cached titles links
    geoXMLUrl = '/ajax/mapping/get/overlays/region?'+locationUrl;
    geoXMLUrl+='&format=JSON&method=getLinks&x='+cc.x+'&y='+cc.y+'&zoom='+map.getZoom();

    // tileUrlTemplate: 'http://domain.com/maps/proxy/regions/?url=http%3A%2F%2Fdomain.com/ajax/mapping/get/cache/?filename=.1.6.0&x={X}&y={Y}&zoom={Z}',

    $.getJSON(geoXMLUrl, function(data) {
      $.each(data, function(i,link) {
        kmlLinks+=encodeURIComponent(link)+',';
      });

      // Builds the location for tiles to be mapped
      tileUrlTemplate = '/maps/proxy/regions/?url='+kmlLinks+'&x={X}&y={Y}&zoom={Z}';
      var tileLayerOverlay = new GTileLayerOverlay(
        new GTileLayer(null, null, null, {
          tileUrlTemplate: tileUrlTemplate,
          isPng:true,
          opacity:1.0
        })
      );
      if (debug) GLog.writeUrl('/maps/proxy/regions/?url='+kmlLinks+'&x={X}&y={Y}&zoom={Z}');
      map.addOverlay(tileLayerOverlay);
    });
  }

Response whilst retrieving links (if cached)

The code behind this simply caches the KML files, if it does not exist, otherwise attempts to create it and also outputs a json request with the files matching the sequence and globs for any files with a similar pattern, all files are suffixed with their page number.

["/ajax/mapping/get/cache/?filename=.1..0&x=250&y=225&zoom=5","/ajax/mapping/get/cache/?filename=.1..1&x=250&y=225&zoom=5"]

Proxying googles tiles and merging the layer ids

    $kmlUrls = urlencode($_GET['url']);
    $cachePath = dirname(__FILE__).'/cache.maps/tiles/';

    $cachedFiles = array_filter(explode(',',rawurldecode($kmlUrls)));
    $hash = sha1(rawurldecode($kmlUrls).".w{$_GET['w']}.h{$_GET['h']}.x{$_GET['x']}.y{$_GET['y']}.{$_GET['zoom']}");
    $cachePath.="{$_GET['x']}.{$_GET['y']}/{$_GET['zoom']}/";
    if (!is_dir($cachePath)) {
      @mkdir($cachePath, 0777, true);
    }

    // Returns image if cached already and aggregated.
    if (file_exists($path = $cachePath.$hash)) {
      header('Content-Type: image/png');
      $fp = fopen($path, 'rb');
      fpassthru($fp);
    }

    // Extract layer id's from KML files that are to be merged.
    $layerIds = array();
    foreach( $cachedFiles AS $kmlFile) {
      $kmlFile="http://{$_SERVER['HTTP_HOST']}{$kmlFile}";

      $url = "http://maps.google.com/maps/gx?q={$kmlFile}&callback=_xdc_._1fsue7g2w";
      @$c = file_get_contents($url);
      if (!$c)
        throw new Exception("Failed to request {$url} - {$c}");
      preg_match_all('/layer_id:"kml:(.*)"/i', $c, $matches);
      if (count($matches)>0 && isset($matches[1][0])) {
        $layerIds[] = "kml:{$matches[1][0]}";
      }
    }

    // Cache locally.
    if (count($layerIds)>0) {
      header('Content-Type: image/png');
      // Aggregate layers into a single image
      $link = "http://mlt0.google.com/mapslt?lyrs=" . implode(',',$layerIds);
      $link.="&x={$_GET['x']}&y={$_GET['y']}&z={$_GET['zoom']}&w={$_GET['w']}&h={$_GET['h']}&source=maps_api";
      echo $c = file_get_contents($link);
      @file_put_contents($path, $c);
    } else {
      // Output 1x1 png
      header('Content-Type: image/png');
      echo base64_decode('iVBORw0KGgoAAAANSUhEUgAAAAEAAAABCAYAAAAfFcSJAAAAC0lEQVQIHWNgAAIAAAUAAY27m/MAAAAASUVORK5CYII=');
    }
  }

Paging GeoXML loading

    function loadGeoXMLPaged(geoXMLUrl) {
      var cc = map.fromLatLngToDivPixel(map.getCenter());
      geoXMLUrl+='&format=JSON&method=getLinks&x='+cc.x+'&y='+cc.y+'&zoom='+map.getZoom();

      if (debug) GLog.writeUrl(geoXMLUrl);

      $.getJSON(geoXMLUrl, function(data) {
	  geoXmlPager = data;
          loadGeoXmlPage();
        });
      }

      var timeoutPID = null;

      function loadGeoXmlPage(){
  	if (data = geoXmlPager.pop()){
	 if (debug)
            GLog.writeUrl(BASE_URL+data);

	 geoXmlStack.push(new GGeoXml(BASE_URL+data));
	 map.addOverlay(geoXmlStack[geoXmlStack.length - 1]);

         GEvent.addListener(geoXmlStack[geoXmlStack.length - 1],"load",function() {
	  timeoutPID = setTimeout("loadGeoXmlPage()", 500);
         });
	}else{
          clearTimeout(timeoutPID);
	  map.setZoom(map.getBoundsZoomLevel(bounds));
          map.setCenter(bounds.getCenter());
          try {
            geoXmlStack[geoXmlStack.length - 1].gotoDefaultViewport(map);
          } catch(e) {}
	}
      }

All the code above has been modified slightly to make it applicable to others, however don’t accept raw input as its simply an example.

I recently came across a peculiar issue that meant dates and times were causing issues with a product we had developed within Australia. The issue being that within “Red Hat Enterprise Linux Server release 5 (Tikanga)” the date within PHP was being read as EST instead of AEST/AEDT, however running “date” from the terminal or running “SELECT NOW()” from MySQL displayed the correct time.

[user@server ~]$ date
Wed Oct 14 22:24:20 EST 2009

[user@server ~]$ php -r'var_dump(date("r"));'
string(51) "Wed, 14 Oct 2009 21:25:07 +1000 Australia/Melbourne"

[user@server ~]$ php -r'var_dump(date("r e"));var_dump(getenv("TZ"));var_dump(ini_get("date.timezone"));var_dump(date_default_timezone_get());';
string(51) "Wed, 14 Oct 2009 21:25:07 +1000 Australia/Melbourne"
bool(false)
string(0) ""
string(19) "Australia/Melbourne"

[user@server ~]$ mysql -uuser -ppassword -e 'SELECT NOW();'
+---------------------+
| NOW()               |
+---------------------+
| 2009-10-14 22:26:12 |
+---------------------+

As you can see php incorrectly gets the time, being an hour off. Running the above on debian worked perfectly fine and comparing the zoneinfo matched my local machine.
[user@server ~]$ md5sum /etc/localtime && md5sum /usr/share/zoneinfo/Australia/Sydney && md5sum /usr/share/zoneinfo/Australia/Melbourne
85285c5495cd5b8834ab62446d9110a9 /etc/localtime
85285c5495cd5b8834ab62446d9110a9 /usr/share/zoneinfo/Australia/Sydney
8a7f0f78d5a146db4bf865ca91cc1c42 /usr/share/zoneinfo/Australia/Melbourne

After a fair amount of digging I ended up coming across the following ticket @478566. Amazingly the ticket is marked as “CLOSED WONTFIX”.

There were a few interesting points from some of the conversations I read.
” Alphabetic time zone abbreviations should not be used as unique identifiers for UTC offsets as they are ambiguous in practice. For example, “EST” denotes 5 hours behind UTC in English-speaking North America, but it denotes 10 or 11 hours ahead of UTC in Australia; and French-speaking North Americans prefer “HNE” to “EST”. twinsun”

Due to different locations in Australia having various interpretations of summer time with start/end dates and clock shifts. As well as the operating system not having zoneinfo data for DEST, AEDT etc (unless you create these yourself) it means you cannot rely on the correct time from php on redhat.

So far I have resorted to the following

[user@server ~]$ php -r ‘date_default_timezone_set(“Etc/GMT-11”); var_dump(date(“r”));’;
string(31) “Wed, 14 Oct 2009 22:24:29 +1100”

I have been migrating a large number of websites and consolidating servers to reduce costs.
As a result it is important to ensure that services are migrated smoothly, planned effectively,
after which I had a think about aspects to consider prior to migrating services.

Planning

• Make a preliminary checklist of services actively in use by each active domain, I.e. FTP, HTTP, SMTP, IMAP, POP3, MySQL etc.
• What maintenance periods do you have available, if at all?
  • What volume of traffic and when are your quietest periods?
• Do you have dedicated infrastructure, sharded, split by service/role?
  • Can parts of the infrastructure be migrated as an individual component
• List core functionality from the domain for testing purposes
  • Ideally this should be wrapped in unit tests as both functional
    • Examples are email, upload (permissions), adding/editing/removing users
• How many servers are you migrating?
  • Large quantities should be automated.
• How critical is the site/service
  • Does it stop 80 staff working?

Specific

• Services
  • Ensure services are initially installed on new server(s).
  • List all configuration files for a particular service (tree).
    • Ensure configuration between each service are identical or compromises are made.
  • List data directories for each service I.e. /var/lib/mysql
    • Can data be transferred automically.
    • Can services be replicated and brought into sync
    • Can data be back filled?
      • I.e Are large log tables required to make the site functional, what is the minimal effort required to bring the site functional?
• SSL
  • Ensure valid certificate exists for any CDN, sub-domain, domain.
• Email
  • Are there any special firewall, configuration requirements?
• DNS
  • Lower the TTL for a domain your preparing to transfer (if possible)
    • Cannot rely on low TTLs, these are cached amongst large corporates, ISPs etc.
  • Ensure the domain is bound to a unique VIP on new servers, if DNS resolution fails, you can put a header(‘Location 10.10.10.10′); in the old site to ensure the domain will resolve correctly.
    • Test this prior to transfer for both HTTP & HTTPS if applicable
• Permissions
  • Do you upload content to the servers, does your code write to the filesystem?
    • Is this writtable?
  • Under which user/group is this written?
• Cache
  • Does your site make use of distributed or local cache?
    • Could there be collisions between different sites, I.e. Do you prefix cache key names based on site?
• Networking
  • Can specific services be migrated prematurely?
    • Repoint via iptables, and keep an eye on bytes passing through the interface till redundant
• Security
  • Were there any firewall restrictions that need to be replicated, either hardware, iptables etc.
  • Chrooted, users copied, ssh keys copied.
• Optimizations
  • Were there any special optimizations, I.e. DnsMasq?, sysctl changes?
• Load balancing
  • Ensure each domain has its own VIP – HTTP_HOST fails in HTTP 1.0 clients
  • Ensure wild cards are not specified within virtual hosts – see above
  • Ensure sites with load balancing and over SSL use TCP requests correctly, in addition see first point.
  • ifdown each VIP in the webserver pool, does it failover with the correct site on all nodes?
• Monitoring
  • If previously had monitoring on servers (should do), has this been replicated to new servers?
• Database (Will vary depending on setup)
  • Is the database replicated?
    • Take LVM snapshots of the raw data on slave and rsync to new servers.
      • Ensure to change configuration such as server id’s, permissions on master, firewall, start service and start replication. Will be ready to start replicating with correct binlog positions etc.
• Other general changes
  • Are there customizations to /etc/hosts get sites working?

Let me know if there is anything you think I have missed.