Development, Analysis And Research

I recently came across a peculiar issue that meant dates and times were causing issues with a product we had developed within Australia. The issue being that within “Red Hat Enterprise Linux Server release 5 (Tikanga)” the date within PHP was being read as EST instead of AEST/AEDT, however running “date” from the terminal or running “SELECT NOW()” from MySQL displayed the correct time.

[user@server ~]$ date
Wed Oct 14 22:24:20 EST 2009

[user@server ~]$ php -r'var_dump(date("r"));'
string(51) "Wed, 14 Oct 2009 21:25:07 +1000 Australia/Melbourne"

[user@server ~]$ php -r'var_dump(date("r e"));var_dump(getenv("TZ"));var_dump(ini_get("date.timezone"));var_dump(date_default_timezone_get());';
string(51) "Wed, 14 Oct 2009 21:25:07 +1000 Australia/Melbourne"
bool(false)
string(0) ""
string(19) "Australia/Melbourne"

[user@server ~]$ mysql -uuser -ppassword -e 'SELECT NOW();'
+---------------------+
| NOW()               |
+---------------------+
| 2009-10-14 22:26:12 |
+---------------------+

As you can see php incorrectly gets the time, being an hour off. Running the above on debian worked perfectly fine and comparing the zoneinfo matched my local machine.


[user@server ~]$ md5sum /etc/localtime && md5sum /usr/share/zoneinfo/Australia/Sydney && md5sum /usr/share/zoneinfo/Australia/Melbourne
85285c5495cd5b8834ab62446d9110a9 /etc/localtime
85285c5495cd5b8834ab62446d9110a9 /usr/share/zoneinfo/Australia/Sydney
8a7f0f78d5a146db4bf865ca91cc1c42 /usr/share/zoneinfo/Australia/Melbourne


After a fair amount of digging I ended up coming across the following ticket @478566. Amazingly the ticket is marked as “CLOSED WONTFIX”.

There were a few interesting points from some of the conversations I read.

” Alphabetic time zone abbreviations should not be used as unique identifiers for UTC offsets as they are ambiguous in practice. For example, “EST” denotes 5 hours behind UTC in English-speaking North America, but it denotes 10 or 11 hours ahead of UTC in Australia; and French-speaking North Americans prefer “HNE” to “EST”. twinsun”

Due to different locations in Australia having various interpretations of summer time with start/end dates and clock shifts. As well as the operating system not having zoneinfo data for DEST, AEDT etc (unless you create these yourself) it means you cannot rely on the correct time from php on redhat.

So far I have resorted to the following

[user@server ~]$ php -r 'date_default_timezone_set("Etc/GMT-11"); var_dump(date("r"));';
string(31) "Wed, 14 Oct 2009 22:24:29 +1100"


I have been migrating a large number of websites and consolidating servers to reduce costs.
As a result it is important to ensure that services are migrated smoothly, planned effectively,
after which I had a think about aspects to consider prior to migrating services.

Planning

• Make a preliminary checklist of services actively in use by each active domain, I.e. FTP, HTTP, SMTP, IMAP, POP3, MySQL etc.
• What maintenance periods do you have available, if at all?
  • What volume of traffic and when are your quietest periods?
• Do you have dedicated infrastructure, sharded, split by service/role?
  • Can parts of the infrastructure be migrated as an individual component
• List core functionality from the domain for testing purposes
  • Ideally this should be wrapped in unit tests as both functional
    • Examples are email, upload (permissions), adding/editing/removing users
• How many servers are you migrating?
  • Large quantities should be automated.
• How critical is the site/service
  • Does it stop 80 staff working?

Specific

• Services
  • Ensure services are initially installed on new server(s).
  • List all configuration files for a particular service (tree).
    • Ensure configuration between each service are identical or compromises are made.
  • List data directories for each service I.e. /var/lib/mysql
    • Can data be transferred automically.
    • Can services be replicated and brought into sync
    • Can data be back filled?
      • I.e Are large log tables required to make the site functional, what is the minimal effort required to bring the site functional?
• SSL
  • Ensure valid certificate exists for any CDN, sub-domain, domain.
• Email
  • Are there any special firewall, configuration requirements?
• DNS
  • Lower the TTL for a domain your preparing to transfer (if possible)
    • Cannot rely on low TTLs, these are cached amongst large corporates, ISPs etc.
  • Ensure the domain is bound to a unique VIP on new servers, if DNS resolution fails, you can put a header(’Location 10.10.10.10′); in the old site to ensure the domain will resolve correctly.
    • Test this prior to transfer for both HTTP & HTTPS if applicable
• Permissions
  • Do you upload content to the servers, does your code write to the filesystem?
    • Is this writtable?
  • Under which user/group is this written?
• Cache
  • Does your site make use of distributed or local cache?
    • Could there be collisions between different sites, I.e. Do you prefix cache key names based on site?
• Networking
  • Can specific services be migrated prematurely?
    • Repoint via iptables, and keep an eye on bytes passing through the interface till redundant
• Security
  • Were there any firewall restrictions that need to be replicated, either hardware, iptables etc.
  • Chrooted, users copied, ssh keys copied.
• Optimizations
  • Were there any special optimizations, I.e. DnsMasq?, sysctl changes?
• Load balancing
  • Ensure each domain has its own VIP – HTTP_HOST fails in HTTP 1.0 clients
  • Ensure wild cards are not specified within virtual hosts – see above
  • Ensure sites with load balancing and over SSL use TCP requests correctly, in addition see first point.
  • ifdown each VIP in the webserver pool, does it failover with the correct site on all nodes?
• Monitoring
  • If previously had monitoring on servers (should do), has this been replicated to new servers?
• Database (Will vary depending on setup)
  • Is the database replicated?
    • Take LVM snapshots of the raw data on slave and rsync to new servers.
      • Ensure to change configuration such as server id’s, permissions on master, firewall, start service and start replication. Will be ready to start replicating with correct binlog positions etc.
• Other general changes
  • Are there customizations to /etc/hosts get sites working?

Let me know if there is anything you think I have missed.

The Project

I was recently working on a project to expose our trading systems via XmlRpc, Rest and SOAP. It was quite an interesting project, which took two of us three weeks to develop (Amongst other things).

This involved creating a testbed, that would automatically generate the payload and response for each protocol. The parameters are introspected for each class method capturing each parameters data type, allowing for user input via standard html forms. This is probably best described with a picture or two.

Most of the documentation was generated via reflection and comments within the docblocks, parameters, notes were also generated making it quick and simple to update. In addition to parsing the start and end line of each method for any applicable error codes/faults that may be returned.

Zend Framework

Using the Zend Framework for the first time in a commercial product was not exactly hassle free, and still has quite a few issues with its webservices implementation. Currently there seems to be quite a bit of confusion regarding its Rest implementation and whether it is to be merged, would be great if someone clarify this.

The main issue I found with the Zend Frameworks implementation of XmlRpc and Rest is that it assumes that the payload it receives is valid. During my development, I tended to mix the payloads from SOAP, XmlRpc and Rest, yet it would assume that simple_xml can parse the input.

For example $this->_sxml is assumed to be a valid object, if not you will either get invalid method call or an undefined index, which doesn’t render well for an xmlrpc server.

    /**
     * Constructor
     *
     * @param string $data XML Result
     * @return void
     */
    public function __construct($data)
    {
        $this->_sxml = simplexml_load_string($data);
    }

    /**
     * toString overload
     *
     * Be sure to only call this when the result is a single value!
     *
     * @return string
     */
    public function __toString()
    {
        if (!$this->getStatus()) {
            $message = $this->_sxml->xpath('//message');
            return (string) $message[0];
        } else {
            $result = $this->_sxml->xpath('//response');
            if (sizeof($result) > 1) {
                return (string) "An error occured.";
            } else {
                return (string) $result[0];
            }
        }
    }

One of the main issues with Rest was that it needed ksort when using the Rest client as the arguments were not necessarily passed in order. This can be “rest.php?method=x&arg1=1&arg0=0″ and it would interpret each arg in the order it received them. This should be sorted in the next release of the ZF.

As the webservices we are exposing needs to have quite good performance with the number of transactions it will be handling and the amount of reflection that Zend Server Reflection (Only noticed after I started profiling) performs and I wanted to optimize any overhead, which got me looking at Zend_XmlRpc_Server_Cache. First thing I did was profile Zend_XmlRpc_Server_Cache, which added a considerable amount of overhead. Looking at its implementation, it uses serialize, which is a relatively slow process and should be avoided, unless there is a large overhead in initializing objects. So most likely Zend_XmlRpc_Server_Cache will not add any benefit. And var_dump’ing out the reflection in XmlRpc spews out a shocking amount of information on some fairly large classes.

  if (!Zend_XmlRpc_Server_Cache::get($cacheFile, $server)) {

  }

Generating WSDL

I tried a number of WSDL generators including the implementation in incubator for ZF, which I found to be the best, yet I still had to write a large chunk of the WSDL by hand and adapt it.

The best way to debug is to run the soap client with verbose mode on, and it will typically tell you the issue straight away.

• Zend_Soap_AutoDiscover: Duplicates an operation in WSDL for methods with parameters that are optional. (ZF-2642)
• Zend_Soap_AutoDiscover: If missing the @return in your docblock the message response in the WSDL is not generated. (ZF-2643)
• AutoDiscover duplicates response if using set class multiple times. (ZF-2641 )
• One of my colleagues typically writes their docblocks with “@return int, comment.”, which the comma caused return types to be dropped with AutoDiscover, more of an issue with Zend Server Reflection

Other odd issues

Raw input bug

Some other obscurities I found was capturing the raw request data. In our local development environment reading the raw request input, and then once again within the Zend Frameworks appears to work fine. However in our pre-production environment it fails to read the second request to read the raw request. (PHP 5.2.2)


if (!isset($HTTP_RAW_POST_DATA)){
$HTTP_RAW_POST_DATA = file_get_contents('php://input');
}


It does seem a little odd that the XmlRpc does not check whether $HTTP_RAW_POST_DATA isset before attempting to re-read raw input.

Internal error: Wrong return type

Whilst running PHPUnit I noticed a very weird quirk in our local dev environment, which essentially did the following… You would expect this to output the contents of an array right? Well between the method call to x and return the result back to method y returns NULL. This is very obscure and i’ve never seen anything like it especially considering it is explicitly set. I had a number of colleagues check this, which had us all scratching our heads. Has anyone else seen anything similar to this?

class test {

  public function x() {
    $ret = array();
    for(...) {
      $ret[] = $row;
    }
    return $ret;
  }

  public function y() {
    $response = $this->x();
    var_dump($response);
  }
}

$t = new test();
$t->y();

Conclusion

Overall the project went pretty well, I’m confident it is now stable especially with the number of tests we ran against it. It is adaptable to other projects that we may need to expose via an API, in total there is about 6000 lines of code alone just testing the 3 different protocols it supports. I would have rather avoided the Rest implementation with ZF as it still needs a lot of work, however XmlRpc is a lot more stable and I would quite happily use again. As there is a lot of overhead with reflection it is not the fastest implementation and was contrasted to some of the heavier web pages we have for some simple functionality. It would be ideal to replace the reflection with something lighter such as an array with the corresponding methods, parameters and types, however I would have to look into that if performance did become a major issue.

PS. Just to note I used PHP’s in built soap server.

I’ve been using Zend Studio for Eclipse (beta) for several weeks in a rewrite of a framework and numerous sites at work and overall I really like the IDE. Its got some great features and being based on the eclipse project makes it really extensible and customizable. With debugging, profiling, code completion, code formatting and more enabled can help with productivity.

A complete list of features can be found at Zend.

Zend Studio for Eclipse consumes quite a lot of memory and the recommended amount of RAM for eclipse based applications is 2GB, however you can control the amount of memory that eclipse will use by editing the zendStudio.ini file.

Whilst I do like the IDE I have found a number of issues with Zend Studio:

Bugs

• 1. There have been a number of issues revolving around the SVN implementation in Zend Studio for Eclipse which causes the application to hang (SVN support is provided to eclipse by a 3rd party plugin (a company called Polarion)):
  • a) When committing files it locks entire directories and often hangs making Zend Studio for Eclipse unusable, if you have files that are not saved and attempt to do so it queues it as a users pending tasks and because the commit has stalled you cannot save the file.

    To resolve this I have to kill the process for Zend Studio, shell into the server and cleanup the the projects src, and sometimes have to re-checkout the directories in a project

• 2. When developing via a samba share, it prompts with an incorrect error and does not attempt to re-authenticate when the samba share needs to re-connect and/or does recognise that it is talking via a remote device.
   
• 3. Auto format adds extra braces to statements, causing syntax errors, strips all comments out of files!
   
• 4. Importing an auto format does not seem to work correctly.
   
• 5. When working with multiple open files it can overwrite the contents with another, I believe this is the case with files of a similar name (I’ve only had this occur once, however a colleague experiences this quite frequently).
   
• 6. Modified file names are prefixed with “>” and when searching for files by pressing a character will not go to that file in PHP Explorer
   
• 7. Templates do not always get replaced e.g. If the system is slow or you type fnc real quick it doesn’t replace with the template for a function.
   
• 8.
  

  If you cause a syntax error, whether deliberate or not, a number of items such as auto formatting does not work, or you cannot open the php manual from the contexts menu for a php function. For example if I type “str_shuffle” then use a context menu to it does not open a manual.

• 9. introducing a syntax error and then removing the syntax error doesn’t clear until you save the document.

  

Resolved

• 1. There was a bug in automatically updating eclipse, which never seemed to work however in the latest release (beta 2) this has now been resolved.

I’ve had alot of experience with other programming languages, however I had to learn C++ from scratch in a very short period of time, a number of weeks ago. This was to develop a real-time stock quote client, the goal was simply to push data from remote servers into our databases, filter what messages it would receive and get something up and running fast as deadlines lingured. This was simple enough, however with the rush the application had its inherent flaws, due to my lack of knowledge of C++, the API, and the goals it had to acomplish.

I’ve since had time to learn a little more C++ and limited time to design the application properly.

The Problems

The core problems with the application:

• refactor, refactor, refactor
• database connection pooling
• Query remote CSP servers^*1
• Query remote CSP servers^*1 from PHP
• Configuration management
• Monitoring
• Flexible Database schema
  • Add columns to database schema dependent on datatype.
  • Log messages in XML per trade message with date/time, columns and values.

Compatible GCC

The first issue was that I used an API from interactive-data, which was compatible with “gcc version 3.2.3″ and is not kept up to date. This meant compiling a compatible gcc from source for 32bit platforms only.


./configure --prefix=/usr/local/gcc/ --mandir=/usr/share/man --infodir=/usr/share/info --enable-shared --enable-threads=posix --disable-checking --with-system-zlib --enable-__cxa_atexit --enable-languages=c,c++,objc,obj-c++

make bootstrap
cd gcc
make
sudo make install


Once having a compatible compiler, I then had to make modifications to the Makefile, move a number of lib/so files to get MySQL to compile and get things working. Unfortunately I did not have a local machine to attach a debugger, so everything was trial and error from the command line with g++32, which makes life difficult identifying runtime errors.

The Logic

Once everything was in place, the logic was fairly simple, foreach field retreived construct a query with the field name, checking the fields values datatype whether it be a datetime, varchar etc. Insert each trade message in a table, update another and if either failed, check if the fault was due to a missing column, if so add it and re-execute queries.

The problem soon arrises when you need to know when each column was actually last updated, with which field, value, datetime and the last insert id for the trade messages. Whilst looping through each trade message, I constructed an XML schema containing the above, however the tricky part is to ensure that it only updates the fragment matching the field in the schema. Not an ideal format to query from a database.

Storing Data

One of the fundemental issues is managing and storing data. For some exchanges you don’t want to store every trade message; simply storing the current data for a number of instruments is enough. Which servers or databases do you peg data to? If one database goes down, how do you handle fault tolerance? MySQL cluster is not a feasible solution, requiring multiple servers and large memory requirements per installation. The databases are highly susceptible to curruption or faults. Also particular sites may require data from multiple exchanges, so seperating trade messages per database is not also ideal.

All of this fundamentally comes down to configuration management.

Configuration

One of the fundamental aspects of the application is configuration management. This contains where data should be stored for a particular exchange, the type of data to store, whether it is per trade message, current data or both. Which servers to source data from, whether it is real time or delayed, whether to source data for bonds, equities, automated trades etc… All queries can be grouped, or to query remote servers. Some of the products for example just for the London Stock Exchange is:

• London Stock Exch – Covered Warrants L1
• London Stock Exch – International Equity Mkt Service L1
• London Stock Exch – International Equity Mkt Service Level 2
• London Stock Exch – UK Equity Mkt Service L1
• London Stock Exch – UK Equity Mkt Service Level 2 (Depth Refresh)
• London Stock Exchange: UK Equity Market Service Level 2

All of which is stored in several database tables and managed via a MySQL database and PHP frontend.

I recently read “Binaries Belong in the Database Too” on sitepoint.com, and thought I would shed some light with regard to my experience of storing files in databases. I’m sure many of you have known this to be a taboo practice, and I would certainly agree depending on the database. A project I worked on for MTV Networks Europe/International required a completely shared nothing architecture. This meant that MTV’s hosting & operations imposed that I stored files in the database, and expressed my hesitation.

The platform:

* Linux
* Apache
* MySQL
* PHP5

The problems

You typically get the common file upload problems with upload_max_filesize, max_input_time, execution time however you also have issues with mysql connections and max packet sizes, mysql chunked streams. Uploads via JUpload allows for large file uploads, however you still encounter TCP/IP connection interuptions and errors. Some of the more major issues I encountered were with the actual management of the data. Currently MySQL has no real support for handling Binary Large Objects, for example if you try to load data in from a file you generally encounter max packet size errors. Although the most fundamental issue is that the MySQL protocol does not send chunked streams for blobs and the client has to load the entire blob into memory. Admittedly memory limits on the server were not too much of an issue, as I was using a 8 CPU 16GB of ram server, however you may not have some of the infrastructure that I had available.

Whilst there were a number of limitations I had to resolve as described above, some of the expectations that I had not anticipated for were some user errors, such as trying to upload 4mb BMP files to be streamed as images for a website. Other factors were that hosting & operations had not expected their adsales department to attempt to upload 120+ mb video files.

DataTypes

Firstly lets look at some of the limitations on the BLOB datatype in MySQL, as you can see there are length limitations on blobs.

TINYBLOB
A BLOB column with a maximum length of 255 (2^8 - 1) bytes.

Alternatives for storing >4Gb BLOBs are:
* Compressing the BLOB so that it fits in 4Gb
* Splitting up the BLOB into 4Gb chunks as separate rows columns.

Tips:

Get Blob length
To find the length in bytes of a stored BLOB. Simply use: SELECT LENGTH(blobcolumn) FROM table.

Get Blob fragment
To retrieve large BLOBs by using repeatedly retrieving only fragments of a BLOB


using substring, ie:
SELECT SUBSTRING(document, 1, 10240) FROM documents WHERE did=3;
and then
SELECT SUBSTRING(document, 10241, 10240) FROM documents WHERE did=3;
etc.

Inserting Blobs
Inserting data into BLOBs. It has to be inserted in hex ie: ‘A’ = 0×41 and ‘AB’ = 0×4142 and so on. The prefix is a zero not a cap o.

If you want to insert binary data into a string column (such as a BLOB), the following characters must be represented by escape sequences:

NUL 	NUL byte (ASCII 0). Represent this character by '' (a backslash followed by an ASCII '0' character).
 	Backslash (ASCII 92). Represent this character by '\'.
' 	Single quote (ASCII 39). Represent this character by '''.
" 	Double quote (ASCII 34). Represent this character by '"'.

When writing applications, any string that might contain any of these special characters must be properly escaped before the string is used as a data value in an SQL statement that is sent to the MySQL server, base64 encoding is a good option.

Indexing Blobs


Blobs can sometimes can be indexed, depending on the storage engine you’re using:
MyISAM, InnoDB, and BDB tables support BLOB and TEXT indexing. However, you must specify a prefix size to be used for the index.This avoids creating index entries that might be huge and thereby defeat any benefits to be gained by that index.The exception is that prefixes are not used for FULLTEXT indexes on TEXT columns. FULLTEXT searches are based on the entire content of the indexed columns, so any prefix you specify is ignored.
MEMORY tables do not support BLOB and TEXT indexes.This is because the MEMORY engine does not support BLOB or TEXT columns at all.

Solution

The solution ended up utilizing 2 memcached servers that cached blobs and objects between the MySQL server, this saved streaming the content directly from MySQL upon each request. Then selecting chunks of data from a binary large object and concatenating the results alleviates maximum packet errors from MySQL. The only other aspects to alleviate are the initial upload, this is entirely upto you, how this is implemented whether it is via JUpload, SCP, FTP, or some other means. Finally increase the above settings. To import / export binary files I wrote a script that queried the database writing out the files, by chunking the data, this script did take a while to execute.

I have heard that Oracle has very good support for handling Binary Large Objects… Maybe thats something to look into..

Pointers.

http://jeremy.zawodny.com/blog/archives/000078.html
http://jeremy.zawodny.com/blog/archives/000840.html
http://www.lentus.se/warehouse/SlidesDW.ppt
http://sunsite.mff.cuni.cz/MIRRORS/ftp.mysql.com/doc/en/BLOB.html

Well, i’ve been rather busy at work, which is nothing short of typical now, however I thought I would post up a script that I wrote to automate the installation of PHP on Windows. In general I’m a very lazy person, and found it irritating going to the PHP site, downloading and manually installing PHP in the same mundane fashion that I have described in the past. Oh, and that even means monitoring the PHP.net site for new releases.

The script I wrote uses a batch file, JScript on the client using WSH,WMI, and a few applications I have bundled with this quick distribution. Which copies, SVN, CVS, and an unzip package (located in the directory bin). It will then download a number of packages that I like to keep tabs on such as:

• Propel
• Prado
• Wact
• Phing
• phpMyAdmin ^{(Which I set the domain in the hosts file to phpMyAdmin)}
• Mojavi
• FCKeditor
• Symfony

Simply execute “CVS.PHP.bat”

Whilst, I have only set this to download CVS phpweb, and extract each. Anyway, I realise that it does struggle with PHP4, however that simply requires some DLL’s into the SAPI dir for example, anyway I haven’t got time to do anymore on this, considering i’m now using a Mac Book Pro.

There are a couple things, I haven’t automated such as virtualhosts, and setting the hosts file, however if someone has some spare time, would be nice to get that working too…

Note: requires Apache 2 and I have this scheduled to run every day at 6.30pm.

download

Anyway for those lazy people enjoy

After upgrading to the very latest build of Plesk the API error seems to have been resolved, and no longer causes an API error of “Domain adding was failed. Error: Can`t resolve ID for IP ()“.

Previously:

[Server 3]# rpm -qa |grep -i psa-api | sort
psa-api-cli-7.5.2-rhel3.build75050130.17
psa-api-common-7.5.2-rhel3.build75050130.17
psa-api-rpc-7.5.2-rhel3.build75050130.17
psa-api-rpc-doc-7.5.2-rhel3.build75050130.17
psa-api-rpc-protocol-7.5.2-rhel3.build75050130.17

Now:

[Server 3]# rpm -qa |grep -i psa-api | sort
psa-api-cli-7.5.4-rhel3.build75051014.16
psa-api-common-7.5.4-rhel3.build75051014.16
psa-api-rpc-7.5.4-rhel3.build75050930.11
psa-api-rpc-doc-7.5.4-rhel3.build75050930.11
psa-api-rpc-protocol-7.5.4-rhel3.build75050930.11

Perhaps upgrading on windows too will also resolve this issue.

I found this a little amazing, but SWSoft don’t actually fully support their own software.

“Unfortunately, SWSoft does not fully support the API that is built in to Plesk and more than likely, they will request that we first perform an upgrade to the latest version of Plesk before they provide any further support.”, Rackspace.

The Fault:
“Domain adding was failed. Error: Can`t resolve ID for IP ()”

NOT Working Correctly
* Server 3 (Plesk 7.5.2)

Working Correctly
* Server 2 (Plesk 7.1.7)
* Server 1 (Plesk 7.5.4)

The IP Address does exist in the domains table, for all servers.

Working Servers:

* Server Name: Server 2

mysql> SELECT * FROM IP_Addresses;
+----+-----------------+-----------------+-------+-----------+--------------------+-------------------+
| id | ip_address | mask | iface | type | ssl_certificate_id | default_domain_id |
+----+-----------------+-----------------+-------+-----------+--------------------+-------------------+
| 2 | 212.100.254.166 | 255.255.255.0 | eth0 | shared | 1 | 47 |
| 3 | 10.230.166.4 | 255.255.255.192 | eth1 | exclusive | 1 | NULL |
| 4 | 192.168.1.20 | 255.255.255.0 | eth0 | exclusive | 1 | 22 |
| 5 | 10.230.166.8 | 255.255.255.192 | eth1 | exclusive | 1 | NULL |
+----+-----------------+-----------------+-------+-----------+--------------------+-------------------+
4 rows in set (0.00 sec)

—————-

Server Name: Server 1

mysql> SELECT * FROM IP_Addresses;

+----+---------------+-----------------+-------+-----------+--------------------+-------------------+
| id | ip_address | mask | iface | type | ssl_certificate_id | default_domain_id |
+----+---------------+-----------------+-------+-----------+--------------------+-------------------+
| 1 | 70.85.198.202 | 255.255.255.248 | eth0 | shared | 1 | NULL |
| 2 | 70.85.198.203 | 255.255.255.248 | eth0 | shared | 1 | NULL |
| 3 | 70.85.198.204 | 255.255.255.248 | eth0 | exclusive | 1 | 65 |
| 4 | 70.85.198.205 | 255.255.255.248 | eth0 | shared | 1 | NULL |
| 5 | 70.85.198.206 | 255.255.255.248 | eth0 | exclusive | 7 | 15 |
+----+---------------+-----------------+-------+-----------+--------------------+-------------------+
5 rows in set (0.00 sec)

—————

Failed Server

Server name: Server 3

mysql> SELECT * FROM IP_Addresses;

+----+---------------+-----------------+-------+-----------+--------------------+-------------------+
| id | ip_address | mask | iface | type | ssl_certificate_id | default_domain_id |
+----+---------------+-----------------+-------+-----------+--------------------+-------------------+
| 4 | 10.230.166.7 | 255.255.255.192 | eth1 | exclusive | 1 | 0 |
| 5 | 192.168.1.22 | 255.255.255.0 | eth0 | shared | 1 | 0 |
| 6 | 192.168.1.223 | 255.255.255.0 | eth0 | shared | 1 | 0 |
+----+---------------+-----------------+-------+-----------+--------------------+-------------------+
3 rows in set (0.00 sec)

——-

One fundamental aspect that I noticed from observation is that Server3 had the columns default_domain_id default value set to 0 where as on all other installations had the value set to NULL. The assumption was that the record 0 for the default IP ID against the domain. Although ALTER’ing the table and testing against each resulted without sucess.

Testing against all IP’s in the waidev6 table results in the following.

* 1. [10.230.166.7] – Domain adding was failed. Error: IP address ‘10.230.166.7′ is not present in client ip pool.
* 2. [192.168.1.22] – Domain adding was failed. Error: Can`t resolve ID for IP ()
* 3. [192.168.1.223] – Domain adding was failed. Error: Can`t resolve ID for IP ()

The XML response for this is…

<?xml version="1.0" encoding="UTF-8"?>

<packet version="1.3.1.0">
  <domain>
    <add>
      <result>
        <status>error</status>
        <errcode>2307</errcode>
        <errtext>Domain adding was failed. Error: Can`t resolve ID for IP ()</errtext>
      </result>
    </add>
  </domain>
</packet>

The payload of the XML document sent across via the XMLRPC request.

<packet version="1.3.1.0">
<domain>
  <add>
    <gen_setup>
      <name>testgen8.com</name>
      <client_id>1</client_id>
      <status>0</status>
      <ip_address>192.168.1.223</ip_address>
      <vrt_hst>
        <ftp_login>catalina0</ftp_login>
        <ftp_password>057177a</ftp_password>
        <ftp_quota>0</ftp_quota>
        <fp>false</fp>
        <fp_ssl>false</fp_ssl>
        <fp_auth>false</fp_auth>
        <fp_admin_login>miaumiaul
        </fp_admin_login>
        <fp_admin_password>lalalalallaa<fp_admin_password>
        <ssl>false</ssl>
        <shell>/bin/false</shell>
        <php>true</php>
        <ssi>false</ssi>
        <cgi>false</cgi>
        <mod_perl>false</mod_perl>
        <mod_python>false</mod_python>
        <asp>false</asp>
        <asp_dot_net>false</asp_dot_net>
        <coldfusion>false</coldfusion>
        <webstat>awstats</webstat>
        <errdocs>false</errdocs>
        <at_domains>true</at_domains>
      </vrt_hst>
      <htype>vrt_hst</htype>
    </gen_setup>
    <hosting>
      <vrt_hst>
        <ftp_login>catalina0</ftp_login>
        <ftp_password>057177a</ftp_password>
        <ftp_quota>0</ftp_quota>
        <fp>false</fp>
        <fp_ssl>false</fp_ssl>
        <fp_auth>false</fp_auth>
        <fp_admin_login>miaumiaul</fp_admin_login>
        <fp_admin_password>lalalalallaa
        </fp_admin_password>
        <ssl>false</ssl>
        <shell>/bin/false</shell>
        <php>true</php>
        <ssi>false</ssi>
        <cgi>false</cgi>
        <mod_perl>false</mod_perl>
        <mod_python>false</mod_python>
        <asp>false</asp>
        <asp_dot_net>false</asp_dot_net>
        <coldfusion>false</coldfusion>
        <webstat>awstats</webstat>
        <errdocs>false</errdocs>
        <at_domains>true</at_domains>
      </vrt_hst>
      <htype>vrt_hst</htype>
    </hosting>
    <prefs>
      <www>true</www>
    </prefs>
   </add>
  </domain>
</packet>

I have also been told that this error also occurs on Windows.

Lately, i’ve been reading Advanced PHP Programming, by George Schlossnagle, which I must say is an excellent book. Below is an excerpt, which I find particularly interesting on the topic of error handling.

“Production Display of Errors

How to notify users of errors is often a political issue. All the large clients I have worked for have had strict rules regarding what to do when a user incurs an error. Business rules have ranged from display of a customized or themed error page to complex logic regarding display of some sort of cached version of the content they were looking for. From a business perspective, this makes complete sense: Your Web presence is your link to your customers, and any bugs in it can color their perceptions of your whole business.

Regardless of the exact content that needs to be returned to a user in case of an unexpected error, the last thing I usually want to show them is a mess of debugging information. Depending on the amount of information in your error messages, that could be a considerable disclosure of information. “, PHP Error Handling

Along with some brief discussions with Håvard Eide on some SQL Injection Attacks, I found in a large public site, the end result was simply terminating the execution, and displaying the error. E.g.

Note: The database is using MySQL 3. MySQL supports multiple result sets, from multiple statements, although this is a flag in MySQL 4.1+. So, when upgrading we have numerous secruity exploits, that has the potential to delete or drop the entire database. Also, with more rigorous permissions on the credentials used would also help alleviate such problems.

“If $userid is passed in, unvalidated, from the end user, a malicious user could pass in this:

$userid = “10; DELETE FROM users;”;

MySQL (like many other RDBMS systems) supports multiple queries inline, if this value is passed in unchecked, you will have lost your user’s table. This is just one of a number of variations on this sort of attack. The moral of the story is that you should always validate any data in queries.”, George Schlossnagle


mysql_query('') or die('MySQL Error: '.mysql_error());


Firstly, if there was a fatal error such as that, someone with a malicious intent can use the debugging information printed to acquire further knowledge and deduce other methods of attack. Also, from a usability point of view, why is the execution terminated? At least give something more meaningful or redirect to an errors page. Ultimately, a message such as that should never, ever, be shown on a live site.

More to the point of this post, displaying a cached version of the page is a great method, it gives the end user the content that they were looking for, and illudes the fundamental issue of deteriorating your users perception. Now, this entails a number of further questions to think about:

• What if it is a dynamic service, such as a search?
• Caching Policy and retention.

Whilst, dynamic services would be very difficult to cater for, and each case would be unique, in terms of search. Some possibilities, are to cache paged results, and apply some business logic on the retention of that cache (We don’t want to serve out-dated content), also state that the user is searching cached content and for any new terms, that haven’t been cached display an error page. This could become very convoluted, and in essence results in how critical the information is or the service!

You also don’t want to display partial pages where execution has ultimately terminated, this should be rather trivial with output buffering. However, it can be a rather gray area to associate, every possibility and as George Schlossnagle mentioned each company has different requirements.

It would be interesting to expand out and detail case examples.

I would be interested to understand better about one of the statements in the book…. “Exceptions expose the possibility of leaking memory.”, if anyone has more information on this, I would be very interested to hear further…

Further Information:

The addslashes() Versus mysql_real_escape_string() Debate
mysql_real_escape_string() versus Prepared Statements
Security Corner: SQL Injection