Development, Analysis And Research

I have been migrating a large number of websites and consolidating servers to reduce costs.
As a result it is important to ensure that services are migrated smoothly, planned effectively,
after which I had a think about aspects to consider prior to migrating services.

Planning

• Make a preliminary checklist of services actively in use by each active domain, I.e. FTP, HTTP, SMTP, IMAP, POP3, MySQL etc.
• What maintenance periods do you have available, if at all?
  • What volume of traffic and when are your quietest periods?
• Do you have dedicated infrastructure, sharded, split by service/role?
  • Can parts of the infrastructure be migrated as an individual component
• List core functionality from the domain for testing purposes
  • Ideally this should be wrapped in unit tests as both functional
    • Examples are email, upload (permissions), adding/editing/removing users
• How many servers are you migrating?
  • Large quantities should be automated.
• How critical is the site/service
  • Does it stop 80 staff working?

Specific

• Services
  • Ensure services are initially installed on new server(s).
  • List all configuration files for a particular service (tree).
    • Ensure configuration between each service are identical or compromises are made.
  • List data directories for each service I.e. /var/lib/mysql
    • Can data be transferred automically.
    • Can services be replicated and brought into sync
    • Can data be back filled?
      • I.e Are large log tables required to make the site functional, what is the minimal effort required to bring the site functional?
• SSL
  • Ensure valid certificate exists for any CDN, sub-domain, domain.
• Email
  • Are there any special firewall, configuration requirements?
• DNS
  • Lower the TTL for a domain your preparing to transfer (if possible)
    • Cannot rely on low TTLs, these are cached amongst large corporates, ISPs etc.
  • Ensure the domain is bound to a unique VIP on new servers, if DNS resolution fails, you can put a header(’Location 10.10.10.10′); in the old site to ensure the domain will resolve correctly.
    • Test this prior to transfer for both HTTP & HTTPS if applicable
• Permissions
  • Do you upload content to the servers, does your code write to the filesystem?
    • Is this writtable?
  • Under which user/group is this written?
• Cache
  • Does your site make use of distributed or local cache?
    • Could there be collisions between different sites, I.e. Do you prefix cache key names based on site?
• Networking
  • Can specific services be migrated prematurely?
    • Repoint via iptables, and keep an eye on bytes passing through the interface till redundant
• Security
  • Were there any firewall restrictions that need to be replicated, either hardware, iptables etc.
  • Chrooted, users copied, ssh keys copied.
• Optimizations
  • Were there any special optimizations, I.e. DnsMasq?, sysctl changes?
• Load balancing
  • Ensure each domain has its own VIP – HTTP_HOST fails in HTTP 1.0 clients
  • Ensure wild cards are not specified within virtual hosts – see above
  • Ensure sites with load balancing and over SSL use TCP requests correctly, in addition see first point.
  • ifdown each VIP in the webserver pool, does it failover with the correct site on all nodes?
• Monitoring
  • If previously had monitoring on servers (should do), has this been replicated to new servers?
• Database (Will vary depending on setup)
  • Is the database replicated?
    • Take LVM snapshots of the raw data on slave and rsync to new servers.
      • Ensure to change configuration such as server id’s, permissions on master, firewall, start service and start replication. Will be ready to start replicating with correct binlog positions etc.
• Other general changes
  • Are there customizations to /etc/hosts get sites working?

Let me know if there is anything you think I have missed.

Recently we had an issue with one of our hosting providers load balancing (LVS), which resulted in some very small outages. As a result we decided to setup our own load balancing that we had full control over, and could manage ourselves. In addition to choosing a better suited weighting algorithm.

Each webserver is setup using ucarp an implementation of Common Address Redundancy Protocol (CARP) allowing failover of a single Virtual IP (VIP) for high availability. We bound multiple VIPs for each host as we noticed some HTTP 1.0 clients incorrectly sending the host address to the server.

There are many ways you can then proxy the webservers and load balance, however we decided to use haproxy. This can also be acheived by pound, apache mod_proxy, mod_backhand etc.

In order to setup ucarp & haproxy:

apt-get install -y haproxy ucarp


Modify /etc/network/interfaces giving each interface a unique ucarp-vid and adjust ucarp-advskew for weighting on each server (increment by one for each server) and set ucarp-master to yes if it is to be the master. Modify the configuration below appropriately.

# The primary network interface
auto eth0
iface eth0 inet static
        address   10.10.10.2 # IP address of server
        netmask   255.255.255.255
        broadcast 10.10.10.10
        gateway   10.10.10.1
        ucarp-vid 3
        ucarp-vip 10.110.10.20 # VIP to listen to
        ucarp-password password
        ucarp-advskew 10
        ucarp-advbase 1
        ucarp-facility local1
        ucarp-master yes
iface eth0:ucarp inet static
        address 10.10.10.20# VIP to listen to
        netmask 255.255.255.255

To bring the interface up, simply run the following:

ifdown eth0; ifup etho0
ifdown eth0:ucarp; ifup eth0:ucarp


In order to configure haproxy:

sed -i -e 's/^ENABLED.*$/ENABLED=1/' /etc/default/haproxy


Reconfigure apache to listen only on local interfaces (/etc/apache2/ports.conf):
So replace “Listen 80″ with


Listen 10.10.10.20:80
Listen 10.10.10.2:80


edit /etc/haproxy/haproxy.cfg

listen web 10.10.10.20:80
        mode http
        balance leastconn
        stats enable
        stats realm Statistics
        stats auth stats:password
        stats scope .
        stats uri /stats?stats
        #persist
        server web1 10.10.10.2:80 check inter 2000 fall 3
        server web2 10.10.10.3:80 check inter 2000 fall 3
        server web3 10.10.10.4:80 check inter 2000 fall 3
        server web4 10.10.10.5:80 check inter 2000 fall 3
        server web5 10.10.10.6:80 check inter 2000 fall 3

Then restart haproxy with /etc/init.d/haproxy restart

After changing your DNS to point to 10.10.10.20 you will be able to see the traffic balanced between the servers by going to the URL http://10.10.10.20/stats?stats with the credentials assigned above and see the bytes balanced between the servers listed.

Some other alternatives are:

• Wackamole
• LVS