Moving Magento store to another server

Custom Magento stores are probably the most complex pieces of web related software out there, and most of the time developers are happy when they come to a point where things (mostly) work. You can imagine the horror Magento developer is up against, when a few weeks before holiday season, Google Analytics gives indication that current hardware infrastructure just won’t cut it. In this article I’ll do my best to guide you trough process of moving your Magento code, files and database to another server. Additionally, I’ll give you some pointers regarding DNS changes required for pointing your domain to another IP address.

Preparation for transfer

The curse of developing ecommerce sites is that downtime can be easily expressed in terms of lost revenue, and this is something we developers must be aware of. With process of moving any site to new server there is a portion of downtime caused by changing IP address in the context of DNS changes propagation. This downtime is unavoidable and not directly under our control, so we really want to minimize amount of downtime caused by moving code and database to another server. The key to achieving fast and effective Magento transfer is in preparation.

Establishing connectivity

First step to our goal is tweaking OpenSSH on both source and destination server to allow secure data transfers without requiring user account passwords. We will accomplish this using Public-Key authentication. Designated user accounts on both of your servers most probably already have private-public key pair generated. If .ssh directory of user account is empty or non existent, you must generate private-public key pairs your self. Creating unencrypted key pair (password-less) is something that isn’t generally recommended but for sake of simplicity this is what we will do here using ssh-keygen command. This should be executed for every server without existing private-public key pair:

ssh-keygen -q -t rsa -N '' -f ~/.ssh/id_rsa

At this point designated user accounts at both servers should have .ssh directory under their home and inside two files named id_rsa and id_rsa.pub. The first file is the private key (keep it private) and the second file is the public key (feel free to share because it can only be used to grant your server access to outside resources). To achieve password-less connectivity between your machines you must do two things. First you must install each server’s public key to every other remote machine’s authorized_keys file. You can use ssh-copy-id command to do so:

# Execute from your source server
ssh-copy-id dstuser@dsthost

# Execute from your destination server
ssh-copy-id srcuser@srchost

Second thing you must take care of is set secure permissions for .ssh directories as well as all parent directories for .ssh (usually only /home):

# Remove write permission from home for group and others
chmod go-w ~/
 
# Secure permissions for .ssh and private key
chmod 700 ~/.ssh
chmod 600 ~/.ssh/id_rsa
 
# A little less secure permissions for public key
chmod 644 ~/.ssh/id_rsa.pub
chmod 644 ~/.ssh/authorized_keys
chmod 644 ~/.ssh/known_hosts

If you can now connect trough ssh from your source server to destination server and vice versa without being asked for password, you did good and you can proceed. If this isn’t the case, then retrace your steps, especially the permissions part.

Server configuration

This is a really broad field that largely depends on the Linux distribution in question, web server of choice (Apache, Nginx), and infrastructure configuration (number of servers involved). Luckily this is something that should be done by your client’s hosting company (sadly this wasn’t the case when this procedure was created, expect more about it in one of my future articles) thus I’ll just cover creation of Magento database on the destination server. I will assume you do not have access to phpMyAdmin on your destination server, so I’ll show you how to do this right from MySQL client console.

First step, decide on MySQL database name, user name and password. After you do this, please document these key pieces of information. Next thing to do is open MySQL client from your destination server as administrative user and create new MySQL user and associated database. Assuming you know how to start MySQL client, here’s code for creation of database user and associated database. Please replace following data with your own:

• db_user – Magento database user
• db_pass – Magento database password
• db_name – Magento database name
• db_host – Magento database host. Most probably localhost but adjust to DB server host name or IP if you’re dealing with dedicated DB server)

CREATE USER 'db_user'@'db_host' IDENTIFIED BY 'db_pass';
GRANT USAGE ON * . * TO 'db_user'@'db_host' IDENTIFIED BY 'db_pass' WITH MAX_QUERIES_PER_HOUR 0 MAX_CONNECTIONS_PER_HOUR 0 MAX_UPDATES_PER_HOUR 0 MAX_USER_CONNECTIONS 0;
CREATE DATABASE IF NOT EXISTS db_name;
GRANT ALL PRIVILEGES ON db_name. * TO 'db_user'@'db_host';

If you have phpMyAdmin available, you can also do this procedure using its interface.

Creating automation scripts

General idea is to create and thoroughly test two shell scripts written in bash with goal of automating key parts of the transfer and eliminating human error from the equation. Important thing to point out is that these scripts should not be taken as they are, rather you should use them as starting points for developing scripts tailored to your specific environment.

First script has purpose of doing initial sync of Magento code between two servers and is to be executed while the site is still up. Updated versions of these scripts will be available inside my GitHub Gists collection, for now lets name this first one magento-transfer-1.sh.

This script requires following data:

• SSH data for source server. We do not need password because we have configured Public-Key authentication.
• Absolute path to web server document root on source web server.
• Absolute path to web server document root on destination web server.

As you can see, the first script uses rsync to do the initial sync. We have selected rsync over scp due to possibility of incremental sync used later on.

Second script is much more complex as it’s intended to be executed after Magento store has been placed in maintenance mode. We will name this script magento-transfer-2.sh. This script does a few things:

• Does incremental rsync of Magento code from source server to account for file changes that occurred after you have executed magento-transfer-1.sh script (media files changes).
• Creates list of unnecessary tables to ignore and feeds this to mysqldump to speed up the dump process. Please keep in mind that this list isn’t as restrictive as when doing developer database dumps, because we’re talking about production database and we would like to preserve information like customer quotes.
• Database is mysqldumped in two steps, first structure and then data. Reason for this is to avoid situation where ignore table directives affect structure as well as data, what causes missing tables on the destination server. After mysqldump, structure and data is united, compressed with gzip and sent trough SSH tunnel. On receiving side data is gunzipped and trigger definers are fixed by replacing trigger definer with name and host values of the new Magento database user. This is the fastest way to transfer large database because we’re avoiding unnecessary disk writes involved when you first do mysqldump then import, as well as manual labor.

Now here’s the magento-transfer-2.sh script code:

This script requires following data:

• SSH data for source server. We do not need password because we have configured Public-Key authentication.
• Absolute path to web server document root on source web server.
• Absolute path to web server document root on destination web server.
• MySQL database data for source server. If you were using separate DB server in the past, you should adjust the SRC_MYSQLDUMP_HOST variable.
• MySQL database data for destination server. If are going to use separate DB server in the future, you should adjust the DST_MYSQLDUMP_HOST variable.

Transfer

This chapter is what the previous chapter was all about. It’s time to place our store in maintenance mode, update Magento configuration and update DNS records with our destination server IP address.

Using automation scripts

First thing we need to do is get our automation scripts to destination server. For this task you can use scp. As far as location on the destination server is concerned, I recommend creating directory named bin inside home directory for your user account. Reason for this is that on nearly all Linux distributions /home/$USER/bin directory is automatically added to system PATH if it exists. This makes it ideal place for dumping shell scripts. Also you must make the scripts executable using following commands:

chmod +x magento-transfer-1.sh
chmod +x magento-transfer-2.sh

Next thing you must do is fill both scripts variables with relevant data like document root paths, SSH and MySQL user names, SSH and MySQL hosts as well as MySQL database names. After you are absolutely sure every piece of data is in it’s place, you can proceed with triggering first script with something like:

~/bin/magento-transfer-1.sh

This will do the initial sync of code and media files.

Following step requires for you to place Magento site in maintenance mode. All you have to do achieve this is create file named ‘maintenance.flag’ in your Magento installation root directory, in our case we’re talking about our source server.

ssh srcuser@srchost
cd /your/magento/installation/root/
touch maintenance.flag

Please keep in mind that store transfer downtime begins with creation of this file, and time that Magento store will spend down is roughly equal to time it takes to execute our second bash script.

Next step is triggering our second script to do incremental sync of code and media files, and transfer of database between source and destination MySQL server. Here’s how:

~/bin/magento-transfer-2.sh

After this process is complete, you should proceed to configuring your destination server environment.

Adjusting Magento environment

After you have transfered your Magento code, files and database to destination server you must do two things. First thing is edit your app/etc/local.xml with new MySQL database name, user, password and new DBMS host (if you are using dedicated DB server).

Second thing is to configure crontab on your new server’s user account to trigger Magento cron. To avoid file permission issues, I advise running Magento cron from crontab belonging to user that your Apache or php-fpm instances are being started under, but this is outside of scope for this article. Changing crontab is done using “crontab -e” command and what you need to do is transfer all crontab entries from your source server (account for possible changes in document root on new environment) viewed using “crontab -l” command. Crontab editor is Vi (Vim) so if you aren’t that good with its insert, save and quit controls, additional instructions are only one Google search away.

Additionally now is the time to configure things like Solr search (requires Java), APC, Redis, Memcached but this is outside of scope for this article.

Final testing

At this point your destination server should hold both files and database, and have it’s configuration adjusted to host your Magento installation. This means you should do some testing to check is everything in order before you instruct your domain registrar to point your domain to new server IP. For you to be able to do any testing, you need assistance of your development system hosts file (/etc/hosts on unix-like operating systems, %systemroot%\system32\drivers\etc\ on Microsoft operating systems) to point your domain to new IP address for testing purpose. Example of hosts file entry to instruct your development system to look up example.com domain at X.X.X.X IP address would be:

X.X.X.X example.com

Now you should navigate to your store from your development machine and browse around, try logging into admin and these kind of things. If the site doesn’t load, go and inspect Magento exception log, system log, system reports and finally your web server software log file. Once everything is in order, you can proceed to changing DNS records.

Changing DNS records

Most domain registrars allow access to interface for editing domain records. If this isn’t the case, you should contact the registrar support to do this for you. Minimal action required for pointing your domain to new IP address is the adjustment of A type record. To prevent issues with sending email, you should additionally adjust TXT type record used to host SPF record, or the newer SPF type record. You should also adjust rDNS for your IP address to match your web server hostname because this is what most mail transfer agents check before relaying email messages. Failure to do so could result with emails ending up in spam folder of customers email account, or worse if messages do not get relayed at all. Reverse DNS record is usually adjusted by the person who controls the in-addr.arpa DNS zone for your IP address. This is usually hosting company, and they most likely provide access to this setting from web interface for your hosting account. Hostname is changed using “hostname” command, for more info you should check manual entry for hostname using “man hostname” command. To make your hostname changes persistent after reboot you must also modify /etc/hostname file accordingly.

After you tweak your DNS records, keep in mind that each record has TTL (time to live) value, and this value determines how long will DNS servers persist it’s values. TTL for A record represents downtime factor you do not have control of, at least not at the time Magento code, files and database transfer is complete.

In this paragraph I will present command used to inspect DNS records for domain. I’m talking about the “dig” command. Here’s example output for querying A and TXT type DNS records for example.com domain:

marko@marko-desktop:~$ dig example.com A TXT
;; Warning, extra type option
 
; <<>> DiG 9.9.2-P1 <<>> example.com A TXT
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 50461
;; flags: qr rd ra; QUERY: 1, ANSWER: 2, AUTHORITY: 2, ADDITIONAL: 5
 
;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;example.com.			IN	TXT
 
;; ANSWER SECTION:
example.com.		51	IN	TXT	"$Id: example.com 1921 2013-10-21 04:00:39Z dknight $"
example.com.		51	IN	TXT	"v=spf1 -all"
 
;; AUTHORITY SECTION:
example.com.		167427	IN	NS	a.iana-servers.net.
example.com.		167427	IN	NS	b.iana-servers.net.
 
;; ADDITIONAL SECTION:
a.iana-servers.net.	519	IN	A	199.43.132.53
a.iana-servers.net.	519	IN	AAAA	2001:500:8c::53
b.iana-servers.net.	519	IN	A	199.43.133.53
b.iana-servers.net.	519	IN	AAAA	2001:500:8d::53
 
;; Query time: 21 msec
;; SERVER: 127.0.0.1#53(127.0.0.1)
;; WHEN: Sat Dec 28 23:00:41 2013
;; MSG SIZE  rcvd: 265

Using procedure outlined in this article, I’ve managed to transfer Magento Enterprise Edition project from old server to new infrastructure consisting of two servers with under 15 minutes of transfer downtime. Of course that time spent for creating this procedure isn’t and shouldn’t be included in downtime 🙂

I hope you find this article useful and I wish you good luck with moving your Magento installation to another server.