Moving Magento store to another server

magento_move_to_another_server

Custom Magento stores are probably the most complex pieces of web related software out there, and most of the time developers are happy when they come to a point where things (mostly) work. You can imagine the horror Magento developer is up against, when a few weeks before holiday season, Google Analytics gives indication that current hardware infrastructure just won’t cut it. In this article I’ll do my best to guide you trough process of moving your Magento code, files and database to another server. Additionally, I’ll give you some pointers regarding DNS changes required for pointing your domain to another IP address.

Preparation for transfer

The curse of developing ecommerce sites is that downtime can be easily expressed in terms of lost revenue, and this is something we developers must be aware of. With process of moving any site to new server there is a portion of downtime caused by changing IP address in the context of DNS changes propagation. This downtime is unavoidable and not directly under our control, so we really want to minimize amount of downtime caused by moving code and database to another server. The key to achieving fast and effective Magento transfer is in preparation.

Establishing connectivity

First step to our goal is tweaking OpenSSH on both source and destination server to allow secure data transfers without requiring user account passwords. We will accomplish this using Public-Key authentication. Designated user accounts on both of your servers most probably already have private-public key pair generated. If .ssh directory of user account is empty or non existent, you must generate private-public key pairs your self. Creating unencrypted key pair (password-less) is something that isn’t generally recommended but for sake of simplicity this is what we will do here using ssh-keygen command. This should be executed for every server without existing private-public key pair:

ssh-keygen -q -t rsa -N '' -f ~/.ssh/id_rsa

At this point designated user accounts at both servers should have .ssh directory under their home and inside two files named id_rsa and id_rsa.pub. The first file is the private key (keep it private) and the second file is the public key (feel free to share because it can only be used to grant your server access to outside resources). To achieve password-less connectivity between your machines you must do two things. First you must install each server’s public key to every other remote machine’s authorized_keys file. You can use ssh-copy-id command to do so:

# Execute from your source server
ssh-copy-id dstuser@dsthost
# Execute from your destination server
ssh-copy-id srcuser@srchost

Second thing you must take care of is set secure permissions for .ssh directories as well as all parent directories for .ssh (usually only /home):

# Remove write permission from home for group and others
chmod go-w ~/
 
# Secure permissions for .ssh and private key
chmod 700 ~/.ssh
chmod 600 ~/.ssh/id_rsa
 
# A little less secure permissions for public key
chmod 644 ~/.ssh/id_rsa.pub
chmod 644 ~/.ssh/authorized_keys
chmod 644 ~/.ssh/known_hosts

If you can now connect trough ssh from your source server to destination server and vice versa without being asked for password, you did good and you can proceed. If this isn’t the case, then retrace your steps, especially the permissions part.

Server configuration

This is a really broad field that largely depends on the Linux distribution in question, web server of choice (Apache, Nginx), and infrastructure configuration (number of servers involved). Luckily this is something that should be done by your client’s hosting company (sadly this wasn’t the case when this procedure was created, expect more about it in one of my future articles) thus I’ll just cover creation of Magento database on the destination server. I will assume you do not have access to phpMyAdmin on your destination server, so I’ll show you how to do this right from MySQL client console.

First step, decide on MySQL database name, user name and password. After you do this, please document these key pieces of information. Next thing to do is open MySQL client from your destination server as administrative user and create new MySQL user and associated database. Assuming you know how to start MySQL client, here’s code for creation of database user and associated database. Please replace following data with your own:

  • db_user – Magento database user
  • db_pass – Magento database password
  • db_name – Magento database name
  • db_host – Magento database host. Most probably localhost but adjust to DB server host name or IP if you’re dealing with dedicated DB server)
CREATE USER 'db_user'@'db_host' IDENTIFIED BY 'db_pass';
GRANT USAGE ON * . * TO 'db_user'@'db_host' IDENTIFIED BY 'db_pass' WITH MAX_QUERIES_PER_HOUR 0 MAX_CONNECTIONS_PER_HOUR 0 MAX_UPDATES_PER_HOUR 0 MAX_USER_CONNECTIONS 0;
CREATE DATABASE IF NOT EXISTS db_name;
GRANT ALL PRIVILEGES ON db_name. * TO 'db_user'@'db_host';

If you have phpMyAdmin available, you can also do this procedure using its interface.

Creating automation scripts

General idea is to create and thoroughly test two shell scripts written in bash with goal of automating key parts of the transfer and eliminating human error from the equation. Important thing to point out is that these scripts should not be taken as they are, rather you should use them as starting points for developing scripts tailored to your specific environment.

First script has purpose of doing initial sync of Magento code between two servers and is to be executed while the site is still up. Updated versions of these scripts will be available inside my GitHub Gists collection, for now lets name this first one magento-transfer-1.sh.

This script requires following data:

  • SSH data for source server. We do not need password because we have configured Public-Key authentication.
  • Absolute path to web server document root on source web server.
  • Absolute path to web server document root on destination web server.

As you can see, the first script uses rsync to do the initial sync. We have selected rsync over scp due to possibility of incremental sync used later on.

Second script is much more complex as it’s intended to be executed after Magento store has been placed in maintenance mode. We will name this script magento-transfer-2.sh. This script does a few things:

  • Does incremental rsync of Magento code from source server to account for file changes that occurred after you have executed magento-transfer-1.sh script (media files changes).
  • Creates list of unnecessary tables to ignore and feeds this to mysqldump to speed up the dump process. Please keep in mind that this list isn’t as restrictive as when doing developer database dumps, because we’re talking about production database and we would like to preserve information like customer quotes.
  • Database is mysqldumped in two steps, first structure and then data. Reason for this is to avoid situation where ignore table directives affect structure as well as data, what causes missing tables on the destination server. After mysqldump, structure and data is united, compressed with gzip and sent trough SSH tunnel. On receiving side data is gunzipped and trigger definers are fixed by replacing trigger definer with name and host values of the new Magento database user. This is the fastest way to transfer large database because we’re avoiding unnecessary disk writes involved when you first do mysqldump then import, as well as manual labor.

Now here’s the magento-transfer-2.sh script code:

This script requires following data:

  • SSH data for source server. We do not need password because we have configured Public-Key authentication.
  • Absolute path to web server document root on source web server.
  • Absolute path to web server document root on destination web server.
  • MySQL database data for source server. If you were using separate DB server in the past, you should adjust the SRC_MYSQLDUMP_HOST variable.
  • MySQL database data for destination server. If are going to use separate DB server in the future, you should adjust the DST_MYSQLDUMP_HOST variable.

Transfer

This chapter is what the previous chapter was all about. It’s time to place our store in maintenance mode, update Magento configuration and update DNS records with our destination server IP address.

Using automation scripts

First thing we need to do is get our automation scripts to destination server. For this task you can use scp. As far as location on the destination server is concerned, I recommend creating directory named bin inside home directory for your user account. Reason for this is that on nearly all Linux distributions /home/$USER/bin directory is automatically added to system PATH if it exists. This makes it ideal place for dumping shell scripts. Also you must make the scripts executable using following commands:

chmod +x magento-transfer-1.sh
chmod +x magento-transfer-2.sh

Next thing you must do is fill both scripts variables with relevant data like document root paths, SSH and MySQL user names, SSH and MySQL hosts as well as MySQL database names. After you are absolutely sure every piece of data is in it’s place, you can proceed with triggering first script with something like:

~/bin/magento-transfer-1.sh

This will do the initial sync of code and media files.

Following step requires for you to place Magento site in maintenance mode. All you have to do achieve this is create file named ‘maintenance.flag’ in your Magento installation root directory, in our case we’re talking about our source server.

ssh srcuser@srchost
cd /your/magento/installation/root/
touch maintenance.flag

Please keep in mind that store transfer downtime begins with creation of this file, and time that Magento store will spend down is roughly equal to time it takes to execute our second bash script.

Next step is triggering our second script to do incremental sync of code and media files, and transfer of database between source and destination MySQL server. Here’s how:

~/bin/magento-transfer-2.sh

After this process is complete, you should proceed to configuring your destination server environment.

Adjusting Magento environment

After you have transfered your Magento code, files and database to destination server you must do two things. First thing is edit your app/etc/local.xml with new MySQL database name, user, password and new DBMS host (if you are using dedicated DB server).

Second thing is to configure crontab on your new server’s user account to trigger Magento cron. To avoid file permission issues, I advise running Magento cron from crontab belonging to user that your Apache or php-fpm instances are being started under, but this is outside of scope for this article. Changing crontab is done using “crontab -e” command and what you need to do is transfer all crontab entries from your source server (account for possible changes in document root on new environment) viewed using “crontab -l” command. Crontab editor is Vi (Vim) so if you aren’t that good with its insert, save and quit controls, additional instructions are only one Google search away.

Additionally now is the time to configure things like Solr search (requires Java), APC, Redis, Memcached but this is outside of scope for this article.

Final testing

At this point your destination server should hold both files and database, and have it’s configuration adjusted to host your Magento installation. This means you should do some testing to check is everything in order before you instruct your domain registrar to point your domain to new server IP. For you to be able to do any testing, you need assistance of your development system hosts file (/etc/hosts on unix-like operating systems, %systemroot%\system32\drivers\etc\ on Microsoft operating systems) to point your domain to new IP address for testing purpose. Example of hosts file entry to instruct your development system to look up example.com domain at X.X.X.X IP address would be:

X.X.X.X example.com

Now you should navigate to your store from your development machine and browse around, try logging into admin and these kind of things. If the site doesn’t load, go and inspect Magento exception log, system log, system reports and finally your web server software log file. Once everything is in order, you can proceed to changing DNS records.

Changing DNS records

Most domain registrars allow access to interface for editing domain records. If this isn’t the case, you should contact the registrar support to do this for you. Minimal action required for pointing your domain to new IP address is the adjustment of A type record. To prevent issues with sending email, you should additionally adjust TXT type record used to host SPF record, or the newer SPF type record. You should also adjust rDNS for your IP address to match your web server hostname because this is what most mail transfer agents check before relaying email messages. Failure to do so could result with emails ending up in spam folder of customers email account, or worse if messages do not get relayed at all. Reverse DNS record is usually adjusted by the person who controls the in-addr.arpa DNS zone for your IP address. This is usually hosting company, and they most likely provide access to this setting from web interface for your hosting account. Hostname is changed using “hostname” command, for more info you should check manual entry for hostname using “man hostname” command. To make your hostname changes persistent after reboot you must also modify /etc/hostname file accordingly.

After you tweak your DNS records, keep in mind that each record has TTL (time to live) value, and this value determines how long will DNS servers persist it’s values. TTL for A record represents downtime factor you do not have control of, at least not at the time Magento code, files and database transfer is complete.

In this paragraph I will present command used to inspect DNS records for domain. I’m talking about the “dig” command. Here’s example output for querying A and TXT type DNS records for example.com domain:

marko@marko-desktop:~$ dig example.com A TXT
;; Warning, extra type option
 
; <<>> DiG 9.9.2-P1 <<>> example.com A TXT
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 50461
;; flags: qr rd ra; QUERY: 1, ANSWER: 2, AUTHORITY: 2, ADDITIONAL: 5
 
;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;example.com.			IN	TXT
 
;; ANSWER SECTION:
example.com.		51	IN	TXT	"$Id: example.com 1921 2013-10-21 04:00:39Z dknight $"
example.com.		51	IN	TXT	"v=spf1 -all"
 
;; AUTHORITY SECTION:
example.com.		167427	IN	NS	a.iana-servers.net.
example.com.		167427	IN	NS	b.iana-servers.net.
 
;; ADDITIONAL SECTION:
a.iana-servers.net.	519	IN	A	199.43.132.53
a.iana-servers.net.	519	IN	AAAA	2001:500:8c::53
b.iana-servers.net.	519	IN	A	199.43.133.53
b.iana-servers.net.	519	IN	AAAA	2001:500:8d::53
 
;; Query time: 21 msec
;; SERVER: 127.0.0.1#53(127.0.0.1)
;; WHEN: Sat Dec 28 23:00:41 2013
;; MSG SIZE  rcvd: 265

Using procedure outlined in this article, I’ve managed to transfer Magento Enterprise Edition project from old server to new infrastructure consisting of two servers with under 15 minutes of transfer downtime. Of course that time spent for creating this procedure isn’t and shouldn’t be included in downtime :)

I hope you find this article useful and I wish you good luck with moving your Magento installation to another server.

Interested in hiring us?

Have a chat with us. You would be surprised how small changes can make your business even more successful.


About Marko Martinovic

Team Leader/Backend Developer

If you ask him – “Marko, how do you relax in your free time?”, he would most probably say: – “I write some code.” Let’s cut to the chase, he is one passionate programmer who currently lives his dream by working for Inchoo.

Read more posts by Marko / Visit Marko's profile

11 comments

  1. i move my magento site from localhost to live domain but on that from front and back side login not done…

    please help…..

  2. Never fails. I figured it out just after I asked the question. I have never had a php installation on a virtual private server not come with the soap client already installed but I guess this one did. I installed the php soap client on my magento server and now it works.

  3. I was able to move my magento to a new server. I did not know about this post at the time but I did the move very close to how it was done here. My problem is that after I made the move anytime I try to open a soap connection it fails. I can reach the WSDL and a web browser so that can not be the issue. I disabled my firewall for testing just to make sure that was not the issue and I still had failures.
    I was connecting fine with php and c# code prior to the move.
    my php client code is

    $client = new SoapClient('https://<my site url>/api/v2_soap/?wsdl');

    The error I am receiving is:Uncaught SoapFault exception: [0] SOAP extension is not loaded. in /Documents/php/Magento_Get_Orders/index.php:15

    Any idea why soap would stop working after the move. I have check all user permissions and have even regenerated the necessary keys just in case.

  4. Hi sir right now i want to transfer my website which is online shopping website in India http://www.bilmos.com right now which is hosted in hostgator shared hosting so i want to transfer it into any cloud server. but i found amazon is so much costly so where i have to host my website please share some cheap and real cloud hosting companies and what is important point we should put into mind when we transfer website from one hosting server to another hosting server,,,,please share best and best price cloud hosting company for http://www.bilmos.com

  5. Why does everyone assume that you have shell access on your Magento host? Why can’t someone do a tutorial for people that chose not to use shell or do not have shell?

    1. @Brent:

      Hello Brent,
      if your Magento host does not allow shell access, you should either change your host, or arrange for your hosting provider to do this procedure for you. On the other hand, if you intentionally choose not to use shell, it would be wise to give up the transfer of Magento to someone who didn’t make that choice.

      Regards

    1. @luis:

      Actually placing site into maintenance mode does the exact same thing (prevents orders from being lost during transition). Maintenance mode additionally stops the API processing to prevent ERP or third party stock management integration from doing changes on database.

      Regards

  6. Would love a ‘Magento upgrade guide’ :)! Please! Trying to get from 1.6 to 1.8, but too scared to do it without an Inchoo guide.

  7. Thanks for the article. It’s even worth reading when installing Magento on a new server. I transfered a jewelry shop one week ago. I wished I had read this page before because I had troubles with DNS.

    One question : is there a way to restore file owner and file permissions automatically (var, media) ?

    1. @Benabee:

      First of all, I’m really glad you find this article useful.

      Concerning topic of keeping file ownership and permissions, yes there is a way and that’s one of the reasons for using rsync for this task. Both scripts from this article use rsync with -a parameter that represents shortcut for -rlptgoD. The -o parameter is for preserving owner, -g for preserving group and -p is for preserving permissions. Keep in mind that -o requires root user account.

      Cheers!

Leave a Reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <blockquote cite=""> <code> <del datetime=""> <em> <strike> <strong>. You may use following syntax for source code: <pre><code>$current = "Inchoo";</code></pre>.