Testing Magento 2 Data Migration Tool

Testing Magento 2 Data Migration Tool

As you are probably aware of, Magento 2 will come with a brand new architecture and database design. To ease up upgrade process, Magento released official Data Migration Tool that will help developers migrate store data from Magento 1 to Magento 2.

Official resources:

Data Migration Tool transfers data from M1 database to M2 database based on set of rules defined in xml files. To be clear, this tool only transfers data. Themes and customizations of your store can’t be automatically changed or migrated to Magento2.

You install tool into empty Magento2 installation with Composer. It’s not an extension, but standalone shell app that requires Magento2 framework to work. It’s located in vendor/magento/data-migration-tool/bin/migrate.

You set it up by editing configuration xml files in vendor/magento/data-migration-tool/etc/ce-to-ce/ folder. On supported default Magento installations, minimum required would be to configure source(M1) and destination(M2) database connections in config.xml:

<source>
<database host="localhost" name="magento1_db" user="root"/>
</source>
<destination>
<database host="localhost" name="magento2_db" user="root"/>
</destination>

config.xml is main configuration file, it defines connection, which steps to execute, path to other specific xmls. map.xml holds main global mapping definitions, which tables or columns to ignore, what is renamed to what, while other xml files are step-specific things.

There are 3 main commands/steps for the migration: settings, data and delta. This and their sub-steps can be seen in config.xml.

cd vendor/magento/data-migration-tool/bin/
php migrate --help
 
Usage:
migrate <mode> --config=path/to/config.xml [options]
migrate --help
 
Possible modes:
settings Migrate system configuration
data Main migration of data
delta Migrate the data is added into Magento after the main migration
 
Available options:
--reset Reset the current position of migration to start from the beginning
--config <value> Path to main configuration file, i.e.: etc/m1_version/config.xml
--verbose <level> Verbosity levels: DEBUG, INFO, ERROR
--help Help information

Migrate Settings

First, you migrate settings, websites and stores to M2. Most of Magento data is related to websites and stores, so this needs to go first.

Migrate Data

Data migration will transfer categories, products, customers, orders, wishlists, ratings, everything you can think of into M2. I believe we’ll even turn off some things, like logs or quotes.

Migrate Delta

Now this is where it gets interesting. After successful data migration, you can always just append new M1 entries to M2 database with delta migration, it will continue where it stopped last time. Delta doesn’t migrate new or changed products or categories (at the moment of writing), only customers, orders, and similar customer related data.

While doing data migration, set of m2_* tables are created in your M1 database with set of triggers that enables tracking changes.

Delta is here to reduce migration time to minimum when going live with M2.

Migrate Media

Media files can simply be copy/pasted to appropriate places in M2 after products and categories are migrated, simple as that.

Customize Migration

Since there is no such thing as default Magento in the real world (XD), developers will always need to configure or customize this tool. The tool itself is flexible, since most of mapping is defined through xml files and filtered through php classes, Magento team really did good job here.

Let’s say we have custom varchar sales_flat_order.custom_column in M1. If we run data migration, we’ll end up with an error:

[ERROR]: Source fields not mapped. Document: sales_flat_order. Fields: custom_column

To ignore it we need to add following to map.xml file.

<source>
<field_rules>
<ignore>
<field>sales_flat_order.custom_column</field>
</ignore>
..

The only remark I have is that maybe unknown columns should be ignored by default. If there is column in M1 that isn’t defined in xml, the tool should ignore it, it would save lot of time.

Let’s say we want to move values to renamed custom_column and modify values in the process:

<source>
<field_rules>
..
<transform>
<field>sales_flat_order.custom_column</field>
<handler class="\Migration\Handler\AddPrefix">
<param name="prefix" value="inchoo_"/>
</handler>
</transform>
<move>
<field>sales_flat_order.custom_column</field>
<to>sales_order.new_custom_column</to>
</move>
..

Example shows that Handler class can be used, or custom one created, for changing values on the fly during migration.

If mapping isn’t enough, whole new custom step can be created. Each step is composed out of Integrity, Data and Volume classes. Integrity is triggered before migration to check if everything is ok with mapping, Data transfers data, Volume checks if everything is ok after migration. Delta class can be added to support delta migrations.

Conclusion

Migration tool is great starting point in M1 to M2 migration, developers can customize it and it will really save tons of work. It’s also fast, since it’s direct database to database migration.

At the moment of writing, only 1.9.1.0 was supported, but I successfully migrated default 1.9.2.0 with sample data and additional tables without many problems. Other M1 versions will be supported when M2 comes out.

So, most likely data migration scenario is.. You setup empty M2 and migrate data from your M1 site. You check, setup and develop everything needed for your new M2 store. When you’re done and ready to switch to M2, migrate delta, go live.

Now, we know how confusing all of this can be. But, we’re here to help you with the process of Magento 2 Upgrade. Feel free to drop us a line

Related Inchoo Services

You made it all the way down here so you must have enjoyed this post! You may also like:

3 best open-source eCommerce platforms in 2021 Zrinka Antolovic
Zrinka Antolovic, | 8

3 best open-source eCommerce platforms in 2021

Moving the validation error message, this time globally Danijel Vrgoc
, | 0

Moving the validation error message, this time globally

What’s New in Magento 2.4? Features and Benefits for Merchants Explained Antonio Gotovac
Antonio Gotovac, | 1

What’s New in Magento 2.4? Features and Benefits for Merchants Explained

23 comments

  1. i got an error like this
    Any solution is there
    [Exception]
    Notice: Undefined offset: 9 in /home/devserve/public_html/magehook/vendor/magento/data-migration-tool/src/Migration/Step/Eav/Data.php on line 291

  2. For everybody with:
    SQLSTATE[23000]: Integrity constraint violation: 1062 Duplicate entry […]

    In my case this happend after trying to run the migration again without resetting the magento2 db. If you run migrate:data you must not have runned the command earlier on the same database. To test the migration I used a clean magento2 db backup and imported it after every migration again.

  3. If i wanted to exclude the products from being migrated, what is the easiest way to achieve this?(aside from removing all of the mapping in map.xml) Is it possible for me to just exclude the step for products in the migration completely?

  4. How migrate only attributes and only categories (with url rewrite)? Without set attribute, sales, customer,settings store. Only attributes, only category. I can’t configure the xml migrate

  5. Hi All,

    I am getting error while doing data migration

    [2016-10-05 09:31:17][ERROR]: Field has ambiguous configuration: catalog_category_entity_int.entity_id

    Please help on this.
    Thank you.

    Regards,
    Swapnil

  6. I’m facing this issue.

    SQLSTATE[23000]: Integrity constraint violation: 1062 Duplicate entry ‘customer-service-1’ for key ‘URL_REWRITE_REQUEST_PATH_STORE_ID’

    Checked for duplicate entry customer-service but didn’t find any.

  7. Does anyone have this issue like this?

    [PDOException]
    SQLSTATE[23000]: Integrity constraint violation: 1062 Duplicate entry ‘12969-1’ for key ‘CATALOGINVENTORY_STOCK_ITEM_PRODUCT_ID_WEBSITE_ID’

  8. I’m trying to move over custom fields that modules have added to our magento tables over the years, but I can’t figure out if there’s some way to get the migration tool to create the new fields.

    In your example, how did you create sales_order.new_custom_column? Did you manually modify the table or is there some function of the migration tool that’ll do that for me?

    1. data migration tool does not migrate custom extensions. So, for sure the table you described wont be migrated. You have to ignore all undefined columns and tables in magento2. So, instead of creating fields in magento2 ignore that fields using below code:

      sales_flat_order.custom_column

      ..

  9. Hey Paulo,
    your error:
    SQLSTATE[23000]: Integrity constraint violation: 1062 Duplicate entry ‘1-Migration_Default’ for key ‘EAV_ATTRIBUTE_SET_ENTITY_TYPE_ID_ATTRIBUTE_SET_NAME’
    is an attribute set duplication error. I have also faced the same..

    If you see your M1 database that you are going to migrate to magento2 there will be “Default” attribute set duplicated.
    So, at the end the solution is delete your all “Default” attribute set from your M1 “eav_attribute_set” table. And then start your migration to fresh blank Magento2 database.

    It worked for me and I hope it would work for u also.

  10. Anyone having this issue?
    ( data migration > EAV Step)

    SQLSTATE[23000]: Integrity constraint violation: 1062 Duplicate entry '1-Migration_Default' for key 'EAV_ATTRIBUTE_SET_ENTITY_TYPE_ID_ATTRIBUTE_SET_NAME'

  11. It does not work very good, I’ve set everything and only core_config_data was imported and it blows all the previous installation twice.

    Who says magento2 is better, it could became but not at this point. There’s a lot of bugs to fix and it is not trustful as expected.

    regards,

    Paulo

  12. Nice Review. Actually I kind of like that unknown fields are not ignored by default. This way, you know which data is lost / you don’t forget valuable fields.

Leave a Reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <blockquote cite=""> <code> <del datetime=""> <em> <s> <strike> <strong>. You may use following syntax for source code: <pre><code>$current = "Inchoo";</code></pre>.

Tell us about your project

Drop us a line. We'd love to know more about your project.