Comments on: Escape from EAV the Magento way

By: Cesar

Cesar — Mon, 25 Jul 2011 13:58:44 +0000

I guess they made a bad decision on using EAV. It is too heavy to query data, to heavy to write data. Too heavy to write queries.

Man! Wonder if you want do to a BI job on it??

I’m about to not get surprised anymore on how many popular systems I’m checking have weird stuff, because of “hmm.. it is not working well”.

I can’t stop thinking: how a system that is supposed to be quick by nature (ecommerce) could implement such slow thing? And being such popular, nobody says “hey, stop with it”?

Note that they are copying data to flat tables. Sounds a workaround for me. Then, why not just make it flat tables for the whole system? Easy and quick for everyone.

*Sigh*

A sad user.

By: Sean

Sean — Fri, 06 May 2011 05:10:33 +0000

@JB

Have a look at some of the generated SQL code here and let us know what you think?

http://chrismckee.co.uk/ever-wondered-why-magento-layered-cache-was-soooo-slow-and-greedy/

By: JB

JB — Thu, 05 May 2011 19:57:36 +0000

They (the Magento developers) have probably not been using the STRAIGHT_JOIN to do their inner joins in Mysql. Had they tried to use that optimization, flattening the tables would be unnecessary. Read the docs on how it replaces an INNER JOIN and then TEST IT. This one change makes Mysql performance much more reasonable with large datasets being self joined.

By: Tim

Tim — Wed, 01 Dec 2010 15:15:57 +0000

They should better think about using a document based db like MongoDB to store Magentos monster objects (catalog/products, customers+adresses, categorys+relations) – 1 request and everything is in place…

By: Sean

Sean — Tue, 16 Nov 2010 20:39:59 +0000

@t-low
How good are views at handling a dozen self joins over 8 tables to covert rows into columnar data? Essentially making a relational pivot table spread over 8 tables?

By: t-low

t-low — Tue, 16 Nov 2010 04:47:38 +0000

Why they haven’t designed views instead of creating flat tables? It could advoid having a lot of data redundant and really large DB Dumps.

By: Sean

Sean — Wed, 29 Sep 2010 05:38:29 +0000

EAV is a simplistic idea that’s been around a long time. Understanding it is not the problem. Getting simplistic single queries out of an EAV database is easy. Getting a listing for reporting on large ranges of product; however, is easy AND horrifically memory intensive bloating time required for data acquisition from seconds to hours.

By: Rolando Garro

Rolando Garro — Wed, 08 Sep 2010 15:11:05 +0000

I just found this eav magento explanation http://www.magentocommerce.com/knowledge-base/entry/magento-for-dev-part-7-advanced-orm-entity-attribute-value/magento-for-dev-part-4-magento-layouts-blocks-and-templates
I feel like giving eav second chance.

By: CmdrAndreysn

CmdrAndreysn — Wed, 08 Sep 2010 06:24:34 +0000

And the performance of EAV got better in a negative direction when we went to 1.4.1.1. It’s not scaling well. Dataflow operations that took 30 minutes under 1.3.1.1 now take 3 hours. The upgrade adds another 100 tables. I recommend lots of hardware and lots of mysql tinkering.

By: Rolando Garro

Rolando Garro — Mon, 16 Aug 2010 17:39:38 +0000

Calling customer model in a webservice used by a point of sales wa s a big problem and we were just using it for getting some basic info, we had to create few functions in a new model using raw sql queries and everything went smooth again.

By: samser

samser — Mon, 26 Oct 2009 19:41:18 +0000

Really a good post.

Yes, my experience with magento database not so good too. For a store, I installed, upgrade, again install, fix … more than 10 times.

By: capncaveman

capncaveman — Mon, 26 Oct 2009 16:37:36 +0000

I am looking at using Magento to build a multiple stores website and move from IXXO cart. I currently have 47,000+ products and eventually I will have approx. 1.2 million.

In addition to tools and accessories we sell service parts for tools. Each manufacturer we carry has roughly 20,000 service part products each. With 50 manufacturers we carry that’s over 1 million service parts alone. Along with the service parts we will be adding .pdf breakdowns of the tools and providing a table below the .pdf image of all the parts associated with it and the user will be able to add them to the cart from that page.

Any idea if Magento can handle this database? Or should I ‘shop’ for another solution? B)

By: Ron Peled

Ron Peled — Wed, 19 Aug 2009 05:28:58 +0000

I am working on a plugin that integrates magento with a POS system. The db performance is terrible largely because of their data abstraction model. I agree their Collections objects are not efficient at all from a db standpoint. There is so much more that needs to be done with Magento – I am starting to think this is not ready for large stores (>2500 products).

By: daim

daim — Sat, 13 Jun 2009 09:08:19 +0000

@Branko Ajzele: Look at http://www.acris-antibodies.com we have many products and filters.
I update the products with a cron scripts all 5 minutes (only modified products) an rebuild every hour the searchindex (15min run this script and i wrote a extension for this, because when you generate the searchindex the search on frontpage dont find any product *magento*)

By: Ben

Ben — Fri, 12 Jun 2009 13:56:08 +0000

@Tomislav – I agree with you all the way, my friend! My use of “ultimate” was meant without the superlative connotation. Rather, I meant to denote that one of the side effects (goals?) of EAV is a maximum degree of normalization. You are spot-on with the assertion that “EAV turns the MySQL in something that it isn’t supposed to be” – it goes against sensibility to have such complexity when trying to represent data at its simplest, doesn’t it?

I find EAV to be a huge PITA, and so must Varien, given that they have implemented some rudimentary data warehousing with the Flat Catalog (improvements?) in the 1.3 branch.

I also agree that today’s underlying data storage technologies will evolve into something more intuitive, just as the programming languages have done. It will be much smarter people than I though who will figure out the most graceful way to represent and store non-static objects for the breadth of Web apps out there… For sure we’ll look back in ten years and say, “I can’t believe we used to work that /hard/!”

By: Tomislav Bilic

Tomislav Bilic — Fri, 12 Jun 2009 13:36:07 +0000

@Ben: “EAV is the ultimate in data normalization”

This could be another topic, but EAV is just a way to go over MySQL disadvantages. Relationship databases are simply not created for this purposes. I have a feeling that EAV turns the MySQL in something that it isn’t supposed to be.

In the years to come, I expect a new database to replace MySQL in open-source world. Amazon SimpleDB maybe?
http://aws.amazon.com/simpledb/

By: Ben

Ben — Fri, 12 Jun 2009 13:22:57 +0000

@inchoo & branko – Another good post, thanks as always for sharing your opinion. I think the downloads area of Magento’s site should use the html blink tag and some sweet animated flame GIFs to tell people to never never never upgrade a live site.

@pct:
- I agree that there is a ton of data in Magento, but I disagree completely that EAV is large because data is duplicated; EAV is the ultimate in data normalization.

- Magento modules have migration files, so running the upgrade should upgrade everything whether by SSH & pear or via MagentoConnect panel (which just runs pear anyway)

- As far as SSH installs & upgrades go (they really are the best way):
Setting up a Staging Server: http://www.crucialwebhost.com/blog/setting-up-a-magento-staging-area/ (There are a number of great tips here)

Miscellaneous info about upgrade permissions and PHP environment:
http://www.magentocommerce.com/boards/viewreply/61020/

By: pct

pct — Fri, 12 Jun 2009 10:16:30 +0000

This is not a comment about EAV in Magento, personally I find the EAV db okay… Biggest complaint I have of it is, that the volume of data in it is huge because so many things are duplicated in it. Especially now that flat thing came. Anyway, I’ll comment about the upgrade part.

Not long ago I upgraded one store with a bit over 30000 products from Magento 1.2.1 to Magento 1.3.1. The upgrade went okay, but only because I knew that it might not be an easy one (I had previously upgraded one site from 1.1.6 to 1.2.1 and that was a rough one).

There are few steps I always follow while upgrading.
1. No live upgrades (or at least close the site so that no one except you can view the frontend or backend). If someone else has access to the site and does page loads (in frontend or in backend), the database will most likely be corrupted after update.
2. Disable cache before doing anything else. Clear it and delete all the files beforehand. Clear all the logs also. And sessions.
3. Modify php.ini or .htaccess, increase the maximum execution time to something like few days and also increase the memory limits to as high as you dare. You might as well increase all the other related numbers. The default limist in php.ini were way too low to do upgrade or the flat db rebuild thing.
4. I haven’t found a way to start the upgrade process from shell. Upgrading the files is easy with pear, but I have no idea how to upgrade the db. And I’ve seen one or two failed upgrades (with older versions like 1.1.2 ->1.1.5) if I go first to the frontend and not to the backend. This is why I always log into the backend before doing the upgrade and refresh the page there (db gets upgraded).

Oh and one last thing. I try to upgrade the Magento sites as often as possible. I think all the upgrades so far have brought some db upgrades and I think that keeping the db version as recent as possible will avoid painful transitions later on. That was one of the reasons 1.1.6 -> 1.2.1 was painful. Another upgrade where I first upgraded to 1.1.8, then to 1.2.0 and after that to 1.2.1 went really well. This is of course only possible if your custom modules don’t touch the db directly but only through the number of layers Magento provides for db abstraction.

By: Branko Ajzele

Branko Ajzele — Fri, 12 Jun 2009 09:52:06 +0000

Hi Daim,

Seems like you have really, really large store. Would love to see how this is working on Magento

Wish you all the best in finding the best solution.

By: daim

daim — Fri, 12 Jun 2009 07:39:32 +0000

We have 200.000 Products with Magento 1.2.1 and i understand you full. We look to migrate to Oxid