Using Zend for intensive data processing

Related Inchoo Services

Sometimes there are cases where a lot of data processing needs to be done and using Magento models and resources is either too slow or too intensive for your solution. This is when Zend framework on which Magento is built upon jumps in. Naturally, you can write raw php/mysql functionality for your needs, but if you want to keep your code clean and reusable, using Zend functionality is the way.

Requests like making feed generator or some other custom scripts that will be too heavy if used with initializing Magento are quite often and can be accomplished by writing a shell script (located in shell directory of Magento project root) which will use only needed resources to complete the task.

Our example script will fetch a couple of basic product attributes, like sku, name, and weight directly from Magento EAV tables.

First step of the script is to create backbone from Magento class Mage_Shell_Abstract that includes Mage.php needed for basic initialization project classes. We will also set up our class and its attributes needed for this example. Lets create a file called inchoo-example.php in our shell directory inside Magento project:

require_once 'abstract.php';
 
class Inchoo_Custom_Script extends Mage_Shell_Abstract
{
 
    public $r = null;
    public $c = null;
    public $entityTypeId = null;
    public $productTypeId = null;
    public $tables = array();
 
    public function _construct()
    {
        /** @var Mage_Core_Model_Resource r */
        /** @var Varien_Db_Adapter_Interface c */
 
        // Init connection and tables
        $this->r = Mage::getSingleton('core/resource');
        $this->c = $this->r->getConnection(Mage_Core_Model_Resource::DEFAULT_WRITE_RESOURCE);
 
        // Entity type for products
        $this->entityTypeId = Mage::getModel('eav/config')
            ->getEntityType(Mage_Catalog_Model_Product::ENTITY)
            ->getEntityTypeId();
 
        // Product type ID
        $this->productTypeId = Mage_Catalog_Model_Product_Type::TYPE_SIMPLE;
 
        // Table names
        $this->tables['cpe'] = $this->r->getTableName('catalog_product_entity');
        $this->tables['ea'] = $this->r->getTableName('eav_attribute');
        $this->tables['cpev'] = $this->r->getTableName('catalog_product_entity_varchar');
        $this->tables['cped'] = $this->r->getTableName('catalog_product_entity_decimal');
        $this->tables['cpei'] = $this->r->getTableName('catalog_product_entity_int');
    }
}
 
$script = new Inchoo_Custom_Script();
$script->run();

Besides extending Mage_Shell_Abstract we made a set up of things we need for script processing in construct, such as determine table names and entity types we will use in data fetching from tables. We will use eav_attribute table to get ID of attribute based on attribute name. ID will be used in catalog_product_entity_decimal table for fetching weight, catalog_product_entity_varchar for fetching name, and catalog_product_entity_int for fetching status of the product (wheater product is enabled or not).

Next step is the core of our application, Zend sql call that will use data we set up in construct:

public function run()
    {
        $queryData = $this->c->select()
            ->from(
                array('cpe' => $this->tables['cpe']),
                array(
                    'ID' => 'cpe.entity_id',
                    'SKU' => 'cpe.sku'
                )
            )
            // Join Name
            ->joinInner(
                array('ea' => $this->tables['ea']),
                "ea.attribute_code = 'name' AND ea.entity_type_id = {$this->entityTypeId}",
                null
            )
            ->joinLeft(
                array('cpev' => $this->tables['cpev']),
                'cpev.attribute_id = ea.attribute_id AND cpev.entity_id = cpe.entity_id',
                array('name' => 'cpev.value')
            )
            // Join Weight
            ->joinInner(
                array('ea1' => $this->tables['ea']),
                "ea1.attribute_code = 'weight' AND ea1.entity_type_id = {$this->entityTypeId}",
                null
            )
            ->joinLeft(
                array('cped' => $this->tables['cped']),
                'cped.attribute_id = ea1.attribute_id AND cped.entity_id = cpe.entity_id',
                array('weight' => 'cped.value')
            )
            // Join Status
            ->joinInner(
                array('ea2' => $this->tables['ea']),
                "ea2.attribute_code = 'status' AND ea2.entity_type_id = {$this->entityTypeId}",
                null
            )
            ->joinInner(
                array('cpei' => $this->tables['cpei']),
                'cpei.attribute_id = ea2.attribute_id AND cpei.entity_id = cpe.entity_id',
                null
            )
            ->where("cpei.value = 1 AND cpe.type_id = '{$this->productTypeId}'")
            ->limit(10);
 
        $values = $this->c->query($queryData)->fetchAll(Zend_Db::FETCH_UNIQUE);
 
        return $values;
    }

Function call to $this->c->select() initializes Varien_Db_Select which handles query build with the help of Zend_Db_Select. Inside app/code/core/Zend/Db/Select.php you can find class specifications and functions used in this example, such as from(), various joins, and other supported wrappers for mysql queries.

If you’re fetching a large amount of data and you run out of memory, you can wrap up sql call in a while loop and call results one by one, which will result in slower execution but will not waste as much memory:

public function run()
    {
        $queryData = $this->c->select()
            ->from(
                array('cpe' => $this->tables['cpe']),
                array(
                    'ID' => 'cpe.entity_id',
                    'SKU' => 'cpe.sku'
                )
            )
            // Join Name
            ->joinInner(
                array('ea' => $this->tables['ea']),
                "ea.attribute_code = 'name' AND ea.entity_type_id = {$this->entityTypeId}",
                null
            )
            ->joinLeft(
                array('cpev' => $this->tables['cpev']),
                'cpev.attribute_id = ea.attribute_id AND cpev.entity_id = cpe.entity_id',
                array('name' => 'cpev.value')
            )
            // Join Weight
            ->joinInner(
                array('ea1' => $this->tables['ea']),
                "ea1.attribute_code = 'weight' AND ea1.entity_type_id = {$this->entityTypeId}",
                null
            )
            ->joinLeft(
                array('cped' => $this->tables['cped']),
                'cped.attribute_id = ea1.attribute_id AND cped.entity_id = cpe.entity_id',
                array('weight' => 'cped.value')
            )
            // Join Status
            ->joinInner(
                array('ea2' => $this->tables['ea']),
                "ea2.attribute_code = 'status' AND ea2.entity_type_id = {$this->entityTypeId}",
                null
            )
            ->joinInner(
                array('cpei' => $this->tables['cpei']),
                'cpei.attribute_id = ea2.attribute_id AND cpei.entity_id = cpe.entity_id',
                null
            )
            ->where("cpei.value = 1 AND cpe.type_id = '{$this->productTypeId}'")
            ->limit(10);
 
        $values = $this->c->query($queryData);
 
        $return = array();
        while($result = $values->fetch()) {
            // Do something with data here
            $return[] = $result;
        }
 
        return $return;
    }

When everything is done you can call script from Magento root with php -f shell/inchoo-example.php.

This is it!

Hopefully, this will help you automatize frequent or one time heavy duty tasks for your project!

You made it all the way down here so you must have enjoyed this post! You may also like:

PHP 7 and Magento – a quick overview Ivona Namjesnik
Ivona Namjesnik, | 2

PHP 7 and Magento – a quick overview

Using Redis cache backend and session storage in Magento Marko Martinovic
Marko Martinovic, | 43

Using Redis cache backend and session storage in Magento

Consuming Magento REST service using Zend_OAuth_Consumer Darko Goles
Darko Goles, | 44

Consuming Magento REST service using Zend_OAuth_Consumer

Leave a Reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <blockquote cite=""> <code> <del datetime=""> <em> <s> <strike> <strong>. You may use following syntax for source code: <pre><code>$current = "Inchoo";</code></pre>.