Handling large queries. each() not helping.

I have process that could potentially result in a query large enough to exhaust my memory limit. It’s fine if the process takes a long time, as long as it eventually gets completed. I’ve been testing ways to limit memory usage with some code thrown into the generic CRUD update function:




//20,000 test items

foreach(Item::find()->each() as $item) {

    $customers = $item->customers; //many to many relation; this line causes a "byte exhausted" crash


    unset($customers);             //doesn't help

    $customers = NULL;             //also doesn't help


    $price = $item->price;         //works just fine; even if I use all() instead of each()

}



Admittedly, I’m not very experienced with PHP and I don’t understand how its garbage collection works. Looking at the memory usage provided by the Yii debug bar, the each() function is working just fine. The problem only seems to come up when I reference something that isn’t purely a member of Item. How do I get PHP to just let go of the memory?

What if you use:




$modelInfo = Item::findAll();

foreach(&modelInfo as $item) {

...

}

I believe the ->find() is being executed each time through the foreach() loop

How many $item->customers are there?

How much RAM do you have?

Why not use:




//20,000 test items

foreach(Item::find()->each() as $item) {

    $customers = [];

    foreach ( $item->getCustomers()->each() as $customer ) {

         $customers[] = $customer;

    }

    // use $customers here


    unset($customers);             //doesn't help

    $customers = NULL;             //also doesn't help


    $price = $item->price;         //works just fine; even if I use all() instead of each()

}



Your problem is that most likely you bring in too much data which exhaust your php memory while you try to store it, so you could also select only the fields you actually need in the queries and process the records as they come in instead of populating a huge array of objects.

  1. Why you want to get 20k rows? Is your algorithm’s logic correct?

  2. If is, you can look at queue managers;

  3. If you don’t want, you can use


::find()->with()->asArray()

The problem appear cause you make count($item) requests when fetching relations in loop.