Yii Framework Forum: Handling large queries. each() not helping. - Yii Framework Forum

Jump to content

Page 1 of 1
  • You cannot start a new topic
  • You cannot reply to this topic

Handling large queries. each() not helping. Running out of memory when calling relation

#1 User is offline   Aver 

  • Newbie
  • Yii
  • Group: Members
  • Posts: 6
  • Joined: 05-June 15

Posted 02 November 2017 - 09:02 AM

I have process that could potentially result in a query large enough to exhaust my memory limit. It's fine if the process takes a long time, as long as it eventually gets completed. I've been testing ways to limit memory usage with some code thrown into the generic CRUD update function:

//20,000 test items
foreach(Item::find()->each() as $item) {
    $customers = $item->customers; //many to many relation; this line causes a "byte exhausted" crash

    unset($customers);             //doesn't help
    $customers = NULL;             //also doesn't help

    $price = $item->price;         //works just fine; even if I use all() instead of each()
}


Admittedly, I'm not very experienced with PHP and I don't understand how its garbage collection works. Looking at the memory usage provided by the Yii debug bar, the each() function is working just fine. The problem only seems to come up when I reference something that isn't purely a member of Item. How do I get PHP to just let go of the memory?
0

#2 User is offline   jkofsky 

  • Master Member
  • PipPipPipPip
  • Yii
  • Group: Members
  • Posts: 687
  • Joined: 17-May 10
  • Location:Pensacola, Florida

Posted 02 November 2017 - 12:37 PM

View PostAver, on 02 November 2017 - 09:02 AM, said:

I have process that could potentially result in a query large enough to exhaust my memory limit. It's fine if the process takes a long time, as long as it eventually gets completed. I've been testing ways to limit memory usage with some code thrown into the generic CRUD update function:

//20,000 test items
foreach(Item::find()->each() as $item) {
    $customers = $item->customers; //many to many relation; this line causes a "byte exhausted" crash

    unset($customers);             //doesn't help
    $customers = NULL;             //also doesn't help

    $price = $item->price;         //works just fine; even if I use all() instead of each()
}


Admittedly, I'm not very experienced with PHP and I don't understand how its garbage collection works. Looking at the memory usage provided by the Yii debug bar, the each() function is working just fine. The problem only seems to come up when I reference something that isn't purely a member of Item. How do I get PHP to just let go of the memory?

What if you use:
$modelInfo = Item::findAll();
foreach(&modelInfo as $item) {
...
}

I believe the ->find() is being executed each time through the foreach() loop
How many $item->customers are there?
How much RAM do you have?
Do, or do not. There is no 'try.' Jedi Master Yoda
0

#3 User is offline   twisted1919 

  • Master Member
  • PipPipPipPip
  • Yii
  • Group: Members
  • Posts: 697
  • Joined: 23-October 10
  • Location:Romania

Posted 02 November 2017 - 03:07 PM

Why not use:
//20,000 test items
foreach(Item::find()->each() as $item) {
    $customers = [];
    foreach ( $item->getCustomers()->each() as $customer ) {
         $customers[] = $customer;
    }
    // use $customers here

    unset($customers);             //doesn't help
    $customers = NULL;             //also doesn't help

    $price = $item->price;         //works just fine; even if I use all() instead of each()
}


Your problem is that most likely you bring in too much data which exhaust your php memory while you try to store it, so you could also select only the fields you actually need in the queries and process the records as they come in instead of populating a huge array of objects.
0

#4 User is offline   umneeq 

  • Standard Member
  • PipPip
  • Yii
  • Group: Members
  • Posts: 149
  • Joined: 10-October 11
  • Location:Russia

Posted 03 November 2017 - 03:23 AM

1. Why you want to get 20k rows? Is your algorithm's logic correct?
2. If is, you can look at queue managers;
3. If you don't want, you can use
::find()->with()->asArray()


The problem appear cause you make count($item) requests when fetching relations in loop.
0

Share this topic:


Page 1 of 1
  • You cannot start a new topic
  • You cannot reply to this topic

1 User(s) are reading this topic
0 members, 1 guests, 0 anonymous users