[2.4] PHP Fatal Error - Memory Exhausted

[Daerik]

Version 2.4
CPU Intel(R) Core(TM) i7 CPU 870 @ 2.93GHz (8 core(s))
OS CentOS 6.9 (Final)
Memory 15.57 GB

PHP Fatal error: Allowed memory size of 536870912 bytes exhausted (tried to allocate 4096 bytes) in /var/www/vhosts/domain.tld/httpdocs/phrets/vendor/troydavisson/phrets/src/Models/Search/Record.php on line 26

With each Search I've noticed the memory is constantly rising using memory_get_peak_usage().

I originally found a work-around by increasing memory_limit to 512M, but now the client wants to include sold properties (essentially tripling the returned results from FlexMLS). Naturally, I can increase the memory_limit, but I just recently updated from 1.x to 2.4. I did not run into the memory issue with the older version.

Update 2017-08-31

I've increased the memory_limit to 1024M with the following fatal error:
PHP Fatal error: Allowed memory size of 1073741824 bytes exhausted (tried to allocate 1 bytes) in /var/www/vhosts/domain.tld/httpdocs/phrets/vendor/troydavisson/phrets/src/Parsers/Search/OneX.php on line 119

Update 2017-08-31 (2)

I've increased the memory_limit to 3G with the following returned log:

Fetching FlexMLS Master Data: CommercialLease
Total Records Found: 291
Memory Used: 16.47 MB of 3 GB

Fetching FlexMLS Master Data: CommercialSale
Total Records Found: 294
Memory Used: Increased from 16.47 MB to 23.12 MB (+6.65 MB) - 23.12 MB of 3 GB

Fetching FlexMLS Master Data: MultiFamily
Total Records Found: 67
Memory Used: Increased from 23.12 MB to 27.36 MB (+4.25 MB) - 27.36 MB of 3 GB

Fetching FlexMLS Master Data: OpenHouse
Total Records Found: 137
Memory Used: 27.36 MB of 3 GB

Fetching FlexMLS Master Data: Rental
Total Records Found: 791
Memory Used: Increased from 27.36 MB to 53.09 MB (+25.72 MB) - 53.09 MB of 3 GB

Fetching FlexMLS Master Data: Residential
Total Records Found: 4418
Memory Used: Increased from 53.09 MB to 301.98 MB (+248.89 MB) - 301.98 MB of 3 GB

Fetching FlexMLS Master Data: VacantLand
Total Records Found: 2262
Memory Used: Increased from 301.98 MB to 378.87 MB (+76.9 MB) - 378.87 MB of 3 GB

Fetching FlexMLS Sold Data: CommercialLease
Total Records Found: 575
Memory Used: 378.87 MB of 3 GB

Fetching FlexMLS Sold Data: CommercialSale
Total Records Found: 404
Memory Used: 378.87 MB of 3 GB

Fetching FlexMLS Sold Data: MultiFamily
Total Records Found: 303
Memory Used: 378.87 MB of 3 GB

Fetching FlexMLS Sold Data: Residential
Total Records Found: 40326
Memory Used: Increased from 378.87 MB to 2.64 GB (+2.27 GB) - 2.64 GB of 3 GB

[steveheinsch]

Are you sure it's PHRETS causing this and not possibly some outside influence? I had an issue where the framework I'm using, laravel, by default stores all of the previously run queries in memory for debugging and other purposes. This caused each loop in my search, where I'm inserting all of the results from my paginated search, to overrun the memory after about 1000 inserts. Once I figured that out and disabled query logging, I could pull hundreds of thousands of listings and store them without consuming more than 7M, for the entire cycle...even when running for hours continuously like pulling down all records since 2000.

[Daerik]

@steveheinsch 1.0rc2 used to run out of memory until I began using FreeResult(). I've checked my logs and see nowhere it is caching the queries. I've updated my issue with more information. For sake of targeting the performance strictly to 2.4, I've disabled all data entry/manipulation and I'm only fetching results.

[1014825794]

Hey sir. We have a problem that can not be solved, so we ask you for help 。 First of all, we have MLS accounts that you got from them. They are: $rets_login_url = "https://pt.rets.crmls.org/contact/rets/login"; $rets_username = "**FANG"; $rets_password = '**!y$jTv'; I can make sure this account is available，I can make sure this account is available, because I can get the data through this account(That is the basic information Housing)， I got the data through this phprets 1.x and put it in the CSV, which is what we need。 I see your document is using this method: GetObject ("property", "photo", "*", "1") ,to get the picture. There are a lot of mlsid fields involved in CSV, but I don't know which field to associate to get the corresponding picture. So please help me, thank you very much!!! Good nice to you!!!

…

------------------ 原始邮件 ------------------ 发件人: "Eric Price";<[email protected]>; 发送时间: 2017年8月31日(星期四) 晚上9:58 收件人: "troydavisson/PHRETS"<[email protected]>; 抄送: "Subscribed"<[email protected]>; 主题: Re: [troydavisson/PHRETS] [2.4] PHP Fatal Error - Memory Exhausted(#167) Update 2017-08-31 I've increased the memory_limit to 1024M with the following fatal error: PHP Fatal error: Allowed memory size of 1073741824 bytes exhausted (tried to allocate 1 bytes) in /var/www/vhosts/domain.tld/httpdocs/phrets/vendor/troydavisson/phrets/src/Parsers/Search/OneX.php on line 119 — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub, or mute the thread.

[1014825794]

Hey sir. We have a problem that can not be solved, so we ask you for help 。 First of all, we have MLS accounts that you got from them. They are: $rets_login_url = "https://pt.rets.crmls.org/contact/rets/login"; $rets_username = "**FANG"; $rets_password = '**!y$jTv'; I can make sure this account is available，I can make sure this account is available, because I can get the data through this account(That is the basic information Housing)， I got the data through this phprets 1.x and put it in the CSV, which is what we need。 I see your document is using this method: GetObject ("property", "photo", "*", "1") ,to get the picture. There are a lot of mlsid fields involved in CSV, but I don't know which field to associate to get the corresponding picture. So please help me, thank you very much!!! Good nice to you!!!

…

------------------ 原始邮件 ------------------ 发件人: "Eric Price";<[email protected]>; 发送时间: 2017年9月1日(星期五) 上午7:04 收件人: "troydavisson/PHRETS"<[email protected]>; 抄送: "Subscribed"<[email protected]>; 主题: Re: [troydavisson/PHRETS] [2.4] PHP Fatal Error - Memory Exhausted(#167) @steveheinsch 1.0rc2 used to run out of memory until I began using FreeResult(). I've checked my logs and see nowhere it is caching the queries. I've updated my issue with more information. For sake of targeting the performance strictly to 2.4, I've disabled all data entry/manipulation and I'm strictly just fetching results. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub, or mute the thread.

[1014825794]

Hey sir. We have a problem that can not be solved, so we ask you for help 。 First of all, we have MLS accounts that you got from them. They are: $rets_login_url = "https://pt.rets.crmls.org/contact/rets/login"; $rets_username = "**FANG"; $rets_password = '**!y$jTv'; I can make sure this account is available，I can make sure this account is available, because I can get the data through this account(That is the basic information Housing)， I got the data through this phprets 1.x and put it in the CSV, which is what we need。 I see your document is using this method: GetObject ("property", "photo", "*", "1") ,to get the picture. There are a lot of mlsid fields involved in CSV, but I don't know which field to associate to get the corresponding picture. So please help me, thank you very much!!! Good nice to you!!!

…

------------------ 原始邮件 ------------------ 发件人: "Steve Heinsch";<[email protected]>; 发送时间: 2017年9月1日(星期五) 凌晨1:52 收件人: "troydavisson/PHRETS"<[email protected]>; 抄送: "Subscribed"<[email protected]>; 主题: Re: [troydavisson/PHRETS] [2.4] PHP Fatal Error - Memory Exhausted(#167) Are you sure it's PHRETS causing this and not possibly some outside influence? I had an issue where the framework I'm using, laravel, by default stores all of the previously run queries in memory for debugging and other purposes. This caused each loop in my search, where I'm inserting all of the results from my paginated search, to overrun the memory after about 1000 inserts. Once I figured that out and disabled query logging, I could pull hundreds of thousands of listings and store them without consuming more than 7M, for the entire cycle...even when running for hours continuously like pulling down all records since 2000. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub, or mute the thread.

[1014825794]

Hey sir. We have a problem that can not be solved, so we ask you for help 。 First of all, we have MLS accounts that you got from them. They are: $rets_login_url = "https://pt.rets.crmls.org/contact/rets/login"; $rets_username = "**FANG"; $rets_password = '**!y$jTv'; I can make sure this account is available，I can make sure this account is available, because I can get the data through this account(That is the basic information Housing)， I got the data through this phprets 1.x and put it in the CSV, which is what we need。 I see your document is using this method: GetObject ("property", "photo", "*", "1") ,to get the picture. There are a lot of mlsid fields involved in CSV, but I don't know which field to associate to get the corresponding picture. So please help me, thank you very much!!! Good nice to you!!!

…

------------------ 原始邮件 ------------------ 发件人: "Eric Price";<[email protected]>; 发送时间: 2017年8月31日(星期四) 晚上9:58 收件人: "troydavisson/PHRETS"<[email protected]>; 抄送: "Subscribed"<[email protected]>; 主题: Re: [troydavisson/PHRETS] [2.4] PHP Fatal Error - Memory Exhausted(#167) Update 2017-08-31 I've increased the memory_limit to 1024M with the following fatal error: PHP Fatal error: Allowed memory size of 1073741824 bytes exhausted (tried to allocate 1 bytes) in /var/www/vhosts/domain.tld/httpdocs/phrets/vendor/troydavisson/phrets/src/Parsers/Search/OneX.php on line 119 — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub, or mute the thread.