Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Laravel local disk append() taking up a lot of memory

Storage::disk('local')
         ->append('/newsletter_lists/'.$contact_file_name.'.csv',
                $contact->email.','.$contact->salutation.','.$contact->lastname.','.$contact->unsubscribelink.','.$contact->bonus
          );

This is the code in Laravel that is writing to csv file inside a foreach loop (chunk loading from database). But as far as I know the append() method, is opening the file everytime, loads the content in memory, adds the new line and saves the file again, which with a large number of records (65 000) takes a long time. I am trying to find a way to append to the end of the file without loading the contents in memory if possible or am I am taking the wrong approach here?

like image 778
TooCooL Avatar asked Oct 29 '25 13:10

TooCooL


2 Answers

I took the @d3L advice and used plain php to append to files and not to use the append() method of laravel, which is loading the content everytime it has to append a line.

$file = fopen(Storage::disk('local')->path('test.csv'), "a+b");
like image 185
TooCooL Avatar answered Nov 01 '25 04:11

TooCooL


For laravel you can use the File facade which uses the Filesystem class rather than the FilesystemAdapter which is limited because it needs to use drivers which do not necessarily have the append feature and thus it has to implement it's own by reading the file and outputting it again which is very inefficient.

File::append(Storage::disk('local')->path('test.csv'), PHP_EOL."text to append")

The former doesn't have this limitation and under the hood uses

file_put_contents($path, $data_to_append, FILE_APPEND);
like image 26
Tofandel Avatar answered Nov 01 '25 03:11

Tofandel



Donate For Us

If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!