Large processes in PhalconPHP

Question

I have webapp that is logging application and I need backup/restore/import/export feature there. I did this successfully with laravel but have some complications with Phalcon. I don't see native functions in phalcon that would split on chunks execution of large php scripts.

The thing is that logs will be backed up and restored as well as imported by users in ADIF format (adif.org) I have parser for that format which converts file to array of arrays then every record should search through another table, containing 2000 regular expressions, and find 3-10 matches there and connect imported records in one table to those in another table (model relation hasMany) That means that every imported record should have quite some processing time. laravel did it somehow with 3500 records imported, I dont know how it will handle more. The average import will contain 10000 records and each of them need to be verified with 2000 regular expression.

The main issue is how to split this huge processing mount into smaller chunks so I wouldnt get timeouts?

Answer 1

Here is the function that could flawlessly do the job with adding 3862 records in one table and as a result of processing of every record add 8119 records in another table:

public function restoreAction()
{
    $this->view->disable();

    $user = Users::findFirst($this->session->auth['id']);

    if ($this->request->isPost()) {
        if ($this->request->isAjax()) {

            $frontCache = new CacheData(array(
                "lifetime" => 21600
            ));

            $cache = new CacheFile($frontCache, array(
                "cacheDir" => "../plc/app/cache/"
            ));

            $cacheKey = $this->request->getPost('fileName').'.cache';

            $records = $cache->get($cacheKey);

            if ($records === null) {
                $rowsPerChunk = 50;
                $adifSource = AdifHelper::parseFile(BASE_URL.'/uploads/'.$user->getUsername().'/'.$this->request->getPost('fileName'));
                $records = array_chunk($adifSource, $rowsPerChunk);
                $key = array_keys($records); 
                $size = count($key); 
            }

            for ($i = 0; $i < $size; $i++) {
                if (!isset($records[$i])) {
                    break;
                }
                set_time_limit(50);
                for ($j=0; $j < $rowsPerChunk; $j++) { 
                    $result = $records[$i][$j];
                    if (!isset($result)) {
                        break;
                    }
                    if(isset($result['call'])) {
                        $p = new PrefixHelper($result['call']);
                    }
                    $bandId = (isset($result['band']) && (strlen($result['band']) > 2)) ? Bands::findFirstByName($result['band'])->getId() : null;
                    $infos = (isset($p)) ? $p->prefixInfo() : null;
                    if (is_array($infos)) {
                        if (isset($result['qsl_sent']) && ($result['qsl_sent'] == 'q')) {
                            $qsl_rcvd = 'R';
                        } else if (isset($result['eqsl_qsl_sent']) && ($result['eqsl_qsl_sent'] == 'c')) {
                            $qsl_rcvd = 'y';
                        } else if (isset($result['qsl_rcvd'])) {
                            $qsl_rcvd = $result['qsl_rcvd'];
                        } else {
                            $qsl_rcvd ='i';
                        }
                        $logRow = new Logs();
                        $logRow->setCall($result['call']);
                        $logRow->setDatetime(date('Y-m-d H:i:s',strtotime($result['qso_date'].' '.$result['time_on'])));
                        $logRow->setFreq(isset($result['freq']) ? $result['freq'] : 0);
                        $logRow->setRst($result['rst_sent']);
                        $logRow->setQslnote(isset($result['qslmsg']) ? $result['qslmsg'] : '');
                        $logRow->setComment(isset($result['comment']) ? $result['comment'] : '');
                        $logRow->setQslRcvd($qsl_rcvd);
                        $logRow->setQslVia(isset($result['qsl_sent_via']) ? $result['qsl_sent_via'] : 'e');
                        $logRow->band_id = $bandId;
                        $logRow->user_id = $this->session->auth['id'];
                        $success = $logRow->save();

                        if ($success) {
                            foreach ($infos as $info) {
                                if (is_object($info)) {
                                    $inf = new Infos();
                                    $inf->setLat($info->lat);
                                    $inf->setLon($info->lon);
                                    $inf->setCq($info->cq);
                                    $inf->setItu($info->itu);
                                    if (isset($result['iota'])) {
                                        $inf->setIota($result['iota']);
                                    }
                                    if (isset($result['pfx'])) {
                                        $inf->setPfx($result['pfx']);
                                    }
                                    if (isset($result['gridsquare'])) {
                                        $inf->setGrid($result['gridsquare']);
                                    } else if (isset($result['grid'])) {
                                        $inf->setGrid($result['grid']);
                                    }
                                    $inf->qso_id = $logRow->getId();
                                    $inf->prefix_id = $info->id;

                                    $infSuccess[] = $inf->save();
                                }
                            }
                        }
                    }
                }
                sleep(1);
            }

        }       
    }
}

I know, the script needs a lot of improvement but for now the task was just to make it work.

Answer 2

I think that the good practice for large processing task in php is console applications, that doesn't have restrictions in execution time and can be setup with more memory for execution.

As for phalcon, it has builtin mechanism for running and processing cli tasks - Command Line Applications (this link will always point to the documentation of a phalcon latest version)