Yii 1.1: backjob

Runs actions as background jobs and can report on their progress
27 followers

BackJob can start an action in the background (by using a request), runs it either as the current user or as anonymous, and monitors its progress.

For running jobs at remote locations, fire-and-forget jobs, interval/cron jobs, please look at the runactions extension.

Requirements

Works with Yii-1.1.14 and up. Needs to be able to use fsockopen and fwrite to itself.

Important: If you want to run background jobs as the current user (which is the default option), you must use some kind of non-blocking session storage, such as CDbHttpSession. Also make sure that user is authorised to access that action.

Installation

Put the source from the zip archive in protected/extensions or from https://github.com/greenhost/BackJob in protected/extensions/backjob.

Configuration:

// Yes, it needs preloads, but it's not resource-heavy (promise!)
'preload' => array(
    'background'
),
 
 
'components' => array(
    'background' => array(
        'class' => 'ext.backjob.EBackJob',
 
        // All other configuration options are optional:
 
        'checkAndCreateTable' => true,  // creates table if it doesn't exist
        'key' => 'sjs&sk&F89fksL*987sdKf' // Random string used to salt the hash used for background-thread-authentication. Optional to change, but you really should.
        'useDb' => true,    // Use a database table
        'useCache' => true, // Use the cache
        'db' => 'db',    // Database component name to use
        'ch' => 'cache', // Cache component name to use
        'tableName' => 'e_background_job', // Name of DB table used
        'cachePrefix' => 'EBackJobPrefix-',  // Prefix used in the cache
        'errorTimeout' => 60, // Nr of seconds after which ALL requests time out, measured from the last update.
        'userAgent' => 'Mozilla/5.0 Firefox/3.6.12', // Useragent used for the background request
        'backlogDays' => 30, // Number of days successfully completed request-entries are retained in the database
        'allBacklogDays' => 60, // Number of days ALL entries (including failed) are retained in the database
    ),
),

Usage

It's possible to start any controllers' action that is reachable through a url, but recommended to have dedicated actions to be run in the background. You'll have to make your own progress reports, otherwise progress will just jump from 0 to 100.

Starting a background job only requires a valid route to that controller/action.

$jobId = Yii::app()->background->start('site/longJob');
// Or, with parameters:
$jobWithParams = Yii::app()->background->start(
    array(
        'site/paramJob', 
        'id'=>$id, 
        'param2'=>true
    )
);

Then you'll probably want to use a time-intervaled ajax request to get the progress. Getting the status of a specific job. This returns an array of the form:

$status = Yii::app()->background->getStatus($jobId);
//This returns an array that looks something like this:
array(
    'progress' => 20, //percentage (integer 0-100) of completeness
    'status' => EBackJob::STATUS_INPROGRESS, // (integer 0-4)
    'start_time' => '2013-11-18 14:11',
    'updated_time' => '2013-11-18 14:11',
    'end_time' => '2013-11-18 14:11',
    'status_text' => 'The complete output of the request, so far',
);

During the background job, the action that actually runs the job itself can update its progress both by echoing and setting the progress counter:

echo "Starting 1<br/>";
Yii::app()->background->update(20);
do_long_function1();
echo "Processing 2<br/>";
Yii::app()->background->update(60);
if(!do_long_function2()){
    echo "Error occurred!";
    Yii::app()->background->fail(); // this also ends the application immediately!
}
echo "Finishing 3<br/>";
Yii::app()->background->update(90);
do_last_function3();
echo "Done<br/>";

If you don't want a list or log of echoed text, but replace it, you can use the update function like this, but make sure that you also finish manually.

Yii::app()->background->update(array('progress'=>60,'status_text'=>'Chugging along now');
Yii::app()->background->finish(array('status_text'=>'And done');

POST and other methods

Any HTTP1.1 method can be used (POST, PUT, DELETE, CHICKEN), and it can send both "GET" data in the url and url-encoded data in the body (sorry, no multipart/file support, you can bypass that by passing temporary filenames). These can be set in the request array using the keys backjobMethod and backjobPostdata respectively. Yes, this means that you can't use your own fields called backjobMethod and backjobPostdata, but this was the best way to add this while maintaining backwards compatibility.

Example: To make a POST call with $\_POST values for id and name and a $\_GET value for page do:

Yii::app()->background->start(array('test/testbackground', 'page'=>1, 'backjobMethod'=>'POST', 'backjobPostdata'=>array('id'=>5, 'name'=>'Name')));

Complete example

class testController extends Controller {
 
    public function actionProgressMonitor(){
        $job = Yii::app()->background->start(array('test/testbackground'));
        $this->render('progress'); // empty file, or containing:
        echo "Progress: <div id='test'></div>";
        echo CHtml::script("$(function(){ setInterval(function(){ $('#test').load('".$this->createUrl('test/getStatus',array('id'=>$job))."');}, 1000);});");
    }
    public function actionGetStatus($id){
        echo json_encode(Yii::app()->background->getStatus($id));
        Yii::app()->end();
    }
    public function actionTestbackground(){
        Yii::app()->background->update(1);
        echo "Job started.";
        sleep(3);
        Yii::app()->background->update(20);
        sleep(3);
        Yii::app()->background->update(40);
        echo "Job in progress.";
        sleep(3);
        Yii::app()->background->update(60);
        sleep(3);
        Yii::app()->background->update(80);
        sleep(3);
        echo "Job done.";
        Yii::app()->end();
    }
}

Changelog

  • 0.50 - Changed the way internal requests are recognised, they are now checked by parameters instead of from-headers, should work nicer with proxied servers, and be more resilient against spoofing. Change the key field in configuration!
  • 0.45 - Added support for other HTTP methods and POST-data.
  • 0.44 - The timeout-setting now also affects the php-timeout setting with set_time_limit, thanks to martijnjonkers.
  • 0.43 - Added backlog-cleanup for the database, so that it won't fill up with completed requests. Keep in mind that there are two different time-scales, one for successfully finished jobs, and one for all jobs including failed ones. Setting these to 0 days will stop cleanup entirely, this might lead to an ever-expanding database! Thanks to Arno S for noticing this omission.
  • 0.42 - Few bugfixes: creation of table, cache was unuseable, a typo
  • 0.41 - Small bugfix
  • 0.40 - Added monitoring thread that waits for the job to end, so requests that end prematurely still finish. Also, you can specify a number of seconds to wait until processing, and multiple requests with the exact same route will be merged together into one request.
  • 0.33 - Https support, better self-recognition, added request field (update your database table!), added global timeout.
  • 0.32 - Fails better.
  • 0.31 - Initial component

Total 20 comments

#19809 report it
Alexandre Rodichevski at 2016/03/19 05:05am
Cookie validation problem

The background session as the current user does not work when the application uses the cookie validation scheme (enableCookieValidation = true). In fact, BackJob reuses the same session cookie for the new background session; this is considered by Yii as a session cookie tampering.

See an alternative approach which works in this case.

#19592 report it
ydakilux at 2015/09/24 05:33am
Yii2 implementation

Hello,

Do you plan to implement a Yii2 version ?

#19204 report it
oma378501 at 2015/04/15 09:55am
send email when finishes job

My question is where I should add the function for send email

Greetings in advance

#18668 report it
maxtorchel at 2014/12/09 11:06am
fsockopen ssl in php 5.6

in php 5.6 fsockopen() function verify certificate enabled by default and if you have self signed cert you get an error 'certificate verify failed'. I fixed it by change 'fsockopen(...)' in line 481 to

stream_socket_client(($port == 443 ? 'ssl://' : '') . $host.':'.$port, $errno, $errstr, $this->errorTimeout,STREAM_CLIENT_CONNECT,$context)

and add into this function context param:

$context = stream_context_create(array(
        'http' => array(
            'header' => 'Cookie: PHPSESSID='.Yii::app()->session->sessionID
        ),
        "ssl"=>array(
            "verify_peer"=>false,
            "verify_peer_name"=>false,
        )
    ));

which disable ssl check. Hope it help someone.

#18358 report it
Ravi Bhalodiya at 2014/10/20 02:12am
Not working on Server

This is very nice extension. I tried to send lots mail using this extension and its is working fine in local and save process time. When I deployed application on server it is not working.

  1. Can I have to do any extra configuration for that??
  2. Can I make call of start function in loop??

Please help me. Thanks in Advance.

#18149 report it
CrisTan at 2014/09/16 11:39am
How to kill the process

Nice extension! Is very usefull for working with a lot of data!

I have one question: How can i kill the backjob, while it is still running, whithout restarting the apache? I was thinking for an extra column in the db to use with a new status, but i didn't figure it out yet.

One problem that i found, when running more than one Backjob: When 1 job is finished, and progress is 100, it will stay in the db as completed, but it will delete from db the other running jobs. They are still running, and doing their job, but i won't get any status from them. It's also affecting the progressMonitor, where i will have a finished job, and the other jobs will be frozen.

Thanks :)

#18102 report it
Siquo at 2014/09/08 10:09am
POST support

Hey Radoo, thanks for the compliment and typo find! Yes, it should support POST (in 20/20 hindsight), which took a bit longer than I expected. I've got that part working, but then it also needed to support DELETE and PUT and arbitrary methods... I'll probably finish that tomorrow and I'll upload the new version then!

Why it's not working on the WAMP stack... I don't know. Caching is a bit hit-and-miss at the moment (badum-tss).

Edit: new version is live, should support both GET and POST and arbitrary methods now. The request body is always url-encoded.

#18039 report it
radoo at 2014/08/30 04:21am
excellent extension

On Windows wamp environment it launches the background job, but the background job seems to stay stuck somehow at the 0% moment. Also, at least on windows, could not get it to work until i set caching to zero in the ext configuration.

The exact same configuration (but with CDbHttpSession from yii core instead of the CHttpDbSession you posted as requirement, although that might be a typo in your Requirements section) works exactly as expected on Linux, haven't tested with cache true tough.

One thing I need to do is pass some variables to the backjob through _POST, not really clear how to do this yet, maybe you could help.

#17947 report it
Siquo at 2014/08/14 10:56am
Re: Not running in background

Hi Steve,

Problems could include your hosting environment (A very low MaxClients value in Apache, although this is unlikely), but most likely you're not using CHttpDbSession for your session storage (see requirements). If you don't wat to use those, a way to work around that is doing the background request as an anonymous user.

#17946 report it
SteveVB at 2014/08/14 10:43am
Not running in background

I tried launching a background job with Yii::app()->background->start And the controller launching it didn't complete till the background job completed. Are there situations where this would happen? How do I get it to actually run in the background as requested?

#17726 report it
Siquo at 2014/07/17 10:06am
timeout

Thanks for spotting that! I included the line in the init() function though, so that both the monitoring thread and the actual job-thread have a (possibly) increased duration.

#17701 report it
martijnjonkers at 2014/07/16 04:31am
timeout

there is a timeout option, but it does not update the global time limit. So when setting the timeout to 60 seconds the monitor will still fail at 30 seconds (or whatever is configured).

I suggest adding set_time_limit in the moonitor() method:

/**
 * The monitor thread. Starts a background request and reports on its progress or failure.
 */
protected function monitor(){
 
        set_time_limit($this->errorTimeout + 5); // add some seconds to handle the timeout
 
        ...
}
#17553 report it
Arno S at 2014/07/01 06:24am
Re: Rights issue

Thanks for the suggestion for the overridden errorHandler. I didn't but I am using the rights module. Therefore my Controller extends RController and that one has an errorHandling on 'accessDenied' which looks like this:

public function accessDenied($message=null)
    {
        if( $message===null )
            $message = Rights::t('core', 'You are not authorized to perform this action.');
 
        $user = Yii::app()->getUser();
        if( $user->isGuest===true )
            $user->loginRequired();
        else
            throw new CHttpException(403, $message);
    }

The BackJob was run as Guest, hence loginRequired generates a nice redirect. I've now overridden the accessDenied in my own Controller like this:

public function accessDenied($message=null)
    {
        if (Yii::app()->background->currentJobId)
            throw new CHttpException(403, 'BackJob is not authorized to perform '.$this->id.'/'.$this->action->id);
 
        return parent::accessDenied($message);
    }

Now I get a correct progress/status/status_text in my DB as well.

Thanks for the support!!

#17551 report it
Siquo at 2014/07/01 05:12am
Debugging BackJob issues

A problem is that due to its nature, debugging asynchronous processes is hard. Logging through the application.log file is your friend.

However, I can't seem to reproduce your error, both on a 404 and a 403 it has status "failed" and the error-response-text in its status-text. It ends successfully and on 100% if it can't find an error in the errorhandler (Yii::app()->errorhandler->error), throwing a 403 CHttpException should automatically add an error though (see method "endRequest"). Have you perhaps overridden the errorhandler?

Edit: updated the "Important" section in the top to remind people that the user should have access to the action.

#17548 report it
Arno S at 2014/07/01 04:44am
Rights issue

I had an issue with BackJob that I'd like to share and hear thoughts on;

From a BackJob I started a controller/action that the current user wasn't authorized for. BackJob was confusing me quite a bit; all progress was set to 100, status to 2 (STATUS_COMPLETED) but the status_text remained empty. It took me some time to find out that I had to authorize the specific controller/action for that user for the BackJob to run correctly.

What is the best way to get a progress/status/status_text that shows that BackJob isn't allowed to run that action?

#17519 report it
Siquo at 2014/06/26 10:53am
Whoops

That was debug code... whoops. I removed that now. I was not sure where exactly to put the cleanDb call, as "finish" doesn't get called every time, and creating a "finally" event is maybe some overkill. It also doesn't have to run every single time, but every X requests maybe, once per day at most.

CHttpSession has a garbage collector event, but using that feels dirty (what if there are no http sessions). Perhaps putting in a probability like CHttpSession would've been better, that's something for next time.

#17518 report it
Arno S at 2014/06/26 09:36am
Different log level

Thanks for accepting the change.

I made another small change to logging code though. I don't think you should log it with CLogger::LEVEL_ERROR. CLogger::LEVEL_INFO is probably better. Perhaps even better if you add a category to allow for CLogFilter filtering. Like so:

Yii::log("Cleaned Backjob DB", CLogger::LEVEL_INFO, __CLASS__);
#17517 report it
Siquo at 2014/06/26 09:12am
Re: Keep the db clean as well

Thanks Arno S! This is clearly something that was missing...

I've added a slightly modified version of your code.

#17509 report it
Arno S at 2014/06/25 04:09am
Keep the db clean as well

I've added the code below to limit the amount of days that are kept in the database table. I call the method on the finish() call.

/**
    * Amount of days to keep the history in DB (0 keeps everything)
    * @var integer
    */
    public $backlogDays=0;
 
    /**
     * Perform database cleanup and limit the amount of backlog
     */
    private function cleanDb() 
    {
        if (0!=$this->backlogDays)
        {
            $this->database->createCommand()->delete($this->tableName, 
                'end_time < DATE_SUB(NOW(), INTERVAL :history DAY) AND status = :status', 
                array(
                    ':history' => $this->backlogDays,
                    ':status' => self::STATUS_COMPLETED,
                ));
        }
    }
#17431 report it
Siquo at 2014/06/11 03:29am
Logout

@Dhaval: If you run Yii::app()->user->logout(), it will destroy the session and session states, so if you rely on those in your background job... I don't know actually, how concurrency would work in that case. Safest way is to assume that won't work.
What will work is either logout using Yii::app()->user-logout(false) which will close the session for the user, but will preserve the session data, or if you don't need the session state, start the job with the second parameter as false, so it will run as the "guest" user: Yii::app()->background->start(array('test/testbackground'), false);

Leave a comment

Please to leave your comment.

Create extension