BackJob can start an action in the background (by using a request), runs it either as the current user or as anonymous, and monitors its progress.
For running jobs at remote locations, fire-and-forget jobs, interval/cron jobs, please look at the runactions extension.
Requirements ¶
Works with Yii-1.1.14 and up. Needs to be able to use fsockopen and fwrite to itself.
Important: If you want to run background jobs as the current user (which is the default option), you must use some kind of non-blocking session storage, such as CDbHttpSession. Also make sure that user is authorised to access that action.
Installation ¶
Put the source from the zip archive in protected/extensions
or from https://github.com/greenhost/BackJob in protected/extensions/backjob
.
Configuration:
// Yes, it needs preloads, but it's not resource-heavy (promise!)
'preload' => array(
'background'
),
'components' => array(
'background' => array(
'class' => 'ext.backjob.EBackJob',
// All other configuration options are optional:
'checkAndCreateTable' => true, // creates table if it doesn't exist
'key' => 'sjs&sk&F89fksL*987sdKf' // Random string used to salt the hash used for background-thread-authentication. Optional to change, but you really should.
'useDb' => true, // Use a database table
'useCache' => true, // Use the cache
'db' => 'db', // Database component name to use
'ch' => 'cache', // Cache component name to use
'tableName' => 'e_background_job', // Name of DB table used
'cachePrefix' => 'EBackJobPrefix-', // Prefix used in the cache
'errorTimeout' => 60, // Nr of seconds after which ALL requests time out, measured from the last update.
'userAgent' => 'Mozilla/5.0 Firefox/3.6.12', // Useragent used for the background request
'backlogDays' => 30, // Number of days successfully completed request-entries are retained in the database
'allBacklogDays' => 60, // Number of days ALL entries (including failed) are retained in the database
),
),
Usage ¶
It's possible to start any controllers' action that is reachable through a url, but recommended to have dedicated actions to be run in the background. You'll have to make your own progress reports, otherwise progress will just jump from 0 to 100.
Starting a background job only requires a valid route to that controller/action.
$jobId = Yii::app()->background->start('site/longJob');
// Or, with parameters:
$jobWithParams = Yii::app()->background->start(
array(
'site/paramJob',
'id'=>$id,
'param2'=>true
)
);
Then you'll probably want to use a time-intervaled ajax request to get the progress. Getting the status of a specific job. This returns an array of the form:
$status = Yii::app()->background->getStatus($jobId);
//This returns an array that looks something like this:
array(
'progress' => 20, //percentage (integer 0-100) of completeness
'status' => EBackJob::STATUS_INPROGRESS, // (integer 0-4)
'start_time' => '2013-11-18 14:11',
'updated_time' => '2013-11-18 14:11',
'end_time' => '2013-11-18 14:11',
'status_text' => 'The complete output of the request, so far',
);
During the background job, the action that actually runs the job itself can update its progress both by echoing and setting the progress counter:
echo "Starting 1<br/>";
Yii::app()->background->update(20);
do_long_function1();
echo "Processing 2<br/>";
Yii::app()->background->update(60);
if(!do_long_function2()){
echo "Error occurred!";
Yii::app()->background->fail(); // this also ends the application immediately!
}
echo "Finishing 3<br/>";
Yii::app()->background->update(90);
do_last_function3();
echo "Done<br/>";
If you don't want a list or log of echoed text, but replace it, you can use the update function like this, but make sure that you also finish manually.
Yii::app()->background->update(array('progress'=>60,'status_text'=>'Chugging along now');
Yii::app()->background->finish(array('status_text'=>'And done');
POST and other methods ¶
Any HTTP1.1 method can be used (POST, PUT, DELETE, CHICKEN), and it can send both "GET" data in the url and url-encoded data in the body (sorry, no multipart/file support, you can bypass that by passing temporary filenames). These can be set in the request array using the keys backjobMethod
and backjobPostdata
respectively. Yes, this means that you can't use your own fields called backjobMethod
and backjobPostdata
, but this was the best way to add this while maintaining backwards compatibility.
Example: To make a POST call with $\_POST
values for id
and name
and a $\_GET
value for page
do:
Yii::app()->background->start(array('test/testbackground', 'page'=>1, 'backjobMethod'=>'POST', 'backjobPostdata'=>array('id'=>5, 'name'=>'Name')));
Complete example ¶
class testController extends Controller {
public function actionProgressMonitor(){
$job = Yii::app()->background->start(array('test/testbackground'));
$this->render('progress'); // empty file, or containing:
echo "Progress: <div id='test'></div>";
echo CHtml::script("$(function(){ setInterval(function(){ $('#test').load('".$this->createUrl('test/getStatus',array('id'=>$job))."');}, 1000);});");
}
public function actionGetStatus($id){
echo json_encode(Yii::app()->background->getStatus($id));
Yii::app()->end();
}
public function actionTestbackground(){
Yii::app()->background->update(1);
echo "Job started.";
sleep(3);
Yii::app()->background->update(20);
sleep(3);
Yii::app()->background->update(40);
echo "Job in progress.";
sleep(3);
Yii::app()->background->update(60);
sleep(3);
Yii::app()->background->update(80);
sleep(3);
echo "Job done.";
Yii::app()->end();
}
}
Changelog ¶
- 0.50 - Changed the way internal requests are recognised, they are now checked by parameters instead of from-headers, should work nicer with proxied servers, and be more resilient against spoofing. Change the
key
field in configuration! - 0.45 - Added support for other HTTP methods and POST-data.
- 0.44 - The timeout-setting now also affects the php-timeout setting with
set_time_limit
, thanks to martijnjonkers. - 0.43 - Added backlog-cleanup for the database, so that it won't fill up with completed requests. Keep in mind that there are two different time-scales, one for successfully finished jobs, and one for all jobs including failed ones. Setting these to 0 days will stop cleanup entirely, this might lead to an ever-expanding database! Thanks to Arno S for noticing this omission.
- 0.42 - Few bugfixes: creation of table, cache was unuseable, a typo
- 0.41 - Small bugfix
- 0.40 - Added monitoring thread that waits for the job to end, so requests that end prematurely still finish. Also, you can specify a number of seconds to wait until processing, and multiple requests with the exact same route will be merged together into one request.
- 0.33 - Https support, better self-recognition, added request field (update your database table!), added global timeout.
- 0.32 - Fails better.
- 0.31 - Initial component
Source?
Any Github url to quickly checkout source code and propose updates and bug fixes?
Github created
Good idea. Github repo to be found here: https://github.com/greenhost/BackJob
email status
Great extension.
I will install it and use it with a current Yii project I am developing.
Is there a way that I can incorporate a function that will email me status information about a current running job (ie: job started, job completed, etc.
Email
@Lloyd: You probably want to extend start(), update() and finish(), or send the emails from the background process itself.
Keep in mind it's a "pull" design like ajax, and not supposed to report on its status by itself, it wants you to request the status.
Logout?
can this process run in background after logout also??
Logout
@Dhaval: If you run
Yii::app()->user->logout()
, it will destroy the session and session states, so if you rely on those in your background job... I don't know actually, how concurrency would work in that case. Safest way is to assume that won't work.What will work is either logout using
Yii::app()->user-logout(false)
which will close the session for the user, but will preserve the session data, or if you don't need the session state, start the job with the second parameter asfalse
, so it will run as the "guest" user:Yii::app()->background->start(array('test/testbackground'), false);
Keep the db clean as well
I've added the code below to limit the amount of days that are kept in the database table. I call the method on the finish() call.
/** * Amount of days to keep the history in DB (0 keeps everything) * @var integer */ public $backlogDays=0; /** * Perform database cleanup and limit the amount of backlog */ private function cleanDb() { if (0!=$this->backlogDays) { $this->database->createCommand()->delete($this->tableName, 'end_time < DATE_SUB(NOW(), INTERVAL :history DAY) AND status = :status', array( ':history' => $this->backlogDays, ':status' => self::STATUS_COMPLETED, )); } }
Re: Keep the db clean as well
Thanks Arno S! This is clearly something that was missing...
I've added a slightly modified version of your code.
Different log level
Thanks for accepting the change.
I made another small change to logging code though. I don't think you should log it with CLogger::LEVEL_ERROR. CLogger::LEVEL_INFO is probably better. Perhaps even better if you add a category to allow for CLogFilter filtering. Like so:
Yii::log("Cleaned Backjob DB", CLogger::LEVEL_INFO, __CLASS__);
Whoops
That was debug code... whoops. I removed that now. I was not sure where exactly to put the cleanDb call, as "finish" doesn't get called every time, and creating a "finally" event is maybe some overkill. It also doesn't have to run every single time, but every X requests maybe, once per day at most.
CHttpSession has a garbage collector event, but using that feels dirty (what if there are no http sessions). Perhaps putting in a probability like CHttpSession would've been better, that's something for next time.
Rights issue
I had an issue with BackJob that I'd like to share and hear thoughts on;
From a BackJob I started a controller/action that the current user wasn't authorized for. BackJob was confusing me quite a bit; all progress was set to 100, status to 2 (STATUS_COMPLETED) but the status_text remained empty.
It took me some time to find out that I had to authorize the specific controller/action for that user for the BackJob to run correctly.
What is the best way to get a progress/status/status_text that shows that BackJob isn't allowed to run that action?
Debugging BackJob issues
A problem is that due to its nature, debugging asynchronous processes is hard. Logging through the application.log file is your friend.
However, I can't seem to reproduce your error, both on a 404 and a 403 it has status "failed" and the error-response-text in its status-text. It ends successfully and on 100% if it can't find an error in the errorhandler (Yii::app()->errorhandler->error), throwing a 403 CHttpException should automatically add an error though (see method "endRequest"). Have you perhaps overridden the errorhandler?
Edit: updated the "Important" section in the top to remind people that the user should have access to the action.
Re: Rights issue
Thanks for the suggestion for the overridden errorHandler. I didn't but I am using the rights module. Therefore my Controller extends RController and that one has an errorHandling on 'accessDenied' which looks like this:
public function accessDenied($message=null) { if( $message===null ) $message = Rights::t('core', 'You are not authorized to perform this action.'); $user = Yii::app()->getUser(); if( $user->isGuest===true ) $user->loginRequired(); else throw new CHttpException(403, $message); }
The BackJob was run as Guest, hence loginRequired generates a nice redirect. I've now overridden the accessDenied in my own Controller like this:
public function accessDenied($message=null) { if (Yii::app()->background->currentJobId) throw new CHttpException(403, 'BackJob is not authorized to perform '.$this->id.'/'.$this->action->id); return parent::accessDenied($message); }
Now I get a correct progress/status/status_text in my DB as well.
Thanks for the support!!
timeout
there is a timeout option, but it does not update the global time limit.
So when setting the timeout to 60 seconds the monitor will still fail at 30 seconds (or whatever is configured).
I suggest adding set_time_limit in the moonitor() method:
/** * The monitor thread. Starts a background request and reports on its progress or failure. */ protected function monitor(){ set_time_limit($this->errorTimeout + 5); // add some seconds to handle the timeout ... }
timeout
Thanks for spotting that! I included the line in the init() function though, so that both the monitoring thread and the actual job-thread have a (possibly) increased duration.
Not running in background
I tried launching a background job with
Yii::app()->background->start
And the controller launching it didn't complete till the background job completed. Are there situations where this would happen? How do I get it to actually run in the background as requested?
Re: Not running in background
Hi Steve,
Problems could include your hosting environment (A very low MaxClients value in Apache, although this is unlikely), but most likely you're not using CHttpDbSession for your session storage (see requirements). If you don't wat to use those, a way to work around that is doing the background request as an anonymous user.
excellent extension
On Windows wamp environment it launches the background job, but the background job seems to stay stuck somehow at the 0% moment. Also, at least on windows, could not get it to work until i set caching to zero in the ext configuration.
The exact same configuration (but with CDbHttpSession from yii core instead of the CHttpDbSession you posted as requirement, although that might be a typo in your Requirements section) works exactly as expected on Linux, haven't tested with cache true tough.
One thing I need to do is pass some variables to the backjob through _POST, not really clear how to do this yet, maybe you could help.
POST support
Hey Radoo, thanks for the compliment and typo find! Yes, it should support POST (in 20/20 hindsight), which took a bit longer than I expected. I've got that part working, but then it also needed to support DELETE and PUT and arbitrary methods... I'll probably finish that tomorrow and I'll upload the new version then!
Why it's not working on the WAMP stack... I don't know. Caching is a bit hit-and-miss at the moment (badum-tss).
Edit: new version is live, should support both GET and POST and arbitrary methods now. The request body is always url-encoded.
How to kill the process
Nice extension! Is very usefull for working with a lot of data!
I have one question: How can i kill the backjob, while it is still running, whithout restarting the apache? I was thinking for an extra column in the db to use with a new status, but i didn't figure it out yet.
One problem that i found, when running more than one Backjob: When 1 job is finished, and progress is 100, it will stay in the db as completed, but it will delete from db the other running jobs. They are still running, and doing their job, but i won't get any status from them. It's also affecting the progressMonitor, where i will have a finished job, and the other jobs will be frozen.
Thanks :)
Not working on Server
This is very nice extension. I tried to send lots mail using this extension and its is working fine in local and save process time. When I deployed application on server it is not working.
Please help me.
Thanks in Advance.
fsockopen ssl in php 5.6
in php 5.6 fsockopen() function verify certificate enabled by default and if you have self signed cert you get an error 'certificate verify failed'. I fixed it by change 'fsockopen(...)' in line 481 to
stream_socket_client(($port == 443 ? 'ssl://' : '') . $host.':'.$port, $errno, $errstr, $this->errorTimeout,STREAM_CLIENT_CONNECT,$context)
and add into this function context param:
$context = stream_context_create(array( 'http' => array( 'header' => 'Cookie: PHPSESSID='.Yii::app()->session->sessionID ), "ssl"=>array( "verify_peer"=>false, "verify_peer_name"=>false, ) ));
which disable ssl check. Hope it help someone.
send email when finishes job
My question is where I should add the function for send email
Greetings in advance
Yii2 implementation
Hello,
Do you plan to implement a Yii2 version ?
Cookie validation problem
The background session as the current user does not work when the application uses the cookie validation scheme (
enableCookieValidation = true
). In fact, BackJob reuses the same session cookie for the new background session; this is considered by Yii as a session cookie tampering.See an alternative approach which works in this case.
If you have any questions, please ask in the forum instead.
Signup or Login in order to comment.