Yii 1.1: yii-dbeanstalkd

Distributed wrapper for beanstalkd client with basic weighted algorithm and graceful handling of connection failures
3 followers

Distributed wrapper for yii for davidpersson's beanstalkd client

This project differs from other yii extensions in that it provides a very trivial method of managing multiple, distributed beanstalkd servers. Other projects, or classes, differ in that they build upon existing implementations but do not offer any method of failover or recovery from a server failure.

Since a single server going down would cause complete failure this must, at the very least:

  1. be aware of multiple servers/connections
  2. be able to choose between them based upon a simple weighted algorithm
  3. be able to use a different server if one is unavailable In the event of a server failure, this class makes no attempt to ensure enqueued items are not lost since. This will allow the user to enqueue in another server and keep going.

Requirements

  • davidpersson's beanstalkd client to be added to your yii project
  • modifications to config files to support these additions

Add davidpersson's beanstalkd client

Clone davidpersson's library and place it in the extensions folder of your yii application, in a folder named 'beanstalk'.

Modify config files to support these additions

Ensure that the Beanstalk.php class from this project has been placed in your components folder.

Ensure this line has been added under the aliases section of your yii config file:

'aliases'=>array(
        // davidpersson's library has been installed to extensions/beanstalk folder
        'Beanstalk'=> 'application.extensions.beanstalk.src',
),

Ensure these lines have been added to the components section of your yii config file:

'components'=>array(
        'beanstalk'=>array(
            'class'=>'application.components.Beanstalk',
            'servers'=>array(
                'server1'=>array(
                    'host'=>'192.168.0.10',
                    'port'=>11300,
                    'weight'=>50,
                    // array of connections/tubes
                    'connections'=>array(),
                ),
                'server2'=>array(
                    'host'=>'192.168.0.11',
                    'port'=>11300,
                    'weight'=>50,
                    // array of connections/tubes
                    'connections'=>array(),
                ),
            ),
        ),
),

Usage

Producer example

/*
 * connect to a server (determines by weight or selects only existing server::
 */
$client = Yii::app()->beanstalk->getClient();
$client->connect();
 
/* 
 * alternatively, one can connect to a server by name:
 */
//$client = Yii::app()->beanstalk->getClient('server1');
//$client->connect();
 
$client->useTube('default');
 
//echo "Tube used: {$client->listTubeUsed()}.\n";
//print_r($client->stats());
 
$jobDetails = [
    'application'=>'beanstalk test',
    'payload'=>'performing a test on: '.date('Y-m-d H:i:s'),
];
$jobDetailString = json_encode($jobDetails);
 
$ret = $client->put(
    0, // priority
    0, // do not wait, put in immediately
    90, // will run within n seconds
    $jobDetailString // job body
);
 
echo "Added $jobDetailString to queue.\n";
echo "Return was: $ret.\n";
 
$client->disconnect();

Consumer Examples

Peeking

$peek = Yii::app()->beanstalk->getClient();
$peek->connect();
$peek->watch('default');
 
$job = $peek->reserve(5);
print_r($job);
 
$result = touch($job['body']);
print_r($job);
print_r($result);
$peek->disconnect();

Consuming

$consumer = Yii::app()->beanstalk->getClient();
$consumer->connect();
$consumer->watch('default');
 
while(true)
{
    $job = $consumer->reserve();
    $result = touch($job['body']);
 
    if( $result )
    {
        // do something with the job request
        echo "Done...\n";
        $consumer->delete($job['id']);
    }
    else
    {
        // handle failure here
        echo "Burying...\n";
        $consumer->bury($job['id']);
    }
}
 
$consumer->disconnect();

Resources

Inspiration

Great related tools

Other

Total 3 comments

#20117 report it
Clem at 2017/08/19 04:08am
RE RE: running the worker?

Thanks for your reply cottonaf, I finally understood that I need to install beanstalk packages on a server and run it, next run the worker from a command line or cronjob. It does the job for me today, but I have read your advices and will investigate on the others solutions you expose here for other projects. Cheers !

#20116 report it
cottonaf at 2017/08/18 02:01pm
RE: running the worker?

Clem,

It has been a few years since I developed this and I ultimately decided to use a different solution which resolved a few more of my concerns, but I will do my best to explain.

To run the worker you can put the consumer code in a yii command, which you can run regularly (using cron, for example) which will check for jobs in the beanstalkd servers. The problem that I encountered that ultimately caused me to abandon this methodology was that since I had two beanstalkd servers I had to check both of them for jobs. This may not be an issue for you, but the solution I came up with involved having two workers (consumers) which ran on a schedule--each one was specifically to check a certain beanstalkd server.

So, create a command, for instance: QueueWorkerCommand, which accepts an argument for the name of the beanstalkd server and then have two entries in cron, one for each beanstalkd server. That will result in both of them getting checked so you can ensure all jobs are processed.

The reasons I was unsatisfied with my efforts were the following (these might cause you to find another solution):

  1. You need two workers (or 'n' workers really, where 'n' is the number of beanstalkd servers)
  2. If one of your beanstalkd servers goes down you lose those jobs (unless you are using the binlog option so that once it comes back online again you will be able to recover it).

An improvement which might make this whole approach worthwhile:

  • use a load balancer to distribute calls to your beanstalkd servers so your application thinks that only one exists and you can just run your consumer more often

Alternative solutions:

  • RabbitMQ, StormMQ, or rely on Redis (laravel uses redis as a queue)
  • An offering by AWS (SQS), or Rackspace (Cloud Queues), which are already highly available and offer a single endpoint for communicating with the queue--this is ultimately what I did.
  • Don't use multiple beanstalkd servers; just use 1. It greatly simplifies the problem but if it goes down then your app will not be able to enqueue jobs (this might be acceptable depending on the task; you may be able to log an exception or store failure data in a table and then re-create the jobs after recovery and rely on error handling in your app to fail silently).

I know this was a bit long and kind of discourages the use of this extension but it might be the best option for you.

#20115 report it
Clem at 2017/08/18 08:37am
running the worker ?

Thanks for your extension, but I don't understand how can I run the worker ? It should be called one time, wright ? When the server starts ?

Leave a comment

Please to leave your comment.

Create extension