Amazon AWS ADK

[size="3"][color="#006400"]/*[/color]

[color="#006400"]Split out from original topic:[/color]

[color="#006400"]how-to-add-the-tarzanaws-tools-as-a-third-party/[/color]

[color="#006400"]*/[/color]

[/size]

I’m successfully using the Amazon AWS ADK within Yii. I’d really like to integrate it within the ActiveRecord class, so I can use extensions and store data in SimpleDB, but that’s a bigger task than I’m ready for as a Yii newbie. :slight_smile:

I toyed with different ways of doing it. In the end, I added this to index.php:


 $aws_path = dirname(__FILE__).'/../../yii-protected/extensions/awssdk/';

Yii::$classMap=require(dirname(__FILE__).'/../../yii-protected/config/classMap.php');

Then created a classMap.php file with each class name and the location of the file like this:


<?php


return(array(

	'sfYamlDumper' => $aws_path . 'lib/yaml/lib/sfYamlDumper.php',

	'sfYamlInline' => $aws_path . 'lib/yaml/lib/sfYamlInline.php',

	...


	'JSON_Exception' => $aws_path . 'utilities/json.class.php',

));

I created that file with an inelegant grep / gedit regex replace query. They are:


$ grep -rn -e "^class" * > classes

Then gedit's Regular Expression Search for: ^([^:]*):\d+:class (\S*).*$ and replace with: '\2' => $aws_path . '\1',

Then I modified sdk.class.php to comment out the auto loader. Instead, classes are auto loaded by Yii.

If there’s interest, I could try packaging this up as an extension. I’m totally new to Yii, so no idea how to do that, but willing to try. :slight_smile:

Love & joy - Callum.

Callum, i just emailed you about this–your technique worked like a charm!!! I had a hard time at first understanding what you meant. You wrote “gedit” and i thought he couldn’t possibly be talking about Gedit on Ubuntu. …either way, for anyone else interested, you have to download and install the regular expression search and replace Gedit plugin here:

http://halfhourhacks.blogspot.com/2008/03/gedit-regular-expression-plugin.html

download the files. place them where they say to place them after extracting them. however, the files you place there are in the folder within the extracted folder. so just extracting them isn’t enough. go one level deep and copy/paste that into the gedit plugins folder.

Otherwise, if you’re like me when i was first looking at Callum’s explanation, i didn’t understand the classMap. Basically, install the entire AWS SDK folder tree in your extensions folder. Then go there at the commandline, and type:


grep -rn -e "^class" * > classMap.php

and then open that file in Gedit on Ubuntu (or any other distro that has it), and first install the regular expressions Gedit plugin. And then execute the plugin by filling the search field with this:


^([^:]*):\d+:class (\S*).*$

and the replace field with this:


'\2' => $aws_path . '\1',

Basically what is happening is his initial grep command is build a list of all the files in the aws sdk folder tree that end in ‘class’ , i.e. it’s producing a list of all the class file names and class names. Then when you run the regular expression in gedit you produce a proper array map of files to include. That’s all. Contact me if you need any help.

Thanks again, Callum!

ps. here’s the classMap.php file:


<?php


return(array(


'S3_Exception' => $aws_path . 'sdk/services/s3.class.php',

'AmazonS3' => $aws_path . 'sdk/services/s3.class.php',

'AmazonSQS' => $aws_path . 'sdk/services/sqs.class.php',

'SQS_Exception' => $aws_path . 'sdk/services/sqs.class.php',

'AmazonRDS' => $aws_path . 'sdk/services/rds.class.php',

'RDS_Exception' => $aws_path . 'sdk/services/rds.class.php',

'CloudFront_Exception' => $aws_path . 'sdk/services/cloudfront.class.php',

'AmazonCloudFront' => $aws_path . 'sdk/services/cloudfront.class.php',

'AmazonEC2' => $aws_path . 'sdk/services/ec2.class.php',

'EC2_Exception' => $aws_path . 'sdk/services/ec2.class.php',

'AmazonIAM' => $aws_path . 'sdk/services/iam.class.php',

'IAM_Exception' => $aws_path . 'sdk/services/iam.class.php',

'AmazonELB' => $aws_path . 'sdk/services/elb.class.php',

'ELB_Exception' => $aws_path . 'sdk/services/elb.class.php',

'AmazonElasticBeanstalk' => $aws_path . 'sdk/services/elasticbeanstalk.class.php',

'ElasticBeanstalk_Exception' => $aws_path . 'sdk/services/elasticbeanstalk.class.php',

'AmazonCloudFormation' => $aws_path . 'sdk/services/cloudformation.class.php',

'CloudFormation_Exception' => $aws_path . 'sdk/services/cloudformation.class.php',

'AmazonCloudWatch' => $aws_path . 'sdk/services/cloudwatch.class.php',

'CloudWatch_Exception' => $aws_path . 'sdk/services/cloudwatch.class.php',

'AmazonSDB' => $aws_path . 'sdk/services/sdb.class.php',

'SDB_Exception' => $aws_path . 'sdk/services/sdb.class.php',

'AmazonEMR' => $aws_path . 'sdk/services/emr.class.php',

'EMR_Exception' => $aws_path . 'sdk/services/emr.class.php',

'AmazonSNS' => $aws_path . 'sdk/services/sns.class.php',

'SNS_Exception' => $aws_path . 'sdk/services/sns.class.php',

'AmazonImportExport' => $aws_path . 'sdk/services/importexport.class.php',

'ImportExport_Exception' => $aws_path . 'sdk/services/importexport.class.php',

'AmazonAS' => $aws_path . 'sdk/services/as.class.php',

'AS_Exception' => $aws_path . 'sdk/services/as.class.php',

'AmazonSES' => $aws_path . 'sdk/services/ses.class.php',

'Email_Exception' => $aws_path . 'sdk/services/ses.class.php',

'sfYaml' => $aws_path . 'sdk/lib/yaml/lib/sfYaml.php',

'sfYamlDumper' => $aws_path . 'sdk/lib/yaml/lib/sfYamlDumper.php',

'sfYamlInline' => $aws_path . 'sdk/lib/yaml/lib/sfYamlInline.php',

'sfYamlParser' => $aws_path . 'sdk/lib/yaml/lib/sfYamlParser.php',

'CachePDO' => $aws_path . 'sdk/lib/cachecore/cachepdo.class.php',

'CacheAPC' => $aws_path . 'sdk/lib/cachecore/cacheapc.class.php',

'CacheFile' => $aws_path . 'sdk/lib/cachecore/cachefile.class.php',

'CacheXCache' => $aws_path . 'sdk/lib/cachecore/cachexcache.class.php',

'CacheMC' => $aws_path . 'sdk/lib/cachecore/cachemc.class.php',

'CacheCore' => $aws_path . 'sdk/lib/cachecore/cachecore.class.php',

'RequestCore' => $aws_path . 'sdk/lib/requestcore/requestcore.class.php',

'ResponseCore' => $aws_path . 'sdk/lib/requestcore/requestcore.class.php',

'RequestCore_Exception' => $aws_path . 'sdk/lib/requestcore/requestcore.class.php',

'Console_ProgressBar' => $aws_path . 'sdk/_samples/lib/ProgressBar.php',

'CFRuntime_Exception' => $aws_path . 'sdk/sdk.class.php',

'CFRuntime' => $aws_path . 'sdk/sdk.class.php',

'CFLoader' => $aws_path . 'sdk/sdk.class.php',

'CFInfo' => $aws_path . 'sdk/utilities/info.class.php',

'CFJSON' => $aws_path . 'sdk/utilities/json.class.php',

'JSON_Exception' => $aws_path . 'sdk/utilities/json.class.php',

'CFStepConfig' => $aws_path . 'sdk/utilities/stepconfig.class.php',

'CFPolicy' => $aws_path . 'sdk/utilities/policy.class.php',

'CFArray' => $aws_path . 'sdk/utilities/array.class.php',

'CFMimeTypes' => $aws_path . 'sdk/utilities/mimetypes.class.php',

'CFStackTemplate' => $aws_path . 'sdk/utilities/stacktemplate.class.php',

'CFComplexType' => $aws_path . 'sdk/utilities/complextype.class.php',

'CFResponse' => $aws_path . 'sdk/utilities/response.class.php',

'CFBatchRequest_Exception' => $aws_path . 'sdk/utilities/batchrequest.class.php',

'CFBatchRequest' => $aws_path . 'sdk/utilities/batchrequest.class.php',

'CFSimpleXML' => $aws_path . 'sdk/utilities/simplexml.class.php',

'CFRequest' => $aws_path . 'sdk/utilities/request.class.php',

'CFManifest' => $aws_path . 'sdk/utilities/manifest.class.php',

'CFUtilities' => $aws_path . 'sdk/utilities/utilities.class.php',

'CFGzipDecode' => $aws_path . 'sdk/utilities/gzipdecode.class.php',

'CFHadoopStep' => $aws_path . 'sdk/utilities/hadoopstep.class.php',


));


and here’s my index.php entry script:


<?php

// change the following paths if necessary

$yii=dirname(__FILE__).'/../../../yii/framework/yii.php';

$config=dirname(__FILE__).'/protected/config/main.php';




$debug = $_SERVER['HTTP_HOST'] != "www.crimetv.com" ? true : false; 

// remove the following lines when in production mode

defined('YII_DEBUG') or define('YII_DEBUG',$debug);

// specify how many levels of call stack should be shown in each log message

defined('YII_TRACE_LEVEL') or define('YII_TRACE_LEVEL',3);


if($debug) {

    defined('YII_TRACE_LEVEL') or define('YII_TRACE_LEVEL',3);

    error_reporting(E_ALL);

    ini_set('display_errors', 'on');

}


require_once($yii);


$aws_path = dirname(__FILE__).'/protected/extensions/aws-sdk/'; //ADD THIS!!!

Yii::$classMap=require(dirname(__FILE__).'/protected/config/classMap.php'); //ADD THIS!!!




Yii::createWebApplication($config)->run();

and since there are several old AWS php libraries, just to be clear, here’s the library everyone should be using:

http://docs.amazonwebservices.com/AWSSDKforPHP/latest/index.html

There’s a Yii extension for a really old S3 library:

http://www.yiiframework.com/extension/es3/

Don’t use that. Use the latest stuff that Amazon is pushing that they have documented in the above link. The stuff in the above link is what Callum’s technique is made for.

Sorry you had so much trouble with the gedit stuff, it wasn’t a very clear explanation. I can’t actually remember how I installed that plugin. I think it would be possible to construct a single command that would output all the necessary data into a file using sed. But you’ve posted the entire file now, so that’s probably easier for most people.

I didn’t make any further progress. I realised I was actually making my life incredibly difficult by using simpledb, so I simply stopped and used a plain MySQL table instead. The model and CRUD generation saved me so much time it was outrageous to have tried simpledb in the first place! :slight_smile:

I’d love to see simpledb implemented as a real database layer in Yii. Some types of query would be outrageously expensive, but most would be fairly easily mapped to simpledb queries I think. I’m very new to Yii and OO programming in general, so it’s a task that’s way beyond me right now.

have you tried AWS’s RDS? it can scale vertically to be extremely powerful. By “vertically” i mean it doesn’t scale it horizontally to have tons of severs working together in a load balanced cluster, but rather each of those boxes can go up and down in size, and they can get extremely large.

That said, the following companies are in the middle of cracking the load-balanced and sharded mysql nut:

http://xeround.com/

http://nimbusdb.com/

http://www.scalebase.com/

…so basically forget SimpleDB. that will never work with ActiveRecord as you seem to have found out. But these solutions are on the verge of facilitating out-of-the-box extreme sql scalability through a single SQL connection you can provide Yii.

I have looked at RDS a little. It’s an interesting offering. The challenge seems to be pricing. It’s priced per instance rather than by usage. That gives SimpleDB a significant advantage I think. I also think it would be possible to map most AR functions onto SimpleDB, at least the basic stuff. Things like joins wouldn’t work, or would be expensive, but basic select / update / delete statements could be handled. It is a totally different design paradigm working with SimpleDB though, so maybe using Active Record is more complex than makes sense. I’m definitely no expert.

yea, i’ve thought about it before too. or rather, mapping yii active record to other distributed data stores like Cassandra, CouchDB, MongoDB, HBase, etc. You could definitely do the basic stuff, and probably pretty easily. But in general, SQL offers so much functionality and even Yii offers so much functionality for building queries through CDbCriteria and ActiveRecord in general, etc, that it would take more work than it’s worth to do it. I could be wrong though. If you have your clear use-cases, you could build mappings for all the active record functionality you use.

Either way, you were talking about the cost of RDS–if you don’t plan to have much traffic or data, then just use MYSQL stored on your EC2 instance. If you have so much data and traffic that you’re thinking of a distributed data store like SimpleDB, then the cost of an RDS instance isn’t an issue anymore since your costs will be high anyway. In which case, your development costs are extremely low if you vertically scale an RDS box to a point where it uses tons of RAM, etc. And once guys like scalebase and xeround 100% nail the automatic sql load-balancing and sharding thing, then your development costs there are also near zero. For a long time now scaling web servers has been completely commoditized. However scaling data has still been extremely expensive in development time and requires lots of experience. Xeround, Scalebase, etc, are all changing that and making it so it’s just as easy to scale data as it is ec2 instances. I’ve tested both those guys’ software. It’s pretty sweet. Scalebase provides you with an SQL connection that acts as a load-balancer to your own mysql servers. So you basically tell scalebase about your mysql servers (which can be on ec2 instances or RDS), and then their load-balancer automatically shards your data and distributes load between them. Xeround provides a fully hosted solution, i.e. they got all your data on their servers. But they’re similar in that they both provide your application with a standard mysql connection–and that’s major. That means your standard simple yii application can benefit from supreme data scalability without having to be architected at the application level to be concerned with how this data is distributed.

So personally, i’d forget SimpleDB altogether. one RDS instance in itself can become massive in size. And their’s support for replication across RDS instances. So you can easily setup a load-balancer across identical RDS mysql servers. That won’t solve “big data” needs, but it will solve plenty of traffic. But give it a year, and nimbusdb, scalebase, xeround, etc, will have solved “big data” through SQL, which everyone currently only thinks is possible with the popular distributed data stores du jour, such as Cassandra.

This topic has been dead for a little while, but the information is excellent! Thanks for this Callum / FaceySpacey.

Anyway, I’m a newb when it comes to Yii, but I’m getting it… I’d say I’ve mastered 40-50% of the concepts at this point.

As I look at the solution to implementing Amazon’s PHP SDK, I’m concerned by the method of including the SDK, directly into the index.php entry script!





$aws_path = dirname(__FILE__).'/protected/extensions/aws-sdk/'; //ADD THIS!!!

Yii::$classMap=require(dirname(__FILE__).'/protected/config/classMap.php'); //ADD THIS!!!

For my situation, I’ve got a website frontend (high traffic) and a user backend (low traffic in comparison). On the backend, I want people to be able to upload files to Amazon S3. So the points at which they would upload something are very specific. So in general, I wouldn’t need the front end user requests loading up the entire Amazon SDK for each request.

Does this process/strategy make sense? If so, I’m going to try to figure out a way to include the Amazon SDK in an on-demand type way, similar to how you can “registerCssFile” or “registerPackage” for CSS/JS files. I like being able to pick and choose what gets included when, both to save on the data transfer, but also file operations.

One thing anyone on Amazon AWS should consider is file operations. More I/O operations WILL cost you 'mo money…, so if every request on my application increased by 67 files, that’d be a pretty significant increase in I/O cost.

I’ll post back what I come up with a little later today.

Again, thanks for this leg-up with getting Amazon AWS integrated with Yii.

Cheers,

Matt

Hi Matt,

To my understanding, but I’m also quite new to the whole Yii thing (about 2 months working nearly full time on it), are you not loading the entire SDK with each request. Instead, you are telling the autoloader where to find the classes for AWS. I guess that you are still loading the classes whenever you give a shout to use them.

Ciao, Emile

you can find the last version of the SDK of AWS and Yii Framework here : YiiAWS