Yii Framework Forum: Asset manager - Yii Framework Forum

Jump to content

  • (2 Pages)
  • +
  • 1
  • 2
  • You cannot start a new topic
  • You cannot reply to this topic

Asset manager

#1 User is online   samdark 

  • Having fun
  • Yii
  • Group: Yii Dev Team
  • Posts: 3,778
  • Joined: 17-January 09
  • Location:Russia

Posted 30 August 2011 - 10:18 AM

Javache posted about improvement Ruby on Rails team made to their assets system. And I think it's a very good feature to have in our asset manager as well.

The basic idea is to use file contents + directory hash instead of just filename + directory hash for assets published.

What for
We will be able to set expire headers for all CSS/JS/images to cache these forever. Since name will change when file content will change, new version will be refreshed immediately and will be cached the same way then.

Moreover, it's much more reliable than relying on file mtime as we're doing now.

Performance
For 20—30 CSS files using hash_file with sha1 isn't significantly slower than just getting mtime of a file.
Yii 1.1 Application Development Cookbook

Enjoying Yii? Star us at github: 1.1 and 2.0.
1

#2 User is offline   jacmoe 

  • Elite Member
  • Yii
  • Group: Moderators
  • Posts: 2,601
  • Joined: 10-October 10
  • Location:Denmark

Posted 30 August 2011 - 10:33 AM

+1 :)
"Less noise - more signal"
0

#3 User is offline   qiang 

  • Yii Project Lead
  • Yii
  • Group: Yii Dev Team
  • Posts: 5,900
  • Joined: 04-October 08
  • Location:DC, USA

Posted 30 August 2011 - 10:49 AM

What if a css file is referring to some images while the names of these image files are modified using this strategy?
0

#4 User is offline   Ben 

  • Standard Member
  • PipPip
  • Yii
  • Group: Members
  • Posts: 274
  • Joined: 15-March 09

Posted 30 August 2011 - 01:41 PM

You had to modify your css file to reflect the changes, so it shouldn't be a problem.

//EDIT:
Now I've got it...
Don't like ads in my sig...
0

#5 User is online   samdark 

  • Having fun
  • Yii
  • Group: Yii Dev Team
  • Posts: 3,778
  • Joined: 17-January 09
  • Location:Russia

Posted 30 August 2011 - 02:15 PM

qiang
Very good question.

Rails implementation can use SCSS and a special helper named image_url() that's used instead of url(). This helper method gives correct paths in the end.

When using regular CSS it's required to use placeholder variable url(<%= asset_path… %>).
Yii 1.1 Application Development Cookbook

Enjoying Yii? Star us at github: 1.1 and 2.0.
0

#6 User is online   samdark 

  • Having fun
  • Yii
  • Group: Yii Dev Team
  • Posts: 3,778
  • Joined: 17-January 09
  • Location:Russia

Posted 30 August 2011 - 02:21 PM

Another ideas from the same Rails pipeline implementation:

1. Allow combining/compressing assets into a single file out of the box in realtime + caching it.
2. Allow pre-processing assets in formats like SCSS into another common format like CSS.

About #1 I think http://www.yiiframew...dedclientscript is pretty close.
Yii 1.1 Application Development Cookbook

Enjoying Yii? Star us at github: 1.1 and 2.0.
3

#7 User is offline   alex-w 

  • Standard Member
  • PipPip
  • Yii
  • Group: Members
  • Posts: 223
  • Joined: 25-November 10

Posted 31 August 2011 - 03:01 AM

Wouldn't this pollute the assets folder?
If every time you change an asset file and it gets published under a new name, what happens to the old one?
0

#8 User is offline   qiang 

  • Yii Project Lead
  • Yii
  • Group: Yii Dev Team
  • Posts: 5,900
  • Joined: 04-October 08
  • Location:DC, USA

Posted 31 August 2011 - 09:56 AM

I think instead of modifying the file names, we can add a new 'version' property to CApplication. CAssetManager can include this version parameter in the directory name generation. As a result, if we change the 'version' property, all asset files will be re-published.

In practice, the version property can be modified by a console script based on the current SVC version number (or some other better mechanism).
Each time we update the production server, this script will be run automatically.

This approach is "coarser" than the file name modification approach. However, it has the benefit that it doesn't need to change our usual workflow (i.e., fixing references in CSS/JS files, which could be very tricky).
0

#9 User is online   samdark 

  • Having fun
  • Yii
  • Group: Yii Dev Team
  • Posts: 3,778
  • Joined: 17-January 09
  • Location:Russia

Posted 31 August 2011 - 01:15 PM

qiang
I don't like fixing references in CSS/JS files either. Your solution will work for medium and large deployments where we can use version control hooks but what about small ones? Of course, you can change version number manually but it's a bit too much for development.
Yii 1.1 Application Development Cookbook

Enjoying Yii? Star us at github: 1.1 and 2.0.
0

#10 User is offline   qiang 

  • Yii Project Lead
  • Yii
  • Group: Yii Dev Team
  • Posts: 5,900
  • Joined: 04-October 08
  • Location:DC, USA

Posted 31 August 2011 - 01:38 PM

What do you mean small ones? And why is this needed during development?
0

#11 User is online   samdark 

  • Having fun
  • Yii
  • Group: Yii Dev Team
  • Posts: 3,778
  • Joined: 17-January 09
  • Location:Russia

Posted 01 September 2011 - 10:15 AM

Small ones are these hosted on shared hosting where you have access to FTP and nothing more. No SSH, no cron.

Typically assets are used for 2 things:

1. Make it easy and reliable to publish non-webroot files.
2. Compress/combine JS/CSS automatically.

During development you generally need only #1 but since compression and combining can hurt it's good to enable it during development or at least during final testing stage.
Yii 1.1 Application Development Cookbook

Enjoying Yii? Star us at github: 1.1 and 2.0.
0

#12 User is online   samdark 

  • Having fun
  • Yii
  • Group: Yii Dev Team
  • Posts: 3,778
  • Joined: 17-January 09
  • Location:Russia

Posted 01 September 2011 - 10:16 AM

Anyway, your version solution looks good and currently I don't have a better one.
Yii 1.1 Application Development Cookbook

Enjoying Yii? Star us at github: 1.1 and 2.0.
0

#13 User is offline   grigori 

  • Junior Member
  • Pip
  • Yii
  • Group: Members
  • Posts: 41
  • Joined: 06-February 11

Posted 01 September 2011 - 06:43 PM

I feel like calculating a hash on each call is no good.
Another idea is implemented in Xenforo avatars - they add ?timestamp=1234 to the end of the url of an image and keep the date the avatar was changed.
0

#14 User is online   samdark 

  • Having fun
  • Yii
  • Group: Yii Dev Team
  • Posts: 3,778
  • Joined: 17-January 09
  • Location:Russia

Posted 01 September 2011 - 07:50 PM

grigori
I've tested hashing the whole file with sha1 and filemtime (I guess that's what is used in xenforo) and can say that both are equally acceptable if you have about 30 assets per page. Calculating simpler hash from a not so long string is even faster.
Yii 1.1 Application Development Cookbook

Enjoying Yii? Star us at github: 1.1 and 2.0.
0

#15 User is offline   mindplay 

  • Advanced Member
  • PipPipPip
  • Yii
  • Group: Members
  • Posts: 397
  • Joined: 03-September 09
  • Location:New York

Posted 20 September 2011 - 08:12 PM

I will briefly describe the approach I use for dynamically generated images (e.g. resized etc.) - I think this approach may potentially be applicable to other types of resources.

I create a folder in the root consistent with the name of a controller - for example, I might create a physical "/images" folder in the public folder, but this will initially be empty.

I then create an ImageController, and a catch-all route for files not physically present in that folder - e.g. "/images/<.*:filename>".

The action method in the ImageController now looks at the filename that was passed, e.g. everything after "/images", parses it in some way, and produces a physical file.

For example, the first time you ask for "/images/320x200/kitten.jpg", the ImageController might parse the width and height (320, 200), load the "kitten.jpg" image from a protected folder, resize the image, create the "320x200" folder and save the resized "kitten.jpg" in that folder.

The idea is that missing resources can be filled in by a controller the first time they are requested - on subsequent requests, the controller (and PHP at all, for that matter) is never even invoked.

This gives theoretically the best possible performance, with the limitation that you have to manually invalidate (erase) cached files - so if the original "kitten.jpg" is modified, you have to find "320x200/kitten.jpg", and potentially other versions, and delete them. They will then get regenerated upon next request, which will be routed to the controller.

During development, you can bypass the function that saves the file, effectively regenerating the file on every request - this is usually more desirable during development.

As mentioned, I believe this pattern is applicable to other types of resources - you can think of a compressed JS or CSS file as a "resized" version of the original resource.
1

#16 User is offline   Psih 

  • Standard Member
  • PipPip
  • Yii
  • Group: Members
  • Posts: 115
  • Joined: 30-June 10

Posted 21 September 2011 - 08:10 AM

I want to add an example of assets manager usage we do in real world (I will describe first and then comment on it).

A month-month and a half ago we started developing the WEB part of a new project witch had to be multi site in essence - same code, different look. Ofcourse theming comes to mind. The tricky part is than every site copy we need is different. Not in "change color scheme, move around some things", but "that's different website with it's own design and layout - only the backend functionality is the same".
Our initial thought was that it will require some clever code and hacking. Guess what? Nothing of the sort! The Asset Manager is a charm - just pass a directory to publish() and it will do the whole directory with all subdirectories. Ha! So we just put JS, images and CSS into a dir in theme and publish it all in one go - all relative paths for CSS and JS stay in place. Every theme just has it's own layout and views - so we are able to tune the HTML for every site individually and they do not have any common HTML, JS or CSS (probably we will figure someting out later with code reusage when we start to make additional sites), but still we have our thing done.

What I'm thinking of is how do you republish the assets if the asset is a whole dir with many files and 1 or 2 files change from the whole set - how do you detect that change? Even more clever question about publishing folders name in assets - it's based only on basename($dir). So even if we make that if we modify some files in folder, make a touch on folder to change it's modification time - nothing will be published under new assets url (like versioning).
Checking for file modifications inside the dir we want to publish is obviously out of the question, but framework cound take into account the filemtime of the dir so it can be updated automatically and under new URL (so we can make assets resources caching permanent or for some long time, witch you usually want). Implementing a touch on a dir on deployment is really a no brainer, and the benifits are hard to ignore. Besides it will make thing even more powerfull :)
0

#17 User is offline   mindplay 

  • Advanced Member
  • PipPipPip
  • Yii
  • Group: Members
  • Posts: 397
  • Joined: 03-September 09
  • Location:New York

Posted 13 January 2012 - 03:15 PM

A couple of thoughts regarding asset management.

First off, we could offer a wealth of features by integrating a third-party component like this one - no point in reinventing the wheel.

The other thing that occurred to me today is, the whole idea of publishing files from a protected folder into a public folder, with generated filename/path based on a hash of the original filename and modification time.

I it's a good time to scrutinize this idea, and wonder if this design is really good or right.

For one, what happens when you clear your assets folder? Currently, any page that has cached an asset URL, is going to get a 404 if the URL of that asset changes.

Perhaps atypical, and you may not care, but in terms of search-indexing, the changing URLs could also be an issue - let's say Google indexes an image you published, perhaps your company logo, and the URL keeps changing, with the old image becoming a 404. (might even affect search engine ranking, not sure.)

As mentioned above, I like the idea of producing assets on-demand, via a controller.

Let's say the URL "http://site.com/assets/28b3cc8e/yii/jquery.js" mapped to an actual controller, which then generated that file. In the case where that file is updated, when the hash no longer matches, that controller could instead return a permanent redirect, thus solving the issue with 404 errors.

In effect, you can put whatever you want in the hash then, it's no longer important - search-engines can update as needed, and browsers showing a cached page will be redirected to get the latest version of each asset.

Of course, this means there has to be some kind of unique identifier for the asset-collection (e.g. module-name) in every URL, since the hash no longer means anything and is only used as a means of forcing cache updates on browsers and other agents.

Thoughts?
0

#18 User is offline   Ben 

  • Standard Member
  • PipPip
  • Yii
  • Group: Members
  • Posts: 274
  • Joined: 15-March 09

Posted 14 January 2012 - 11:15 AM

Two things that come to my mind:

1) Permanent redirect might be the wrong answer. If a resource has been changed, it is necessary to force the client to fetch the new version. Don't know if there are appropriate headers for this purpose.

2) Runtime overhead. Links to published resources should be handled by webserver only, without the need to invoke php, yii or anything else. A fallback to invoke an action for assets that do no longer exist could help in some cases. But this requires URL rewriting and not everyone might have access to the web server config. So I think this should only be an optional feature.
Don't like ads in my sig...
0

#19 User is offline   mindplay 

  • Advanced Member
  • PipPipPip
  • Yii
  • Group: Members
  • Posts: 397
  • Joined: 03-September 09
  • Location:New York

Posted 16 January 2012 - 08:33 PM

View PostBen, on 14 January 2012 - 11:15 AM, said:

Permanent redirect might be the wrong answer.


Might be, I'm not 100% sure. My thinking here, is that once the resource has been updated, it's URL changes to something new, and so the new URL is "permanent" until (or if) the resource changes again. There may be a more appropriate 3xx status code for this.

View PostBen, on 14 January 2012 - 11:15 AM, said:

2) Runtime overhead.


See my explanation above regarding dynamically generated images - I use this technique currently, for precisely that reason: there is no runtime overhead, PHP doesn't even load, except the very first time a client asks for a resource.

I generate the image on-demand the first time, and on subsequent requests, Apache sees that the file now exists, and PHP never gets invoked; if you purge the folder of generated resource, subsequent requests will simply cause those file to regenerate on the next request - in other words, it's perfectly safe to purge the cache folder.

The same pattern would work for any kind of resource - though some resources, such as striped and minified JavaScript resources, would require some configuration, e.g. to specify which files to stripe, and how to name the collected resource... When the resource is purged, the controller goes back to that configuration and figures out how to assemble the requested resource - this can be as labor-intensive as it needs to be, since it'll only happen once...
0

#20 User is offline   dmapes123 

  • Newbie
  • Yii
  • Group: Members
  • Posts: 12
  • Joined: 21-December 10

Posted 30 April 2012 - 11:48 AM

Thanks for considering the best way to implement this guys. If there could be a way to tie this functionality into AWS' S3 and Cloudfront that would be FANTASTIC!
Duncan Mapes
Challenge Media Group LLC
ideas@challengemediagroup.com | http://www.challengemediagroup.com
Twitter: http://twitter.com/duncanmapes | LinkedIn: http://linkedin.com/in/duncanmapes
0

Share this topic:


  • (2 Pages)
  • +
  • 1
  • 2
  • You cannot start a new topic
  • You cannot reply to this topic

1 User(s) are reading this topic
0 members, 1 guests, 0 anonymous users