Discussion: General way to set portal image sizes / caching

[Jakob]

Hey guys,

I want to start a discussion about how the image sizes for portal output should be set in the future.
At the moment, the size of images is sometimes set in portal classes, e.g. for module news:

Code: Alles auswählen

$arrNews["news_image"] = "<img src=\""._webpath_."/image.php?image=".urlencode($objNews->getStrImage())."&amp;maxWidth=400&amp;maxHeight=400\" >";

or set in element settings, e.g. for module gallery/element image

or is set nowhere, e.g. element paragraph :

Code: Alles auswählen

$strReturn .= "<img src=\"".$this->arrElementData["absatz_bild"]."\" alt=\"".$this->arrElementData["absatz_bild"]."\" /><br /><br />\n";

As our aim is to be as flexible as possible without touching the classes we should think about an other and in my opinion general way to set these sizes.


My idea would be to set the image sizes only in the templates. So we could add more template sections e.g. for module news:

Code: Alles auswählen

<news_list_image>
    <img src="_webpath_/image.php?image=%%news_image%%&amp;maxWidth=200&amp;maxHeight=200" alt="%%news_title%%" />
</news_list_image>

<news_detail_image>
    <img src="_webpath_/image.php?image=%%news_image%%&amp;maxWidth=400&amp;maxHeight=400" alt="%%news_title%%" />
</news_detail_image>

And I think this would be even more flexible (of course it's some more code, but therefor very flexible):

Code: Alles auswählen

<news_list>
<div class="newsList">
    <div class="newsListHeader">
<a href="%%news_more_link_href%%"><img src="_webpath_/image.php?image=%%news_image%%&amp;maxWidth=200&amp;maxHeight=200" alt="%%news_title%%" /></a>
        <div class="newsListTitle">
            <h2><a href="%%news_more_link_href%%">%%news_title%%</a></h2>
        </div>
        <div class="newsListMore">%%news_start_date%%</div>
        <div class="clearer"></div>
    </div>
    <div class="newsListTeaser">
        <div>%%news_intro%% %%news_more_link%%</div>
    </div>
</div>
</news_list>

<news_list_with_image>
<div class="newsList">
    <div class="newsListHeader">
        <div class="newsListTitle">
            <h2><a href="%%news_more_link_href%%">%%news_title%%</a></h2>
        </div>
        <div class="newsListMore">%%news_start_date%%</div>
        <div class="clearer"></div>
    </div>
    <div class="newsListTeaser">
        <div>%%news_intro%% %%news_more_link%%</div>
    </div>
</div>
</news_list_with_image>


<news_detail>
<div class="newsDetail">
    <h2>%%news_title%%</h2> %%news_start_date%%
    <p class="newsTeaser">%%news_intro%%</p>
    <p>%%news_text%%</p>
    <p>%%news_back_link%%</p>
</div>
</news_detail>

<news_detail_with_image>
<div class="newsDetail">
    <h2>%%news_title%%</h2> %%news_start_date%%
    <p class="newsTeaser">%%news_intro%%</p>
    <img src="_webpath_/image.php?image=%%news_image%%&amp;maxWidth=400&amp;maxHeight=400" alt="%%news_title%%" />
    <p>%%news_text%%</p>
    <p>%%news_back_link%%</p>
</div>
</news_detail_with_image>

We could do the same in module gallery:

Code: Alles auswählen

<piclist_pic>
    <div style="text-align: center;">
        <div><a href="%%pic_href%%"><img src="_webpath_/image.php?image=%%pic%%&amp;maxWidth=150&amp;maxHeight=150" /></a></div>
        <div>%%name%%</div>
    </div>
</piclist_pic>

And also in the paragraph/image element we could integrate templates as well, which would be great anyway in my opinion. The template could be set in the element general-settings which will may come with 3.3.0. E.g. it could look like this for element paragraph:

Code: Alles auswählen

<paragraph>
    <h2>%%paragraph_title%%</h2>
    <p>%%paragraph_content%%<p>
</paragraph>

<paragraph_with_image>
    <h2>%%paragraph_title%%</h2>
    <img src="_webpath_/image.php?image=%%paragraph_image%%&amp;maxWidth=150&amp;maxHeight=150" alt="%%paragraph_title%%" />
    <p>%%paragraph_content%%<p>
</paragraph_with_image>

<paragraph_with_link>
    <h2>%%paragraph_title%%</h2>
    <p>%%paragraph_content%%<p>
    <a href="%%paragraph_link%%">%%lang_readmore%%</a>
</paragraph_with_link>

<paragraph_with_image_and_link>
    <h2>%%paragraph_title%%</h2>
    <a href="%%paragraph_link%%"><img src="_webpath_/image.php?image=%%paragraph_image%%&amp;maxWidth=150&amp;maxHeight=150" alt="%%paragraph_title%%" /></a>
    <p>%%paragraph_content%%<p>
    <a href="%%paragraph_link%%">%%lang_readmore%%</a>
</paragraph_with_image_and_link>


The template-way would also be very flexible in calling the image.php, so instead of using the maxWidth/maxHeight parameters you could also use the new introduced (3.2.1) fixedWidth/fixedHeight parameters if you want real equal image sizes. Or if needed you could add the quality param to set the compression level...

But lets discuss this, maybe you guys have some other ideas? Where would you prefer setting the image sizes?
Thanks for your feedback!

[Jakob]

To blow the whole thing up I got another idea I'm thinking on for quite a while. It's actually a little off-topic but I want to share my thoughts with you guys.

What about calling our dynamically resized images like this?:

Code: Alles auswählen

www.kajona.de/portal/pics/upload/your-picture.150x150.max.jpg
www.kajona.de/portal/pics/upload/your-picture.150x150.fixed.jpg

or in case you only want to set one size
www.kajona.de/portal/pics/upload/your-picture.0x150.max.jpg

instead of

Code: Alles auswählen

www.kajona.de/image.php?image=/portal/pics/upload/your-picture.jpg&maxWidth=150&maxHeight=150
www.kajona.de/image.php?image=/portal/pics/upload/your-picture.jpg&fixedWidth=150&fixedWidth=150

Via mod_rewrite we could translate these URLs to the usual image.php-calls. So that's just looking nice and not more or less, but the more interesting thing comes now:

With mod_rewrite it's possible to check if a file exists. So if we would change a few things on how we save cached images we could let mod_rewrite check, if the requested image is already cached or if it has to be generated from scratch. This would lead to an massive performance boost when serving resized images since we only have to call PHP once and all following requests on this resized image will be served only by the webserver.

But there's one big problem till now: I didn't found a way how we can check if the original image file has been modified so its cached version has to be invalidated. It seems this is only possible with the RewriteMap feature which will call a custom shell script - which is way to complicated for our easy-to-use CMS. Furthermore the whole idea works (of course) only when mod_rewrite is available.
Anyway, I just wanted to let you know about what I'm thinking, maybe you have some cool ideas.

[Stefan]

Just a few thought about your second post:
I like the idea, but I wonder how we'll get the checksum for the cached images.
Actually, we're using an algorithm in order to create a checksum-based filename out of a few environment- and image-variables.
When working only within the apache rewriting parts, we don't have access to those vars - so how can we generate the checksum (aka the filename) to look for?

[Jakob]

Just a few thought about your second post:
I like the idea, but I wonder how we'll get the checksum for the cached images.
Actually, we're using an algorithm in order to create a checksum-based filename out of a few environment- and image-variables.
When working only within the apache rewriting parts, we don't have access to those vars - so how can we generate the checksum (aka the filename) to look for?

Jep, this is exactly the problem - furthermore the invalidation of cached images is the problem. As you're right, we can't create a checksum with mod_rewrite, I would save the cached images in a way like this:

Code: Alles auswählen

www.kajona.de/portal/pics/cache/upload/your-picture.150x150.max.jpg
www.kajona.de/portal/pics/cache/upload/gallery1/your-picture.150x150.max.jpg

or without subfolders like

www.kajona.de/portal/pics/cache/upload_gallery1_your-picture.150x150.max.jpg

So the creation of a cached image is not a problem at all, we actually don't need a checksum here. But there's no way to check later on if the original image has changed after the cached version was created. We could delete the cached version of a picture when the original one is edited through the filemanager, but in case you upload a new version via FTP you have to empty the cache manually.

[Stefan]

As you said - there's no way to find out if an image changed.
I don't see anyway right now how to solve this - but we'll keep an eye on this.

[Jakob]

Regarding to my second post:
As I got really bad image delivery performance on one project I just compared the image delivery speed for an 57 KB image, first through our image.php and second via direct access to the cached image file. Then I modified a few files so image.php doesn't make any DB calls anymore (static image cache path...) and compared the speed as well.

The results are not that reliable as I did them on a live mass-hosting server (which has bad performance in general), just disabled caching in Firefox and made 3x 15 requests, tracking the delivery speed with Firebug. Anyway, this are the results in ms:

Code: Alles auswählen

through image.php:
    556
    1006
    1450
    975
    771
    842
    934
    1110
    983
    846
    1010
    885
    1006
    985
    988

direct http request to cached image file:
    101
    98
    99
    97
    102
    128
    428
    351
    129
    96
    102
    125
    114
    124
    97
    99

modified image.php so it doesn't make any DB calls:
    402
    752
    720
    529
    475
    739
    741
    392
    641
    760
    451
    725
    668
    690
    395
    667

I think we should work on an optimization of delivering dynamically processed images, since our current image.php is a massive bottleneck which affects the server-side and above all noticeable the user-experience, especially when requesting a huge number of images (e.g. in gallery-lists).
I created a ticket for this - we'll see what we can do. I'm thinking about lightweighting the image.php (in most cases their is no need to have a dynamic cache path, no need to check the session, to include a lot of system-classes, ...) and preparing the image-caching to allow static image delivery through rewrite rules as described in my post. So this can optionally be enabled in cases where the problem with the invalidation of cached images is not that relevant.

[Stefan]

Well, before we'll digg into the code, we should make reliable tests.
What you did is creating a few numbers, but you can't rely on them. There are too many external factors influencing the results.
so, what you should do is creating a test-szenario on your local machine.
This reduces the following factors:
- CPU usage on the hosters' machine
- DB usage on the hosters' machine
- Network latency on the hosters' subnet
- Network latency on the cariieres' network

I for myself didn't run in any trouble regarding the speed of image-generation yet. I mean, what do you expect if you buy a 0,50€/month webhosting package? It's undoubtedly clear that the performance will be rather poor.

But, I'm thinking of another improvement, eliminating quite a few speed-issues.
In your case, so the image.php, the DB-connection is opened to read the system-settings, only.
What about the following:
The system settings are loaded only at the first request and being stored to session. Every further call will load them from the session.
The DB-class itself will get some kind of lazy-connection creation. So, the connection to the db will not be opend during instantiation, but just at the first real query.
I don't know if this'll realy work, but it would save time dramatically. The inclusion of files isn't expensive whereas the opening of the db-connection is.
And - before we throw away code & co: make a few profiler dumps to spot out the methods and files the time gets' consumed. Otherwise it's like guessing the parts and pointing to code with eyes closed.

[Jakob]

Well, before we'll digg into the code, we should make reliable tests.

Yep!

What you did is creating a few numbers, but you can't rely on them. There are too many external factors influencing the results.

This is what I wrote and I know how to do reliable tests, but thanks for your explanation

I for myself didn't run in any trouble regarding the speed of image-generation yet. I mean, what do you expect if you buy a 0,50€/month webhosting package? It's undoubtedly clear that the performance will be rather poor.

Sure, but this is where many Kajona installations run on. This is why I wanted to see the speed on a live-server, not to get reliable data, more to get an rough impression how fast images are served in a typical worst-case live environment. If you run a small site the image delivery won't be a problem at all, but with many images on a site and a growing number of requests the current image.php will be one of the first bottlenecks. In general, the usage of PHP and the DB for serving already processed and cached (so static) images (in many, but of course not all cases) gives me a reason to think it through.

In your case, so the image.php, the DB-connection is opened to read the system-settings, only.

That's not true, enable db-logging and have a look. Requesting one already cached image through the image.php produces 2 Selects and 1 Update:

Code: Alles auswählen

10:04:42, 08-10-09      D:\_webserver\_module\kajona\system\class_carrier.php    Row 55, function loadConfigsDatabase
SELECT * FROM kajona_system_config ORDER BY system_config_module ASC

10:04:42, 08-10-09      D:\_webserver\_module\kajona\system\class_modul_system_session.php    Row 69, function getRow
SELECT * FROM kajona_session WHERE session_id = '10b86a84acd9cc663c83'

10:04:42, 08-10-09      D:\_webserver\_module\kajona\system\class_session.php    Row 560, function updateObjectToDb
UPDATE kajona_session SET
                      session_phpid =  '5e1d1a8470e35a4d5c2f0585f47d643c',
                      session_userid = '',
                      session_groupids =  'ffb47d74aca7743e872b',
                      session_releasetime =  1254992682,
                      session_loginstatus = 'loggedout',
                      session_loginprovider = '',
                      session_lasturl =  'image=%2Fportal%2Fpics%2Fupload%2Fsamples%2FP3197800.JPG&maxWidth=150&maxHeight=110'
                    WHERE session_id = '10b86a84acd9cc663c83'


What about the following:
The system settings are loaded only at the first request and being stored to session. Every further call will load them from the session.
The DB-class itself will get some kind of lazy-connection creation. So, the connection to the db will not be opend during instantiation, but just at the first real query.
I don't know if this'll realy work, but it would save time dramatically. The inclusion of files isn't expensive whereas the opening of the db-connection is.

Sounds interesting, could be an idea.

[KojiKui]

Hi just the last few days i played with performance optimisation. I believe an improvement could be use expire headers.
All dynamic images send 1981/11/19 as expire date. So they are loaded on every page request. Most images are loaded more than once per user, if we set the expire header maybe 1 week in the future we could reduce requests.
We need only change a few rows in image.php.
What do you think about?

[Jakob]

Yes, this is a great additional idea, I also thought about it. The only thing is that it could happen, that the users see changes on the pictures delayed - since the browser doesn't check on every request if the picture was modified anymore. But I think for most cases this is okay. Maybe we can think about adding a config-value to enable/disable things like that.

[KojiKui]

Yes a config would be fine.
We should also think about using mod_expire in the htaccess, for files like css, js, etc.

[Jakob]

For larger projects I always added a get-variable, e.g. /portal/scripts/kajona.js?100 to have a little control over the delivery of js/css/swf files when using expire-headers. So in case of Kajona we could e.g. add a variable in the global_includes.php, so the value can be accessed in the templates. Everytime you update your js/css/swf files you can manually change the value and force the browser to reload the file. I think this is the most reliable way of "remotly clearing" the browser cache.
What do you think? Would this make sense for Kajona? For smaller projects it's too complicated but for larger ones it'll get useful in my eyes. We just have to watch out that we don't add 50 new settings to set up caching stuff

[KojiKui]

I´m not sure if we really need that complexity. I don´t like get variables just for cache clearing. We should turn of expire header in default .htaccess with the possibility to turn on. Set expire time for js/ css/ swf to 1 week.
In small (normal) projects everybody could wait for a week if these files changed. If you´ve got a release date for changes in larger projects you could edit the expire date more than one week before to the release date. So everybody who visits the site before the update will get the correct expire date.

[Jakob]

Yes, I agree, for larger projects the release date should be known so you could manually adjust the expire date. Do you have experience if all browsers stick to this date or are there some specials?

[Jakob]

What do you think about following lines in the .htaccess for static files?

Code: Alles auswählen

<FilesMatch "\.(js|css|jpg|jpeg|png|gif|ico|swf)$">
    Header set Cache-Control "max-age=604800, public"
    <IfModule expires_module>
        ExpiresActive On
        ExpiresDefault "access plus 1 week"
    </IfModule>
</FilesMatch>

I played around on my local server and also on a live server at HostEurope - unfortunately they don't loaded mod_expires, this is why I also used mod_headers. So in case mod_expires is not available, only the Cache-Control headers will be sent. Do you guys know if you can do calculations in the .htaccess? If yes we wouldn't need mod_expires, we could send all the headers via mod_headers.

I also added this for our image.php, it's working great but of course it's failing when requesting a captcha-image. So I think we have to send the headers for the image.php via PHP, I don't see any chance to do it through .htaccess?!
This is what's working fine exept of the captcha:

Code: Alles auswählen

<FilesMatch "image.php$">
    Header set Cache-Control "max-age=604800, public"
    Header unset Pragma
    Header unset Expires
    <IfModule expires_module>
        ExpiresActive On
        ExpiresDefault "access plus 1 week"
    </IfModule>
</FilesMatch>