User Tools

Site Tools


new_rotation_faq

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revision Previous revision
Next revision
Previous revision
new_rotation_faq [2017/10/16 12:23]
127.0.0.1 external edit
new_rotation_faq [2022/08/05 07:46] (current)
admin [What is mod_rewrite]
Line 1: Line 1:
 +====== New Rotation FAQ ======
 +
 +===== What is mod_rewrite =====
 +
 +mod_rewrite - That's an addon that helps to create nice URLs that do not exist on server. There's a mod for nginx and lighttpd as well.
 +
 +Let's let by default you have urls like http://domain/gallery/cool-gallery/index.html, in fact there's no such path /gallery/cool-gallery/index.html on server. Mod_rewrite transforms it into http://domain/scj/cgi/out.php?url=content&slug=cool-gallery , "nice" URL is for SE and users.
 +
 +.htaccess is a file that sets a list of these rewrite urls for apache. 
 +
 +
 +I'll try to explain how it works:
 +
 +As I said before there's such path as /gallery/cool-gallery/index.html on server. You have to use an url like /scj/cgi/out.php?url=content&slug=cool-gallery to see a gallery, and this url doesn't look nice. That's why we add a special rule into .htaccess that says how to transform urls 
 +
 +
 +  RewriteRule ^gallery/(.*)/index.html$ /scj/cgi/out.php?url=content&slug=$1
 +
 +That means if URL starts with  gallery/, then we have "something" (.*) upto "/" symbol, and then it ends with "/index.html" - we have to transform it into /scj/cgi/out.php?url=content&slug=$1 where $1 is something that was in (.*)
 +
 +In this example $1 = "cool-gallery". So Apache will transform it into  /scj/cgi/out.php?url=content&slug=cool-gallery and you'll see a gallery.
 +
 +You see urls like /gallery/cool-gallery/index.html at your site because you have something like href="/gallery/<!--GALLERY_SLUG-->/index.html in your templates
 +
 +That means that if you'd like to change URLs - you need to change rewrites.
 +
 +
 +===== Rotation - Groups shows that I have 10 galleries while Rotation - List thumbs - 20 =====
 +
 +Each gallery can be assigned to more then 1 group. You can notice it while gallery edit: there are 2 select fields for groups - main and ext. Rotation - groups counts using main group so we can see real amount of galleries in DB, while Rotation - list thumbs shows galleries using Ext groups , the way you'll see it on site.
 +
 +
 +===== Where pages are stored =====
 +
 +With New Rotation you get a lot of pages. Old style rotation was like you create a finite number of pages (usually up to ten)
 +
 +Pretty common situation: 300 categories, 7 pages in each, 3 ways to sort galleries inside category (CTR, date, duration) - it's 6300 pages.
 +
 +It doesn't make sense to create those pages each minute so script creates it when someone requests it, stores in cache and uses cached version until it expires.
 +
 +Thus we have 2 points to remember:
 +
 +1. It takes time to create pages. if cache is empty it is called 'cold startup', we have to "warm up" cache , ie create some mostly visited pages.
 +
 +2. Once cache is up it doesn't really matters how many visitors you have as it almost does not consume any resources to give away a page from cache. 
 +
 +
 +Thus if you have 1 site with  50k or 500k traffic - it doesn't affect your server load too much as pages in most case will be cached. But if you have 5 sites with 10k traffic on each - it won't be the same at 1 site with 50k traffic as we have to create a full set of pages for each site.
 +
 +
 +Beside that your layout affects load: ie if you have 3 options to sort content it will be "total amount of pages * 3", while if you have just 2 ways to soft it (it depends design actually , ether you have a link to another way to soft it or not) - you'll get less page to create => less load on server.
 +
 +===== Thumbs (Galleries) are not getting deleted =====
 +
 +  * up to update 46 there was a problem with deletion of large parts of thumb, script was not able to delete lat's say 1000 thumb during 30 seconds browser timeout time.
 +  * that's why since update 46 thumbs won't be actually deleted once you press 'Delete', instead those thumbs will be marked as 'Marked for deletion' and once every 10 minutes cron script will check for those thumbs and actually delete it.
 +  * you want to force that action you have to run following command in SSG
 +  * <code>cd /PATH_TO_/scj/bin/; env HTTP_HOST=yourdomain.com php rotation.php process_deleted=true </code>
 +  * there's one exclusion - if you delete thumbs in 'List Thumbs' and total amount of thumbs being deleted is less then 30.
 +
 +
 +===== What is cache =====
 +
 +Script does not generate static page each minute or something like that. Instead it creates a page once someone request it and saves it to cache with expiration time = CACHE_TIME, so each next the same request will get this page from cache till expiration time.
 +
 +Yo can set CACHE_TIME in Rotation - Settings , but if for some reason you want to overwrite it somewhere - you can add in common.php 
 +
 +  if (!defined('CACHE_TIME')) define('CACHE_TIME', 900);
 +
 +
 +If you want a page without cache - add &skip_cache=true to any request for example http://domain.com/?skip_cache=true. Note that this request will only show a page to you and will NOT save this page to cache.
 +
 +If you want script to recreate a page AND save it to cache - you have to go to Rotation - Special - Recreate visited pages and it will add a cookie to your browser so each page you visit will be recreated and saved to cache.
 +
 +For example: A page was created at 0 seconds. Cache time = 1000 seconds. 
 +
 +At 200th second you change design. If you or any other surfer open that page at this time - you will not see new design. 
 +
 +If you open with 'skip_cache' - you'll see new design, surfers - not.
 +
 +If you open with 'Recreate cookie' - you'll see new design and surfers will see new design.
 +
 +
 +===== How much memory consumes cache =====
 +
 +It 100% depends on your site.
 +
 +We cache ready-made HTML in cache. So let's say your index page is 100k. 
 +Plus you have 100 categories, each category page 100k, and you have 50 pages within category , so that's 100 category * 50 pages * 100k = 500M
 +
 +but a user can see those pages in 3 sort modes so 500 * 3 ..
 +
 +Also let's say you have 100 000 galleries , each page is also 100k. So that's 10gb of html.
 +
 +Of course, HTML gets compressed so it consumes less memory, but you can get an idea.
 +
 +Usually when the cache is full an old record get deleted and a new one - stored in cache. Thus if a surfer requests an old one - we'll have to generate a page again rather then just pull it out of cache.
 +
 +===== Cache Engines (New) =====
 +
 +The main idea of cache is generate a page once and then show it to every visitor until cache expires. So we need a place to store cache data
 +
 +**Regular Files** 
 +Usually (bu default) we store cache in file (scj/cache - file cache). 
 +Pros - works everywhere, no need to setup anything. 
 +Cons - file cache means a lot of files on disk, and it's slow.
 +
 +**Memcached**
 +
 +Pros: easy to setup, easy to manage.
 +Cons: you lose all your cache data when server reboots or memcache is restarted. So each time after that you'll need some time to "warm up" cache ie create pages.
 +
 +Also unless you have more then one instance of memcached - all your cache data will be stored within one instance and if you click at "clean cache" at one site , this will kill cache for all sites.
 +
 +
 +**That's why there are a couple of new cache engines**
 +
 +The easiest one to migrate to is **Couchbase** 
 +**CouchBase** - it's a successor of MemBase (inspired by memcached I believe)
 +
 +The main advantage of it is that you don't have to do a lot to start using it.
 +  - Ask admin to install couchbase
 +  - add the same strings as for memcache
 +
 +<code>
 +$config['memcached_host'] = '127.0.0.1';
 +$config['memcached_port'] = '11211';
 +</code>
 +
 +that's it.
 +
 +
 +**Redis** - another great solution written from scratch. 
 +  - ask admin to install redis
 +  - ask admin to install redis extension for php
 +  - add to includes/config.php
 +<code>
 +$config['redis_host'] = '127.0.0.1';
 +$config['redis_port'] = '6379';
 +$config['redis_database'] = 0; 
 +$config['redis_password'] = '';
 +
 +if your admin prefers to use sockets then 
 +
 +$config['redis_host'] = '/tmp/redis.sock';
 +$config['redis_port'] = '0';
 +
 +
 +</code>
 +
 +That's it.
 +
 +
 +===== How to add a big list of thumbs =====
 +
 +Note, if you use version 2+ you can add big files as-is, no need to do any extra work.
 +
 +If you have really big list to import it could be really hard due to some reasons like size of POST is usually limited, server won't have enough time to process it and so on. Beside that adding a lot of galleries really fast may overload your server.
 +
 +Good idea is load load this list smoothly using import sets. You have to 
 +
 +  * create and upload file cut.php 
 +
 +<code>
 +
 +<?php
 +
 +error_reporting(E_ALL & ~E_NOTICE);
 +
 +if (!$_GET['file']) die("\n You have to pass file=... parameter in URL");
 +if (!file_exists($_GET['file'])) die("\n File {$_GET['file']} does NOT exists. Check path. ");
 +if (!is_writeable($_GET['file'])) die("\n File {$_GET['file']} is NOT writeable. Set 666 permissions. ");
 +
 +if (!file_exists('./tmp.file') or !is_writeable('./tmp.file')) die("\n File tmp.file does not exists OR is NOT writeable. Set 666 permissions. ");
 +
 +echo passthru("head -n500 {$_GET['file']} ");
 +if (!$_GET['test']) exec("tail -n +500 {$_GET['file']} > tmp.file; cp tmp.file {$_GET['file']} ");
 +
 +</code>
 +
 +  * upload to the same location your dump and set permissions 666
 +  * Upload empty file tmp.file and also set permissions 666
 +  * Add ImportSet with URL http://yourdomain/path_to/cut.php?file=DUMP_NAME&test=true
 +  * Click Test and select fields
 +  * cut.php outputs 500 lines and cuts dump by those 500 lines. This way you'll add everything as usual import set - gradually and smoothly. 
 +  * &test=true means do not cut dump by 500 lines this time. You have to remove this parameter after test. So your final URL will be http://yourdomain/path_to/cut.php?file=DUMP_NAME
 +
 +That's it.
 +
 +
 +===== Cache Engines (New) =====
 +
 +If you read [[New Rotation FAQ]] you should know that script generates a page on request only. We do not generate million of pages by cron or something like that. When the page is generated we have to store if somewhere in cache. But we have different types of cache you can use.
 +
 +
 +**File cache** 
 +A basic one. All cached pages are stored in files (scj/cache folder)
 +Pros - works everywhere, you don't have to do anything.
 +Cons - it's a file cache, it's slow, not so good cached by OS disk operation and script has to check all files for expiration date and so on.
 +
 +**Memcached**
 +it was a big step ahead some time ago. 
 +Pros: everybody know it. There's no proble for admin to setup it. 
 +Cons: sometimes it's hard to find out how much memory is actually still free, data gets fragmented, if you reboot server - you lose all cached data and need time to "warm up" cache after reboot. Memcache can not segregate cached data by popularity and stores everything in memory. It's really painful to reset cache just for 1 site for example.
 +
 +**That's why there are some other cache engines you worth using**
 +
 +New cache engines (basically NoSQL dbs) deliver you more performance while consuming less CPU and HDD.
 +
 +Easiest solution - **Couchbase** which was built on MemBase, based on memcachedb, which was inspired  memcached. It's really easy to start using it
 +
 +  - ask admin to setup CouchBase
 +  - add 2 lines to scj/includes/config.php
 +<code>
 +$config['memcached_host'] = '127.0.0.1';
 +$config['memcached_port'] = '11211';
 +</code>
 +
 +and that's it. You can use it with versions 48-49 too.
 +
 +**Redis** - another nice key-value product . To you it you have to
 +  - ask admin to setup Redis 
 +  - ask admin to add redis module for php
 +  - add following to scj/includes/config.php
 +<code>
 +$config['redis_host'] = '127.0.0.1';
 +$config['redis_port'] = '6379';
 +$config['redis_database'] = 0; 
 +$config['redis_password'] = '';
 +</code>
 +
 +There are few new lines. redis_password - if you have a dedicated server most likely you don't want to setup it. redis_database - it's kinda number of slot in redis DB. Useful if you want to separate caches for your sites for some reason.
 +
 +
 +Any of those 2 new engines are better then memcache and definitely better then file cache. I'd recommend to use file cache in emergency case only.
 +
 +
 +===== Custom Sitemaps =====
 +
 +Right after installation you should have a default sitemap template. Basically sitemap is a regular template, the only difference is that you have to output a special header to identify that it's a xml format.
 +
 +<code>
 +<? echo '<?xml version="1.0" encoding="UTF-8"?>' ?>
 +
 +</code>
 +
 +and then goes the rest of template with galleries 
 +
 +<code>
 +<thumb num=1-100>
 +
 +<url>
 +    <loc>http://<?=$_SERVER['HTTP_HOST']?>/gallery/<!--GALLERY_SLUG-->/index.html</loc>
 +    <lastmod><!--ACTIVATION_DATE--></lastmod>
 +    <changefreq>daily</changefreq>
 +    <priority>0.5</priority>
 +</url>
 +
 +</thumb>
 +
 +</code>
 +
 +
 +In case if you want to output let's say categories just replace it with a proper tag
 +
 +For example
 +
 +<code>
 +<category num=1-100>
 +
 +<url>
 +    <loc>http://<?=$_SERVER['HTTP_HOST']?>/category/<!--CATEGORY_NAME-->/</loc>
 +    <changefreq>daily</changefreq>
 +    <priority>1</priority>
 +</url>
 +
 +</category>
 +
 +</code>
 +
 +and so on. Don't forget to change urls and other parameters in subtemplate.
 +
 +If you need to list category page you have to use <!--TOTAL_ITEMS--> tag to calculate how many page you have in each category
 +
 +for example 
 +
 +
 +<code>
 +<category num=1-100>
 +
 +<url>
 +    <loc>http://<?=$_SERVER['HTTP_HOST']?>/category/<!--CATEGORY_NAME-->/</loc>
 +    <changefreq>daily</changefreq>
 +    <priority>1</priority>
 +</url>
 +
 +then list pages
 +
 +<?
 +$per_page = 100;
 +
 +for ($i = 1; $i <= ceil('<!--TOTAL_ITEMS-->'/$per_page); $i++ )  {
 +?>
 +
 +<a href='/category/<!--CATEGORY_NAME-->/<?=$i?>/'>Page <?=$i?></a>
 +
 +<?
 +}
 +?>
 +
 +
 +
 +</category>
 +
 +</code>
 +
 +
 +===== Why order of thumbs differs in admin area and on site =====
 +
 +In most cases order should be the same order but you should keep in mind few settings:
 +
 +  - Test positions start (do not test new thumbs at positions higher then ) ie if a thumb is considered as new if won't be shown in the top position on site, but you can see it the top position in admin are
 +  - % of test places on page (% of places on each page where we test NEW thumbs ) some thumbs might not be in the same order as well 
 +
 +
 +===== eval()'d code =====
 +
 +If you see something like "Parse error: some error, in /some/path/scj/tube/index.php(0) : eval()'d code on line 1717" - it means you have an error in php code in your template
 +
 +Check php tags like (<?php vs <?), commas. Or  you can remove your phpcode peace by peace to see what part of code causes the error.
 +
 +===== How to import a large cvs =====
 +
 +Note, with version 2.X you can add as large cvs as you want. Written below relates to 1.X only.
 +
 +There's a logical issue with large cvss. 
 +A file can be too large to be uploaded with browser, it can take too much time to process it that causes timeouts and so on.
 +
 +So some people decided to use importsets to inport large file and it would be file if they don't forget about it.
 +
 +A common issue that 90% forget about those importsets and the script tries to import let's say 500 000 galleries every hour and that takes a lot of resources every hour.
 +
 +Besides that when you import a large file at once it creates a spike in server load that might cause a notable slow down in server's performance. 
 +
 +So here's a good way to import large files:
 +
 +  * create a file called cut.php 
 +
 +<code>
 +
 +<?php
 +
 +error_reporting(E_ALL & ~E_NOTICE);
 +
 +if (!$_GET['file']) die("\n You have to pass file=... parameter in URL");
 +if (!file_exists($_GET['file'])) die("\n File {$_GET['file']} does NOT exists. Check path. ");
 +if (!is_writeable($_GET['file'])) die("\n File {$_GET['file']} is NOT writeable. Set 666 permissions. ");
 +
 +if (!file_exists('./tmp.file') or !is_writeable('./tmp.file')) die("\n File tmp.file does not exists OR is NOT writeable. Set 666 permissions. ");
 +
 +echo passthru("head -n500 {$_GET['file']} ");
 +if (!$_GET['test']) exec("tail -n +500 {$_GET['file']} > tmp.file; cp tmp.file {$_GET['file']} ");
 +
 +</code>
 +
 +  * upload a cvs file to the same location and chmod 666 it
 +  * Upload emtry file tmp.file and chmod 666 it as well
 +  * Add ImportSet with URL http://yourdomain/path_to/cut.php?file=cvs_file_name&test=true
 +  * Click Test and select fields
 +  * cut.php outputs  500 lines and cuts out thoses 500 files from cvs file. So on each run you'll add 500 new galleries.  Note, you can change this amount in the code above.
 +  * &test=true means "do not cut out those 500 lines". That's necessary for test purposes.later you should remove it and it should be like http://yourdomain/path_to/cut.php?file=cvs_file_name
 +
 +That's it.
 +
 +So this was an entire large cvs file will be addded and even if you forget to remove that import set it won't create any load on a server.
 +
 +
 +===== Lazy load, load on scroll, pinterest and so on  =====
 +
 +The base idea here is to load some content when user scrolls down the end of a page. So user doesn't have to reload an entire page but gets "infinite" scroll instead. 
 +
 +Let's sat we have a template "index" with 50 thumbs
 +
 +<code>
 +<thumb num=1-50>
 +<a href='/...'><img src='<!--THUMB_URL-->'></a>
 +</thumb>
 +</code>
 +
 +
 +In this example we use jquery.wookmark JS library (example here http://demo.smartcj.com/scroll/) Actually there are a plenty of libs like this one
 +
 +When user reaches the end of a page we need to load some new content so here's an URL for it
 +
 +  var P_BASE = '/?force_template=index_scroll_ajax&page=';
 +  
 +  
 +as you can see we load template "index_scroll_ajax" with a variable &page=.. So what jquery.wookmark does it changing that last parameter &page= each time loading /?force_template=index_scroll_ajax&page=2, /?force_template=index_scroll_ajax&page=3 and so on
 +
 +Template index_scroll_ajax where we output 10 thumbs
 +
 +<code>
 +<thumb num=1-10>
 +<a href='/...'><img src='<!--THUMB_URL-->'></a>
 +</thumb>
 +</code>
 +
 +
 +
 +===== Thumbs\galleries are not being deleted =====
 +
 +Thumbs are not deleted right away when you click "delete" in admin area because we need to reload a page right away (to avoid timeout). So instead щи being deleted thumbs are marked as 'to delete'. Every 10 minutes rotation.php crontab job checks for thumbs with this mark and actually delete them.
 +
 +if you want to force deletion:
 +
 +<code>cd /PATH_TO_/scj/bin/; env HTTP_HOST=yourdomain.com php rotation.php process_deleted=true </code>
 +
 +
 +===== It does not grab this gallery  =====
 +
 +Some galleries might have some peculiarities and are created in a way that the script can not download it.  We try to implement it as good as we can but if there's a gallery you'd like to add but the script can not parse it - you can use so called 'gateway' for it:
 +
 +  * let's say a gallery http://gallery/ contains an image http://gallery/image.jpg but the script can not see it
 +  * we create file http://your_server/gate.php
 +  * and import the gallery  as http://your_server/gate.php?url=http://gallery/ 
 +  * your script gate.php gets URL,  parses the real gallery and outputs it in a regular way , for example 
 +
 +<code>
 +<a href='http://image1'><img src='http://thumb1'></a>
 +<a href='http://image2'><img src='http://thumb2'></a>
 +and so on
 +</code>
 +
 +**ImportSets**
 +
 +Let's say we have importset http://sponsor/dump.php that outputs something like 
 +
 +url|thumb_url|some_id
 +
 +and we'd like to replace some_id some something more meaningful like http://test/modified_id
 +
 +
 +  * gate.php?url=http://sponsor/dump.php
 +  * the gate gets URL, downloads it and replaces some_id with http://test/modified_id and outputs, so the script get's dump in a way you want it to be
 +  * we add gate.php?url=http://sponsor/dump.php in import set instead of http://sponsor/dump.php
 +
 +
 +
 +===== Fast Navigation  =====
 +
 +If you have a lot of galleries (millions) it takes a lot of time to count total amount of galleries, actually it takes 90% of time. So it makes sense to replace <pagination> with <!--PREV_PAGE--> <!--NEXT_PAGE-->
 +
 +if you use <pagination> the script has to go thought all the DB and calculate total amount of galleries, it takes time\load, if you use <!--PREV_PAGE--> <!--NEXT_PAGE--> - no need to count, and to be honest users dont care.
 +