This shows you the differences between two versions of the page.
Both sides previous revision Previous revision Next revision | Previous revision | ||
new_rotation_faq [2019/09/13 15:17] admin [Rotation - Groups shows that I have 10 galleries while Rotation - List thumbs - 20] |
new_rotation_faq [2022/08/05 07:46] (current) admin [What is mod_rewrite] |
||
---|---|---|---|
Line 1: | Line 1: | ||
+ | ====== New Rotation FAQ ====== | ||
+ | |||
+ | ===== What is mod_rewrite ===== | ||
+ | |||
+ | mod_rewrite - That's an addon that helps to create nice URLs that do not exist on server. There' | ||
+ | |||
+ | Let's let by default you have urls like http:// | ||
+ | |||
+ | .htaccess is a file that sets a list of these rewrite urls for apache. | ||
+ | |||
+ | |||
+ | I'll try to explain how it works: | ||
+ | |||
+ | As I said before there' | ||
+ | |||
+ | |||
+ | RewriteRule ^gallery/ | ||
+ | |||
+ | That means if URL starts with gallery/, then we have " | ||
+ | |||
+ | In this example $1 = " | ||
+ | |||
+ | You see urls like / | ||
+ | |||
+ | That means that if you'd like to change URLs - you need to change rewrites. | ||
+ | |||
+ | |||
+ | ===== Rotation - Groups shows that I have 10 galleries while Rotation - List thumbs - 20 ===== | ||
+ | |||
+ | Each gallery can be assigned to more then 1 group. You can notice it while gallery edit: there are 2 select fields for groups - main and ext. Rotation - groups counts using main group so we can see real amount of galleries in DB, while Rotation - list thumbs shows galleries using Ext groups , the way you'll see it on site. | ||
+ | |||
+ | |||
+ | ===== Where pages are stored ===== | ||
+ | |||
+ | With New Rotation you get a lot of pages. Old style rotation was like you create a finite number of pages (usually up to ten) | ||
+ | |||
+ | Pretty common situation: 300 categories, 7 pages in each, 3 ways to sort galleries inside category (CTR, date, duration) - it's 6300 pages. | ||
+ | |||
+ | It doesn' | ||
+ | |||
+ | Thus we have 2 points to remember: | ||
+ | |||
+ | 1. It takes time to create pages. if cache is empty it is called 'cold startup', | ||
+ | |||
+ | 2. Once cache is up it doesn' | ||
+ | |||
+ | |||
+ | Thus if you have 1 site with 50k or 500k traffic - it doesn' | ||
+ | |||
+ | |||
+ | Beside that your layout affects load: ie if you have 3 options to sort content it will be "total amount of pages * 3", while if you have just 2 ways to soft it (it depends design actually , ether you have a link to another way to soft it or not) - you'll get less page to create => less load on server. | ||
+ | |||
+ | ===== Thumbs (Galleries) are not getting deleted ===== | ||
+ | |||
+ | * up to update 46 there was a problem with deletion of large parts of thumb, script was not able to delete lat's say 1000 thumb during 30 seconds browser timeout time. | ||
+ | * that's why since update 46 thumbs won't be actually deleted once you press ' | ||
+ | * you want to force that action you have to run following command in SSG | ||
+ | * < | ||
+ | * there' | ||
+ | |||
+ | |||
+ | ===== What is cache ===== | ||
+ | |||
+ | Script does not generate static page each minute or something like that. Instead it creates a page once someone request it and saves it to cache with expiration time = CACHE_TIME, so each next the same request will get this page from cache till expiration time. | ||
+ | |||
+ | Yo can set CACHE_TIME in Rotation - Settings , but if for some reason you want to overwrite it somewhere - you can add in common.php | ||
+ | |||
+ | if (!defined(' | ||
+ | |||
+ | |||
+ | If you want a page without cache - add & | ||
+ | |||
+ | If you want script to recreate a page AND save it to cache - you have to go to Rotation - Special - Recreate visited pages and it will add a cookie to your browser so each page you visit will be recreated and saved to cache. | ||
+ | |||
+ | For example: A page was created at 0 seconds. Cache time = 1000 seconds. | ||
+ | |||
+ | At 200th second you change design. If you or any other surfer open that page at this time - you will not see new design. | ||
+ | |||
+ | If you open with ' | ||
+ | |||
+ | If you open with ' | ||
+ | |||
+ | |||
+ | ===== How much memory consumes cache ===== | ||
+ | |||
+ | It 100% depends on your site. | ||
+ | |||
+ | We cache ready-made HTML in cache. So let's say your index page is 100k. | ||
+ | Plus you have 100 categories, each category page 100k, and you have 50 pages within category , so that's 100 category * 50 pages * 100k = 500M | ||
+ | |||
+ | but a user can see those pages in 3 sort modes so 500 * 3 .. | ||
+ | |||
+ | Also let's say you have 100 000 galleries , each page is also 100k. So that's 10gb of html. | ||
+ | |||
+ | Of course, HTML gets compressed so it consumes less memory, but you can get an idea. | ||
+ | |||
+ | Usually when the cache is full an old record get deleted and a new one - stored in cache. Thus if a surfer requests an old one - we'll have to generate a page again rather then just pull it out of cache. | ||
+ | |||
+ | ===== Cache Engines (New) ===== | ||
+ | |||
+ | The main idea of cache is generate a page once and then show it to every visitor until cache expires. So we need a place to store cache data | ||
+ | |||
+ | **Regular Files** | ||
+ | Usually (bu default) we store cache in file (scj/cache - file cache). | ||
+ | Pros - works everywhere, no need to setup anything. | ||
+ | Cons - file cache means a lot of files on disk, and it's slow. | ||
+ | |||
+ | **Memcached** | ||
+ | |||
+ | Pros: easy to setup, easy to manage. | ||
+ | Cons: you lose all your cache data when server reboots or memcache is restarted. So each time after that you'll need some time to "warm up" cache ie create pages. | ||
+ | |||
+ | Also unless you have more then one instance of memcached - all your cache data will be stored within one instance and if you click at "clean cache" at one site , this will kill cache for all sites. | ||
+ | |||
+ | |||
+ | **That' | ||
+ | |||
+ | The easiest one to migrate to is **Couchbase** | ||
+ | **CouchBase** - it's a successor of MemBase (inspired by memcached I believe) | ||
+ | |||
+ | The main advantage of it is that you don't have to do a lot to start using it. | ||
+ | - Ask admin to install couchbase | ||
+ | - add the same strings as for memcache | ||
+ | |||
+ | < | ||
+ | $config[' | ||
+ | $config[' | ||
+ | </ | ||
+ | |||
+ | that's it. | ||
+ | |||
+ | |||
+ | **Redis** - another great solution written from scratch. | ||
+ | - ask admin to install redis | ||
+ | - ask admin to install redis extension for php | ||
+ | - add to includes/ | ||
+ | < | ||
+ | $config[' | ||
+ | $config[' | ||
+ | $config[' | ||
+ | $config[' | ||
+ | |||
+ | if your admin prefers to use sockets then | ||
+ | |||
+ | $config[' | ||
+ | $config[' | ||
+ | |||
+ | |||
+ | </ | ||
+ | |||
+ | That's it. | ||
+ | |||
+ | |||
+ | ===== How to add a big list of thumbs ===== | ||
+ | |||
+ | Note, if you use version 2+ you can add big files as-is, no need to do any extra work. | ||
+ | |||
+ | If you have really big list to import it could be really hard due to some reasons like size of POST is usually limited, server won't have enough time to process it and so on. Beside that adding a lot of galleries really fast may overload your server. | ||
+ | |||
+ | Good idea is load load this list smoothly using import sets. You have to | ||
+ | |||
+ | * create and upload file cut.php | ||
+ | |||
+ | < | ||
+ | |||
+ | <?php | ||
+ | |||
+ | error_reporting(E_ALL & ~E_NOTICE); | ||
+ | |||
+ | if (!$_GET[' | ||
+ | if (!file_exists($_GET[' | ||
+ | if (!is_writeable($_GET[' | ||
+ | |||
+ | if (!file_exists(' | ||
+ | |||
+ | echo passthru(" | ||
+ | if (!$_GET[' | ||
+ | |||
+ | </ | ||
+ | |||
+ | * upload to the same location your dump and set permissions 666 | ||
+ | * Upload empty file tmp.file and also set permissions 666 | ||
+ | * Add ImportSet with URL http:// | ||
+ | * Click Test and select fields | ||
+ | * cut.php outputs 500 lines and cuts dump by those 500 lines. This way you'll add everything as usual import set - gradually and smoothly. | ||
+ | * & | ||
+ | |||
+ | That's it. | ||
+ | |||
+ | |||
+ | ===== Cache Engines (New) ===== | ||
+ | |||
+ | If you read [[New Rotation FAQ]] you should know that script generates a page on request only. We do not generate million of pages by cron or something like that. When the page is generated we have to store if somewhere in cache. But we have different types of cache you can use. | ||
+ | |||
+ | |||
+ | **File cache** | ||
+ | A basic one. All cached pages are stored in files (scj/cache folder) | ||
+ | Pros - works everywhere, you don't have to do anything. | ||
+ | Cons - it's a file cache, it's slow, not so good cached by OS disk operation and script has to check all files for expiration date and so on. | ||
+ | |||
+ | **Memcached** | ||
+ | it was a big step ahead some time ago. | ||
+ | Pros: everybody know it. There' | ||
+ | Cons: sometimes it's hard to find out how much memory is actually still free, data gets fragmented, if you reboot server - you lose all cached data and need time to "warm up" cache after reboot. Memcache can not segregate cached data by popularity and stores everything in memory. It's really painful to reset cache just for 1 site for example. | ||
+ | |||
+ | **That' | ||
+ | |||
+ | New cache engines (basically NoSQL dbs) deliver you more performance while consuming less CPU and HDD. | ||
+ | |||
+ | Easiest solution - **Couchbase** which was built on MemBase, based on memcachedb, which was inspired | ||
+ | |||
+ | - ask admin to setup CouchBase | ||
+ | - add 2 lines to scj/ | ||
+ | < | ||
+ | $config[' | ||
+ | $config[' | ||
+ | </ | ||
+ | |||
+ | and that's it. You can use it with versions 48-49 too. | ||
+ | |||
+ | **Redis** - another nice key-value product . To you it you have to | ||
+ | - ask admin to setup Redis | ||
+ | - ask admin to add redis module for php | ||
+ | - add following to scj/ | ||
+ | < | ||
+ | $config[' | ||
+ | $config[' | ||
+ | $config[' | ||
+ | $config[' | ||
+ | </ | ||
+ | |||
+ | There are few new lines. redis_password - if you have a dedicated server most likely you don't want to setup it. redis_database - it's kinda number of slot in redis DB. Useful if you want to separate caches for your sites for some reason. | ||
+ | |||
+ | |||
+ | Any of those 2 new engines are better then memcache and definitely better then file cache. I'd recommend to use file cache in emergency case only. | ||
+ | |||
+ | |||
+ | ===== Custom Sitemaps ===== | ||
+ | |||
+ | Right after installation you should have a default sitemap template. Basically sitemap is a regular template, the only difference is that you have to output a special header to identify that it's a xml format. | ||
+ | |||
+ | < | ||
+ | <? echo '<? | ||
+ | |||
+ | </ | ||
+ | |||
+ | and then goes the rest of template with galleries | ||
+ | |||
+ | < | ||
+ | <thumb num=1-100> | ||
+ | |||
+ | <url> | ||
+ | < | ||
+ | < | ||
+ | < | ||
+ | < | ||
+ | </ | ||
+ | |||
+ | </ | ||
+ | |||
+ | </ | ||
+ | |||
+ | |||
+ | In case if you want to output let's say categories just replace it with a proper tag | ||
+ | |||
+ | For example | ||
+ | |||
+ | < | ||
+ | < | ||
+ | |||
+ | <url> | ||
+ | < | ||
+ | < | ||
+ | < | ||
+ | </ | ||
+ | |||
+ | </ | ||
+ | |||
+ | </ | ||
+ | |||
+ | and so on. Don't forget to change urls and other parameters in subtemplate. | ||
+ | |||
+ | If you need to list category page you have to use < | ||
+ | |||
+ | for example | ||
+ | |||
+ | |||
+ | < | ||
+ | < | ||
+ | |||
+ | <url> | ||
+ | < | ||
+ | < | ||
+ | < | ||
+ | </ | ||
+ | |||
+ | then list pages | ||
+ | |||
+ | <? | ||
+ | $per_page = 100; | ||
+ | |||
+ | for ($i = 1; $i <= ceil('< | ||
+ | ?> | ||
+ | |||
+ | <a href='/ | ||
+ | |||
+ | <? | ||
+ | } | ||
+ | ?> | ||
+ | |||
+ | |||
+ | |||
+ | </ | ||
+ | |||
+ | </ | ||
+ | |||
+ | |||
+ | ===== Why order of thumbs differs in admin area and on site ===== | ||
+ | |||
+ | In most cases order should be the same order but you should keep in mind few settings: | ||
+ | |||
+ | - Test positions start (do not test new thumbs at positions higher then ) ie if a thumb is considered as new if won't be shown in the top position on site, but you can see it the top position in admin are | ||
+ | - % of test places on page (% of places on each page where we test NEW thumbs ) some thumbs might not be in the same order as well | ||
+ | |||
+ | |||
+ | ===== eval()' | ||
+ | |||
+ | If you see something like "Parse error: some error, in / | ||
+ | |||
+ | Check php tags like (<?php vs <?), commas. Or you can remove your phpcode peace by peace to see what part of code causes the error. | ||
+ | |||
+ | ===== How to import a large cvs ===== | ||
+ | |||
+ | Note, with version 2.X you can add as large cvs as you want. Written below relates to 1.X only. | ||
+ | |||
+ | There' | ||
+ | A file can be too large to be uploaded with browser, it can take too much time to process it that causes timeouts and so on. | ||
+ | |||
+ | So some people decided to use importsets to inport large file and it would be file if they don't forget about it. | ||
+ | |||
+ | A common issue that 90% forget about those importsets and the script tries to import let's say 500 000 galleries every hour and that takes a lot of resources every hour. | ||
+ | |||
+ | Besides that when you import a large file at once it creates a spike in server load that might cause a notable slow down in server' | ||
+ | |||
+ | So here's a good way to import large files: | ||
+ | |||
+ | * create a file called cut.php | ||
+ | |||
+ | < | ||
+ | |||
+ | <?php | ||
+ | |||
+ | error_reporting(E_ALL & ~E_NOTICE); | ||
+ | |||
+ | if (!$_GET[' | ||
+ | if (!file_exists($_GET[' | ||
+ | if (!is_writeable($_GET[' | ||
+ | |||
+ | if (!file_exists(' | ||
+ | |||
+ | echo passthru(" | ||
+ | if (!$_GET[' | ||
+ | |||
+ | </ | ||
+ | |||
+ | * upload a cvs file to the same location and chmod 666 it | ||
+ | * Upload emtry file tmp.file and chmod 666 it as well | ||
+ | * Add ImportSet with URL http:// | ||
+ | * Click Test and select fields | ||
+ | * cut.php outputs | ||
+ | * & | ||
+ | |||
+ | That's it. | ||
+ | |||
+ | So this was an entire large cvs file will be addded and even if you forget to remove that import set it won't create any load on a server. | ||
+ | |||
+ | |||
+ | ===== Lazy load, load on scroll, pinterest and so on ===== | ||
+ | |||
+ | The base idea here is to load some content when user scrolls down the end of a page. So user doesn' | ||
+ | |||
+ | Let's sat we have a template " | ||
+ | |||
+ | < | ||
+ | <thumb num=1-50> | ||
+ | <a href='/ | ||
+ | </ | ||
+ | </ | ||
+ | |||
+ | |||
+ | In this example we use jquery.wookmark JS library (example here http:// | ||
+ | |||
+ | When user reaches the end of a page we need to load some new content so here's an URL for it | ||
+ | |||
+ | var P_BASE = '/? | ||
+ | | ||
+ | | ||
+ | as you can see we load template " | ||
+ | |||
+ | Template index_scroll_ajax where we output 10 thumbs | ||
+ | |||
+ | < | ||
+ | <thumb num=1-10> | ||
+ | <a href='/ | ||
+ | </ | ||
+ | </ | ||
+ | |||
+ | |||
+ | |||
+ | ===== Thumbs\galleries are not being deleted ===== | ||
+ | |||
+ | Thumbs are not deleted right away when you click " | ||
+ | |||
+ | if you want to force deletion: | ||
+ | |||
+ | < | ||
+ | |||
+ | |||
+ | ===== It does not grab this gallery | ||
+ | |||
+ | Some galleries might have some peculiarities and are created in a way that the script can not download it. We try to implement it as good as we can but if there' | ||
+ | |||
+ | * let's say a gallery http:// | ||
+ | * we create file http:// | ||
+ | * and import the gallery | ||
+ | * your script gate.php gets URL, parses the real gallery and outputs it in a regular way , for example | ||
+ | |||
+ | < | ||
+ | <a href=' | ||
+ | <a href=' | ||
+ | and so on | ||
+ | </ | ||
+ | |||
+ | **ImportSets** | ||
+ | |||
+ | Let's say we have importset http:// | ||
+ | |||
+ | url|thumb_url|some_id | ||
+ | |||
+ | and we'd like to replace some_id some something more meaningful like http:// | ||
+ | |||
+ | |||
+ | * gate.php? | ||
+ | * the gate gets URL, downloads it and replaces some_id with http:// | ||
+ | * we add gate.php? | ||
+ | |||
+ | |||
+ | |||
+ | ===== Fast Navigation | ||
+ | |||
+ | If you have a lot of galleries (millions) it takes a lot of time to count total amount of galleries, actually it takes 90% of time. So it makes sense to replace < | ||
+ | |||
+ | if you use < | ||
+ | |||