Table of Contents
Trade CMS
Thumb CMS - this is the next step in the development of SmartCJ.
We have always tried to support all possible server configurations, including those running on PHP 4 and MySQL 3.23, despite the fact that support for these versions has been discontinued ten years ago. Now it's time to take the next step in many ways.
What's new
The new branch has a number of quality changes for all aspects of the script.
- The database was optimized to enable fast work with databases of 200-300k thumbs in each category. Previously this was a problem, as groups with a large number of thumbs seriously slowed down the site.
- Crons can be run as often as you like, this will only affect the speed of statistics updates and history saves. At the same time, crons control their own execution to avoid running duplicates. However, due to point 1, this may not be necessary.
- Database selections and updates have been optimized
The script has become completely “invisible”:
- You can place the script in any folder
- Rewrites are simplified, you can make any links on the site
- The script doesn't set any cookies in Cookie Engine = Cache mode or sets encoded ones in Cookie Engine = MD5 mode
- You can remove TCMS Rotation Parameter and the script will still be able to count thumb statistics, even if there are multiple on a gallery + Cell K
Thus the question of ranking sites by Google depending on the script will be completely removed.
A number of unique features have been added:
- Thumb Split - uniquification of thumbs by cutting them into multiple parts
- FaceDetect - automation of thumb cutting, taking into account the position of the face to create higher quality thumbs automatically
Installation
Installation has become even simpler and should please admins. Now you don't need to download anything, just give the command in the shell
curl -sS http://smartcj.com/updates2/install | php --
and the script will be installed in interactive mode. PS yes, two dashes at the end are required
you can also specify all parameters directly in the command line
curl -sS http://smartcj.com/updates2/install | php -- mysql_host= mysql_user= mysql_pass= mysql_name= scj_folder= domain= admin_email=
Default Rewrites
RewriteEngine On
RewriteBase /
RewriteCond %{REQUEST_FILENAME} !-f
RewriteCond %{REQUEST_FILENAME} !-d
RewriteRule ^([^/]{2})/(.*)$ $2?force_lng=$1&%{QUERY_STRING} [L]
# add &dont_count_in_trade=true if you dont want to count these clicks
RewriteRule ^category/([^/]+)/$ /script_folder_name/cgi/out.php?url=content&p=100&skip_sell=true&group_name=$1&order=ctr&%{QUERY_STRING} [L]
RewriteRule ^category/([^/]+)/([^/]+)/([^/]+)/$ /script_folder_name/cgi/out.php?url=content&p=100&skip_sell=true&group_name=$1&order=$2&page=$3&%{QUERY_STRING} [L]
#slug galleries
RewriteRule ^gallery/([^/]+)/index.html$ /script_folder_name/cgi/out.php?url=content&slug=$1&%{QUERY_STRING} [L]
RewriteRule ^gallery/([^/]+)/([0-9]{1,2}).html$ /script_folder_name/cgi/out.php?url=content&slug=$1&item_id=$2&%{QUERY_STRING} [L]
#tags
RewriteRule ^tag/([^/]+)/([^/]+)/([^/]+)/$ /?tag=$1&page=$3&order=$2&%{QUERY_STRING} [L]
RewriteRule ^tag/([^/]+)/([^/]+)/$ /?tag=$1&page=$2&%{QUERY_STRING} [L]
RewriteRule ^tag/([^/]+)/$ /?tag=$1&%{QUERY_STRING} [L]
#model galleries
RewriteRule ^model_galleries/([^/]+)/$ /?model_slug=$1&force_template=model_galleries [L]
RewriteRule ^model_galleries/([^/]+)/([^/]+)/$ /?model_slug=$1&page=$2&force_template=model_galleries [L]
RewriteRule ^model/([^/]+)/$ /?model_slug=$1&force_template=model_galleries&%{QUERY_STRING} [L]
RewriteRule ^model/([^/]+)/([^/]+)/([^/]+)/$ /?model_slug=$1&force_template=model_galleries&order=$2&page=$3&%{QUERY_STRING} [L]
RewriteRule ^search/([^/]+)/([^/]+)/$ /?search=$1&page=$2&%{QUERY_STRING}
RewriteRule ^search/([^/]+)/([^/]+)/([^/]+)/$ /?search=$1&order=$2&page=$3&%{QUERY_STRING}
For nginx
server {
....
location /scj/scj2.lic {
return 404;
}
location / {
.....
if (!-e $request_filename) {
rewrite "^/([^/]{2})/(.*)$" /$2?force_lng=$1&$query_string ;
}
}
location /gallery {
rewrite ^/gallery/([^/]+)/index.html$ /script_folder_name/cgi/out.php?url=content&slug=$1&$query_string ;
rewrite "^/gallery/([^/]+)/([0-9]{1,2}).html$" /script_folder_name/cgi/out.php?url=content&slug=$1&item_id=$2&$query_string ;
}
location /category {
rewrite ^/category/([^/]+)/$ /script_folder_name/cgi/out.php?url=content&p=100&dont_count_in_trade=true&skip_sell=true&group_name=$1&order=ctr&$query_string ;
rewrite ^/category/([^/]+)/([^/]+)/([^/]+)/$ /script_folder_name/cgi/out.php?url=content&p=100&dont_count_in_trade=true&skip_sell=true&group_name=$1&order=$2&page=$3&$query_string ;
}
location /tag {
rewrite ^/tag/([^/]+)/([^/]+)/([^/]+)/$ /?tag=$1&page=$3&order=$2&$query_string ;
rewrite ^/tag/([^/]+)/([^/]+)/$ /?tag=$1&page=$2&$query_string ;
rewrite ^/tag/([^/]+)/$ /?tag=$1&$query_string ;
}
location /model_galleries {
rewrite ^/model_galleries/([^/]+)/$ /?model_slug=$1&force_template=model_galleries ;
rewrite ^/model_galleries/([^/]+)/([^/]+)/$ /?model_slug=$1&page=$2&force_template=model_galleries ;
}
location /model {
rewrite ^/model/([^/]+)/$ /?model_slug=$1&force_template=model_galleries&$query_string ;
rewrite ^/model/([^/]+)/([^/]+)/([^/]+)/$ /?model_slug=$1&force_template=model_galleries&order=$2&page=$3&$query_string ;
}
location /search {
rewrite ^/search/([^/]+)/([^/]+)/$ /?search=$1&page=$2&$query_string;
rewrite ^/search/([^/]+)/([^/]+)/([^/]+)/$ /?search=$1&order=$2&page=$3&$query_string;
}
Search
By default, the search uses a construct like WHERE description LIKE '%search_term%'. In the rotation settings there is an item
Search fields If you don't use Sphinx the script will use Mysql built-in functionality. It's slower so we limit amount of fields Please, read the last WIKI about the last option.
By default the script searches in description, in this item you can configure search in Title (Alt), or in both fields at once, which of course creates somewhat more load.
The best option is to configure Sphinx which is described below.
But there is a compromise option: FullText Search for Mysql (the last of the Search fields option items). Its point is that Mysql also knows how to search taking into account morphology, but before version 5.6 it did this only for MyISAM tables.
So if you want to use this option, which is somewhat simpler than the Sphinx option in the setup, you need to add an index for the rot_gallery_data* tables
ALTER TABLE `rot_gallery_data1` ADD FULLTEXT ( `alt` , `description` )
And switch the option in settings to FullText search.
Sphinx Config
Please note that in Sphinx Settings in 99% of cases you need to write 127.0.0.1
Example config
You need to make changes in the section
searchd
{
listen = your_port:mysql41
the main thing here is adding mysql41, a new way for sphinx to listen over the mysql protocol
Next is an example for the database
source your_name_source
{
type = mysql
sql_host = localhost
sql_user =
sql_pass =
sql_db =
sql_port = 3306 # optional, default is 3306
sql_query_pre = SET NAMES utf8
sql_query = SELECT gi.gallery_id, UNIX_TIMESTAMP(gi.activation_date) as date, alt, description, gi.duration, sponsor_id, gs.total_ctr, gi.content_type, \
(SELECT group_concat(tag_name) FROM rot_gal2tag g2t \
LEFT JOIN rot_tags as t on t.tag_id = g2t.tag_id \
WHERE g2t.gallery_id = gi.gallery_id) as tags, \
(SELECT group_concat(tag_id) FROM rot_gal2tag g2t \
WHERE g2t.gallery_id = gi.gallery_id) as tag_ids, \
(SELECT group_concat(name) FROM rot_groups \
WHERE rot_groups.id in (SELECT group_id FROM rot_gallery_stats1 WHERE rot_gallery_stats1.gallery_id = gi.gallery_id AND group_id != 0) ) as group_names, \
\
(SELECT group_concat(name) FROM rot_groups \
WHERE rot_groups.id in (SELECT group_id FROM rot_gallery_stats1 WHERE rot_gallery_stats1.gallery_id = gi.gallery_id AND group_id != 0) ) as group_names, \
(SELECT group_concat(gss.group_id) FROM rot_gallery_stats1 as gss \
WHERE gss.gallery_id = gi.gallery_id AND group_id != 0) as categories \
FROM rot_gallery_info AS gi \
JOIN rot_gallery_data1 AS gd ON gi.gallery_id = gd.gallery_id \
JOIN rot_gallery_stats1 AS gs ON gs.gallery_id = gi.gallery_id \
WHERE gallery_status = 'active' and gallery_type = 0 \
and gs.best_thumb = 'yes' and gs.group_id = 0 AND gs.group_id NOT IN (SELECT group_id FROM rot_groups_data WHERE status != 1)
sql_attr_timestamp = date
sql_attr_uint = duration
sql_attr_uint = sponsor_id
sql_attr_float = total_ctr
sql_attr_uint = content_type
sql_attr_multi = uint categories from field;
sql_attr_multi = uint tag_ids from field;
}
index your_name_index
{
source = your_name_source
path = /your_path
docinfo = extern
morphology = stem_en
charset_type = utf-8
}
Sphinx Delta Config
When creating an index sphinx creates an index of the current state of the database and if there were any changes in the database after indexing, sphinx won't find new data because it was added after indexing.
The logical conclusion from this: we need to re-index after each change in the database, however the problem is that if the database is large enough, indexing can take a long time. And indexing is a resource-intensive process.
To solve this problem, delta index was invented - this is an index in which changes that occurred from the moment of full indexing to the moment of delta index creation will fall. Since a small amount of data is indexed, the operation is very fast.
When searching, sphinx will search both in the main index and in the delta index, and thus will see all changes. Creating a delta index can be done as often as you like.
So:
in the main part that you already have, you need to add after
sql_query_pre = SET NAMES utf8
line
sql_query_pre = UPDATE rot_settings SET value = (SELECT MAX(gallery_id) FROM rot_gallery_info) WHERE name = 'sphinx_max_gallery_id'
Now you need to add the creation of delta index
source delta : your_name_source
{
sql_query_pre = SET NAMES utf8
sql_query = SELECT gi.gallery_id, UNIX_TIMESTAMP(gi.activation_date) as date, \
\
gd.alt, gd.description, gi.duration, gi.sponsor_id, gs.total_ctr, gi.content_type, rot_sponsors.sponsor_name, rot_sponsors.sponsor_site,\
\
\
(SELECT group_concat(tag_id) FROM rot_gal2tag g2t \
WHERE g2t.gallery_id = gi.gallery_id) as tag_ids, \
\
# TAGS START you can remove this part up to TAG END \
# this part indexes tags, if you remove it - tags wont be indexed \
# ligther index = less memory usage and faster search \
(SELECT group_concat(tag_name) FROM rot_gal2tag g2t \
LEFT JOIN rot_tags as t on t.tag_id = g2t.tag_id \
WHERE g2t.gallery_id = gi.gallery_id) as tags, \
\
# TAGS END \
\
\
# GROUP START you can remove this part up to TAG END \
# this part indexes group names, if you remove it - it wont be indexed \
# ligther index = less memory usage and faster search \
(SELECT group_concat(name) FROM rot_groups \
WHERE rot_groups.id in (SELECT group_id FROM rot_gallery_stats1 \
WHERE rot_gallery_stats1.gallery_id = gi.gallery_id AND group_id != 0) \
) as group_names, \
\
\
# GROUP END \
\
# Models START you can remove this part up to MODEL END \
# this part indexes model names, if you remove it - it wont be indexed \
# ligther index = less memory usage and faster search \
(SELECT group_concat(model_name) FROM rot_models \
WHERE rot_models.model_id in (SELECT model_id FROM rot_gal2model \
WHERE rot_gal2model.gallery_id = gi.gallery_id) \
) as model_names, \
\
# MODEL END \
\
\
(SELECT group_concat(distinct(gss.group_id)) FROM rot_gallery_stats1 as gss \
WHERE gss.gallery_id = gi.gallery_id AND group_id != 0) as categories \
\
FROM rot_gallery_info AS gi \
JOIN rot_sponsors on rot_sponsors.sponsor_id = gi.sponsor_id \
JOIN rot_gallery_data1 AS gd ON gi.gallery_id = gd.gallery_id \
\
\
JOIN rot_gallery_stats1 AS gs ON gs.gallery_id = gi.gallery_id \
WHERE gallery_status = 'active' and gallery_type = 0 \
and gs.best_thumb = 'yes' and gs.group_id = 0 \
AND gs.group_id NOT IN (SELECT group_id FROM rot_groups_data WHERE status != 1) and gi.gallery_id > ( SELECT value FROM rot_settings WHERE name = 'sphinx_max_gallery_id' )
}
index delta : your_name_index
{
source = delta
path = /your_full_path/data/delta
}
As you can see the query is the same, the only difference is the presence of sphinx_max_gallery_id in the selection.
After this you need to add indexing of new records to the cron as often as you like with the line
indexer --rotate delta
After creating the index, you need to specify delta in the Sphinx Delta Index settings of the rotation.
and once a day for example you can attach the delta index to the main one
indexer --rotate --merge your_name_index delta
Done.