import/export

diszk
Posts: 63
Joined: Tue Jan 17, 2012 12:07 am

import/export

Post by diszk »

Hello,

some other questions popped up in my mind:

are there any ways to import/export galleries thru command line (export galleries to file, rsync it and import from file)

also are there any ways to completely disable statistics for the master? its not a live site, just a container so I think there is no need to process stats

if I disable bookmark/ip/browser/etc processing at "Processed data" page, could this increase the performance?

many thanks in advance!
admin
Site Admin
Posts: 37202
Joined: Wed Sep 10, 2008 11:43 am

Re: import/export

Post by admin »

Hi !

Stats for master - if you don't add galleries there you can even switch off crontab jobs

export - create a regular template , name it 'test' for example

Code: Select all

<thumb num=1-100>
<!--URL-->|<!--DESCRIPTION-->
</thumb>
and you can add

master/scj/tube/?force_template=test&order=date

as an import set at another site
Don't forget to run script update
diszk
Posts: 63
Joined: Tue Jan 17, 2012 12:07 am

Re: import/export

Post by diszk »

Thank you, my problem with this way is that your app will process the already processed and exported galleries once again and as I see, its a one by one thing. So there is no way to import lets say 10k galleries in one batch?

And what if I add 10k galleries with the method you described but with some passive status (and set hotlink images which are on my cdn server actually so no processing needed) and then I just mass edit them and set the status to active?
admin
Site Admin
Posts: 37202
Joined: Wed Sep 10, 2008 11:43 am

Re: import/export

Post by admin »

First of all - what's the point to export\import the same content , why don't you use master\slave?
Don't forget to run script update
diszk
Posts: 63
Joined: Tue Jan 17, 2012 12:07 am

Re: import/export

Post by diszk »

Hello, sorry for superlong reply time, so as usual, I'm over-complicating everything, but honestly, just playing so if you have to do anything serious just ignore me entirely:).

So my "problem" was that even if I set the grabber at master site to grab 1 gallery per run, something locked up rot_gallery_info table for 20+ minutes and that caused a "Cannot connect to database..." error (I'm not sure about this, I'm low-skilled at mysql but saw the queries with mytop) and also even if it everything would went smooth, I could only grab 1-2-3-4 galleries per run because of black filtering, face detection, imagemagick processing and uploading to my cdn and also 25 thumbs per galleries so its a bit heavy.

So I deployed 3x5$ vps as a "grabber cluster" and deployed a "supervisor" vps which will decide which grabber instance should process the new galleries (didn't decide the way, but maybe it could check the load of the grabber instance, or querying the database for actual count of "to_grab" status ). Once one grabbed gallery count reached lets say 10k at an instance then I export them at /admin/?operation=rotation&sub=rot_list& as a post request as you told me earlier and when export done, I set those galleries to "to_delete" status.

After that, daily 1-2 times, all the exported galleries in files get merged into one and get transferred to the master instance server and then import them as hotlinked galleries.

The import/export automatization isn't done yet, but if I can do it manually, then I could do it by post requests as well and the results are daily 9k-11k grabbed galleries which is pretty awesome in my eyes with a ~20$ config(this success is because of your optimized app, not because of me, I'm not trying to act as a smartass).
And also I can't see the lock table "problem" with this way (which means nothing as I don't know the inner processes and such also maybe this lock table thing doesn't have any correlation with the grabber)

but there are some things:

1) when I want to delete the grabbed and exported galleries from the grabber cluster, I have to change the thumb save method to local, so the thumbs won't get deleted from my cdn but until the deletion process running, I cannot grab new galleries so are there any ways to speed up the deletion process? as I see its deleting 100 galleries per run...or maybe deleting galleries in real time without cron?

2) in the far future when you really bored, could you please add a new field to /admin/?operation=rotation&sub=rot_list& export place? a new field would be "group custom var 1" or "group custom name" because my group names have the first letter uppercase and the custom var/custom name are the lowercase words

Also any thoughts about this? besides that I'm and idiot because I'm complicating everything for basically nothing serious:) Many thanks even for reading this!
admin
Site Admin
Posts: 37202
Joined: Wed Sep 10, 2008 11:43 am

Re: import/export

Post by admin »

I must say you've done a lot to setup a config like this one )
But what do you think if we spent the same time investigating 'DB lock issue' ?
Don't forget to run script update
diszk
Posts: 63
Joined: Tue Jan 17, 2012 12:07 am

Re: import/export

Post by diszk »

Then the problem would be solved but still couldn't grab too much galleries per day, so the general reason for this config was the speed of the grabber. Anyway if you have any idea how could I check the root of this lock table thing, I would be glad. Also this issue is popped up only after the sphinx update, before that I could grab that 1-2-3 galleries without even see the lock tables query at mytop, but I don't know if this matters or not
admin
Site Admin
Posts: 37202
Joined: Wed Sep 10, 2008 11:43 am

Re: import/export

Post by admin »

Did you check hdd load ?
Don't forget to run script update
diszk
Posts: 63
Joined: Tue Jan 17, 2012 12:07 am

Re: import/export

Post by diszk »

yes, iotop is constantly running and not much load but there is an actual lock running, the mytop output: (queries are chunked because of the length). The disk read is approx 300K and write is mostly 0 but for some seconds it reaches 40M but its a ssd and can do 100M+

time query
545 Sleep
545 Query Sending select tag_id as id, tag_name as name, ( SELECT group_concat(gs.thumb_id order by tested_thumb DESC, gs.total_ctr DESC) FROM rot_gallery
485 Query Waiting LOCK TABLES rot_gallery_info WRITE
425 Sleep
425 Query Waiting SELECT gs.thumb_id, gi.sponsor_id, gi.url, gi.crop_profile_id, gi.content_count, gt.thumb_url, gs.total_shows, gs.total_ctr FROM rot_gal
314 Sleep
314 Query Waiting SELECT gs.thumb_id, gi.sponsor_id, gi.url, gi.crop_profile_id, gi.content_count, gt.thumb_url, gs.total_shows, gs.total_ctr FROM rot_gal
304 Query Waiting LOCK TABLES rot_gallery_info WRITE
Last edited by diszk on Sun Feb 04, 2018 2:21 pm, edited 1 time in total.
diszk
Posts: 63
Joined: Tue Jan 17, 2012 12:07 am

Re: import/export

Post by diszk »

maybe wrong mysql config, I'll try to calculate the ram usage etc again with the config calc and update you with the results
Post Reply