Grabbing speed up

filthlab
Posts: 118
Joined: Tue May 30, 2017 6:49 am

Grabbing speed up

Post by filthlab »

Hi,
How to find out which process takes most of the time from these lines:

2017-06-16 05:01:33: Processing http://www.drtuber.com/video/3369213/am ... uckdatecom (1359262) (19.372843027115, 0.10102701187134)
2017-06-16 05:01:33: Gallery description is empty: Update with 'admin added' (19.375967979431, 0.0031230449676514)
2017-06-16 05:01:33: Content type: 1 (19.383626937866, 0.0076560974121094)
2017-06-16 05:01:33: Creating thumb (320x180) (Crop profile: 1) (19.385671138763, 0.0020420551300049)
2017-06-16 05:01:33: Downloading img http://pics.drtuber.com/media/videos/tm ... ayer/6.jpg (../tmp/1359262/tmp//588645.jpg) (19.385740041733, 6.8187713623047E-5)
2017-06-16 05:01:33: Source Size 105219 (19.436568021774, 0.050843000411987)
2017-06-16 05:01:33: Make thumb ../tmp/1359262/tmp//588645.jpg (19.436862945557, 0.00027990341186523)
2017-06-16 05:01:33: Creating thumb ../tmp/1359262/tmp//72588.jpg.tmp.jpg -> ../tmp/1359262/tmp//72588.jpg (19.437283039093, 0.00041699409484863)
2017-06-16 05:01:33: Identify /usr/bin//identify -verbose ../tmp/1359262/tmp//72588.jpg.tmp.jpg (19.43748497963, 0.00019407272338867)
2017-06-16 05:01:33: Size: 960x540 (../tmp/1359262/tmp//72588.jpg.tmp.jpg) thumb: 320x180
thumb_ratio: 1.7777777777778 img_ratio: 1.7777777777778 to ../tmp/1359262/tmp//72588.jpg (19.483212947845, 0.045722007751465)
2017-06-16 05:01:33: Cut image (19.48334813118, 0.00013089179992676)
2017-06-16 05:01:33: Face detect Cmd : /home/domains//face --input=../tmp/1359262/tmp//72588.jpg.tmp.jpg --cascade=/home/domains/1.xml (19.483412027359, 6.1988830566406E-5)
2017-06-16 05:01:33: Output : sh: 1: /home/domains//face: not found
(19.487561941147, 0.0041580200195312)
2017-06-16 05:01:33: CMD: /usr/bin//convert -crop +35+0 +repage -crop -35-40 +repage -resize 320x180^ -gravity Center -crop 320x180+0+0 -quality 100 -filter Lanczos -normalize -unsharp 1x0.6+1 -unsharp 1x0.2+1 -enhance -modulate 100,102 ../tmp/1359262/tmp//72588.jpg.tmp.jpg ../tmp/1359262/tmp//72588.jpg (19.487744092941, 0.00017094612121582)
2017-06-16 05:01:33: Identify /usr/bin//identify -verbose ../tmp/1359262/tmp//72588.jpg (19.783210992813, 0.29547095298767)
2017-06-16 05:01:33: Detect img size for ../tmp/1359262/tmp//72588.jpg (320 x 180) (19.797949075699, 0.01473593711853)
2017-06-16 05:01:33: Rot save (2443969): title:Amateur blonde mommy and her toy Karissa from 1fuckdatecom, desc:() (19.80765914917, 0.0097041130065918)
2017-06-16 05:01:33: Saving ../thumbs/2/443//969_mommy_her_from.jpg (source ../tmp/1359262/tmp//72588.jpg) (19.807858943939, 0.00019693374633789)
2017-06-16 05:01:33: Cleanup tmp folder ../tmp/1359262 (19.808213949203, 0.00035285949707031)

What does it mean these numbers "(19.808213949203, 0.00035285949707031)" to end of the every line?
Any idea how can I speed up the grabber process according to the lines above?
admin
Site Admin
Posts: 37202
Joined: Wed Sep 10, 2008 11:43 am

Re: Grabbing speed up

Post by admin »

1. speed

2017-06-16 05:01:33: Processing http://www.drtuber.com/video/3369213/am ...
2017-06-16 05:01:33: Cleanup tmp folder ../tmp/1359262

less than a second, how fast do you think it should be?

2. (19.808213949203, 0.00035285949707031) 19 seconds - total process runnings time, 0.00035 since previous record

3. How it happens that you port to a russian part ? I move your topics to the english part everytime
Don't forget to run script update
filthlab
Posts: 118
Joined: Tue May 30, 2017 6:49 am

Re: Grabbing speed up

Post by filthlab »

1. According to the log it grabs more than 1 gallery per second, probably about 1.5 per second. So, it should be about 130k per 24h. But now I'm getting about 60-70k per 24h, which is strange a bit for me.

2.
since previous record
What is the record, and is this number in seconds?
I have some lines with "6.1988830566406E-5", is this means 6 seconds.

Btw, didn't find where to turn off Face detect, I don't need it.

3. I'm very sorry for this. Didn't noticed I'm on Russian language. It is fixed now. Sorry again for the inconvenience.

Also, I'm sorry if my questions seems to you stupid, but I'm a newbie with your script, and I'm just trying to figure out how is everything working.
admin
Site Admin
Posts: 37202
Joined: Wed Sep 10, 2008 11:43 am

Re: Grabbing speed up

Post by admin »

1. you can check the entire log for a day, may be remote host was slow at some point for example
I have some lines with "6.1988830566406E-5", is this means 6 seconds.
E-5 means "5 digits after 0.
Btw, didn't find where to turn off Face detect, I don't need it.
crop profile - edit

NP, feel free to ask any questions here
Don't forget to run script update
filthlab
Posts: 118
Joined: Tue May 30, 2017 6:49 am

Re: Grabbing speed up

Post by filthlab »

crop profile - edit
I saw this before but there is just "Face Detect Path". I should just remove the path and this option will be off?

Btw, is there an option to keep the log files smaller? Now gallery_grabber.log is about 2GB, cj.log about 3GB, and rotation.log - 2GB. And all they rising.
admin
Site Admin
Posts: 37202
Joined: Wed Sep 10, 2008 11:43 am

Re: Grabbing speed up

Post by admin »

Face detect - yes, just delete that line

Log size - should be within

Log files max length, lines

if more then that - check if logs for 'rotate logs' , it runs once a day.
Don't forget to run script update
filthlab
Posts: 118
Joined: Tue May 30, 2017 6:49 am

Re: Grabbing speed up

Post by filthlab »

Log files max length, lines: I have 1000
check if logs for 'rotate logs' , it runs once a day
Not sure how to check this.
admin
Site Admin
Posts: 37202
Joined: Wed Sep 10, 2008 11:43 am

Re: Grabbing speed up

Post by admin »

search for

rotate logs

in cron.log
Don't forget to run script update
filthlab
Posts: 118
Joined: Tue May 30, 2017 6:49 am

Re: Grabbing speed up

Post by filthlab »

I have it just twice on the day the script has been installed. Is there something missing in the cron tab?
admin
Site Admin
Posts: 37202
Joined: Wed Sep 10, 2008 11:43 am

Re: Grabbing speed up

Post by admin »

May be
how often do you run cron.php ?
Don't forget to run script update
Post Reply