Gallery Grabber Threads

burn
Posts: 262
Joined: Wed Feb 01, 2017 11:27 am

Gallery Grabber Threads

Post by burn »

День добрый.
На новых сиджах медленно стали грабиться галеры.
Те же дампы на старых намного быстрее. Старые на этом же сервере.
Грабится примерно по 3-4.
Выставляю Gallery Grabber Threads в 100-200 - ничего не меняется и по логам и в "Current average speed" пишет 3-4.
Сервер не нагружен: ЛА 10, 32 ядра
Крон раз в 5 мин.

Code: Select all

2021-06-05 17:32:00: Processing https://www.pornhub.com/view_video.php?viewkey=1661419764 (176403) (0.011178016662598, 0.0098631381988525)
2021-06-05 17:32:05: Set gallery status to active  (5.4431891441345, 0.010816097259521)2021-06-05 17:32:05: processed_galleries = 2, memory = 26.6 Mb limit 512M (5.4459660053253, 0.0027689933776855)
2021-06-05 17:32:05: Processing https://www.pornhub.com/view_video.php?viewkey=ph5ac1f3be9232f (176404) (5.4515891075134, 0.0056159496307373)
2021-06-05 17:32:10: Set gallery status to active  (9.8466761112213, 0.0017690658569336)2021-06-05 17:32:10: processed_galleries = 3, memory = 27.6 Mb limit 512M (9.8475029468536, 0.00082492828369141)
2021-06-05 17:32:10: Processing https://www.pornhub.com/view_video.php?viewkey=ph5826ceb0f2b77 (176405) (9.8497030735016, 0.0021970272064209)
2021-06-05 17:32:13: Set gallery status to active  (12.871883153915, 0.0018618106842041)2021-06-05 17:37:00: processed_galleries = 1, memory = 4 Mb limit 512M (0.00094008445739746, 0.00094509124755859)
2021-06-05 17:37:00: Processing https://www.pornhub.com/view_video.php?viewkey=ph5bad263f27ffb (176407) (0.010069131851196, 0.0091249942779541)
2021-06-05 17:37:01: Processing https://www.pornhub.com/view_video.php?viewkey=ph566562ccc4c85 (176408) (1.4751770496368, 0.0034630298614502)
2021-06-05 17:37:06: Set gallery status to active  (6.3935132026672, 0.0022909641265869)2021-06-05 17:37:06: processed_galleries = 3, memory = 26.8 Mb limit 512M (6.3946180343628, 0.0010988712310791)
2021-06-05 17:37:06: Processing https://www.pornhub.com/view_video.php?viewkey=165689273 (176409) (6.3983981609344, 0.003777027130127)
2021-06-05 17:37:10: Set gallery status to active  (10.439781188965, 0.0017569065093994)2021-06-05 17:37:10: processed_galleries = 4, memory = 26.8 Mb limit 512M (10.440680027008, 0.00089597702026367)
2021-06-05 17:37:10: Processing https://www.pornhub.com/view_video.php?viewkey=ph5a533e740d918 (176410) (10.443266153336, 0.0025839805603027)
2021-06-05 17:37:12: Processing https://www.pornhub.com/view_video.php?viewkey=ph56dded9614a6c (176411) (11.791800022125, 0.0027260780334473)
2021-06-05 17:37:16: Set gallery status to active  (16.002367973328, 0.0023131370544434)2021-06-05 17:37:16: processed_galleries = 6, memory = 26.8 Mb limit 512M (16.003540992737, 0.001168966293335)
2021-06-05 17:37:16: Processing https://www.pornhub.com/view_video.php?viewkey=ph59496bf0d1c5b (176412) (16.006613016129, 0.0030701160430908)
2021-06-05 17:37:19: Set gallery status to active  (19.030066013336, 0.0049529075622559)2021-06-05 17:37:19: processed_galleries = 7, memory = 19.5 Mb limit 512M (19.031466007233, 0.0013968944549561)
2021-06-05 17:37:19: Processing https://www.pornhub.com/view_video.php?viewkey=ph5b6c0ff728f4a (176413) (19.03439116478, 0.0029220581054688)
2021-06-05 17:37:23: Set gallery status to active  (23.03404712677, 0.002187967300415)2021-06-05 17:42:00: processed_galleries = 1, memory = 4 Mb limit 512M (0.001011848449707, 0.0010180473327637)
2021-06-05 17:42:00: Processing https://www.pornhub.com/view_video.php?viewkey=107313928 (176415) (0.010648965835571, 0.0096371173858643)
2021-06-05 17:42:05: Set gallery status to active  (5.48286485672, 0.0053369998931885)2021-06-05 17:47:00: processed_galleries = 1, memory = 4 Mb limit 512M (0.001276969909668, 0.001284122467041)
2021-06-05 17:47:00: Processing https://www.pornhub.com/view_video.php?viewkey=ph567c7c556bac0 (176417) (0.012966156005859, 0.011688947677612)
2021-06-05 17:47:06: Set gallery status to active  (6.1631669998169, 0.0080680847167969)2021-06-05 17:52:00: processed_galleries = 1, memory = 4 Mb limit 512M (0.0010359287261963, 0.0010418891906738)
2021-06-05 17:52:00: Processing https://www.pornhub.com/view_video.php?viewkey=ph5a5b1e4da627c (176419) (0.010966062545776, 0.0099270343780518)
2021-06-05 17:52:06: Set gallery status to active  (5.702693939209, 0.0035080909729004)2021-06-05 17:52:06: processed_galleries = 2, memory = 26.6 Mb limit 512M (5.7039148807526, 0.0012130737304688)
2021-06-05 17:52:06: Processing https://www.pornhub.com/view_video.php?viewkey=185982585 (176420) (5.7074229717255, 0.0035068988800049)
2021-06-05 17:52:10: Set gallery status to active  (9.7432799339294, 0.0018038749694824)2021-06-05 17:57:00: processed_galleries = 1, memory = 4 Mb limit 512M (0.0010690689086914, 0.0010750293731689)
2021-06-05 17:57:00: Processing https://www.pornhub.com/view_video.php?viewkey=1247813958 (176422) (0.010745048522949, 0.009674072265625)
2021-06-05 17:57:05: Set gallery status to active  (5.2571499347687, 0.0061030387878418)2021-06-05 18:12:00: processed_galleries = 1, memory = 4 Mb limit 512M (0.0010771751403809, 0.0010831356048584)
2021-06-05 18:12:00: Processing https://www.pornhub.com/view_video.php?viewkey=ph5a59179ad929b (176426) (0.010696172714233, 0.0096151828765869)
2021-06-05 18:12:06: Set gallery status to active  (5.2794940471649, 0.0047140121459961)2021-06-05 18:22:00: processed_galleries = 1, memory = 4 Mb limit 512M (0.0011310577392578, 0.0011389255523682)
2021-06-05 18:22:00: Processing https://www.pornhub.com/view_video.php?viewkey=748228653 (176429) (0.011747121810913, 0.010611057281494)
2021-06-05 18:22:05: Set gallery status to active  (5.422110080719, 0.0063021183013916)2021-06-05 18:22:05: processed_galleries = 2, memory = 26.9 Mb limit 512M (5.4259519577026, 0.0038361549377441)
Attachments
Снимок экрана от 2021-06-05 18-30-53.png
Снимок экрана от 2021-06-05 18-30-53.png (28.02 KiB) Viewed 1085 times
admin
Site Admin
Posts: 37202
Joined: Wed Sep 10, 2008 11:43 am

Re: Gallery Grabber Threads

Post by admin »

2021-06-05 17:37:01: Processing https://www.pornhub.com/view_video.php? ... 562ccc4c85 (176408) (1.4751770496368, 0.0034630298614502)
2021-06-05 17:37:06: Set gallery status to active (6.3935132026672, 0.0022909641265869)2021-06-05 17:37:06: processed_galleries = 3, memory = 26.8 Mb limit 512M (6.3946180343628, 0.0010988712310791)


вот 2 строки на галеру и между ниси ничего?
Don't forget to run script update
burn
Posts: 262
Joined: Wed Feb 01, 2017 11:27 am

Re: Gallery Grabber Threads

Post by burn »

Есть конечно, так вывел чтоб нагляднее было.

Code: Select all

2021-06-05 17:37:01: Processing https://www.pornhub.com/view_video.php?viewkey=ph566562ccc4c85 (176408) (1.4751770496368, 0.0034630298614502)
2021-06-05 17:37:01: Loading gallery https://www.pornhub.com/view_video.php?viewkey=ph566562ccc4c85 (1.4785120487213, 0.0033299922943115)
2021-06-05 17:37:06: Gallery description is empty: Update  with 'Add admin'  (5.9697251319885, 4.4912099838257)
2021-06-05 17:37:06: Downloading files for a custom gallery (5.9759380817413, 0.0062038898468018)
2021-06-05 17:37:06: Content type: 4 (5.9764111042023, 0.00046896934509277)
2021-06-05 17:37:06: Creating thumb from http://www.domain.pro/api/v1/pictures/thumb/300x250/data__parse__644__files__acfd38f26f89d88aa193364670edb1c0__desi-teen-couple-having-fun-in-bf-room_0.jpg  (400x0) (Crop profile: 1)  (5.9771749973297, 0.00076198577880859)
2021-06-05 17:37:06: Downloading img http://www.domain.pro/api/v1/pictures/thumb/300x250/data__parse__644__files__acfd38f26f89d88aa193364670edb1c0__desi-teen-couple-having-fun-in-bf-room_0.jpg to ../tmp/176408/tmp//230095.jpg, ref https://www.pornhub.com/view_video.php?viewkey=ph566562ccc4c85  (5.9774701595306, 0.00029397010803223)
2021-06-05 17:37:06: Source Size: 13200 (6.2572782039642, 0.27982187271118)
2021-06-05 17:37:06: Dupe check   (6.2574701309204, 0.00016093254089355)
2021-06-05 17:37:06: Downloading thumb http://www.domain.pro/api/v1/pictures/thumb/300x250/data__parse__644__files__acfd38f26f89d88aa193364670edb1c0__desi-teen-couple-having-fun-in-bf-room_0.jpg and make a copy (6.2580420970917, 0.00056695938110352)
2021-06-05 17:37:06: Identify /usr/bin//identify -verbose ../tmp/176408/tmp//230095.jpg (6.2797889709473, 0.021749973297119)
2021-06-05 17:37:06: Identify /usr/bin//identify -verbose ../tmp/176408/tmp//230095.jpg.thumb.jpg (6.3317470550537, 0.051965951919556)
2021-06-05 17:37:06: Detect img size for ../tmp/176408/tmp//230095.jpg.thumb.jpg (300 x 250) (6.3862860202789, 0.054526090621948)
2021-06-05 17:37:06: Saving thumb (346229): title:desi teen couple having fun in bf room, desc:, function rotation_get_thumb_path (force name: ) to folder 346/ filename 229_teen-having-in (6.3887569904327, 0.0024490356445312)
2021-06-05 17:37:06: Saving location: local to folder ../thumbs//346/  (6.3889579772949, 0.00019598007202148)
2021-06-05 17:37:06: save_name /229_teen-having-in.jpg (6.3890750408173, 0.00011181831359863)
2021-06-05 17:37:06: Saving file ../tmp/176408/tmp//230095.jpg.thumb.jpg (13200 bytes) to ../thumbs//346///229_teen-having-in.jpg  (6.3891839981079, 0.00010514259338379)
2021-06-05 17:37:06: ../thumbs//346///229_teen-having-in.jpg saved (13200 bytes) (6.3894741535187, 0.00028586387634277)
2021-06-05 17:37:06: Update flv_thumb_url for this gallery (6.3899869918823, 0.00050711631774902)
2021-06-05 17:37:06: Cleanup tmp folder: ../tmp/176408 (6.3912160396576, 0.0012280941009521)
2021-06-05 17:37:06: Set gallery status to active  (6.3935132026672, 0.0022909641265869)2021-06-05 17:37:06: processed_galleries = 3, memory = 26.8 Mb limit 512M (6.3946180343628, 0.0010988712310791)
admin
Site Admin
Posts: 37202
Joined: Wed Sep 10, 2008 11:43 am

Re: Gallery Grabber Threads

Post by admin »

Тут вижу что все время ушло на загрузку галеры самой галеры
можете сами по времени проверить

те понятно что если уаленный сервак не будет отдавать ее быстро то и грабит быстро никак не сможет

какие еще примеры из лога ест ?
Don't forget to run script update
burn
Posts: 262
Joined: Wed Feb 01, 2017 11:27 am

Re: Gallery Grabber Threads

Post by burn »

Если качает 6 сек, то за минуту должно было 10 галер скачать, а за пять 50.
А качает по 3 и перестает.
admin
Site Admin
Posts: 37202
Joined: Wed Sep 10, 2008 11:43 am

Re: Gallery Grabber Threads

Post by admin »

тогда надо больше лога что б виеть что происходит, например вижу разрыв в 5 минут между записями, те можно крон стоит не каждую минуту а раз в 5 минут
Don't forget to run script update
burn
Posts: 262
Joined: Wed Feb 01, 2017 11:27 am

Re: Gallery Grabber Threads

Post by burn »

Да, крон ротатора раз в пять мин, писал об этом в первом посте.
admin
Site Admin
Posts: 37202
Joined: Wed Sep 10, 2008 11:43 am

Re: Gallery Grabber Threads

Post by admin »

да, сорри не заметил

а зачем раз в 5 минут если не нагружен?
Don't forget to run script update
burn
Posts: 262
Joined: Wed Feb 01, 2017 11:27 am

Re: Gallery Grabber Threads

Post by burn »

Да как то по дефолту так уже ставлю, со временем все равно нагрузка вырастит.
admin
Site Admin
Posts: 37202
Joined: Wed Sep 10, 2008 11:43 am

Re: Gallery Grabber Threads

Post by admin »

так может поменять назад когда вырастет
Don't forget to run script update
Post Reply