Big Base Demo

admin
Site Admin
Posts: 37202
Joined: Wed Sep 10, 2008 11:43 am

Re: Big Base Demo

Post by admin »

добавил в ап что б не считало
Don't forget to run script update
darkover
Posts: 1518
Joined: Sat Apr 13, 2013 8:59 pm

Re: Big Base Demo

Post by darkover »

Привет!
Пишу в этом треде. т.к. всеж это касается в большей степени именно большой базы под не совсем стандартное решение - база всех тубов для наполнения сиджев/тубов, писать свое было лень, а в смарте все это уже есть: импорт/экспорт, категоризация, модели и сфинкс и работа с большими дампами. Сейчас работает такой агрегатор в котором 30 лямов галер (Your Build 20220826131).

Но я решил немного переделать, и дабы не трогать то, что работает и настроено уже год с лишним, запилил на другом сервере тож агрегатор (Your Build 202401111327), дабы в последствии перейти на него.

Столкнулся я вот с чем:
1) Это попадание дюпов галер в rot_gallery_info и соответственно в другие таблицы.
Выяснил я это решив просто посмотреть уникальность в rot_gallery_info

Code: Select all

SELECT url, count(url)
FROM rot_gallery_info
GROUP BY url
HAVING count(url)>1;

Отображение строк 0 - 24 (1183 всего, Запрос занял 5.3906 сек.)
1183 дюпа на 700к галер.

Code: Select all

В настройках ротации :
Check for dupe description - no
Check for dupe thumbs - no (тумбы не создаются в импорт сете, там указан hotlink )

В настройках import set:
Check DB for Dupes	- on
Copy Thumbs to server - hotlink thumbs

Кодировка в базе rot_gallery_info
url - utf8mb3_bin
source_url - utf8mb3_bin
Вот логи по одной галерке, которая аж за троилась:

В дампе одна строка с этой галерой.

Code: Select all

rotation.log

2024-01-13 23:44:37: Importing gallery line: https://drtvid.com/v/MdAB|https://drtvid.com/e/MdAB|Amateur gives husband blowjob live on cam| поскипал  (1115.9122030735, 2.5033950805664E-5)
2024-01-13 23:44:37: Importing gallery line: https://drtvid.com/v/MdAB|https://drtvid.com/e/MdAB|Amateur gives husband blowjob live on cam| поскипал  (516.44523286819, 3.6001205444336E-5)
2024-01-13 23:44:37: Importing gallery line: https://drtvid.com/v/MdAB|https://drtvid.com/e/MdAB|Amateur gives husband blowjob live on cam| поскипал  (156.18445515633, 3.504753112793E-5)
2024-01-13 23:44:37: Set tags for gallery_id 76166  (156.18935918808, 0.0049021244049072)
2024-01-13 23:44:37: Set tags for gallery_id 76165  (1115.9187269211, 0.0065250396728516)
2024-01-13 23:44:37: Set tags for gallery_id 76167  (1655.9667329788, 0.0054769515991211)
2024-01-13 23:44:37: Set tags for gallery_id 76168  (516.45140504837, 0.0061700344085693)
2024-01-13 23:44:37: Import result: OK (1536.7332921028, 0.37010502815247)

Code: Select all


   Отображение строк 0 -  2 (3 всего, Запрос занял 0.0002 сек.)

SELECT *  FROM `rot_gallery_info` WHERE `url` = 'https://drtvid.com/v/MdAB'


gallery_id	gallery_type	url	source_url	sponsor_id	content_count	поскипал 
76165	0	https://drtvid.com/v/MdAB	https://drtvid.com/v/MdAB	17	10000	поскипал 
76166	0	https://drtvid.com/v/MdAB	https://drtvid.com/v/MdAB	17	10000	поскипал
76168	0	https://drtvid.com/v/MdAB	https://drtvid.com/v/MdAB	17	10000	поскипал

Code: Select all

gallery_grabber.log
2024-01-13 23:45:37: Processing https://drtvid.com/v/MdAB (76165) (28.684059143066, 0.009537935256958)
2024-01-13 23:45:37: Gallery description is empty: Update  with 'empty'  (28.68532705307, 0.0012691020965576)
2024-01-13 23:45:37: Set tags for gallery_id 76165  (28.686778068542, 0.0014479160308838)
2024-01-13 23:45:37: Content type: 4 (28.786545038223, 0.099764823913574)
2024-01-13 23:45:37: Creating thumb from https://pics.drtuber.com/media/videos/tmb/9069635/preview/11.jpg  (240x180) (Crop profile: 1)  (28.787038087845, 0.00049281120300293)
2024-01-13 23:45:37: File ext jpg (28.787060022354, 1.978874206543E-5)
2024-01-13 23:45:37: Skip download as it's a hotlink and we don't need to check for dupes (28.787068128586, 8.1062316894531E-6)
2024-01-13 23:45:37: Creating thumb from https://pics.drtuber.com/media/videos/tmb/9069635/preview/1.jpg  (240x180) (Crop profile: 1)  (28.787082195282, 1.5020370483398E-5)
2024-01-13 23:45:37: File ext jpg (28.787099123001, 1.5974044799805E-5)
2024-01-13 23:45:37: Skip download as it's a hotlink and we don't need to check for dupes (28.787110090256, 9.0599060058594E-6)
2024-01-13 23:45:37: Creating thumb from https://pics.drtuber.com/media/videos/tmb/9069635/preview/2.jpg  (240x180) (Crop profile: 1)  (28.787123203278, 1.215934753418E-5)
2024-01-13 23:45:37: File ext jpg (28.787137031555, 1.2874603271484E-5)
2024-01-13 23:45:37: Skip download as it's a hotlink and we don't need to check for dupes (28.787145137787, 7.8678131103516E-6)
2024-01-13 23:45:37: Creating thumb from https://pics.drtuber.com/media/videos/tmb/9069635/preview/3.jpg  (240x180) (Crop profile: 1)  (28.78715801239, 1.215934753418E-5)
2024-01-13 23:45:37: File ext jpg (28.787173032761, 1.2874603271484E-5)
2024-01-13 23:45:37: Skip download as it's a hotlink and we don't need to check for dupes (28.787182092667, 7.8678131103516E-6)
2024-01-13 23:45:37: Creating thumb from https://pics.drtuber.com/media/videos/tmb/9069635/preview/4.jpg  (240x180) (Crop profile: 1)  (28.787196159363, 1.2874603271484E-5)
2024-01-13 23:45:37: File ext jpg (28.787212133408, 1.4066696166992E-5)
2024-01-13 23:45:37: Skip download as it's a hotlink and we don't need to check for dupes (28.787222146988, 8.1062316894531E-6)
2024-01-13 23:45:37: Creating thumb from https://pics.drtuber.com/media/videos/tmb/9069635/preview/5.jpg  (240x180) (Crop profile: 1)  (28.787234067917, 1.0967254638672E-5)
2024-01-13 23:45:37: File ext jpg (28.787247180939, 1.215934753418E-5)
2024-01-13 23:45:37: Skip download as it's a hotlink and we don't need to check for dupes (28.787256002426, 7.1525573730469E-6)
2024-01-13 23:45:37: insert into rot_thumbs  (28.787271976471, 1.5974044799805E-5)
2024-01-13 23:45:37: insert into rot_gallery_stats (28.787425994873, 0.0001530647277832)
2024-01-13 23:45:37: Saving thumb (76165): full_filepath: 1.jpg  (28.788200139999, 0.00077199935913086)
2024-01-13 23:45:37: Saving location: local to folder ../thumbs//user_thumbs//76/165  (28.788218021393, 1.6927719116211E-5)
2024-01-13 23:45:37: save_name 1. (28.788305044174, 8.7976455688477E-5)
2024-01-13 23:45:37: Saving  - not a file??? (28.788315057755, 7.8678131103516E-6)
2024-01-13 23:45:37: Saving thumb (76165): full_filepath: 2.jpg  (28.788326025009, 9.7751617431641E-6)
2024-01-13 23:45:37: Saving location: local to folder ../thumbs//user_thumbs//76/165  (28.788336038589, 7.8678131103516E-6)
2024-01-13 23:45:37: save_name 1. (28.788352012634, 1.6212463378906E-5)
2024-01-13 23:45:37: Saving  - not a file??? (28.788357019424, 4.7683715820312E-6)
2024-01-13 23:45:37: Saving thumb (76165): full_filepath: 3.jpg  (28.78836607933, 8.1062316894531E-6)
2024-01-13 23:45:37: Saving location: local to folder ../thumbs//user_thumbs//76/165  (28.788377046585, 1.215934753418E-5)
2024-01-13 23:45:37: save_name 1. (28.788410186768, 3.1948089599609E-5)
2024-01-13 23:45:37: Saving  - not a file??? (28.788421154022, 9.0599060058594E-6)
2024-01-13 23:45:37: Saving thumb (76165): full_filepath: 4.jpg  (28.788444042206, 2.1934509277344E-5)
2024-01-13 23:45:37: Saving location: local to folder ../thumbs//user_thumbs//76/165  (28.788453102112, 7.8678131103516E-6)
2024-01-13 23:45:37: save_name 1. (28.788469076157, 1.5020370483398E-5)
2024-01-13 23:45:37: Saving  - not a file??? (28.788473129272, 4.0531158447266E-6)
2024-01-13 23:45:37: Saving thumb (76165): full_filepath: 5.jpg  (28.788480997086, 7.1525573730469E-6)
2024-01-13 23:45:37: Saving location: local to folder ../thumbs//user_thumbs//76/165  (28.788488149643, 6.9141387939453E-6)
2024-01-13 23:45:37: save_name 1. (28.788501024246, 1.2874603271484E-5)
2024-01-13 23:45:37: Saving  - not a file??? (28.788520097733, 1.8835067749023E-5)
2024-01-13 23:45:37: Cleanup tmp folder: ../tmp/76165 (28.788624048233, 1.6927719116211E-5)
2024-01-13 23:45:37: Set gallery status to active  (28.79013299942, 0.001507043838501)2024-01-13 23:45:37: processed_galleries = 494, memory = 6.6 Mb limit 512M (59.408946990967, 0.00063610076904297)
2024-01-13 23:45:37: Processing https://drtvid.com/v/MdAB (76168) (59.440761089325, 0.045356035232544)
2024-01-13 23:45:37: Gallery description is empty: Update  with 'empty'  (59.44544005394, 0.0046792030334473)
2024-01-13 23:45:37: Set tags for gallery_id 76168  (59.446877002716, 0.0014338493347168)
2024-01-13 23:45:37: Content type: 4 (59.512028932571, 0.06515097618103)
2024-01-13 23:45:37: Creating thumb from https://pics.drtuber.com/media/videos/tmb/9069635/preview/11.jpg  (240x180) (Crop profile: 1)  (59.512516975403, 0.00048708915710449)
2024-01-13 23:45:37: File ext jpg (59.512547969818, 2.9087066650391E-5)
2024-01-13 23:45:37: Skip download as it's a hotlink and we don't need to check for dupes (59.512562036514, 1.1920928955078E-5)
2024-01-13 23:45:37: Creating thumb from https://pics.drtuber.com/media/videos/tmb/9069635/preview/1.jpg  (240x180) (Crop profile: 1)  (59.512581110001, 1.6927719116211E-5)
2024-01-13 23:45:37: File ext jpg (59.512597084045, 1.4066696166992E-5)
2024-01-13 23:45:37: Skip download as it's a hotlink and we don't need to check for dupes (59.512607097626, 9.0599060058594E-6)
2024-01-13 23:45:37: Creating thumb from https://pics.drtuber.com/media/videos/tmb/9069635/preview/2.jpg  (240x180) (Crop profile: 1)  (59.512620925903, 1.215934753418E-5)
2024-01-13 23:45:37: File ext jpg (59.512635946274, 1.2874603271484E-5)
2024-01-13 23:45:37: Skip download as it's a hotlink and we don't need to check for dupes (59.51264500618, 8.8214874267578E-6)
2024-01-13 23:45:37: Creating thumb from https://pics.drtuber.com/media/videos/tmb/9069635/preview/3.jpg  (240x180) (Crop profile: 1)  (59.512658119202, 1.1920928955078E-5)
2024-01-13 23:45:37: File ext jpg (59.512671947479, 1.3113021850586E-5)
2024-01-13 23:45:37: Skip download as it's a hotlink and we don't need to check for dupes (59.51268196106, 9.0599060058594E-6)
2024-01-13 23:45:37: Creating thumb from https://pics.drtuber.com/media/videos/tmb/9069635/preview/4.jpg  (240x180) (Crop profile: 1)  (59.512696027756, 1.1920928955078E-5)
2024-01-13 23:45:37: File ext jpg (59.512710094452, 1.3113021850586E-5)
2024-01-13 23:45:37: Skip download as it's a hotlink and we don't need to check for dupes (59.512718915939, 8.1062316894531E-6)
2024-01-13 23:45:37: Creating thumb from https://pics.drtuber.com/media/videos/tmb/9069635/preview/5.jpg  (240x180) (Crop profile: 1)  (59.512732028961, 1.2874603271484E-5)
2024-01-13 23:45:37: File ext jpg (59.512746095657, 1.1920928955078E-5)
2024-01-13 23:45:37: Skip download as it's a hotlink and we don't need to check for dupes (59.512754917145, 8.1062316894531E-6)
2024-01-13 23:45:37: insert into rot_thumbs  (59.512772083282, 1.5974044799805E-5)
2024-01-13 23:45:37: insert into rot_gallery_stats (59.512963056564, 0.00019001960754395)
2024-01-13 23:45:37: Saving thumb (76168): full_filepath: 1.jpg  (59.513631105423, 0.00066709518432617)
2024-01-13 23:45:37: Saving location: local to folder ../thumbs//user_thumbs//76/168  (59.513664007187, 3.1948089599609E-5)
2024-01-13 23:45:37: save_name 1. (59.513756036758, 8.9883804321289E-5)
2024-01-13 23:45:37: Saving  - not a file??? (59.513767004013, 8.8214874267578E-6)
2024-01-13 23:45:37: Saving thumb (76168): full_filepath: 2.jpg  (59.513780117035, 1.215934753418E-5)
2024-01-13 23:45:37: Saving location: local to folder ../thumbs//user_thumbs//76/168  (59.513792037964, 1.0967254638672E-5)
2024-01-13 23:45:37: save_name 1. (59.513808965683, 1.6927719116211E-5)
2024-01-13 23:45:37: Saving  - not a file??? (59.513814926147, 5.0067901611328E-6)
2024-01-13 23:45:37: Saving thumb (76168): full_filepath: 3.jpg  (59.513824939728, 1.0013580322266E-5)
2024-01-13 23:45:37: Saving location: local to folder ../thumbs//user_thumbs//76/168  (59.513833999634, 7.8678131103516E-6)
2024-01-13 23:45:37: save_name 1. (59.513849020004, 1.4066696166992E-5)
2024-01-13 23:45:37: Saving  - not a file??? (59.513854026794, 4.0531158447266E-6)
2024-01-13 23:45:37: Saving thumb (76168): full_filepath: 4.jpg  (59.513862133026, 8.1062316894531E-6)
2024-01-13 23:45:37: Saving location: local to folder ../thumbs//user_thumbs//76/168  (59.513869047165, 6.9141387939453E-6)
2024-01-13 23:45:37: save_name 1. (59.513884067535, 1.3113021850586E-5)
2024-01-13 23:45:37: Saving  - not a file??? (59.513889074326, 4.0531158447266E-6)
2024-01-13 23:45:37: Saving thumb (76168): full_filepath: 5.jpg  (59.513896942139, 6.9141387939453E-6)
2024-01-13 23:45:37: Saving location: local to folder ../thumbs//user_thumbs//76/168  (59.513904094696, 6.9141387939453E-6)
2024-01-13 23:45:37: save_name 1. (59.513917922974, 1.4066696166992E-5)
2024-01-13 23:45:37: Saving  - not a file??? (59.513922929764, 5.0067901611328E-6)
2024-01-13 23:45:37: Cleanup tmp folder: ../tmp/76168 (59.513930082321, 6.9141387939453E-6)
2024-01-13 23:45:37: Set gallery status to active  (59.515235900879, 0.0013070106506348)2024-01-13 23:45:37: processed_galleries = 491, memory = 6.6 Mb limit 512M (59.443475008011, 0.00038599967956543)
2024-01-13 23:45:37: Processing https://drtvid.com/v/LWfl (76173) (59.467054128647, 0.023580074310303)
2024-01-13 23:45:37: Gallery description is empty: Update  with 'empty'  (59.468927145004, 0.0018730163574219)
2024-01-13 23:45:37: Set tags for gallery_id 76173  (59.470293998718, 0.0013628005981445)
2024-01-13 23:45:37: Content type: 4 (59.534504175186, 0.064209938049316)
2024-01-13 23:45:37: Creating thumb from https://pics.drtuber.com/media/videos/tmb/9002969/preview/3.jpg  (240x180) (Crop profile: 1)  (59.534885168076, 0.00038003921508789)
2024-01-13 23:45:37: File ext jpg (59.534903049469, 1.6927719116211E-5)
2024-01-13 23:45:37: Skip download as it's a hotlink and we don't need to check for dupes (59.534911155701, 8.1062316894531E-6)
2024-01-13 23:45:37: Creating thumb from https://pics.drtuber.com/media/videos/tmb/9002969/preview/1.jpg  (240x180) (Crop profile: 1)  (59.534921169281, 1.0013580322266E-5)
2024-01-13 23:45:37: File ext jpg (59.534929990768, 8.8214874267578E-6)
2024-01-13 23:45:37: Skip download as it's a hotlink and we don't need to check for dupes (59.534934997559, 5.0067901611328E-6)
2024-01-13 23:45:37: Creating thumb from https://pics.drtuber.com/media/videos/tmb/9002969/preview/2.jpg  (240x180) (Crop profile: 1)  (59.53494310379, 7.1525573730469E-6)
2024-01-13 23:45:37: File ext jpg (59.534950017929, 6.9141387939453E-6)
2024-01-13 23:45:37: Skip download as it's a hotlink and we don't need to check for dupes (59.534955024719, 3.814697265625E-6)
2024-01-13 23:45:37: Creating thumb from https://pics.drtuber.com/media/videos/tmb/9002969/preview/3.jpg  (240x180) (Crop profile: 1)  (59.534962177277, 5.9604644775391E-6)
2024-01-13 23:45:37: File ext jpg (59.53497004509, 6.9141387939453E-6)
2024-01-13 23:45:37: Skip download as it's a hotlink and we don't need to check for dupes (59.534979104996, 8.1062316894531E-6)
2024-01-13 23:45:37: Creating thumb from https://pics.drtuber.com/media/videos/tmb/9002969/preview/4.jpg  (240x180) (Crop profile: 1)  (59.534989118576, 8.1062316894531E-6)
2024-01-13 23:45:37: File ext jpg (59.534996032715, 6.9141387939453E-6)
2024-01-13 23:45:37: Skip download as it's a hotlink and we don't need to check for dupes (59.535001039505, 3.814697265625E-6)
2024-01-13 23:45:37: Creating thumb from https://pics.drtuber.com/media/videos/tmb/9002969/preview/5.jpg  (240x180) (Crop profile: 1)  (59.535007953644, 6.1988830566406E-6)
2024-01-13 23:45:37: File ext jpg (59.535015106201, 6.9141387939453E-6)
2024-01-13 23:45:37: Skip download as it's a hotlink and we don't need to check for dupes (59.535020112991, 5.0067901611328E-6)
2024-01-13 23:45:37: insert into rot_thumbs  (59.535029172897, 9.0599060058594E-6)
2024-01-13 23:45:37: insert into rot_gallery_stats (59.535197973251, 0.0001680850982666)
2024-01-13 23:45:37: Saving thumb (76173): full_filepath: 1.jpg  (59.535832166672, 0.00063490867614746)
2024-01-13 23:45:37: Saving location: local to folder ../thumbs//user_thumbs//76/173  (59.535849094391, 1.5020370483398E-5)
2024-01-13 23:45:37: save_name 1. (59.53590798378, 5.9127807617188E-5)
2024-01-13 23:45:37: Saving  - not a file??? (59.53591799736, 7.8678131103516E-6)
2024-01-13 23:45:37: Saving thumb (76173): full_filepath: 2.jpg  (59.535928010941, 8.8214874267578E-6)
2024-01-13 23:45:37: Saving location: local to folder ../thumbs//user_thumbs//76/173  (59.535937070847, 8.1062316894531E-6)
2024-01-13 23:45:37: save_name 1. (59.535952091217, 1.5020370483398E-5)
2024-01-13 23:45:37: Saving  - not a file??? (59.535958051682, 4.0531158447266E-6)
2024-01-13 23:45:37: Saving thumb (76173): full_filepath: 3.jpg  (59.535966157913, 8.1062316894531E-6)
2024-01-13 23:45:37: Saving location: local to folder ../thumbs//user_thumbs//76/173  (59.535973072052, 6.9141387939453E-6)
2024-01-13 23:45:37: save_name 1. (59.535987138748, 1.4066696166992E-5)
2024-01-13 23:45:37: Saving  - not a file??? (59.535992145538, 4.0531158447266E-6)
2024-01-13 23:45:37: Saving thumb (76173): full_filepath: 4.jpg  (59.535999059677, 5.9604644775391E-6)
2024-01-13 23:45:37: Saving location: local to folder ../thumbs//user_thumbs//76/173  (59.536005973816, 6.1988830566406E-6)
2024-01-13 23:45:37: save_name 1. (59.536020040512, 1.2874603271484E-5)
2024-01-13 23:45:37: Saving  - not a file??? (59.536025047302, 4.0531158447266E-6)
2024-01-13 23:45:37: Saving thumb (76173): full_filepath: 5.jpg  (59.536034107208, 9.0599060058594E-6)
2024-01-13 23:45:37: Saving location: local to folder ../thumbs//user_thumbs//76/173  (59.536046028137, 1.0967254638672E-5)
2024-01-13 23:45:37: save_name 1. (59.536069154739, 2.1934509277344E-5)
2024-01-13 23:45:37: Saving  - not a file??? (59.536076068878, 4.0531158447266E-6)
2024-01-13 23:45:37: Cleanup tmp folder: ../tmp/76173 (59.536082029343, 5.9604644775391E-6)
2024-01-13 23:45:37: Set gallery status to active  (59.537127017975, 0.0010449886322021)2024-01-13 23:45:37: processed_galleries = 278, memory = 6.6 Mb limit 512M (33.06130194664, 0.00085306167602539)
2024-01-13 23:45:37: Processing https://drtvid.com/v/MdAB (76166) (33.063122034073, 0.001816987991333)
2024-01-13 23:45:37: Gallery description is empty: Update  with 'empty'  (33.064409017563, 0.0012872219085693)
2024-01-13 23:45:37: Set tags for gallery_id 76166  (33.065864086151, 0.0014538764953613)
2024-01-13 23:45:37: Content type: 4 (33.210663080215, 0.14479899406433)
2024-01-13 23:45:37: Creating thumb from https://pics.drtuber.com/media/videos/tmb/9069635/preview/11.jpg  (240x180) (Crop profile: 1)  (33.211235046387, 0.00057411193847656)
2024-01-13 23:45:37: File ext jpg (33.211277008057, 3.7193298339844E-5)
2024-01-13 23:45:37: Skip download as it's a hotlink and we don't need to check for dupes (33.211302042007, 2.288818359375E-5)
2024-01-13 23:45:37: Creating thumb from https://pics.drtuber.com/media/videos/tmb/9069635/preview/1.jpg  (240x180) (Crop profile: 1)  (33.211326122284, 2.1934509277344E-5)
2024-01-13 23:45:37: File ext jpg (33.211349010468, 2.1219253540039E-5)
2024-01-13 23:45:37: Skip download as it's a hotlink and we don't need to check for dupes (33.211364030838, 1.2874603271484E-5)
2024-01-13 23:45:37: Creating thumb from https://pics.drtuber.com/media/videos/tmb/9069635/preview/2.jpg  (240x180) (Crop profile: 1)  (33.211383104324, 1.6927719116211E-5)
2024-01-13 23:45:37: File ext jpg (33.211400985718, 1.4781951904297E-5)
2024-01-13 23:45:37: Skip download as it's a hotlink and we don't need to check for dupes (33.211409091949, 6.9141387939453E-6)
2024-01-13 23:45:37: Creating thumb from https://pics.drtuber.com/media/videos/tmb/9069635/preview/3.jpg  (240x180) (Crop profile: 1)  (33.211418151855, 8.1062316894531E-6)
2024-01-13 23:45:37: File ext jpg (33.211428165436, 9.0599060058594E-6)
2024-01-13 23:45:37: Skip download as it's a hotlink and we don't need to check for dupes (33.2114341259, 5.0067901611328E-6)
2024-01-13 23:45:37: Creating thumb from https://pics.drtuber.com/media/videos/tmb/9069635/preview/4.jpg  (240x180) (Crop profile: 1)  (33.211442947388, 8.1062316894531E-6)
2024-01-13 23:45:37: File ext jpg (33.211452960968, 9.0599060058594E-6)
2024-01-13 23:45:37: Skip download as it's a hotlink and we don't need to check for dupes (33.211459159851, 5.9604644775391E-6)
2024-01-13 23:45:37: Creating thumb from https://pics.drtuber.com/media/videos/tmb/9069635/preview/5.jpg  (240x180) (Crop profile: 1)  (33.211467027664, 6.9141387939453E-6)
2024-01-13 23:45:37: File ext jpg (33.21147608757, 8.1062316894531E-6)
2024-01-13 23:45:37: Skip download as it's a hotlink and we don't need to check for dupes (33.211483001709, 5.9604644775391E-6)
2024-01-13 23:45:37: insert into rot_thumbs  (33.21150302887, 1.8835067749023E-5)
2024-01-13 23:45:37: insert into rot_gallery_stats (33.211744070053, 0.00023984909057617)
2024-01-13 23:45:37: Saving thumb (76166): full_filepath: 1.jpg  (33.212476015091, 0.00072884559631348)
2024-01-13 23:45:37: Saving location: local to folder ../thumbs//user_thumbs//76/166  (33.212499141693, 2.1934509277344E-5)
2024-01-13 23:45:37: save_name 1. (33.212608098984, 0.00010991096496582)
2024-01-13 23:45:37: Saving  - not a file??? (33.21262216568, 1.0967254638672E-5)
2024-01-13 23:45:37: Saving thumb (76166): full_filepath: 2.jpg  (33.212635993958, 1.4066696166992E-5)
2024-01-13 23:45:37: Saving location: local to folder ../thumbs//user_thumbs//76/166  (33.212656021118, 1.7881393432617E-5)
2024-01-13 23:45:37: save_name 1. (33.212689161301, 3.1948089599609E-5)
2024-01-13 23:45:37: Saving  - not a file??? (33.212702989578, 1.0967254638672E-5)
2024-01-13 23:45:37: Saving thumb (76166): full_filepath: 3.jpg  (33.212722063065, 1.7881393432617E-5)
2024-01-13 23:45:37: Saving location: local to folder ../thumbs//user_thumbs//76/166  (33.212734937668, 1.1920928955078E-5)
2024-01-13 23:45:37: save_name 1. (33.21276807785, 3.1948089599609E-5)
2024-01-13 23:45:37: Saving  - not a file??? (33.212783098221, 1.2874603271484E-5)
2024-01-13 23:45:37: Saving thumb (76166): full_filepath: 4.jpg  (33.212801933289, 1.8119812011719E-5)
2024-01-13 23:45:37: Saving location: local to folder ../thumbs//user_thumbs//76/166  (33.212815999985, 1.1920928955078E-5)
2024-01-13 23:45:37: save_name 1. (33.21283698082, 1.978874206543E-5)
2024-01-13 23:45:37: Saving  - not a file??? (33.212847948074, 1.0967254638672E-5)
2024-01-13 23:45:37: Saving thumb (76166): full_filepath: 5.jpg  (33.212866067886, 1.5974044799805E-5)
2024-01-13 23:45:37: Saving location: local to folder ../thumbs//user_thumbs//76/166  (33.212882995605, 1.6212463378906E-5)
2024-01-13 23:45:37: save_name 1. (33.212918043137, 3.4093856811523E-5)
2024-01-13 23:45:37: Saving  - not a file??? (33.212933063507, 1.2874603271484E-5)
2024-01-13 23:45:37: Cleanup tmp folder: ../tmp/76166 (33.212947130203, 1.2874603271484E-5)
2024-01-13 23:45:37: Set gallery status to active  (33.216485977173, 0.0035381317138672)2024-01-13 23:45:37: processed_galleries = 337, memory = 6.6 Mb limit 512M (37.839073896408, 0.00052809715270996)
2024-01-13 23:45:37: Processing https://xh.video/v/3k7 (76164) (37.861101865768, 0.022027969360352)
2024-01-13 23:45:37: Gallery description is empty: Update  with 'empty'  (37.862180948257, 0.0010771751403809)
2024-01-13 23:45:37: Set tags for gallery_id 76164  (37.86358499527, 0.0014009475708008)
2024-01-13 23:45:37: Content type: 4 (38.018405914307, 0.15482091903687)
2024-01-13 23:45:37: Creating thumb from https://thumb-v3.xhcdn.com/a/demiRdoVsUqoFXOgyh8GpQ/000/244/023/2000x2000.3.jpg  (240x180) (Crop profile: 1)  (38.018862962723, 0.00045394897460938)
2024-01-13 23:45:37: File ext jpg (38.018893003464, 3.8862228393555E-5)
2024-01-13 23:45:37: Skip download as it's a hotlink and we don't need to check for dupes (38.018915891647, 1.215934753418E-5)
2024-01-13 23:45:37: insert into rot_thumbs  (38.018946886063, 3.0040740966797E-5)
2024-01-13 23:45:37: insert into rot_gallery_stats (38.019175052643, 0.00022697448730469)
2024-01-13 23:45:37: Cleanup tmp folder: ../tmp/76164 (38.019654989243, 0.00047802925109863)
2024-01-13 23:45:37: Set gallery status to active  (38.020663976669, 0.0010080337524414)2024-01-13 23:45:37: processed_galleries = 288, memory = 6.6 Mb limit 512M (31.784096002579, 0.00024580955505371)
2024-01-13 23:45:37: Processing https://xh.video/v/Nm1 (76175) (31.795778036118, 0.011680841445923)
2024-01-13 23:45:37: Gallery description is empty: Update  with 'empty'  (31.797079086304, 0.0012998580932617)
2024-01-13 23:45:37: Set tags for gallery_id 76175  (31.798738956451, 0.0016570091247559)
2024-01-13 23:45:37: Content type: 4 (31.912315130234, 0.11357593536377)
2024-01-13 23:45:37: Creating thumb from https://thumb-v3.xhcdn.com/a/BtHQACKWm-RDLmoUeAuECw/000/219/533/2000x2000.4.jpg  (240x180) (Crop profile: 1)  (31.913223981857, 0.00090909004211426)
2024-01-13 23:45:37: File ext jpg (31.91325712204, 3.0994415283203E-5)
2024-01-13 23:45:37: Skip download as it's a hotlink and we don't need to check for dupes (31.913269996643, 1.0967254638672E-5)
2024-01-13 23:45:37: insert into rot_thumbs  (31.913290977478, 2.0027160644531E-5)
2024-01-13 23:45:37: insert into rot_gallery_stats (31.913502931595, 0.00021100044250488)
2024-01-13 23:45:37: Cleanup tmp folder: ../tmp/76175 (31.914819002151, 0.0013139247894287)
2024-01-13 23:45:37: Set gallery status to active  (31.917108058929, 0.0022878646850586)2024-01-13 23:45:37: processed_galleries = 322, memory = 6.6 Mb limit 512M (36.517471075058, 0.00043487548828125)
2024-01-13 23:45:37: Processing https://xh.video/v/Nc4 (76169) (36.556241989136, 0.038772106170654)
2024-01-13 23:45:37: Gallery description is empty: Update  with 'empty'  (36.557650089264, 0.0014069080352783)
2024-01-13 23:45:37: Set tags for gallery_id 76169  (36.559370040894, 0.0017168521881104)
2024-01-13 23:45:37: Content type: 4 (36.673558950424, 0.11418914794922)
2024-01-13 23:45:37: Creating thumb from https://thumb-v1.xhcdn.com/a/TpP4i_oEDQ-sIBQIM0C9tg/000/231/181/2000x2000.3.jpg  (240x180) (Crop profile: 1)  (36.675441980362, 0.0018820762634277)
2024-01-13 23:45:37: File ext jpg (36.675471067429, 2.598762512207E-5)
2024-01-13 23:45:37: Skip download as it's a hotlink and we don't need to check for dupes (36.675482988358, 1.0967254638672E-5)
2024-01-13 23:45:37: insert into rot_thumbs  (36.675503015518, 1.9073486328125E-5)
2024-01-13 23:45:37: insert into rot_gallery_stats (36.675746917725, 0.000244140625)
2024-01-13 23:45:37: Cleanup tmp folder: ../tmp/76169 (36.677995920181, 0.0022480487823486)
2024-01-13 23:45:37: Set gallery status to active  (36.680769920349, 0.0027740001678467)2024-01-13 23:45:37: processed_galleries = 323, memory = 6.6 Mb limit 512M (36.68155503273, 0.00078201293945312)
По дюпам все.

2) момент который раздражает:
Скрипт создает каталоги в user_thums, но ничего туда не пишет естественно при настройке импорт сетах - хотлинковать.
Т.е. по моей логике, при полном хотлинке всего, скрипт должен взять дампы и просто зафигачить их в базу, разложив все как надо ( как ему указано ), проверить на дюпы и на этом все.
admin
Site Admin
Posts: 37202
Joined: Wed Sep 10, 2008 11:43 am

Re: Big Base Demo

Post by admin »

Приветствую!

а можно плз админку и PMA туда? мы б вместе протестировали попадание дупов
Don't forget to run script update
darkover
Posts: 1518
Joined: Sat Apr 13, 2013 8:59 pm

Re: Big Base Demo

Post by darkover »

admin wrote: Mon Jan 15, 2024 2:01 pm Приветствую!

а можно плз админку и PMA туда? мы б вместе протестировали попадание дупов
В ЛС с трактатом о наболевшем )
admin
Site Admin
Posts: 37202
Joined: Wed Sep 10, 2008 11:43 am

Re: Big Base Demo

Post by admin »

вроде все решили тут )
Don't forget to run script update
darkover
Posts: 1518
Joined: Sat Apr 13, 2013 8:59 pm

Re: Big Base Demo

Post by darkover »

admin wrote: Tue Feb 20, 2024 9:55 pm вроде все решили тут )
Ну вроде да )
admin
Site Admin
Posts: 37202
Joined: Wed Sep 10, 2008 11:43 am

Re: Big Base Demo

Post by admin »

Отлично )
Don't forget to run script update
Post Reply