Gallery dupes

[filthlab]

Hi,
I'm importing galleries with Sets, but there are a lot of galleries dupes.

I have in the Sets:
Check DB for Dupes: Yes

And in the Settings:
Check for dupe thumbs: Yes

A while after I started the Set, I accidentally deleted queue with "Click here to delete queue".
Do you think this might be the problem in my case?

Here is the sample of dupes from the log:

2017-06-07 02:29:23: Processing http://www.vporn.com/anal/first-anal-qu ... e/1054946/ (844220) (4.2053430080414, 0.014203786849976)
2017-06-07 02:29:23: Gallery description is empty: Update with 'admin added' (4.2060608863831, 0.00071501731872559)
2017-06-07 02:29:23: Content type: 1 (4.2082810401917, 0.0022189617156982)
2017-06-07 02:29:23: Creating thumb (320x180) (Crop profile: 1) (4.20876288414, 0.00048089027404785)
2017-06-07 02:29:23: Downloading img http://th.vporn.com/t/1054946/720x408/130.jpg (../tmp/844220/tmp//702277.jpg) (4.2088270187378, 6.1988830566406E-5)
2017-06-07 02:29:24: Dupe check 626df268e22888640351692852381864 (4.7156729698181, 0.50684881210327)
2017-06-07 02:29:24: Source Size 81279 (4.7160770893097, 0.00039887428283691)
2017-06-07 02:29:24: Make thumb ../tmp/844220/tmp//702277.jpg (4.71617603302, 9.4890594482422E-5)
2017-06-07 02:29:24: Creating thumb ../tmp/844220/tmp//43163.jpg.tmp.jpg -> ../tmp/844220/tmp//43163.jpg (4.7163279056549, 0.00015091896057129)
2017-06-07 02:29:24: Identify /usr/bin//identify -verbose ../tmp/844220/tmp//43163.jpg.tmp.jpg (4.7163980007172, 6.6995620727539E-5)
2017-06-07 02:29:24: Size: 720x408 (../tmp/844220/tmp//43163.jpg.tmp.jpg) thumb: 320x180
thumb_ratio: 1.7777777777778 img_ratio: 1.7647058823529 to ../tmp/844220/tmp//43163.jpg (4.7492160797119, 0.032819032669067)
2017-06-07 02:29:24: Cut image (4.749340057373, 0.00011992454528809)
2017-06-07 02:29:24: Face detect Cmd : /home/domains//face --input=../tmp/844220/tmp//43163.jpg.tmp.jpg --cascade=/home/domains/1.xml (4.7494049072266, 6.3180923461914E-5)
2017-06-07 02:29:24: Output : sh: 1: /home/domains//face: not found
(4.7538180351257, 0.0044200420379639)
2017-06-07 02:29:24: CMD: /usr/bin//convert -crop +33+0 +repage -crop -33-40 +repage -resize 320x180^ -gravity Center -crop 320x180+0+0 -quality 100 -filter Lanczos -normalize -unsharp 1x0.6+1 -unsharp 1x0.2+1 -enhance -modulate 100,102 ../tmp/844220/tmp//43163.jpg.tmp.jpg ../tmp/844220/tmp//43163.jpg (4.7539839744568, 0.00015497207641602)
2017-06-07 02:29:24: Identify /usr/bin//identify -verbose ../tmp/844220/tmp//43163.jpg (5.0634388923645, 0.30945992469788)
2017-06-07 02:29:24: Detect img size for ../tmp/844220/tmp//43163.jpg (320 x 180) (5.0756430625916, 0.012201070785522)
2017-06-07 02:29:24: Rot save (1041853): title:First Anal Quest Stephanie, desc:() (5.0785849094391, 0.0029361248016357)
2017-06-07 02:29:24: Saving ../thumbs/1/041//853_First.jpg (source ../tmp/844220/tmp//43163.jpg) (5.0787160396576, 0.00012898445129395)
2017-06-07 02:29:24: Cleanup tmp folder ../tmp/844220 (5.0789120197296, 0.00019383430480957)

2017-06-07 02:29:23: Processing http://www.vporn.com/anal/first-anal-qu ... e/1054946/ (844221) (4.367292881012, 0.0020301342010498)
2017-06-07 02:29:23: Gallery description is empty: Update with 'admin added' (4.3678438663483, 0.0005490779876709)
2017-06-07 02:29:23: Content type: 1 (4.3703548908234, 0.0025088787078857)
2017-06-07 02:29:23: Creating thumb (320x180) (Crop profile: 1) (4.3706858158112, 0.00032901763916016)
2017-06-07 02:29:23: Downloading img http://th.vporn.com/t/1054946/720x408/130.jpg (../tmp/844221/tmp//223441.jpg) (4.3707418441772, 5.5074691772461E-5)
2017-06-07 02:29:24: Dupe check 626df268e22888640351692852381864 (4.9255938529968, 0.55485391616821)
2017-06-07 02:29:24: Source Size 81279 (4.9258317947388, 0.00023293495178223)
2017-06-07 02:29:24: Make thumb ../tmp/844221/tmp//223441.jpg (4.9259169101715, 8.3208084106445E-5)
2017-06-07 02:29:24: Creating thumb ../tmp/844221/tmp//28834.jpg.tmp.jpg -> ../tmp/844221/tmp//28834.jpg (4.9260687828064, 0.00014996528625488)
2017-06-07 02:29:24: Identify /usr/bin//identify -verbose ../tmp/844221/tmp//28834.jpg.tmp.jpg (4.9261558055878, 8.2969665527344E-5)
2017-06-07 02:29:24: Size: 720x408 (../tmp/844221/tmp//28834.jpg.tmp.jpg) thumb: 320x180
thumb_ratio: 1.7777777777778 img_ratio: 1.7647058823529 to ../tmp/844221/tmp//28834.jpg (4.9626770019531, 0.036521196365356)
2017-06-07 02:29:24: Cut image (4.9628047943115, 0.00012397766113281)
2017-06-07 02:29:24: Face detect Cmd : /home/domains//face --input=../tmp/844221/tmp//28834.jpg.tmp.jpg --cascade=/home/domains/1.xml (4.9628658294678, 5.9127807617188E-5)
2017-06-07 02:29:24: Output : sh: 1: /home/domains//face: not found
(4.9644320011139, 0.0015699863433838)
2017-06-07 02:29:24: CMD: /usr/bin//convert -crop +33+0 +repage -crop -33-40 +repage -resize 320x180^ -gravity Center -crop 320x180+0+0 -quality 100 -filter Lanczos -normalize -unsharp 1x0.6+1 -unsharp 1x0.2+1 -enhance -modulate 100,102 ../tmp/844221/tmp//28834.jpg.tmp.jpg ../tmp/844221/tmp//28834.jpg (4.9645569324493, 0.00011801719665527)
2017-06-07 02:29:24: Identify /usr/bin//identify -verbose ../tmp/844221/tmp//28834.jpg (5.3118000030518, 0.34724807739258)
2017-06-07 02:29:24: Detect img size for ../tmp/844221/tmp//28834.jpg (320 x 180) (5.3240528106689, 0.012250185012817)
2017-06-07 02:29:24: Rot save (1041854): title:First Anal Quest Stephanie, desc:() (5.3256928920746, 0.001633882522583)
2017-06-07 02:29:24: Saving ../thumbs/1/041//854_Anal_Stephanie.jpg (source ../tmp/844221/tmp//28834.jpg) (5.3258218765259, 0.00012683868408203)
2017-06-07 02:29:24: Cleanup tmp folder ../tmp/844221 (5.3260657787323, 0.00024294853210449)

[admin]

Hi !

Did you try to add the same gallery again and it worked ?

[filthlab]

No, I'm importing from a dump, and I'm sure this url occurs only 1 time in this dump. I've checked this.

[admin]

Ok, lets create a dump with 3 galleries and add it to test

Does it add those 3 galleries more then once ?

[filthlab]

Right now grabbing is running. There is 428664 already in queue.
I've added a new set with 4 galleries, but nothing happened. Probably they are in queue.

[admin]

Well, it's gonna take awhile to process 500k galleries

We can install a copy on another domain (or even the same one, just another folder) and test it there

[filthlab]

I've tested all with about 100 galleries, and there was no problem. Also, I've tested with a big dump (about 200k galleries), and there was no problem either.

Btw, it took about 2 days to delete all these 200k (about 15k was in Active and the rest in To grab). Is there a way delete much faster from the database? The site is still not live, so I'm ready to delete all imported galleries (130804 Active and 428265 To grab), to make the tests again.

[admin]

http://smartcj.com/wiki/doku.php?id=new ... ng_deleted

[filthlab]

Hi again,
I've deleted all galleries in the database. Also, I've deleted all Sets. Then I've created a new Set with a dump with about 200k galleries. Now I'm getting the same results - a lot of gallery dupes.
See one of the dupes from the log. Keep in mind that both records are not one after another.

2017-06-07 22:15:16: Processing http://www.vporn.com/1080p/hot-skinny-b ... -dad/5370/ (1077362) (5.6154401302338, 0.10067987442017)
2017-06-07 22:15:16: Gallery description is empty: Update with 'admin added' (5.630341053009, 0.014891147613525)
2017-06-07 22:15:16: Content type: 1 (5.7011880874634, 0.070843935012817)
2017-06-07 22:15:16: Creating thumb (320x180) (Crop profile: 1) (5.7110130786896, 0.0098259449005127)
2017-06-07 22:15:16: Downloading img http://th.vporn.com/t/5370/720x408/9.jpg (../tmp/1077362/tmp//519546.jpg) (5.7111201286316, 0.00010299682617188)
2017-06-07 22:15:17: Dupe check f2adcd14f90c7e67dc0669624aeb81c8 (6.3390290737152, 0.62792801856995)
2017-06-07 22:15:17: Source Size 166376 (6.339802980423, 0.00075507164001465)
2017-06-07 22:15:17: Make thumb ../tmp/1077362/tmp//519546.jpg (6.3401999473572, 0.00038480758666992)
2017-06-07 22:15:17: Creating thumb ../tmp/1077362/tmp//28927.jpg.tmp.jpg -> ../tmp/1077362/tmp//28927.jpg (6.3410511016846, 0.00084710121154785)
2017-06-07 22:15:17: Identify /usr/bin//identify -verbose ../tmp/1077362/tmp//28927.jpg.tmp.jpg (6.3414559364319, 0.00038886070251465)
2017-06-07 22:15:17: Size: 720x408 (../tmp/1077362/tmp//28927.jpg.tmp.jpg) thumb: 320x180
thumb_ratio: 1.7777777777778 img_ratio: 1.7647058823529 to ../tmp/1077362/tmp//28927.jpg (6.3872909545898, 0.04581618309021)
2017-06-07 22:15:17: Cut image (6.3874850273132, 0.00018906593322754)
2017-06-07 22:15:17: Face detect Cmd : /home/domains//face --input=../tmp/1077362/tmp//28927.jpg.tmp.jpg --cascade=/home/domains/1.xml (6.3875761032104, 8.9168548583984E-5)
2017-06-07 22:15:17: Output : sh: 1: /home/domains//face: not found
(6.3928871154785, 0.0053300857543945)
2017-06-07 22:15:17: CMD: /usr/bin//convert -crop +33+0 +repage -crop -33-40 +repage -resize 320x180^ -gravity Center -crop 320x180+0+0 -quality 100 -filter Lanczos -normalize -unsharp 1x0.6+1 -unsharp 1x0.2+1 -enhance -modulate 100,102 ../tmp/1077362/tmp//28927.jpg.tmp.jpg ../tmp/1077362/tmp//28927.jpg (6.3931200504303, 0.00020813941955566)
2017-06-07 22:15:18: Identify /usr/bin//identify -verbose ../tmp/1077362/tmp//28927.jpg (7.0290610790253, 0.6359429359436)
2017-06-07 22:15:18: Detect img size for ../tmp/1077362/tmp//28927.jpg (320 x 180) (7.04185795784, 0.012794971466064)
2017-06-07 22:15:18: Rot save (1331425): title:Hot skinny blonde bitch gets fucked hard by her black step dad, desc:() (7.0473189353943, 0.0054540634155273)
2017-06-07 22:15:18: Saving ../thumbs/1/331//425_bitch_gets.jpg (source ../tmp/1077362/tmp//28927.jpg) (7.0474579334259, 0.00013613700866699)
2017-06-07 22:15:18: Cleanup tmp folder ../tmp/1077362 (7.0476679801941, 0.00020909309387207)
.....................
.....................
2017-06-07 22:15:16: Processing http://www.vporn.com/1080p/hot-skinny-b ... -dad/5370/ (1077361) (5.7205040454865, 0.12923002243042)
2017-06-07 22:15:16: Gallery description is empty: Update with 'admin added' (5.7267091274261, 0.0061960220336914)
2017-06-07 22:15:16: Content type: 1 (5.7312920093536, 0.0045809745788574)
2017-06-07 22:15:16: Creating thumb (320x180) (Crop profile: 1) (5.7317290306091, 0.00043511390686035)
2017-06-07 22:15:16: Downloading img http://th.vporn.com/t/5370/720x408/9.jpg (../tmp/1077361/tmp//699008.jpg) (5.7317950725555, 6.3896179199219E-5)
2017-06-07 22:15:17: Dupe check f2adcd14f90c7e67dc0669624aeb81c8 (6.3443729877472, 0.61259484291077)
2017-06-07 22:15:17: Source Size 166376 (6.3453412055969, 0.00095200538635254)
2017-06-07 22:15:17: Make thumb ../tmp/1077361/tmp//699008.jpg (6.3456161022186, 0.00026893615722656)
2017-06-07 22:15:17: Creating thumb ../tmp/1077361/tmp//58681.jpg.tmp.jpg -> ../tmp/1077361/tmp//58681.jpg (6.3462371826172, 0.00061893463134766)
2017-06-07 22:15:17: Identify /usr/bin//identify -verbose ../tmp/1077361/tmp//58681.jpg.tmp.jpg (6.3464870452881, 0.00023913383483887)
2017-06-07 22:15:17: Size: 720x408 (../tmp/1077361/tmp//58681.jpg.tmp.jpg) thumb: 320x180
thumb_ratio: 1.7777777777778 img_ratio: 1.7647058823529 to ../tmp/1077361/tmp//58681.jpg (6.402626991272, 0.056133031845093)
2017-06-07 22:15:17: Cut image (6.4028210639954, 0.00018811225891113)
2017-06-07 22:15:17: Face detect Cmd : /home/domains//face --input=../tmp/1077361/tmp//58681.jpg.tmp.jpg --cascade=/home/domains/1.xml (6.4029121398926, 8.9168548583984E-5)
2017-06-07 22:15:17: Output : sh: 1: /home/domains//face: not found
(6.4082231521606, 0.0053300857543945)
2017-06-07 22:15:17: CMD: /usr/bin//convert -crop +33+0 +repage -crop -33-40 +repage -resize 320x180^ -gravity Center -crop 320x180+0+0 -quality 100 -filter Lanczos -normalize -unsharp 1x0.6+1 -unsharp 1x0.2+1 -enhance -modulate 100,102 ../tmp/1077361/tmp//58681.jpg.tmp.jpg ../tmp/1077361/tmp//58681.jpg (6.4084560871124, 0.00020813941955566)
2017-06-07 22:15:18: Identify /usr/bin//identify -verbose ../tmp/1077361/tmp//58681.jpg (7.0479810237885, 0.63952612876892)
2017-06-07 22:15:18: Detect img size for ../tmp/1077361/tmp//58681.jpg (320 x 180) (7.0611650943756, 0.01318097114563)
2017-06-07 22:15:18: Rot save (1331426): title:Hot skinny blonde bitch gets fucked hard by her black step dad, desc:() (7.0634500980377, 0.0022788047790527)
2017-06-07 22:15:18: Saving ../thumbs/1/331//426_hard_black_dad.jpg (source ../tmp/1077361/tmp//58681.jpg) (7.0635640621185, 0.00011396408081055)
2017-06-07 22:15:18: Cleanup tmp folder ../tmp/1077361 (7.063863992691, 0.00029802322387695)

[admin]

ok, send me this dump and what to do to repeat the issue