Page 1 of 3

Import Big CSV

Posted: Tue Aug 02, 2016 2:36 pm
by happydg
Hello,

I would like to import big csv files with a lot of galleries ( like 600k galleries ), but browser times out before ending. Is there a way to do it server side and not on the browser?

Thank you.

Re: Import Big CSV

Posted: Wed Aug 03, 2016 7:20 am
by admin

Re: Import Big CSV

Posted: Wed Aug 03, 2016 1:16 pm
by happydg
1) What Period, hours should i set if i have 5 sets adding galleries so it doesn't overwhelm my dedicated server?

2) I added a big csv file ( 200 mo ) and after clicking on test it says: "This source is too big (bigger then 600kb) to be parsed at a regular pace. You can add it, BUT it will be parsed ONE TIME ONLY. Ie it will NOT be parsed every X hours."

So it won't parse 500 new videos per hour like cut.php should do?

Thank you.

Re: Import Big CSV

Posted: Thu Aug 04, 2016 7:27 am
by admin
1. you can set 1, should be fine anyway

2. did you add it with &test=true as wiki says ?

Re: Import Big CSV

Posted: Thu Aug 04, 2016 9:28 am
by happydg
2. I forgot to add it, my mistake.

Re: Import Big CSV

Posted: Thu Aug 04, 2016 9:35 am
by admin
ok

Re: Import Big CSV

Posted: Thu Aug 04, 2016 12:45 pm
by happydg
Even with &test=true i get:

"This source is too big (bigger then 600kb) to be parsed at a regular pace. You can add it, BUT it will be parsed ONE TIME ONLY. Ie it will NOT be parsed every X hours."

I sent you the link via pm.

Thank you.

Re: Import Big CSV

Posted: Thu Aug 04, 2016 5:15 pm
by admin
What url exactly are you trying to add ?

Re: Import Big CSV

Posted: Thu Aug 04, 2016 5:19 pm
by admin
Usually that message pops up when import part is bigger then 600k

So probably 500 lines are too much, try to lower to to 200 for rexample

Re: Import Big CSV

Posted: Tue Aug 09, 2016 9:27 am
by happydg
I imported the xhamster.com big dump. It's 1.8 gb and has millions of galleries.

I used this way to import it http://smartcj.com/wiki/doku.php?id=new ... _large_cvs with 150 galleries per hour.

It just imported 11k galleries and the xhamster.csv file is now at 0kb