Page 1 of 3
Import Big CSV
Posted: Tue Aug 02, 2016 2:36 pm
by happydg
Hello,
I would like to import big csv files with a lot of galleries ( like 600k galleries ), but browser times out before ending. Is there a way to do it server side and not on the browser?
Thank you.
Re: Import Big CSV
Posted: Wed Aug 03, 2016 7:20 am
by admin
Re: Import Big CSV
Posted: Wed Aug 03, 2016 1:16 pm
by happydg
1) What Period, hours should i set if i have 5 sets adding galleries so it doesn't overwhelm my dedicated server?
2) I added a big csv file ( 200 mo ) and after clicking on test it says: "This source is too big (bigger then 600kb) to be parsed at a regular pace. You can add it, BUT it will be parsed ONE TIME ONLY. Ie it will NOT be parsed every X hours."
So it won't parse 500 new videos per hour like cut.php should do?
Thank you.
Re: Import Big CSV
Posted: Thu Aug 04, 2016 7:27 am
by admin
1. you can set 1, should be fine anyway
2. did you add it with &test=true as wiki says ?
Re: Import Big CSV
Posted: Thu Aug 04, 2016 9:28 am
by happydg
2. I forgot to add it, my mistake.
Re: Import Big CSV
Posted: Thu Aug 04, 2016 9:35 am
by admin
ok
Re: Import Big CSV
Posted: Thu Aug 04, 2016 12:45 pm
by happydg
Even with &test=true i get:
"This source is too big (bigger then 600kb) to be parsed at a regular pace. You can add it, BUT it will be parsed ONE TIME ONLY. Ie it will NOT be parsed every X hours."
I sent you the link via pm.
Thank you.
Re: Import Big CSV
Posted: Thu Aug 04, 2016 5:15 pm
by admin
What url exactly are you trying to add ?
Re: Import Big CSV
Posted: Thu Aug 04, 2016 5:19 pm
by admin
Usually that message pops up when import part is bigger then 600k
So probably 500 lines are too much, try to lower to to 200 for rexample
Re: Import Big CSV
Posted: Tue Aug 09, 2016 9:27 am
by happydg
I imported the xhamster.com big dump. It's 1.8 gb and has millions of galleries.
I used this way to import it
http://smartcj.com/wiki/doku.php?id=new ... _large_cvs with 150 galleries per hour.
It just imported 11k galleries and the xhamster.csv file is now at 0kb