Differences

This shows you the differences between two versions of the page.

--- import_sets [2015/06/10 15:09] – admin
+++ import_sets [2026/03/21 10:11] (current) – external edit 127.0.0.1
@@ Line 1: / Line 1: @@
 ====== Import Sets ======
-Various sponsors provide URLs in various formats: RSS (and some sponsors use nonstandard fields), XML, dump. Before update 47 you could add only standard RSS feed and It was inconvenient for automatic feeding of you site. Import Set introduced for take care of this. With Import Sets you can automatically import galleries from virtually any source with any options including Custom Galleries, additional thumbs and so on.
+Import Sets let you automatically pull galleries from any source at regular intervals and apply the same actions you'd use for manual imports (custom galleries, multi-category assignment, thumbnail generation, etc.).
-How it works: Rotation - Import Sets, click "Add New". It opens the Import form with only few exceptions: input field for feed url (RSS, dump, XML), period of regrab, type of grab and delimiter. So type url to your gallery dump for example, click test, select what you want to import from this dump, select options as with regular import and click save. Script will import data from this dump every X hours.
+To set up an import set:
-** Hints **
+. Go to Rotation > Import Sets and click "Add New"
+. Enter the URL of your data source (dump or RSS feed)
+. Set the re-grab interval (24 = every 24 hours)
+. Specify the delimiter (usually "|" for dumps)
+. Click Test—the script analyzes the URL and identifies fields
+. For any unidentified fields, skip them or manually select the field type
+. Configure import options just like manual import
+. Click Save
-  * Sometimes there's no direct URL to thumb in RSS and it looks like "some text <img src=....>". Select "Parse Thumb",  not "Thumb" in such cases.
+Done. The import runs automatically on schedule.
-  * Delimiter - a division of the fields, it usually '|' for dumps, for RSS 'item' or 'video'
-  * Description (Strip tags) use it if RSS have something like <a href='...'> Some Description </a> while you need only Some Description.
+===== Tips =====
-**Note:** there are 2 types of import sets: Add and Delete. So urls found in Add type - will be added to DB, in type = delele - will be deleted from DB.
+- **RSS thumbnails:** Sometimes RSS feeds don't provide direct thumbnail URLs but instead contain HTML like <img src=....>. For these fields, select "Parse Thumb" instead of "Thumb" so the script extracts the image URL from the HTML tag.
+- **Delimiters:** The delimiter separates fields in your data source. In dumps, it's usually | (pipe). In RSS feeds, items are typically separated by `<item>` tags, though sometimes `<video>` is used. To find the right delimiter, open the URL in your browser, view the page source, and look for the repeating pattern that separates records.
-===== Import Sets Type Deletion и Hamster (как пример, но актуально и для других) =====
-В принципе это относится не только к Хамстеру, но поскольку вопрос возник именно на его примере - описываю  проблему на его примере.
+===== Deletion Feed =====
-Вводная: Хамстер дает ембеды, вы решили сделать на эмбедах сайт. Дабы было постоянное автоматическое пополнение сиджа настроили Import Set и добавили туда дамп Хамстера
+Import Sets come in two types: **Add** (imports new galleries) and **Deletion** (removes galleries from your database).
+Deletion feeds are efficient for automatic gallery removal. Instead of running [[Gallery Checker]] to test every link (which loads both your server and the sponsor's), you can subscribe to a deletion feed. Sponsors provide a list of deleted URLs, and the script checks it every X hours, automatically removing those galleries from your database.
+The script supports partial URL matching. For example, if your URL is http://sponsor.com/gallery/index.html?id=your_id but the deletion feed only provides http://sponsor.com/gallery/index.html, it still matches and deletes the gallery.
+**Aggregating deletion feeds across multiple sites**
+If you run multiple sites that share gallery URLs, you can aggregate deletion feeds in one location. For example:
+- You have 10 deletion feed sources plus your own custom galleries
+- These galleries are used across 100 other sites
+- Instead of adding the 10 feeds to each of the 100 sites, add them once to a central "base" site
+On the base site, set the import type to "mark inactive" (offline). Then create a deletion list template:
+<code>
+<thumb num=1-100 order=date gallery_status=inactive>
+gallery URL
+</thumb>
+</code>
+Access the list via http://domain.com/?force_template=deletion_list (shows the last 100 inactive galleries). Now you only need to add this single URL as a Deletion import set on your other 100 sites.
+**Note for master-slave networks:** If you run master-slave sites sharing one database, you only need to add the deletion feed to one site—all slaves will automatically see the deletions.
+===== Large import sets =====
+Normally, import sets check a URL on a schedule (once per day, once every X hours) and add newly available galleries. This works well when sources add 10-20 to 1000 URLs daily.
+For very large one-time imports (1 million+ URLs), add the file as an import set as usual. The script downloads and processes all URLs until the entire file is imported. Once complete, the import set automatically pauses—it won't check for daily updates since the source was a one-time bulk file.
+===== Deletion feeds with embed imports (example: xHamster) =====
+This applies to any video source, but xHamster is a common example.
+**The scenario:**
+You set up an import set to add xHamster embeds:
   http://partners.xhamster.com/2export.php?ch=!&cnt=4&tmb=4&tcnt=10&tl=on&ord=1&url=off&em=1&ttl=on&chs=on&sz=on&dlm=|
-  который выдает в формате #EMBED|#THUMB|#TITLE|#CHANNEL|#DURATION|
-Как бы этого достаточно, все работает и все хорошо.
+This provides: #EMBED|#THUMB|#TITLE|#CHANNEL|#DURATION|
-Вопрос возникнет позже - когда надо будет добавить автоматическое удаление галер удаленных на хамстере.
+Everything works initially. Later, you want to automatically remove galleries deleted on xHamster. xHamster provides a deletion list, but your import set has no URL field (#EMBED|#THUMB|#TITLE|#CHANNEL|#DURATION|), so the script can't match deleted videos.
-хамстер выдает список урлов удаленных видео, **но в нашем то импорт сете урла нет (#EMBED|#THUMB|#TITLE|#CHANNEL|#DURATION|**) и потому Смарт не сможет на автомате удалять удаленные видео (сорри за тавтологию но смысл надеюсь ясен).
+**The solution:**
-**Что делать:** урл дампа хамстера надо поменять так, что б там появился урл гали, примерно так
+Modify the import URL to include the gallery URL:
   http://partners.xhamster.com/2export.php?ch=!&cnt=4&tmb=4&tcnt=10&tl=on&ord=1&url=on&em=1&ttl=on&chs=on&sz=on&dlm=|
-  тогда дамп будет выдавать урл гали #EMBED|#THUMB|#URL|#TITLE|#CHANNEL|#DURATION|
-  те добавлено #URL
+Now it provides: #EMBED|#THUMB|#URL|#TITLE|#CHANNEL|#DURATION|
+SmartCJ stores each gallery with its source URL and can automatically match and delete videos when they appear in the deletion feed. This also applies if you use [[Gallery Checker]] instead.
+**Background:**
+Before deletion feeds, [[Tube Import]] let you auto-import videos, but tubes delete content regularly. [[Gallery Checker]] was created to detect deletions by keyword matching (for example, searching for "Video deleted" on xhamster.com pages). However, checking 500k+ URLs this way was inefficient. When tubes started offering deletion RSS feeds with specific deleted URLs, the Import Set gained a Deletion type for instant, accurate removal.
+For both [[Gallery Checker]] and deletion imports, you need the original gallery URL. Deletion feeds are preferred over Gallery Checker when available.
-Смарт будет добавлять гали в базу с пометкой source url = ... и соответственно сможет автоматически удалять у себя в базе удаленные видео хамстера.
+**Option 2: Mismatch between import URL and deletion feed**
+Sometimes the URLs in your import don't match the deletion feed. For example:
+- You imported: http://gallery/1234.html
+- The tube changed URLs to: http://gallery/1234/
+- The deletion feed provides only: 1234
-PS Это так же актуально если вы проверяете доступность галер с помощью [[Gallery Checker]].
+Store the deletion feed identifier in a custom field. For example, import as: url|custom_var1 (where custom_var1 = 1234)
-Будет не лишним сделать небольшое отступление: еще до импортсетов появился [[Tube Import]], который может автоматически добавлять видео с тубов к вам. Однако тубы периодически удаляют контент и соответственно у вас на сайте оказывались ссылки на удаленое видео или , если видео было добавлено как ембед, переставало работать. Дабы избежать такой ситуации появился [[Gallery Checker]], который по кейвордам проверяд не удалено ли видео. Если кратко, то у [[Gallery Checker]] есть список вида domain- keyword, например xhamster.com - video was deleted и чекер проходит по всем видео и проверяет, что на страницах такого слова нет и значит видео все еще работает.
+Then configure the Deletion import set to match by custom_var1 instead of the main URL field.
-Однако с ростом баз стало ясно, что если в базе 500к урлов, то появляется некоторая проблема с проверкой такого кол-ва урлов, и как раз к этому моменту у продвинутых тубов начали появляться deletion rss - те рсс со списками удаленных урлов, и соответственно можно было не проверять все урлы с помощью [[Gallery Checker]], а сразу точечно удалять нужные урлы - для этого у импорт сета появился тип deletion.
+**Important:** The script checks both field value AND sponsor name when matching. This prevents deleting the wrong gallery if multiple sponsors happen to have ID 1234.
+===== Paginated import feeds =====
-Те и для [[Gallery Checker]] и для импорт сет deletion type  - надо знать исходный урл. Если у туба\спонсора есть deletion rss - предпочтительнее юзать его. Если нет - [[Gallery Checker]].
+Some feeds limit results to 50 items per page and require you to manually increment the page parameter. Use the {PAGE} tag to automate this. The script starts with 1 on the first run and increments by 1 each subsequent run, walking through all pages. The import stops when the URL returns no results.
+If you edit the import set, the page counter resets to 1.
-===== Import Sets - personal name =====
+===== Custom names for import sets =====
-Для удобства пользования у каждого импортсета появилось опциональное имя. Можно его не вводить и список импортсетов будет выглядеть как раньше, а можно дать каждому импорсету персональное имя.
+You can optionally assign a descriptive name to each import set. This makes it easier to identify them in the admin interface, though it's not required.