Differences

This shows you the differences between two versions of the page.

--- import_sets [2015/06/10 15:07] – admin
+++ import_sets [2026/03/21 10:11] (current) – external edit 127.0.0.1
@@ Line 1: / Line 1: @@
 ====== Import Sets ======
-Various sponsors provide URLs in various formats: RSS (and some sponsors use nonstandard fields), XML, dump. Before update 47 you could add only standard RSS feed and It was inconvenient for automatic feeding of you site. Import Set introduced for take care of this. With Import Sets you can automatically import galleries from virtually any source with any options including Custom Galleries, additional thumbs and so on.
+Import Sets let you automatically pull galleries from any source at regular intervals and apply the same actions you'd use for manual imports (custom galleries, multi-category assignment, thumbnail generation, etc.).
-How it works: Rotation - Import Sets, click "Add New". It opens the Import form with only few exceptions: input field for feed url (RSS, dump, XML), period of regrab, type of grab and delimiter. So type url to your gallery dump for example, click test, select what you want to import from this dump, select options as with regular import and click save. Script will import data from this dump every X hours.
+To set up an import set:
-** Hints **
+. Go to Rotation > Import Sets and click "Add New"
+. Enter the URL of your data source (dump or RSS feed)
+. Set the re-grab interval (24 = every 24 hours)
+. Specify the delimiter (usually "|" for dumps)
+. Click Test—the script analyzes the URL and identifies fields
+. For any unidentified fields, skip them or manually select the field type
+. Configure import options just like manual import
+. Click Save
-  * Sometimes there's no direct URL to thumb in RSS and it looks like "some text <img src=....>". Select "Parse Thumb",  not "Thumb" in such cases.
+Done. The import runs automatically on schedule.
-  * Delimiter - a division of the fields, it usually '|' for dumps, for RSS 'item' or 'video'
-  * Description (Strip tags) use it if RSS have something like <a href='...'> Some Description </a> while you need only Some Description.
+===== Tips =====
-**Note:** there are 2 types of import sets: Add and Delete. So urls found in Add type - will be added to DB, in type = delele - will be deleted from DB.
+- **RSS thumbnails:** Sometimes RSS feeds don't provide direct thumbnail URLs but instead contain HTML like <img src=....>. For these fields, select "Parse Thumb" instead of "Thumb" so the script extracts the image URL from the HTML tag.
+- **Delimiters:** The delimiter separates fields in your data source. In dumps, it's usually | (pipe). In RSS feeds, items are typically separated by `<item>` tags, though sometimes `<video>` is used. To find the right delimiter, open the URL in your browser, view the page source, and look for the repeating pattern that separates records.
-===== Import Sets Type Deletion и Hamster  =====
-This is just an example based on xhamster, but it can be used with any other data sourse.
-Let's say you import embedded videos from xhamster, to do so you added 'Import Set' like
+===== Deletion Feed =====
+Import Sets come in two types: **Add** (imports new galleries) and **Deletion** (removes galleries from your database).
+Deletion feeds are efficient for automatic gallery removal. Instead of running [[Gallery Checker]] to test every link (which loads both your server and the sponsor's), you can subscribe to a deletion feed. Sponsors provide a list of deleted URLs, and the script checks it every X hours, automatically removing those galleries from your database.
+The script supports partial URL matching. For example, if your URL is http://sponsor.com/gallery/index.html?id=your_id but the deletion feed only provides http://sponsor.com/gallery/index.html, it still matches and deletes the gallery.
+**Aggregating deletion feeds across multiple sites**
+If you run multiple sites that share gallery URLs, you can aggregate deletion feeds in one location. For example:
+- You have 10 deletion feed sources plus your own custom galleries
+- These galleries are used across 100 other sites
+- Instead of adding the 10 feeds to each of the 100 sites, add them once to a central "base" site
+On the base site, set the import type to "mark inactive" (offline). Then create a deletion list template:
+<code>
+<thumb num=1-100 order=date gallery_status=inactive>
+gallery URL
+</thumb>
+</code>
+Access the list via http://domain.com/?force_template=deletion_list (shows the last 100 inactive galleries). Now you only need to add this single URL as a Deletion import set on your other 100 sites.
+**Note for master-slave networks:** If you run master-slave sites sharing one database, you only need to add the deletion feed to one site—all slaves will automatically see the deletions.
+===== Large import sets =====
+Normally, import sets check a URL on a schedule (once per day, once every X hours) and add newly available galleries. This works well when sources add 10-20 to 1000 URLs daily.
+For very large one-time imports (1 million+ URLs), add the file as an import set as usual. The script downloads and processes all URLs until the entire file is imported. Once complete, the import set automatically pauses—it won't check for daily updates since the source was a one-time bulk file.
+===== Deletion feeds with embed imports (example: xHamster) =====
+This applies to any video source, but xHamster is a common example.
+**The scenario:**
+You set up an import set to add xHamster embeds:
   http://partners.xhamster.com/2export.php?ch=!&cnt=4&tmb=4&tcnt=10&tl=on&ord=1&url=off&em=1&ttl=on&chs=on&sz=on&dlm=|
-  format #EMBED|#THUMB|#TITLE|#CHANNEL|#DURATION|
-It it fine and works good.
+This provides: #EMBED|#THUMB|#TITLE|#CHANNEL|#DURATION|
-But you might get into a trouble later, if a video will be deleted. xhamster provides a list of deleted videos as a set of urls, but when you imported videos you didn't import URL (#EMBED|#THUMB|#TITLE|#CHANNEL|#DURATION|**)
+Everything works initially. Later, you want to automatically remove galleries deleted on xHamster. xHamster provides a deletion list, but your import set has no URL field (#EMBED|#THUMB|#TITLE|#CHANNEL|#DURATION|), so the script can't match deleted videos.
+**The solution:**
-**What to do:** just add URL to the Import Set, for example
+Modify the import URL to include the gallery URL:
   http://partners.xhamster.com/2export.php?ch=!&cnt=4&tmb=4&tcnt=10&tl=on&ord=1&url=on&em=1&ttl=on&chs=on&sz=on&dlm=|
-  ie now it looks like  #EMBED|#THUMB|**#URL**|#TITLE|#CHANNEL|#DURATION|
+Now it provides: #EMBED|#THUMB|#URL|#TITLE|#CHANNEL|#DURATION|
+SmartCJ stores each gallery with its source URL and can automatically match and delete videos when they appear in the deletion feed. This also applies if you use [[Gallery Checker]] instead.
+**Background:**
+Before deletion feeds, [[Tube Import]] let you auto-import videos, but tubes delete content regularly. [[Gallery Checker]] was created to detect deletions by keyword matching (for example, searching for "Video deleted" on xhamster.com pages). However, checking 500k+ URLs this way was inefficient. When tubes started offering deletion RSS feeds with specific deleted URLs, the Import Set gained a Deletion type for instant, accurate removal.
+For both [[Gallery Checker]] and deletion imports, you need the original gallery URL. Deletion feeds are preferred over Gallery Checker when available.
+**Option 2: Mismatch between import URL and deletion feed**
+Sometimes the URLs in your import don't match the deletion feed. For example:
+- You imported: http://gallery/1234.html
+- The tube changed URLs to: http://gallery/1234/
+- The deletion feed provides only: 1234
+Store the deletion feed identifier in a custom field. For example, import as: url|custom_var1 (where custom_var1 = 1234)
+Then configure the Deletion import set to match by custom_var1 instead of the main URL field.
+**Important:** The script checks both field value AND sponsor name when matching. This prevents deleting the wrong gallery if multiple sponsors happen to have ID 1234.
+===== Paginated import feeds =====
+Some feeds limit results to 50 items per page and require you to manually increment the page parameter. Use the {PAGE} tag to automate this. The script starts with 1 on the first run and increments by 1 each subsequent run, walking through all pages. The import stops when the URL returns no results.
+If you edit the import set, the page counter resets to 1.
-And now you can use Deletion RSS provided from xhamster, and add it as Import Set :: type = deletion. The script will automatically delete videos from this RSS.
+===== Custom names for import sets =====
+You can optionally assign a descriptive name to each import set. This makes it easier to identify them in the admin interface, though it's not required.