====== Import Sets ======

Import Sets let you automatically pull galleries from any source at regular intervals and apply the same actions you'd use for manual imports (custom galleries, multi-category assignment, thumbnail generation, etc.).

To set up an import set:

1. Go to Rotation > Import Sets and click "Add New"
2. Enter the URL of your data source (dump or RSS feed)
3. Set the re-grab interval (24 = every 24 hours)
4. Specify the delimiter (usually "|" for dumps)
5. Click Test—the script analyzes the URL and identifies fields
6. For any unidentified fields, skip them or manually select the field type
7. Configure import options just like manual import
8. Click Save

Done. The import runs automatically on schedule.

===== Tips =====

- **RSS thumbnails:** Sometimes RSS feeds don't provide direct thumbnail URLs but instead contain HTML like <img src=....>. For these fields, select "Parse Thumb" instead of "Thumb" so the script extracts the image URL from the HTML tag.

- **Delimiters:** The delimiter separates fields in your data source. In dumps, it's usually | (pipe). In RSS feeds, items are typically separated by `<item>` tags, though sometimes `<video>` is used. To find the right delimiter, open the URL in your browser, view the page source, and look for the repeating pattern that separates records.



===== Deletion Feed =====

Import Sets come in two types: **Add** (imports new galleries) and **Deletion** (removes galleries from your database).

Deletion feeds are efficient for automatic gallery removal. Instead of running [[Gallery Checker]] to test every link (which loads both your server and the sponsor's), you can subscribe to a deletion feed. Sponsors provide a list of deleted URLs, and the script checks it every X hours, automatically removing those galleries from your database.

The script supports partial URL matching. For example, if your URL is http://sponsor.com/gallery/index.html?id=your_id but the deletion feed only provides http://sponsor.com/gallery/index.html, it still matches and deletes the gallery.

**Aggregating deletion feeds across multiple sites**

If you run multiple sites that share gallery URLs, you can aggregate deletion feeds in one location. For example:

- You have 10 deletion feed sources plus your own custom galleries
- These galleries are used across 100 other sites
- Instead of adding the 10 feeds to each of the 100 sites, add them once to a central "base" site

On the base site, set the import type to "mark inactive" (offline). Then create a deletion list template:

<code>

<thumb num=1-100 order=date gallery_status=inactive>
gallery URL
</thumb>

</code>

Access the list via http://domain.com/?force_template=deletion_list (shows the last 100 inactive galleries). Now you only need to add this single URL as a Deletion import set on your other 100 sites.

**Note for master-slave networks:** If you run master-slave sites sharing one database, you only need to add the deletion feed to one site—all slaves will automatically see the deletions.
===== Large import sets =====

Normally, import sets check a URL on a schedule (once per day, once every X hours) and add newly available galleries. This works well when sources add 10-20 to 1000 URLs daily.

For very large one-time imports (1 million+ URLs), add the file as an import set as usual. The script downloads and processes all URLs until the entire file is imported. Once complete, the import set automatically pauses—it won't check for daily updates since the source was a one-time bulk file.

===== Deletion feeds with embed imports (example: xHamster) =====

This applies to any video source, but xHamster is a common example.

**The scenario:**

You set up an import set to add xHamster embeds:

  http://partners.xhamster.com/2export.php?ch=!&cnt=4&tmb=4&tcnt=10&tl=on&ord=1&url=off&em=1&ttl=on&chs=on&sz=on&dlm=|

This provides: #EMBED|#THUMB|#TITLE|#CHANNEL|#DURATION|

Everything works initially. Later, you want to automatically remove galleries deleted on xHamster. xHamster provides a deletion list, but your import set has no URL field (#EMBED|#THUMB|#TITLE|#CHANNEL|#DURATION|), so the script can't match deleted videos.

**The solution:**

Modify the import URL to include the gallery URL:

  http://partners.xhamster.com/2export.php?ch=!&cnt=4&tmb=4&tcnt=10&tl=on&ord=1&url=on&em=1&ttl=on&chs=on&sz=on&dlm=|

Now it provides: #EMBED|#THUMB|#URL|#TITLE|#CHANNEL|#DURATION|

SmartCJ stores each gallery with its source URL and can automatically match and delete videos when they appear in the deletion feed. This also applies if you use [[Gallery Checker]] instead.

**Background:**

Before deletion feeds, [[Tube Import]] let you auto-import videos, but tubes delete content regularly. [[Gallery Checker]] was created to detect deletions by keyword matching (for example, searching for "Video deleted" on xhamster.com pages). However, checking 500k+ URLs this way was inefficient. When tubes started offering deletion RSS feeds with specific deleted URLs, the Import Set gained a Deletion type for instant, accurate removal.

For both [[Gallery Checker]] and deletion imports, you need the original gallery URL. Deletion feeds are preferred over Gallery Checker when available.

**Option 2: Mismatch between import URL and deletion feed**

Sometimes the URLs in your import don't match the deletion feed. For example:
- You imported: http://gallery/1234.html
- The tube changed URLs to: http://gallery/1234/
- The deletion feed provides only: 1234

Store the deletion feed identifier in a custom field. For example, import as: url|custom_var1 (where custom_var1 = 1234)

Then configure the Deletion import set to match by custom_var1 instead of the main URL field.

**Important:** The script checks both field value AND sponsor name when matching. This prevents deleting the wrong gallery if multiple sponsors happen to have ID 1234.
===== Paginated import feeds =====

Some feeds limit results to 50 items per page and require you to manually increment the page parameter. Use the {PAGE} tag to automate this. The script starts with 1 on the first run and increments by 1 each subsequent run, walking through all pages. The import stops when the URL returns no results.

If you edit the import set, the page counter resets to 1.

===== Custom names for import sets =====

You can optionally assign a descriptive name to each import set. This makes it easier to identify them in the admin interface, though it's not required.