TA-Helper only showing the first 12 videos for each channel.

[bytePatrol]

Have everything setup & working and love the script but I noticed that it only generates 12 symbolic links for each channel regardless if the channel has 20 or 500 videos, each channel somehow only has 12 symlinks created with TA-Helper script. I cant find any obvious settings to adjust it from 12 or see any logical reason why it's limited to 12 but hoping I'm overlooking something simple here.

[RoninTech]

Hi,

The helper script should simply iterate over what has been grabbed by TA.

Has TA actually grabbed more than 12 videos from each channel and the helper script is ignoring anything beyond 12 videos? Or did TA not grab more than 12 videos?

If TA did grab all the channel videos but the helper script didn't process beyond 12 then the helper script may have the QUICK option set to true and it found an already processed video (symlink already exists) so it stopped.

If TA didn't grab more than 12 videos for each channel then this is a question for TA.

Maybe generate a log file with LOGGING_LEVEL set to "DEBUG"? If you see "Symlink exists for " + title, then the helper is stopping because it has found an already processed video.

Hope that helps.

[focustense]

I don't know if this is still maintained, but for anyone who stumbles on this, I had to make several changes to the script to get it working with current TA builds, and one of them was around pagination. There are actually a few problems, some with the script and some apparently with TA's API.

Ultimately it all boils down to the QUICK setting, which stops processing all videos for a given channel as soon as it sees any video that already has a symlink. This is probably a good idea for anyone who's got thousands of videos archived; if your library is small and/or your server is fast, you could just disable QUICK and that should fix all pagination problems. I wanted to make sure this script scales to however large my collection eventually gets, so here's what I did to get QUICK mode working:

1. The default sort order for the /videos endpoint (I don't think channels/{channel}/videos works anymore) does not list the most recently downloaded videos first, and in fact I think it may produce the oldest videos first, which means by default, all but the first sync for any given channel will stop after the very first video. This is easy to resolve by setting the URL parameters to {'channel': channel_id, 'sort': 'downloaded', 'order': 'desc'}.

2. TA-Helper works by looking at pagination.current_page and incrementing it, which you'd think should work. But for some reason, TA's page 0 is the same as page 1. This means that the logic as written will always see duplicate/already-processed videos as soon as it hits the second page (or 1 in the page param) and quit early. This is the proximate cause of only getting 12 results, even when starting from scratch. One workaround would be to just skip page index 1 and start pagination at page 2; however, since TA includes a next_pages field, I went with using next_pages[0] instead. You'll notice that the "next page" after the initial request is 2, not 1.

3. Unfortunately there seems to be an unrelated bug in TA, where next_pages is empty for the second-to-last page, so I also had to fall back to the "current page plus one" logic when next_pages is empty. No need for this workaround if you use the "skip page 1" hack instead.

A brief "diff" here that I'm collapsing into two snippets:

Old code

chan_url = url+channel['channel_id']+"/video/"

# (other code)

chan_videos = requests.get(chan_url, headers=headers)

# (other code)

while chan_videos_json['paginate']['last_page']:
    chan_videos_json = requests.get(chan_url, headers=headers, params={'page': chan_videos_json['paginate']['current_page'] + 1}).json()

to this:

chan_url = TA_SERVER + '/api/video'
chan_params = {'channel': channel['channel_id'], 'sort': 'downloaded', 'order': 'desc'}

# (unchanged code)

chan_videos = requests.get(chan_url, headers=headers, params=chan_params)

# (unchanged code)

while chan_videos_json['paginate']['last_page']:
    if len(chan_videos_json['paginate']['next_pages']) > 0:
        chan_params['page'] = chan_videos_json['paginate']['next_pages'][0]
    else:
        chan_params['page'] = chan_videos_json['paginate']['current_page'] + 1
    chan_videos_json = requests.get(chan_url, headers=headers, params=chan_params).json()

Hope it helps someone. I'd post an entire file/diff but I made a bunch of other unrelated changes so I think it would only add confusion if I did.