Skip to content

Conversation

@AudricV
Copy link
Member

@AudricV AudricV commented Sep 17, 2023

  • I carefully read the contribution guidelines and agree to them.
  • I have tested the API against NewPipe.
  • I agree to create a pull request for NewPipe as soon as possible to make it compatible with the changed API.

This PR replaces fetching playlist items with an initial playlist continuation, in order to workaround the Shorts UI in playlists, restricting access to only the 100th first items, as no continuation is currently provided. Reel items returned don't also give upload date, uploader info and precise view count.

This required to add a protocol buffer definition file, for which its structure is based on reverse engineering of playlists continuations sent by WEB InnerTube client, received from InnerTube responses.

Java classes of this file are generated for the Java Lite runtime of Protobuf with the Protobuf Gradle plugin, as the lite version is enough for our use cases. This plugin ships in JARs Protobuf definitions, which should be avoided (I didn't found something for NewPipeExtractor working on assemble Gradle tasks, only for NewPipe).

As Protobuf generated classes are parsed by Checkstyle and do not follow our style rules at all, an exclusion rule has been for them.

As a result of this change, testMoreRelatedItems method of YoutubePlaylistExtractorTest.ShortsUI test class has been enabled.

Fixes #1273
Fixes TeamNewPipe/NewPipe#12341

@AudricV AudricV added enhancement New feature or request youtube service, https://www.youtube.com/ labels Sep 17, 2023
@AudricV AudricV force-pushed the yt_continuations-playlists-shorts-ui branch from bccde47 to 2e3f6b4 Compare September 22, 2023 15:10
@AudricV AudricV marked this pull request as draft October 6, 2023 18:54
@AudricV AudricV mentioned this pull request Feb 13, 2025
1 task
@AudricV AudricV force-pushed the yt_continuations-playlists-shorts-ui branch from 2e3f6b4 to b7ea744 Compare April 18, 2025 13:29
@AudricV AudricV marked this pull request as ready for review April 18, 2025 13:29
@AudricV AudricV force-pushed the yt_continuations-playlists-shorts-ui branch 2 times, most recently from e797b4b to 43082e1 Compare April 18, 2025 14:11
Copy link
Member

@Stypox Stypox left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Have you tried this to exclude the *.proto from the .jar? google/protobuf-gradle-plugin#390

Other than that, code looks good to me. Thanks!

@Stypox Stypox added this to v0.28.x May 16, 2025
@Stypox Stypox moved this to Todo in v0.28.x May 16, 2025
@AudricV AudricV moved this from Todo to In Progress in v0.28.x Jul 7, 2025
@Stypox Stypox self-assigned this Jul 16, 2025
AudricV and others added 7 commits July 16, 2025 13:26
query parameters

This can be used to pass Google APIs standard parameters to requests.
YouTube doesn't return currently a continuation, if applicable, for
Shorts UI playlists, restricting access to the 100th first items. The
reel items returned don't give also upload date, uploader info and
precise view count.

Using a continuation which requests the first page of the playlist
allows currently to get access to continuations, if applicable, and
also standard video elements instead of Shorts ones, making extraction
of upload date, uploader info and precise view count again possible.

This method is used for all playlist types, the original request is
still made, but now only returns what we need.

It requires to add a protocol buffer definition file, for which its
structure is based on reverse engineering of playlists continuations
sent by WEB InnerTube client, received from InnerTube responses.

Java classes of this file are generated for the Java Lite runtime of
Protobuf with the Protobuf Gradle plugin, as the lite version is enough
for our use cases. This plugin ships in JARs Protobuf definitions,
which should be avoided.

As Protobuf classes are parsed by Checkstyle checks and do not follow
our style rules at all, an exclusion rule has been for them.
…dItems

As continuations are now again available for playlists with Shorts UI, more
items' tests can be enabled.
@Stypox Stypox force-pushed the yt_continuations-playlists-shorts-ui branch from 62d0fac to a3e2d48 Compare July 16, 2025 11:43
@Stypox Stypox merged commit a9a4181 into TeamNewPipe:dev Jul 16, 2025
3 of 4 checks passed
@github-project-automation github-project-automation bot moved this from In Progress to Done in v0.28.x Jul 16, 2025
@Stypox
Copy link
Member

Stypox commented Jul 16, 2025

I rebased and fixed *.proto ending up in the JAR.

Before:

image

After:

image

@AudricV AudricV deleted the yt_continuations-playlists-shorts-ui branch July 16, 2025 13:16
FineFindus added a commit to FineFindus/NewPipe that referenced this pull request Jul 26, 2025
Fixes, an issue, where the proto-generated files where removed by
proguard, leading to an error.

Ref: TeamNewPipe/NewPipeExtractor#1104
whistlingwoods pushed a commit to whistlingwoods/NewPipeExtractor that referenced this pull request Aug 13, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request youtube service, https://www.youtube.com/

Projects

Status: Done

Development

Successfully merging this pull request may close these issues.

[YouTube] Shorts only playlists show up as empty [YT] Playlists that only contains shorts/ShortsUI does not return any items

2 participants