Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Bug]: Manual update - raise ValueError("failed to add item to index") #803

Open
2 tasks done
Dontkickmi22 opened this issue Aug 29, 2024 · 1 comment
Open
2 tasks done
Labels
bug Something isn't working

Comments

@Dontkickmi22
Copy link

Dontkickmi22 commented Aug 29, 2024

I've read the documentation

Operating System

Docker (TA 0.4.10 / ES8.14.3 /Redis-Stack latest) on Debian ARM64

Your Bug Report

Describe the bug

manage.py ta_fix_channels stopped with ValueError("failed to add item to index") with certain channels.

Steps To Reproduce

  1. python manage.py ta_fix_channels entered in container console
  2. displays number of channels queued
  3. settings page displays reindexing channel progress
  4. container log display below errors :
    • failed to extract thumbnail, use fallback
    • {"type":"parse_exception","reason":"[value] required property is missing","property_name":"value","processor_type":"set"}
    • raised unexpected: ValueError('failed to add item to index')
    • ValueError: failed to add item to index
  5. manual update process stops.

Expected behavior

Process finish with no errors.

Relevant log output

Log from machine A:

[2024-08-29 09:59:28,049: WARNING/ForkPoolWorker-4] 1NYdNEABJdY: no remote match found
[2024-08-29 09:59:28,049: WARNING/ForkPoolWorker-4] UCM_i6qQqbPBGyyeaUrD_1KQ: nothing to update
[2024-08-29 09:59:33,062: WARNING/ForkPoolWorker-4] UC7ZCaVIHCBE5i0c5Yt7ha_g: get metadata from es
[2024-08-29 09:59:33,072: WARNING/ForkPoolWorker-4] UC7ZCaVIHCBE5i0c5Yt7ha_g: get metadata from youtube
[2024-08-29 09:59:33,939: WARNING/ForkPoolWorker-4] UC7ZCaVIHCBE5i0c5Yt7ha_g: download channel thumbnail
[2024-08-29 09:59:34,574: WARNING/ForkPoolWorker-4] UC7ZCaVIHCBE5i0c5Yt7ha_g: failed to extract thumbnail, use fallback
[2024-08-29 09:59:34,601: WARNING/ForkPoolWorker-4] UC7ZCaVIHCBE5i0c5Yt7ha_g: failed to extract thumbnail, use fallback
[2024-08-29 09:59:34,795: WARNING/ForkPoolWorker-4] {"error":{"root_cause":[{"type":"parse_exception","reason":"[value] required property is missing","property_name":"value","processor_type":"set"}],"type":"parse_exception","reason":"[value] required property is missing","property_name":"value","processor_type":"set"},"status":400}
[2024-08-29 09:59:34,795: WARNING/ForkPoolWorker-4] {'description': 'UC7ZCaVIHCBE5i0c5Yt7ha_g', 'processors': [{'set': {'field': 'channel.channel_active', 'value': True}}, {'set': {'field': 'channel.channel_description', 'value': ''}}, {'set': {'field': 'channel.channel_id', 'value': 'UC7ZCaVIHCBE5i0c5Yt7ha_g'}}, {'set': {'field': 'channel.channel_last_refresh', 'value': 1724896773}}, {'set': {'field': 'channel.channel_name', 'value': 'Sun Sun'}}, {'set': {'field': 'channel.channel_subs', 'value': None}}, {'set': {'field': 'channel.channel_subscribed', 'value': False}}, {'set': {'field': 'channel.channel_tags', 'value': False}}, {'set': {'field': 'channel.channel_banner_url', 'value': False}}, {'set': {'field': 'channel.channel_thumb_url', 'value': 'https://yt3.googleusercontent.com/ytc/AIdro_mJzkr4dE7b5Qp7ovo2jxAFjIY12WJXF0YhssQBG1Y=s900-c-k-c0x00ffffff-no-rj'}}, {'set': {'field': 'channel.channel_tvart_url', 'value': False}}, {'set': {'field': 'channel.channel_views', 'value': 0}}]}
[2024-08-29 09:59:34,811: WARNING/ForkPoolWorker-4] 2d5f3a38-c9f9-4c06-9c69-11c147865bfa Failed callback
[2024-08-29 09:59:34,826: ERROR/ForkPoolWorker-4] Task check_reindex[2d5f3a38-c9f9-4c06-9c69-11c147865bfa] raised unexpected: ValueError('failed to add item to index')
Traceback (most recent call last):
  File "/root/.local/lib/python3.11/site-packages/celery/app/trace.py", line 453, in trace_task
    R = retval = fun(*args, **kwargs)
                 ^^^^^^^^^^^^^^^^^^^^
  File "/root/.local/lib/python3.11/site-packages/celery/app/trace.py", line 736, in __protected_call__
    return self.run(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/home/tasks.py", line 203, in check_reindex
    handler.reindex_all()
  File "/app/home/src/index/reindex.py", line 276, in reindex_all
    self.reindex_type(name, index_config)
  File "/app/home/src/index/reindex.py", line 291, in reindex_type
    reindex(youtube_id)
  File "/app/home/src/index/reindex.py", line 378, in _reindex_single_channel
    channel.sync_to_videos()
  File "/app/home/src/index/channel.py", line 167, in sync_to_videos
    _, _ = ElasticWrap(ingest_path).put(data)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/home/src/es/connect.py", line 113, in put
    raise ValueError("failed to add item to index")
ValueError: failed to add item to index

====================================================================
Log from machine B
====================================================================
[2024-08-29 09:55:10,991: WARNING/ForkPoolWorker-4] UCAnjgApA-zlc7yNzNItwlBA: download channel thumbnail
[2024-08-29 09:55:16,213: WARNING/ForkPoolWorker-4] {"error":{"root_cause":[{"type":"parse_exception","reason":"[value] required property is missing","property_name":"value","processor_type":"set"}],"type":"parse_exception","reason":"[value] required property is missing","property_name":"value","processor_type":"set"},"status":400}
������
8-29 09:55:16,214: WARNING/ForkPoolWorker-4] {'description': 'UCAnjgApA-zlc7yNzNItwlBA', 'processors': [{'set': {'field': 'channel.channel_active', 'value': True}}, {'set': {'field': 'channel.channel_description', 'value': 'Welcome to Reol OFFICIAL CHANNEL\n\nReol Official Site:https://www.reol.jp/\n\n1993年11月9日生まれ。シンガーソングライターであり、自身のアーティスト活動全般をセルフ・プロデュースするマルチ・クリエイター。2018年、“Reol”名義で活動を開始。\n\n1stフルアルバム「事実上」をリリース後、アジアを含め全9都市を回る大規模なツアーを開催し大成功を収め、翌年リリースした「文明EP」はiTunes総合チャート1位・オリコン初登場7位と自身最高位を記録した。2020年1月には「App Store」のCMに起用された楽曲「HYPE MODE」が収録された最新アルバム「金字塔」を発売し、同年7月YouTubeがグローバル規模で実施するキャンペーン “Artist On The Rise” に日本人女性アーティストとして初選出。7月29日にボートレースCMソング「第六感」を配信リリース。8月16日には、初の無観客ライヴ「Reol Japan Tour 2020 ハーメルンの大号令 -接続編-」がYouTubeで生配信。\n\n現在、YouTube登録者数は140万人を突破。総再生回数は7億回を超え、国内外から人気を集めている。\n\nReol Twitter:https://twitter.com/RRReol\nReol Official Twitter:https://twitter.com/RRReol_official\nReol Instagram:https://www.instagram.com/rrreol999\n'}}, {'set': {'field': 'channel.channel_id', 'value': 'UCAnjgApA-zlc7yNzNItwlBA'}}, {'set': {'field': 'channel.channel_last_refresh', 'value': 1724896510}}, {'set': {'field': 'channel.channel_name', 'value': 'Reol'}}, {'set': {'field': 'channel.channel_subs', 'value': None}}, {'set': {'field': 'channel.channel_subscribed', 'value': True}}, {'set': {'field': 'channel.channel_tags', 'value': False}}, {'set': {'field': 'channel.channel_banner_url', 'value': 'https://yt3.googleusercontent.com/2nRR_cqnAWn-GDM__BioGhjWTeX5vExEgoAZso2c574cEd7H73K3ye6fgWqQmazbt9RzujSqUNM=w2560-fcrop64=1,00005a57ffffa5a8-k-c0xffffffff-no-nd-rj'}}, {'set': {'field': 'channel.channel_thumb_url', 'value': 'https://yt3.googleusercontent.com/gNuLgCbNpwVn_qZ9O5eIj_B7B2U-18WyQwUkcuKh8C4-0zbEkdi6BaThaxbM9ZlQ3dxWspmsM3I=s900-c-k-c0x00ffffff-no-rj'}}, {'set': {'field': 'channel.channel_tvart_url', 'value': 'https://yt3.googleusercontent.com/2nRR_cqnAWn-GDM__BioGhjWTeX5vExEgoAZso2c574cEd7H73K3ye6fgWqQmazbt9RzujSqUNM=s0'}}, {'set': {'field': 'channel.channel_views', 'value': 0}}]}
[2024-08-29 09:55:16,228: WARNING/ForkPoolWorker-4] 604a235b-e0ba-4136-9599-93e2de89b872 Failed callback
[2024-08-29 09:55:16,245: ERROR/ForkPoolWorker-4] Task check_reindex[604a235b-e0ba-4136-9599-93e2de89b872] raised unexpected: ValueError('failed to add item to index')
Traceback (most recent call last):
  File "/root/.local/lib/python3.11/site-packages/celery/app/trace.py", line 453, in trace_task
    R = retval = fun(*args, **kwargs)
                 ^^^^^^^^^^^^^^^^^^^^
  File "/root/.local/lib/python3.11/site-packages/celery/app/trace.py", line 736, in __protected_call__
    return self.run(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/home/tasks.py", line 203, in check_reindex
    handler.reindex_all()
  File "/app/home/src/index/reindex.py", line 276, in reindex_all
    self.reindex_type(name, index_config)
  File "/app/home/src/index/reindex.py", line 291, in reindex_type
    reindex(youtube_id)
  File "/app/home/src/index/reindex.py", line 378, in _reindex_single_channel
    channel.sync_to_videos()
  File "/app/home/src/index/channel.py", line 167, in sync_to_videos
    _, _ = ElasticWrap(ingest_path).put(data)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/home/src/es/connect.py", line 113, in put
    raise ValueError("failed to add item to index")
ValueError: failed to add item to index

Anything else?

skip troubled channel instead of a stop (halt)?

Thanks for your time.

@Walmarpole
Copy link

Walmarpole commented Sep 19, 2024

Same issue. I can confirm same as above with specific channels on 'latest' branch docker build. Shell log posted below:

2024-09-18 21:28:49 [2024-09-18 21:28:49,947: INFO/MainProcess] Task check_reindex[fc51500e-ccc5-4ecf-aca6-30feb8edc68f] received
2024-09-18 21:28:49 [2024-09-18 21:28:49,949: WARNING/ForkPoolWorker-32] check_reindex create callback
2024-09-18 21:28:49 [2024-09-18 21:28:49,952: WARNING/ForkPoolWorker-32] [task][check_reindex] reindex {'channel': ['UCzGrGrvf9g8CVVzh_LvGf-g']}
2024-09-18 21:28:49 [2024-09-18 21:28:49,968: WARNING/ForkPoolWorker-32] UCzGrGrvf9g8CVVzh_LvGf-g: get metadata from es
2024-09-18 21:28:49 [2024-09-18 21:28:49,974: WARNING/ForkPoolWorker-32] UCzGrGrvf9g8CVVzh_LvGf-g: get metadata from youtube
2024-09-18 21:28:52 [2024-09-18 21:28:52,810: WARNING/ForkPoolWorker-32] WARNING: [youtube] tLVVhfJqohU: nsig extraction failed: Some formats may be missing
2024-09-18 21:28:52          Install PhantomJS to workaround the issue. Please download it from https://phantomjs.org/download.html
2024-09-18 21:28:52          n = Nbc6T-IasNB0Ydh87 ; player = https://www.youtube.com/s/player/a9d81eca/player_ias.vflset/en_US/base.js
2024-09-18 21:28:54 [2024-09-18 21:28:54,084: WARNING/ForkPoolWorker-32] UCzGrGrvf9g8CVVzh_LvGf-g: download channel thumbnail
2024-09-18 21:28:56 [2024-09-18 21:28:56,414: WARNING/ForkPoolWorker-32] {"error":{"root_cause":[{"type":"parse_exception","reason":"[value] required property is missing","property_name":"value","processor_type":"set"}],"type":"parse_exception","reason":"[value] required property is missing","property_name":"value","processor_type":"set"},"status":400}
2024-09-18 21:28:56 [2024-09-18 21:28:56,414: WARNING/ForkPoolWorker-32] {'description': 'UCzGrGrvf9g8CVVzh_LvGf-g', 'processors': [{'set': {'field': 'channel.channel_active', 'value': True}}, {'set': {'field': 'channel.channel_description', 'value': 'Nirvana on Vevo - Official Music Videos, Live Performances, Interviews and more...'}}, {'set': {'field': 'channel.channel_id', 'value': 'UCzGrGrvf9g8CVVzh_LvGf-g'}}, {'set': {'field': 'channel.channel_last_refresh', 'value': 1726720134}}, {'set': {'field': 'channel.channel_name', 'value': 'NirvanaVEVO'}}, {'set': {'field': 'channel.channel_subs', 'value': None}}, {'set': {'field': 'channel.channel_subscribed', 'value': False}}, {'set': {'field': 'channel.channel_tags', 'value': ['vevo music video official nirvana']}}, {'set': {'field': 'channel.channel_banner_url', 'value': 'https://yt3.googleusercontent.com/icXgmwCoMtg49hJBRlRnjmScm4-_dkDZAX6ghByMNU5XDfqluUx4ItdL_1vmRKARAi6fbB-kVWI=w2560-fcrop64=1,00005a57ffffa5a8-k-c0xffffffff-no-nd-rj'}}, {'set': {'field': 'channel.channel_thumb_url', 'value': 'https://yt3.googleusercontent.com/ytc/AIdro_nEzankJ2oekN31HOpZjk5u-eEyCyLcxEjvL1bWrYjyv5M=s900-c-k-c0x00ffffff-no-rj'}}, {'set': {'field': 'channel.channel_tvart_url', 'value': 'https://yt3.googleusercontent.com/icXgmwCoMtg49hJBRlRnjmScm4-_dkDZAX6ghByMNU5XDfqluUx4ItdL_1vmRKARAi6fbB-kVWI=s0'}}, {'set': {'field': 'channel.channel_views', 'value': 0}}]}
2024-09-18 21:28:56 [2024-09-18 21:28:56,418: WARNING/ForkPoolWorker-32] fc51500e-ccc5-4ecf-aca6-30feb8edc68f Failed callback
2024-09-18 21:28:56 [2024-09-18 21:28:56,420: ERROR/ForkPoolWorker-32] Task check_reindex[fc51500e-ccc5-4ecf-aca6-30feb8edc68f] raised unexpected: ValueError('failed to add item to index')
2024-09-18 21:28:56 Traceback (most recent call last):
2024-09-18 21:28:56   File "/root/.local/lib/python3.11/site-packages/celery/app/trace.py", line 453, in trace_task
2024-09-18 21:28:56     R = retval = fun(*args, **kwargs)
2024-09-18 21:28:56                  ^^^^^^^^^^^^^^^^^^^^
2024-09-18 21:28:56   File "/root/.local/lib/python3.11/site-packages/celery/app/trace.py", line 736, in __protected_call__
2024-09-18 21:28:56     return self.run(*args, **kwargs)
2024-09-18 21:28:56            ^^^^^^^^^^^^^^^^^^^^^^^^^
2024-09-18 21:28:56   File "/app/home/tasks.py", line 203, in check_reindex
2024-09-18 21:28:56     handler.reindex_all()
2024-09-18 21:28:56   File "/app/home/src/index/reindex.py", line 276, in reindex_all
2024-09-18 21:28:56     self.reindex_type(name, index_config)
2024-09-18 21:28:56   File "/app/home/src/index/reindex.py", line 291, in reindex_type
2024-09-18 21:28:56     reindex(youtube_id)
2024-09-18 21:28:56   File "/app/home/src/index/reindex.py", line 378, in _reindex_single_channel
2024-09-18 21:28:56     channel.sync_to_videos()
2024-09-18 21:28:56   File "/app/home/src/index/channel.py", line 167, in sync_to_videos
2024-09-18 21:28:56     _, _ = ElasticWrap(ingest_path).put(data)
2024-09-18 21:28:56            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-09-18 21:28:56   File "/app/home/src/es/connect.py", line 113, in put
2024-09-18 21:28:56     raise ValueError("failed to add item to index")
2024-09-18 21:28:56 ValueError: failed to add item to index
2024-09-18 21:28:56 [2024-09-18 21:28:56,421: WARNING/ForkPoolWorker-32] fc51500e-ccc5-4ecf-aca6-30feb8edc68f return callback

@bbilly1 bbilly1 added the bug Something isn't working label Sep 22, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

3 participants