WAIT: What Anime Is This?

Anime Scene Reverse Search Engine

27 May 2016

Tags: CBIR, Anime

soruly

https://github.com/soruly/whatanime.ga

https://github.com/soruly/whatanime.ga-WebExtension

@soruly

* The Motivation

.link https://anilist.co/forum/thread/1536/1 Anime Guessing Game

.link http://myanimelist.net/forum/?topicid=1404673 Help Identifying The Anime

.link https://yuki.la/wsr/111302 Know the anime but can't find the scene

Image reverse search engines

- [[https://images.google.com/][Google Image]] - Results are limited

- [[https://www.tineye.com/][TinEye]] - never works on anime

- [[https://iqdb.org][iqdb]] - tailored for doujin artwork, not anime

- [[https://saucenao.com/][SauceNAO]] - covers iqdb plus game CG

.caption tried to include anime many years ago but never works, see [[https://saucenao.com/status.html][database status]]

- [[https://whatanime.ga][whatanime.ga]] - tailored for anime

* Demo

.image 2016-05-whatanime.ga/demo.jpg _ 500

.caption search results on the left, anime info below is provided by [[https://anilist.co][anilist.co]]

* How does it work?

[[https://en.wikipedia.org/wiki/Content-based_image_retrieval][Content-based image retrieval (CBIR)]]

There is a long list of CBIR engine/libraries/projects on [[https://en.wikipedia.org/wiki/List_of_CBIR_engines][wikipedia]]

whatanime.ga uses [[http://www.lire-project.net/][Lire]] which has a [[https://bitbucket.org/dermotte/liresolr][solr plugin]]

Common image descriptors:

- Color Layout

- Edge Histogram

- Opponent Histogram

- ScalableColor

- etc.

whatanime.ga only uses Color Layout due to Hardware limitations

* Brief idea of Color Layout

- one of the [[http://mpeg.chiariglione.org/standards/mpeg-7/visual][MPEG-7 standard]]

- Raw Image -> Partition to 8x8 blocks -> take average color of each block -> convert color space to YCbCr -> DCT transform (quantize) -> Zigzag scanning

.image 2016-05-whatanime.ga/inko.jpg _ 240

.caption Raw image

.image 2016-05-whatanime.ga/inko64.png _ 240

.caption extracted image feature: FQYLBAQRFgoYFBANEBIQDw0QCw0PDxAeEhEQDhAfDQ8PEA8=

* Use CBIR for analyzing video

Raw Video -> FFmpeg extract all frames -> Lire extract image features -> deduplicate images with same image features with a running window of 12 frames -> append timestamp -> load into solr

.image 2016-05-whatanime.ga/extracted.png _ 680

.caption extracted image features after deduplication and appending timestamp

* Hash distribution of Color Layout on 15000 hours of Anime

.image 2016-05-whatanime.ga/histogram.jpg _ 800

.caption 14 Dec 2015 (blue) compared to 18 Apr 2016 (red). Graph shifted upwards evenly

* Hash distribution of Color Layout on 15000 hours of Anime

.image 2016-05-whatanime.ga/histogram.png _ 800

.caption hash sorted by popularity

.caption search is almost instant if it found a match under the 10000 line

* Some optimization on searching

whatanime.ga is targeted for almost exact match, not visually similar images

- Assumption: consider similarity < 90% as no match

- Stop searching once a good result (Similarity > 90%) is found

- No need to continue searching after 10% of extracted image feature is searched

- warmup data with [[https://github.com/hoytech/vmtouch][vmtouch]]

- cache high confidence search results in redis

- cache less confidence search results in redis too, but expire sooner

more detail posted on [[http://www.semanticmetadata.net/2016/03/04/lire-use-case-what-anime-is-this/][semanticmetadata.net]]

* Statistics

- 15600 hours video from 2395 anime series analyzed (~8.4TB)

.caption it took 2 quad core-i7 machines 2 months to complete

- 414 Million analyzed frames after deduplication, 76GB in solr

- Automatically analyze new animes

- indexing speed: ~65 seconds for an 24-minute 720p anime (~35s extract, ~25s hashing, ~5s dedup and transform)

- runs on an i7-4770 with 32GB RAM

- search time: varies between 0-40 seconds );

- 500+ users of its Chrome Extension ;)

- Referrals from partner website SauceNao (thanks [[https://twitter.com/SauceNAO][Xamayon]])

- Hundreds of fans sharing it on [[https://twitter.com/search?f=tweets&vertical=default&q=whatanime.ga&src=savs][Twitter]], [[http://www.plurk.com/w/#whatanime.ga?time=365][plurk]], [[https://www.reddit.com/r/newsokur/comments/4932zx/][reddit]], [[https://www.google.com/webhp?#q=site:vk.com+%22whatanime.ga%22][VK]]

* What's Next?

Kind of experiment right now

- badly written sourcecode

- misplaced anime not cleaned up yet

Keep a close eye on [[https://github.com/joshstar/AniList-API-Docs][AniList API]], it may launch APIv2 later this year

Develop a Firefox Add-on

Wait for facebook to approve the [[https://twitter.com/soruly/status/720633945189015553][Messenger Bot]]

Develop more IM bots, any ideas?

Other Potential usage:

- something like youtube contentID

- auto tagger

- advertise for commercial online anime websites / anime producers

What about search by audio?

- possible, [[https://github.com/worldveil/dejavu][library]] exists for this purpose

* Credit

Dr. Mathias Lux (LIRE Project)

[[http://www.lire-project.net]]

.caption Lux Mathias, Savvas A. Chatzichristofis. Lire: Lucene Image Retrieval – An Extensible Java CBIR Library. In proceedings of the 16th ACM International Conference on Multimedia, pp. 1085-1088, Vancouver, Canada, 2008

.caption [[http://www.morganclaypool.com/doi/abs/10.2200/S00468ED1V01Y201301ICR025][Visual Information Retrieval with Java and LIRE]]

*

*

*

*

*

*

*

*

*

*

*

*