ããã«ã¡ã¯ãããã¯ããã°ã§ã¯æ¥åã¨å ¨ãé¢ä¿ãªãMLãããã®ã好ããªã¨ã³ã¸ãã¢ãªã³ã°ã°ã«ã¼ãã¼ãã©ã«ããã¸ã£ã¼ã®å¤§å£ã§ãã
ç§ã¯æ®æ®µã¯å°±æ¥å¾ã¯å¦»ã¨ã¢ãã¡ã観ã¦ãããã¨ãå¤ããã§ãããå æ¥"å°é¢å¸«ãã¡"ã観ã¦ãããæã家ã§ã¯ãã©ããä¸æçãªãã¼ã ã¨ãªã£ã¦ãã¾ãã æè¿è¦ã¦ãã"å ¨é åç°å¸¸è§£æ±ºå®¤"ã§ã¯ãæ¥æ¬ç¥è©±ã®ç¥ã転çãã¦äººéã¨ãã¦æ®ããã¦ããã¨ãããã¢ãã¡ã£ã½ããã³ãã¢è¨å®ã§ã¡ãã£ã¨å¥½ãã§ãã ä¾ãã°ãã¢ã¡ãã¦ãºã¡ããã³ãã¯å¤©éããã¨ãã¦æ®ããã¦ãã¾ãã
ãããã夢ã®ãã話ãæ³åããã¨ãã¿ãªãããæ¥å¸¸ã«æ½ãç¥ãè¦ã¤ãåºããããªãã¾ãããï¼ã¨ãããã¨ã§ãä»åã¯å§åå¤æ師AIãä½ãã¾ãã ç§ã¯ãã£ã±ãã¢ãã¡ã好ããªã®ã§ãæ¥å¸¸ã®ä¸ã«æ½ãã§ããã¢ãã¡ãããååã¨ããã®ãè¦ã¤ãã«è¡ãã¾ãï¼
ãã¡ãã¯ã¨ã ã¹ãªã¼ Advent Calendar 2024 6æ¥ç®ã®è¨äºã§ããæ¨æ¥ã¯AIã»æ©æ¢°å¦ç¿ãã¼ã ã®ä¸æµ¦ããã®ãKubernetesã¨PyTorch Lightningã«ããå»çAIéçºç°å¢ã¨ãã®Tipsã§ããï¼
- å§åå¤æãã§ããã¨ã¯ã©ããããã¨ããåé¡è¨å®
- ç¹å¾´1: ã¾ãã¯1-gram/2-gramç¹å¾´ãçªã£è¾¼ãã§ãååã®çãããã¢ããªã³ã°
- ç¹å¾´2: ç»æ°ç¹å¾´
- çµæã¯?
- ã¡ãªã¿ã«å®å¨ã®äººéå士ã¯èå¥ã§ããã®ï¼
- We are hiring !!
å§åå¤æãã§ããã¨ã¯ã©ããããã¨ããåé¡è¨å®
å§åå¤æãã§ããã¨ã¯ã©ãããåé¡è¨å®ãã¨èããã¨ãã¤ã¾ããååã ããå ¥åã¨ãã¦ãã®äººã®å±æ§ãå¤æã§ãããã¨ããäºã«ãªãããã§ãã ä»åã¯ã¢ãã¡ãããååã®è¨ºæãä½ãããã®ã§ãã¤ã¾ããååã ããå ¥åã¨ãã¦ã¢ãã¡ãã£ã©ãã©ãããå¤å¥ã§ãããã¨ããåé¡ã解ãã¦ããç¶æ ã¨è¨ãããã§ãã
æ¬å½ã¯æ¥æ¬äººã®å ¨ååãæã£ã¦ãã¦ãã¢ãã¡ãã£ã©ã¨å¤å¥ã§ãããããããããã®ã§ãããæµç³ã«ãã®ãããªãã¼ã¿ã¯å ¬éããã¦ããããä»åã¯ã"è¡è°é¢è°å¡"ã"ãã©ã¤ã ä¸å ´ä¼æ¥ã®ä»£è¡¨"ã"ã¢ãã¡ãã£ã©"ã®3ãã¼ã¿ã»ããã使ã£ã¦èå¥åé¡ã解ãã¦ã¿ã¾ãããªããç°¡åããããã¼ã¿ãé¤ãããã«ãã¢ãã¡ãã£ã©ã®ãã¡ãååã®ãã¹ã¦ãæ¼¢åã§æ§æããã¦ãããã£ã©ã¯ã¿ã¼ã®ã¿ã対象ã¨ãã¾ãã
ç¹å¾´1: ã¾ãã¯1-gram/2-gramç¹å¾´ãçªã£è¾¼ãã§ãååã®çãããã¢ããªã³ã°
ã¢ãã¡ãã£ã©ãããã®ä»®èª¬ã¨ãã¦ãå®å¨ã®äººéã¨åºå¥ãããããããã»è¦ããããããããã«ãä½è ã¯çããååãã¤ããã¡ãªã®ã§ã¯ãã¨ãããã¨ãèãããã¾ãã
ããã§ãæ¼¢åã®é »åº¦ç¹å¾´ã¨ååã»ååã®é »åº¦ç¹å¾´ãçªã£è¾¼ãã°èå¥ç²¾åº¦ãé«ãããããã§ãããä»åéãããã¼ã¿ã§ã¯ååã¨ååãåå²ããã¦ããªããã®ãå¤ãã£ãã®ã§ãã·ã³ãã«ã«1-gram,2-gramã®one hot vectorãä½ããããããé »åº¦ä¸ä½100ã®ãã®ãåºç¾ãããã©ããã®200次å ã®ç¹å¾´ã¨ãã¾ããã
ç° 202 ä¸ 128 é 104 è¤ 87 é 81
太é 28 ä¸é 24 é´æ¨ 17 ä½è¤ 15 ç°ä¸ 13
ç¹å¾´2: ç»æ°ç¹å¾´
ãã¦ãå§åå¤æ師ã¨ããããã«ã¯ãåç»ã®æ å ±ãè¦ã¦å¤å®ã§ããªãã¨ãã¡ã§ããããããã§éè¦ã«ãªãã®ããæåæ å ±åºç¤æ´åäºæ¥ã§ãããªãã¨ãè¶ è©³ç´°ãªæ¼¢åã®æ å ±ãã¡ã¤ã«ãã¯ãªã¨ã¤ãã£ãã»ã³ã¢ã³ãºã§é å¸ãã¦ããã¦ããã®ã§ãï¼
ãã®è©³ç´°ãã¼ã¿ãå ã«ã
- ååã®ç»æ°ã®ç·å
- ååã®ç»æ°ã®æ大
- ååã®ç»æ°ã®æå°
- ååã®ç»æ°ã®æ¨æºåå·®
- 常ç¨ã¾ãã¯äººåæ¼¢åç
- 常ç¨ã¾ãã¯äººåæ¼¢åå¤æ°
- 常ç¨æ¼¢åç
- 常ç¨æ¼¢åå¤æ°
ã¨ããç¹å¾´éã追å ãã¾ããã
df_orig.assign(**{ 'raw_kanjidata': lambda _df: _df["name"].map(lambda name: kanji_data.merge(pd.DataFrame(list(name), columns=["font"]))) }).assign(**{ 'ç»æ°raw': lambda _df: _df["name"].map(lambda name: kanji_data.merge(pd.DataFrame(list(name), columns=["font"]))['ç·ç»æ°(åè)'].tolist()) }).assign(**{ 'ç»æ°mean': lambda _df: _df["ç»æ°raw"].map(lambda x: sum(x) / len(x)), 'ç»æ°max': lambda _df: _df["ç»æ°raw"].map(lambda x: max(x)), 'ç»æ°min': lambda _df: _df["ç»æ°raw"].map(lambda x: min(x)), 'ç»æ°std': lambda _df: _df["ç»æ°raw"].map(lambda x: pd.Series(x).std()), 'ç»æ°sum': lambda _df: _df["ç»æ°raw"].map(lambda x: sum(x)), }).assign(**{ '常ç¨äººåæ¼¢åç': lambda _df: _df["raw_kanjidata"].map(lambda kanjidata: kanjidata["æ¼¢åæ½ç"].isin(["常ç¨æ¼¢å", "人åç¨æ¼¢å"]).mean()), '常ç¨äººåæ¼¢åå¤æ°': lambda _df: _df["raw_kanjidata"].map(lambda kanjidata: (~kanjidata["æ¼¢åæ½ç"].isin(["常ç¨æ¼¢å", "人åç¨æ¼¢å"])).sum()), '常ç¨æ¼¢åå¤æ°': lambda _df: _df["raw_kanjidata"].map(lambda kanjidata: (kanjidata["æ¼¢åæ½ç"] != "常ç¨æ¼¢å").sum()), '常ç¨æ¼¢åç': lambda _df: _df["raw_kanjidata"].map(lambda kanjidata: (kanjidata["æ¼¢åæ½ç"] == "常ç¨æ¼¢å").mean()), '人åç¨æ¼¢åç': lambda _df: _df["raw_kanjidata"].map(lambda kanjidata: (kanjidata["æ¼¢åæ½ç"] == "人åç¨æ¼¢å").mean()), })
çµæã¯?
ãªãã¨ãååã ãããå ¥ãã¦ãªãã«ãããããããAUC=0.7ã®èå¥å¨ãä½æãããã¨ãåºæ¥ã¦ãã¾ãã¾ãã
æå¤ã¨ã§ããããã§ããã
æãã¢ãã¡ãããã®ããã¢ãã¡ãã£ã©ã©ã³ãã³ã°ãè¦ã¦ã¿ã¾ã
- æ¬å®ç å¤æ¥é½ (ãã¡ãè飾åºäºæå ¬ååæ´¾åºæ)
- æ¬å®ç 檸檬 (ãã¡ãè飾åºäºæå ¬ååæ´¾åºæ)
- æ¥åå°æ¬¡é (ãã£ããã³ç¿¼)
- 泥水次éé· (éé)
- çåå°åé (è±ã®æ ¶æ¬¡ *1 )
ãªãã¨ãè¦äºã«ã¸ã£ã³ãä½åã«ç¬å ããã¦ãã¾ãã¾ãããæ¼¢åã®ååã«çµã£ã¦ããã®ã§ããç¨åº¦å¤ãä½åãå¤ããªããã¡ã¨ãããããªè¦ç´ ãããã®ã§ãããããã¸ã£ã³ãã¯ã¢ãã¡ãã£ã©ã®ååã®æ¦å¿µãä½ã£ã¦ããã¨ãè¨ããã®ããããã¾ãããã
1ä½ã»2ä½ã¯ã¾ãã«ãé常ã®ååã§ã¯ä½¿ãããªããããªé£ããæ¼¢åã ã£ããã6æåã¨ãã£ãé·ãååã ã£ãããã¾ãã3,4,5ä½ã¯"é"ã®å(4,5ä½ã¯æ¬¡é)ãå ±éãã¦ã¾ãããè±ã®æ ¶æ¬¡ãééã®ããã«ãä¸å®æ´å²ç©è¦ç´ ãããããå¤é¢¨ãªååãã¢ã¤ã³ã³ã¨ãã¦ä½¿ãããã¡ã¨ãããã¨ãªãããã¾ãã
ã§ã¯ãéã«ãæ¬å½ã¯è¡è°é¢è°å¡ãªã®ã«ã誤ã£ã¦ã¢ãã¡ãã£ã©ã¨å¤å®ããã¦ãã¾ã£ããæãã¢ãã¡ãã£ã©ãããååã®è¡è°é¢è°å¡ã©ã³ãã³ã°
- è¾»æ¸ äºº (èªç±æ°ä¸»å )
- åæ¬ç«å¤ªé (èªç±æ°ä¸»å )
- è¼æ¨ç§å (å½æ°æ°ä¸»å )
- å°æ³é²æ¬¡é (èªç±æ°ä¸»å )
- æ¢ æè¡ (æ¥æ¬ç¶æ°ã®ä¼)
ãã¯ããéãå ¥ã£ã¦ãã¾ããããå°æ³é²æ¬¡éããã¯ã¢ãã¡ãã£ã©ä¸ä½ã®æ¬¡éãå«ãã§ã¾ãããããªãã¨ãªãå¾åãæ´ãã¾ãã
ã§ã¯ããå°ãæ·±å ããã¦è¦ãããã«ãAIåã¯ã©ãã«çç®ãã¦ãããã1ä½ã®è¾»æ¸ 人ãããä¾ã«è¦ã¦ã¿ã¾ãããã
ç»æ°ç¹å¾´ãå§åçä¸ä½ã§ãããã¯ãAIåãå§åå¤æã¯ç»æ°ã§ãããã§ããã¼ãminãå°ããã¦(人ã®åã2ç»)ãstdã大ãã(5,11,2ç»ã¨æ¯ãå¹ )ã¨ãããã¨ãã¢ãã¡ãããã®ç¹å¾´ã«ãªã£ã¦ãã¾ããç»æ°ã®ãã©ã³ã¹ãè¦ã¦ãããªãã¦æ¬å½ã«å§åå¤æã£ã½ãã§ããããããã«èãã¦ã¿ãã¨ã"æè¤ä¸"*2ã¿ãããªæ¯ãå¹ ã®å¤§ããååã¯ãªããç¾å®é¢ããã¦ã«ãã³ã¤ã¤ã§ãããã ã¾ããé¢ç½ãç¹å¾´ã¨ãã¦ã常ç¨æ¼¢åçã¨ããã®ãä¸ä½ã«æ¥ã¦ãã¾ããããã¯è¾»æ¸ 人ããã®è¾»ã1ã¤ç¹ã®ã¤ãã§ã常ç¨æ¼¢åã§ã¯ãªãããã§ãããããã«ã¢ãã¡ãã£ã©ããããæãã¾ããã
åç»ã®ãã©ã³ã¹ã®è©±ã¯é¢ç½ããã§ããã¡ãã£ã¨æã£ã¦ã¿ãã¨ã ãããè¦ã¦ã¿ãã¨ãéè²(éã¢ãã¡)ã¯ãã ã£ã¨éã¾ã£ã¦ããã®ã«å¯¾ãã¦ã赤è²(ã¢ãã¡)ã¯å ¨ä½ã«æ£ã£ã¦ãã¾ããã¤ã¾ãæå°ç»æ°ãå¤ã(é£ããæ¼¢åã°ãã)ãåæ£ã大ãã(åç»ãã¢ã³ãã©ã³ã¹)ãæå°ç»æ°ãå°ããããªã©ãããããã®æ¹åã«å°ã£ã¦ããã¨ã¢ãã¡ããããã§ã¦ããã¨ãããã¨ã§ãããç´å¾ã
ã¡ãªã¿ã«ã¢ãã¡ãããã¯ã¼ã¹ã㯠äºç©ä¿æ¨¹(capeta)ãä¸éè±å¹¸(èªç±æ°ä¸»å ) ã§ããããäºäººå ±ç¢ºãã«åç»ããã©ã³ã¹æ´ã£ã¦ã¾ãããæå°ç»æ°ãæ¨æºçãª4ã§ããã å ãã¦ãAIåã«ããã¨ãäºãä¸ãéã¨ããæ¼¢åã¯ã¢ãã¡ãã£ã©ããããªãã¿ããã§ããä¸é家ã®äºã¤åã¯ãããï¼
ã¡ãªã¿ã«å®å¨ã®äººéå士ã¯èå¥ã§ããã®ï¼
ãã£ããå®å¨ã®äººéã社é·ã¨æ¿æ²»å®¶ããã¤ãã¦ããã®ã ããããããã®ï¼ã¤ã®å±æ§ãéå®ã§ããã®ãè¦ã¦ã¿ããã§ãããã
çµæã¨ãã¦ã¯AUC=0.55ã¨ãå ¨ãåºæ¥ã¦ãªãã¨ã¯è¨ããªããã®ã®ãããªãé£ããã¨ãããã¨ããããã¾ããã ãããåºæ¥ã¦ãã¾ãã¨ãçã¾ãã§è·æ¥ã決ã¾ã£ã¦ãã¾ãã¨ãããã¨ãªã®ã§ãã§ãã«ããã®ã¯ãããã¨ã§ããã
ãã ãå¹ãã¦ããç¹å¾´ãã¤ã¶ãã«è¦ã¦ããã¨é¢ç½ããã¨ããããã¾ããå®ã¯äºæ³ãã¦ç¨æãã¦ããã§ããããã®èå¥ã«ã¯1-gramã®ç¹å¾´ãæãå¹ãã¾ã(ãããä»ã®ç¹å¾´ããããã¨ç²¾åº¦ãä¸ãã)
ãã¦ãæ¿æ²»å®¶ã«å¹ãæåã観ã¦ã¿ãã¨
"å"ã"太"ã"ä¸"ã"é"ããããã¯ååã«ä½¿ãããæ¼¢åã§ãããå¤é¢¨ãªååãã¤ãã家æãå¹ãã¨ããã®ã¯ãã¡ããã¡ãç´å¾ã§ããã¼ããã¨ã¯é·ç·é·å¥³ãç¶ãå¾åã®è¦ç´ ãï¼ï¼
åé¡ã¯"ç°"ã"æ¨"ã"è¤"ããããã¯ååã«ä½¿ãããæ¼¢åã§ããããããã¯è¦ªã決ãããã®ã§ã¯ãªãã®ã§æããã¦ãªããããç§ã¯å¤§æ²³ãã©ã"å ãåã¸"ã観ã¦ãã®ã§ãç¾ä»£ã«ããã¦ãè¤ãæ¿æ²»å®¶ã«åãç¶ããã¦ããã¨ããé°è¬è«ãèªãã¦æºè¶³ã§ã(ããã®ãï¼)(ååã®æ¹ã¯ãããªã«å¯ä¸åº¦ãé«ããªãããæ¬å½ã«é°è¬è«ãªã®ã§ãã¾ãçã«åããªãã§ãã ããw)
We are hiring !!
ã¨ã ã¹ãªã¼ã§ã¯ãæ¥å¸¸ã®ã©ããªçåã§ãæè¡ã§è§£æ±ºãããããªã®ã¼ã¯ãæè¿ãã¦ã¾ãï¼ æ°åã»ä¸éããããã®æ¡ç¨ã ãã§ãªããã«ã¸ã¥ã¢ã«é¢è«ãã¤ã³ã¿ã¼ã³ã常æåéãã¦ãã¾ãï¼ *3