📰 TL;DR • 🔎 Evaluation • 🧪 Training • 🔮 Synthetic Data Generation • 📜 Citation • 🙏 Acknowledgement
How to effectively and efficiently obtain code preferences and judgements is an important yet under-studied topic!
To this end, our work provides:
- CodeFavor: an open recipe to train code preference models with from-scratch data!
- Commit-Instruct: code commits -> code preference
- Critic-Evol: code critique & revising -> code preference
- CodePrefBench: 1364 code preference tasks covering both verifiable and human objectives:
- Code Correctness
- Code Efficiency
- Code Security
- Human Preference
- Study: our paper provides comprehensive studies!
- Human studies: quantifying the cost and performance of human preference based 18 developers
- Case studies: our Appendix case-studies model preferences over code correctness, efficiency, and security
- Controlled experiments: impact of data, comment, criteria, modeling, etc. on training preference models
- Python requirements: 3.10 or higher.
conda create -n codefavor python=3.10 -y
conda activate codefavor
pip install -r requirements.txt# OpenAI server
python codefavor/evaluate.py --model-id "gpt-4o-2024-05-13" --model-type openai --concurrency 80
# Other OpenAI-compatible servers (vLLM, DeepSeek APIs, etc.)
python codefavor/evaluate.py --model-id "google/gemma-2-27b-it" --model-type openai --concurrency 80 --model-url http://localhost:8000/v1
# Claude models via Bedrock
python codefavor/evaluate.py --model-id "anthropic.claude-3-sonnet-20240229-v1:0" --model-type bedrock --concurrency 10
# Pairwise RM
python codefavor/evaluate.py --model-id ./models/mix-cls-mistral-7b-it_bs32_ep1_lr5e-6-l3-70b/checkpoint-688 --model-type pair-rm- Supported
--model-type:huggingface,openai,bedrock,pair-rm, andgoogle
git clone https://github.com/axolotl-ai-cloud/axolotl.git axolotl-dep
cd axolotl-dep
pip install torch==2.3.0
pip install packaging ninja wandb
pip install -e '.[flash-attn,deepspeed]'python scripts/axolotl/prepare_data.py \
--decomposed-dataset datasets/train/editpackft-Llama-3-70B-Instruct.commit_instruct.decompose.jsonl \
--judge-type classification --both-order
python scripts/axolotl/prepare_data.py \
--decomposed-dataset datasets/train/Llama-3-8B-Instruct-SOSS.teacher.Llama-3-70B-Instruct.critic_evol.decompose.jsonl \
--judge-type classification --both-orderaccelerate launch -m axolotl.cli.train \
scripts/axolotl/recipe/gemma/cls-commit-instruct-from-llama3-70b.yaml \
--deepspeed scripts/axolotl/zero3.json
# or use `torchrun` if your `accelerate` is complaining
torchrun --nproc_per_node 8 -m axolotl.cli.train \
scripts/axolotl/recipe/gemma/cls-commit-instruct-from-llama3-70b.yaml \
--deepspeed scripts/axolotl/zero3.json# Support OpenAI and Bedrock interface
# OAI interface
python codefavor/prompt/commit_instruct.py --model-id "deepseek-chat" --model-type "openai" --concurrency 256 --dataset editpackft --model-url "https://api.deepseek.com/v1"
# Bedrock interface
python codefavor/prompt/commit_instruct.py --model-id "meta.llama3-1-405b-instruct-v1:0" --model-type "bedrock" --concurrency 10 --dataset editpackftpython codefavor/prompt/critic_evol.py --weak-dataset ./datasets/train/Llama-3-8B-Instruct-SOSS.jsonl \
--model-id "deepseek-coder" --model-url "https://api.deepseek.com/v1"
python codefavor/prompt/critic_evol.py --weak-dataset ./datasets/train/Llama-3-8B-Instruct-SOSS.jsonl \
--model-id "meta.llama3-1-405b-instruct-v1:0" --concurrency 10- Pairwise training code is partially adopted from https://github.com/RLHFlow/RLHF-Reward-Modeling/tree/main/pair-pm
@article{codefavor,
title = {Learning Code Preference via Synthetic Evolution},
author = {Liu, Jiawei and Nguyen, Thanh and Shang, Mingyue and Ding, Hantian and Li, Xiaopeng and Yu, Yu and Kumar, Varun and Wang, Zijian},
journal = {arXiv preprint arXiv:2410.03837},
year = {2024},
}- Our training code is partially adapted from RLHFlow.
- Our evaluation code is partially adapted from RepoQA.
- The seed corpus used in this paper comes from EditPackFT and Self-OSS-Instruct.
This source code is being released solely for academic and scientific reproducibility purposes, in support of the methods and findings described in the associated publication. Pull requests are not being accepted in order to maintain the code exactly as it was used in the paper, but interested parties are encouraged to open an issue requesting open source community development.