first commit

iflytek · Aug 25, 2022 · 4eeaf8a · 4eeaf8a
1 parent 372a75d
commit 4eeaf8a
Show file tree

Hide file tree

Showing 278 changed files with 20,726 additions and 128 deletions.
diff --git a/.gitignore b/.gitignore
@@ -1,129 +1,28 @@
-# Byte-compiled / optimized / DLL files
-__pycache__/
-*.py[cod]
-*$py.class
-
-# C extensions
-*.so
-
-# Distribution / packaging
-.Python
-build/
-develop-eggs/
-dist/
-downloads/
-eggs/
-.eggs/
-lib/
-lib64/
-parts/
-sdist/
-var/
-wheels/
-pip-wheel-metadata/
-share/python-wheels/
-*.egg-info/
-.installed.cfg
-*.egg
-MANIFEST
-
-# PyInstaller
-#  Usually these files are written by a python script from a template
-#  before PyInstaller builds the exe, so as to inject date/other infos into it.
-*.manifest
-*.spec
-
-# Installer logs
-pip-log.txt
-pip-delete-this-directory.txt
-
-# Unit test / coverage reports
-htmlcov/
-.tox/
-.nox/
-.coverage
-.coverage.*
-.cache
-nosetests.xml
-coverage.xml
-*.cover
-*.py,cover
-.hypothesis/
-.pytest_cache/
-
-# Translations
-*.mo
-*.pot
-
-# Django stuff:
-*.log
-local_settings.py
-db.sqlite3
-db.sqlite3-journal
-
-# Flask stuff:
-instance/
-.webassets-cache
-
-# Scrapy stuff:
-.scrapy
-
-# Sphinx documentation
-docs/_build/
-
-# PyBuilder
-target/
-
-# Jupyter Notebook
+# IFLearner file
+log
+doc/build
+doc/docs/api/reference
+dist
+build
+iflearner.egg-info
+examples/homo/**/data
+examples/homo/**/t10*
+examples/homo/**/train-*
+data
+examples/homo/**/client*
+iflearner/business/util/client*
 .ipynb_checkpoints
+*.log
 
-# IPython
-profile_default/
-ipython_config.py
-
-# pyenv
-.python-version
-
-# pipenv
-#   According to pypa/pipenv#598, it is recommended to include Pipfile.lock in version control.
-#   However, in case of collaboration, if having platform-specific dependencies or dependencies
-#   having no cross-platform support, pipenv may install dependencies that don't work, or not
-#   install all needed dependencies.
-#Pipfile.lock
-
-# PEP 582; used by e.g. github.com/David-OConnor/pyflow
-__pypackages__/
-
-# Celery stuff
-celerybeat-schedule
-celerybeat.pid
-
-# SageMath parsed files
-*.sage.py
-
-# Environments
-.env
-.venv
-env/
-venv/
-ENV/
-env.bak/
-venv.bak/
-
-# Spyder project settings
-.spyderproject
-.spyproject
-
-# Rope project settings
-.ropeproject
-
-# mkdocs documentation
-/site
+# macOS
+.DS_Store
 
-# mypy
-.mypy_cache/
-.dmypy.json
-dmypy.json
+# Editor
+.vscode
+.idea
 
-# Pyre type checker
-.pyre/
+# Byte-compiled / optimized / DLL files
+*.pyc
+__pycache__/
+*.py[cod]
+*$py.class
diff --git a/LICENSE b/LICENSE
@@ -186,7 +186,7 @@
       same "printed page" as the copyright notice for easier
       identification within third-party archives.
 
-   Copyright [yyyy] [name of copyright owner]
+   Copyright 2022 TURING
 
    Licensed under the Apache License, Version 2.0 (the "License");
    you may not use this file except in compliance with the License.

diff --git a/MANIFEST.in b/MANIFEST.in
@@ -0,0 +1 @@
+include requirements.txt
diff --git a/README.md b/README.md
@@ -1,2 +1,48 @@
-# iflearner
-A Powerful and Lightweight Federated Learning Framework
+![](https://img.shields.io/badge/language-python-blue.svg)
+![](https://img.shields.io/badge/license-Apache-000000.svg)
+![Docs](https://github.com/iflytek/iflearner/workflows/DeployDocs/badge.svg)
+
+# iFLearner - A Powerful and Lightweight Federated Learning Framework
+[DOCS](https://iflytek.github.io/iflearner/) | [中文](https://iflytek.github.io/iflearner/zh/)
+
+iFLearner is a federated learning framework, which provides a secure computing framework based on 
+data privacy security protection, mainly for federated modeling in deep learning scenarios. Its security bottom 
+layer supports various encryption technologies such as homomorphic encryption, secret sharing, and differential 
+privacy. The algorithm layer supports various deep learning network models, and supports mainstream frameworks 
+such as Tensorflow, Mxnet, and Pytorch.
+
+## Architecture
+![iFLeaner Arch](./doc/docs/images/iFLearner框架设计.jpg)
+
+The design of iFLearner is based on a few guiding principles:
+
+* **Event-driven mechanism**: Use an event-driven programming paradigm to build federated learning, that is, 
+  to regard federated learning as the process of sending and receiving messages between participants,
+  and describe the federated learning process by defining message types and the behavior of processing messages.
+
+* **Training framework abstraction**: Abstract deep learning backend, compatible with support for multiple 
+  types of framework backends such as Tensorflow and Pytorch.
+
+* **High scalability: modular design**, users can customize aggregation strategies, encryption modules,
+  and support algorithms in various scenarios.
+
+* **Lightweight and simple**: The framework is Lib level, light enough, and users can simply transform their deep 
+  learning algorithms into federated learning algorithms.
+
+## Documentation
+[iFLeaner Docs](https://iflytek.github.io/iflearner/):
+* [Installation](https://iflytek.github.io/iflearner/quick_start/installation/)
+* [Quickstart (TensorFlow)](https://iflytek.github.io/iflearner/quick_start/quickstart_tensorflow/)
+* [Quickstart (PyTorch)](https://iflytek.github.io/iflearner/quick_start/quickstart_pytorch/)
+* [Quickstart (MXNet)](https://iflytek.github.io/iflearner/quick_start/quickstart_mxnet/)
+* [Quickstart (keras)](https://iflytek.github.io/iflearner/quick_start/quickstart_keras/)
+
+## Contributor
+[iFLeaner FAQ](https://iflytek.github.io/iflearner/tutorial/contributor_guide/)
+
+## FAQ
+[iFLeaner FAQ](https://iflytek.github.io/iflearner/faq/faq/)
+
+## License
+[Apache License 2.0](LICENSE)
+
diff --git a/README_zh.md b/README_zh.md
@@ -0,0 +1,41 @@
+![](https://img.shields.io/badge/language-python-blue.svg)
+![](https://img.shields.io/badge/license-Apache-000000.svg)
+![Docs](https://github.com/iflytek/iflearner/workflows/DeployDocs/badge.svg)
+
+# iFLearner - 一个强大且轻量的联邦学习框架
+[DOCS](https://iflytek.github.io/iflearner/zh/) | [英文](https://iflytek.github.io/iflearner/)
+
+iFLearner是一个强大且轻量的联邦学习框架，提供了一种基于数据隐私安全保护的安全计算框架，
+主要针对深度学习场景下的联邦建模。其安全底层支持同态加密、秘密共享、差分隐私等多种加密技术，
+算法层支持各类深度学习网络模型，并且同时支持Tensorflow、Mxnet、Pytorch等主流框架。
+
+## 架构
+![iFLeaner Arch](./doc/docs/images/iFLearner框架设计.jpg)
+
+iFLearner主要基于以下原则进行设计:
+* **事件驱动机制**: 使用事件驱动的编程范式来构建联邦学习，即将联邦学习看成是参与方之间收发消息的过程，
+  通过定义消息类型以及处理消息的行为来描述联邦学习过程。
+
+* **训练框架抽象**: 抽象深度学习后端，兼容支持Tensorflow、Pytorch等多类框架后端。
+
+* **扩展性高**：模块式设计，用户可以自定义聚合策略，加密模块，同时支持各类场景下的算法。
+
+* **轻量且简单**：该框架Lib级别，足够轻量，同时用户可以简单改造自己的深度学习算法为联邦学习算法。
+
+
+## 文档
+[iFLeaner Docs](https://iflytek.github.io/iflearner/zh/):
+* [Installation](https://iflytek.github.io/iflearner/zh/quick_start/installation/)
+* [Quickstart (TensorFlow)](https://iflytek.github.io/iflearner/zh/quick_start/quickstart_tensorflow/)
+* [Quickstart (PyTorch)](https://iflytek.github.io/iflearner/zh/quick_start/quickstart_pytorch/)
+* [Quickstart (MXNet)](https://iflytek.github.io/iflearner/zh/quick_start/quickstart_mxnet/)
+* [Quickstart (keras)](https://iflytek.github.io/iflearner/zh/quick_start/quickstart_keras/)
+
+## Contributor
+[iFLeaner FAQ](https://iflytek.github.io/iflearner/zh/tutorial/contributor_guide/)
+
+## FAQ
+[iFLeaner FAQ](https://iflytek.github.io/zh/iflearner/faq/faq/)
+
+## License
+[Apache License 2.0](LICENSE)
diff --git a/dev/format.sh b/dev/format.sh
@@ -0,0 +1,14 @@
+#!/bin/bash
+set -e
+
+cd "$( cd "$( dirname "${BASH_SOURCE[0]}" )" >/dev/null 2>&1 && pwd )"/../
+
+# core
+python -m isort -rc iflearner/
+python -m black --config pyproject.toml --line-length 88 iflearner/
+python -m docformatter -i -r iflearner/
+
+# examples
+python -m isort -rc examples/
+python -m black examples/
+python -m docformatter -i -r examples/
diff --git a/dev/pack.sh b/dev/pack.sh
@@ -0,0 +1,9 @@
+#!/usr/bin/env bash
+set -e
+cd "$( cd "$( dirname "${BASH_SOURCE[0]}" )" >/dev/null 2>&1 && pwd )"/../
+
+rm -rf dist iflearner.egg-info || true
+python setup.py sdist
+
+# whl
+## python setup.py bdist_wheel
diff --git a/dev/publish.sh b/dev/publish.sh
@@ -0,0 +1,22 @@
+#!/usr/bin/env bash
+set -e
+cd "$( cd "$( dirname "${BASH_SOURCE[0]}" )" >/dev/null 2>&1 && pwd )"/../
+
+
+#First, Configture setuptools tools
+# Linux
+## vim ~/.pypirc
+# Windows
+## C:\Users\Username\.pypirc
+: <<'COMMENT'
+ [distutils]
+ index-servers=pypi
+
+ [pypi]
+ repository=https://upload.pypi.org/legacy/
+ username=
+ password=
+COMMENT
+
+# Second, Upload to pypi
+twine upload dist/* -r pypi
diff --git a/dev/test.sh b/dev/test.sh
@@ -0,0 +1,13 @@
+#!/bin/bash
+set -e
+cd "$( cd "$( dirname "${BASH_SOURCE[0]}" )" >/dev/null 2>&1 && pwd )"/../
+
+echo $PWD
+echo "=== test.sh ==="
+
+python -m isort iflearner/ examples/  && echo "- isort:         done" &&
+python -m docformatter -rc iflearner/  examples/   && echo "- docformatter:  done" &&
+python -m black  --check --line-length 88 iflearner/  examples/      && echo "- black:         done" &&
+python -m mypy  --config-file mypy.ini iflearner/     && echo "- mypy:          done" &&
+python -m flake8 iflearner/ && echo "- flake8:        done" &&
+echo "- All Python checks passed"
diff --git a/doc/README.md b/doc/README.md
@@ -0,0 +1,24 @@
+# introduce
+
+## install dependencies
+```shell
+pip install -r requirements.txt
+````
+
+## Modify the content of the document
+Built with the mkdocs tool, the document structure is defined in the mkdocs.yml file, and the specific content of the document is in the docs directory.
+
+This document is edited in markdown syntax. If new documents need to be added, please edit mkdocs.yaml and add chapters.
+
+## Local debugging documentation
+```shell
+mkdocs serve -a 127.0.0.1:8030
+````
+After executing the above command, you can view the content of the generated document through the http://127.0.0.1:8030 address.
+> When the document is modified, the page content will be updated automatically.
+
+## Local build documentation
+```shell
+mkdocs build
+````
+After executing the above command, the static files of the documentation site will be generated in the site directory, and the generated static files can be accessed by proxy.
diff --git a/doc/README_zh.md b/doc/README_zh.md
@@ -0,0 +1,27 @@
+# 介绍
+
+## 安装依赖
+```shell
+pip install -r requirements.txt
+```
+
+## 修改文档内容
+采用mkdocs工具构建，文档结构定义在 mkdocs.yml 文件中，文档的具体内容均在 docs 目录中。
+
+本文档采用 markdown 语法编辑，如有新的文档需要添加，请编辑 mkdocs.yaml 添加章节即可。 
+
+## 本地调试文档
+```shell
+mkdocs serve -a 127.0.0.1:8030
+```     
+执行上述命令后，可通过 http://127.0.0.1:8030 地址查看生成的文档内容.
+> 当修改文档后，页面内容会自动更新。
+
+## 本地构建文档
+```shell
+mkdocs build
+```
+执行上述命令后，会在 site 目录下生成文档站点的静态文件, 可将生成的静态文件进行代理访问。
+
+
+
diff --git a/doc/docs/about/changelog.md b/doc/docs/about/changelog.md
@@ -0,0 +1,10 @@
+# changelog
+All notable changes to this project will be recorded in this file.
+
+## iflearner
+## [Unreleased]
+#### Add
+- Support the underlying communication protocol of GRPC, and complete the abstraction of the upper layer protocol
+- Support access to deep learning frameworks such as Tensorflow, Pytorch, Mxnet, Keras, etc.
+- Support common aggregation strategies, and support users to customize their own aggregation strategies
+- Support SMPC, differential privacy security encryption strategy