Clone the repository

git clone [email protected]:tesera/pygypsy.git
cd pygypsy

Setup the commit hooks

ln -s "$(pwd)/git-hooks/pre-commit.sh" .git/hooks/pre-commit

• Fork the repository
• Clone your fork to your development machine
• Create a branch
• Make a change: update code, update docs, update tests, and update README if appropriate)
• Run the tests, linter, and build the package
• Submit a pull request against the dev branch of the upstream repository (the pull request will be against dev by default)
• Select a peer reviewer and schedule a review
• Conduct the review. - Please record as much as possible in the pull request using comments and review feature on github. - An outline of a good review procedure is given here.
• Revise pull request if necessary and continue reviewing-revising loop until reviewer(s) are satisfied.
• Code is merged into dev branch

Releases will occur when enough new features or fixes have been merged into the development branch. 'Enough' is at the discretion of the mainters.

Several development tasks are defined in docker-compose.yml

Note - the base directory is mounted to a volume in the container - /opt/pygypsy. This way you do not need to rebuild the image every time you change a file or requirement.

Start the dev container and setup a venv

docker-compose run dev
virtualenv venv -p python2.7
. venv/bin/activate
pip install -e .[dev,test,lint,docs]

Then it is possible to use the docker-compose tasks

docker-compose run test # run tests
docker-compose run docs # build docs
docker-compose run dev # run bash
docker-compose run lint # run linter

Any functional changes or additions should be tested. Add a test for your changes and update old tests, if required.

Run the tests as follows

docker-compose run test

# or, if set up with virtualenv outside of container
py.test -s -v tests/

To save time, it is often convenient only to test the sections of the code you are actively working. This can be done as follows

docker-compose run test bash # only run this if you are using docker

py.test -s -v tests/test_data_prep.py # test one file
py.test -s -v tests/test_data_prep.py::test_prep_plottable # test one function

Linting checks the code for style and bugs.

Run the linter as follows

docker-compose run lint

# or, if set up with virtualenv outside of container
bash bin/lint.sh

If you are familiar with pylint, you can use the pylint command directly.

All new code should satisfy the linting standards!

Documentation is built automatically from docstrings and hosted at pygypsy docs and administration is done at Read The Docs.

See http://www.sphinx-doc.org/en/stable/rest.html for the syntax

You can build the docs locally as follows:

docker-compose run docs

# or, if set up with virtualenv outside of container
cd docs
sphinx-apidoc -o ./source ../pygypsy
make html
make coverage

Commit hooks run automatically when committing to the repository for the following quality control items:

• debug breakpoints
• linting

You have to symlink from the commit hooks provided to your local git hooks directory as described in Getting Started:

You can override the commit hook by using the -n option when running git commit. This is however discouraged!

There are many strategies suitable for profiling.

A good initial strategy is to run cProfile on a script as follows:

python -m cProfile -s cumtime  "$(which pygypsy)" simulate data/raw_plottable.csv.prepped > profile.txt

It is very easy to be misled by the profiler, cProfile has 2x overhead, and there a multitude of possible solutions to performance issues from internal optimizations to better use of library functions.

Do not attempt to optimize unless it is absolutely necessary, and discuss your ideas with other developers before trying to implement them.

Cython is used and aviable for use for performance limiting areas of the code. Familiarize yourself with cython before revising / adding code which uses cython.

The implication of cython use is that the modules written in cython must be recompiled in order for changes in those modules to take effect (e.g. before running tests). This can be done in either of the following ways:

- `pip install -e .`
- `python setup.py build_ext --inplace`

Sometimes it is useful to inlcude the results of ad-hoc analyses of pygypsy's behaviour.

For this purpose, a directory called notebooks/ is available, where jupyter notebooks can be saved.

If you would like to do an ad-hoc analyses, the procedure is as follows

1. File an issue describing the problem to be solved/reason for the analysis
2. Create a branch for your analysis using the following format

<issue-number>-<desciption>

for example

#32-address-testing-findings

2. Ensure the analysis extras are installed

pip install -e .[analysis]

If you are using docker remember to first enter the docker container with docker-compose run dev.

3. Start the jupyter notebook server

docker-compose run --service-ports notebook # if using docker
jupyter notebook --notebook-dir notebooks # if not using docker

4. Create a notebook using the same name as was used for the branch
5. Conduct the analysis & revise the source code as necessary

Do not commit your data used in your analysis

Notebooks are /not/ a replacement for unit tests! It is required to make suitable unit tests for the finding of an analysis before a pull request associated with an analysis will be merged.

The env/dev.env file is not required for most developers. It is required to deploy on PyPI, update coveralls, and to run the S3 tests. PyPI and coveralls should never be used locall - they should only be used from the continuous integration service. If you would like to run the tests for data on S3, you can create a env/dev.env file in your clone with the appropriate variables. Make sure not to commit it to the repository!

GYPSY_BUCKET=secrets
AWS_ACCESS_KEY_ID=secrets
AWS_SECRET_ACCESS_KEY=secrets
AWS_REGION=secrets
PYPI_USER=secrets
PYPI_PASSWORD=secrets
COVERALLS_REPO_TOKEN=secrets

This repository is open - do not commit sensitive data if you do not want it to become publicly accessible!

The project continuious integration service authenticates against AWS, PyPI, and the coveralls service.

Credentials for those services are limited to this project and they are encrypted. However,they are available unencrypted in the continuous integration environment; maintainers be warned!

• Create a new branch named release-x.y.z from dev where:
  • x.y.z is the version increment using semantic versioning, familiarize with semantic versioning before doing a release
  • in short, x,y,z should be incrememnted for backwards incompatible public api changes, backwards compatible public api changes, and backwards compatible bug fixes
• Make sure all issues tagged with the release's milestone are closed or moved to a future milestone
• Make sure dependencies listed in setup.py are up to date, including their minimum versions
• Make sure tests are passing
• Update changelog with summary of changes since previous release
  • the command below can be used to get a list of changes since the previous release; summarize and prepend
  • git log `git describe --tags --abbrev=0`..HEAD --oneline
• Open pull request with target of master
• When pull request is merged, create a release on github
  • when this is done, a build will be released to PYPI via the CI service
• Merge master back to dev
• In Read The Docs, activate and build the documentation for the release
• Publish the pygypsy release on zenodo as described `here https://guides.github.com/activities/citable-code/>`__
• Once the new release is on PyPI, the `conda-forge feedstock

<https://github.com/conda-forge/pygypsy-feedstock>`__ conda-forge release should be updated to build and deploy for conda. This can be done by bumping the version in the meta.yaml file. You will also need to update the sha256 for the package, which can be obtained from pypi