Modernize packaging and test automation (GHA, Tox) #602

bittner · 2023-07-06T11:37:45Z

This package used to run the test suite with Travis CI, which was acquired in 2019 by Idera. Even though they are claiming to continue their support for Open Source projects the acquisition has made a lot of projects to move to the then-new GHA (GitHub Actions), which is integrated natively in GitHub nowadays.

Python packaging has moved focus to pyproject.toml, which is now officially endorsed and favored by the Python Packaging User Guide. The file has also become the home for (the configuration sections of) a lot of popular code quality and testing tools, such as black, ruff and pytest. Along with Pytest and Tox to run tests in an isolated fashion, this encompasses a modern Python packaging setup.

The future package causes an ImportError in certain situations with Python 3 (when used with projects having a src layout, and PYTHONPATH=src), which the six package seems not to do. We can avoid this issue in general by installing future (and using it in the source code) only for Python 2. For future reference, the specific error looks like below.

Installation error of future package (expand to see)

error: subprocess-exited-with-error
  
  × python setup.py egg_info did not run successfully.
  │ exit code: 1
  ╰─> [39 lines of output]
      Fatal Python error: initsite: Failed to import the site module
      Error processing line 1 of /.../lib/python3.7/site-packages/_virtualenv.pth:
      
      Traceback (most recent call last):
        File "/usr/local/lib/python3.7/site.py", line 168, in addpackage
          exec(line)
        File "<string>", line 1, in <module>
        File "/.../lib/python3.7/site-packages/_virtualenv.py", line 7, in <module>
          from contextlib import suppress
        File "/usr/local/lib/python3.7/contextlib.py", line 5, in <module>
          from collections import deque
        File "/usr/local/lib/python3.7/collections/__init__.py", line 27, in <module>
          from reprlib import recursive_repr as _recursive_repr
        File "/tmp/pip-install-gm5gpb34/future_0738a84497904a72a0b4996ecfb0b330/src/reprlib/__init__.py", line 7, in <module>
          raise ImportError('This package should not be accessible on Python 3. '
      ImportError: This package should not be accessible on Python 3. Either you are trying to run from the python-future src folder or your installation of python-future is corrupted.
      
      During handling of the above exception, another exception occurred:
      
      Traceback (most recent call last):
        File "/usr/local/lib/python3.7/site.py", line 579, in <module>
          main()
        File "/usr/local/lib/python3.7/site.py", line 562, in main
          known_paths = venv(known_paths)
        File "/usr/local/lib/python3.7/site.py", line 494, in venv
          addsitepackages(known_paths, [sys.prefix])
        File "/usr/local/lib/python3.7/site.py", line 349, in addsitepackages
          addsitedir(sitedir, known_paths)
        File "/usr/local/lib/python3.7/site.py", line 207, in addsitedir
          addpackage(sitedir, name, known_paths)
        File "/usr/local/lib/python3.7/site.py", line 178, in addpackage
          import traceback
        File "/usr/local/lib/python3.7/traceback.py", line 3, in <module>
          import collections
        File "/usr/local/lib/python3.7/collections/__init__.py", line 27, in <module>
          from reprlib import recursive_repr as _recursive_repr
        File "/tmp/pip-install-gm5gpb34/future_0738a84497904a72a0b4996ecfb0b330/src/reprlib/__init__.py", line 7, in <module>
          raise ImportError('This package should not be accessible on Python 3. '
      ImportError: This package should not be accessible on Python 3. Either you are trying to run from the python-future src folder or your installation of python-future is corrupted.
      [end of output]
  
  note: This error originates from a subprocess, and is likely not a problem with pip.
error: metadata-generation-failed
× Encountered error while generating package metadata.
╰─> See above for output.
note: This is an issue with the package mentioned above, not pip.
hint: See above for details.

Changes / Improvements

Use GitHub Actions (replacing the obsolete Travis CI)
Use Tox to run tests, linting and packaging jobs
Use modern pyproject.toml for packaging (replacing setup.{py,cfg})
Use six instead of future where possible (avoid side-effects with Python 3)
Install future for Python 2 only (avoid affecting Python 3 installations)
Exclude source code of tests from source distribution package

Test-drive this change

You can install autograd with the changes from this PR via, e.g.

pip install "autograd @ git+https://github.com/bittner/autograd@feature/modernize-packaging-tests"

... or with scipy as an optional dependency:

pip install "autograd[scipy] @ git+https://github.com/bittner/autograd@feature/modernize-packaging-tests"

Releases to PyPI, Codacy/Coveralls

@j-towns For GHA to publish new releases on PyPI when a Git tag is being pushed, you need to add two secret values to the project settings (Settings > Secrets and variables). The token is generated in your PyPI account settings.

PYPI_USERNAME
PYPI_PASSWORD

Similar for the Codacy (or Coveralls) integration, if you want to use it.

- Use GitHub Actions (replacing the obsolete Travis CI) - Use Tox to run tests, linting and packaging jobs - Use modern `pyproject.toml` for packaging (replacing `setup.{py,cfg}`) - Use `six` instead of `future` where possible (avoid side-effects with Python 3) - Install `future` for Python 2 only (avoid affecting Python 3 installations) - Exclude source code of tests from source distribution package

bittner · 2023-07-21T13:55:35Z

Any feedback on this? Are the changes fine? Any adaptions needed before this can be merged?

thatlittleboy · 2023-07-22T03:01:30Z

I stumbled onto this PR while locating the reason why future was being installed as part of my dependencies. I'll also appreciate if this PR could be reviewed and merged.

j-towns · 2023-07-22T13:03:24Z

Hi @bittner, thanks for this. It would be good to have CI working again. In principal the changes in this PR look ok, except that it doesn't really make sense for us to do code style checks, since Autograd's code has its own idiosyncratic style (which was a conscious decision). Could you remove the black checks from the GHA/tox configs, and the references to them in CONTRIBUTING.md? It might also make sense to configure ruff not to check style, if that is straightforward to do.

bittner · 2023-07-24T17:35:23Z

Could you remove the black checks from the GHA/tox configs, and the references to them in CONTRIBUTING.md?

Done.

It might also make sense to configure ruff not to check style, if that is straightforward to do.

I have added a configuration section for Ruff in pyproject.toml. Ruff is commented out in the GHA setup, though, so it doesn't run at all.

A separate PR for (activating) Ruff

I would suggest to make additional changes to the Ruff configuration in a separate PR, which also allows to discuss certain decisions. Some issues, for example, are likely not due to an idiosyncratic style but mistakes or leftovers (e.g. "variable imported but never unused", "variable assigned but never used") and some suggested changes may improve maintainability without affecting implemented features (e.g. unsorted imports, trailing whitespace on empty lines).

Afterwards, you may decide to activate linting in GHA, or remove Ruff entirely as a QA tool (it provides little value if we disable the majority of rules just for the sake of respecting the current baseline).

bittner · 2023-07-26T10:48:33Z

@j-towns Does that look good enough now for getting it merged? - Thanks for taking another look! 👀

j-towns · 2023-07-26T13:24:46Z

I have made a few changes and will attempt a merge now. The changes were:

Tox was running tests in an environment without Scipy. Ideally we would want GHA to test in environments with and without Scipy. For now though it would be ok to just test with Scipy, so I added Scipy to tox.ini.
Add myself as an author to pyproject.toml.
Remove codacy as I want to minimise the amount of time I spend on setup now and we can set that up later if desired.

j-towns · 2023-07-26T13:25:55Z

Thanks again for this pr!

j-towns · 2023-07-27T11:11:48Z

@bittner do you have the time/motivation to fix the failing tests? IMO it might be best to just drop the PyPy tests (I don't think we've ever claimed to support PyPy), but the others would be nice to fix. It also appears that GHA is now running a 'none' and a 'scipy' version of each test, but SciPy is being installed and used on both, which is not the intention.

bittner · 2023-07-28T10:26:37Z

do you have the time/motivation to fix the failing tests?

I'll give it a shot.

It also appears that GHA is now running a 'none' and a 'scipy' version of each test, but SciPy is being installed and used on both, which is not the intention.

I'll take a look at that, too.

bittner · 2023-07-28T12:39:42Z

In #602 (comment) above you write:

For now though it would be ok to just test with Scipy, so I added Scipy to tox.ini.

Re your observation:

It also appears that GHA is now running a 'none' and a 'scipy' version of each test, but SciPy is being installed and used on both, which is not the intention.

The pipeline setup is prepared in a way that it can run the tests with and without scipy (i.e. dependencies: scipy and dependencies: none). By removing the extras = scipy line in tox.ini this feature would be restored.

EDIT We could turn the pipeline's dependency installation code into actually installing the extras, that would be slightly harder to read but certainly be more accurate. You may also want to rename the "dependencies" variable to "extras" to be more blunt, e.g.

-     run: python -m pip install ${{ matrix.dependencies }}
+     run: python -m pip install .[${{ matrix.extras }}]

bittner · 2023-07-28T13:55:41Z

For some other issue, the GitHub hosted runners have started to remove older Python versions. The Ubuntu 20.04 VM image used to still ship Python 2.7 and 3.5 upwards, but supporting those Python versions seems to have been discontinued, finally.

I don't see an easy alternative. We either have to install older Python versions in the pipeline ourselves (e.g. using pyenv or similar) or accept the risk of having outdated Python versions not tested automatically.

bittner added 2 commits July 24, 2023 19:05

Remove Black as a QA tool from both Tox and GHA

03655ca

Add initial configuration for Ruff linter

3b8cb05

j-towns added 3 commits July 26, 2023 15:16

Tox run tests with scipy

9659398

Add j-towns to authors in pyproject.toml

8702d46

Rm codacy from GHA

3e1e95d

j-towns merged commit bd4f375 into HIPS:master Jul 26, 2023

bittner deleted the feature/modernize-packaging-tests branch July 28, 2023 10:27

bittner mentioned this pull request Jul 28, 2023

Fix GHA pipeline, install scipy using conda #604

Merged

bittner mentioned this pull request Aug 22, 2023

Install extras for testing via Tox, test with SciPy+PyPy only on Ubuntu #605

Merged

bittner mentioned this pull request Feb 14, 2024

Release new version 1.6.3 on PyPI #616

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Modernize packaging and test automation (GHA, Tox) #602

Modernize packaging and test automation (GHA, Tox) #602

Modernize packaging and test automation (GHA, Tox) #602

Modernize packaging and test automation (GHA, Tox) #602

Conversation

Changes / Improvements

Test-drive this change

Releases to PyPI, Codacy/Coveralls

A separate PR for (activating) Ruff