FWIW, I have the same issue with PyFtdi: how to run CI, for example with GitHub Actions...

I looked at various HW solutions: STM32, MSP430, even small FPGAs.

The problem here is that it requires a LOT of work to make a configurable remote device to simulate and/or catch errors, and be versatile enough to be reconfigured for each kind of tests. It would be definitely great, but it is nearly the same work effort than the project I want to test... I have not enough spare time to deal with both. For now, I kept this idea in mind as it would definitely be a interesting solution. Moreover to test PyUSB you would need a HS capable device to validate real use cases, which brings its own challenges.

Another issue is that it reduces the count of people that can run the tests, as they need the test HW, and the tools to program it. It is not something you can download and easily get up and running whenever you want to troubleshoot an issue, unfortunately.

I ended up writing a virtual backend so that it is possible to describe virtual USB devices using simple Yaml config files and test several APIs.

Obviously, it is not possible to test everything this way, but it gives much more confidence in the changes as it enables to catch many errors nonetheless.

This virtual backend is added on demand to the list of PyUSB backend candidates. This means that it is not possible to exercise the code of the existing backends, only the core and util modules of PyUSB - but I guess a similar solution could be implemented to test a backend by faking ctypes...

Yaml files describe the USB device hierarchy (device, configurations, interfaces, endpoints) and support multiple devices. The virtual backend then handles the control and bulk requests.

Vendor calls are redirected to the second emulation level which handles the FTDI HW API, so it can emulate the HW behavior. At this point the test code may access this virtual HW to check what’s going on and handle the virtualized IOs.

It’s still a work in progress but I already found several issues that have been silently hiding in PyFtdi for years, and I’ve been able to develop new features and test them first with the virtual HW then check how it behaves with the real one. The most challenging part will be to simulate the real time behavior of an actual HW.

A similar solution might be useful for PyUSB, at least to check some parts of its base code.

I’m interested in whatever solution is implemented to test PyUSB (HW, virtual, any other idea) as I think I have the same needs.

HTH,
Manu

pyusb test suite #235

Description

Activity

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions