Pre-requisite for testing: When testing the latest release 1.0.0 on Windows, we cloned the repo and used Anaconda Navigator to launch a Notebook Server on the Local machine. We create a virtual environment for testing and periodically delete and recreate the virtual environment when we see error that don't make sense

The runtime platform is as follow:
Python 3.12.7
Windows-11-10.0.22631-SP0
3.12.7 | packaged by Anaconda, Inc. | (main, Oct 4 2024, 13:17:27) [MSC v.1929 64 bit (AMD64)]

All the transform use pip install of their respective module. for example:
When testing the doc_chunk transform with the python runtime, the following command is used:
pip install --user data-prep-toolkit-transforms[doc_chunck]
When testing the doc_chunk transform with the ray runtime, the following command is used:
pip install --user data-prep-tookit-transforms[ray,doc_chunk]

All tests use the Notebook provided by the transform developers in the transform folder. such as: used were taken from their respective transform such as data-prep-kit/transforms/language/doc_chunk/doc_chunk-python.ipynb.

When the developer provided a notebook for each of the runtimes (pythan and ray), both notebooks were tested.

Notebooks that have been tested and shown to work on Windows:

1. doc_chunk
2. doc_quality
3. html2parqet
4. pii_redactor
5. text_encoder
6. doc_id
7. ededup
8. filter
9. hap
10. profiler
11. resize
12. Tokenization

Notebook that have failed to run and for which an issue was created to be addressed in release 1.0.1:
pdf2parquet- Python Runtime: data-prep-kit/transforms/language/pdf2parquet/pdf2parquet-python.ipynb
pdf2parquet- Ray Runtime: data-prep-kit/transforms/language/pdf2parquet/pdf2parquet-ray.ipynb

Notebooks that have failed and require additional investigation for identifying the proper fix (Unable to provide estimate for fix release yet):
web2parquet: /data-prep-kit/transforms/universal/web2parquet/web2parquet.ipynb

Notebooks that have failed that require workaround from developer. Estimate fix coming with 1.0.1
Developers are currently required to modify the notebook and pip install fasttext-wheel
lang_id: /data-prep-kit/transforms/language/lang_id/lang_id.ipynb

lang_id depends on fasttext. Developers need to first install fasttext-wheel on Windows prior to importing this module.