@cragwolfe if we want to keep some tests in this repo, how do we maintain parity as new connectors are added and new tests for these connectors are added to the new repo? Will we need to update this repo to have those tests as well? This would only persist the tests that existed here before we moved the ingest work to the new repo.

@cragwolfe if we want to keep some tests in this repo, how do we maintain parity as new connectors are added and new tests for these connectors are added to the new repo? Will we need to update this repo to have those tests as well? This would only persist the tests that existed here before we moved the ingest work to the new repo.

yes, my main concern is not about adding connectors in the future, it is preserving the tests we have (which are basically two tests in one: output correctness and ingest smoke test). in practice, this means that new ingest tests which relied on new connectors wouldn't immediately be added to this repo, at least not until a pypi package was available with the new connector (or merged to main if we just want to clone the repo in tests). or, if we feel like the separate unstructured-ingest repo tests are sufficient end-to-end for the new connectors, we can skip adding them to this repo.

I advocate for using the ingest tests that have been moved over to the new ingest repo as a valid smoke test since the version of unstructured is pinned there for testing. This alleviated the burden of matching new patterns and connectors in both places as they get added to the new repo. If for any reason, the ingest tests break due to a bump in the unstructured repo, we can work to remedy that. This will also make the CI process in this repo more streamline for developers given the overhead running the ingest tests add.