Geofileops is a python toolbox to process large vector files faster.
Most typical GIS operations are available: e.g. buffer, dissolve, erase/difference, intersection, union,...
The spatial operations are tested on geopackage and shapefile input files, but geopackage is recommended as it will give better performance. General layer and file operations can be used on the file formats supported by GDAL.
The full documentation is available on readthedocs.
Different techniques are used under the hood to be able to process large files as fast as possible:
- process data in batches
- subdivide/merge complex geometries on the fly
- process data in different passes
- use all available CPUs
The following chart gives an impression of the speed improvement that can be expected when processing larger files. The benchmarks typically use input file(s) with 500K polygons, ran on a Windows PC with 12 cores and include I/O.