Skip to content

Introduction

PyPI - Python Version PyPI - License PyPI Conda (channel only)

Docs Tests GitHub issues Codestyle black Changelog

Universal Pathlib is a Python library that extends the pathlib_abc.JoinablePath, pathlib_abc.Readable, and pathlib_abc.Writable API to give you a unified, Pythonic interface for working with files, whether they're on your local machine, in S3, on GitHub, or anywhere else. Built on top of filesystem_spec, it brings the convenienve of a pathlib.Path-like interface to cloud storage, remote filesystems, and more! ✨

If you enjoy working with Python's pathlib objects to operate on local file system paths, universal pathlib provides the same interface for many supported filesystem_spec implementations, from cloud-native object storage like Amazon's S3 Storage, Google Cloud Storage, Azure Blob Storage, to http, sftp, memory stores, and many more...

If you're familiar with filesystem_spec , then universal pathlib provides a convenient way to handle the path, protocol and storage options of a object stored on a fsspec filesystem in a single container (upath.UPath). And it further provides a pathlib interface to do path operations on the fsspec urlpath.

The great part is, if you're familiar with the pathlib.Path API, you can immediately switch from working with local paths to working on remote and virtual filesystem by simply using the UPath class:

# Local files: use pathlib
from pathlib import Path
local_file = Path("data/file.txt")
content = local_file.read_text()

# S3 files: use boto3/s3fs
import boto3
s3 = boto3.client('s3')
obj = s3.get_object(Bucket='bucket', Key='data/file.txt')
content = obj['Body'].read().decode('utf-8')

# Different APIs, different patterns 😫

# All files: use UPath! ✨
from upath import UPath

local_file = UPath("data/file.txt")
s3_file = UPath("s3://bucket/data/file.txt")

# Same API everywhere! 🎉
content = local_file.read_text()
content = s3_file.read_text()

Learn more about why you should use Universal Pathlib →

Quick Start 🚀

Installation

pip install universal-pathlib

Installing for specific filesystems

To use cloud storage or other remote filesystems, install the necessary fsspec extras:

pip install "universal-pathlib" "fsspec[s3,gcs,azure]"

See the Installation Guide for more details.

TL;DR Examples

from upath import UPath

# Works with local paths
local_path = UPath("documents/notes.txt")
local_path.write_text("Hello, World!")
print(local_path.read_text())  # "Hello, World!"

# Works with S3
s3_path = UPath("s3://my-bucket/data/processed/results.csv")
if s3_path.exists():
    data = s3_path.read_text()

# Works with HTTP
http_path = UPath("https://example.com/data/file.json")
if http_path.exists():
    content = http_path.read_bytes()

# Works with many more! 🌟

Currently supported filesystems

  • file: and local: Local filesystem
  • memory: Ephemeral filesystem in RAM
  • az:, adl:, abfs: and abfss: Azure Storage (requires adlfs)
  • data: RFC 2397 style data URLs (requires fsspec>=2023.12.2)
  • ftp: FTP filesystem
  • github: GitHub repository filesystem
  • http: and https: HTTP(S)-based filesystem
  • hdfs: Hadoop distributed filesystem
  • gs: and gcs: Google Cloud Storage (requires gcsfs)
  • hf: Hugging Face Hub (requires huggingface_hub)
  • s3: and s3a: AWS S3 (requires s3fs)
  • sftp: and ssh: SFTP and SSH filesystems (requires paramiko)
  • smb: SMB filesystems (requires smbprotocol)
  • webdav:, webdav+http: and webdav+https: WebDAV (requires webdav4[fsspec])

Untested Filesystems

Other fsspec-compatible filesystems likely work through the default implementation. If you encounter issues, please report it our issue tracker! We're happy to add official support!

Getting Help ❓

Need help? We're here for you!

Before Opening an Issue

Please check if your question has already been answered in the documentation or existing issues.

License 📃

Universal Pathlib is distributed under the MIT license, making it free and open source software. Use it freely in your projects!

Ready to get started?

Install Now