feat: add file tree and s3 tree docs (#54)

tconbeer · Jan 22, 2024 · 0610998 · 0610998
1 parent ebac59d
commit 0610998
Show file tree

Hide file tree

Showing 10 changed files with 155 additions and 8 deletions.
diff --git a/src/docs/bindings.md b/src/docs/bindings.md
@@ -17,6 +17,7 @@ menuOrder: 200
 - <Key>F9</Key>, <Key>ctrl+b</Key> Toggle the sidebar.
 - <Key>F10</Key> Toggle full screen mode for the current widget.
 - <Key>ctrl+e</Key> Write the returned data to a CSV, Parquet, or JSON file.
+- <Key>ctrl+r</Key> Refresh the Data Catalog.
 
 ## Query Editor Bindings
 
@@ -83,8 +84,15 @@ _With the autocomplete list open:_
 
 ## Data Catalog Bindings
 
-- <Key>enter</Key> Select the current item.
-- <Key>space</Key> Toggle the expand/collapses state of the current item.
-- <Key>up</Key> Move the cursor up.
-- <Key>down</Key> Move the cursor down.
-- <Key>ctrl+enter</Key>, <Key>ctrl+j</Key> Insert the current name into the Query Editor.
+- <Key>ctrl+enter</Key>,<Key>ctrl+j</Key> Insert the current name into the Query Editor.
+- <Key>ctrl+c</Key> Copy the current name to the clipboard.
+
+### Switching Tabs
+
+- <Key>j</Key> Switch to the previous tab.
+- <Key>k</Key> Switch to the next tab.
+
+### Moving the Cursor
+
+- <Key>up</Key>,<Key>down</Key> Move the cursor one row.
+- <Key>enter</Key>,<Key>space</Key> Toggle the expand/collapsed state of the current item.
diff --git a/src/docs/config-file.md b/src/docs/config-file.md
@@ -1,6 +1,6 @@
 ---
 title: Using Config Files
-menuOrder: 3
+menuOrder: 6
 ---
 
 <script>

diff --git a/src/docs/files/index.md b/src/docs/files/index.md
@@ -0,0 +1,25 @@
+---
+title: Viewing Files
+menuOrder: 2
+---
+
+<script>
+    import Key from "$lib/components/key.svelte"
+    import file_tree from "$lib/assets/docs/file-tree.png"
+</script>
+
+Harlequin's Data Catalog optionally displays file trees for either local files or remote objects stored in Amazon S3 (or another object storage service that provides an S3-compatible API.)
+
+To view the file tree, use your mouse to select the "Files" tab, or focus on the Data Catalog with <Key>F6</Key> and then switch tabs with <Key>k</Key> or <Key>j</Key>. Insert file paths into the query editor with <Key>ctrl+enter</Key> or <Key>ctrl+j</Key>, or copy them to the clipboard with <Key>ctrl+c</Key>.
+
+<div class="flex flex-wrap justify-center py-2">
+    <figure>
+        <img src={file_tree} alt="Example of the file tree."  class="h-auto w-full max-h-80">
+        <figcaption class="text-center text-sm text-purple font-bold">Example of the file tree.</figcaption>
+    </figure>
+</div>
+
+Keep reading for:
+
+1. [Local Files](local)
+2. [Remote Objects (S3)](remote)
diff --git a/src/docs/files/local.md b/src/docs/files/local.md
@@ -0,0 +1,18 @@
+---
+title: Local Files
+menuOrder: 3
+---
+
+Harlequin's Data Catalog will show remote files in a second tab in the Data Catalog if Harlequin is initialized with the `--show-files` option (alias `-f`). `--show-files` takes an absolute or relative file path to a directory as its argument:
+
+For example, an absolute path:
+
+```bash
+harlequin --show-files /path/to/my/data
+```
+
+For the current working directory:
+
+```bash
+harlequin -f .
+```
diff --git a/src/docs/files/remote.md b/src/docs/files/remote.md
@@ -0,0 +1,88 @@
+---
+title: Remote Objects (S3)
+menuOrder: 4
+---
+
+## Installation
+
+Before viewing remote objects, you must install the `boto3` package in the same environment as Harlequin. You can do this by installing Harlequin with the `s3` extra:
+
+```bash
+pip install harlequin[s3]
+```
+
+## Compatibility
+
+Harlequin works with any object storage system that provides an S3-compatible API, including Amazon S3, Google Cloud Storage, and Minio; basically, if you can query the storage with `boto3` and manage credentials with the AWS CLI, Harlequin can display your files.
+
+## Authentication
+
+Harlequin relies on the AWS CLI credential toolchain for authentication. Specifically, it will attempt to connect to your storage using whatever credentials are currently active (or default) in your AWS CLI config, **including the region**. You can use the credentials from another profile by setting the `AWS_PROFILE` environment variable.
+
+For Google Cloud Storage please see [the docs](https://cloud.google.com/storage/docs/authentication/hmackeys) on the XML API and HMAC authentication. After generating HMAC Keys, you can use the AWS CLI (`aws configure`) to store these keys in a profile accessible to Harlequin.
+
+Your user must have `ListObjects` or the equivalent permission to view a bucket's objects in Harlequin.
+
+Currently this is not configurable; if that doesn't work for you, please [open an issue](https://github.com/tconbeer/harlequin/issues/new/choose).
+
+## Usage
+
+Harlequin will display the remote tree if it is initialized with the `--show-s3` option (alias `--s3`). This option takes a reference to remote object storage.
+
+### Displaying All Buckets
+
+Use Harlequin with `--show-s3 all` to display all Amazon S3 buckets that the authenticated user has access to. (This is not advised if you have access to millions of objects in S3):
+
+```bash
+harlequin --show-s3 all
+```
+
+For GCS or another endpoint that supports `ListBuckets`, provide the endpoint url without a path:
+
+```bash
+harlequin --show-s3 "https://storage.googleapis.com"
+```
+
+### Displaying a Single Bucket
+
+Use `--show-s3` with an argument that represents a bucket and (optionally) an endpoint url and key prefix:
+
+```bash
+harlequin --show-s3 my-bucket
+```
+
+```bash
+harlequin --show-s3 my-bucket/my-prefix
+```
+
+A one-liner to set the AWS Profile and connect to a GCS bucket, filtering for a prefix:
+
+```bash
+AWS_PROFILE=gcs harlequin --s3 "https://storage.googleapis.com/my-gcs-bucket/my-prefix"
+```
+
+Harlequin takes any of the following formats:
+
+```
+# Amazon S3 Formats
+all
+my-bucket
+my-bucket/my-prefix
+s3://my-bucket
+s3://my-bucket/my-prefix
+https://s3.amazonaws.com/my-bucket
+https://s3.amazonaws.com/my-bucket/my-prefix
+https://my-bucket.s3.amazonaws.com
+https://my-bucket.s3.amazonaws.com/my-prefix
+
+# Google Cloud Storage Formats
+https://storage.googleapis.com
+https://storage.googleapis.com/my-bucket
+https://storage.googleapis.com/my-bucket/my-prefix
+https://my-bucket.storage.googleapis.com
+https://my-bucket.storage.googleapis.com/my-prefix
+
+# Minio, AWS PrivateLink, etc.
+https://my-storage.com/my-bucket/
+https://my-storage.com/my-bucket/my-prefix
+```
diff --git a/src/docs/getting-started.md b/src/docs/getting-started.md
@@ -15,8 +15,16 @@ After installing Python 3.8 or above, install Harlequin using `pip` or `pipx` wi
 pipx install harlequin
 ```
 
+You may want to install Harlequin with one or more extras, which provide additional features like [additional database support](adapters) or [remote file viewing](files/remote). That would look like this:
+
+```bash
+pipx install harlequin[postgres,s3]
+```
+
 ## Using Harlequin with DuckDB
 
+Harlequin defaults to using its DuckDB database adapter.
+
 From any shell, to open one or more DuckDB database files:
 
 ```bash

diff --git a/src/docs/themes.md b/src/docs/themes.md
@@ -1,6 +1,6 @@
 ---
 title: Choosing a Theme
-menuOrder: 2
+menuOrder: 5
 ---
 
 You can set a theme for Harlequin, passing the name of any [Pygments style](https://pygments.org/styles/) to the `--theme` or `-t` option.

diff --git a/src/lib/assets/docs/file-tree.png b/src/lib/assets/docs/file-tree.png
diff --git a/src/lib/assets/features/catalog.png b/src/lib/assets/features/catalog.png
diff --git a/src/routes/features.svelte b/src/routes/features.svelte
@@ -15,7 +15,7 @@
   <li>
     <Feature
       title="Data Catalog"
-      body="View tables, columns, and their types across one or more attached databases."
+      body="View tables, columns, and their types across one or more attached databases. Optionally view local files or S3 objects."
       src={catalog}
     />
   </li>