change references, examples from "credential" to "connection". Update…

…d run/connection examples with new connection type syntax
turbot · vhadianto · Oct 22, 2024 · Sep 18, 2024 · Sep 18, 2024 · Sep 18, 2024
commit e1bfd37a3784a14ef36edbd59162e920124c47ac
diff --git a/docs/build/input.md b/docs/build/input.md
@@ -7,7 +7,7 @@ sidebar_label: Ask for Input
 
 Flowpipe allows you to optimize and automate your operational and business processes, but there are times when human interaction is required.  For example, a manager may be required to approve a privilege escalation request.  Furthermore, we live in an era of constant communication across multiple channels.  People prefer to use specific tools (Email, Slack, etc.) for communication and expect these interactions to occur in their tools.  The Flowpipe `input` step primitive provides the ability to prompt for human input via multiple channels and wait for a response.
 
-To interact with the various tools and services, Flowpipe also includes an `integration` configuration resource.  This allows you to configure Flowpipe for 2-way communication with your tool (Slack, email, etc).  `integration` is a configuration resource (defined in `.fpc` files) like `credential`, thus the configuration is installation level, not mod level; you can define the integrations for your installation once and refer to them from any mods.
+To interact with the various tools and services, Flowpipe also includes an `integration` configuration resource.  This allows you to configure Flowpipe for 2-way communication with your tool (Slack, email, etc).  `integration` is a configuration resource (defined in `.fpc` files) like `connection`, thus the configuration is installation level, not mod level; you can define the integrations for your installation once and refer to them from any mods.
 
 Sending notifications is a common pattern, and often users will want to route a request to more than one user, group, or channel, and via more than one delivery mechanism.  For instance, you may want to request approval via Slack AND email.  The `notifier` resource allows you to define a list of integrations to send notifications to. Like `integration`, a `notifier` is an installation-level configuration resource.
 

diff --git a/docs/flowpipe-hcl/variable.md b/docs/flowpipe-hcl/variable.md
@@ -32,7 +32,7 @@ variable "secret_access_key" {
 
 variable "session_token" {
   type        = string
-  description = "The token that users must pass to the service API to use the temporary credentials."
+  description = "The token that users must pass to the service API to use the temporary connections."
   default     = ""
 }
 

diff --git a/docs/reference/cli/index.md b/docs/reference/cli/index.md
@@ -39,7 +39,7 @@ sidebar_label: Flowpipe CLI
   <tr> 
     <td nowrap="true"> <inlineCode>--config-path</inlineCode> </td> 
     <td>  
-    Sets the search path for <a href = "/docs/reference/config-files">configuration files</a>. This argument accepts a colon-separated list of directories.  All configuration files (<inlineCode>*.fpc</inlineCode>) will be loaded from each path, with decreasing precedence.  The default is <inlineCode>.:$FLOWPIPE_INSTALL_DIR/config</inlineCode> (<inlineCode>.:~/.flowpipe/config</inlineCode>).  This allows you to manage your <a href="/docs/reference/config-files/workspace"> workspaces </a> and <a href="/docs/reference/config-files/credential">credentials</a> centrally in the <inlineCode>~/.flowpipe/config</inlineCode> directory, but override them in the working directory / mod location if desired.
+    Sets the search path for <a href = "/docs/reference/config-files">configuration files</a>. This argument accepts a colon-separated list of directories.  All configuration files (<inlineCode>*.fpc</inlineCode>) will be loaded from each path, with decreasing precedence.  The default is <inlineCode>.:$FLOWPIPE_INSTALL_DIR/config</inlineCode> (<inlineCode>.:~/.flowpipe/config</inlineCode>).  This allows you to manage your <a href="/docs/reference/config-files/workspace"> workspaces </a> and <a href="/docs/reference/config-files/connection">connections</a> centrally in the <inlineCode>~/.flowpipe/config</inlineCode> directory, but override them in the working directory / mod location if desired.
     </td> 
   </tr>
 

diff --git a/docs/reference/config-files/index.md b/docs/reference/config-files/index.md
@@ -5,6 +5,6 @@ sidebar_label: Configuration Files
 
 # Configuration Files
 
-Configuration resources like [workspaces](/docs/reference/config-files/workspace) and [credentials](/docs/reference/config-files/credential) are defined using HCL in one or more Flowpipe config files.  
+Configuration resources like [workspaces](/docs/reference/config-files/workspace) and [connections](/docs/reference/config-files/connection) are defined using HCL in one or more Flowpipe config files.  
 
-Flowpipe will load ALL configuration files (`*.fpc`) from every directory in the [configuration search path](/docs/reference/env-vars/flowpipe_config_path), with decreasing precedence.  By default, the configuration search path includes the current directory (or `--mod-location` if specified) followed by the `config` directory in the [FLOWPIPE_INSTALL_DIR](/docs/reference/env-vars/flowpipe_install_dir): `.:$FLOWPIPE_INSTALL_DIR/config`.  This allows you to manage your workspaces and credentials centrally in the `~/.flowpipe/config` directory, but override them in the working directory / mod location if desired.
+Flowpipe will load ALL configuration files (`*.fpc`) from every directory in the [configuration search path](/docs/reference/env-vars/flowpipe_config_path), with decreasing precedence.  By default, the configuration search path includes the current directory (or `--mod-location` if specified) followed by the `config` directory in the [FLOWPIPE_INSTALL_DIR](/docs/reference/env-vars/flowpipe_install_dir): `.:$FLOWPIPE_INSTALL_DIR/config`.  This allows you to manage your workspaces and connections centrally in the `~/.flowpipe/config` directory, but override them in the working directory / mod location if desired.
diff --git a/docs/reference/env-vars/flowpipe_config_path.md b/docs/reference/env-vars/flowpipe_config_path.md
@@ -7,7 +7,7 @@ sidebar_label: FLOWPIPE_CONFIG_PATH
 
  Sets the search path for [configuration files](/docs/reference/config-files).  `FLOWPIPE_CONFIG_PATH` accepts a colon-separated list of directories.  
 
- All configuration files (`*.fpc`) will be loaded from each path, with decreasing precedence.  The default is the mod location, followed by the `config` directory in the [FLOWPIPE_INSTALL_DIR](/docs/reference/env-vars/flowpipe_install_dir): `.:$FLOWPIPE_INSTALL_DIR/config`.  This allows you to manage your [workspaces](/docs/reference/config-files/workspace) and [credentials](/docs/reference/config-files/credential) centrally in the `~/.flowpipe/config` directory, but override them in the working directory/mod-location if desired.
+ All configuration files (`*.fpc`) will be loaded from each path, with decreasing precedence.  The default is the mod location, followed by the `config` directory in the [FLOWPIPE_INSTALL_DIR](/docs/reference/env-vars/flowpipe_install_dir): `.:$FLOWPIPE_INSTALL_DIR/config`.  This allows you to manage your [workspaces](/docs/reference/config-files/workspace) and [connections](/docs/reference/config-files/connection) centrally in the `~/.flowpipe/config` directory, but override them in the working directory/mod-location if desired.
 
 
 ## Usage

diff --git a/docs/run/connections.md b/docs/run/connections.md
@@ -0,0 +1,213 @@
+---
+title: Managing Connections
+sidebar_label: Connections
+---
+
+# Managing Connections
+
+Flowpipe **Connections** provide a mechanism for defining and sharing secrets in your Flowpipe environment.
+
+
+## Defining / Declaring  Connections
+`connection` is a top-level resource in a config file (.fpc). Connections are defined in configuration files (`.fpc`), NOT mod files (`.fp`).  Like `workspace`, a `connection` can be defined in any .fpc file in the config directory (`~/.flowpipe/config/*.fpc`) or in the root of the `mod-location` (usually `$PWD/*.fpc`). 
+
+While it is *possible* to add connections in `.fpc` in a mod repo, you should usually avoid doing so:
+  - They often contain secrets
+  - Different users will have different secrets - you should not share connections
+
+Connections are defined in a `connection` block.  They have a type label and a name.
+
+```hcl
+connection "aws" "aws_01" {
+  access_key = "ASIAQGDFAKEKGUI5MCEU"
+  secret_key = "QhLNLGM5MBkXiZm2k2tfake+TduEaCkCdpCSLl6U"
+}
+```
+
+There is a type for each service:  `aws`, `gcp`, `azure`, `github`, `slack`, etc...  The arguments and attributes vary by type. 
+
+```hcl
+connection "gcp" "gcp_01" {
+  credentials    = "~/.config/gcloud/application_default_credentials.json"
+}
+connection "slack" "slack" {
+  token    = "xoxp-asd7f8fd0fake9890"
+}
+```
+
+
+You can use the `env()` function to set connections to the value of an environment variable:
+```hcl
+connection "slack" "default" {
+  token    = env("SLACK_TOKEN")
+}
+```
+
+### Default connections
+Flowpipe also creates a single **implicit default** connection for each connection type.  The connection is named `default`, eg `connection.aws.default`, `connection.github.default`, etc.  The intent of the default connection is that Flowpipe can "just work" with your existing setup, and is similar to the default connection that Steampipe creates for each plugin.  For example, the AWS default connection will use the standard AWS SDK credential resolution mechanism to create a Flowpipe connection that will use the same credential that the `aws` CLI would use. 
+
+You can override the default by simply creating a connection for that type that is named `default`:
+
+```hcl
+connection "slack" "default" {
+  token    = "xoxp-mydefaulttoken"
+}
+```
+
+----
+
+## Using / Referencing Connections 
+
+Although connections are defined in config files (not mod files), Flowpipe makes all the connections available to mods using the `connection` variable.  This special variable is a map of all connection types by type and name, so you can reference it with standard HCL syntax: `connection.{type}.{name}` eg `connection.aws.aws01` or `connection[{type}][{name}]` eg `connection["aws"["aws01"]`.
+
+You can reference them directly in a mod:
+```hcl
+pipeline "ex1" {
+  step "container" "aws" {
+    image = "public.ecr.aws/aws-cli/aws-cli"
+    cmd   = [ "s3", "ls" ]
+    env   = connection.aws.aws01.env
+  } 
+}
+```
+
+Since they are a map, you can even loop through them:
+
+```hcl
+pipeline "ex1" {
+  step "container" "aws" {
+    for_each = connection.aws
+    image    = "public.ecr.aws/aws-cli/aws-cli"
+    cmd      = [ "s3", "ls" ]
+    env      = connection.aws[each.value].env
+  } 
+}
+```
+
+Or reference them dynamically:
+<!--
+ - this can be especially useful when [importing](#importing-connections) Steampipe connections:
+-->
+
+```hcl
+pipeline "ex1" {
+  step "query" "get_running_instances" {
+    sql = <<-EOQ
+      select
+        instance_id,
+        region,
+        account_id,
+        _ctx ->> 'connection_name' as connection_name,
+        instance_state,
+        tags
+      from
+        aws_ec2_instance
+      where
+        instance_state == 'running'
+        and tags ->> 'env' like 'dev%'
+    EOQ
+  }
+
+  step "container" "stop_instances" {
+    for_each = step.query.get_running_instances.rows
+    image    = "public.ecr.aws/aws-cli/aws-cli"
+    cmd     = [ "ec2", "stop-instances" 
+                "--instance-ids", each.value["instance_id"
+                "--region", each.value["region"]]
+    ]
+    env   = connection.aws[each.value.connection_name].env
+  } 
+}
+```
+
+### Passing connections
+Generally, referencing connections literally by name in a mod is bad practice, especially in library mods because the details are instance-specific - You are making assumptions about what connections have been set up, including the names of those connections.  Instead, you should generally pass in the connection as a pipeline parameter. 
+
+Flowpipe supports `connection` as a `type` for `param` and `variable`, in addition to the builtin `string`, `number`, and `bool` types.  You may define the type as `connection`, in which case the variable or param accepts any connection, or `connection.{type}` (e.g `connection.aws`) to limit the types of connections that may be passed.  
+
+By convention, you should set the default value of the variable to the [default connection](#default-connections) so that your mod will work even if the user does not pass the argument or variable.
+
+
+
+
+```hcl
+pipeline "ex01" {
+
+  param "conn" {
+    type = connection.slack
+    default = connection.slack.default
+  }
+
+  step "http" "send_msg" {
+    url = "https://slack.com/...."
+
+    request_headers = {
+      Content-Type = "application/json; charset=utf-8"
+      Authorization = "Bearer ${param.conn.token}"
+    }
+  }
+
+  output "results_http" {
+    value = step.http.send_msg.response_body
+  }
+
+}
+```
+
+You can run the pipeline without argument to use your slack default connections:
+
+```bash
+flowpipe pipeline run ex01
+```
+
+If you want to use a different connection instead, you can pass it as an argument:
+
+```bash
+flowpipe pipeline run ex01 --arg conn=connection.slack.my_connection
+```
+
+
+### Dynamic Connections
+
+Some connections support dynamic attributes that contain temporary credentials.  Cloud services like AWS and GCP provide many mechanisms for defining credentials (static keys, SSO, assumed roles, profiles,etc) but in all cases can be exchanged for a "standard" format such as an access token or AWS key pair.
+
+For example, you can specify AWS connections in many formats:
+```hcl
+connection "aws" "aws_static" {
+  access_key = "ASIAQGDFAKEKGUI5MCEU"
+  secret_key = "QhLNLGM5MBkXiZm2k2tfake+TduEaCkCdpCSLl6U"
+}
+
+connection "aws" "aws_profile" {
+  profile = "awx_prod_01"
+}
+
+```
+
+For *all* of these cases you can resolve to an access key / secret key / session token, which are available as attributes. Additionally, the `aws` connection provides an `env` attribute which is a map of the [credential-related environment variables](https://docs.aws.amazon.com/sdk-for-php/v3/developer-guide/guide_credentials_environment.html) (`AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY`, `AWS_SESSION_TOKEN`) 
+
+When writing a library mod, you should use these dynamic attributes where possible, as it simplifies the code and minimizes the number of connection-related parameters:
+
+```hcl
+pipeline "ex1" {
+  param "conn" {
+    type    = connection.aws
+    default = connection.aws.default
+  }
+
+  step "container" "aws" {
+    image = "public.ecr.aws/aws-cli/aws-cli"
+    cmd   = [ "s3", "ls" ]
+    env   = param.conn.env
+  } 
+}
+```
+
+When using dynamic connection attributes, the credentials are resolved as late as possible.  When the *step instance* runs that uses the connection, it is fetched (from the cache if not expired) or generated.  This is important because the dynamic credentials are usually temporary and will expire and become invalid. You can tell Flowpipe how long to cache these temporary connections with the `ttl` argument.  This specifies the number of seconds to cache the connection.  Not all connections support (or require) `ttl`, and for those that do, the default value varies by type.  Note that the cache is in-memory only, and does not persist when Flowpipe stops.
+
+```hcl
+connection "gcp" " my_creds" {
+  credentials = "~/.config/gcloud/application_default_credentials.json"
+  ttl         = 900 # 15 minutes
+}
+```
diff --git a/docs/run/index.md b/docs/run/index.md
@@ -13,7 +13,7 @@ Flowpipe always runs in the context of a [mod](/docs/build), which is a collecti
 Flowpipe loads the mod from the current directory by default, but you can pass the [--mod-location](/docs/reference/cli) flag or set the [FLOWPIPE_MOD_LOCATION](/docs/reference/env-vars/flowpipe_mod_location) to set it to a different path.  The event store is also written to this mod location, in the `.flowpipe/store/` subdirectory.
 
 ## Configuration Files
-Flowpipe will load [configuration files](/docs/reference/config-files) (`*.fpc`) according to the configuration search path.  You can change this path with the `--config-path` argument or the [FLOWPIPE_CONFIG_PATH](/docs/reference/env-vars/flowpipe_config_path) environment variable, but it defaults to `.:$FLOWPIPE_INSTALL_DIR/config` (`.:~/.flowpipe/config`).  This allows you to manage your [workspaces](/docs/run/workspaces) and [credentials](/docs/run/credentials) centrally in the `~/.flowpipe/config` directory, but override them in the working directory / mod location if desired.
+Flowpipe will load [configuration files](/docs/reference/config-files) (`*.fpc`) according to the configuration search path.  You can change this path with the `--config-path` argument or the [FLOWPIPE_CONFIG_PATH](/docs/reference/env-vars/flowpipe_config_path) environment variable, but it defaults to `.:$FLOWPIPE_INSTALL_DIR/config` (`.:~/.flowpipe/config`).  This allows you to manage your [workspaces](/docs/run/workspaces) and [connections](/docs/run/connections) centrally in the `~/.flowpipe/config` directory, but override them in the working directory / mod location if desired.
 
 
 ## Operating Modes

diff --git a/docs/sidebar.json b/docs/sidebar.json
@@ -15,7 +15,7 @@
     "items": [
       "run/pipelines",
       "run/server",
-      "run/credentials",
+      "run/connections",
       "run/workspaces"
     ]
   },
-Original file line number
+Diff line change
@@ Expand Up / @@ -39,7 +39,7 @@ sidebar_label: Flowpipe CLI @@
       <tr>
         <td nowrap="true"> <inlineCode>--config-path</inlineCode> </td>
         <td>
-        Sets the search path for <a href = "/docs/reference/config-files">configuration files</a>. This argument accepts a colon-separated list of directories.  All configuration files (<inlineCode>*.fpc</inlineCode>) will be loaded from each path, with decreasing precedence.  The default is <inlineCode>.:$FLOWPIPE_INSTALL_DIR/config</inlineCode> (<inlineCode>.:~/.flowpipe/config</inlineCode>).  This allows you to manage your <a href="/docs/reference/config-files/workspace"> workspaces </a> and <a href="/docs/reference/config-files/credential">credentials</a> centrally in the <inlineCode>~/.flowpipe/config</inlineCode> directory, but override them in the working directory / mod location if desired.
+        Sets the search path for <a href = "/docs/reference/config-files">configuration files</a>. This argument accepts a colon-separated list of directories.  All configuration files (<inlineCode>*.fpc</inlineCode>) will be loaded from each path, with decreasing precedence.  The default is <inlineCode>.:$FLOWPIPE_INSTALL_DIR/config</inlineCode> (<inlineCode>.:~/.flowpipe/config</inlineCode>).  This allows you to manage your <a href="/docs/reference/config-files/workspace"> workspaces </a> and <a href="/docs/reference/config-files/connection">connections</a> centrally in the <inlineCode>~/.flowpipe/config</inlineCode> directory, but override them in the working directory / mod location if desired.
         </td>
       </tr>
@@ Expand Down @@