CLI tool for extracting AtScale models, generating SML semantic models, and generating BI workbooks (Tableau, Excel, Power BI).

Upcoming features:

• Google Sheets
• Rudy's aggregate util
• Perspectives
• -- apply plan should show command
• graphql output not going to output
• web interface better + REST
• tableau, mstr, ssas conversion
  • find Hive dialect in a workbook
• Apply style to SML
• Add kubectl management commands; for example reading log files, updating passwords
• Calculation groups as shared objects

Connect to a live AtScale instance or read local SML files to produce a portable model.yaml capturing all metrics and dimension hierarchies.

flowchart LR
    ATS["AtScale Instance"] --> A["extract-model-from-atscale"] --> MODEL["model.yaml"]
    SML["SML Files"] --> B["extract-model-from-sml"] --> MODEL

Generate a complete AtScale SML semantic model from a live database connection or a DDL file, extract existing schema DDL for inspection, or run ad-hoc SQL against any registered connection, or run the analysis-suggestions engine against an extracted model to surface the highest-value metric combinations.

flowchart LR
    DB[("Database")] --> A["extract-ddl-from-connection"] --> DDL["DDL (.sql)"]
    ATS["AtScale Instance"] --> B["generate-ddl-from-atscale"] --> DDL
    DDL --> C["generate-sml-from-ddl"] --> SML["SML Files"]
    DB --> D["generate-sml-from-connection"] --> SML
    XML["AtScale XML"] --> G["generate-sml-from-xml"] --> SML
    SML2A["SML Dir A"] --> H["generate-shared-model-plan"] --> PLAN["RECOMMENDATION.md + option-N.yml"]
    SML2B["SML Dir B"] --> H
    PLAN --> I["apply-shared-model-plan-option"] --> SHARED["shared/dimensions, datasets, models"]
    DB --> E["execute-sql-on-connection"] --> OUT["Results (stdout)"]
    MODEL["model.yaml"] --> F["generate-metrics-from-model"] --> METRICS["metrics/*.yml"]
    SML --> J["apply-style-to-sml"] --> SML

Profile an existing database's schema and value distributions, then generate matching synthetic DDL and CSV data — either written to local files or loaded directly into a target database.

flowchart LR
    DB[("Source Database")] --> A["extract-data-shape-from-connection"] --> SHAPE["data-shape.json"]
    SHAPE --> B["generate-ddl-from-data-shape"] --> DDL["DDL (.sql)"]
    SHAPE --> C["generate-data-from-data-shape"] --> CSV["Synthetic CSVs"]
    SHAPE --> D["generate-data-from-data-shape-to-connection"] --> TARGET[("Target Database")]

Generate a namespace definition from a model, then produce ready-to-open Tableau, Excel, and Power BI workbooks with optional field-label aliases.

flowchart LR
    MODEL["model.yaml"] --> A["generate-namespace-from-model"] --> NS["namespace.yaml"]
    NS --> B["generate-tableau-from-namespace"] --> TWB["tableau.twb"]
    NS --> C["generate-excel-from-namespace"] --> XLSX["workbook.xlsx"]
    NS --> D["generate-powerbi-from-namespace"] --> PBI["output/powerbi/"]
    CONN["connections.yaml"] --> B & C & D
    ALIASES["aliases.yaml (opt.)"] -.-> B & C & D

Capture queries from AtScale's Postgres backend, replay them through a load harness, enrich results with execution metadata, and compare two runs side-by-side to detect regressions in row counts, duration, or error behavior.

flowchart TD
    SML["SML Files"] --> G["generate-queries-from-sml"] --> QJSON["queries/*.json"]
    MODEL["model.yaml"] --> H["generate-queries-from-model"] --> QJSON
    ATSDB[("AtScale Postgres")] --> A["extract-query-stats-from-atscale"] --> STATS["occurrences.csv"]
    ATSDB --> B["extract-queries-from-atscale"] --> QJSON
    QJSON --> C["execute-atscale-query-harness"] --> RUN["run_results/*.csv"]
    ATS["AtScale Instance"] --> C
    CONN["connections.yaml"] --> D["execute-query-on-connection"] --> QOUT["Query Output"]
    RUN --> E["generate-enhanced-query-results"] --> ECSV["*_enhanced.csv"]
    ATSDB --> E
    RUN --> F["execute-run-analysis"]
    ECSV --> F
    F --> SUMMARY["summary.txt"]
    F --> COMPARISON["comparison.csv"]
    F --> OUTLIERS["outliers.csv"]

Expose every operation as a GraphQL mutation and REST endpoint via an embedded HTTP server.

flowchart LR
    CLIENT["HTTP Client"] --> A["execute-web-services"] --> OPS["All Operations (GraphQL / REST)"]

flowchart LR
    A["version"] --> VER["@atscale/[email protected] (stdout)"]

Bootstrap and manage an AtScale instance — generate Helm install values, register data sources and SML repositories, deploy catalogs, and inspect live configuration state.

flowchart LR
    HOSTNAME["Hostname"] --> A["generate-atscale-install-yaml"] --> VALUES["values.yaml (Helm)"]
    CONN["connections.yaml"] --> B["atscale-create-data-source"] --> ATS["AtScale Instance"]
    CONN --> C["atscale-create-repo"] --> ATS
    CONN --> D["atscale-deploy-catalog"] --> ATS
    ATS --> E["atscale-list-data-sources"] --> INFO["AtScale Info (stdout)"]
    ATS --> F["atscale-list-repos"] --> INFO
    ATS --> G["atscale-list-deployments"] --> INFO
    ATS --> H["atscale-list-model-errors"] --> INFO

• Setup
• Operations
  • Model Extraction
    • extract-model-from-atscale
    • extract-model-from-sml
  • SML Creation and Manipulation
    • execute-sql-on-connection
    • extract-ddl-from-connection
    • generate-sml-from-connection
    • generate-sml-from-ddl
    • generate-sml-from-xml
    • generate-shared-model-plan
    • apply-shared-model-plan-option
    • apply-style-to-sml
    • generate-ddl-from-atscale
    • generate-metrics-from-model
  • Synthetic Data Generation
    • extract-data-shape-from-connection
    • generate-ddl-from-data-shape
    • generate-data-from-data-shape
    • generate-data-from-data-shape-to-connection
  • Visualization and Namespace Processing
    • BI Tool Feature Comparison
    • generate-namespace-from-model
    • generate-tableau-from-namespace
    • generate-excel-from-namespace
    • generate-powerbi-from-namespace
  • Testing / Query Processing
    • generate-queries-from-sml
    • generate-queries-from-model
    • extract-query-stats-from-atscale
    • extract-queries-from-atscale
    • execute-atscale-query-harness
    • execute-query-on-connection
    • generate-enhanced-query-results
    • execute-run-analysis
  • AtScale Config
    • generate-atscale-install-yaml
    • atscale-list-data-sources
    • atscale-create-data-source
    • atscale-list-repos
    • atscale-create-repo
    • atscale-list-deployments
    • atscale-deploy-catalog
    • atscale-list-model-errors
  • Web Services
    • execute-web-services
  • Utilities
    • version
• Reference Documentation
• Extract AtScale Model Workflow
• Connection YAML (connections.yaml)
• SML Style Config (sml.style.yaml)
• Model YAML (model.yaml)
• Namespace YAML (namespace.yaml)
• Aliases YAML (aliases.yaml)

Prerequisites:

• Node.js 18+
• macOS only: Xcode Command Line Tools (required for native module compilation):

  xcode-select --install
  brew install npm 
  brew install tsc

Install globally from npm:

sudo npm install -g @atscale/ps-utils

Build from source:

npm install
npm run build

The docs/ directory contains extended reference material:

↑ Table of Contents

Connects to a live AtScale instance via MDX and extracts a model's metrics and attributes into a model.yaml file. This file is the input for generate-tableau-from-namespace.

./atscale-utils extract-model-from-atscale \
  --model "Telemetry" \
  --connection-file "./connections.yaml" \
  --connection-name "ats_connection" \
  --output-model-file "./model.yaml"

GitHub Actions workflow: See Extract AtScale Model Workflow.

↑ Table of Contents

Reads a local SML directory (produced by generate-sml-from-connection or generate-sml-from-ddl) and outputs a model.yaml file in the same format as extract-model-from-atscale. Use this to generate a Tableau workbook without a live AtScale connection.

./atscale-utils extract-model-from-sml \
  --sml-dir "./sml-output" \
  --output-model-file "./model.yaml"

With optional overrides:

./atscale-utils extract-model-from-sml \
  --sml-dir "./sml-output" \
  --model-name "SalesModel" \
  --connection-name "snow_demo" \
  --output-model-file "./model.yaml"

↑ Table of Contents

Reads a SQL file, splits it into individual statements, and executes each one against a named database connection. Works with DDL (CREATE TABLE, DROP TABLE, ALTER TABLE, CREATE VIEW), DML (INSERT, UPDATE, DELETE), and mixed files.

./atscale-utils execute-sql-on-connection \
  --sql-file "./schema/migrations/001_init.sql" \
  --connection-file "./connections.yaml" \
  --connection-name "snow_demo"

Preview statements without running them:

./atscale-utils execute-sql-on-connection \
  --sql-file "./schema.sql" \
  --connection-name "snow_demo" \
  --dry-run true

Skip failed statements and continue:

./atscale-utils execute-sql-on-connection \
  --sql-file "./schema.sql" \
  --connection-name "snow_demo" \
  --on-error continue

↑ Table of Contents

Connects to a live database, reads schema metadata for each table in the target schema, and writes CREATE TABLE DDL statements to a file (or stdout). Useful for capturing schema snapshots, seeding DDL files for generate-sml-from-ddl, or comparing schema drift.

./atscale-utils extract-ddl-from-connection \
  --connection-file "./connections.yaml" \
  --connection-name "snow_demo" \
  --schema "PUBLIC" \
  --output-file "./schema.ddl"

Extract only specific tables or wildcard patterns:

./atscale-utils extract-ddl-from-connection \
  --connection-file "./connections.yaml" \
  --connection-name "snow_demo" \
  --schema "PUBLIC" \
  --tables "Dim*,FactInternetSales" \
  --output-file "./schema.ddl"

↑ Table of Contents

Connects to a live database, introspects its schema, runs semantic model inference, and writes a complete set of AtScale SML files to a directory.

Inference engine capabilities:

• Composite keys — tables with multi-column primary keys produce key_columns arrays in SML level attributes.
• FK-based classification — fact vs. dimension classification uses foreign key graph topology. Tables that only receive FKs (high in-degree) are classified as dimensions. Tables with FK references to multiple tables plus numeric payload columns are classified as facts.
• Bridge / cross-reference tables — junction tables with FKs to ≥2 tables and ≤1 payload column are automatically classified as shared dimensions (the AtScale pattern for bridge tables). A [BRIDGE TABLE] advisory is emitted in the model warnings.
• Naming convention patterns — prefix/suffix patterns take priority over structural heuristics: dim_* / *_dim, fct_* / *_fct / fact_* / *_fact, lkp_* / ref_* / lookup_*, bridge_* / xref_* / junction_* / map_*, etc.
• information_schema FK queries — foreign key metadata is read from INFORMATION_SCHEMA.REFERENTIAL_CONSTRAINTS and KEY_COLUMN_USAGE for accurate composite-key support. Falls back to the driver-level API when information_schema is not accessible (e.g. Snowflake).
• One relationship per hierarchy — when a dimension has multiple hierarchies, one model relationship is emitted per hierarchy leaf level so all hierarchies are visible in BI tools.
• Distinct count estimate — measures inferred as distinct-countable entities use distinct count estimate aggregation (preferred over distinct count for AtScale aggregation engine compatibility).

./atscale-utils generate-sml-from-connection \
  --connection-file "./connections.yaml" \
  --connection-name "ats_connection" \
  --model-name "Telemetry" \
  --output-dir "./sml-output"

With optional overrides:

./atscale-utils generate-sml-from-connection \
  --connection-file "./connections.yaml" \
  --connection-name "snow_demo" \
  --model-name "SalesModel" \
  --output-dir "./sml-output" \
  --schema "SALES" \
  --catalog-name "Sales Analytics" \
  --pii-severity "HIGH" \
  --sample-size 500 \
  --fact-tables "FactInternetSales,FactResellerSales" \
  --camel-case-files true \
  --camel-case-measures true

Style parameters (--pii-severity, --fact-tables, --catalog-name, --camel-case-files, --camel-case-measures, --label-style, --sample-size, --min-hierarchies-per-dim, --max-hierarchies-per-dim) can also be set in an SML style config file. CLI flags take priority over the file. After generation, effective settings are always written to <output-dir>/sml.style.yaml regardless of the input config path.

Output layout:

<output-dir>/
  catalog.yml
  connections/<connectionName>.yml
  datasets/<table>.yml
  dimensions/<dimension>.yml
  metrics/<metric>.yml
  models/<modelName>.yml
  sml.style.yaml   ← effective settings used for this generation

↑ Table of Contents

Parses a SQL DDL file (CREATE TABLE / CREATE VIEW statements) and generates AtScale SML files without a live database connection. Useful for offline model generation and CI pipelines.

All inference capabilities described under generate-sml-from-connection (composite keys, bridge table detection, naming patterns, one-relationship-per-hierarchy) apply equally to the DDL path. FK constraints declared in the DDL (FOREIGN KEY (col1, col2) REFERENCES …) are parsed and used for composite join inference.

./atscale-utils generate-sml-from-ddl \
  --ddl-file "./schema.sql" \
  --output-dir "./sml-output"

With optional overrides:

./atscale-utils generate-sml-from-ddl \
  --ddl-file "./schema.sql" \
  --model-name "SalesModel" \
  --output-dir "./sml-output" \
  --connection-name "my_warehouse" \
  --catalog-name "Sales Analytics" \
  --schema "SALES" \
  --pii-severity "LOW" \
  --fact-tables "FactInternetSales,FactResellerSales" \
  --camel-case-files true \
  --camel-case-measures true

Style parameters (--pii-severity, --fact-tables, --catalog-name, --camel-case-files, --camel-case-measures, --label-style, --min-hierarchies-per-dim, --max-hierarchies-per-dim) can also be set in an SML style config file. CLI flags take priority over the file. After generation, effective settings are always written to <output-dir>/sml.style.yaml regardless of the input config path.

Output layout: Same as generate-sml-from-connection (including sml.style.yaml).

↑ Table of Contents

Reads an AtScale XML project file (schema version project_2_0) and converts it to AtScale SML YAML files. No database connection is required — the conversion runs entirely from the XML model definition.

Dimensions, metrics, datasets, catalog, connection, and model files are all emitted based on the XML structure. Relationships are inferred from the cube's key-ref logical sections: cross-table FKs (complete="false") are mapped to separate dimension datasets, and degenerate dimensions (complete="true") are mapped as self-joins within the fact table. Role-played dimensions (role_play), include_default_drillthrough, metric folders, dataset column definitions, and the immutable flag are all extracted from the XML when present. The connection name is auto-detected from <physical><connection id="..."> if --connection-name is not supplied. Schema-level dimensions that have no join path to the cube are omitted.

./atscale-utils generate-sml-from-xml \
  --xml-file "./MyModel.xml" \
  --output-dir "./sml-output"

With optional overrides:

./atscale-utils generate-sml-from-xml \
  --xml-file "./MyModel.xml" \
  --output-dir "./sml-output" \
  --connection-name "my_bq_conn" \
  --connection-type "bigquery" \
  --connection-db "my-project-id" \
  --connection-schema "my_dataset" \
  --catalog-name "My Catalog"

Output layout:

<output-dir>/
  catalog.yml
  connections/<connection-name>.yml
  datasets/<dataset-name>.yml      (one per XML <data-set>)
  dimensions/<dim-name>.yml        (one per referenced dimension)
  metrics/<metric-name>.yml        (one per measure or inline expression)
  calculations/<calc-name>.yml     (one per schema-level calculated member)
  models/<cube-name>.yml           (one per XML <cube>)

↑ Table of Contents

Analyses one or more SML output directories to identify opportunities for sharing or reusing dimensions, datasets, and model structures across projects. Uses Jaccard-based fuzzy subtree matching to compute similarity scores and emits a human-readable recommendation report plus machine-readable option YAML files.

Each option falls into one of three categories:

./atscale-utils generate-shared-model-plan \
  --input-dirs "./project-a,./project-b" \
  --output-dir "./plan-output"

With threshold override:

./atscale-utils generate-shared-model-plan \
  --input-dirs "./project-a,./project-b,./project-c" \
  --output-dir "./plan-output" \
  --threshold 0.5

Output layout:

<output-dir>/
  RECOMMENDATION.md              — options with diagrams, change lists, and pros/cons
  option-1-<kind>.yml            — machine-readable changes for option 1
  option-2-<kind>.yml            — machine-readable changes for option 2
  ...

↑ Table of Contents

Applies a machine-readable recommendation YAML produced by generate-shared-model-plan to create shared SML files. Three kinds are supported:

./atscale-utils apply-shared-model-plan-option \
  --plan-file "./shared-plan/option-1-dataset-consolidation.yml" \
  --shared-dir "./shared"

To also delete the local source copies after writing the shared file:

./atscale-utils apply-shared-model-plan-option \
  --plan-file "./shared-plan/option-11-shared-dimension-library.yml" \
  --shared-dir "./shared" \
  --remove-sources

Preview without touching disk:

./atscale-utils apply-shared-model-plan-option \
  --plan-file "./shared-plan/option-31-base-model-extraction.yml" \
  --shared-dir "./shared" \
  --dry-run

An APPLY_REPORT.md is written to <shared-dir> summarising every action taken and the deployment steps required.

↑ Table of Contents

Re-applies display labels to an existing SML directory using a style config. Reads datasets, dimensions, and metrics YAML files in-place and rewrites their label fields according to the active style, then writes STYLE.md and STYLE_CHANGES.md summarising the conventions and every label change made.

./atscale-utils apply-style-to-sml \
  --sml-dir "./sml-output"

With optional overrides:

./atscale-utils apply-style-to-sml \
  --sml-dir "./sml-output" \
  --sml-config-file "./sml-output/sml.style.yaml" \
  --label-style camel-case \
  --catalog-name "Sales Analytics"

Output: Updates datasets/*.yml, dimensions/*.yml, and metrics/*.yml labels in-place; writes STYLE.md and STYLE_CHANGES.md to <sml-dir>.

↑ Table of Contents

Generates DDL (CREATE TABLE statements) by reading table and column metadata from an AtScale data source via the REST API. No direct database connection is required — AtScale acts as the metadata broker.

./atscale-utils generate-ddl-from-atscale \
  --connection-file "./connections.yaml" \
  --atscale-connection-name "my_atscale" \
  --data-source-name "snowflake_prod" \
  --database "MY_DATABASE" \
  --schema "PUBLIC"

With optional filters and output file:

./atscale-utils generate-ddl-from-atscale \
  --connection-file "./connections.yaml" \
  --atscale-connection-name "my_atscale" \
  --data-source-name "snowflake_prod" \
  --database "MY_DATABASE" \
  --schema "PUBLIC" \
  --tables "fact_*,dim_*" \
  --output-file "./schema.ddl"

Output: One CREATE TABLE statement per matched table. Output includes a header comment with data source name, database, schema, and timestamp.

Foreign keys: The AtScale metadata API does not expose FK relationships. FK constraints are not included in the output; a header comment documents this. Use extract-ddl-from-connection if FK constraints are required.

↑ Table of Contents

Reads a model.yaml file, reconstructs a SemanticModel from its mdx and sql sections, and runs the analysis-suggestions engine to produce a ranked list of suggested metric × dimension combinations. Each suggestion includes a relevance score, analysis type, the measure being analysed, and the dimension hierarchy to slice by.

./atscale-utils generate-metrics-from-model \
  --model-file "./model.yaml"

With options:

./atscale-utils generate-metrics-from-model \
  --model-file "./model.yaml" \
  --model-name "SalesModel" \
  --max-suggestions 20 \
  --min-score 0.6 \
  --include-tuples true \
  --format yaml \
  --output-file "./suggestions.yaml"

The suggestion-tuning parameters (--max-suggestions, --min-score, --include-tuples) can also be set in an SML style config file. CLI flags take priority over the file. After generation, effective settings are written to sml.style.yaml in the output file's directory (or the working directory when writing to stdout).

↑ Table of Contents

Connects to a live database, reads an SML model to understand the semantic layer structure, and extracts a statistical fingerprint of the data — capturing hierarchy level cardinalities, rollup ratios, leaf-level fact densities, measure distributions, and conformed dimension overlap.

No actual data values are written. The output is a YAML fingerprint file that fully describes the statistical shape of the model without divulging any specific records. The file contains enough information to reconstruct plausible DDL and generate synthetic data that is statistically equivalent to the original.

Large fact tables are automatically sampled via TABLESAMPLE SYSTEM or a LIMIT-based fallback (see --target-fact-rows and --no-tablesample). Sample sizes are computed using the Cochran formula (z² × 0.25 / e²) with finite-population correction, guaranteeing statistical significance without reading the entire table.

./atscale-utils extract-data-shape-from-connection \
  --connection-file "./connections.yaml" \
  --connection-name "snow_demo" \
  --sml-path        "./sml-output" \
  --output-file     "./data-shape.yaml"

With sampling tuning:

./atscale-utils extract-data-shape-from-connection \
  --connection-file    "./connections.yaml" \
  --connection-name    "snow_demo" \
  --sml-path           "./sml-output" \
  --target-fact-rows   50000 \
  --target-column-rows 5000 \
  --no-tablesample          # use for MySQL / MariaDB

Output: A data-shape.yaml fingerprint file containing:

• Dimension hierarchy level cardinalities, null key fractions, and rollup ratios (P50/P95/shape)
• Leaf-level fact densities (avg/stddev/P50/P90/P99), coverage fraction, and cold-member fraction
• Measure distributions (null fraction, min/max/mean, percentiles, additivity classification)
• Pairwise conformed dimension overlap across facts (intersection/union fraction)

By default, all entity names are replaced with opaque sequential IDs (D1, D1.H1, D1.H1.L3, F1, F1.M2) and the mapping is discarded. Pass --preserve-meta-data to retain the original physical names in a metadata: block so that downstream generate-data-from-data-shape-to-connection runs create tables that match the SML model schema.

See STATISTICS.md for the full algorithm description.

↑ Table of Contents

Reads a data-shape.yaml fingerprint file produced by extract-data-shape-from-connection and emits CREATE TABLE DDL statements. No database connection is required.

Table and column names are synthetic — the original names are not stored in the fingerprint. The same fingerprint always produces identical DDL regardless of when it is run.

./atscale-utils generate-ddl-from-data-shape \
  --input-file "./data-shape.yaml" \
  --output-file "./schema.sql"

With dialect selection:

./atscale-utils generate-ddl-from-data-shape \
  --input-file  "./data-shape.yaml" \
  --dialect     snowflake \
  --output-file "./schema.sql"

Output: One CREATE TABLE statement per dimension and fact. Dimension tables are emitted first so FOREIGN KEY references resolve correctly.

Dialect notes:

• bigquery — PRIMARY KEY and FOREIGN KEY constraints are omitted (not supported)
• snowflake — integer types are mapped to NUMBER(n,0), decimals to NUMBER(18,4)
• All other dialects — standard ANSI SQL types (SMALLINT, INTEGER, BIGINT, DECIMAL(18,4), VARCHAR(200))

See STATISTICS.md §Phase 7 for the reconstruction algorithm.

↑ Table of Contents

Reads a data-shape.yaml fingerprint file and generates statistically equivalent synthetic data, writing one CSV file per table to an output directory. No database connection is required.

./atscale-utils generate-data-from-data-shape \
  --input-file "./data-shape.yaml" \
  --output-dir "./data"

With a scale factor and reproducible seed:

./atscale-utils generate-data-from-data-shape \
  --input-file    "./data-shape.yaml" \
  --output-dir    "./data" \
  --scale-factor  0.01 \
  --seed          42

Output: One CSV per table — dimensions first, then facts. Column names match those produced by generate-ddl-from-data-shape.

See STATISTICS.md §Phase 8 for the generation algorithm.

↑ Table of Contents

End-to-end pipeline: reads a data-shape.yaml fingerprint, generates synthetic data in memory, and loads it directly into a live database. Combines generate-data-from-data-shape and generate-ddl-from-data-shape into a single step.

./atscale-utils generate-data-from-data-shape-to-connection \
  --connection-file "./connections.yaml" \
  --connection-name "snow_demo" \
  --input-file      "./data-shape.yaml" \
  --drop-if-exists  true \
  --create-tables   true \
  --dialect         snowflake

With scale factor and batch tuning:

./atscale-utils generate-data-from-data-shape-to-connection \
  --connection-file "./connections.yaml" \
  --connection-name "pg_sandbox" \
  --input-file      "./data-shape.yaml" \
  --scale-factor    0.1 \
  --seed            42 \
  --schema          PUBLIC \
  --batch-size      1000

Operation order: DROP facts → DROP dims → CREATE dims → CREATE facts → INSERT dims (parallel) → INSERT facts (parallel). Dimensions are inserted in parallel since they have no inter-table FK dependencies. Facts are inserted in parallel after all dimension inserts complete, ensuring FK constraints are respected throughout.

See STATISTICS.md §Phase 8 for the generation algorithm.

↑ Table of Contents

Reads a model.yaml file and automatically generates a namespace YAML by running the analysis-suggestions engine against the model's measures and dimensions. The output is ready to pass directly to generate-tableau-from-namespace or generate-excel-from-namespace.

Each suggestion becomes a worksheet:

• trend → graphType: line (measure over time, title suffixed with granularity e.g. "by Week")
• comparison → graphType: line with colorField (measure over time, broken down by a second dimension)
• breakdown / distribution → graphType: bar
• ranking → graphType: bar with limit: 10 and sortDirection: desc

Up to six summary-statistic scorecards (graphType: text) are prepended automatically. All worksheets are arranged in a single auto-sized dashboard. Time-based line charts include an xAxisGranularity field ("day" for DATE_DOUBLE columns, "week" for DATETIME columns).

./atscale-utils generate-namespace-from-model \
  --model-file "./model.yaml" \
  --output-file "./namespace.yaml"

With optional overrides:

./atscale-utils generate-namespace-from-model \
  --model-file "./model.yaml" \
  --model-name "SalesModel" \
  --title "Sales Analytics" \
  --max-suggestions 20 \
  --min-score 0.6 \
  --output-file "./namespace.yaml"

↑ Table of Contents

Generates a Tableau .twb workbook from a namespace YAML definition and a model YAML file.

./atscale-utils generate-tableau-from-namespace \
  --namespace-file "./resources/namespaces/telemetry/overview.yaml" \
  --model-file "./model.yaml" \
  --connection-file "./connections.yaml" \
  --connection-name "ats_connection" \
  --tableau-version 2025 \
  --target-file "./tableau.twb"

With an aliases file:

./atscale-utils generate-tableau-from-namespace \
  --namespace-file "./namespace.yaml" \
  --model-file "./model.yaml" \
  --connection-file "./connections.yaml" \
  --connection-name "ats_connection" \
  --aliases-file "./aliases.yaml" \
  --target-file "./tableau.twb"

See Namespace YAML for the full namespace format reference.

↑ Table of Contents

Generates an Excel workbook (.xlsx) from a namespace YAML and a model YAML. No external dependencies beyond the npm packages.

Each dashboard in the namespace produces one visible sheet containing:

• One chart per tile (bar, line, pie, or area) styled from the worksheet graphType
• CUBE formula data sections in far-right columns of the dashboard sheet — Excel evaluates these against AtScale via MDX/XMLA when the workbook is connected
• An OLAP pivot table on the hidden _Connections sheet — click Data → Refresh All in Excel to load live data
• Number formatting applied from the worksheet format field (integer, decimal:N, percent:N, currency:N)
• Granularity-aware set expressions for time axes when xAxisGranularity is set and the model hierarchy has a matching level

./atscale-utils generate-excel-from-namespace \
  --namespace-file "analysis/namespace.yaml" \
  --model-file     "model.yaml" \
  --connection-file "connections.yaml" \
  --connection-name "ats_connection" \
  --target-file    "analysis/workbook.xlsx"

With an aliases file:

./atscale-utils generate-excel-from-namespace \
  --namespace-file "analysis/namespace.yaml" \
  --model-file     "model.yaml" \
  --connection-file "connections.yaml" \
  --connection-name "ats_connection" \
  --aliases-file   "aliases.yaml" \
  --target-file    "analysis/workbook.xlsx"

The MDX connection uses Provider=MSOLAP.8 pointed at the AtScale XMLA endpoint (<mdx.url>/xmla/<organization_id>). Open the workbook in Excel and click Data → Refresh All to load live data.

↑ Table of Contents

Generates a Power BI project folder (.pbip) from a namespace YAML and a model YAML. The output can be opened directly in Power BI Desktop.

One page is created per worksheet in the namespace. The graphType controls the visual type on each page:

Connection requirement: The named connection must have an mdx: block, and the referenced user must have a token field (not password) — Power BI connects via the AtScale MDX URL with token authentication.

Output layout:

output/<target-folder>/
  <target-folder>.pbip
  <target-folder>.SemanticModel/
    definition.pbism
    modelReference.json
  <target-folder>.Report/
    definition.pbir
    definition/
      report.json
      version.json
      pages/
        <uuid>/
          page.json
          visuals/<uuid>/visual.json

./atscale-utils generate-powerbi-from-namespace \
  --namespace-file "analysis/namespace.yaml" \
  --model-file     "model.yaml" \
  --connection-file "connections.yaml" \
  --connection-name "ats_connection" \
  --target-folder  "powerbi"

The connections.yaml user entry for Power BI must include a token field:

users:
  admin:
    username: admin
    token: "<AtScale API token>"

connections:
  ats_connection:
    mdx:
      url: http://template.atscale-se-demo.com
      user: admin
      organization_id: default
      catalog_name: Telemetry

↑ Table of Contents

Reads an SML directory and generates two query JSON files — one XMLA (MDX) and one SQL — both compatible with execute-atscale-query-harness. Each file provides complete coverage of the model:

• Metric totals — one query per metric with no dimensional breakdown (verifies the measure computes without errors and returns a value)
• Level breakdowns — one query per hierarchy level across all dimensions, selecting all model metrics broken down by that level (verifies dimensional slicing at every granularity)

XMLA query formats:

Metric total:

SELECT {[Measures].[m_metric_name]} ON COLUMNS
FROM [ModelName]

Level breakdown:

SELECT {[Measures].[m1], [Measures].[m2], …} ON COLUMNS,
  NON EMPTY [Dim Name].[Hierarchy Name].[Level Name].MEMBERS ON ROWS
FROM [ModelName]

SQL query formats:

Metric total:

SELECT "m_metric_name"
FROM "ModelName"

Level breakdown:

SELECT "level_column", "m1", "m2", …
FROM "ModelName"
GROUP BY "level_column"
ORDER BY "level_column"

# Generate queries from an SML directory
./atscale-utils generate-queries-from-sml \
  --sml-dir "./sml" \
  --xmla-output-file "./queries/model_xmla.json" \
  --sql-output-file "./queries/model_sql.json"

# Pass the output directly to the harness
./atscale-utils execute-atscale-query-harness \
  --connection-file "./connections.yaml" \
  --connection-name "my_model" \
  --query-file "./queries/model_xmla.json" \
  --protocol xmla \
  --output-dir "./run_results"

↑ Table of Contents

Reads a model.yaml file (output of extract-model-from-atscale or extract-model-from-sml) and generates the same XMLA and SQL query JSON files as generate-queries-from-sml. Use this operation when a model.yaml is already available instead of a raw SML directory.

Coverage is identical: one grand-total query per metric and one per-level breakdown query per hierarchy level across all dimensions.

# Generate queries from a model.yaml
./atscale-utils generate-queries-from-model \
  --model-file "./model.yaml" \
  --xmla-output-file "./queries/model_xmla.json" \
  --sql-output-file "./queries/model_sql.json"

# Multiple models in one file — select by name
./atscale-utils generate-queries-from-model \
  --model-file "./model.yaml" \
  --model-name "Telemetry" \
  --xmla-output-file "./queries/telemetry_xmla.json" \
  --sql-output-file "./queries/telemetry_sql.json"

↑ Table of Contents

Paginates through the AtScale query history REST API and writes a CSV occurrence matrix showing how many user queries involved each (dimension attribute × measure) pair. Mirrors the query_histogram_updated.ipynb notebook analysis.

./atscale-utils extract-query-stats-from-atscale \
  --connection-file "./connections.yaml" \
  --connection-name "ats_connection" \
  --model "MyModel" \
  --output-dir "./query-stats" \
  --window-days 30

With a monthly breakdown:

./atscale-utils extract-query-stats-from-atscale \
  --connection-file "./connections.yaml" \
  --connection-name "ats_connection" \
  --model "MyModel" \
  --output-dir "./query-stats" \
  --monthly true \
  --monthly-year 2025

Outputs:

• {output-dir}/{catalog}_{model}_occurrences.csv — occurrence count for every (attribute, measure) pair in the model
• {output-dir}/{catalog}_{model}_monthly_occurrences.csv — month-by-month counts (only when --monthly true)

The connections.yaml entry must have an mdx: block with url, organization_id, catalog_name, and user. The user entry needs username and password (installer mode) or username and password for cloud OAuth2.

↑ Table of Contents

Connects to the AtScale internal Postgres backend and extracts deduplicated query history for one or more models. Outputs one JSON file per (model, protocol) pair that can be consumed directly by execute-atscale-query-harness.

Supports two config formats:

• connections.yaml — the standard connections file used by this project (pass --connection-name to select the entry)
• systems.properties — legacy properties file; connection details and model list are read automatically

# Using a connections.yaml file
./atscale-utils extract-queries-from-atscale \
  --connection-file "./connections.yaml" \
  --connection-name "ats_postgres" \
  --models "SalesModel,InventoryModel" \
  --days 60 \
  --protocol all \
  --output-dir "./queries"

# Using a systems.properties file
./atscale-utils extract-queries-from-atscale \
  --connection-file "./systems.properties" \
  --models "SalesModel"

* Required when using a connections.yaml file.

Outputs: One JSON file per (model, protocol) pair written to --output-dir:

• {model}_sql_queries.json — container SQL queries (pgsql language)
• {model}_sql_installer_queries.json — installer SQL queries (sql/Hive language)
• {model}_xmla_queries.json — XMLA/MDX queries (analysis language)

Each file is a JSON array of query records with fields: queryName, queryLanguage, originalText, originalTextHash (SHA-256), outboundText, cubeName, projectId, aggregateUsed, numTimes, elapsedTimeInSeconds, avgResultSetSize, atscaleQueryId.

The connections.yaml entry must have a sql: block with dialect: postgres pointing at the AtScale Postgres backend (typically port 25432, database atscale).

↑ Table of Contents

Replays extracted queries against a live AtScale instance, measuring response time and row count for each query. Supports SQL and XMLA/MDX protocols, concurrent workers, throttling, and both one-pass and timed-duration run modes. Output is a CSV suitable for performance analysis.

Supports three input modes:

• --query-file — JSON file produced by extract-queries-from-atscale
• --ingest-file — ingest CSV (sampler_name,sql_text or sampler_name,atscale_query_id,sql_text)
• --task-file — executor task YAML/JSON (runs all tasks sequentially, inferring protocol from simulationClass)

Supports two connection config formats: connections.yaml or systems.properties.

# Direct mode — XMLA queries from a JSON file
./atscale-utils execute-atscale-query-harness \
  --connection-file "./connections.yaml" \
  --connection-name "ats_connection" \
  --query-file "./queries/SalesModel_xmla_queries.json" \
  --protocol xmla \
  --concurrent-users 5 \
  --output-dir "./run_results"

# Direct mode — SQL queries from an ingest CSV
./atscale-utils execute-atscale-query-harness \
  --connection-file "./connections.yaml" \
  --connection-name "ats_connection" \
  --ingest-file "./ingest/sales_queries.csv" \
  --protocol sql \
  --concurrent-users 3 \
  --throttle-ms 100

# Timed duration run (loop queries for 10 minutes with 5 concurrent users)
./atscale-utils execute-atscale-query-harness \
  --connection-file "./connections.yaml" \
  --connection-name "ats_connection" \
  --query-file "./queries/SalesModel_xmla_queries.json" \
  --protocol xmla \
  --concurrent-users 5 \
  --duration-minutes 10

# Task-file mode — run all executor tasks from a YAML file
./atscale-utils execute-atscale-query-harness \
  --connection-file "./systems.properties" \
  --connection-name "SalesModel" \
  --task-file "./executor_tasks/tasks.yaml"

Output CSV columns: run_id, task_name, model, query_name, run_query_uuid, original_atscale_query_id, protocol, status, duration_ms, row_count, checksum, error, timestamp, original_text_hash

• run_query_uuid — UUID generated per individual query execution; correlates this CSV row with the comment injected into the executed query (when --annotate-queries true)
• original_atscale_query_id — the query ID recorded in AtScale's query log when the query was originally captured
• row_count — number of rows returned (SQL) or number of <Value> elements within <CellData> in the XMLA response (MDX). 0 when no data is returned or on error.
• checksum — SHA1 hex digest of the result data. For SQL, computed over all rows serialised deterministically (columns sorted alphabetically, values tab-separated, rows newline-separated). For XMLA, computed over the SOAP <Body> content only (the <Header> is excluded because it contains per-request session IDs and timestamps). Empty when row_count = 0 or when the query fails.

Output filename:

• Task-file mode: derived from runLogFileName in the task definition (.log → .csv)
• Direct mode: {run-id}_{connection-name}.csv

Injection step compatibility (task-file mode):

↑ Table of Contents

Executes one or more queries from a query file against a live connection and writes the results to output file(s). Intended for ad-hoc inspection and debugging without the overhead of a full harness run.

--query-name supports shell-style wildcards:

• * — matches any sequence of characters
• ? — matches exactly one character

When the pattern matches a single query, results are written to --output-file directly. When it matches multiple queries, each result is written to a separate file derived from --output-file: {dir}/{query_name}{ext}.

Accepts the same query file formats as execute-atscale-query-harness:

• JSON — array of QueryRecord objects produced by extract-queries-from-atscale
• CSV — Gatling ingest format (sampler_name,sql_text or sampler_name,atscale_query_id,sql_text)

Output format depends on protocol:

• SQL — CSV with column headers and data rows
• XMLA — raw SOAP XML response body

# Execute one query by exact name
./atscale-utils execute-query-on-connection \
  --connection-file "./connections.yaml" \
  --connection-name "ats_connection" \
  --query-file "./queries/SalesModel_xmla_queries.json" \
  --protocol xmla \
  --query-name "Total Revenue by Region" \
  --output-file "./output/revenue_by_region.xml"

# Execute all queries whose names start with "sales_" — writes one CSV per query
# to ./output/sales_<query_name>.csv
./atscale-utils execute-query-on-connection \
  --connection-file "./connections.yaml" \
  --connection-name "ats_connection" \
  --query-file "./ingest/sales_queries.csv" \
  --protocol sql \
  --query-name "sales_*" \
  --output-file "./output/placeholder.csv"

# Execute all queries in the file
./atscale-utils execute-query-on-connection \
  --connection-file "./connections.yaml" \
  --connection-name "ats_connection" \
  --query-file "./queries/SalesModel_xmla_queries.json" \
  --protocol xmla \
  --query-name "*" \
  --output-file "./output/placeholder.xml"

↑ Table of Contents

Enriches a run-results CSV from execute-atscale-query-harness with the AtScale query_id, outbound SQL, and optionally the execution plan from the target data source. The correlation relies on the annotation comment that execute-atscale-query-harness prepends to every query:

/* {"run_id":"<run_id>","run_query_uuid":"<uuid>","original_text_hash":"<sha256>"} */

The operation connects to the AtScale internal Postgres backend, searches the queries, subqueries, queries_planned, and query_results tables for rows whose query_text starts with that comment, extracts the run_query_uuid, and joins enriched data back to the CSV. When --target-connection-name is provided the operation also connects to the target database and runs EXPLAIN on each outbound query. Identical outbound queries are cached so each unique SQL is explained only once.

Requirements: --annotate-queries must have been true (the default) when the harness run was executed.

./atscale-utils generate-enhanced-query-results \
  --results-file "./run_results/2026-04-21-ABC123_ats_connection.csv" \
  --connection-file "./connections.yaml" \
  --connection-name "ats_connection"

# Custom output path and wider look-back window
./atscale-utils generate-enhanced-query-results \
  --results-file "./run_results/2026-04-21-ABC123_ats_connection.csv" \
  --connection-file "./connections.yaml" \
  --connection-name "ats_connection" \
  --output-file "./run_results/enhanced.csv" \
  --days 14

Output: The input CSV with the following columns appended on the right. Rows with no AtScale match have empty values.

↑ Table of Contents

Compares two run logs from execute-atscale-query-harness, or two enhanced CSVs from generate-enhanced-query-results, on a query-by-query basis. Queries are matched by a configurable join key.

When the same join-key value appears multiple times in a file (e.g. the same query was run several times), rows are sorted by timestamp and paired positionally. Extra occurrences that cannot be paired are reported as unmatched in the summary.

Output files:

• --summary-file — plain-text report containing:

  • Input file metadata (paths, types, row counts, run IDs, unique join-key counts)
  • Match statistics
  • Queries only present in file A
  • Queries only present in file B
  • Row-count and duration mismatch counts (with a reference to --outliers-file)
  • Queries with mismatched status or error messages

• --comparison-file — CSV with one row per matched pair:

  • join_key_value, query_name, occurrence
  • Flag columns: row_count_mismatch, duration_outside_variance, error_mismatch
  • Per-side columns: a_status, b_status, a_duration_ms, b_duration_ms, a_row_count, b_row_count, a_checksum, b_checksum, a_error, b_error, a_timestamp, b_timestamp
  • Delta columns: duration_delta_ms, duration_delta_pct
  • Enhanced timing columns (when present in either input): a_run_duration_ms, b_run_duration_ms, a_run_inbound_ms, b_run_inbound_ms, a_run_query_planning_ms, b_run_query_planning_ms, a_run_outbound_ms, b_run_outbound_ms, a_run_wait_ms, b_run_wait_ms, a_run_execute, b_run_execute, a_run_fetch_ms, b_run_fetch_ms

• --outliers-file — filtered subset of --comparison-file containing only the pairs flagged for a row-count mismatch or a duration outside the variance threshold. Same CSV schema as --comparison-file.

# Compare two run logs using the default join key (original_text_hash)
./atscale-utils execute-run-analysis \
  --file-a "./run_results/2026-04-21-ABC123_model.csv" \
  --file-b "./run_results/2026-04-22-DEF456_model.csv" \
  --summary-file "./analysis/summary.txt" \
  --comparison-file "./analysis/comparison.csv" \
  --outliers-file "./analysis/outliers.csv"

# Compare enhanced outputs with a 10% duration variance threshold
./atscale-utils execute-run-analysis \
  --file-a "./run_results/2026-04-21_enhanced.csv" \
  --file-b "./run_results/2026-04-22_enhanced.csv" \
  --duration-variance-pct 10 \
  --summary-file "./analysis/summary.txt" \
  --comparison-file "./analysis/comparison.csv" \
  --outliers-file "./analysis/outliers.csv"

# Join by original_atscale_query_id instead of text hash
./atscale-utils execute-run-analysis \
  --file-a "./run_results/run_a.csv" \
  --file-b "./run_results/run_b.csv" \
  --join-key original_atscale_query_id \
  --summary-file "./analysis/summary.txt" \
  --comparison-file "./analysis/comparison.csv" \
  --outliers-file "./analysis/outliers.csv"

↑ Table of Contents

Generates a Helm values.yaml for deploying AtScale on Kubernetes. Accepts a hostname for the ingress domain and optionally a pre-existing TLS certificate. If no certificate is supplied, a self-signed RSA-2048 / SHA-256 certificate is generated automatically for the provided hostname (valid 365 days).

The tlsCrt and tlsKey fields in the output are base64-encoded PEM strings — the PEM content is base64-encoded a second time, as required by the AtScale Helm chart.

# Generate with a self-signed certificate (auto-generated)
./atscale-utils generate-atscale-install-yaml \
  --hostname "atscale.example.com"

# With a license key and existing certificate
./atscale-utils generate-atscale-install-yaml \
  --hostname     "atscale.example.com" \
  --license-key  "XXXX-XXXX-XXXX-XXXX" \
  --cert-file    "./tls.crt" \
  --key-file     "./tls.key" \
  --output-file  "./helm/values.yaml"

Output: A values.yaml ready to pass to helm install atscale ... --values values.yaml.

↑ Table of Contents

Lists the data warehouses (data sources) registered in an AtScale instance and writes the result as JSON to stdout.

./atscale-utils atscale-list-data-sources \
  --connection-file "./connections.yaml" \
  --atscale-connection-name "my_atscale"

Output: Pretty-printed JSON array to stdout. Each entry contains id, name, connectionId, and a connections array of {id, name} sub-connection objects.

The connections file must have an atscale: block in the named entry. See AtScale REST atscale fields for the connection format.

↑ Table of Contents

Registers a data warehouse (data source) in an AtScale instance. Reads the connection details from a named sql: entry in the connections file and posts them to the appropriate AtScale API endpoint based on the dialect (snowflake, databricks, or bigquery).

./atscale-utils atscale-create-data-source \
  --connection-file "./connections.yaml" \
  --atscale-connection-name "my_atscale" \
  --new-connection-name "snow_prod" \
  --aggregate-schema "ATSCALE_AGGS"

With all options:

./atscale-utils atscale-create-data-source \
  --connection-file "./connections.yaml" \
  --atscale-connection-name "my_atscale" \
  --new-connection-name "snow_prod" \
  --aggregate-schema "ATSCALE_AGGS" \
  --name "Production Snowflake" \
  --connection-id "snow_prod" \
  --access-users "admin,atscale-user"

Output: JSON response from AtScale ({id, created}) written to stdout.

The SQL connection's sql.dialect controls which API endpoint is called:

↑ Table of Contents

Lists the git repositories registered in an AtScale instance.

./atscale-utils atscale-list-repos \
  --connection-file "./connections.yaml" \
  --atscale-connection-name "my_atscale"

Output: Pretty-printed JSON array to stdout. Each entry contains id, name, url, and optional visibleBranchesPattern and defaultBranch.

↑ Table of Contents

Registers a git repository in an AtScale instance.

./atscale-utils atscale-create-repo \
  --connection-file "./connections.yaml" \
  --atscale-connection-name "my_atscale" \
  --name "my-sml-repo" \
  --url "https://github.com/myorg/sml.git" \
  --default-branch "main"

Output: JSON response from AtScale containing the created repository {id, name, url, type}.

↑ Table of Contents

Lists the deployed catalogs (semantic models) in an AtScale instance.

./atscale-utils atscale-list-deployments \
  --connection-file "./connections.yaml" \
  --atscale-connection-name "my_atscale"

Output: Pretty-printed JSON array to stdout. Each entry contains id, name, caption, publishedAt, publishedBy, and a models array.

↑ Table of Contents

Reads local SML files, uploads them to an AtScale git-backed repository, and publishes the catalog. The operation:

1. Collects all *.yml files from --sml-dir
2. Looks up the repository by --repo-name (or uses --repo-id directly)
3. Derives the project name as {catalog.unique_name}_{repo.defaultBranch} (overridable via --project-name)
4. Generates the catalog XML and calls the AtScale deploy endpoint
5. Automatically acquires an auth_session cookie via Keycloak when the deploy endpoint requires one — no manual browser cookie needed

Either apiToken (on the user entry in the users: block) or username/password credentials must be available. When apiToken is set on the user entry, the username/password from the same entry are used to acquire the session cookie for the deploy endpoint, while the API token is used for all other REST calls.

./atscale-utils atscale-deploy-catalog \
  --connection-file "./connections.yaml" \
  --atscale-connection-name "my_atscale" \
  --sml-dir "./sml-output" \
  --repo-name "my-sml-repo"

Either --repo-id or --repo-name must be provided. When only --repo-name is given, the operation calls atscale-list-repos to look up the corresponding UUID.

Output: JSON response from AtScale, e.g. {"tableau":[],"permissions":{"isSuccessful":true}}.

Connection format (apiToken and credentials both live on the user entry — the API token is used for all REST calls, while username/password are used to acquire the session cookie for the deploy endpoint):

users:
  vm_user:
    apiToken: "<api-token>"       # used for /wapi/p/ endpoints
    username: atscale-kc-admin    # used to acquire auth_session cookie for deploy
    password: "<password>"

connections:
  my_atscale:
    atscale:
      url: https://atscale.example.com
      user: vm_user

↑ Table of Contents

Validates an SML model and lists any structural or engine-level problems.

Runs in two phases:

1. Structural — local cross-reference check of all SML YAML files (datasets, dimensions, level attributes, model relationships).
2. Engine (if Phase 1 passes) — calls POST /catalog/validate-model to validate column joinability and uniqueness against the actual data warehouse.

Supports two source modes — provide exactly one of --sml-dir, --repo-name, or --repo-id.

Local mode (validate SML files on disk):

./atscale-utils atscale-list-model-errors \
  --connection-file "./connections.yaml" \
  --atscale-connection-name "my_atscale" \
  --sml-dir "./sml" \
  --model-name "sales_demo"

Remote mode (clone an AtScale-connected git repo and validate):

./atscale-utils atscale-list-model-errors \
  --connection-file "./connections.yaml" \
  --atscale-connection-name "my_atscale" \
  --repo-name "my-sml-repo" \
  --branch "main" \
  --model-name "sales_demo"

† Provide exactly one of --sml-dir, --repo-name, or --repo-id.

Output: JSON with model, problems array (each entry has phase, severity, message, optional location), and summary with errors/warnings counts.

↑ Table of Contents

Starts a GraphQL HTTP server that dynamically exposes every registered operation as a mutation. The schema is built at startup from the live operation registry — no configuration required.

File parameters (names ending in -file) accept either a local path string or a multipart-uploaded file via the Upload scalar.

See GRAPHQL.md for the full schema reference and per-operation documentation.

./atscale-utils execute-web-services
./atscale-utils execute-web-services --port 4000 --host 0.0.0.0

Example GraphQL call:

mutation {
  generateSmlFromXml(input: {
    xmlFile: "/data/project.xml"
    outputDir: "/output/sml"
  }) {
    success
    output
    error
  }
}

↑ Table of Contents

Prints the installed version of atscale-utils to stdout.

atscale-utils version
atscale-utils --version

Output: @atscale/ps-utils@<version>

This operation takes no parameters.

The extract-model-from-atscale workflow (.github/workflows/extract-model-from-atscale.yml) runs manually from the Actions tab.

Add a repository secret named ATSCALE_CONNECTION_FILE containing the full contents of your connections YAML:

Settings → Secrets and variables → Actions → New repository secret

1. Go to Actions → Extract AtScale Model → Run workflow
2. Fill in the inputs and click Run workflow
3. Download the extracted model from the Artifacts section of the run

- uses: actions/checkout@v4
- uses: actions/setup-node@v4
  with:
    node-version: 20
    cache: npm
- run: npm install && npm run build
- name: Write connection file
  run: echo "${{ secrets.ATSCALE_CONNECTION_FILE }}" > connection.yml
- name: Extract AtScale model
  run: |
    ./atscale-utils extract-model-from-atscale \
      --model "Telemetry" \
      --connection-file connection.yml \
      --connection-name "ats_connection" \
      --output-model-file "model.yml"

Passed to operations via --connection-file. Contains user credentials and named connection definitions.

Security: Do not commit this file to source control. Add it to .gitignore or supply it via a CI secret.

users:
  <user_key>:
    ...           # credentials — see below

connections:
  <connection_name>:
    installer: true | false     # optional: marks as an AtScale installer connection
    mdx:                        # optional: required for extract-model-from-atscale
      url: http://<atscale_host>
      user: <user_key>
      organization_id: default
      catalog_name: <AtScale catalog name>
    sql:                        # required for SQL operations and SML generation
      dialect: postgres | snowflake | redshift | databricks | bigquery | iris
      ...
    atscale:                    # required for atscale-list-data-sources and other REST operations
      url: https://<atscale_host>
      user: <user_key>          # key from users block, or use username/password inline
      # username: admin         # alternative: inline credentials
      # password: secret

users:
  db_user:
    username: myuser
    password: mypassword

users:
  snowflake_user:
    username: [email protected]
    privateKeyPath: resources/keys/snowflake_key.p8   # PKCS8 private key file
    privateKeyPassword: ""                             # empty string if unencrypted
    # Alternative: pre-encoded PKCS8 DER base64
    # privateKeyBase64: "<base64-der-pkcs8>"

users:
  databricks_user:
    token: "dapi1234567890abcdef"

users:
  bigquery_sa:
    key_file: resources/keys/bigquery-service-account.json
    # Alternative: inline base64-encoded JSON key
    # key_base64: "<base64-encoded-json>"

users:
  my_user:
    apiToken: "<api-token>"   # profile icon → API Token → Generate
    # Optional: also add username/password if atscale-deploy-catalog is needed
    # username: atscale-kc-admin
    # password: "<password>"

users:
  admin:
    username: admin
    password: "<password>"

connections:
  ats_connection:
    installer: true
    mdx:
      url: http://template.atscale-se-demo.com
      user: admin
      organization_id: default
      catalog_name: Telemetry
    sql:
      dialect: postgres
      server: template.atscale-se-demo.com
      port: 15432
      database: atscale
      schema: Telemetry
      user: admin

users:
  snowflake_user:
    username: [email protected]
    privateKeyPath: resources/keys/snowflake_key.p8
    privateKeyPassword: ""

connections:
  snow_demo:
    sql:
      dialect: snowflake
      account: da37161
      warehouse: COMPUTE_WH
      database: MY_DATABASE
      schema: MY_SCHEMA
      role: SYSADMIN
      snowflake_user: snowflake_user

Redshift uses the Postgres wire protocol. The pg driver is used directly — no AWS SDK required.

users:
  rs_user:
    username: awsuser
    password: mypassword

connections:
  redshift_prod:
    sql:
      dialect: redshift
      server: my-cluster.abc123.us-east-1.redshift.amazonaws.com
      port: 5439
      database: analytics
      schema: public
      user: rs_user

Connects to a Databricks SQL Warehouse via the official @databricks/sql driver.

users:
  databricks_user:
    token: "dapi1234567890abcdef1234567890abcdef"

connections:
  databricks_prod:
    sql:
      dialect: databricks
      host: adb-1234567890123456.7.azuredatabricks.net
      path: /sql/1.0/warehouses/abc1234567890def
      catalog: main
      schema: sales
      databricks_user: databricks_user

Connects via the @google-cloud/bigquery client. Authentication uses a service account key file or Application Default Credentials (ADC).

ADC: If bigquery_user is omitted, the driver uses the environment's Application Default Credentials (GOOGLE_APPLICATION_CREDENTIALS env var or gcloud login). This is the recommended approach when running in GCP (Cloud Run, GKE, etc.).

users:
  bigquery_sa:
    key_file: resources/keys/bigquery-service-account.json

connections:
  bq_prod:
    sql:
      dialect: bigquery
      project_id: my-gcp-project-123
      dataset: analytics
      location: US
      bigquery_user: bigquery_sa

connections:
  bq_adc:
    sql:
      dialect: bigquery
      project_id: my-gcp-project-123
      dataset: analytics

Used by atscale-list-data-sources and other operations that call the AtScale public REST API. Authenticates via Keycloak OIDC (the standard AtScale auth flow — the same token endpoint used by atscale-ai-link).

† Either user (referencing a users: entry with apiToken or username/password) or inline username/password must be provided.

Keycloak client troubleshooting:

users:
  my_user:
    apiToken: "<api-token>"   # profile icon → API Token → Generate

connections:
  my_atscale:
    atscale:
      url: https://atscale.example.com
      user: my_user

users:
  admin:
    username: admin
    password: "<password>"

connections:
  my_atscale:
    atscale:
      url: https://atscale.example.com
      user: admin

connections:
  my_atscale:
    atscale:
      url: https://atscale.example.com
      username: admin
      password: "<password>"

Connects to InterSystems IRIS via the intersystems-iris community driver.

Note: The IRIS community npm package has a single-connection-per-process limitation. It is suitable for sequential operations but not concurrent workloads.

users:
  iris_user:
    username: _SYSTEM
    password: SYS

connections:
  iris_prod:
    sql:
      dialect: iris
      server: iris.example.com
      port: 1972
      namespace: MYAPP
      user: iris_user

An optional YAML file that stores SML generation parameters so you don't have to repeat them on every command. Used by generate-sml-from-ddl, generate-sml-from-connection, and generate-metrics-from-model.

Priority: CLI flags always override the file. Any parameter not supplied on the CLI falls back to the file, then to the hardcoded default.

Input vs. output: --sml-config-file is the input path only. After generation, the effective settings (all values including defaults) are always written to <output-dir>/sml.style.yaml — a fixed location independent of the input path. If --sml-config-file points to the same file (e.g. you pass --sml-config-file sml-output/sml.style.yaml), it is simply overwritten.

Reference copy: A fully annotated reference file with all parameters and their defaults lives at docs/sml.style.yaml. Copy it to your working directory as a starting point.

Style guide: docs/STYLE.md documents the naming conventions, casing rules, and generation settings that sml.style.yaml controls.

# ── generate-sml-from-ddl / generate-sml-from-connection ─────────────────────
pii-severity: MEDIUM          # "HIGH" | "MEDIUM" | "LOW" | "none"
fact-tables:                  # force-classify tables as facts (list or empty)
  - FactSales
  - FactOrders
catalog-name: My Catalog      # display name; omit to default to model-name
camel-case-files: false       # camelCase SML filenames
camel-case-measures: false    # camelCase metric labels (deprecated — use label-style)
label-style: title-case       # label style for all SML labels (title-case/camel-case/none)
sample-size: 250              # rows per table for type inference (0 to disable)
                              # note: always 0 for generate-sml-from-ddl
min-hierarchies-per-dim: 1   # drop dimensions with fewer than this many hierarchies
max-hierarchies-per-dim: 4   # keep at most this many hierarchies per dimension

# ── generate-metrics-from-model ───────────────────────────────────────────────
max-suggestions: 25           # maximum suggestions to output
min-score: 0.5                # minimum relevance score [0–1]
include-tuples: true          # include multi-dimension suggestions

First run (no file yet — all defaults apply):

./atscale-utils generate-sml-from-ddl \
  --ddl-file schema.sql \
  --output-dir sml-output
# → sml-output/sml.style.yaml written with all effective settings

Edit sml-output/sml.style.yaml, then re-run pointing --sml-config-file at it. The output overwrites the same file:

./atscale-utils generate-sml-from-ddl \
  --ddl-file schema.sql \
  --output-dir sml-output \
  --sml-config-file sml-output/sml.style.yaml
# → sml-output/sml.style.yaml overwritten with updated effective settings

Or keep a project-level sml.style.yaml in the working directory (the default input path) and let the output land separately in the output directory:

# Reads from ./sml.style.yaml (CWD default), writes to sml-output/sml.style.yaml
./atscale-utils generate-sml-from-connection \
  --connection-file connections.yaml \
  --connection-name snow_demo \
  --model-name SalesModel \
  --output-dir sml-output

Auto-generated by extract-model-from-atscale or extract-model-from-sml. Describes one or more AtScale models with two parallel sections per model: mdx (MDX query metadata) and sql (column metadata used by the BI generators).

<ModelName>:
  data_source: <connection-name>
  mdx:
    metrics: [...]
    attributes: {...}
  sql:
    table_name: <ModelName>
    columns:
      <column_name>:
        alias: false
        name: <column_name>
        data_type: <DATA_TYPE>
        label: <Human-readable label>
        description: ""
        role: measure | dimension
        type: quantitative | nominal | ordinal
        aggregation: sum | avg | min | max | count | countd   # measures only
        folder: ""

Column names in sql.columns are what you reference in namespace YAML fields (xAxis, yAxis, measures, colorField, filters[].field). When an --aliases-file is supplied, sql.columns is extended with alias entries and the original columns are preserved under sql.rawColumns.

Telemetry:
  data_source: ats_connection
  sql:
    table_name: Telemetry
    columns:
      m_query_id_count:
        alias: false
        name: m_query_id_count
        data_type: INT8
        label: Total Queries
        description: ""
        role: measure
        type: quantitative
        aggregation: count
        folder: ""
      query_hour:
        alias: false
        name: query_hour
        data_type: DATETIME
        label: Query Hour
        description: ""
        role: dimension
        type: nominal
        folder: ""

Drives the BI workbook generators. Defines worksheets and dashboards.

version: 1
title: <Workbook title>
description: <Description>

worksheets:
  <worksheet_key>:
    title: <Display name>
    model: <ModelName>          # key from model.yaml
    graphType: bar | line | text
    ...

dashboards:
  <dashboard_key>:
    ...

Horizontal bar chart. xAxis = measure (bar length), yAxis = dimension (bar labels).

top_users:
  title: Top 10 Most Active Users
  model: Telemetry
  graphType: bar
  xAxis: m_query_id_count       # measure column
  yAxis: user_hash              # dimension column
  limit: 10                     # optional: top-N rows
  sortDirection: desc           # optional: asc | desc
  colorField: service           # optional: dimension for color encoding
  format: integer               # optional: integer | decimal:N | percent:N | currency:N
  filters:
    - field: user_hash
      excludeNull: true

Time-series line chart. xAxisGranularity selects a specific level from the time hierarchy when the AtScale model has named levels (e.g. "Day", "Week"). If the hierarchy has only one level the field is recorded but has no effect on the set expression.

queries_over_time:
  title: Queries by Week
  model: Telemetry
  graphType: line
  xAxis: query_hour             # typically a DATETIME dimension
  xAxisGranularity: week        # optional: day | week | month | quarter | year
  yAxis: m_query_id_count       # measure (Y axis)
  colorField: service           # optional: dimension for series breakdown
  filters:
    - field: succeeded
      excludeNull: true

Single-number KPI scorecard.

total_queries:
  title: Total Queries
  model: Telemetry
  graphType: text
  measures:
    - m_query_id_count
  format: integer               # optional: integer | decimal:N | percent:N | currency:N

Applies a number format to measure values in Excel. Accepted values:

filters:
  - field: <column_name>        # must exist in model's sql.columns (or aliases)
    excludeNull: true           # exclude null/blank members

dashboards:
  my_dashboard:
    title: Telemetry Dashboard
    description: ""

    size:
      width: 1800               # total pixel width
      height: 4900              # total pixel height
      hSegments: 3              # grid columns
      vSegments: 49             # grid rows

    categoryHeaders:            # optional: full-width section title zones
      - label: "Summary Statistics"
        x: 0
        y: 0
        colSpan: 3
        rowSpan: 1

    tiles:
      - worksheet: total_queries
        x: 0                    # grid column (0-based)
        y: 1                    # grid row (0-based)
        colSpan: 1
        rowSpan: 2
        category: Summary Statistics

Grid cell size = width / hSegments × height / vSegments px. x + colSpan must not exceed hSegments.

See resources/namespaces/telemetry/overview.yaml.

An optional file passed via --aliases-file to generate-tableau-from-namespace and generate-excel-from-namespace. It lets you define friendly names for model columns so that namespace files can reference an alias instead of the underlying column key.

When an aliases file is provided, sql.columns in the model is extended: each alias entry adds a copy of the referenced column under the alias name. The original columns remain unchanged. The pre-merge snapshot is preserved as sql.rawColumns, and the aliases map is attached as model.aliases.

Any field in the namespace that accepts a column name (xAxis, yAxis, measures, colorField, filters[].field) can use either the original column key or any defined alias.

The file is a flat YAML map of alias_name: original_column_key:

<alias>: <original_column_key>

query_time: query_hour
total_queries: m_query_id_count
response_time_avg: m_epoch_sql_wall_time_avg
subquery_count: m_subquery_count_sum
success_rate: cm_query_success_pct

With this file, a namespace worksheet can use response_time_avg as a yAxis and it will resolve to the m_epoch_sql_wall_time_avg column in the model.

See example/aliases.yaml for the full example.