Skip to content

Add iotdb-thingsboard-table module for ThingsBoard on IoTDB Table Mode#110

Open
PDGGK wants to merge 1 commit into
apache:masterfrom
PDGGK:pr1
Open

Add iotdb-thingsboard-table module for ThingsBoard on IoTDB Table Mode#110
PDGGK wants to merge 1 commit into
apache:masterfrom
PDGGK:pr1

Conversation

@PDGGK

@PDGGK PDGGK commented Jun 14, 2026

Copy link
Copy Markdown

What this adds

A new Maven module iotdb-thingsboard-table that implements ThingsBoard's historical telemetry DAO SPI on top of Apache IoTDB 2.0.8 Table Mode (relational SQL via ITableSession / Tablet writes). This is the first PR of a staged series; it delivers the write + raw-read foundation, packaged so it activates only on an explicit opt-in and otherwise stays completely inert.

DAO implementation

  • IoTDBTableBaseDaoITableSessionPool wiring (Spring constructor injection), iotdb.* configuration binding, and getEntry() mapping of the five typed columns (bool_v/long_v/double_v/str_v/json_v) with fail-fast on the single-typed-column schema invariant.
  • IoTDBTableTimeseriesDao.save() — an async bounded-queue batch writer: rows are mapped to a sparse IoTDB Tablet (TAG columns declared low-cardinality-first — entity_type, tenant_id, key, entity_id — plus one populated typed FIELD per DataType), flushed by size/linger, with retry+backoff on connection errors, reject-on-full back-pressure (fails the future rather than blocking the caller, with reject counters and a rate-limited WARN — at most one per 10s with cumulative counts — so dropped points are never silent), and a graceful shutdown drain.
  • findAllAsync (raw) + remove + savePartition — the non-aggregated read path (half-open ranges, escaped key/order, rows mapped back to BasicTsKvEntry), the delete path, and the partition no-op, all on a bounded read thread pool. Per the ThingsBoard AbstractChunkedAggregationTimeseriesDao contract, a query is routed to raw when aggregation == NONE || interval < 1.
  • Aggregation, latest telemetry and attributes arrive in later PRs. The not-yet-implemented positive-interval aggregation path in IoTDBTableTimeseriesDao throws UnsupportedOperationException and is unreachable by default thanks to the explicit raw-only opt-in. IoTDBTableLatestDao / IoTDBTableAttributesDao ship only as unregistered inert skeletons (not Spring beans, no ThingsBoard interface binding), so no configuration selector can route traffic to a non-working DAO.

Activation & runtime (inert by default, opt-in foundation)

  • Spring auto-configuration with classpath isolationIoTDBTableConfiguration is an @AutoConfiguration registered via META-INF/spring/...AutoConfiguration.imports + META-INF/spring.factories. The outer class carries a string-based @ConditionalOnClass(name="org.thingsboard.server.dao.timeseries.TimeseriesDao") and contains no bean method or annotation that force-loads a ThingsBoard type, so on a non-ThingsBoard classpath Spring evaluates the condition from ASM metadata and skips the module without a NoClassDefFoundError. All ThingsBoard-referencing beans live in a nested @Configuration; @ConditionalOnMissingBean(type=...) uses the string form.
  • Inert until explicitly opted in — because this PR delivers only the write + raw-read path (aggregation lands in a later PR), the live TimeseriesDao, session pool, writer and schema bootstrap activate only when BOTH database.ts.type=iotdb-table AND iotdb.ts.experimental-raw-only=true are set. A normal deployment (selector alone, or neither) gets nothing, so the not-yet-implemented aggregation path is never reachable through the public SPI. Activation uses a small case-insensitive Condition, not SpEL.
  • Module-owned session pool — the module registers its session pool under a dedicated bean name and injects it everywhere by qualifier, so a host-provided ITableSessionPool can never silently substitute for it.
  • No silent half-activation — if the backend is enabled but the host already provides a conflicting non-IoTDB TimeseriesDao, a BeanFactoryPostProcessor fails startup with a clear message (trusting only resolvable bean types, fail-closed) rather than leaving the pool/bootstrap running while the DAO is shadowed.
  • IoTDBTableSchemaBootstrap — creates the IoTDB database/table on first activation (same opt-in guard + @ConditionalOnProperty(iotdb.schema.bootstrap, matchIfMissing=true)), so the first write does not fail on a missing table. The configured database name is validated before it is spliced into DDL.
  • iotdb.* config is bound/validated only when the backend is selected (@EnableConfigurationProperties sits on the conditional nested config), so an unrelated host with stray iotdb.* properties is unaffected.

ThingsBoard compile surface (Strategy F)

ThingsBoard's dao/common artifacts are not published to Maven Central, so the SPI/value types the module compiles against are provided as a compile-only source surface under src/provided/java and excluded from the built jar (org/thingsboard/**, org/apache/commons/**); at runtime the real ThingsBoard classpath provides them. The surface is kept in sync with ThingsBoard v4.3.1.2 (each stub file carries a provenance header with the verified version + date), and StrategyFContractTest pins the exact TimeseriesDao SPI signatures the DAO depends on so any accidental drift fails the build. Integration tests run against target/classes (which retains the compile surface).

Reactor / build impact

  • The module is added to the root reactor through a profile activated by <jdk>[17,)</jdk> (it uses Java-17 language features), so existing JDK 8/11 root builds skip it and only 17/21 jobs build it — no change to current CI on older JDKs.
  • The parent tsfile.version is left untouched at 2.1.1. The iotdb-session 2.0.8 Tablet write API needs tsfile 2.3.0, so the bump is a module-local property override in iotdb-thingsboard-table/pom.xml — only this module resolves the newer tsfile; every other connector keeps 2.1.1, so there is zero blast radius on the rest of the reactor.
  • The module's Testcontainers integration tests run only under an explicit -Piotdb-table-it profile, so a default mvn install runs unit tests only and does not require Docker.

Tests

73 unit tests (type mapping, batch flush/linger, back-pressure, retry, shutdown drain + forced-stop settle guarantees, reject-WARN rate limiting, raw read SQL/mapping, half-open delete, blank-key fail-fast, read-pool reject/drain, auto-config discovery + classpath isolation, named-pool ownership, conflict-guard fail-fast, schema bootstrap + DDL column-order pins, Strategy-F contract) plus 9 Testcontainers integration tests against a real apache/iotdb:2.0.8-standalone container (round-trip all five types, order/limit, half-open end, scoped delete, key escaping, fresh-database bootstrap through a database-bound pool). mvn apache-rat:check is clean.

Known limitation (documented, in the active path)

This affects the write + raw-read surface delivered here, not a deferred placeholder: if the same (tenant, entity, key, ts) point has its data type changed across two separate flushes, the two typed columns coexist on that row, and the raw read fails fast on the single-typed-column invariant (IoTDBTableBaseDao.getEntry) rather than silently returning a wrong value. Within a single flush the writer already de-duplicates such a type change. Full delete-then-insert reconciliation across flushes is deferred to a later PR in the series. The behaviour is pinned by a unit test and an integration test and documented in the module README.

Context

This module is the first deliverable of a Google Summer of Code 2026 project to add an Apache IoTDB 2.x Table Mode storage backend to ThingsBoard. The design (schema, current-state analysis, Spring activation, and the staged-PR plan) was shared and discussed earlier on the [email protected] list.

Implements ThingsBoard's historical telemetry TimeseriesDao SPI on Apache
IoTDB 2.0.8 Table Mode (ITableSession SQL + Tablet writes). This first PR
delivers the write + raw-read foundation: IoTDBTableBaseDao, an async
bounded-queue batch writer, and the raw findAllAsync/remove read-delete
path. Aggregation, latest telemetry and attributes land in later PRs.

The module is inert by default: the live DAO/pool/writer/schema-bootstrap
activate only on an explicit database.ts.type=iotdb-table plus
iotdb.ts.experimental-raw-only=true opt-in, the auto-configuration is
classpath-isolated from a non-ThingsBoard runtime, and ThingsBoard SPI
types are a compile-only source surface excluded from the built jar
(Strategy F). It is added to the reactor through a JDK-17 profile and
overrides tsfile only within its own module pom, so the rest of the
reactor is unaffected.

Signed-off-by: Zihan Dai <[email protected]>

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi, this is your first pull request in IoTDB project. Thanks for your contribution! IoTDB will be better because of you.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant