Add iotdb-thingsboard-table module for ThingsBoard on IoTDB Table Mode#110
Open
PDGGK wants to merge 1 commit into
Open
Add iotdb-thingsboard-table module for ThingsBoard on IoTDB Table Mode#110PDGGK wants to merge 1 commit into
PDGGK wants to merge 1 commit into
Conversation
Implements ThingsBoard's historical telemetry TimeseriesDao SPI on Apache IoTDB 2.0.8 Table Mode (ITableSession SQL + Tablet writes). This first PR delivers the write + raw-read foundation: IoTDBTableBaseDao, an async bounded-queue batch writer, and the raw findAllAsync/remove read-delete path. Aggregation, latest telemetry and attributes land in later PRs. The module is inert by default: the live DAO/pool/writer/schema-bootstrap activate only on an explicit database.ts.type=iotdb-table plus iotdb.ts.experimental-raw-only=true opt-in, the auto-configuration is classpath-isolated from a non-ThingsBoard runtime, and ThingsBoard SPI types are a compile-only source surface excluded from the built jar (Strategy F). It is added to the reactor through a JDK-17 profile and overrides tsfile only within its own module pom, so the rest of the reactor is unaffected. Signed-off-by: Zihan Dai <[email protected]>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this adds
A new Maven module
iotdb-thingsboard-tablethat implements ThingsBoard's historical telemetry DAO SPI on top of Apache IoTDB 2.0.8 Table Mode (relational SQL viaITableSession/ Tablet writes). This is the first PR of a staged series; it delivers the write + raw-read foundation, packaged so it activates only on an explicit opt-in and otherwise stays completely inert.DAO implementation
IoTDBTableBaseDao—ITableSessionPoolwiring (Spring constructor injection),iotdb.*configuration binding, andgetEntry()mapping of the five typed columns (bool_v/long_v/double_v/str_v/json_v) with fail-fast on the single-typed-column schema invariant.IoTDBTableTimeseriesDao.save()— an async bounded-queue batch writer: rows are mapped to a sparse IoTDB Tablet (TAG columns declared low-cardinality-first —entity_type, tenant_id, key, entity_id— plus one populated typed FIELD perDataType), flushed by size/linger, with retry+backoff on connection errors, reject-on-full back-pressure (fails the future rather than blocking the caller, with reject counters and a rate-limited WARN — at most one per 10s with cumulative counts — so dropped points are never silent), and a graceful shutdown drain.findAllAsync(raw) +remove+savePartition— the non-aggregated read path (half-open ranges, escaped key/order, rows mapped back toBasicTsKvEntry), the delete path, and the partition no-op, all on a bounded read thread pool. Per the ThingsBoardAbstractChunkedAggregationTimeseriesDaocontract, a query is routed to raw whenaggregation == NONE || interval < 1.IoTDBTableTimeseriesDaothrowsUnsupportedOperationExceptionand is unreachable by default thanks to the explicit raw-only opt-in.IoTDBTableLatestDao/IoTDBTableAttributesDaoship only as unregistered inert skeletons (not Spring beans, no ThingsBoard interface binding), so no configuration selector can route traffic to a non-working DAO.Activation & runtime (inert by default, opt-in foundation)
IoTDBTableConfigurationis an@AutoConfigurationregistered viaMETA-INF/spring/...AutoConfiguration.imports+META-INF/spring.factories. The outer class carries a string-based@ConditionalOnClass(name="org.thingsboard.server.dao.timeseries.TimeseriesDao")and contains no bean method or annotation that force-loads a ThingsBoard type, so on a non-ThingsBoard classpath Spring evaluates the condition from ASM metadata and skips the module without aNoClassDefFoundError. All ThingsBoard-referencing beans live in a nested@Configuration;@ConditionalOnMissingBean(type=...)uses the string form.TimeseriesDao, session pool, writer and schema bootstrap activate only when BOTHdatabase.ts.type=iotdb-tableANDiotdb.ts.experimental-raw-only=trueare set. A normal deployment (selector alone, or neither) gets nothing, so the not-yet-implemented aggregation path is never reachable through the public SPI. Activation uses a small case-insensitiveCondition, not SpEL.ITableSessionPoolcan never silently substitute for it.TimeseriesDao, aBeanFactoryPostProcessorfails startup with a clear message (trusting only resolvable bean types, fail-closed) rather than leaving the pool/bootstrap running while the DAO is shadowed.IoTDBTableSchemaBootstrap— creates the IoTDB database/table on first activation (same opt-in guard +@ConditionalOnProperty(iotdb.schema.bootstrap, matchIfMissing=true)), so the first write does not fail on a missing table. The configured database name is validated before it is spliced into DDL.iotdb.*config is bound/validated only when the backend is selected (@EnableConfigurationPropertiessits on the conditional nested config), so an unrelated host with strayiotdb.*properties is unaffected.ThingsBoard compile surface (Strategy F)
ThingsBoard's
dao/commonartifacts are not published to Maven Central, so the SPI/value types the module compiles against are provided as a compile-only source surface undersrc/provided/javaand excluded from the built jar (org/thingsboard/**,org/apache/commons/**); at runtime the real ThingsBoard classpath provides them. The surface is kept in sync with ThingsBoard v4.3.1.2 (each stub file carries a provenance header with the verified version + date), andStrategyFContractTestpins the exactTimeseriesDaoSPI signatures the DAO depends on so any accidental drift fails the build. Integration tests run againsttarget/classes(which retains the compile surface).Reactor / build impact
<jdk>[17,)</jdk>(it uses Java-17 language features), so existing JDK 8/11 root builds skip it and only 17/21 jobs build it — no change to current CI on older JDKs.tsfile.versionis left untouched at2.1.1. The iotdb-session 2.0.8 Tablet write API needs tsfile2.3.0, so the bump is a module-local property override iniotdb-thingsboard-table/pom.xml— only this module resolves the newer tsfile; every other connector keeps2.1.1, so there is zero blast radius on the rest of the reactor.-Piotdb-table-itprofile, so a defaultmvn installruns unit tests only and does not require Docker.Tests
73 unit tests (type mapping, batch flush/linger, back-pressure, retry, shutdown drain + forced-stop settle guarantees, reject-WARN rate limiting, raw read SQL/mapping, half-open delete, blank-key fail-fast, read-pool reject/drain, auto-config discovery + classpath isolation, named-pool ownership, conflict-guard fail-fast, schema bootstrap + DDL column-order pins, Strategy-F contract) plus 9 Testcontainers integration tests against a real
apache/iotdb:2.0.8-standalonecontainer (round-trip all five types, order/limit, half-open end, scoped delete, key escaping, fresh-database bootstrap through a database-bound pool).mvn apache-rat:checkis clean.Known limitation (documented, in the active path)
This affects the write + raw-read surface delivered here, not a deferred placeholder: if the same
(tenant, entity, key, ts)point has its data type changed across two separate flushes, the two typed columns coexist on that row, and the raw read fails fast on the single-typed-column invariant (IoTDBTableBaseDao.getEntry) rather than silently returning a wrong value. Within a single flush the writer already de-duplicates such a type change. Full delete-then-insert reconciliation across flushes is deferred to a later PR in the series. The behaviour is pinned by a unit test and an integration test and documented in the module README.Context
This module is the first deliverable of a Google Summer of Code 2026 project to add an Apache IoTDB 2.x Table Mode storage backend to ThingsBoard. The design (schema, current-state analysis, Spring activation, and the staged-PR plan) was shared and discussed earlier on the
[email protected]list.