5.8.2

Release Date: 2024-11-12

Enhancements

Core MQTT Functionalities

• #14059 Added a new configuration option for the retainer to cap message expiry intervals for retained messages. This enables garbage collection to remove messages sooner if storage is running low.

• #14072 Updated the virtual machine to use Unicode for its printable range. This improvement enhances the readability of certain binary data in messages. For instance, a binary previously displayed as <<116,101,115,116,228,184,173,230,150,135>> will now be formatted as <<"test中文"/utf8>>, providing clearer representation.

MQTT Durable Sessions

• #14130 Reduced CPU usage for idle durable sessions.

  Previously, idle durable sessions periodically woke up to refresh the list of DS streams. With this change, stream discovery is now event-based, significantly lowering CPU consumption during idle periods. Additionally, the update reduces the delay in notifying sessions of new streams, effectively eliminating the long-tail latency in end-to-end processing.

REST API

• #13889 Enhanced the performance of the /api/v5/monitor_current and /api/v5/metrics APIs.

  Previously, these APIs queried clustered nodes sequentially in a loop. Now, the queries are sent in parallel, reducing response time. The latency is now primarily dependent on the slowest node in the cluster.

  Additionally, a node parameter was added to the /api/v5/monitor_current API, allowing targeted queries to a single node instead of the entire cluster. For instance, using ?aggregate=false&[email protected] will return data exclusively for the specified node.

EMQX Clustering

• #13903 Added logs to inform the user when a replicant node cannot find a core node with the same release version as its own.

Security

• #13923 Added zone support in authentication, authorization, and mountpoint templates.

  Previously, to reference a client's zone in authentication or authorization rules, users needed to access it through client_attrs. Now, the ${zone} placeholder can be used directly in these templates, simplifying rule creation and enabling zone-specific configurations.

  For example, the following ACL rule uses ${zone} to dynamically apply permissions based on a client’s assigned zone: {allow, all, all, ["${zone}/${username}/#"]}.

• #14102 Added support for SSL private key passphrase from a secret file.

  EMQX can now read the passphrase from a secret file if password is configured as ...ssl_options.password = "file://{path-to-secret-file}".

Data Integration

• #14065 Added a new queuing_bytes metric for data integration. This metric shows the RAM and/or disk resources consumed by buffering for a specific action.

Observability

• #14096 Exposed emqx_conf_sync_txid as a Prometheus metric, allowing for monitoring the configuration file synchronization status of each node in the cluster.

MQTT over QUIC

• #13814 Connection Scope Keepalive for MQTT over QUIC Multi-Stream:

  Introduced a new feature to keep MQTT connections alive when data streams remain active, even if the control stream is idle.

  Previously, clients were required to send MQTT.PINGREQ on idle control streams to keep the connection alive. Now, a shared state tracks activity across all streams for each connection. This shared state is used to determine if the connection is still alive, reducing the risk of keepalive timeouts due to Head-of-Line (HOL) blocking.

• #14112 Added support ssl_options.hibernate_after in QUIC listener to reduce memory footprint of QUIC transport.

Bug Fixes

Core MQTT Functionality

• #13931 Updated the gen_rpc library to version 3.4.1, which includes a fix to prevent client socket initialization errors from escalating to the node level on the server side.

• #13969 Optimized the periodic cleanup of expired retained messages to ensure efficient resource usage, particularly in cases with a large volume of expired messages.

• #14068 Added the handle_frame_error/2 callback to all gateway implementation modules to handle message parsing errors.

• #14037 Improved the internal database bootstrap process to better tolerate temporary unavailability of peer nodes, particularly when a new node joins an existing cluster.

• #14116 Fixed an issue where the default configuration for the retainer was generated incorrectly after joining a cluster.

MQTT Durable Sessions

• #14042 Fix crash in the durable session after updates to subscription parameters (such as QoS, no_local, upgrade_qos, ...).

• #14052 Corrected memory usage reporting from cgroups when in use.

• #14055 Updated the /clients_v2 API to properly respect all filtering arguments when querying offline clients with durable sessions. Previously, only the username filter was applied, while other filtering arguments were ignored.

• #14151 Fixed handling of the conn_state filter in the /clients_v2 API for offline clients with durable sessions. Previously, these clients could be incorrectly selected with conn_state=connected.

• #14057 Resolved a compatibility issue that prevented the Messages DS database from starting due to a slightly different database configuration schema. This issue occurred when upgrading EMQX from version 5.7.x with session durability enabled.

REST API

• #14023 Fixed an issue with the GET /monitor HTTP API where returned values could appear higher than actual values, depending on the requested time window. For data points within a 1-hour window, this distortion is only visual on the Dashboard. However, for data points older than 1 hour, the data distortion is permanent.

  The affected metrics include:

  • disconnected_durable_sessions
  • subscriptions_durable
  • subscriptions
  • topics
  • connections
  • live_connections

EMQX Clustering

• #13996 Fixed an intermittent crash occurring when using emqx conf fix to resolve configuration discrepancies, particularly if a configuration key was missing on one of the nodes.

Security

• #13922 Updated the CRL (Certificate Revocation List) cache to use the full Distribution Point (DP) URL as the cache key. Previously, only the path part of the URL was used, causing conflicts when multiple DPs shared the same path.

• #13924 Fixed an issue where JWK keys could leak into debug logs upon JWT authentication failure.

Data Integration

• #13916 Fixed an issue where the parent metric failed was not incremented when a rule’s failed.no_result or failed.exception metrics were updated.

• #14001 Resolved a race condition where a resource (such as a connector, action, source, authentication, or authorization) could falsely report a connected, healthy channel after a brief disconnection. This issue could result in excessive action_not_found log entries when the race condition occurred.

• #13913 Fixed an issue with the actions and source HTTP APIs where a 500 status code would be returned if a timeout occurred while attempting to update or delete a resource.

• #14101 Resolved an issue where deleting a resource would fail if a source and an action were both created with the same name.

Observability

• #13909 Fixed log formatting for cases where the payload cannot be displayed as readable UTF-8 Unicode characters.

• #14061 Improved log information when emqx_cm:request_stepdown/3 fails.

  In scenarios where a client channel needs to terminate another channel with the same ClientID, a race condition may occur if the target channel has already been closed or terminated. In such cases, error logs and stack traces that provide no useful information will no longer be generated.

• #14070 Removed the connector's state from error and warning logs due to its potential length. For issue analysis, the connector's state can now be accessed through emqx_resource:list_instances_verbose/0. Below is an example of a log entry before this change:

  pid: <0.43914.0>, connector: connector:sqlserver:connector-05a2e105, reason: [Microsoft][ODBC Driver 17 for SQL Server][SQL Server]Argument data type varchar is invalid for argument 2 of dateadd function. SQLSTATE IS: 42000, state: {"resource_opts":{"start_timeout":5000,"start_after_created":true,"health_check_interval":15000},"pool_name":"connector:sqlserver:connector-05a2e105","installed_channels":{"action:sqlserver:action-4b033621:connector:sqlserver:connector-05a2e105":{"sql_templates":{"batch_insert_t...
  

5.8.1

Release Date: 2024-10-14

Make sure to check the breaking changes and known issues before upgrading to EMQX 5.8.1.

Important Changes

• #13956 Updated the gen_rpc library to version 3.4.1, which includes a node crash issue.
  Previously, if a node is force shutdown down while RPC channels are being established, it may cause a cluster peer node to crash.

Enhancements

Core MQTT Functionalities

• #13525 Added new configuration item shared_subscription_initial_sticky_pick to specify the strategy for making the initial pick when shared_subscription_strategy is set to sticky.

• #13942 The HTTP client now automatically reconnects if no activity is detected for 10 seconds after the latest request has expired.
  Previously, it would wait indefinitely for a server response, causing timeouts if the server dropped requests.

  This change impacts below components.

  • HTTP authentication
  • HTTP authorization
  • Webhook (HTTP connector)

Authentication and Authorization

• #13863 EMQX now supports ${cert_common_name} placeholder in topic name templates for raw ACL rules.

• #13792 The banned-clients API GET /banned supports querying the rules using filters in the query string.

  The available filters are:

  • clientid
  • username
  • peerhost
  • like_clientid
  • like_username
  • like_peerhost
  • like_peerhost_net

  When adding a new banned client entry, the default expiration time for entries without the until parameter specified has been changed from 1 year to infinite.

Rule Engine

• #13773 Disabled rule actions now do not trigger out_of_service warnings.

  Previously, if an action is disabled, there would be a warning log with msg: out_of_service,
  and the actions.failed counter was incremented for the rule.

  After this enhancement, disabled action will result in a debug level log with msg: discarded,
  and the newly introduced counter actions.discarded will be incremented.

MQTT over QUIC

• #13814 Connection Scope Keepalive for MQTT over QUIC Multi-Stream:

  This update introduces a new feature to maintain MQTT connections over QUIC multi-streams, even when the control stream is idle but other data streams are active.

  Previously, clients had to send MQTT.PINGREQ on idle control streams to keep the connection alive. Now, a shared state is maintained for each connection, monitoring activity across all streams. This shared state helps determine if the connection is still active, reducing the risk of keepalive timeouts caused by Head-of-Line (HOL) blocking and improving overall connection stability.

Bug Fixes

Core MQTT Functions

• #13702 Clean up the corresponding exclusive subscriptions when a node goes down.

• #13708 Fixed an issue which may cause shared subscription 'sticky' strategy to degrade to 'random'.

• #13733 Made cacertfile optional when configuring https listener from emqx ctl conf load command.

• #13742 Fixed when subscribing with + as the first level, or # as a wildcard, retained messages with topics starting with $ are incorrectly received.

• #13754 Fixed an issue when websocket connection would break consistently on its own.

• #13756 Introduced more randomness to broker assigned client IDs.

• #13790 The default heartbeat interval for the MQTT connector has been reduced from 300 seconds to 160 seconds.

  This change helps maintain the underlying TCP connection by preventing timeouts due to the idle limits
  imposed by load balancers or firewalls, which typically range from 3 to 5 minutes depending on the cloud provider.

• #13832 Fixed that the Publish endpoint would have a 500 error when persistent session were enabled.

• #13842 Fixed a UTF-8 string validation exception.

Upgrade and Migration

• #13731 Resolved an issue that prevented clusters running on EMQX 5.4.0 from upgrading to EMQX 5.8.0. This fix introduces a migration procedure to update specific internal database tables created in version 5.4.0 to align with the new schema.

Breaking Changes

• #13792 The default expiration time for a banned item that is created without an until value is now infinity (previsouly capped at 1 year limit).

• #13742 Fixed an issue when a client would receive retained messages for a topic starting with $ when it subscribed to topic # or +.

  This fix satisfies the requirement of MQTT-4.7.2-1.

Enhancements

Core MQTT Functionality

• #13009 Updated the log level for message receiving pause due to rate limiting from debug to warning. The log message socket_receive_paused_by_rate_limit is throttled to avoid excessive logging.

Authentication and Authorization

• #12418 Enhanced JWT authentication to support claims verification using a list of objects:

  [
    {
      name = "claim_name",
      value = "${username}"
    },
    ...
  ]
  

  Expected values are now treated as templates, consistent with other authenticators, allowing for arbitrary expressions such as ${username} and ${clientid}. Previousy, only fixed "${username}" "${clientid}" values were supported for interpolation.

  Improved the documentation for the verify_claims parameter.

• #13229 Added support for ${cert_pem} placeholder in authentication templates.

• #13534 Added trace logging to indicate when the superuser bypasses the authorization check.

Data Integrations

• #13144 Changed the log level to warning and added throttling for the log message data_bridge_buffer_overflow when bridge buffers overflow and messages are dropped. Previously, these events were logged at the info level and were not visible with the default log settings.

• #13492 Enhanced the GET /connectors and GET /connectors/:id APIs to include lists of actions and sources that depend on a specific connector. Additionally, the GET /actions, GET /sources, GET /actions/:id, and GET /sources/:id APIs now return the list of rules associated with a specific action or source.

• #13505 Added the ability to filter rules in the HTTP API based on the IDs of data integration actions or sources used.

• #13506 Introduced the peername field to all rule engine events that already include the peerhost field. The peername field is a string formatted as IP:PORT.

• #13516 Added a direct_dispatch argument to the republish action.

  When direct_dispatch is set to true (or rendered as true from template) the message is dispatched directly to subscribers. This feature helps prevent the triggering of additional rules or the recursive activation of the same rule.

• #13573 Introduced client_attrs to the SQL context for client connectivity events and the message publish event.
  Users can now access client attributes within rule SQL statements, such as SELECT client_attrs.attr1 AS attribute1, and utilize ${attribute1} in data integration actions.

• #13640 Added two new SQL functions for rules: coalesce/2 and coalesce_ne/2.

  These functions simplify handling null values in rule SQL expressions. For instance, instead of using:

  SELECT
    CASE
      WHEN is_null(payload.path.to.value) THEN
        0
      ELSE
        payload.path.to.value
    END AS my_value
  

  you can now write a more concise expression: SELECT coalesce(payload.path.to.value, 0) AS my_value.

Operations

• #13202 Introduced the emqx ctl conf cluster_sync fix command to address cluster configuration inconsistencies. This command synchronizes the configuration of all nodes with the configuration of the node that has the highest tnx_id, ensuring consistency across the cluster.

• #13250 Added a new value for cluster.discovery_strategy: singleton. By choosing this option, there will be effectively no clustering, and the node will reject connection attempts to and from other nodes.

• #13370 Added a new version of wildcard_optimized storage layout for durable storage, offering the following improvements:

  • The new layout does not have an inherent latency.

  • MQTT messages are serialized into a more space-efficient format.

• #13524 Added the emqx ctl exclusive CLI interface to manage exclusive topics more effectively. It allows administrators to better manage and troubleshoot exclusive topic subscriptions, ensuring that subscription states are accurately reflected and preventing unexpected failures.

• #13597 Added thin wrapper functions for plugins to store and manage the certificate files used by the plugins themselves. This fix prevents plugin certificates from being inadvertently deleted by the certificate garbage collection (GC) function.

• #13626 Added a new command emqx ctl listeners enable <Identifier> <Bool> to enable/disable a listener.

• #13493 Upgraded the RPC library gen_rpc to version 3.4.0. This update changes the default RPC server socket option from true to active-100, which introduces back-pressure to peer nodes when the RPC server experiences heavy load.

• #13665 Added a new metric emqx_actions_count to the prometheus endpoint. It contains the number of all actions added by all rules, including Republish actions and Console Output actions.

Bug Fixes

Core MQTT Functionality

• #12944 Fixed an issue that caused a crash when clients with non-UTF8 client IDs attempted to connect with strict_mode=false.

• #13006 Improved the validation of retained, delayed, and taken-over session messages to ensure they comply with banned client ID rules implemented through regular expression matching. Previously, certain messages, such as those delayed due to network issues or taken over by another session, could bypass the client ID bans set by regular expressions.

Authentication and Authorization

• #13024 Added a default ACL deny rule to reject subscriptions to the +/# topic pattern. Since EMQX by default rejects subscriptions to # topic, for completeness, it should reject +/# as well.

• #13040 Improved HTTP authentication:

  • Improved error logging for cases where the HTTP Content-Type header is missing or unrecognized, providing more detailed information.
  • Fixed an issue causing double encoding of query parameters in authentication HTTP requests
  • Enhanced error messages when a POST method with a JSON content type is configured for authentication requests but the JSON template fails to render into valid JSON. This can occur, for example, when a template contains a placeholder like ${password} but receives a non-UTF8 password input, leading to better transparency and easier debugging for such scenarios.

• #13196 Added a limit to the built-in authorization database, restricting the number of Access Control List (ACL) rules per client or user to a default of 100.

• #13584 Fixed an issue with creating HTTP authorization that resulted in errors when the HTTP header list was empty.

• #13618 Improved the type specifications for the authorization/sources endpoint to provide clearer and more concise error messages.

• #13624 Fixed an issue in the built-in authorizer where updating rules for a client or user could result in the total number of rules exceeding the max_rules limit.

• #13678 Made the deletion of an authenticator in the chain an idempotent operation, ensuring that deleting a non-existing authenticator always succeeds.

Data Integrations

• #13207 Improved the republish rule engine action to accurately reflect the success and failure of message publishing. Previously, the success metrics were incremented even when the republish action failed to deliver the message to any subscribers. Now, if the action detects that a message fails to reach any subscriber, the failure metrics are correctly incremented.

• #13425 Improved the MQTT connector error log messages to provide clearer and more detailed information.

• #13589 Fixed an issue where creating a rule with a string "null" for ID via the HTTP API was allowed, which could lead to an inconsistent configuration.

Operations

• #13078 Improved validation and error handling in the EMQX Management API to ensure that requests with a JSON body include the Content-Type: application/json header. If the header is missing for APIs that expect JSON input, the server now correctly responds with a 415 Unsupported Media Type status code instead of 400 Bad Request.

• #13225 Enhanced security in authentication and authorization APIs by redacting sensitive data such as passwords. Previously, the APIs could return the original password values in responses. With this update, sensitive information is replaced with ****** to prevent accidental exposure and protect user credentials.

Gateways

• #13607 Fixed an issue where the QoS level for CoAP subscriptions displayed through the API did not match the actual QoS level being...

Enhancements

• #13317 Added a new per-authorization source metric type: ignore. This metric increments when an authorization source attempts to authorize a request but encounters scenarios where the authorizer is not applicable or encounters an error, resulting in an undecidable outcome.

• #13336 Added functionality to initialize authentication data in the built-in database of an empty EMQX node or cluster using a bootstrap file in CSV or JSON format. This feature introduces new configuration entries, bootstrap_file and bootstrap_type.

• #13348 Added a new field payload_encode in the log configuration to determine the format of the payload in the log data.

• #13436 Added the option to add custom request headers to JWKS requests.

• #13507 Introduced a new built-in function getenv in the rule engine and variform expression to facilitate access to environment variables. This function adheres to the following constraints:

  • Prefix EMQXVAR_ is added before reading from OS environment variables. For example, getenv('FOO_BAR') is to read EMQXVAR_FOO_BAR.
  • These values are immutable once loaded from the OS environment.

• #13521 Resolved an issue where LDAP query timeouts could cause the underlying connection to become unusable, potentially causing subsequent queries to return outdated results. The fix ensures the system reconnects automatically in case of a timeout.

• #13528 Applied log throttling for the event of unrecoverable errors in data integrations.

• #13548 EMQX now can optionally invoke the on_config_changed/2 callback function when the plugin configuration is updated via the REST API. This callback function is assumed to be exported by the <PluginName>_app module.
  For example, if the plugin name and version are my_plugin-1.0.0, then the callback function is assumed to be my_plugin_app:on_config_changed/2.

• #13386 Added support for initializing a list of banned clients on an empty EMQX node or cluster with a bootstrap file in CSV format. The corresponding config entry to specify the file path is banned.bootstrap_file. This file is a CSV file with , as its delimiter. The first line of this file must be a header line. All valid headers are listed here:

  • as :: required
  • who :: required
  • by :: optional
  • reason :: optional
  • at :: optional
  • until :: optional

  See the Configuration Manual for details on each field.

  Each row in the rest of this file must contain the same number of columns as the header line, and the column can be omitted then its value is undefined.

Bug Fixes

• #13222 Resolved issues with flags checking and error handling associated with the Will message in the CONNECT packet.
  For detailed specifications, refer to:

  • MQTT-v3.1.1-[MQTT-3.1.2-13], MQTT-v5.0-[MQTT-3.1.2-11]
  • MQTT-v3.1.1-[MQTT-3.1.2-14], MQTT-v5.0-[MQTT-3.1.2-12]
  • MQTT-v3.1.1-[MQTT-3.1.2-15], MQTT-v5.0-[MQTT-3.1.2-13]

• #13307 Updated ekka library to version 0.19.5. This version of ekka utilizes mria 0.8.8, enhancing auto-heal functionality. Previously, the auto-heal worked only when all core nodes were reachable. This update allows to apply auto-heal once the majority of core nodes are alive. For details, refer to the Mria PR.

• #13334 Implemented strict mode checking for the PasswordFlag in the MQTT v3.1.1 CONNECT packet to align with protocol specifications.

  Note: To ensure bug-to-bug compatibility, this check is performed only in strict mode.

• #13344 Resolved an issue where the POST /clients/:clientid/subscribe/bulk API would not function correctly if the node receiving the API request did not maintain the connection to the specified clientid.

• #13358 Fixed an issue when the reason in the authn_complete_event event was incorrectly displayed.

• #13375 The value infinity has been added as default value to the listener configuration fields max_conn_rate, messages_rate, and bytes_rate.

• #13382 Updated the emqtt library to version 0.4.14, which resolves an issue preventing emqtt_pools from reusing pools that are in an inconsistent state.

• #13389 Fixed an issue where the Derived Key Length for pbkdf2 could be set to a negative integer.

• #13389 Fixed an issue where topics in the authorization rules might be parsed incorrectly.

• #13393 Fixed an issue where plugin applications failed to restart after a node joined a cluster, resulting in hooks not being properly installed and causing inconsistent states.

• #13398 Fixed an issue where ACL rules were incorrectly cleared when reloading the built-in database for authorization using the command line.

• #13403 Addressed a security issue where environment variable configuration overrides were inadvertently logging passwords. This fix ensures that passwords present in environment variables are not logged.

• #13408 Resolved a function_clause crash triggered by authentication attempts with invalid salt or password types. This fix enhances error handling to better manage authentication failures involving incorrect salt or password types.

• #13419 Resolved an issue where crash log messages from the /configs API were displaying garbled hints. This fix ensures that log messages related to API calls are clear and understandable.

• #13422 Fixed an issue where the option force_shutdown.max_heap_size could not be set to 0 to disable this tuning.

• #13442 Fixed an issue where the health check interval configuration for actions/sources was not being respected. Previously, EMQX ignored the specified health check interval for actions and used the connector's interval instead. The fix ensures that EMQX now correctly uses the health check interval configured for actions/sources, allowing for independent and accurate health monitoring frequencies.

• #13503 Fixed an issue where connectors did not adhere to the configured health check interval upon initial startup, requiring an update or restart to apply the correct interval.

• #13515 Fixed an issue where the same client could not subscribe to the same exclusive topic when the node was down for some reason.

• #13527 Fixed an issue in the Rule Engine where executing a SQL test for the Message Publish event would consistently return no results when a $bridges/... source was included in the FROM clause.

• #13541 Fixed an issue where disabling CRL checks for a listener required a listener restart to take effect.

• #13552 Added a startup timeout limit for EMQX plugins with a default timeout of 10 seconds. Before this update, problematic plugins could cause runtime errors during startup, leading to potential issues where the main startup process might hang when EMQX is stopped and restarted.

Breaking Changes

• #12947 For JWT authentication, a new boolean option disconnect_after_expire has been added with default value set to true. When enabled, the client will be disconnected after the JWT token expires.

  Previously, the clients with actual JWTs could connect to the broker and stay connected even after the JWT token expired. Now, the client will be disconnected after the JWT token expires. To preserve the previous behavior, set disconnect_after_expire to false.

Enhancements

• #12983 Add new rule engine event $events/client_check_authn_complete for authentication completion event.

• #13180 Improved client message handling performance when EMQX is running on Erlang/OTP 26 and increased message throughput by 10% in fan-in mode.

• #13191 Upgraded EMQX Docker images to run on Erlang/OTP 26.

  EMQX had been running on Erlang/OTP 26 since v5.5 except for docker images which were on Erlang/OTP 25. Now all releases are on Erlang/OTP 26.

• #13242 Significantly increased the startup speed of EMQX dashboard listener.

Bug Fixes

• #13156 Resolved an issue where the Dashboard Monitoring pages would crash following the update to EMQX v5.7.0.

• #13164 Fixed HTTP authorization request body encoding.

  Before this fix, the HTTP authorization request body encoding format was taken from the accept header. The fix is to respect the content-type header instead. Also added access templating variable for v4 compatibility. The access code of SUBSCRIBE action is 1 and PUBLISH action is 2.

• #13238 Improved the logged error messages when an HTTP authorization request with an unsupported content-type header is returned.

• #13258 Fix an issue where the MQTT-SN gateway would not restart correctly due to incorrect startup order of gateway dependencies.

• #13273 Fixed and improved handling of URIs in several configurations. The fix includes the following improvement details:

  • Authentication and authorization configurations: Corrected a previous error where valid pathless URIs such as https://example.com?q=x were mistakenly rejected. These URIs are now properly recognized as valid.
  • Connector configurations: Enhanced checks to ensure that URIs with potentially problematic components, such as user info or fragment parts, are no longer erroneously accepted.

• #13276 Fixed an issue in the durable message storage mechanism where parts of the internal storage state were not correctly persisted during the setup of new storage generations. The concept of "generation" is used internally and is crucial for managing message expiration and cleanup. This could have manifested as messages being lost after a restart of EMQX.

• #13291 Fixed an issue where durable storage sites that were down being reported as up.

• #13290 Fixed an issue where the command $ bin/emqx ctl rules show rule_0hyd would produce no output when used to display rules with a data integration action attached.

• #13293 Improved the restoration process from data backups by automating the re-indexing of imported retained messages. Previously, re-indexing required manual intervention using the emqx ctl retainer reindex start CLI command after importing a data backup file.

  This fix also extended the functionality to allow exporting retained messages to a backup file when the retainer.backend.storage_type is configured as ram. Previously, only setups with disc as the storage type supported exporting retained messages.

• #13140 Fixed an issue that caused text traces for the republish action to crash and not display correctly.

• #13148 Fixed an issue where a 500 HTTP status code could be returned by /connectors/:connector-id/start when there is a timeout waiting for the resource to be connected.

• #13181 EMQX now forcefully shut down the connector process when attempting to stop a connector, if such operation times out. This fix also improved the clarity of error messages when disabling an action or source fails due to an unresponsive underlying connector.

• #13216 Respect clientid_prefix config for MQTT bridges. Since EMQX v5.4.1, the MQTT client IDs are restricted to a maximum of 23 bytes. Previously, the system factored the clientid_prefix into the hash of the original, longer client ID, affecting the final shortened ID. The fix includes the following change details:

  • Without Prefix: The behavior remains unchanged. EMQX hashes the long client IDs (exceeding 23 bytes) to fit within the 23-byte limit.
  • With Prefix:
    • Prefix ≤ 19 bytes: The prefix is retained, and the remaining portion of the client ID is hashed into a 4-byte space, ensuring the total length does not exceed 23 bytes.
    • Prefix ≥ 20 bytes: EMQX will not attempt to shorten the client ID, fully preserving the configured prefix regardless of length.

Enhancements

Security

• #12947 For JWT authentication, support new disconnect_after_expire option. When enabled, the client will be disconnected after the JWT token expires.

Note: This is a breaking change. This option is enabled by default, so the default behavior is changed. Previously, the clients with actual JWTs could connect to the broker and stay connected even after the JWT token expired. Now, the client will be disconnected after the JWT token expires. To preserve the previous behavior, set disconnect_after_expire to false.

Data Processing and Integration

• #12671 An unescape function has been added to the rule engine SQL language to handle the expansion of escape sequences in strings. This addition has been done because string literals in the SQL language don't support any escape codes (e.g., \n and \t). This enhancement allows for more flexible string manipulation within SQL expressions.

Extensibility

• #12872 Implemented Client Attributes feature. It allows setting additional properties for each client using key-value pairs. Property values can be generated from MQTT client connection information (such as username, client ID, TLS certificate) or set from data accompanying successful authentication returns. Properties can be used in EMQX for authentication, authorization, data integration, and MQTT extension functions. Compared to using static properties like client ID directly, client properties offer greater flexibility in various business scenarios, simplifying the development process and enhancing adaptability and efficiency in development work.
  Initialization of client_attrs
  The client_attrs fields can be initially populated from one of the following clientinfo fields:

  • cn: The common name from the TLS client's certificate.
  • dn: The distinguished name from the TLS client's certificate, that is, the certificate "Subject".
  • clientid: The MQTT client ID provided by the client.
  • username: The username provided by the client.
  • user_property: Extract a property value from 'User-Property' of the MQTT CONNECT packet.

  Extension through Authentication Responses
  Additional attributes may be merged into client_attrs from authentication responses. Supported
  authentication backends include:

  • HTTP: Attributes can be included in the JSON object of the HTTP response body through a
    client_attrs field.
  • JWT: Attributes can be included via a client_attrs claim within the JWT.

  Usage in Authentication and Authorization
  If client_attrs is initialized before authentication, it can be used in external authentication
  requests. For instance, ${client_attrs.property1} can be used within request templates
  directed at an HTTP server for authenticity validation.

  • The client_attrs can be utilized in authorization configurations or request templates, enhancing
    flexibility and control. Examples include: In acl.conf, use {allow, all, all, ["${client_attrs.namespace}/#"]} to apply permissions based on the namespace attribute.
  • In other authorization backends, ${client_attrs.namespace} can be used within request templates to dynamically include client attributes.

• #12910 Added plugin configuration management and schema validation. For EMQX enterprise edition, one can also annotate the schema with metadata to facilitate UI rendering in the Dashboard. See more details in the plugin template and plugin documentation.

Operations and Management

• #12923 Provided more specific error when importing wrong format into builtin authenticate database.

• #12940 Added ignore_readonly argument to PUT /configs API.
  Before this change, EMQX would return 400 (BAD_REQUEST) if the raw config included read-only root keys (cluster, rpc, and node).
  After this enhancement it can be called as PUT /configs?ignore_readonly=true, EMQX will in this case ignore readonly root config keys, and apply the rest. For observability purposes, an info level message is logged if any readonly keys are dropped.
  Also fixed an exception when config has bad HOCON syntax (returns 500). Now bad syntax will cause the API to return 400 (BAD_REQUEST).

• #12957 Started building packages for macOS 14 (Apple Silicon) and Ubuntu 24.04 Noble Numbat (LTS).

Bug Fixes

Security

• #12887 Fixed MQTT enhanced auth with sasl scram.

• #12962 TLS clients can now verify server hostname against wildcard certificate. For example, if a certificate is issued for host *.example.com, TLS clients is able to verify server hostnames like srv1.example.com.

MQTT

• #12996 Fixed process leak in emqx_retainer application. Previously, client disconnection while receiving retained messages could cause a process leak.

Data Processing and Integration

• #12653 The rule engine function bin2hexstr now supports bitstring inputs with a bit size that is not divisible by 8. Such bitstrings can be returned by the rule engine function subbits.

• #12657 The rule engine SQL-based language previously did not allow putting any expressions as array elements in array literals (only constants and variable references were allowed). This has now been fixed so that one can use any expressions as array elements.
  The following is now permitted, for example:

  select
  [21 + 21, abs(-abs(-2)), [1 + 1], 4] as my_array
  from "t/#"

• #12932 Previously, if a HTTP action request received a 503 (Service Unavailable) status, it was marked as a failure and the request was not retried. This has now been fixed so that the request is retried a configurable number of times.

• #12948 Fixed an issue where sensitive HTTP header values like Authorization would be substituted by ****** after updating a connector.

• #13118 Fix a performance issue in the rule engine template rendering.

Observability

• #12765 Make sure stats subscribers.count subscribers.max contains shared-subscribers. It only contains non-shared subscribers previously.

Operations and Management

• #12812 Made resource health checks non-blocking operations. This means that operations such as updating or removing a resource won't be blocked by a lengthy running health check.

• #12830 Made channel (action/source) health checks non-blocking operations. This means that operations such as updating or removing an action/source data integration won't be blocked by a lengthy running health check.

• #12993 Fixed listener config update API when handling an unknown zone.
  Before this fix, when a listener config is updated with an unknown zone, for example {"zone": "unknown"}, the change would be accepted, causing all clients to crash whens connected.
  After this fix, updating the listener with an unknown zone name will get a "Bad request" response.

• #13012 The MQTT listerners config option access_rules has been improved in the following ways:

  • The listener no longer crash with an incomprehensible error message if a non-valid access rule is configured. Instead a configuration error is generated.
  • One can now add several rules in a single string by separating them by comma (for example, "allow 10.0.1.0/24, deny all").

• #13041 Improved HTTP authentication error log message. If HTTP content-type header is missing for POST method, it now emits a meaningful error message instead of a less readable exception with stack trace.

• #13077 This fix makes EMQX only read action configurations from the global configuration when the connector starts or restarts, and instead stores the latest configurations for the actions in the connector. Previously, updates to action configurations would sometimes not take effect without disabling and enabling the action. This means that an action could sometimes run with the old (previous) configuration even though it would look like the action configuration has been updated successfully.

• #13090 Attempting to start an action or source whose connector is disabled will no longer attempt to start the connector itself.

Gateways

• #12909 Fixed UDP listener process handling on errors or closure, The fix ensures the UDP listener is cleanly stopped and restarted as needed if these error conditions occur.

• #13001 Fixed an issue where the syskeeper forwarder would never reconnect when the connection was lost.

• #13010 Fixed the issue where the JT/T 808 gateway could not correctly reply to the REGISTER_ACK message when requesting authentication from the registration service failed.

Breaking Changes

• #12947 For JWT authentication,...

Bug Fixes

• #12759 EMQX now automatically removes invalid backup files that fail during upload due to schema validation errors. This fix ensures that only valid configuration files are displayed and stored, enhancing system reliability.

• #12766 Renamed message_queue_too_long error reason to mailbox_overflow

  mailbox_overflow. The latter is consistent with the corresponding config parameter: force_shutdown.max_mailbox_size.

• #12773 Upgraded HTTP client libraries.

  The HTTP client library (gun-1.3) incorrectly appended a :portnumber suffix to the Host header for
  standard ports (http on port 80, https on port 443). This could cause compatibility issues with servers or gateways performing strict Host header checks (e.g., AWS Lambda, Alibaba Cloud HTTP gateways), leading to errors such as InvalidCustomDomain.NotFound or "The specified CustomDomain does not exist."

• #12802 Improved how EMQX handles node removal from clusters via the emqx ctl cluster leave command. Previously, nodes could unintentionally rejoin the same cluster (unless it was stopped) if the configured cluster discovery_strategy was not manual. With the latest update, executing the cluster leave command now automatically disables cluster discovery for the node, preventing it from rejoining. To re-enable cluster discovery, use the emqx ctl discovery enable command or simply restart the node.

• #12814 Improved error handling for the /clients/{clientid}/mqueue_messages and /clients/{clientid}/inflight_messages APIs in EMQX. These updates address:

  • Internal Timeout: If EMQX fails to retrieve the list of Inflight or Mqueue messages within the default 5-second timeout, likely under heavy system load, the API will return 500 error with the response {"code":"INTERNAL_ERROR","message":"timeout"}, and log additional details for troubleshooting.
  • Client Shutdown: Should the client connection be terminated during an API call, the API now returns a 404 error, with the response {"code": "CLIENT_SHUTDOWN", "message": "Client connection has been shutdown"}. This ensures clearer feedback when client connections are interrupted.

• #12824 Updated the statistics metrics subscribers.count and subscribers.max to include shared subscribers. Previously, these metrics accounted only for non-shared subscribers.

• #12826 Fixed issues related to the import functionality of source data integrations and retained messages in EMQX. Before this update:

  • The data integration sources specified in backup files were not being imported. This included configurations under the sources.mqtt category with specific connectors and parameters such as QoS and topics.
  • Importing the mnesia table for retained messages was not supported.

• #12843 Fixed cluster_rpc_commit transaction ID cleanup procedure on replicator nodes after executing the emqx ctl cluster leave command. Previously, failing to properly clear these transaction IDs impeded configuration updates on the core node.

• #12885 Fixed an issue in EMQX where users were unable to view "Retained Messages" under the "Monitoring" menu in the Dashboard.

  The "Retained messages" backend API uses the qlc library. This problem was due to a permission issue where the qlc library's file_sorter function tried to use a non-writable directory, /opt/emqx, to store temporary files, resulting from recent changes in directory ownership permissions in Docker deployments.

  This update modifies the ownership settings of the /opt/emqx directory to emqx:emqx, ensuring that all necessary operations, including retained messages retrieval, can proceed without access errors.

Bug Fixes

• #12759 EMQX now automatically removes invalid backup files that fail during upload due to schema validation errors. This fix ensures that only valid configuration files are displayed and stored, enhancing system reliability.

• #12766 Renamed message_queue_too_long error reason to mailbox_overflow

  mailbox_overflow. The latter is consistent with the corresponding config parameter: force_shutdown.max_mailbox_size.

• #12773 Upgraded HTTP client libraries.

  The HTTP client library (gun-1.3) incorrectly appended a :portnumber suffix to the Host header for
  standard ports (http on port 80, https on port 443). This could cause compatibility issues with servers or gateways performing strict Host header checks (e.g., AWS Lambda, Alibaba Cloud HTTP gateways), leading to errors such as InvalidCustomDomain.NotFound or "The specified CustomDomain does not exist."

• #12802 Improved how EMQX handles node removal from clusters via the emqx ctl cluster leave command. Previously, nodes could unintentionally rejoin the same cluster (unless it was stopped) if the configured cluster discovery_strategy was not manual. With the latest update, executing the cluster leave command now automatically disables cluster discovery for the node, preventing it from rejoining. To re-enable cluster discovery, use the emqx ctl discovery enable command or simply restart the node.

• #12814 Improved error handling for the /clients/{clientid}/mqueue_messages and /clients/{clientid}/inflight_messages APIs in EMQX. These updates address:

  • Internal Timeout: If EMQX fails to retrieve the list of Inflight or Mqueue messages within the default 5-second timeout, likely under heavy system load, the API will return 500 error with the response {"code":"INTERNAL_ERROR","message":"timeout"}, and log additional details for troubleshooting.
  • Client Shutdown: Should the client connection be terminated during an API call, the API now returns a 404 error, with the response {"code": "CLIENT_SHUTDOWN", "message": "Client connection has been shutdown"}. This ensures clearer feedback when client connections are interrupted.

• #12824 Updated the statistics metrics subscribers.count and subscribers.max to include shared subscribers. Previously, these metrics accounted only for non-shared subscribers.

• #12826 Fixed issues related to the import functionality of source data integrations and retained messages in EMQX. Before this update:

  • The data integration sources specified in backup files were not being imported. This included configurations under the sources.mqtt category with specific connectors and parameters such as QoS and topics.
  • Importing the mnesia table for retained messages was not supported.

• #12843 Fixed cluster_rpc_commit transaction ID cleanup procedure on replicator nodes after executing the emqx ctl cluster leave command. Previously, failing to properly clear these transaction IDs impeded configuration updates on the core node.

• #12882 Fixed an issue with the RocketMQ action in EMQX data integration, ensuring that messages are correctly routed to their configured topics. Previously, when multiple actions shared a single RocketMQ connector, all messages were mistakenly sent to the topic configured for the first action. This fix involves starting a distinct set of RocketMQ workers for each topic, preventing cross-topic message delivery errors.

• #12885 Fixed an issue in EMQX where users were unable to view "Retained Messages" under the "Monitoring" menu in the Dashboard.

  The "Retained messages" backend API uses the qlc library. This problem was due to a permission issue where the qlc library's file_sorter function tried to use a non-writable directory, /opt/emqx, to store temporary files, resulting from recent changes in directory ownership permissions in Docker deployments.

  This update modifies the ownership settings of the /opt/emqx directory to emqx:emqx, ensuring that all necessary operations, including retained messages retrieval, can proceed without access errors.

Enhancements

• #12251 Optimized the performance of the RocksDB-based persistent sessions, achieving a reduction in RAM usage and database request frequency. Key improvements include:

  • Introduced dirty session state to avoid frequent mria transactions.
  • Introduced an intermediate buffer for the persistent messages.
  • Used separate tracks of PacketIds for QoS1 and QoS2 messages.
  • Limited the number of continuous ranges of inflight messages to 1 per stream.

• #12326 Enhanced session tracking with registration history. EMQX now has the capability to monitor the history of session registrations, including those that have expired. By configuring broker.session_history_retain, EMQX retains records of expired sessions for a specified duration.

  • Session count API: Use the API GET /api/v5/sessions_count?since=1705682238 to obtain a count of sessions across the cluster that remained active since the given UNIX epoch timestamp (with seconds precision). This enhancement aids in analyzing session activity over time.

  • Metrics expansion with cluster sessions gauge: A new gauge metric, cluster_sessions, is added to better track the number of sessions within the cluster. This metric is also integrated into Prometheus for easy monitoring:

    # TYPE emqx_cluster_sessions_count gauge
    emqx_cluster_sessions_count 1234
    

    NOTE: Please consider this metric as an approximate estimation. Due to the asynchronous nature of data collection and calculation, exact precision may vary.

• #12338 Introduced a time-based garbage collection mechanism to the RocksDB-based persistent session backend. This feature ensures more efficient management of stored messages, optimizing storage utilization and system performance by automatically purging outdated messages.

• #12398 Exposed the swagger_support option in the Dashboard configuration, allowing for the enabling or disabling of the Swagger API documentation.

• #12467 Started supporting cluster discovery using AAAA DNS record type.

• #12483 Renamed emqx ctl conf cluster_sync tnxid ID to emqx ctl conf cluster_sync inspect ID.

  For backward compatibility, tnxid is kept, but considered deprecated and will be removed in 5.7.

• #12499 Enhanced client banning capabilities with extended rules, including:

  • Matching clientid against a specified regular expression.
  • Matching client's username against a specified regular expression.
  • Matching client's peer address against a CIDR range.

  Important Notice: Implementing a large number of broad matching rules (not specific to an individual clientid, username, or host) may affect system performance. It's advised to use these extended ban rules judiciously to maintain optimal system efficiency.

• #12509 Implemented API to re-order all authenticators / authorization sources.

• #12517 Configuration files have been upgraded to accommodate multi-line string values, preserving indentation for enhanced readability and maintainability. This improvement utilizes """~ and ~""" markers to quote indented lines, offering a structured and clear way to define complex configurations. For example:

  rule_xlu4 {
    sql = """~
      SELECT
        *
      FROM
        "t/#"
    ~"""
  }
  

  See HOCON 0.42.0 release notes for details.

• #12520 Implemented log throttling. The feature reduces the volume of logged events that could potentially flood the system by dropping all but the first occurance of an event within a configured time window.
  Log throttling is applied to the following log events that are critical yet prone to repetition:

  • authentication_failure
  • authorization_permission_denied
  • cannot_publish_to_topic_due_to_not_authorized
  • cannot_publish_to_topic_due_to_quota_exceeded
  • connection_rejected_due_to_license_limit_reached
  • dropped_msg_due_to_mqueue_is_full

• #12561 Implemented HTTP APIs to get the list of client's in-flight and message queue (mqueue) messages. These APIs facilitate detailed insights and effective control over message queues and in-flight messaging, ensuring efficient message handling and monitoring.

  To get the first chunk of data:

  • GET /clients/{clientid}/mqueue_messages?limit=100
  • GET /clients/{clientid}/inflight_messages?limit=100

  Alternatively, for the first chunks without specifying a start position:

  • GET /clients/{clientid}/mqueue_messages?limit=100&position=none
  • GET /clients/{clientid}/inflight_messages?limit=100&position=none

  To get the next chunk of data:

  • GET /clients/{clientid}/mqueue_messages?limit=100&position={position}
  • GET /clients/{clientid}/inflight_messages?limit=100&position={position}

  Where {position} is a value (opaque string token) of meta.position field from the previous response.

  Ordering and Prioritization:

  • Mqueue Messages: These are prioritized and sequenced based on their queue order (FIFO), from higher to lower priority. By default, mqueue messages carry a uniform priority level of 0.
  • In-Flight Messages: Sequenced by the timestamp of their insertion into the in-flight storage, from oldest to newest.

• #12590 Removed mfa meta data from log messages to improve clarity.

• #12641 Improved text log formatter fields order. The new fields order is as follows:

  tag > clientid > msg > peername > username > topic > [other fields]

• #12670 Added field shared_subscriptions to endpoint /monitor_current and /monitor_current/nodes/:node.

• #12679 Upgraded docker image base from Debian 11 to Debian 12.

• #12700 Started supporting "b" and "B" unit in bytesize hocon fields. For example, all three fields below will have the value of 1024 bytes:

  bytesize_field = "1024b"
  bytesize_field2 = "1024B"
  bytesize_field2 = 1024
  

• #12719 The /clients API has been upgraded to accommodate queries for multiple clientids and usernames simultaneously, offering a more flexible and powerful tool for monitoring client connections. Additionally, this update introduces the capability to customize which client information fields are included in the API response, optimizing for specific monitoring needs.

  Examples of Multi-Client/Username Queries:

  • To query multiple clients by ID: /clients?clientid=client1&clientid=client2
  • To query multiple users: /clients?username=user11&username=user2
  • To combine multiple client IDs and usernames in one query: /clients?clientid=client1&clientid=client2&username=user1&username=user2

  Examples of Selecting Fields for the Response:

  • To include all fields in the response: /clients?fields=all (Note: Omitting the fields parameter defaults to returning all fields.)
  • To specify only certain fields: /clients?fields=clientid,username

• #12381 Added new SQL functions: map_keys(), map_values(), map_to_entries(), join_to_string(), join_to_string(), join_to_sql_values_string(), is_null_var(), is_not_null_var().

  For more information on the functions and their usage, refer to Built-in SQL Functions the documentation.

• #12336 Performance enhancement. Created a dedicated async task handler pool to handle client session cleanup tasks.

• #12725 Implemented REST API to list the available source types.

• #12746 Added username log field. If MQTT client is connected with a non-empty username the logs and traces will include username field.

• #12785 Added timestamp_format configuration option to log handlers. This new option allows for the following settings:

  • auto: Automatically determines the timestamp format based on the log formatter being used.
    Utilizes rfc3339 format for text formatters, and epoch format for JSON formatters.

  • epoch: Represents timestamps in microseconds precision Unix epoch format.

  • rfc3339: Uses RFC3339 compliant format for date-time strings. For example, 2024-03-26T11:52:19.777087+00:00.

Bug Fixes

• #11868 Fixed a bug where will messages were not published after session takeover.

• #12347 Implemented an update to ensure that messages processed by the Rule SQL for the MQTT egress data bridge are always rendered as valid, even in scenarios where the data is incomplete or lacks certain placeholders defined in the bridge configuration. This adjustment prevents messages from being incorrectly deemed invalid and subsequently discarded by the MQTT egress data bridge, as was the case previously.

  When variables in payload and topic templates are undefined, they are now rendered as empty strings instead of the literal undefined string.

• #12472 Fixed an issue whe...

Enhancements

• #12326 Enhanced session tracking with registration history. EMQX now has the capability to monitor the history of session registrations, including those that have expired. By configuring broker.session_history_retain, EMQX retains records of expired sessions for a specified duration.

  • Session count API: Use the API GET /api/v5/sessions_count?since=1705682238 to obtain a count of sessions across the cluster that remained active since the given UNIX epoch timestamp (with seconds precision). This enhancement aids in analyzing session activity over time.

  • Metrics expansion with cluster sessions gauge: A new gauge metric, cluster_sessions, is added to better track the number of sessions within the cluster. This metric is also integrated into Prometheus for easy monitoring:

    # TYPE emqx_cluster_sessions_count gauge
    emqx_cluster_sessions_count 1234
    

    NOTE: Please consider this metric as an approximate estimation. Due to the asynchronous nature of data collection and calculation, exact precision may vary.

• #12398 Exposed the swagger_support option in the Dashboard configuration, allowing for the enabling or disabling of the Swagger API documentation.

• #12467 Started supporting cluster discovery using AAAA DNS record type.

• #12483 Renamed emqx ctl conf cluster_sync tnxid ID to emqx ctl conf cluster_sync inspect ID.

  For backward compatibility, tnxid is kept, but considered deprecated and will be removed in 5.7.

• #12495 Introduced new AWS S3 connector and action.

• #12499 Enhanced client banning capabilities with extended rules, including:

  • Matching clientid against a specified regular expression.
  • Matching client's username against a specified regular expression.
  • Matching client's peer address against a CIDR range.

  Important Notice: Implementing a large number of broad matching rules (not specific to an individual clientid, username, or host) may affect system performance. It's advised to use these extended ban rules judiciously to maintain optimal system efficiency.

• #12509 Implemented API to re-order all authenticators / authorization sources.

• #12517 Configuration files have been upgraded to accommodate multi-line string values, preserving indentation for enhanced readability and maintainability. This improvement utilizes """~ and ~""" markers to quote indented lines, offering a structured and clear way to define complex configurations. For example:

  rule_xlu4 {
    sql = """~
      SELECT
        *
      FROM
        "t/#"
    ~"""
  }
  

  See HOCON 0.42.0 release notes for details.

• #12520 Implemented log throttling. The feature reduces the volume of logged events that could potentially flood the system by dropping all but the first occurance of an event within a configured time window.
  Log throttling is applied to the following log events that are critical yet prone to repetition:

  • authentication_failure
  • authorization_permission_denied
  • cannot_publish_to_topic_due_to_not_authorized
  • cannot_publish_to_topic_due_to_quota_exceeded
  • connection_rejected_due_to_license_limit_reached
  • dropped_msg_due_to_mqueue_is_full

• #12561 Implemented HTTP APIs to get the list of client's in-flight and message queue (mqueue) messages. These APIs facilitate detailed insights and effective control over message queues and in-flight messaging, ensuring efficient message handling and monitoring.

  To get the first chunk of data:

  • GET /clients/{clientid}/mqueue_messages?limit=100
  • GET /clients/{clientid}/inflight_messages?limit=100

  Alternatively, for the first chunks without specifying a start position:

  • GET /clients/{clientid}/mqueue_messages?limit=100&position=none
  • GET /clients/{clientid}/inflight_messages?limit=100&position=none

  To get the next chunk of data:

  • GET /clients/{clientid}/mqueue_messages?limit=100&position={position}
  • GET /clients/{clientid}/inflight_messages?limit=100&position={position}

  Where {position} is a value (opaque string token) of meta.position field from the previous response.

  Ordering and Prioritization:

  • Mqueue Messages: These are prioritized and sequenced based on their queue order (FIFO), from higher to lower priority. By default, mqueue messages carry a uniform priority level of 0.
  • In-Flight Messages: Sequenced by the timestamp of their insertion into the in-flight storage, from oldest to newest.

• #12590 Removed mfa meta data from log messages to improve clarity.

• #12641 Improved text log formatter fields order. The new fields order is as follows:

  tag > clientid > msg > peername > username > topic > [other fields]

• #12670 Added field shared_subscriptions to endpoint /monitor_current and /monitor_current/nodes/:node.

• #12679 Upgraded docker image base from Debian 11 to Debian 12.

• #12700 Started supporting "b" and "B" unit in bytesize hocon fields.

  For example, all three fields below will have the value of 1024 bytes:

  bytesize_field = "1024b"
  bytesize_field2 = "1024B"
  bytesize_field2 = 1024
  

• #12719 The /clients API has been upgraded to accommodate queries for multiple clientids and usernames simultaneously, offering a more flexible and powerful tool for monitoring client connections. Additionally, this update introduces the capability to customize which client information fields are included in the API response, optimizing for specific monitoring needs.

  Examples of Multi-Client/Username Queries:

  • To query multiple clients by ID: /clients?clientid=client1&clientid=client2
  • To query multiple users: /clients?username=user11&username=user2
  • To combine multiple client IDs and usernames in one query: /clients?clientid=client1&clientid=client2&username=user1&username=user2

  Examples of Selecting Fields for the Response:

  • To include all fields in the response: /clients?fields=all (Note: Omitting the fields parameter defaults to returning all fields.)
  • To specify only certain fields: /clients?fields=clientid,username

• #12330 The Cassandra bridge has been split into connector and action components. They are backwards compatible with the bridge HTTP API. Configuration will be upgraded automatically.

• #12353 The OpenTSDB bridge has been split into connector and action components. They are backwards compatible with the bridge HTTP API. Configuration will be upgraded automatically.

• #12376 The Kinesis bridge has been split into connector and action components. They are backwards compatible with the bridge HTTP API. Configuration will be upgraded automatically.

• #12386 The GreptimeDB bridge has been split into connector and action components. They are backwards compatible with the bridge HTTP API. Configuration will be upgraded automatically.

• #12423 The RabbitMQ bridge has been split into connector, action and source components. They are backwards compatible with the bridge HTTP API. Configuration will be upgraded automatically.

• #12425 The ClickHouse bridge has been split into connector and action components. They are backwards compatible with the bridge HTTP API. Configuration will be upgraded automatically.

• #12439 The Oracle bridge has been split into connector and action components. They are backwards compatible with the bridge HTTP API. Configuration will be upgraded automatically.

• #12449 The TDEngine bridge has been split into connector and action components. They are backwards compatible with the bridge HTTP API. Configuration will be upgraded automatically.

• #12488 The RocketMQ bridge has been split into connector and action components. They are backwards compatible with the bridge HTTP API. Configuration will be upgraded automatically.

• #12512 The HStreamDB bridge has been split into connector and action components. They are backwards compatible with the bridge HTTP API. Configuration will be upgraded automatically, however, it is recommended to do the upgrade manually as new fields have been added to the configuration.

• #12543 The DynamoDB bridge has been split into connector and action components. They are backwards compatible with the bridge HTTP API. Configuration will be upgraded automatically.

• #12595 The Kafka Consumer bridge has been split into connector and source components. They are backwards compatible with the bridge HTTP API. Configuration will be upgraded automatically.

• #12619 The Microsoft SQL Server bridge has been split into connector...