Hi users!
We have released v1.16.6 on 2024-08-20. ChangeLog is here.
This release is a maintenance release of v1.16 series. In this release, some bugs were fixed with backporting from v1.17.x.
yaml_parser: $log_level notation in YAML was supportedIn this release, we added a support for $log_level in YAML config file syntax.
In the previous versions, if you used log_level in YAML configuration, it causes a following warning:
[warn]: #0 'log_level' is deprecated parameter name. use '@log_level' instead.
But @log_level is invalid in YAML format. In YAML format, it should be $log_level as same as
similar parameter such as $tag and $type, but not supported before.
Since Fluentd v1.16.6, $log_level has been supported! (In v1.17.x, it was supported since v1.17.1)
--plugin (-p) option not to overwrite default valueIn this release, we have fixed a bug that fluentd command: --plugin (-p) option could not overwrite default value.
This bug was unexpectedly introduced since v1.16.0.
Now it was reverted to the original behavior.
In this release, we fixed a LoadError with console gem v1.25 or later.
Since console gem v1.25 or later, it's internal change causes a LoadError in Fluentd. This error breaks existing plugin such as fluent-plugin-prometheus which provides monitoring REST API.
With supporting console gem v1.25 or later, this issue has solved!
Enjoy logging!
Hi users!
We have released v1.17.1 on 2024-08-19. ChangeLog is here.
This release is a new release of v1.17 series. In this release, we added some new options for some plugins and fixed bugs of Parser.
yaml_parser: $log_level notation in YAML was supportedIn this release, we added a support for $log_level in YAML config file syntax.
In the previous versions, if you used log_level in YAML configuration, it causes a following warning:
[warn]: #0 'log_level' is deprecated parameter name. use '@log_level' instead.
But @log_level is invalid in YAML format. In YAML format, it should be $log_level as same as
similar parameter such as $tag and $type, but not supported before.
Since Fluentd v1.17.1, $log_level has been supported now!
out_http: compress option was supportedIn this release, we added a new option compress for the out_http plugin.
You can compress HTTP request body by specifying compress gzip.
To keep compatibility, the default configuration is set as compress text (no compression).
This feature was contributed by @rockliffelewis. Thanks!
in_sample: the default behavior was changedIn this release, we changed the default behavior of in_sample plugin and added a new option reuse_record for it.
The default behavior has changed to copy sample data to avoid the impact of destructive changes by subsequent plugins.
In the previous versions, when in_sample plugin is used with the subsequent filter parser plugin which uses remove_key_name_field, it raises the following error repeatedly.
#0 dump an error event: error_class=ArgumentError error="message does not exist"
This error occurs when key_name and remove_key_name_field removes key from record with destructive change in filter parser.
It affects generated sample data after twice or later.
(This error is not limited to filter parser plugin. The similar destructive change will be done with the subsequent plugins, this error is reproducible)
The new default behavior (reuse_record false) avoids this problem, but it increases the load when generating large amounts of sample data.
You can use this new parameter to have the same performance as before with reuse_record.
In this release, we fixed a LoadError with console gem v1.25 or later.
Since console gem v1.25 or later, it's internal change causes a LoadError in Fluentd. This error breaks existing plugin such as fluent-plugin-prometheus which provides monitoring REST API.
With supporting console gem v1.25 or later, this issue has solved!
Enjoy logging!
Hi users!
We have released fluent-package v5.1.0. fluent-package is a stable distribution package of Fluentd. (successor of td-agent v4)
This is a maintenance release of v5.x series. Bundled ruby version was upgraded to 3.2.5 and fluentd v1.17.0 was shipped!
In this release, focused on updating bundled ones.
In LTS version, conservative updating policy is enabled, so updating gem is very limited to bug fix or security fix. In contrast to LTS version, v5.1.0 bundles more recent components because of standard version.
As we already announced in Scheduled support lifecycle announcement about Fluent Package blog article, in normal release channel, we will ship the latest version (v1.17.0) of Fluentd.
So if you want to try using latest version of Fluentd, v5.1.0 is one for you. If you use fluent-package in enterprise services, keep using fluent-package 5.0.x (LTS).
Please see the download page.
We plan to ship the next LTS version of fluent-package v5.0.5 on Oct, 2024. The content of updates are still in T.B.D.
As it was already announced Drop schedule announcement about EOL of Treasure Agent (td-agent) 4, td-agent v4 was reached EOL in Dec, 2023.
After reached EOL, td-agent v4.5.3 on Windows was released because there was a crash bug during startup on Windows. It was backported fix from fluent-package v5 as it is critical in some case. Even though this was a exceptional maintenance release, but there is no change to the fact that we already stopped maintaining td-agent v4.
We strongly recommend migrating from td-agent v4 to fluent-package v5 (LTS). See Upgrade to fluent-package v5
Hi users!
We have released fluent-package v5.0.4. fluent-package is a stable distribution package of Fluentd. (successor of td-agent v4)
This is a maintenance release of v5.0.x LTS series. As bundled Ruby was updated to 3.2.4 and a foolproof mechanism was implemented to prevent launching duplicated Fluentd instances, we recommend upgrading to fluent-package v5.0.4!
In this release, a foolproof mechanism was implemented to prevent launching Fluentd wrongly if the service is already running.
As you know, you can check the version of Fluentd with fluentd --version, but there is a case that fluentd -v is executed wrongly to
do it.
When already running Fluentd as a service, fluentd -v launches a duplicated Fluentd instance with the same fluentd configuration.
If you launch duplicated Fluentd instances, it causes the corruption of processing Fluentd buffer.
To prevent such a situation, a foolproof was implemented now.
For example, if Fluentd is running as a service, launching Fluentd causes an error to block it.
Here is the example on Windows:
> fluentd
Error: Can't start duplicate Fluentd instance with the default config.
To start Fluentd, please do one of the following:
(Caution: Please be careful not to start multiple instances with the same config.)
- Stop the Fluentd Windows service 'fluentdwinsvc'.
- Specify the config path explicitly by '-c' ('--config').
Even though if you wrongly launch Fluentd to check version with -v (It should be --version to show version), then it causes the following error.
> fluentd -v
Error: Can't start duplicate Fluentd instance with the default config.
To take the version, please use '--version', not '-v' ('--verbose').
To start Fluentd, please do one of the following:
(Caution: Please be careful not to start multiple instances with the same config.)
- Stop the Fluentd Windows service 'fluentdwinsvc'.
- Specify the config path explicitly by '-c' ('--config')
Note that this foolproof feature is intended to block launching duplicated Fluentd instance, you can explicitly launch Fluentd by specifying a specific option to pass it even though already Fluentd is running as a service.
On Windows:
-c (--config)--dry-run--reg-winsvc--reg-winsvc-fluentdopt--show-plugin-configOn Linux:
-c (--config)--dry-run--show-plugin-configWe plan to ship the next LTS version of fluent-package v5.0.5 on Oct, 2024. The content of updates are still in T.B.D.
As it was already announced Drop schedule announcement about EOL of Treasure Agent (td-agent) 4, td-agent v4 was reached EOL in Dec, 2023.
After reached EOL, td-agent v4.5.3 on Windows was released because there was a crash bug during startup on Windows. It was backported fix from fluent-package v5 as it is critical in some case. Even though this was a exceptional maintenance release, but there is no change to the fact that we already stopped maintaining td-agent v4.
We strongly recommend migrating from td-agent v4 to fluent-package v5 (LTS). See Upgrade to fluent-package v5
Please see the download page.
Hi users!
We have released v1.17.0 on 2024-04-30. ChangeLog is here.
This release is a new release of v1.17 series. In this release, we added some new features for some plugins and fixed bugs of Parser.
in_tail: Add glob_policy option for expanding glob capability of path and exclude_pathIn this release, we added a new option glob_policy for in_tail plugin.
In previous versions, we can use only * in glob patterns for path and exclude_path option.
Example:
path /path/to/*
exclude_path ["/path/to/*.gz", "/path/to/*.zip"]
From this version, we can also use [], ?, and {} in glob patterns depending on the glob_policy option.
Example:
path "[0-1][2-3].log"
glob_policy extended
Please see the document and #4401 for more information.
out_http: Support AWS Signature Version 4 authenticationIn this release, we added a new option aws_sigv4 for the method setting of out_http plugin.
By using this option, out_http can use AWS Signature Version 4.
For example, this allows out_http to write to Amazon OpenSearch Ingestion.
Please see the document and #4459 for more information.
out_http: Add option to reuse connectionsIn this release, we add a new option reuse_connections for out_http plugin.
This option will improve throughput of out_http.
Please see the document and #4330 for more information.
in_http: Recognize CSP reports as JSON dataIn this release, we make the data type of the request where the Content-Type is application/csp-report be considered JSON by default.
Now, in_http can receive Content Security Policy's report by default.
Please see #4282 for more information.
The record data in an event of Fluentd must be a hash object.
However, in the previous versions, some parser plugins could return a non-hash object, such as an array. It could cause errors in subsequent processing.
In this release, the following parser plugins have been fixed.
The changes are as follows:
Here are the details for each case.
Config:
<source>
@type tcp
tag test.tcp
<parse>
@type json
</parse>
</source>
<match test.**>
@type stdout
</match>
Send an array data:
$ netcat 0.0.0.0 5170
[{"k":"v"}, {"k2":"v2"}]
The result before this version:
{datetime} test.tcp: [{"k":"v"},{"k2":"v2"}]
The result after this version:
{datetime} test.tcp: {"k":"v"}
{datetime} test.tcp: {"k2":"v2"}
Config:
<source>
@type tcp
tag test.tcp
<parse>
@type json
null_empty_string
</parse>
</source>
<match test.**>
@type stdout
</match>
Send an array data:
$ netcat 0.0.0.0 5170
[{"k":"v"}, {"k2":"v2"}]
The result before this version:
{datetime} [error]: #0 unexpected error on reading data host="xxx" port=xxx error_class=NoMethodError error="undefined method `each_key' for [{\"k\"=>\"v\"}, {\"k2\"=>\"v2\"}]:Array"
The result after this version:
{datetime} test.tcp: {"k":"v"}
{datetime} test.tcp: {"k2":"v2"}
In the previous versions, filter_parser could return an array record based on this wrong behavior.
From this release, it can not return multiple parsed results anymore and Fluentd outputs a warning log in this case.
This behavior should improve in the future.
Here is an example.
<source>
@type sample
tag test.array
sample {"message": "[{\"k\":\"v\"}, {\"k2\":\"v2\"}]"}
</source>
<filter test.**>
@type parser
key_name message
<parse>
@type json
</parse>
</filter>
<match test.**>
@type stdout
</match>
The result before this version:
{datetime} test.array: [{"k":"v"},{"k2":"v2"}]
The result after this version:
{datetime} [warn]: #0 dump an error event: error_class=Fluent::Plugin::Parser::ParserError error="Could not emit the event. The parser returned multiple results, but currently filter_parser plugin only returns the first parsed result. Raw data: '[{\"k\":\"v\"}, {\"k2\":\"v2\"}]'" location=nil tag="test.array" time=xxx record={"k2"=>"v2"}
{datetime} test.array: {"k":"v"}
These are the major changes for this release.
In addition, some performance improvements have been included! Please see ChangeLog for details!
Enjoy logging!
Hi users!
We have released fluent-package v5.0.3. fluent-package is a stable distribution package of Fluentd. (successor of td-agent v4)
This is a maintenance release of v5.0.x LTS series. As significant slow starting service and crash issues during startup on Windows were fixed, we recommend upgrading to fluent-package v5.0.3!
%~dp0 correctly.We plan to ship next LTS version of fluent-package v5.0.4 on June, 2024. The content of updates are still in T.B.D.
As it was already announced Drop schedule announcement about EOL of Treasure Agent (td-agent) 4, td-agent v4 was reached EOL in Dec, 2023.
After reached EOL, td-agent v4.5.3 on Windows was released because there was a crash bug during startup on Windows. It was backported fix from fluent-package v5 as it is critical in some case. Even though this was a exceptional maintenance release, but there is no change to the fact that we already stopped maintaining td-agent v4.
We strongly recommend migrating from td-agent v4 to fluent-package v5 (LTS). See Upgrade to fluent-package v5
Please see the download page.
Hi users!
We have released v1.16.5 on 2024-03-27. ChangeLog is here.
The previous version v1.16.4 has a serious Buffer bug when it processes large data exceeding chunk size limit. So, we have released v1.16.5 urgently, and have fixed the bug. So, please don't use v1.16.4.
Sorry for troubling.
In the previous version v1.16.4, we fixed a Buffer bug (Please see Fluentd v1.16.4 has been released for details).
There was a race condition problem with that fix, and similar Buffer errors explained in Fluentd v1.16.4 has been released could occur.
emit transaction failed: ...send an error event stream to @ERROR: ...These errors are the same as the Buffer error of Fluentd v1.16.4 has been released, but it would be more likely to happen. The cause is a race condition when processing large data exceeding chunk size limit. So, in that case, these errors can occur depending on the timing.
So, please don't use v1.16.4.
See #4447 for more information.
Sorry for troubling.
Hi users!
We have released v1.16.4 on 2024-03-14. ChangeLog is here.
In this release, we fixed several bugs and improved performance.
In previous versions, when Buffer processes large data exceeding the chunk size limit, it sometimes fails and causes an emit error.
It is usually unlikely, but it may occur when Buffer receives data with extremely large records that are distributed unevenly in the data.
If you set @ERROR label, the data is routed to that label, and Fluentd outputs a warning log message: send an error event stream to @ERROR: ....
However, if you don't set @ERROR label, the data is discarded, and Fluentd outputs a warning log message: emit transaction failed: ....
In addition, Input plugins that do not consider an emit error may stop working, such as in_windows_eventlog2.
Here is an example of the warning message:
2024-03-22 14:13:35 +0900 [warn]: #0 emit transaction failed: error_class=IOError error="closed stream" location="/path/to/fluentd/lib/fluent/plugin/buffer/file_chunk.rb:82:in `pos'" tag="test"
If you see these warnings, please update Fluentd.
See #4342 for more information.
in_tail: Fix tail watchers in rotate_wait state not being managed. #4334
Enjoy logging!
Hi users!
We have released fluent-package v5.0.2 and td-agent v4.5.2. fluent-package is a stable distribution package of Fluentd.
This is a maintenance release of v5.0.x LTS series.
As significant in_tail bugs (wrongly stopping tailing logs) were fixed in latest release, we recommend upgrading to fluent-package v5.0.2!
in_tail.
See Fluentd v1.16.3 and v1.16.2 have been released blog article about details.We plan to ship next LTS version of fluent-package v5.0.3 on Feb, 2024. The content of updates are still in T.B.D.
This is a exceptional maintenance release of v4.5.x series.
Fluentd was updated to 1.16.3 because it contains significant bug fixes about in_tail.
Note that td-agent will not be updated anymore.
See Drop schedule announcement about EOL of Treasure Agent (td-agent) 4.
Please see the download page.
Hi users!
We have released v1.16.3 (changelog) on 2023-11-14. We also describe the content of v1.16.2 (changelog) on 2023-07-14 here.
In these releases, in_tail issues that have been reported for years are fixed!
And other some bug fixes too.
in_tail: Fix in_tail wrongly stopping tailing the current target file and causing handle leaksThis issue has been reported for years. It was looking very serious, but the cause remained unclear until recently.
At last, we found different causes when the follow_inodes option is true and when it is false (default).
We fixed the problem of follow_inodes true case in v1.16.2, and fixed the problem of follow_inodes false case in v1.16.3.
Especially, in follow_inodes false case, the trouble was mainly caused by rotate_wait option.
In past versions, please avoid using large value for this option.
In particular, if you make it greater than refresh_interval option, it will surely cause this issue.
In Fluentd v1.15.1 (td-agent v4.4.0) or later, the following warning log is recorded when this issue occurs.
Skip update_watcher because watcher has been already updated by other inotify event
If this issue occurs, please restart Fluentd, and please update Fluentd to the latest version. fluent-package v5.0.2 will be released in this month and it contains these fixes.
See the following PRs for more information.
in_tail: Fix possible log duplication of follow_inodes option. #4237, #4239in_forward: Fix corrupted data possibly breaking other data. #4178NoMethodError. #4303Enjoy logging!
Fluentd is an open source data collector to simplify log management.
2024-08-29: Scheduled support lifecycle announcement about Fluent Package v6
2023-08-29: Drop schedule announcement about EOL of Treasure Agent (td-agent) 4
2023-08-29: Scheduled support lifecycle announcement about Fluent Package
2023-07-31: Upgrade to fluent-package v5
2024-11-08: fluent-package v5.0.5 has been released
2024-08-29: Scheduled support lifecycle announcement about Fluent Package v6
2024-08-20: Fluentd v1.16.6 has been released
2024-08-19: Fluentd v1.17.1 has been released
2024-08-02: fluent-package v5.1.0 has been released
2024-07-02: fluent-package v5.0.4 has been released
2024-04-30: Fluentd v1.17.0 has been released
2024-03-29: fluent-package v5.0.3 has been released
2024-03-27: Fluentd v1.16.5 has been released
2024-03-14: Fluentd v1.16.4 has been released
Want to learn the basics of Fluentd? Check out these pages.
Couldn't find enough information? Let's ask the community!
You need commercial-grade support from Fluentd committers and experts?
©2010-2024 Fluentd Project. ALL Rights Reserved.
Fluentd is a hosted project under the Cloud Native Computing Foundation (CNCF). All components are available under the Apache 2 License.