Apache Airflow version

2.10.3

If "Other Airflow 2 version" selected, which one?

No response

What happened?

Hello all, hope you doing well.

While trying to send logs to elasticsearch directly threw the elastic adapter (inside airflow conf), it does not work.
--> today we pass threw file share (azure) mounted as PV inside K8S, then logstash pipeline, but it costs a lots per years ...

I think the adapter is broken, even when trying to test a connection directly inside the webserver, we have an error :

'ESConnection' object has no attribute 'close'

Then when trying to send logs to elasticsearch, it does not try to send logs and it cannot connect to the elasticsearch when trying to get back logs.

Anyway, while trying to look for logs inside a DAG we also have this error :

elasticsearch.AuthenticationException: AuthenticationException(401, 'security_exception', 'missing authentication credentials for REST request [/airflow-logs-*/_count]') airflow-webserver-1 | 172.18.0.1 - - [06/Dec/2024:11:26:46 +0000] "GET /api/v1/dags/debug_airflow_to_elastic/dagRuns/manual__2024-12-06T11:26:37.042642+00:00/taskInstances/print_debug_message/logs/1?full_content=false HTTP/1.1" 500 1588 "http://localhost:8080/dags/debug_airflow_to_elastic/grid?dag_run_id=manual__2024-12-06T11%3A26%3A37.042642%2B00%3A00&task_id=print_debug_message&tab=logs" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36"

But the I tried with an api key, user password and even both but cannot get rid of it.

I think there is a bug around it or did we do something wrong.

Thousands thanks !

Benjamin

What you think should happen instead?

No response

How to reproduce

Just add a connection to elasticsearch, try to connect to it.

Then add remote logging inside conf.

It does not try to send logs and it cannot connect to the elasticsearch.

Operating System

Kubernetes and docker compose. (both d'ont work)

Versions of Apache Airflow Providers

No response

Deployment

Official Apache Airflow Helm Chart

Deployment details

[core] sensitive_var_conn_names = key,login,secret,pass,auth hide_sensitive_var_conn_fields = True max_map_length = 16396 expose_config = non-sensitive-only load_examples = False test_connection = Enabled [webserver] show_trigger_form_if_no_params = True allow_testing_connections = Enabled [logging] remote_logging = True remote_log_conn_id = elasticsearch_default logging_level = INFO [elasticsearch] host = ************************************ write_stdout = True json_format = True index_patterns = airflow-logs-* [elasticsearch_config] verify_certs=False

Anything else?

No response

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct