Skip to content

ActiveMQ Artemis Scaler - connection refused using exact same scaled object working on 2.13.1 but not on 2.14 or higher #6333

New issue
New issue
Closed
Closed
ActiveMQ Artemis Scaler - connection refused using exact same scaled object working on 2.13.1 but not on 2.14 or higher#6333
Labels
bugSomething isn't workingstaleAll issues that are marked as stale due to inactivity
@GrayBlueJay

Description

@GrayBlueJay
GrayBlueJay
opened on Nov 14, 2024

Report

After upgrading from 2.13.1 to 2.14 (also tried on 2.16) existing scaled objects that worked on 2.13.1 now get a connection refused error in 2.14 and 2.16 (have not tested 2.15).

When rolling back to 2.13.1 the scaled objects are able to connect without issue.

Trigger definition:

  Triggers:
    Authentication Ref:
      Name:  trigger-auth-kedartemis
    Metadata:
      Broker Address:       DEV_File_Notify
      Broker Name:          tscore-artemis
      Management Endpoint:  192.168.62.224:8161
      Queue Length:         100
      Queue Name:           DEV_File_Notify
    Type:                   artemis-queue

keda-operator logs for scaled object in version 2.16:

2024-11-14T20:17:43Z    ERROR   artemis_queue_scaler    Unable to access the artemis management endpoint        {"type": "ScaledObject", "namespace": "dev", "name": "tscorefilenotify", "managementEndpoint": "192.168.62.224:8161", "error": "Get \"http://192.168.62.224:8161/console/jolokia/read/org.apache.activemq.artemis:broker=%22tscore-artemis%22,component=addresses,address=%22DEV_File_Notify%22,subcomponent=queues,routing-type=%22anycast%22,queue=%22DEV_File_Notify%22/MessageCount\": dial tcp 192.168.62.224:8161: connect: connection refused"}
2024-11-14T20:17:43Z    ERROR   artemis_queue_scaler    Unable to access the artemis management endpoint        {"type": "ScaledObject", "namespace": "dev", "name": "tscorefilenotify", "managementEndpoint": "192.168.62.224:8161", "error": "Get \"http://192.168.62.224:8161/console/jolokia/read/org.apache.activemq.artemis:broker=%22tscore-artemis%22,component=addresses,address=%22DEV_File_Notify%22,subcomponent=queues,routing-type=%22anycast%22,queue=%22DEV_File_Notify%22/MessageCount\": dial tcp 192.168.62.224:8161: connect: connection refused"}
2024-11-14T20:17:43Z    ERROR   scale_handler   error getting scale decision    {"scaledObject.Namespace": "dev", "scaledObject.Name": "tscorefilenotify", "scaler": "artemisScaler", "error": "Get \"http://192.168.62.224:8161/console/jolokia/read/org.apache.activemq.artemis:broker=%22tscore-artemis%22,component=addresses,address=%22DEV_File_Notify%22,subcomponent=queues,routing-type=%22anycast%22,queue=%22DEV_File_Notify%22/MessageCount\": dial tcp 192.168.62.224:8161: connect: connection refused"}
2024-11-14T20:18:13Z    ERROR   artemis_queue_scaler    Unable to access the artemis management endpoint        {"type": "ScaledObject", "namespace": "dev", "name": "tscorefilenotify", "managementEndpoint": "192.168.62.224:8161", "error": "Get \"http://192.168.62.224:8161/console/jolokia/read/org.apache.activemq.artemis:broker=%22tscore-artemis%22,component=addresses,address=%22DEV_File_Notify%22,subcomponent=queues,routing-type=%22anycast%22,queue=%22DEV_File_Notify%22/MessageCount\": dial tcp 192.168.62.224:8161: connect: connection refused"}
2024-11-14T20:18:13Z    ERROR   artemis_queue_scaler    Unable to access the artemis management endpoint        {"type": "ScaledObject", "namespace": "dev", "name": "tscorefilenotify", "managementEndpoint": "192.168.62.224:8161", "error": "Get \"http://192.168.62.224:8161/console/jolokia/read/org.apache.activemq.artemis:broker=%22tscore-artemis%22,component=addresses,address=%22DEV_File_Notify%22,subcomponent=queues,routing-type=%22anycast%22,queue=%22DEV_File_Notify%22/MessageCount\": dial tcp 192.168.62.224:8161: connect: connection refused"}
2024-11-14T20:18:13Z    ERROR   scale_handler   error getting scale decision    {"scaledObject.Namespace": "dev", "scaledObject.Name": "tscorefilenotify", "scaler": "artemisScaler", "error": "Get \"http://192.168.62.224:8161/console/jolokia/read/org.apache.activemq.artemis:broker=%22tscore-artemis%22,component=addresses,address=%22DEV_File_Notify%22,subcomponent=queues,routing-type=%22anycast%22,queue=%22DEV_File_Notify%22/MessageCount\": dial tcp 192.168.62.224:8161: connect: connection refused"}
2024-11-14T20:18:43Z    INFO    Reconciling ScaledObject        {"controller": "scaledobject", "controllerGroup": "keda.sh", "controllerKind": "ScaledObject", "ScaledObject": {"name":"tscorefilenotify","namespace":"dev"}, "namespace": "dev", "name": "tscorefilenotify", "reconcileID": "bffbdbf3-9c3a-4cc0-84be-e15b442a16a2"}
2024-11-14T20:18:43Z    INFO    Detected resource targeted for scaling  {"controller": "scaledobject", "controllerGroup": "keda.sh", "controllerKind": "ScaledObject", "ScaledObject": {"name":"tscorefilenotify","namespace":"dev"}, "namespace": "dev", "name": "tscorefilenotify", "reconcileID": "bffbdbf3-9c3a-4cc0-84be-e15b442a16a2", "resource": "apps/v1.Deployment", "name": "tscorefilenotify"}

keda-operator logs after rolling back to 2.13.1:

2024-11-14T20:33:32Z    INFO    Reconciling ScaledObject        {"controller": "scaledobject", "controllerGroup": "keda.sh", "controllerKind": "ScaledObject", "ScaledObject": {"name":"tscorefilenotify","namespace":"dev"}, "namespace": "dev", "name": "tscorefilenotify", "reconcileID": "ea9191c1-df3a-4d7d-a5e1-94e040b0e636"}
2024-11-14T20:33:32Z    INFO    Detected resource targeted for scaling  {"controller": "scaledobject", "controllerGroup": "keda.sh", "controllerKind": "ScaledObject", "ScaledObject": {"name":"tscorefilenotify","namespace":"dev"}, "namespace": "dev", "name": "tscorefilenotify", "reconcileID": "ea9191c1-df3a-4d7d-a5e1-94e040b0e636", "resource": "apps/v1.Deployment", "name": "tscorefilenotify"}
2024-11-14T20:33:32Z    INFO    Updated HPA according to ScaledObject   {"controller": "scaledobject", "controllerGroup": "keda.sh", "controllerKind": "ScaledObject", "ScaledObject": {"name":"tscorefilenotify","namespace":"dev"}, "namespace": "dev", "name": "tscorefilenotify", "reconcileID": "ea9191c1-df3a-4d7d-a5e1-94e040b0e636", "HPA.Namespace": "dev", "HPA.Name": "keda-hpa-tscorefilenotify"}
2024-11-14T20:33:32Z    INFO    Initializing Scaling logic according to ScaledObject Specification      {"controller": "scaledobject", "controllerGroup": "keda.sh", "controllerKind": "ScaledObject", "ScaledObject": {"name":"tscorefilenotify","namespace":"dev"}, "namespace": "dev", "name": "tscorefilenotify", "reconcileID": "ea9191c1-df3a-4d7d-a5e1-94e040b0e636"}
2024-11-14T20:33:32Z    INFO    scaleexecutor   Successfully updated ScaleTarget        {"scaledobject.Name": "tscorefilenotify", "scaledObject.Namespace": "dev", "scaleTarget.Name": "tscorefilenotify", "Original Replicas Count": 0, "New Replicas Count": 1}
2024-11-14T20:33:35Z    INFO    Reconciling ScaledObject        {"controller": "scaledobject", "controllerGroup": "keda.sh", "controllerKind": "ScaledObject", "ScaledObject": {"name":"tscorefilenotify","namespace":"dev"}, "namespace": "dev", "name": "tscorefilenotify", "reconcileID": "071d2235-44cd-4608-8cc9-337d7651a6c3"}
2024-11-14T20:33:36Z    INFO    Detected resource targeted for scaling  {"controller": "scaledobject", "controllerGroup": "keda.sh", "controllerKind": "ScaledObject", "ScaledObject": {"name":"tscorefilenotify","namespace":"dev"}, "namespace": "dev", "name": "tscorefilenotify", "reconcileID": "071d2235-44cd-4608-8cc9-337d7651a6c3", "resource": "apps/v1.Deployment", "name": "tscorefilenotify"}
2024-11-14T20:39:32Z    INFO    scaleexecutor   Successfully set ScaleTarget replicas count to ScaledObject minReplicaCount     {"scaledobject.Name": "tscorefilenotify", "scaledObject.Namespace": "dev", "scaleTarget.Name": "tscorefilenotify", "Original Replicas Count": 1, "New Replicas Count": 0}

Please let me know if you need additional information

Expected Behavior

I do not see any changes documented in the scaler for Artemis or note any breaking changes in the 2.14 release notes and therefore expect that the scaler would continue to work as it did in 2.13.1

Actual Behavior

Connection refused error being thrown when using an artemis scaler that worked in 2.13.1 but now no longer works in version >= 2.14

Steps to Reproduce the Problem

  1. Create artemis scaler in version < 2.14
  2. Confirm artemis scaler works
  3. Upgrade Keda to version >= 2.14
  4. Observer connection refused error

Logs from KEDA operator

2024-11-14T20:33:32Z    INFO    Reconciling ScaledObject        {"controller": "scaledobject", "controllerGroup": "keda.sh", "controllerKind": "ScaledObject", "ScaledObject": {"name":"tscorefilenotify","namespace":"dev"}, "namespace": "dev", "name": "tscorefilenotify", "reconcileID": "ea9191c1-df3a-4d7d-a5e1-94e040b0e636"}
2024-11-14T20:33:32Z    INFO    Detected resource targeted for scaling  {"controller": "scaledobject", "controllerGroup": "keda.sh", "controllerKind": "ScaledObject", "ScaledObject": {"name":"tscorefilenotify","namespace":"dev"}, "namespace": "dev", "name": "tscorefilenotify", "reconcileID": "ea9191c1-df3a-4d7d-a5e1-94e040b0e636", "resource": "apps/v1.Deployment", "name": "tscorefilenotify"}
2024-11-14T20:33:32Z    INFO    Updated HPA according to ScaledObject   {"controller": "scaledobject", "controllerGroup": "keda.sh", "controllerKind": "ScaledObject", "ScaledObject": {"name":"tscorefilenotify","namespace":"dev"}, "namespace": "dev", "name": "tscorefilenotify", "reconcileID": "ea9191c1-df3a-4d7d-a5e1-94e040b0e636", "HPA.Namespace": "dev", "HPA.Name": "keda-hpa-tscorefilenotify"}
2024-11-14T20:33:32Z    INFO    Initializing Scaling logic according to ScaledObject Specification      {"controller": "scaledobject", "controllerGroup": "keda.sh", "controllerKind": "ScaledObject", "ScaledObject": {"name":"tscorefilenotify","namespace":"dev"}, "namespace": "dev", "name": "tscorefilenotify", "reconcileID": "ea9191c1-df3a-4d7d-a5e1-94e040b0e636"}
2024-11-14T20:33:32Z    INFO    scaleexecutor   Successfully updated ScaleTarget        {"scaledobject.Name": "tscorefilenotify", "scaledObject.Namespace": "dev", "scaleTarget.Name": "tscorefilenotify", "Original Replicas Count": 0, "New Replicas Count": 1}
2024-11-14T20:33:35Z    INFO    Reconciling ScaledObject        {"controller": "scaledobject", "controllerGroup": "keda.sh", "controllerKind": "ScaledObject", "ScaledObject": {"name":"tscorefilenotify","namespace":"dev"}, "namespace": "dev", "name": "tscorefilenotify", "reconcileID": "071d2235-44cd-4608-8cc9-337d7651a6c3"}
2024-11-14T20:33:36Z    INFO    Detected resource targeted for scaling  {"controller": "scaledobject", "controllerGroup": "keda.sh", "controllerKind": "ScaledObject", "ScaledObject": {"name":"tscorefilenotify","namespace":"dev"}, "namespace": "dev", "name": "tscorefilenotify", "reconcileID": "071d2235-44cd-4608-8cc9-337d7651a6c3", "resource": "apps/v1.Deployment", "name": "tscorefilenotify"}
2024-11-14T20:39:32Z    INFO    scaleexecutor   Successfully set ScaleTarget replicas count to ScaledObject minReplicaCount     {"scaledobject.Name": "tscorefilenotify", "scaledObject.Namespace": "dev", "scaleTarget.Name": "tscorefilenotify", "Original Replicas Count": 1, "New Replicas Count": 0}

KEDA Version

2.13.1

Kubernetes Version

1.30

Platform

Amazon Web Services

Scaler Details

ActiveMq Artemis

Anything else?

No response

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't workingstaleAll issues that are marked as stale due to inactivity

    Type

    No type

    Projects

    Roadmap - KEDA Core

    Status

    Ready To Ship

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions