You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
linkis1.2.0+dss1.1.0+dolphinscheduler
为了解决hive脚本在dolphinscheduler中执行时查找默认的hive-2.3.3引擎，将dolphinscheduler/lib/下的linkis-computation-governance-common-1.1.1.jar和linkis-label-common-1.1.1.jar替换成了linkis1.2.0中的linkis-computation-governance-common-1.2.0.jar，linkis-label-common-1.2.0.jar，然后再运行shell、python、hive脚本时都会报错，日志如下：
[INFO] 2022-10-19 20:45:37.633 - [taskAppId=TASK-2-9-19]:[115] - create dir success /appcom/tmp/dolphinscheduler/exec/process/3/2/9/19
[INFO] 2022-10-19 20:45:37.648 - [taskAppId=TASK-2-9-19]:[88] - shell task params {"rawScript":"export LINKIS_TYPE='linkis.python.python'\nexport PROXY_USER='hadoop'\nexport LINKIS_VERSION='1.0.0'\nexport JOB_COMMAND='{"script":"4d0bd28b-f8d2-4ab8-a308-096b45da1721.python"}'\nexport JOB_PARAMS='{"variable":{},"configuration":{"special":{},"runtime":{},"startup":{}}}'\nexport JOB_RESOURCES='[{"fileName":"4d0bd28b-f8d2-4ab8-a308-096b45da1721.python","resourceId":"68db3470-e19c-4edf-b3c8-ee894c238463","version":"v000001"}]'\nexport JOB_SOURCE='{"nodeName":"python_3189","workspaceName":"bdapWorkspace","projectName":"test1","flowName":"workflow1"}'\nexport CONTEXT_ID='{"type":"HAWorkFlowContextID","value":"{\"instance\":null,\"backupInstance\":null,\"user\":\"hadoop\",\"workspace\":\"bdapWorkspace\",\"project\":\"test1\",\"flow\":\"workflow1\",\"contextId\":\"8-8--cs_1_devcs_1_dev28\",\"version\":\"v000004\",\"env\":\"BDAP_DEV\"}"}'\nexport LINKIS_GATEWAY_URL='http://192.168.1.36:9111'\nexport DS_VERSION='1.3.8'\nexport RUN_DATE='${global_run_date}'\nexport JOB_LABELS='{"DSSEnv":"dev"}'\nexport FLOW_PROPERTIES='[{"user.to.proxy":"hadoop"}]'\nsh /appcom/Install/dolphinscheduler/dss-dolphinscheduler-client/bin/dss-dolphinscheduler-client.sh bdapWorkspace-test1-workflow1-python_3189"}
[INFO] 2022-10-19 20:45:37.652 - [taskAppId=TASK-2-9-19]:[154] - raw script : export LINKIS_TYPE='linkis.python.python'
export PROXY_USER='hadoop'
export LINKIS_VERSION='1.0.0'
export JOB_COMMAND='{"script":"4d0bd28b-f8d2-4ab8-a308-096b45da1721.python"}'
export JOB_PARAMS='{"variable":{},"configuration":{"special":{},"runtime":{},"startup":{}}}'
export JOB_RESOURCES='[{"fileName":"4d0bd28b-f8d2-4ab8-a308-096b45da1721.python","resourceId":"68db3470-e19c-4edf-b3c8-ee894c238463","version":"v000001"}]'
export JOB_SOURCE='{"nodeName":"python_3189","workspaceName":"bdapWorkspace","projectName":"test1","flowName":"workflow1"}'
export CONTEXT_ID='{"type":"HAWorkFlowContextID","value":"{"instance":null,"backupInstance":null,"user":"hadoop","workspace":"bdapWorkspace","project":"test1","flow":"workflow1","contextId":"8-8--cs_1_devcs_1_dev28","version":"v000004","env":"BDAP_DEV"}"}'
export LINKIS_GATEWAY_URL='http://192.168.1.36:9111'
export DS_VERSION='1.3.8'
export RUN_DATE='20221018'
export JOB_LABELS='{"DSSEnv":"dev"}'
export FLOW_PROPERTIES='[{"user.to.proxy":"hadoop"}]'
sh /appcom/Install/dolphinscheduler/dss-dolphinscheduler-client/bin/dss-dolphinscheduler-client.sh bdapWorkspace-test1-workflow1-python_3189
[INFO] 2022-10-19 20:45:37.653 - [taskAppId=TASK-2-9-19]:[155] - task execute path : /appcom/tmp/dolphinscheduler/exec/process/3/2/9/19
[INFO] 2022-10-19 20:45:37.655 - [taskAppId=TASK-2-9-19]:[87] - tenantCode user:ef215fb4c, task dir:2_9_19
[INFO] 2022-10-19 20:45:37.655 - [taskAppId=TASK-2-9-19]:[92] - create command file:/appcom/tmp/dolphinscheduler/exec/process/3/2/9/19/2_9_19.command
[INFO] 2022-10-19 20:45:37.656 - [taskAppId=TASK-2-9-19]:[111] - command : #!/bin/sh
BASEDIR=$(cd dirname $0; pwd)
cd $BASEDIR
source /appcom/Install/dolphinscheduler/conf/env/dolphinscheduler_env.sh
/appcom/tmp/dolphinscheduler/exec/process/3/2/9/19/2_9_19_node.sh
[INFO] 2022-10-19 20:45:37.662 - [taskAppId=TASK-2-9-19]:[330] - task run command:
sudo -u ef215fb4c sh /appcom/tmp/dolphinscheduler/exec/process/3/2/9/19/2_9_19.command
[INFO] 2022-10-19 20:45:37.665 - [taskAppId=TASK-2-9-19]:[211] - process start, process id is: 16046
[INFO] 2022-10-19 20:45:38.666 - [taskAppId=TASK-2-9-19]:[138] - -> /usr/java/jdk1.8.0_181-cloudera/bin/java
command: java -DserviceName=dss-dophinscheduler-client-bdapWorkspace-test1-workflow1-python_3189 -Xmx256M -XX:+UseG1GC -Xloggc:/appcom/Install/dolphinscheduler/dss-dolphinscheduler-client/logs/dss-dophinscheduler-client-bdapWorkspace-test1-workflow1-python_3189.gc -cp /appcom/Install/dolphinscheduler/dss-dolphinscheduler-client/conf:/appcom/Install/dolphinscheduler/dss-dolphinscheduler-client/lib/* com.webank.wedatasphere.dss.plugins.dolphinscheduler.linkis.client.DSSDolphinSchedulerClient
[INFO] 2022-10-19 20:45:39.667 - [taskAppId=TASK-2-9-19]:[138] - -> SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/appcom/Install/dolphinscheduler/dss-dolphinscheduler-client/lib/log4j-slf4j-impl-2.17.2.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/appcom/Install/dolphinscheduler/dss-dolphinscheduler-client/lib/slf4j-simple-1.7.30.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
[INFO] 2022-10-19 20:45:40.670 - [taskAppId=TASK-2-9-19]:[138] - -> 2022-10-19 20:45:39.877 WARN [main] org.apache.linkis.common.conf.BDPConfiguration$ 50 warn - ******************************** Notice: The Linkis configuration file is linkis.properties ! ***************************
2022-10-19 20:45:39.902 WARN [main] org.apache.linkis.common.conf.BDPConfiguration$ 50 warn - ******************************** Notice: The Linkis serverConf file null is not exists! ***************************
2022-10-19 20:45:39.914 INFO [main] com.webank.wedatasphere.dss.plugins.dolphinscheduler.linkis.client.DSSDolphinSchedulerClient$ 41 info - The DolphinScheduler Version => 1.3.8.
2022-10-19 20:45:39.915 WARN [main] com.webank.wedatasphere.dss.plugins.dolphinscheduler.linkis.client.DSSDolphinSchedulerClient$ 50 warn - For the lower DolphinScheduler versions(< 2.0), we will parse the yarn applicationId to application__\d+_\d+.
2022-10-19 20:45:39.916 WARN [main] com.webank.wedatasphere.dss.plugins.dolphinscheduler.linkis.client.DSSDolphinSchedulerClient$ 50 warn - Why? Because the lower DolphinScheduler versions will block the thread until the yarn application is completed.
2022-10-19 20:45:39.919 INFO [main] com.webank.wedatasphere.dss.plugins.dolphinscheduler.linkis.client.DSSDolphinSchedulerClient$ 44 info - linkistype => linkis.python.python.
2022-10-19 20:45:39.920 INFO [main] com.webank.wedatasphere.dss.plugins.dolphinscheduler.linkis.client.DSSDolphinSchedulerClient$ 44 info - linkis.version => 1.0.0.
2022-10-19 20:45:39.920 INFO [main] com.webank.wedatasphere.dss.plugins.dolphinscheduler.linkis.client.DSSDolphinSchedulerClient$ 44 info - run_date => 20221018.
2022-10-19 20:45:39.921 INFO [main] com.webank.wedatasphere.dss.plugins.dolphinscheduler.linkis.client.DSSDolphinSchedulerClient$ 44 info - wds.linkis.flow.contextID => {"type":"HAWorkFlowContextID","value":"{"instance":null,"backupInstance":null,"user":"hadoop","workspace":"bdapWorkspace","project":"test1","flow":"workflow1","contextId":"8-8--cs_1_devcs_1_dev28","version":"v000004","env":"BDAP_DEV"}"}.
2022-10-19 20:45:39.925 INFO [main] org.apache.linkis.common.conf.Configuration$ 41 info - gatewayUrl is 192.168.1.36:9111
2022-10-19 20:45:39.926 INFO [main] org.apache.linkis.common.conf.Configuration$ 41 info - gatewayUrl is 192.168.1.36:9111
2022-10-19 20:45:39.927 INFO [main] com.webank.wedatasphere.dss.plugins.dolphinscheduler.linkis.client.DSSDolphinSchedulerClient$ 44 info - wds.linkis.gateway.url.v1 => http://192.168.1.36:9111.
2022-10-19 20:45:39.927 INFO [main] com.webank.wedatasphere.dss.plugins.dolphinscheduler.linkis.client.DSSDolphinSchedulerClient$ 44 info - proxy.user => hadoop.
2022-10-19 20:45:39.928 INFO [main] com.webank.wedatasphere.dss.plugins.dolphinscheduler.linkis.client.DSSDolphinSchedulerClient$ 44 info - jobSource => {"nodeName":"python_3189","workspaceName":"bdapWorkspace","projectName":"test1","flowName":"workflow1"}.
2022-10-19 20:45:39.928 INFO [main] com.webank.wedatasphere.dss.plugins.dolphinscheduler.linkis.client.DSSDolphinSchedulerClient$ 44 info - jobParams => {"variable":{},"configuration":{"special":{},"runtime":{},"startup":{}}}.
2022-10-19 20:45:39.929 INFO [main] com.webank.wedatasphere.dss.plugins.dolphinscheduler.linkis.client.DSSDolphinSchedulerClient$ 44 info - jobCommand => {"script":"4d0bd28b-f8d2-4ab8-a308-096b45da1721.python"}.
2022-10-19 20:45:39.929 INFO [main] com.webank.wedatasphere.dss.plugins.dolphinscheduler.linkis.client.DSSDolphinSchedulerClient$ 44 info - jobResources => [{"fileName":"4d0bd28b-f8d2-4ab8-a308-096b45da1721.python","resourceId":"68db3470-e19c-4edf-b3c8-ee894c238463","version":"v000001"}].
2022-10-19 20:45:39.930 INFO [main] com.webank.wedatasphere.dss.plugins.dolphinscheduler.linkis.client.DSSDolphinSchedulerClient$ 44 info - flowProperties => [{"user.to.proxy":"hadoop"}].
2022-10-19 20:45:39.930 INFO [main] com.webank.wedatasphere.dss.plugins.dolphinscheduler.linkis.client.DSSDolphinSchedulerClient$ 44 info - jobLabels => {"DSSEnv":"dev"}.
2022-10-19 20:45:40.423 INFO [main] com.webank.wedatasphere.dss.plugins.dolphinscheduler.linkis.client.job.DolphinSchedulerJobBuilder 109 build - the contextMap is:{type=HAWorkFlowContextID, value={"instance":null,"backupInstance":null,"user":"hadoop","workspace":"bdapWorkspace","project":"test1","flow":"workflow1","contextId":"8-8--cs_1_devcs_1_dev28","version":"v000004","env":"BDAP_DEV"}}
2022-10-19 20:45:40.424 INFO [main] com.webank.wedatasphere.dss.plugins.dolphinscheduler.linkis.client.job.DolphinSchedulerJobBuilder 111 build - try to get workspace str by workspaceName bdapWorkspace.
[INFO] 2022-10-19 20:45:41.672 - [taskAppId=TASK-2-9-19]:[138] - -> 2022-10-19 20:45:41.038 INFO [main] org.reflections.Reflections 229 scan - Reflections took 532 ms to scan 12 urls, producing 207 keys and 2140 values
[INFO] 2022-10-19 20:45:42.625 - [taskAppId=TASK-2-9-19]:[238] - process has exited, execute path:/appcom/tmp/dolphinscheduler/exec/process/3/2/9/19, processId:16046 ,exitStatusCode:1 ,processWaitForStatus:true ,processExitValue:1
[INFO] 2022-10-19 20:45:42.673 - [taskAppId=TASK-2-9-19]:[138] - -> 2022-10-19 20:45:41.987 INFO [main] org.apache.linkis.httpclient.dws.DWSHttpClient 119 org$apache$linkis$httpclient$AbstractHttpClient$$addAttempt$1 - invoke http://192.168.1.36:9111/api/rest_j/v1/dss/framework/workspace/getWorkSpaceStr?workspaceName=bdapWorkspace taken: 140
2022-10-19 20:45:42.099 INFO [main] com.webank.wedatasphere.dss.plugins.dolphinscheduler.linkis.client.job.DolphinSchedulerJobBuilder 113 build - Got workspace str {"workspaceId":224,"workspaceName":"bdapWorkspace","cookies":{"linkis_user_session_ticket_id_v1":"M7UZXQP9Ld01gJnIFIFPPbI6lx7afyeX2HLiVea4FcQ=","dataworkcloud_inner_request":"true","workspaceName":"bdapWorkspace","workspaceId":"224"},"dssUrl":"http://wdss01:9111"}.
2022-10-19 20:45:42.125 INFO [main] com.webank.wedatasphere.dss.plugins.dolphinscheduler.linkis.client.DSSDolphinSchedulerClient$ 44 info - Use workspace client1X:http://192.168.1.36:9111
2022-10-19 20:45:42.403 INFO [main] org.apache.linkis.httpclient.dws.DWSHttpClient 119 org$apache$linkis$httpclient$AbstractHttpClient$$addAttempt$1 - invoke http://192.168.1.36:9111/api/rest_j/v1/filesystem/openScriptFromBML?resourceId=68db3470-e19c-4edf-b3c8-ee894c238463&fileName=4d0bd28b-f8d2-4ab8-a308-096b45da1721.python&version=v000001 taken: 274
2022-10-19 20:45:42.477 INFO [main] com.webank.wedatasphere.dss.plugins.dolphinscheduler.linkis.client.DSSDolphinSchedulerClient$ 44 info - Get execution code from workspace client,bml resource id 68db3470-e19c-4edf-b3c8-ee894c238463, version is v000001
2022-10-19 20:45:42.478 INFO [main] com.webank.wedatasphere.dss.plugins.dolphinscheduler.linkis.client.DSSDolphinSchedulerClient$ 44 info - SUBMIT CODE
2022-10-19 20:45:42.478 INFO [main] com.webank.wedatasphere.dss.plugins.dolphinscheduler.linkis.client.DSSDolphinSchedulerClient$ 44 info - print("hello, world!!!!")
2022-10-19 20:45:42.479 INFO [main] com.webank.wedatasphere.dss.plugins.dolphinscheduler.linkis.client.DSSDolphinSchedulerClient$ 44 info - ************************************SUBMIT CODE************************************
2022-10-19 20:45:42.479 INFO [main] com.webank.wedatasphere.dss.plugins.dolphinscheduler.linkis.client.DSSDolphinSchedulerClient$ 44 info - Start to put variable and configuration
2022-10-19 20:45:42.480 INFO [main] com.webank.wedatasphere.dss.plugins.dolphinscheduler.linkis.client.DSSDolphinSchedulerClient$ 44 info - Finished to put variable and configuration
2022-10-19 20:45:42.482 INFO [main] com.webank.wedatasphere.dss.linkis.node.execution.service.impl.BuildJobActionImpl 62 prepareYarnLabel - Set Yarn cluster label is: {labels={"DSSEnv":"dev"}}
2022-10-19 20:45:42.486 INFO [main] com.webank.wedatasphere.dss.linkis.node.execution.service.impl.BuildJobActionImpl 84 parseExecutionCodeFor1X - The parseExecutionCodeFor1X code for the job is print("hello, world!!!!")
2022-10-19 20:45:42.549 INFO [main] org.apache.linkis.manager.label.builder.factory.LabelBuilderFactoryContext 87 labelBuilderInitRegister - Succeed to register label builder: org.apache.linkis.manager.label.builder.CombinedLabelBuilder
Exception in thread "main" java.lang.NoSuchMethodError: org.apache.linkis.common.utils.Logging.$init$(Lorg/apache/linkis/common/utils/Logging;)V
at org.apache.linkis.manager.label.entity.engine.EngineType$.<init>(EngineType.scala:24)
at org.apache.linkis.manager.label.entity.engine.EngineType$.<clinit>(EngineType.scala)
at org.apache.linkis.manager.label.entity.engine.EngineType.SPARK(EngineType.scala)
at org.apache.linkis.manager.label.utils.EngineTypeLabelCreator.init(EngineTypeLabelCreator.java:48)
at org.apache.linkis.manager.label.utils.EngineTypeLabelCreator.<clinit>(EngineTypeLabelCreator.java:39)
at com.webank.wedatasphere.dss.linkis.node.execution.service.impl.BuildJobActionImpl.getSubmitAction(BuildJobActionImpl.java:131)
at com.webank.wedatasphere.dss.linkis.node.execution.execution.impl.LinkisNodeExecutionImpl.runJob(LinkisNodeExecutionImpl.java:107)
at com.webank.wedatasphere.dss.plugins.dolphinscheduler.linkis.client.DSSDolphinSchedulerClient$.main(DSSDolphinSchedulerClient.scala:75)
at com.webank.wedatasphere.dss.plugins.dolphinscheduler.linkis.client.DSSDolphinSchedulerClient.main(DSSDolphinSchedulerClient.scala)
reacted with thumbs up emoji reacted with thumbs down emoji reacted with laugh emoji reacted with hooray emoji reacted with confused emoji reacted with heart emoji reacted with rocket emoji reacted with eyes emoji
-
linkis1.2.0+dss1.1.0+dolphinscheduler
为了解决hive脚本在dolphinscheduler中执行时查找默认的hive-2.3.3引擎，将dolphinscheduler/lib/下的linkis-computation-governance-common-1.1.1.jar和linkis-label-common-1.1.1.jar替换成了linkis1.2.0中的linkis-computation-governance-common-1.2.0.jar，linkis-label-common-1.2.0.jar，然后再运行shell、python、hive脚本时都会报错，日志如下：
[INFO] 2022-10-19 20:45:37.633 - [taskAppId=TASK-2-9-19]:[115] - create dir success /appcom/tmp/dolphinscheduler/exec/process/3/2/9/19
[INFO] 2022-10-19 20:45:37.648 - [taskAppId=TASK-2-9-19]:[88] - shell task params {"rawScript":"export LINKIS_TYPE='linkis.python.python'\nexport PROXY_USER='hadoop'\nexport LINKIS_VERSION='1.0.0'\nexport JOB_COMMAND='{"script":"4d0bd28b-f8d2-4ab8-a308-096b45da1721.python"}'\nexport JOB_PARAMS='{"variable":{},"configuration":{"special":{},"runtime":{},"startup":{}}}'\nexport JOB_RESOURCES='[{"fileName":"4d0bd28b-f8d2-4ab8-a308-096b45da1721.python","resourceId":"68db3470-e19c-4edf-b3c8-ee894c238463","version":"v000001"}]'\nexport JOB_SOURCE='{"nodeName":"python_3189","workspaceName":"bdapWorkspace","projectName":"test1","flowName":"workflow1"}'\nexport CONTEXT_ID='{"type":"HAWorkFlowContextID","value":"{\"instance\":null,\"backupInstance\":null,\"user\":\"hadoop\",\"workspace\":\"bdapWorkspace\",\"project\":\"test1\",\"flow\":\"workflow1\",\"contextId\":\"8-8--cs_1_devcs_1_dev28\",\"version\":\"v000004\",\"env\":\"BDAP_DEV\"}"}'\nexport LINKIS_GATEWAY_URL='http://192.168.1.36:9111'\nexport DS_VERSION='1.3.8'\nexport RUN_DATE='${global_run_date}'\nexport JOB_LABELS='{"DSSEnv":"dev"}'\nexport FLOW_PROPERTIES='[{"user.to.proxy":"hadoop"}]'\nsh /appcom/Install/dolphinscheduler/dss-dolphinscheduler-client/bin/dss-dolphinscheduler-client.sh bdapWorkspace-test1-workflow1-python_3189"}
[INFO] 2022-10-19 20:45:37.652 - [taskAppId=TASK-2-9-19]:[154] - raw script : export LINKIS_TYPE='linkis.python.python'
export PROXY_USER='hadoop'
export LINKIS_VERSION='1.0.0'
export JOB_COMMAND='{"script":"4d0bd28b-f8d2-4ab8-a308-096b45da1721.python"}'
export JOB_PARAMS='{"variable":{},"configuration":{"special":{},"runtime":{},"startup":{}}}'
export JOB_RESOURCES='[{"fileName":"4d0bd28b-f8d2-4ab8-a308-096b45da1721.python","resourceId":"68db3470-e19c-4edf-b3c8-ee894c238463","version":"v000001"}]'
export JOB_SOURCE='{"nodeName":"python_3189","workspaceName":"bdapWorkspace","projectName":"test1","flowName":"workflow1"}'
export CONTEXT_ID='{"type":"HAWorkFlowContextID","value":"{"instance":null,"backupInstance":null,"user":"hadoop","workspace":"bdapWorkspace","project":"test1","flow":"workflow1","contextId":"8-8--cs_1_devcs_1_dev28","version":"v000004","env":"BDAP_DEV"}"}'
export LINKIS_GATEWAY_URL='http://192.168.1.36:9111'
export DS_VERSION='1.3.8'
export RUN_DATE='20221018'
export JOB_LABELS='{"DSSEnv":"dev"}'
export FLOW_PROPERTIES='[{"user.to.proxy":"hadoop"}]'
sh /appcom/Install/dolphinscheduler/dss-dolphinscheduler-client/bin/dss-dolphinscheduler-client.sh bdapWorkspace-test1-workflow1-python_3189
[INFO] 2022-10-19 20:45:37.653 - [taskAppId=TASK-2-9-19]:[155] - task execute path : /appcom/tmp/dolphinscheduler/exec/process/3/2/9/19
[INFO] 2022-10-19 20:45:37.655 - [taskAppId=TASK-2-9-19]:[87] - tenantCode user:ef215fb4c, task dir:2_9_19
[INFO] 2022-10-19 20:45:37.655 - [taskAppId=TASK-2-9-19]:[92] - create command file:/appcom/tmp/dolphinscheduler/exec/process/3/2/9/19/2_9_19.command
[INFO] 2022-10-19 20:45:37.656 - [taskAppId=TASK-2-9-19]:[111] - command : #!/bin/sh
BASEDIR=$(cd
dirname $0; pwd)cd $BASEDIR
source /appcom/Install/dolphinscheduler/conf/env/dolphinscheduler_env.sh
/appcom/tmp/dolphinscheduler/exec/process/3/2/9/19/2_9_19_node.sh
[INFO] 2022-10-19 20:45:37.662 - [taskAppId=TASK-2-9-19]:[330] - task run command:
sudo -u ef215fb4c sh /appcom/tmp/dolphinscheduler/exec/process/3/2/9/19/2_9_19.command
[INFO] 2022-10-19 20:45:37.665 - [taskAppId=TASK-2-9-19]:[211] - process start, process id is: 16046
[INFO] 2022-10-19 20:45:38.666 - [taskAppId=TASK-2-9-19]:[138] - -> /usr/java/jdk1.8.0_181-cloudera/bin/java
command: java -DserviceName=dss-dophinscheduler-client-bdapWorkspace-test1-workflow1-python_3189 -Xmx256M -XX:+UseG1GC -Xloggc:/appcom/Install/dolphinscheduler/dss-dolphinscheduler-client/logs/dss-dophinscheduler-client-bdapWorkspace-test1-workflow1-python_3189.gc -cp /appcom/Install/dolphinscheduler/dss-dolphinscheduler-client/conf:/appcom/Install/dolphinscheduler/dss-dolphinscheduler-client/lib/* com.webank.wedatasphere.dss.plugins.dolphinscheduler.linkis.client.DSSDolphinSchedulerClient
[INFO] 2022-10-19 20:45:39.667 - [taskAppId=TASK-2-9-19]:[138] - -> SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/appcom/Install/dolphinscheduler/dss-dolphinscheduler-client/lib/log4j-slf4j-impl-2.17.2.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/appcom/Install/dolphinscheduler/dss-dolphinscheduler-client/lib/slf4j-simple-1.7.30.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
[INFO] 2022-10-19 20:45:40.670 - [taskAppId=TASK-2-9-19]:[138] - -> 2022-10-19 20:45:39.877 WARN [main] org.apache.linkis.common.conf.BDPConfiguration$ 50 warn - ******************************** Notice: The Linkis configuration file is linkis.properties ! ***************************
2022-10-19 20:45:39.902 WARN [main] org.apache.linkis.common.conf.BDPConfiguration$ 50 warn - ******************************** Notice: The Linkis serverConf file null is not exists! ***************************
2022-10-19 20:45:39.914 INFO [main] com.webank.wedatasphere.dss.plugins.dolphinscheduler.linkis.client.DSSDolphinSchedulerClient$ 41 info - The DolphinScheduler Version => 1.3.8.
2022-10-19 20:45:39.915 WARN [main] com.webank.wedatasphere.dss.plugins.dolphinscheduler.linkis.client.DSSDolphinSchedulerClient$ 50 warn - For the lower DolphinScheduler versions(< 2.0), we will parse the yarn applicationId to application__\d+_\d+.
2022-10-19 20:45:39.916 WARN [main] com.webank.wedatasphere.dss.plugins.dolphinscheduler.linkis.client.DSSDolphinSchedulerClient$ 50 warn - Why? Because the lower DolphinScheduler versions will block the thread until the yarn application is completed.
2022-10-19 20:45:39.919 INFO [main] com.webank.wedatasphere.dss.plugins.dolphinscheduler.linkis.client.DSSDolphinSchedulerClient$ 44 info - linkistype => linkis.python.python.
2022-10-19 20:45:39.920 INFO [main] com.webank.wedatasphere.dss.plugins.dolphinscheduler.linkis.client.DSSDolphinSchedulerClient$ 44 info - linkis.version => 1.0.0.
2022-10-19 20:45:39.920 INFO [main] com.webank.wedatasphere.dss.plugins.dolphinscheduler.linkis.client.DSSDolphinSchedulerClient$ 44 info - run_date => 20221018.
2022-10-19 20:45:39.921 INFO [main] com.webank.wedatasphere.dss.plugins.dolphinscheduler.linkis.client.DSSDolphinSchedulerClient$ 44 info - wds.linkis.flow.contextID => {"type":"HAWorkFlowContextID","value":"{"instance":null,"backupInstance":null,"user":"hadoop","workspace":"bdapWorkspace","project":"test1","flow":"workflow1","contextId":"8-8--cs_1_devcs_1_dev28","version":"v000004","env":"BDAP_DEV"}"}.
2022-10-19 20:45:39.925 INFO [main] org.apache.linkis.common.conf.Configuration$ 41 info - gatewayUrl is 192.168.1.36:9111
2022-10-19 20:45:39.926 INFO [main] org.apache.linkis.common.conf.Configuration$ 41 info - gatewayUrl is 192.168.1.36:9111
2022-10-19 20:45:39.927 INFO [main] com.webank.wedatasphere.dss.plugins.dolphinscheduler.linkis.client.DSSDolphinSchedulerClient$ 44 info - wds.linkis.gateway.url.v1 => http://192.168.1.36:9111.
2022-10-19 20:45:39.927 INFO [main] com.webank.wedatasphere.dss.plugins.dolphinscheduler.linkis.client.DSSDolphinSchedulerClient$ 44 info - proxy.user => hadoop.
2022-10-19 20:45:39.928 INFO [main] com.webank.wedatasphere.dss.plugins.dolphinscheduler.linkis.client.DSSDolphinSchedulerClient$ 44 info - jobSource => {"nodeName":"python_3189","workspaceName":"bdapWorkspace","projectName":"test1","flowName":"workflow1"}.
2022-10-19 20:45:39.928 INFO [main] com.webank.wedatasphere.dss.plugins.dolphinscheduler.linkis.client.DSSDolphinSchedulerClient$ 44 info - jobParams => {"variable":{},"configuration":{"special":{},"runtime":{},"startup":{}}}.
2022-10-19 20:45:39.929 INFO [main] com.webank.wedatasphere.dss.plugins.dolphinscheduler.linkis.client.DSSDolphinSchedulerClient$ 44 info - jobCommand => {"script":"4d0bd28b-f8d2-4ab8-a308-096b45da1721.python"}.
2022-10-19 20:45:39.929 INFO [main] com.webank.wedatasphere.dss.plugins.dolphinscheduler.linkis.client.DSSDolphinSchedulerClient$ 44 info - jobResources => [{"fileName":"4d0bd28b-f8d2-4ab8-a308-096b45da1721.python","resourceId":"68db3470-e19c-4edf-b3c8-ee894c238463","version":"v000001"}].
2022-10-19 20:45:39.930 INFO [main] com.webank.wedatasphere.dss.plugins.dolphinscheduler.linkis.client.DSSDolphinSchedulerClient$ 44 info - flowProperties => [{"user.to.proxy":"hadoop"}].
2022-10-19 20:45:39.930 INFO [main] com.webank.wedatasphere.dss.plugins.dolphinscheduler.linkis.client.DSSDolphinSchedulerClient$ 44 info - jobLabels => {"DSSEnv":"dev"}.
2022-10-19 20:45:40.423 INFO [main] com.webank.wedatasphere.dss.plugins.dolphinscheduler.linkis.client.job.DolphinSchedulerJobBuilder 109 build - the contextMap is:{type=HAWorkFlowContextID, value={"instance":null,"backupInstance":null,"user":"hadoop","workspace":"bdapWorkspace","project":"test1","flow":"workflow1","contextId":"8-8--cs_1_devcs_1_dev28","version":"v000004","env":"BDAP_DEV"}}
2022-10-19 20:45:40.424 INFO [main] com.webank.wedatasphere.dss.plugins.dolphinscheduler.linkis.client.job.DolphinSchedulerJobBuilder 111 build - try to get workspace str by workspaceName bdapWorkspace.
[INFO] 2022-10-19 20:45:41.672 - [taskAppId=TASK-2-9-19]:[138] - -> 2022-10-19 20:45:41.038 INFO [main] org.reflections.Reflections 229 scan - Reflections took 532 ms to scan 12 urls, producing 207 keys and 2140 values
[INFO] 2022-10-19 20:45:42.625 - [taskAppId=TASK-2-9-19]:[238] - process has exited, execute path:/appcom/tmp/dolphinscheduler/exec/process/3/2/9/19, processId:16046 ,exitStatusCode:1 ,processWaitForStatus:true ,processExitValue:1
[INFO] 2022-10-19 20:45:42.673 - [taskAppId=TASK-2-9-19]:[138] - -> 2022-10-19 20:45:41.987 INFO [main] org.apache.linkis.httpclient.dws.DWSHttpClient 119 org$apache$linkis$httpclient$AbstractHttpClient$$addAttempt$1 - invoke http://192.168.1.36:9111/api/rest_j/v1/dss/framework/workspace/getWorkSpaceStr?workspaceName=bdapWorkspace taken: 140
2022-10-19 20:45:42.099 INFO [main] com.webank.wedatasphere.dss.plugins.dolphinscheduler.linkis.client.job.DolphinSchedulerJobBuilder 113 build - Got workspace str {"workspaceId":224,"workspaceName":"bdapWorkspace","cookies":{"linkis_user_session_ticket_id_v1":"M7UZXQP9Ld01gJnIFIFPPbI6lx7afyeX2HLiVea4FcQ=","dataworkcloud_inner_request":"true","workspaceName":"bdapWorkspace","workspaceId":"224"},"dssUrl":"http://wdss01:9111"}.
2022-10-19 20:45:42.125 INFO [main] com.webank.wedatasphere.dss.plugins.dolphinscheduler.linkis.client.DSSDolphinSchedulerClient$ 44 info - Use workspace client1X:http://192.168.1.36:9111
2022-10-19 20:45:42.403 INFO [main] org.apache.linkis.httpclient.dws.DWSHttpClient 119 org$apache$linkis$httpclient$AbstractHttpClient$$addAttempt$1 - invoke http://192.168.1.36:9111/api/rest_j/v1/filesystem/openScriptFromBML?resourceId=68db3470-e19c-4edf-b3c8-ee894c238463&fileName=4d0bd28b-f8d2-4ab8-a308-096b45da1721.python&version=v000001 taken: 274
2022-10-19 20:45:42.477 INFO [main] com.webank.wedatasphere.dss.plugins.dolphinscheduler.linkis.client.DSSDolphinSchedulerClient$ 44 info - Get execution code from workspace client,bml resource id 68db3470-e19c-4edf-b3c8-ee894c238463, version is v000001
2022-10-19 20:45:42.478 INFO [main] com.webank.wedatasphere.dss.plugins.dolphinscheduler.linkis.client.DSSDolphinSchedulerClient$ 44 info - SUBMIT CODE
2022-10-19 20:45:42.478 INFO [main] com.webank.wedatasphere.dss.plugins.dolphinscheduler.linkis.client.DSSDolphinSchedulerClient$ 44 info - print("hello, world!!!!")
Beta Was this translation helpful? Give feedback.
All reactions