Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

dolphinscheduler任务同步成功,执行失败 #389

Closed
iLay1678 opened this issue Nov 5, 2024 · 10 comments
Closed

dolphinscheduler任务同步成功,执行失败 #389

iLay1678 opened this issue Nov 5, 2024 · 10 comments

Comments

@iLay1678
Copy link

iLay1678 commented Nov 5, 2024

java.lang.AbstractMethodError: Receiver class com.qlangtech.tis.plugin.dolphinscheduler.task.TISDatasyncTaskChannel does not define or inherit an implementation of the resolved method abstract parseParameters(Ljava/lang/String;)Lorg/apache/dolphinscheduler/plugin/task/api/parameters/AbstractParameters; of interface org.apache.dolphinscheduler.plugin.task.api.TaskChannel.
	at org.apache.dolphinscheduler.server.worker.utils.TaskExecutionContextUtils.downloadResourcesIfNeeded(TaskExecutionContextUtils.java:71)
	at org.apache.dolphinscheduler.server.worker.runner.WorkerTaskExecutor.beforeExecute(WorkerTaskExecutor.java:226)
	at org.apache.dolphinscheduler.server.worker.runner.WorkerTaskExecutor.run(WorkerTaskExecutor.java:165)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
	at java.base/java.lang.Thread.run(Thread.java:834)

确认已替换worker节点libs中的jar包

@iLay1678
Copy link
Author

iLay1678 commented Nov 5, 2024

另外问一下支持的dolphinscheduler版本,但是dolphinscheduler jdk版本是8,修改的dolphinscheduler-task-datasync是用jdk11编译的

@baisui1981
Copy link
Member

确认已替换worker节点libs中的jar包

  • TIS 测试使用的是 apache-dolphinscheduler-3.2.2-bin 这个版本,估计是你使用的版本比较低不支持,可以换一个新一点的版本试试。
  • TIS的目前使用jdk11 版本编译打包的,你本地可以装一个jdk11 试试,ds 使用 jdk8 应该是兼容 jdk11 的

@iLay1678
Copy link
Author

iLay1678 commented Nov 5, 2024

使用jdk11会报Could not initialize class org.apache.dolphinscheduler.extract.base.client.SingletonJdkDynamicRpcClientProxyFactory

@baisui1981
Copy link
Member

使用jdk11会报Could not initialize class org.apache.dolphinscheduler.extract.base.client.SingletonJdkDynamicRpcClientProxyFactory

你的DS 啥版本的?

@iLay1678
Copy link
Author

iLay1678 commented Nov 5, 2024

docker部署的3.2.2,针对worker容器做了一些映射

dolphinscheduler-worker:
    image: ${HUB}/dolphinscheduler-worker:${TAG}
    profiles: ["all"]
    env_file: .env
    healthcheck:
      test: [ "CMD", "curl", "http://localhost:1235/actuator/health" ]
      interval: 30s
      timeout: 5s
      retries: 3
    depends_on:
      dolphinscheduler-zookeeper:
        condition: service_healthy
    volumes:
      - ./data/worker:/tmp/dolphinscheduler
      - ./logs:/opt/dolphinscheduler/logs
      - ./shared:/opt/soft
      - ./resource:/dolphinscheduler
      - ./worker-libs:/opt/dolphinscheduler/libs
      - ./worker-conf:/opt/dolphinscheduler/conf
      - ./jdk/jdk-11:/opt/java/openjdk

环境配置里设置了下java_home,现在能运行了,但是下载依赖报错

java.lang.RuntimeException: may be there is some errors when connect to TIS host, you would better to deploy the local infrastructure to enable TIS data synchronize by manual
        at com.qlangtech.tis.plugin.dolphinscheduler.task.TISDatasyncTask.init(TISDatasyncTask.java:101)
        at org.apache.dolphinscheduler.server.worker.runner.WorkerTaskExecutor.beforeExecute(WorkerTaskExecutor.java:239)
        at org.apache.dolphinscheduler.server.worker.runner.WorkerTaskExecutor.run(WorkerTaskExecutor.java:164)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base/java.lang.Thread.run(Thread.java:834)
Caused by: com.qlangtech.tis.lang.TisException: maxRetry:1,url:http://mirror.qlangtech.com/4.0.1/tis/tis-data.tar.gz
        at com.qlangtech.tis.lang.TisException.create(TisException.java:171)
        at com.qlangtech.tis.manage.common.ConfigFileContext.processContent(ConfigFileContext.java:136)
        at com.qlangtech.tis.manage.common.ConfigFileContext.processContent(ConfigFileContext.java:112)
        at com.qlangtech.tis.manage.common.ConfigFileContext.processContent(ConfigFileContext.java:108)
        at com.qlangtech.tis.manage.common.HttpUtils.get(HttpUtils.java:109)
        at com.qlangtech.tis.extension.model.UpdateCenter.copyTarToLocal(UpdateCenter.java:171)
        at com.qlangtech.tis.extension.model.UpdateCenter.copyDataTarToLocal(UpdateCenter.java:157)
        at com.qlangtech.tis.plugin.dolphinscheduler.task.TISDatasyncTask.init(TISDatasyncTask.java:80)
        ... 5 common frames omitted
Caused by: java.net.SocketTimeoutException: Read timed out
        at java.base/java.net.SocketInputStream.socketRead0(Native Method)
        at java.base/java.net.SocketInputStream.socketRead(SocketInputStream.java:115)
        at java.base/java.net.SocketInputStream.read(SocketInputStream.java:168)
        at java.base/java.net.SocketInputStream.read(SocketInputStream.java:140)
        at java.base/java.io.BufferedInputStream.fill(BufferedInputStream.java:252)
        at java.base/java.io.BufferedInputStream.read1(BufferedInputStream.java:292)
        at java.base/java.io.BufferedInputStream.read(BufferedInputStream.java:351)
        at java.base/sun.net.www.http.HttpClient.parseHTTPHeader(HttpClient.java:746)
        at java.base/sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:689)
        at java.base/sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1604)
        at java.base/sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1509)
        at java.base/java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:527)
        at com.qlangtech.tis.manage.common.ConfigFileContext.getNetInputStream(ConfigFileContext.java:183)
        at com.qlangtech.tis.manage.common.ConfigFileContext.processContent(ConfigFileContext.java:120)

@baisui1981
Copy link
Member

docker 所在网络环境能连公网不?ds 运行是会去公网下载 资源包 http://mirror.qlangtech.com/4.0.1/tis/tis-data.tar.gz

@iLay1678
Copy link
Author

iLay1678 commented Nov 5, 2024

可以

@baisui1981
Copy link
Member

可以

未必 ,你进入docker 容器 执行一下 wget http://mirror.qlangtech.com/4.0.1/tis/tis-data.tar.gz 试试看

@iLay1678
Copy link
Author

iLay1678 commented Nov 5, 2024

可以了,经排查发现是docker网络mtu问题,OpenStack开的机器mtu默认是1450,docker网络mtu是1500导致read timeout

@iLay1678 iLay1678 closed this as completed Nov 5, 2024
@baisui1981
Copy link
Member

可以了,经排查发现是docker网络mtu问题,OpenStack开的机器mtu默认是1450,docker网络mtu是1500导致read timeout

完美,有啥问题随时交流。多提宝贵意见 😃

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants