Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Bug] [Data Development - Project] SQL jobs for Dinky-v1.20-1.2.1 do not support SQL-Hints for Flink-1.20(STATE_TTL) #4252

Open
2 of 3 tasks
adu-shzz opened this issue Feb 25, 2025 · 1 comment
Open
2 of 3 tasks

[Bug] [Data Development - Project] SQL jobs for Dinky-v1.20-1.2.1 do not support SQL-Hints for Flink-1.20(STATE_TTL) #4252

adu-shzz opened this issue Feb 25, 2025 · 1 comment
Assignees
@Zzm0809
Labels
Bug Something isn't working

Comments

@adu-shzz
Copy link

adu-shzz commented Feb 25, 2025 •
edited
Loading

Search before asking

  • I had searched in the issues and found no similar issues.

What happened

如标题所述，在 Dinky-1.20-1.2.1 中，创建一个 FlinkSQL 作业，使用 Flink-1.20 的 SQL-Hint /*+ STATE_TTL('tmp0' = '90s') */ ，在做 SQL 检查操作时报错，而直接在 Flink 的 sql-client 终端中执行相同的 SQL，一切正常。

Dinky SQL 中的报错信息如下：

Error message:
 org.apache.flink.table.api.ValidationException: The options of following hints cannot match the name of input tables or views: 
`tmp0` in `STATE_TTL`
	at org.apache.flink.table.planner.plan.optimize.QueryHintsResolver.validateHints(QueryHintsResolver.java:336)
	at org.apache.flink.table.planner.plan.optimize.QueryHintsResolver.resolve(QueryHintsResolver.java:83)
	at org.apache.flink.table.planner.plan.optimize.CommonSubGraphBasedOptimizer.optimize(CommonSubGraphBasedOptimizer.scala:81)
	at org.apache.flink.table.planner.delegation.PlannerBase.optimize(PlannerBase.scala:320)
	at org.apache.flink.table.planner.delegation.PlannerBase.getExplainGraphs(PlannerBase.scala:534)
	at org.apache.flink.table.planner.delegation.StreamPlanner.explain(StreamPlanner.scala:103)
	at org.apache.flink.table.planner.delegation.StreamPlanner.explain(StreamPlanner.scala:51)
	at org.dinky.executor.CustomTableEnvironmentImpl.explainStatementSet(CustomTableEnvironmentImpl.java:260)
	at org.dinky.executor.Executor.explainStatementSet(Executor.java:368)
	at org.dinky.job.runner.JobSqlRunner.explain(JobSqlRunner.java:154)
	at org.dinky.explainer.Explainer.explainSql(Explainer.java:137)
	at org.dinky.job.JobManager.explainSql(JobManager.java:352)
	at org.dinky.service.task.FlinkSqlTask.explain(FlinkSqlTask.java:58)
	at org.dinky.service.impl.TaskServiceImpl.explainTask(TaskServiceImpl.java:497)
	at org.dinky.service.impl.TaskServiceImpl$$FastClassBySpringCGLIB$$22087f7c.invoke(<generated>)
	at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218)
	at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:792)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
	at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:762)
	at org.springframework.aop.aspectj.MethodInvocationProceedingJoinPoint.proceed(MethodInvocationProceedingJoinPoint.java:89)
	at org.dinky.aop.UdfClassLoaderAspect.round(UdfClassLoaderAspect.java:55)
	at sun.reflect.GeneratedMethodAccessor448.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethodWithGivenArgs(AbstractAspectJAdvice.java:634)
	at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethod(AbstractAspectJAdvice.java:624)
	at org.springframework.aop.aspectj.AspectJAroundAdvice.invoke(AspectJAroundAdvice.java:72)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
	at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:762)
	at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:97)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
	at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:762)
	at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:707)
	at org.dinky.service.impl.TaskServiceImpl$$EnhancerBySpringCGLIB$$db0087ce.explainTask(<generated>)
	at org.dinky.controller.TaskController.explainSql(TaskController.java:188)
	at org.dinky.controller.TaskController$$FastClassBySpringCGLIB$$5b68bd32.invoke(<generated>)
	at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218)
	at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:792)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
	at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:762)
	at org.springframework.aop.framework.adapter.AfterReturningAdviceInterceptor.invoke(AfterReturningAdviceInterceptor.java:57)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
	at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:762)
	at org.springframework.aop.aspectj.MethodInvocationProceedingJoinPoint.proceed(MethodInvocationProceedingJoinPoint.java:89)
	at org.dinky.aop.TaskOperationPermissionAspect.processAround(TaskOperationPermissionAspect.java:82)
	at sun.reflect.GeneratedMethodAccessor604.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethodWithGivenArgs(AbstractAspectJAdvice.java:634)
	at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethod(AbstractAspectJAdvice.java:624)
	at org.springframework.aop.aspectj.AspectJAroundAdvice.invoke(AspectJAroundAdvice.java:72)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:175)
	at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:762)
	at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:97)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
	at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:762)
	at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:707)
	at org.dinky.controller.TaskController$$EnhancerBySpringCGLIB$$88e93e6a.explainSql(<generated>)
	at sun.reflect.GeneratedMethodAccessor1156.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:205)
	at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:150)
	at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:117)
	at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:895)
	at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:808)
	at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:87)
	at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:1072)
	at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:965)
	at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:1006)
	at org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:909)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:517)
	at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:883)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:584)
	at io.undertow.servlet.handlers.ServletHandler.handleRequest(ServletHandler.java:74)
	at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:129)
	at io.undertow.websockets.jsr.JsrWebSocketFilter.doFilter(JsrWebSocketFilter.java:173)
	at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:67)
	at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131)
	at org.springframework.web.filter.CorsFilter.doFilterInternal(CorsFilter.java:91)
	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117)
	at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:67)
	at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131)
	at com.alibaba.druid.support.http.WebStatFilter.doFilter(WebStatFilter.java:114)
	at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:67)
	at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131)
	at org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:100)
	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117)
	at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:67)
	at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131)
	at cn.dev33.satoken.filter.SaPathCheckFilterForServlet.doFilter(SaPathCheckFilterForServlet.java:55)
	at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:67)
	at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131)
	at org.springframework.web.filter.FormContentFilter.doFilterInternal(FormContentFilter.java:93)
	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117)
	at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:67)
	at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131)
	at org.springframework.boot.actuate.metrics.web.servlet.WebMvcMetricsFilter.doFilterInternal(WebMvcMetricsFilter.java:96)
	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117)
	at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:67)
	at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131)
	at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:201)
	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117)
	at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:67)
	at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131)
	at io.undertow.servlet.handlers.FilterHandler.handleRequest(FilterHandler.java:84)
	at io.undertow.servlet.handlers.security.ServletSecurityRoleHandler.handleRequest(ServletSecurityRoleHandler.java:62)
	at io.undertow.servlet.handlers.ServletChain$1.handleRequest(ServletChain.java:68)
	at io.undertow.servlet.handlers.ServletDispatchingHandler.handleRequest(ServletDispatchingHandler.java:36)
	at io.undertow.servlet.handlers.RedirectDirHandler.handleRequest(RedirectDirHandler.java:68)
	at io.undertow.servlet.handlers.security.SSLInformationAssociationHandler.handleRequest(SSLInformationAssociationHandler.java:117)
	at io.undertow.servlet.handlers.security.ServletAuthenticationCallHandler.handleRequest(ServletAuthenticationCallHandler.java:57)
	at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
	at io.undertow.security.handlers.AbstractConfidentialityHandler.handleRequest(AbstractConfidentialityHandler.java:46)
	at io.undertow.servlet.handlers.security.ServletConfidentialityConstraintHandler.handleRequest(ServletConfidentialityConstraintHandler.java:64)
	at io.undertow.security.handlers.AuthenticationMechanismsHandler.handleRequest(AuthenticationMechanismsHandler.java:60)
	at io.undertow.servlet.handlers.security.CachedAuthenticatedSessionHandler.handleRequest(CachedAuthenticatedSessionHandler.java:77)
	at io.undertow.security.handlers.AbstractSecurityContextAssociationHandler.handleRequest(AbstractSecurityContextAssociationHandler.java:43)
	at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
	at io.undertow.servlet.handlers.SendErrorPageHandler.handleRequest(SendErrorPageHandler.java:52)
	at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
	at io.undertow.servlet.handlers.ServletInitialHandler.handleFirstRequest(ServletInitialHandler.java:275)
	at io.undertow.servlet.handlers.ServletInitialHandler.access$100(ServletInitialHandler.java:79)
	at io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:134)
	at io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:131)
	at io.undertow.servlet.core.ServletRequestContextThreadSetupAction$1.call(ServletRequestContextThreadSetupAction.java:48)
	at io.undertow.servlet.core.ContextClassLoaderSetupAction$1.call(ContextClassLoaderSetupAction.java:43)
	at io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:255)
	at io.undertow.servlet.handlers.ServletInitialHandler.access$000(ServletInitialHandler.java:79)
	at io.undertow.servlet.handlers.ServletInitialHandler$1$1.run(ServletInitialHandler.java:106)
	at java.security.AccessController.doPrivileged(Native Method)
	at io.undertow.servlet.handlers.ServletInitialHandler$1.handleRequest(ServletInitialHandler.java:103)
	at io.undertow.server.Connectors.executeRootHandler(Connectors.java:393)
	at io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:852)
	at org.jboss.threads.ContextClassLoaderSavingRunnable.run(ContextClassLoaderSavingRunnable.java:35)
	at org.jboss.threads.EnhancedQueueExecutor.safeRun(EnhancedQueueExecutor.java:2019)
	at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.doRunTask(EnhancedQueueExecutor.java:1558)
	at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1449)
	at org.xnio.XnioWorker$WorkerThreadFactory$1$1.run(XnioWorker.java:1282)
	at java.lang.Thread.run(Thread.java:748)

What you expected to happen

期望：能像 Flink sql-client 终端中一样正常使用 SQL-Hints(STATE_TTL)。

How to reproduce

重现步骤：
1、在 Dinky 的数据开发界面中，创建一个 FlinkSQL 作业。
2、编辑作业内容，写入以下 SQL：

CREATE TEMPORARY VIEW IF NOT EXISTS view_tmp
  COMMENT '视图' AS
    WITH cte0 AS (
        SELECT  /*+ STATE_TTL('tmp0' = '90s') */ -- 这一行不能成功通过 Dinky 的 SQL 检查
                user_id,
                TUMBLE_START(evt_time, INTERVAL '60' SECOND) AS win_start,
                TUMBLE_ROWTIME(evt_time, INTERVAL '60' SECOND) AS win_end,

                MIN(evt_ts) AS evt_ts_min,
                MAX(evt_ts) AS evt_ts_max

        FROM ${DEF_LAKEHOUSE_DB}.user_data /*+ OPTIONS(
            'scan.parallelism' = '1',
            'consumer-id' = '123456789-00',
            'consumer.mode' = 'at-least-once',
            'scan.mode' = 'latest-full',
            'read.batch-size' = '100000'
        ) */ AS tmp0
        GROUP BY TUMBLE(evt_time, INTERVAL '60' SECOND), user_id
    )
    SELECT  CAST(win_end AS DATE) AS date_part,
           user_id, win_start, win_end, (evt_ts_max - evt_ts_min) AS duration
    FROM cte0;

3、点击 Dinky 的检查按钮，会发现报错信息。

Anything else

No response

Version

1.2.1

Are you willing to submit PR?

  • Yes I am willing to submit a PR!

Code of Conduct
@adu-shzz adu-shzz added Bug Something isn't working Waiting for reply Waiting for reply labels Feb 25, 2025
@github-actions GitHub Actions
Copy link

Hello @adu-shzz, this issue is about web, so I assign it to @Zzm0809. If you have any questions, you can comment and reply.

你好 @adu-shzz, 这个 issue 是关于 web 的，所以我把它分配给了 @Zzm0809。如有任何问题，可以评论回复。

@github-actions github-actions bot changed the title [Bug] [数据开发 - 项目] Dinky-v1.20-1.2.1 的 SQL 作业，不支持 Flink-1.20 的 SQL-Hints(STATE_TTL) [Bug] [Data Development - Project] SQL jobs for Dinky-v1.20-1.2.1 do not support SQL-Hints for Flink-1.20(STATE_TTL) Feb 25, 2025
@aiwenmo aiwenmo removed the Waiting for reply Waiting for reply label Feb 27, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@Zzm0809 Zzm0809

Labels
Bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@aiwenmo @adu-shzz @Zzm0809