-
Notifications
You must be signed in to change notification settings - Fork 642
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[ISSUE #4787]integration connector #4790
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Since the demo code is just a general idea, I've only looked at the general idea, so I'll leave some minor review comments for now. I think it should be possible to realize "when start runtime, if there are local Connectors, they will be started together with SPI mechanism, and the communication method will be intra-process communication". I agree with this idea, and look forward to hearing from more people in the community.
But there is a question, should we consider this case: the local Connector has already started by itself?
因为该demo代码只是大致的思路,所以只看了个大概,一些细小的review意见就暂时不提了。我觉得该思路应该可以实现“启动runtime时,如果本地有Connectors会利用SPI机制一并启动,并且通信方式将是进程内通信”。我赞同该思路,同时期待社区更多人的看法。
但是有个疑问,要不要考虑这种情况:本地的Connector已经自己启动过了?
Sorry, I have a bad cold these days, so I think it may be necessary to use a form like file lock. After connector is activated, the Worker will generate a file named Connector-instance-mian at a fixed location. When the connector is booted either integrally or separately, it will get an error that cannot be started if the file exists (init timing), and delete the file when the connector stops (shutdown timing). The only drawback is the need to manually delete this file if the connector stops unexpectedly (such as when the process is killed). |
I see in the issue that you have raised a lot of queries about the community's planning for this part of the demand, so let's leave this minor detail aside for now. I'll keep an eye on these queries as well #4787 (comment) . 我看到issue中您提了很多关于社区对这部分需求的规划的疑问,所以这个次要的细节问题暂不考虑吧。对于这些疑问(#4787 (comment) ),我也会继续关注。 |
Fixes #4787
########
UnMergeable!!! its demo ,不能合并这只是用于讨论
########
Motivation
Integrate connector with minimal modification
Very simple integration using connecter
Modifications
Integrating 'connecter' means that 'connecter' has to have another running mode, and integrating into 'runtime' has some problems to solve
集成
connecter
,意味着connecter
要有另一种运行模式,集成到runtime
中有一些问题需要解决The first problem that needs to be solved is that the main method is not called, but
connecter
is a plug-in module, and it is troublesome to hard-code main once for each plug-in. So the method I think of is that all theconnecter
module's startup class inherits aEmbeddableConnector
interface, and uses theEventmesh-SPI
to load, so that the changes are broughtEmbeddableConnector
All startup classes in
Connector
(in this case,FileConnectServer
)All files referred to by
SPI
inConnector
首先需要解决的问题就是
main
方法没有被调用,但是connecter
是插件化的模块,为每一个插件都硬编码一次main
这样很麻烦,所以我想到的方法是所有connecter
模块的启动类都继承一个EmbeddableConnector
接口,使用Eventmesh
的SPI
进行加载,这样带来的修改就有EmbeddableConnector
所有
Connector
中的启动类(这个例子中的启动类是FileConnectServer
)所有
Connector
中SPI
涉及的文件Second, the original interaction between
connector
andruntime
usestcp protocol,
and the purpose of integration is to reduce serialization and communication consumption. Therefore, interaction needs to be realized in other ways. The method I think is to use the form of service. connector itself only has two roles,sink
andsource
, which are publish and subscribe in terms of function, so this service needs to use publish and subscribe functions. Meanwhile, in order to preserve the original structure and design to the greatest extent, the connector uses atcp adapter
(the adapter does not implement network communication in the middle). Instead, the service is called.) So the design changes areIInnerPubSubService Indicates the interface of the service
InnerPubSubService implementation
IntegrationCloudEventTCPClientAdapter TCP adapter
Application must have a static variable holding the service
The SinkWorker passes the service to the tcp adapter
SourceWorker transfers the service tcp adapter
其次需要解决的就是原有
connector
和runtime
之间的交互使用的是tcp协议
而集成的目的就是减少序列化和通信消耗,所以需要通过其他方式对交互进行实现,我想到的方法是使用服务的形式,connector
本身其实只有两种角色,sink
和source
,从功能来说就是发布和订阅,那么这个服务就需要用发布和订阅的功能即可,同时为了最大程度保留原有结构和设计,使用tcp适配器(适配器中间不在实现网络通信,而是对服务进行调用)所以设计的修改有IInnerPubSubService 服务的接口
InnerPubSubService 服务的实现
IntegrationCloudEventTCPClientAdapter tcp适配器
Application 要有静态变量持有服务
SinkWorker 传递服务给tcp适配器
SourceWorker 传递服务tcp适配器
Unfinished changes:
The implementation of InnerPubSubService
未完成的修改:
InnerPubSubService
的具体实现达成的效果:
只要
gradle
引用了对应的connector
插件就会自动处理 内嵌connector
相关逻辑Results achieved:
The built-in
connector
logic is automatically handled whenevergradle
references the correspondingconnector
plug-in