Introduce TexeraWorkflowCompilingService for Workflow Compilation #2796
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
bobbai00
force-pushed
the
jiadong-separate-workflow-compiler
branch
from
2bf266e to
5738e9d
Compare
Yicong-Huang
requested changes
Aug 21, 2024
core/amber/src/main/scala/edu/uci/ics/texera/web/TexeraCompilerWebApplication.scala
Outdated
Show resolved
Hide resolved
core/amber/src/main/scala/edu/uci/ics/texera/web/TexeraCompilerWebApplication.scala
Outdated
Show resolved
Hide resolved
core/amber/src/main/scala/edu/uci/ics/texera/web/TexeraCompilerWebApplication.scala
Outdated
Show resolved
Hide resolved
core/amber/src/main/scala/edu/uci/ics/texera/web/TexeraCompilerWebApplication.scala
Outdated
Show resolved
Hide resolved
core/amber/src/main/scala/edu/uci/ics/texera/web/TexeraCompilerWebApplication.scala
Outdated
Show resolved
Hide resolved
core/amber/src/main/scala/edu/uci/ics/texera/web/TexeraCompilerWebApplication.scala
Outdated
Show resolved
Hide resolved
core/amber/src/main/scala/edu/uci/ics/texera/web/TexeraCompilerWebApplication.scala
Outdated
Show resolved
Hide resolved
core/amber/src/main/scala/edu/uci/ics/texera/web/resource/WorkflowCompilationResource.scala
Outdated
Show resolved
Hide resolved
bobbai00
force-pushed
the
jiadong-separate-workflow-compiler
branch
from
5738e9d to
a9f102d
Compare
bobbai00
changed the title
bobbai00
force-pushed
the
jiadong-separate-workflow-compiler
branch
from
d34d7ca to
730561a
Compare
|
@bobbai00 please reply to previous comments so I know what's changed or to expect. |
Yicong-Huang
approved these changes
Sep 4, 2024
.../amber/src/main/scala/edu/uci/ics/amber/engine/architecture/deploysemantics/PhysicalOp.scala
Outdated
Show resolved
Hide resolved
.../amber/src/main/scala/edu/uci/ics/amber/engine/architecture/deploysemantics/PhysicalOp.scala
Outdated
Show resolved
Hide resolved
core/amber/src/main/scala/edu/uci/ics/texera/web/TexeraWorkflowCompilingService.scala
Outdated
Show resolved
Hide resolved
core/amber/src/main/scala/edu/uci/ics/texera/web/TexeraWorkflowCompilingService.scala
Outdated
Show resolved
Hide resolved
core/amber/src/main/scala/edu/uci/ics/texera/web/resource/WorkflowCompilationResource.scala
Show resolved
Hide resolved
core/amber/src/main/scala/edu/uci/ics/texera/web/TexeraCompilerWebApplication.scala
Outdated
Show resolved
Hide resolved
core/amber/src/main/scala/edu/uci/ics/texera/web/TexeraCompilerWebApplication.scala
Outdated
Show resolved
Hide resolved
bobbai00
force-pushed
the
jiadong-separate-workflow-compiler
branch
from
13c567d to
b777b06
Compare
This reverts commit c31a0ff.
bobbai00
force-pushed
the
jiadong-separate-workflow-compiler
branch
from
75c1fd5 to
f15a724
Compare
PurelyBlank
pushed a commit
that referenced
this pull request
Dec 4, 2024
) This PR adds a standalone web service called `TexeraWorkflowCompilingService`. ## The purpose of having a standalone compilation service Separate the Texera Workflow Compilation with Workflow Execution. ## The configuration of this service ### 1. Web Server Config All the configuration items are put in `core/amber/src/main/resources/texera-compiler-web-config.yml`. By default, this server runs at port 9090. ### 2. Application Config This service shares the same application configuration file, i.e. `core/amber/src/main/resources/application.conf`. To ensure that JWT token can be recognized by both `TexeraWebApplication` and `TexeraWorkflowCompilingService`, `user-sys.jwt.256-bit-secret` must NOT be set as random, instead it must be set as a fixed 256-bit string. ## The Endpoint provided by `TexeraWorkflowCompilingService` A single endpoint is provided by this web application: ``` POST /api/texera/compilation/{wid} ``` - request format: the serialized workflow json string - response: - physicalPlan: Option[PhysicalPlan] // the physical plan of the workflow, if None, the compilation is failed - operatorInputSchemas: Map[String, List[Option[List[Attribute]]]]. // from `OpId` to list of `Schema`(List of attributes) indexed by port id - operatorErrors: Map[String, String] // from `OpId` to error of this operator ## Future TODO after this PR 1. Use this web service when GUI is doing the workflow editing, maintain the physical plan in GUI. 2. Change all the deployment scripts, including `terminate-daemon.sh`, `deploy-daemon.sh` and `Dockerfile` to launch & terminate this web service 3. Clean up the code in `core/amber/src/main/scala/edu/uci/ics/texera/workflow/common/workflow/WorkflowCompiler.scala` and `core/amber/src/main/scala/edu/uci/ics/texera/web/service/WorkflowExecutionService.scala`
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR adds a standalone web service called
TexeraWorkflowCompilingService.The purpose of having a standalone compilation service
Separate the Texera Workflow Compilation with Workflow Execution.
The configuration of this service
1. Web Server Config
All the configuration items are put in
core/amber/src/main/resources/texera-compiler-web-config.yml. By default, this server runs at port 9090.2. Application Config
This service shares the same application configuration file, i.e.
core/amber/src/main/resources/application.conf. To ensure that JWT token can be recognized by bothTexeraWebApplicationandTexeraWorkflowCompilingService,user-sys.jwt.256-bit-secretmust NOT be set as random, instead it must be set as a fixed 256-bit string.The Endpoint provided by
TexeraWorkflowCompilingServiceA single endpoint is provided by this web application:
OpIdto list ofSchema(List of attributes) indexed by port idOpIdto error of this operatorFuture TODO after this PR
terminate-daemon.sh,deploy-daemon.shandDockerfileto launch & terminate this web servicecore/amber/src/main/scala/edu/uci/ics/texera/workflow/common/workflow/WorkflowCompiler.scalaandcore/amber/src/main/scala/edu/uci/ics/texera/web/service/WorkflowExecutionService.scala