Merge in recent changes (#1)

* Update README.md * Fix CVEs in 3.16. (conductor-oss#46) * Upgraded ES7 to 7.17.16, alpine to 3.19 * Update ElasticSearch to 7.17.16, some sdk tests are failing. * Server would not run, StackOverflow - 54742540 had this handy fix. * Use the new image in test container. * Update Spring Boot to 3.2.1. One failing test in end to end. * Handle the change in exceptions from Spring Framework. * Update AWS SDK for CVE fix. * Replace generic import with specific class. * Removed dependencies.lock file from projects. No longer used. * prevents orkes-queues from pulling in old version of condcutor (conductor-oss#55) * Documentation link update in Readme (conductor-oss#54) * Update README.md * Revert "Update README.md" This reverts commit fe16da9. * Update README.md * Change dependency group,version (conductor-oss#48) * Remove unnecessary Autowired from Constructors (conductor-oss#36) * Remove unnecessary Autowired from Constructors * Add @Autowired back to property classes --------- Co-authored-by: tiffany jernigan <[email protected]> * upgrade from gradle 7.6.2 to gradle 8.5 (conductor-oss#28) * Fix redundant taskDef in DB (conductor-oss#17) * Add imports file that define auto configure classes supported by spring-boot 2.7 (conductor-oss#60) * update the workflow template * fixes * Update README.md * Left menu URL fix for docs site (conductor-oss#77) Update mkdocs.yml by @RizaFarheen * Update source.md (conductor-oss#75) changed excuted -> executed * Update README.md (conductor-oss#72) againt -> against * Optimise the postgres queries for popping messages (conductor-oss#76) * Feature - use environment variable to set ES6 support mode for Nashorn script evaluator (conductor-oss#88) Use environment variable CONDUCTOR_NASHORN_ES6_ENABLED=true to set ES6 support mode for Nashorn script evaluator. * Fix an issue with the postgres indices as they weren't being used properly (conductor-oss#92) * Upgrade the postgres lib to prevent CVE-2024-1597 (conductor-oss#94) * Enable finer-grained control over the postgres indexing and searching (conductor-oss#93) * Enable the searching of the JSON field in Postgres to be configured * Enable the indexing of tasks to be enabled / disabled * Implement removeWorkflow for postgres index * Enable the migration to run without always recreating an index (conductor-oss#98) * Feature: Use TaskStatusListener.onTaskInProgress() (conductor-oss#58) * apply task-task-status-listener * fix test error * prevent NPE * Fix SpringBoot imports file location error (conductor-oss#89) * Implementation of Postgres PollDataDAO with read and write caching (conductor-oss#82) * Initial implementation of an experimental queue optimisation using Postgres LISTEN/NOTIFY (conductor-oss#87) * Initial implementation of an experimental queue optimisation using Postgres LISTEN/NOTIFY * Use Optional instead of returning null for getSize() * Update connection method for queue listener to make it more thread safe * Only apply the queue notify migrations if configured * Update the postgres docs with all configurable properties (conductor-oss#99) * Fix ElasticSearch health check failure when ES sets password (conductor-oss#59) * Add option to only index workflows on Postgres when their status changes (conductor-oss#83) * Implement postgres lock DAO * add postgres locking mention in docs * rename locking db migration * refactor query - the pg now() function is already UTC * fix test * fix pg query * simplify query * Adding redis username support for clusters (conductor-oss#105) Co-authored-by: Anup Chatterjee <[email protected]> * fix tests * timeunit ms * Update README.md * Workflow and task status notifier Co-authored-by: ToastedTaco <[email protected]> Co-authored-by: BabyBlue0214 <[email protected]> Co-authored-by: Elijah Spicer <[email protected]> * Fix PostgresIndexQueryBuilder null pointer exception (conductor-oss#116) * Throw IllegalArgumentException when query string does not match regex * Unit test for exception * Update README.md (conductor-oss#113) * Add a migration to relax the size constraint on task_index columns (conductor-oss#114) * Fix of exception handler trying to write response to a disconnected client (conductor-oss#109) Why: On Conductor server side, an exception gets raised `Failure in @ExceptionHandler com.netflix.conductor.rest.controllers.ApplicationExceptionMapper#handleAll(HttpServletRequest, Throwable)`. Cause is attempting to write while the client aborted the connection `org.apache.catalina.connector.ClientAbortException: java.io.IOException: Broken pipe` What: Added a ClientAbortException handler. Testing done: Local run. Not trivial to emulate the issue in a test. Such a test may rely on timing out the client before server responded. Signed-off-by: Iva Avramova <[email protected]> * Workflow bulk deletion API and action module options (conductor-oss#110) * Initial commit to add bulk workflow deletion * Added an API endpoint for bulk deletion * Add archiveworkflow param * Added bulk delete action to BulkActionModule * Added archive option to bulk delete * Addition of terminateRemove() Functions (conductor-oss#9) * Added an API endpoint for bulk deletion * Fixed WorkflowBulkServiceTest, added test cases for deleteWorkflow an… (conductor-oss#10) * Fixed WorkflowBulkServiceTest, added test cases for deleteWorkflow and terminateRemove, and added WorkflowResourceTest test for single terminateRemove * pass spotlessjava --------- Co-authored-by: JeffP <[email protected]> --------- Co-authored-by: jeffp1 <[email protected]> Co-authored-by: Ayush Thengne <[email protected]> Co-authored-by: Jeff P <[email protected]> Co-authored-by: Doe1111 <[email protected]> * revert WorkflowDef changes * Join task fixes (conductor-oss#117) * fix join completing with errors when a task has failed or some tasks are not terminal yet * add more tests * add more comments to join routine * mark join task synchronous * Revert unrelated changes * Status Notifier Properties Cleanup * fix tests * fix formatting * update the tests for the JOIN task fix * Fix the name of the task index migration * Update README.md * Fix of exception handler trying to write response to a disconnected client Why: On Conductor server side, an exception gets raised `Failure in @ExceptionHandler com.netflix.conductor.rest.controllers.ApplicationExceptionMapper#handleAll(HttpServletRequest, Throwable)`. Cause is attempting to write while the client aborted the connection `org.apache.catalina.connector.ClientAbortException: java.io.IOException: Broken pipe`, while the exception was wrapped in `ClosedChannelException`. What: Added a ClosedChannelException handler for such exceptions root caused by ClientAbortException. Testing done: Local run. Not trivial to emulate the issue in a test. Such a test may rely on timing out the client before server responded. Signed-off-by: Iva Avramova <[email protected]> * Fix of exception handler trying to write response to a disconnected client Why: On Conductor server side, an exception gets raised `Failure in @ExceptionHandler com.netflix.conductor.rest.controllers.ApplicationExceptionMapper#handleAll(HttpServletRequest, Throwable)`. Cause is attempting to write while the client aborted the connection `org.apache.catalina.connector.ClientAbortException: java.io.IOException: Broken pipe`, while the exception was nested in `ClosedChannelException`. What: Added a ClosedChannelException handler for exception chain containing ClientAbortException. Testing done: Local run. Not trivial to emulate the issue in a test. Such a test may rely on timing out the client before server responded. Signed-off-by: Iva Avramova <[email protected]> * fix: Fix error with headers in webhook * Added logic to remove stale iteration outputs based on keepLastN parameter * Remove unused code block * Remove unused code block * Fixed formatting issues * Updating DoWhileTask keepLastN iteration endIndex * style: Fix style * added tests; generalized the exception Signed-off-by: Iva Avramova <[email protected]> * added tests; generalized the exception Signed-off-by: Iva Avramova <[email protected]> * Why: Ignoring a particular exception based on its stack trace should be rather an alerting concern. What: Reverted ClientAbortedException specific handler. Added test coverage for the existing exception handler. Testing done: local run Signed-off-by: Iva Avramova <[email protected]> * Update README.md * if status-listener error, task not executed * change the point of cancellation * Wrap PGExecutionDAO createTasks in a single transaction * publish roadmap * add link to the roadmap * Create codacy.yml * Delete .github/workflows/codacy.yml * Bump io.micrometer:micrometer-registry-datadog from 1.12.1 to 1.13.0 Bumps [io.micrometer:micrometer-registry-datadog](https://github.com/micrometer-metrics/micrometer) from 1.12.1 to 1.13.0. - [Release notes](https://github.com/micrometer-metrics/micrometer/releases) - [Commits](micrometer-metrics/micrometer@v1.12.1...v1.13.0) --- updated-dependencies: - dependency-name: io.micrometer:micrometer-registry-datadog dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <[email protected]> * Fix for concurrentExecLimit * Bump com.diffplug.spotless from 5.0.0 to 6.25.0 Bumps com.diffplug.spotless from 5.0.0 to 6.25.0. --- updated-dependencies: - dependency-name: com.diffplug.spotless dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <[email protected]> --------- Signed-off-by: Iva Avramova <[email protected]> Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: Viren Baraiya <[email protected]> Co-authored-by: Dennis Caldwell <[email protected]> Co-authored-by: Brian Hawkins <[email protected]> Co-authored-by: Riza Farheen <[email protected]> Co-authored-by: Hoony <[email protected]> Co-authored-by: tiffany jernigan <[email protected]> Co-authored-by: tiffany jernigan <[email protected]> Co-authored-by: HOA PHAN <[email protected]> Co-authored-by: Richard Meng <[email protected]> Co-authored-by: Young-Zen <[email protected]> Co-authored-by: Kunal Kumar <[email protected]> Co-authored-by: Ikko Eltociear Ashimine <[email protected]> Co-authored-by: Ben Pirt <[email protected]> Co-authored-by: Vasiliy Pankov <[email protected]> Co-authored-by: Boney <[email protected]> Co-authored-by: Bert Verstraete <[email protected]> Co-authored-by: Anup Chatterjee <[email protected]> Co-authored-by: Anup Chatterjee <[email protected]> Co-authored-by: CollinDewey <[email protected]> Co-authored-by: ToastedTaco <[email protected]> Co-authored-by: BabyBlue0214 <[email protected]> Co-authored-by: Elijah Spicer <[email protected]> Co-authored-by: omadaan <[email protected]> Co-authored-by: ivakoleva <[email protected]> Co-authored-by: Olivia Bailey <[email protected]> Co-authored-by: jeffp1 <[email protected]> Co-authored-by: Ayush Thengne <[email protected]> Co-authored-by: Jeff P <[email protected]> Co-authored-by: Doe1111 <[email protected]> Co-authored-by: Juan Cruz <[email protected]> Co-authored-by: Srihari Malagi <[email protected]> Co-authored-by: Srihari K. Malagi <[email protected]> Co-authored-by: Martini <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: manan164 <[email protected]>
codaxy · Jun 11, 2024 · 1426741 · 1426741
1 parent 2e309b2
commit 1426741
Show file tree

Hide file tree

Showing 149 changed files with 4,078 additions and 20,091 deletions.
diff --git a/README.md b/README.md
@@ -1,6 +1,6 @@
 ![Conductor](docs/img/logo.svg)
 
-[![Github release](https://img.shields.io/github/v/release/Netflix/conductor.svg)](https://GitHub.com/Netflix/conductor/releases)
+[![Github release](https://img.shields.io/github/v/release/conductor-oss/conductor.svg)](https://GitHub.com/Netflix/conductor-oss/releases)
 [![License](https://img.shields.io/github/license/conductor-oss/conductor.svg)](http://www.apache.org/licenses/LICENSE-2.0)
 
 
@@ -28,21 +28,22 @@ Please update your forks to point to this repo.  This will ensure your commits a
 git remote set-url origin https://github.com/conductor-oss/conductor
 ```
 > [!IMPORTANT]  
-> **Follow the steps below if you have an active PR againt the Netflix/Conductor repository**
+> **Follow the steps below if you have an active PR against the Netflix/Conductor repository**
 > 1. Fork **this** repository
 > 2. Update your local repository to change the remote to this repository
 > 3. Send a PR against the `main` branch
 
-## Releases
-The latest version is [![Github release](https://img.shields.io/github/v/release/Netflix/conductor.svg)](https://GitHub.com/Netflix/conductor/releases)
+## Conductor OSS Roadmap
+[See the roadmap for the Conductor](ROADMAP.md)
 
-The next scheduled release is in Dec 2023.
+## Releases
+The latest version is [![Github release](https://img.shields.io/github/v/release/conductor-oss/conductor.svg)](https://GitHub.com/conductor-oss/conductor/releases)
 
 ## Resources
-#### [Slack Community](https://join.slack.com/t/orkes-conductor/shared_invite/zt-xyxqyseb-YZ3hwwAgHJH97bsrYRnSZg)
-We have an active [community](https://join.slack.com/t/orkes-conductor/shared_invite/zt-xyxqyseb-YZ3hwwAgHJH97bsrYRnSZg) of Conductor users and contributors on the channel.
-#### [Documentation Site](https://orkes.io/content)
-[Documentation](https://orkes.io/content) and tutorial on how to use Conductor
+#### [Slack Community](https://join.slack.com/t/orkes-conductor/shared_invite/zt-2hmxn0i3n-_W~a9rWMbvMoYmlJo3Y15g)
+We have an active [community](https://join.slack.com/t/orkes-conductor/shared_invite/zt-2hmxn0i3n-_W~a9rWMbvMoYmlJo3Y15g) of Conductor users and contributors on the channel.
+#### [Documentation Site](https://docs.conductor-oss.org/)
+[Documentation](https://docs.conductor-oss.org/) and tutorial on how to use Conductor
 
 [Discussion Forum](https://github.com/conductor-oss/conductor/discussions): Please use the forum for questions and discussing ideas and join the community.
 
@@ -61,28 +62,17 @@ The runnable server is in server/ module.
 Follow the steps below to launch the docker container:
 
 ```shell
-
-# Create volumes for persistent stores
-# Used to create a persistent volume that will preserve the 
-docker volume create postgres
-docker volume create redis
-
-docker run --init -p 8080:8080 -p 1234:5000 --mount source=redis,target=/redis \
---mount source=postgres,target=/pgdata conductoross/conductor-standalone:3.15.0
-```
-
-Navigate to http://localhost:1234 once the container starts to launch UI.
-
-## Docker Containers for production usage
-```shell
-docker pull conductoross/conductor:3.15.0
+docker compose -f docker/docker-compose.yaml up
 ```
+* Navigate to http://localhost:5000 once the container starts to launch UI.
+* APIs are accessible at http://localhost:8080
+* Swagger Docs:http://localhost:8080/swagger-ui/index.html?configUrl=/api-docs/swagger-config#/
 
 
 ## Database Requirements
 
 * The default persistence used is Redis
-* The indexing backend is [Elasticsearch](https://www.elastic.co/) (6.x)
+* The indexing backend is [Elasticsearch](https://www.elastic.co/) (7.x)
 
 ## Other Requirements
 * JDK 17+

diff --git a/ROADMAP.md b/ROADMAP.md
@@ -0,0 +1,75 @@
+# Conductor OSS Roadmap
+
+
+## New Features
+### Type safety for workflow inputs and task input/output through JSON Schema
+
+* Allow type safe workflows and workers with support for JSON schema and protobuf
+* Enable scaffolding code generation for workers through schema for workers using CLI tool
+
+### New System Tasks
+
+* Database task to work with relational & no-sql databases
+* Polling support for HTTP task
+* Operators
+ * * For..Each with parallel and sequential execution
+ * * Improved While loop
+ * *  Try..Catch for improved error handling at the task level
+
+### LLM Integrations
+Conductor is a perfect platform to build your next LLM powered application or incorporating genAI into your applications.
+Enable system tasks for LLM integrations that lets you work with various language models for:
+1. Text completion
+2. Chat completion with memory
+3. Embedding generation
+
+### CLI for Conductor
+Allow developers to manage their conductor instance via CLI.
+
+* Manage metadata
+* Query and manage workflow executions (terminate, pause, resume, retry)
+* Start | Stop manage conductor server
+
+### Support Python as a scripting language for INLINE task
+Extend usability of Conductor by allowing lightweight python code as INLINE tasks.
+
+### New APIs for workflow state management
+
+* Synchronous execution of workflows
+* update workflow variables
+* Update tasks synchronously
+
+## SDKs
+
+* Rust
+* Kotlin
+* C++
+* Ruby
+* Swift
+* Flutter / Dart 
+* PHP
+
+### Worker metrics on server
+Expose an endpoint on the server that can be used by workers to publish worker specific metrics.
+This will allow monitoring metrics for all the workers in a distributed system across the entire system. 
+
+## Testing
+Infrastructure to make workflows easier to test and debug right from the UI and IDE.
+
+### Workflow Debugger
+
+* Ability to debug your workflows during development just like you would do when you write code
+* All functionality of a debugger
+* Breakpoints add/remove
+* Step to next
+* Drop to a certain task that was already executed. (going back in time)
+* Ability to inspect, modify, add input / output parameters
+* Watch Windows to see values of interesting &nbsp;parameters during execution
+* Attaching to a certain WF execution
+* Remote Task debugging (with SDK Support).. Enable step by step execution in a task worker from the server
+
+## Maintenance
+
+1. Deprecate support for Elasticsearch 6
+2. Update support for newer versions of Elasticsearch
+2. Improve/Fix JOIN task performance (less about making it performant and more about just fixing the usability) &nbsp;- Done
diff --git a/amqp/dependencies.lock b/amqp/dependencies.lock