Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[FM v0.14.0]--Add & update some function and connector configs #259

Merged
merged 5 commits into from
May 29, 2023
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion docs/function-mesh/function-mesh-overview.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@ This document gives a brief introduction into Function Mesh.

## What is Function Mesh

Function Mesh is a Kubernetes operator that enables users to run Pulsar Functions and Pulsar connectors natively on Kubernetes, unlocking the full power of Kubernetes features and resources.
Function Mesh is a Kubernetes operator that enables users to run Pulsar Functions and Pulsar connectors natively on Kubernetes, unlocking the full power of Kubernetes' features and resources.

By providing a serverless framework that enables users to organize a collection of Pulsar Functions and connectors, Function Mesh simplifies the process of creating complex streaming jobs. Function Mesh is a valuable tool for users who are seeking cloud-native serverless streaming solutions. Key benefits include:

Expand Down
2 changes: 1 addition & 1 deletion docs/overview.md
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,7 @@ At the current stage, Function Mesh has the following components:
- Be easily deployed directly on Kubernetes clusters, including [Minikube](https://github.com/kubernetes/minikube) and [Kind](https://kind.sigs.k8s.io/docs/user/quick-start/), without special dependencies.
- Use [CustomResourceDefinitions (CRD)](https://kubernetes.io/docs/concepts/extend-kubernetes/api-extension/custom-resources/) to define Functions, Source, Sink, and Mesh. Using CRD makes Function Mesh naturally integrate with the Kubernetes ecosystem.
- Integrate with Kubernetes secrets seamlessly to read secrets directly. This would help improve the overall security for the Pulsar Functions.
- Leverage the Kubernetess auto-scaler to auto-scale instances for functions based on the CPU usage. In future, Function Mesh will support auto-scaling based on the backlog.
- Leverage the Kubernetes's auto-scaler to auto-scale instances for functions based on the CPU usage. In future, Function Mesh will support auto-scaling based on the backlog.
- Utilize the full power scheduling capability provided by Kubernetes. Therefore, users do not need to write any customized codes to communicate with the Kubernetes API server.
- Allow one function to talk to multiple different Pulsar clusters, which are defined as config maps.
- Support function registry for function package management. We are going to introduce the Pulsar registry in Pulsar 2.8.0 for function package management. Then, the function package can be reused by different functions.
Expand Down
2 changes: 1 addition & 1 deletion docs/overview/overview.md
Original file line number Diff line number Diff line change
Expand Up @@ -115,7 +115,7 @@ Figure 7. The Function Mesh architecture
- Be easily deployed directly on Kubernetes clusters, including [Minikube](https://github.com/kubernetes/minikube) and [Kind](https://kind.sigs.k8s.io/docs/user/quick-start/), without special dependencies.
- Use [CustomResourceDefinitions (CRD)](https://kubernetes.io/docs/concepts/extend-kubernetes/api-extension/custom-resources/) to define Functions, source, sink, and Mesh. Using CRD makes Function Mesh naturally integrate with the Kubernetes ecosystem.
- Integrate with Kubernetes secrets seamlessly to read secrets directly. This helps improve the overall security for the Pulsar Functions.
- Leverage the Kubernetess auto-scaler to auto-scale instances for functions based on the CPU and memory usage.
- Leverage the Kubernetes's auto-scaler to auto-scale instances for functions based on the CPU and memory usage.
- Utilize the full power scheduling capability provided by Kubernetes. Therefore, you do not need to write any customized codes to communicate with the Kubernetes API server.
- Allow one function to talk to multiple different Pulsar clusters, which are defined as config maps.
- Support the function registry for function package management. The function package can be reused by different functions.
Expand Down
6 changes: 3 additions & 3 deletions docs/reference/crd-config/function-crd.md
Original file line number Diff line number Diff line change
Expand Up @@ -27,9 +27,9 @@ This table lists Pulsar Function configurations.
| `deadLetterTopic` | The topic where all messages that were not processed successfully are sent. This parameter is not supported in Python Functions. |
| `funcConfig` | Pulsar Functions configurations in YAML format. |
| `logTopic` | The topic to which the logs of a Pulsar Function are produced. |
| `autoAck` | Whether or not the framework acknowledges messages automatically. This field is required. You can set it to `true` or `false`.|
| `autoAck` (Deprecated) | Whether or not the framework acknowledges messages automatically. This field is required. You can set it to `true` or `false`.|
| `maxMessageRetry` | How many times to process a message before giving up. |
| `processingGuarantee` | The processing guarantees (delivery semantics) applied to the function. Available values: `atleast_once`, `atmost_once`, `effectively_once`. When you set `ProcessingGuarantees` to `effectively_once`, the runtime will set the subscription type to `FAILOVER`. By default, the subscription type is set to `SHARED`.|
| `processingGuarantee` | The processing guarantees (delivery semantics) applied to the function. Available values: `atleast_once`, `atmost_once`, `effectively_once`, and `manual`. When you set `ProcessingGuarantees` to `effectively_once`, the runtime will set the subscription type to `FAILOVER`. By default, the subscription type is set to `SHARED`. The `manual` option is only available for the runner image v2.11.0 or above.|
| `forwardSourceMessageProperty` | Configure whether to pass message properties to a target topic. |
| `retainOrdering` | The function consumes and processes messages in order. When you set `retainOrdering`, the runtime will set the subscription type to `FAILOVER`. By default, the subscription type is set to `SHARED`. |
| `retainKeyOrdering`| Configure whether to retain the key order of messages. When you set `retainKeyOrdering`, the runtime will set the subscription type to `KEY_SHARED`. By default, the subscription type is set to `SHARED`. |
Expand Down Expand Up @@ -221,7 +221,7 @@ The output topics of a Pulsar Function. This table lists options available for t
| `topics` | The output topic of a Pulsar Function (If none is specified, no output is written). |
| `sinkSerdeClassName` | The map of output topics to SerDe class names (as a JSON string). |
| `sinkSchemaType` | The built-in schema type or custom schema class name to be used for messages sent by the function.|
| `producerConf` | The producer specifications. Available options: < br />- `maxPendingMessages`: the maximum number of pending messages. <br />- `maxPendingMessagesAcrossPartitions`: the maximum number of pending messages across partitions. <br />- `useThreadLocalProducers`: configure whether the producer uses a thread. <br />- `cryptoConfig`: cryptography configurations of the producer. <br />- `batchBuilder`: support key-based batcher.
| `producerConf` | The producer specifications. Available options: <br />- `batchBuilder`: The type of batch construction method. Support the key-based batcher. <br />- `compressionType`: the message data compression type used by a producer. Available options are `LZ4`, `NONE`, `ZLIB`, `ZSTD`, and `SNAPPY`. By default, it is set to `LZ4`. This option is only available for the runner image v3.0.0 or above. <br />- `cryptoConfig`: the cryptography configurations of the producer. <br />- `maxPendingMessages`: the maximum number of pending messages. <br />- `maxPendingMessagesAcrossPartitions`: the maximum number of pending messages across all partitions. <br />- `useThreadLocalProducers`: configure whether the producer uses a thread. |
| `customSchemaSinks` | The map of output topics to Schema class names (as a JSON string). |

## Resources
Expand Down
8 changes: 4 additions & 4 deletions docs/reference/crd-config/sink-crd-config.md
Original file line number Diff line number Diff line change
Expand Up @@ -25,16 +25,16 @@ This table lists sink configurations.
| `sinkConfig` | The sink connector configurations in YAML format.|
| `timeout` | The message timeout in milliseconds. |
| `negativeAckRedeliveryDelayMs`| The number of redelivered messages due to negative acknowledgement. |
| `autoAck` | Whether or not the framework acknowledges messages automatically. This field is required. You can set it to `true` or `false`.|
| `autoAck` (Deprecated) | Whether or not the framework acknowledges messages automatically. This field is required. You can set it to `true` or `false`.|
| `maxMessageRetry` | How many times to process a message before giving up. |
| `processingGuarantee` | The processing guarantees (delivery semantics) applied to the sink connector. Available values: `atleast_once`, `atmost_once`, `effectively_once`. When you set `ProcessingGuarantees` to `effectively_once`, the runtime will set the subscription type to `FAILOVER`. By default, the subscription type is set to `SHARED`.|
| `processingGuarantee` | The processing guarantees (delivery semantics) applied to the sink connector. Available values: `atleast_once`, `atmost_once`, `effectively_once`, and `manual`. When you set `ProcessingGuarantees` to `effectively_once`, the runtime will set the subscription type to `FAILOVER`. By default, the subscription type is set to `SHARED`. The `manual` option is only available for the runner image v2.11.0 or above.|
| `retainOrdering` | The sink connector consumes and processes messages in order. When you set `retainOrdering`, the runtime will set the subscription type to `FAILOVER`. By default, the subscription type is set to `SHARED`. |
| `retainKeyOrdering`| Configure whether to retain the key order of messages. When you set `retainKeyOrdering`, the runtime will set the subscription type to `KEY_SHARED`. By default, the subscription type is set to `SHARED`. |
| `deadLetterTopic` | The topic where all messages that were not processed successfully are sent. |
| `subscriptionName` | The subscription name of the sink connector if you want a specific subscription-name for the input-topic consumer. |
| `subscriptionName` | The subscription name of the sink connector if you want a specific subscription name for the input-topic consumer. |
| `cleanupSubscription` | Configure whether to clean up subscriptions. |
| `subscriptionPosition` | The subscription position. |
| `pulsar` | The configurations about the Pulsar cluster. For details, see [messaging](#messaging). |
| `pulsar` | The configurations of the Pulsar cluster. For details, see [messaging](#messaging). |

## Annotations

Expand Down
6 changes: 3 additions & 3 deletions docs/reference/crd-config/source-crd-config.md
Original file line number Diff line number Diff line change
Expand Up @@ -23,10 +23,10 @@ This table lists source configurations.
| `cleanupImage` | The image that is used to delete the subscriptions created or used by a source when the source is deleted. If no clean-up image is set, the runner image will be used. |
| `maxReplicas`| The maximum number of instances that you want to run for this source connector. When the value of the `maxReplicas` parameter is greater than the value of `replicas`, it indicates that the source controller automatically scales the source connector based on the CPU usage. By default, `maxReplicas` is set to 0, which indicates that auto-scaling is disabled. |
| `sourceConfig` | The source connector configurations in YAML format. |
| `processingGuarantee` | The processing guarantees (delivery semantics) applied to the source connector. Available values: `atleast_once`, `atmost_once`, `effectively_once`.|
| `processingGuarantee` | The processing guarantees (delivery semantics) applied to the source connector. Available values: `atleast_once`, `atmost_once`, `effectively_once`, and `manual`. The `manual` option is only available for the runner image v2.11.0 or above.|
| `forwardSourceMessageProperty` | Configure whether to pass message properties to a target topic. |
| `batchSourceConfig` | The batch source configurations in YAML format. You can configure the following properties. <br/> - `discoveryTriggererClassName`: the class that is used for triggering the discovery process. <br/> - `discoveryTriggererConfig`: the configurations that are required for initiating the discovery Triggerer. |
| `pulsar` | The configurations about the Pulsar cluster. For details, see [messaging](#messaging). |
| `pulsar` | The configurations of the Pulsar cluster. For details, see [messaging](#messaging). |

## Annotations

Expand Down Expand Up @@ -176,7 +176,7 @@ The output topics of a Pulsar Function. This table lists options available for t
| `topics` | The output topic of a Pulsar Function (If none is specified, no output is written). |
| `sinkSerdeClassName` | The map of output topics to SerDe class names (as a JSON string). |
| `sinkSchemaType` | The built-in schema type or custom schema class name to be used for messages sent by the function.|
| `producerConf` | The producer specifications. Available options: <br />- `maxPendingMessages`: the maximum number of pending messages. <br />- `maxPendingMessagesAcrossPartitions`: the maximum number of pending messages across partitions. <br />- `useThreadLocalProducers`: configure whether the producer uses a thread. <br />- `cryptoConfig`: cryptography configurations of the producer. <br />- `batchBuilder`: support key-based batcher.
| `producerConf` | The producer specifications. Available options: <br />- `batchBuilder`: The type of batch construction method. Support the key-based batcher. <br />- `compressionType`: the message data compression type used by a producer. Available options are `LZ4`, `NONE`, `ZLIB`, `ZSTD`, and `SNAPPY`. By default, it is set to `LZ4`. This option is only available for the runner image v3.0.0 or above. <br />- `cryptoConfig`: the cryptography configurations of the producer. <br />- `maxPendingMessages`: the maximum number of pending messages. <br />- `maxPendingMessagesAcrossPartitions`: the maximum number of pending messages across all partitions. <br />- `useThreadLocalProducers`: configure whether the producer uses a thread. |
| `customSchemaSinks` | The map of output topics to Schema class names (as a JSON string). |

## Resources
Expand Down
2 changes: 1 addition & 1 deletion src/pages/index.js
Original file line number Diff line number Diff line change
Expand Up @@ -121,7 +121,7 @@ const twoSections = [
<ul>
<li>
A Kubernetes Operator to run Pulsar Functions and I/O connectors
natively on Kubernetes, unlocking the full power of Kubernetes
natively on Kubernetes, unlocking the full power of Kubernetes'
application deployment, scaling and management.
</li>
<li>
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -26,7 +26,7 @@ This table lists sink configurations.
| `processingGuarantee` | The processing guarantees (delivery semantics) applied to the sink connector. Available values: `atleast_once`, `atmost_once`, `effectively_once`.|
| `retainOrdering` | The sink connector consumes and processes messages in order. |
| `deadLetterTopic` | The topic where all messages that were not processed successfully are sent. |
| `subscriptionName` | The subscription name of the sink connector if you want a specific subscription-name for the input-topic consumer. |
| `subscriptionName` | The subscription name of the sink connector if you want a specific subscription name for the input-topic consumer. |
| `cleanupSubscription` | Configure whether to clean up subscriptions. |
| `subscriptionPosition` | The subscription position. |

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@ This document gives a brief introduction into Function Mesh.

## What is Function Mesh

Function Mesh is a Kubernetes operator that enables users to run Pulsar Functions and Pulsar connectors natively on Kubernetes, unlocking the full power of Kubernetes features and resources.
Function Mesh is a Kubernetes operator that enables users to run Pulsar Functions and Pulsar connectors natively on Kubernetes, unlocking the full power of Kubernetes' features and resources.

By providing a serverless framework that enables users to organize a collection of Pulsar Functions and connectors, Function Mesh simplifies the process of creating complex streaming jobs. Function Mesh is a valuable tool for users who are seeking cloud-native serverless streaming solutions. Key benefits include:

Expand Down
2 changes: 1 addition & 1 deletion versioned_docs/version-0.1.10/functions/function-crd.md
Original file line number Diff line number Diff line change
Expand Up @@ -29,7 +29,7 @@ This table lists Pulsar Function configurations.
| `forwardSourceMessageProperty` | Configure whether to pass message properties to a target topic. |
| `retainOrdering` | Function consumes and processes messages in order. |
| `retainKeyOrdering`| Configure whether to retain the key order of messages. |
| `subscriptionName` | Pulsar Functions subscription name if you want a specific subscription-name for the input-topic consumer. |
| `subscriptionName` | Pulsar Functions' subscription name if you want a specific subscription name for the input-topic consumer. |
| `cleanupSubscription` | Configure whether to clean up subscriptions. |
| `subscriptionPosition` | The subscription position. |

Expand Down
2 changes: 1 addition & 1 deletion versioned_docs/version-0.1.10/overview.md
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,7 @@ At the current stage, Function Mesh has the following components:
- Be easily deployed directly on Kubernetes clusters, including [Minikube](https://github.com/kubernetes/minikube) and [Kind](https://kind.sigs.k8s.io/docs/user/quick-start/), without special dependencies.
- Use [CustomResourceDefinitions (CRD)](https://kubernetes.io/docs/concepts/extend-kubernetes/api-extension/custom-resources/) to define Functions, Source, Sink, and Mesh. Using CRD makes Function Mesh naturally integrate with the Kubernetes ecosystem.
- Integrate with Kubernetes secrets seamlessly to read secrets directly. This would help improve the overall security for the Pulsar Functions.
- Leverage the Kubernetess auto-scaler to auto-scale instances for functions based on the CPU usage. In future, Function Mesh will support auto-scaling based on the backlog.
- Leverage the Kubernetes's auto-scaler to auto-scale instances for functions based on the CPU usage. In future, Function Mesh will support auto-scaling based on the backlog.
- Utilize the full power scheduling capability provided by Kubernetes. Therefore, users do not need to write any customized codes to communicate with the Kubernetes API server.
- Allow one function to talk to multiple different Pulsar clusters, which are defined as config maps.
- Support function registry for function package management. We are going to introduce the Pulsar registry in Pulsar 2.8.0 for function package management. Then, the function package can be reused by different functions.
Expand Down
2 changes: 1 addition & 1 deletion versioned_docs/version-0.1.10/overview/overview.md
Original file line number Diff line number Diff line change
Expand Up @@ -115,7 +115,7 @@ Figure 7. The Function Mesh architecture
- Be easily deployed directly on Kubernetes clusters, including [Minikube](https://github.com/kubernetes/minikube) and [Kind](https://kind.sigs.k8s.io/docs/user/quick-start/), without special dependencies.
- Use [CustomResourceDefinitions (CRD)](https://kubernetes.io/docs/concepts/extend-kubernetes/api-extension/custom-resources/) to define Functions, source, sink, and Mesh. Using CRD makes Function Mesh naturally integrate with the Kubernetes ecosystem.
- Integrate with Kubernetes secrets seamlessly to read secrets directly. This would help improve the overall security for the Pulsar Functions.
- Leverage the Kubernetess auto-scaler to auto-scale instances for functions based on the CPU usage. In future, Function Mesh will support auto-scaling based on the backlog.
- Leverage the Kubernetes's auto-scaler to auto-scale instances for functions based on the CPU usage. In future, Function Mesh will support auto-scaling based on the backlog.
- Utilize the full power scheduling capability provided by Kubernetes. Therefore, users do not need to write any customized codes to communicate with the Kubernetes API server.
- Allow one function to talk to multiple different Pulsar clusters, which are defined as config maps.
- Support function registry for function package management. We are going to introduce the Pulsar registry in Pulsar 2.8.0 for function package management. Then, the function package can be reused by different functions.
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -26,7 +26,7 @@ This table lists sink configurations.
| `processingGuarantee` | The processing guarantees (delivery semantics) applied to the sink connector. Available values: `atleast_once`, `atmost_once`, `effectively_once`.|
| `retainOrdering` | The sink connector consumes and processes messages in order. |
| `deadLetterTopic` | The topic where all messages that were not processed successfully are sent. |
| `subscriptionName` | The subscription name of the sink connector if you want a specific subscription-name for the input-topic consumer. |
| `subscriptionName` | The subscription name of the sink connector if you want a specific subscription name for the input-topic consumer. |
| `cleanupSubscription` | Configure whether to clean up subscriptions. |
| `subscriptionPosition` | The subscription position. |

Expand Down
Loading