Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[FLINK-37522][docs] Add description for the config about file system connection limit #26326

Open
wants to merge 1 commit into
base: master
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 4 additions & 4 deletions docs/content.zh/docs/deployment/filesystems/common.md
Original file line number Diff line number Diff line change
Expand Up @@ -50,9 +50,9 @@ fs.default-scheme: <default-fs>
要限制文件系统的连接数,可将下列配置添加至 Flink 配置中。设置限制的文件系统由其 scheme 指定:

```yaml
fs.<scheme>.limit.total: (数量,0/-1 表示无限制)
fs.<scheme>.limit.input: (数量,0/-1 表示无限制)
fs.<scheme>.limit.output: (数量,0/-1 表示无限制)
fs.<scheme>.limit.total: (数量,0/-1 表示无连接限制)
fs.<scheme>.limit.input: (数量,0/-1 表示无输入连接限制)
fs.<scheme>.limit.output: (数量,0/-1 表示无输出连接限制)
fs.<scheme>.limit.timeout: (毫秒,0 表示无穷)
fs.<scheme>.limit.stream-timeout: (毫秒,0 表示无穷)
```
Expand All @@ -63,4 +63,4 @@ fs.<scheme>.limit.stream-timeout: (毫秒,0 表示无穷)

连接数是按每个 TaskManager/文件系统来进行限制的。因为文件系统的创建是按照 scheme 和 authority 进行的,所以不同的 authority 具有独立的连接池,例如 `hdfs://myhdfs:50010/` 和 `hdfs://anotherhdfs:4399/` 会有单独的连接池。

{{< top >}}
{{< top >}}
8 changes: 4 additions & 4 deletions docs/content/docs/deployment/filesystems/common.md
Original file line number Diff line number Diff line change
Expand Up @@ -52,9 +52,9 @@ To limit a specific file system's connections, add the following entries to the
its scheme.

```yaml
fs.<scheme>.limit.total: (number, 0/-1 mean no limit)
fs.<scheme>.limit.input: (number, 0/-1 mean no limit)
fs.<scheme>.limit.output: (number, 0/-1 mean no limit)
fs.<scheme>.limit.total: (number, 0/-1 means there is no connection limit)
fs.<scheme>.limit.input: (number, 0/-1 means there is no input connection limit)
fs.<scheme>.limit.output: (number, 0/-1 means there is no output connection limit)
fs.<scheme>.limit.timeout: (milliseconds, 0 means infinite)
fs.<scheme>.limit.stream-timeout: (milliseconds, 0 means infinite)
```
Expand All @@ -69,4 +69,4 @@ Limit enforcement on a per TaskManager/file system basis.
Because file systems creation occurs per scheme and authority, different
authorities have independent connection pools. For example `hdfs://myhdfs:50010/` and `hdfs://anotherhdfs:4399/` will have separate pools.

{{< top >}}
{{< top >}}
Original file line number Diff line number Diff line change
Expand Up @@ -527,23 +527,41 @@ public static String[] mergeListsToArray(List<String> base, List<String> append)
* open. Unlimited be default.
*/
public static ConfigOption<Integer> fileSystemConnectionLimit(String scheme) {
return ConfigOptions.key("fs." + scheme + ".limit.total").intType().defaultValue(-1);
return ConfigOptions.key("fs." + scheme + ".limit.total")
.intType()
.defaultValue(-1)
.withDescription(
"The connection limit for the file system "
+ scheme
+ ". The valid value must >= -1. Values of -1 or 0 means there is no connection limit.");
}

/**
* The total number of input connections that a file system for the given scheme may open.
* Unlimited be default.
*/
public static ConfigOption<Integer> fileSystemConnectionLimitIn(String scheme) {
return ConfigOptions.key("fs." + scheme + ".limit.input").intType().defaultValue(-1);
return ConfigOptions.key("fs." + scheme + ".limit.input")
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: as the 3 configurations only differ with the key and first word(s) of the description, could we define a method with the common code. Maybe a common method with connectionLimitType.
send in "total" "input" or "output" as connectionLimitType
key would have
.limit. + connectionLimitType
and the description could start
"The " + connectionLimitType + " connection limit ....

WDYT?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is a good idea. But I'm not sure if it is worth!

.intType()
.defaultValue(-1)
.withDescription(
"Input connection limit for the file system "
+ scheme
+ ". The valid value must >= -1. Values of -1 or 0 means there is no input connection limit.");
}

/**
* The total number of output connections that a file system for the given scheme may open.
* Unlimited be default.
*/
public static ConfigOption<Integer> fileSystemConnectionLimitOut(String scheme) {
return ConfigOptions.key("fs." + scheme + ".limit.output").intType().defaultValue(-1);
return ConfigOptions.key("fs." + scheme + ".limit.output")
.intType()
.defaultValue(-1)
.withDescription(
"Output connection limit for the file system "
+ scheme
+ ". The valid value must >= -1. Values of -1 or 0 means there is no output connection limit.");
}

/**
Expand Down