Skip to content

Update description of MAX_WRITE_SPEED #20852

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 3 commits into
base: master
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion sql-statements/sql-statement-admin-alter-ddl.md
Original file line number Diff line number Diff line change
Expand Up @@ -23,7 +23,7 @@ The following are the supported parameters for different DDL jobs and their corr
- `ADD INDEX`:
- `THREAD`: the concurrency of the DDL job. The initial value is set by `tidb_ddl_reorg_worker_cnt`.
- `BATCH_SIZE`: the batch size. The initial value is set by [`tidb_ddl_reorg_batch_size`](/system-variables.md#tidb_ddl_reorg_batch_size).
- `MAX_WRITE_SPEED`: the maximum bandwidth limit for importing index records into each TiKV. The initial value is set by [`tidb_ddl_reorg_max_write_speed`](/system-variables.md#tidb_ddl_reorg_max_write_speed-new-in-v6512-v755-and-v850).
- `MAX_WRITE_SPEED`: the maximum bandwidth limit for importing index records into TiKV on each TiDB node. The initial value is set by [`tidb_ddl_reorg_max_write_speed`](/system-variables.md#tidb_ddl_reorg_max_write_speed-new-in-v6512-v755-and-v850).

Currently, the preceding parameters only work for `ADD INDEX` jobs that are submitted and running after [`tidb_enable_dist_task`](/system-variables.md#tidb_enable_dist_task-new-in-v710) is disabled.

Expand Down
2 changes: 1 addition & 1 deletion sql-statements/sql-statement-import-into.md
Original file line number Diff line number Diff line change
Expand Up @@ -151,7 +151,7 @@ The supported options are described as follows:
| `DISK_QUOTA='<string>'` | All file formats | Specifies the disk space threshold that can be used during data sorting. The default value is 80% of the disk space in the TiDB [temporary directory](https://docs.pingcap.com/tidb/stable/tidb-configuration-file#temp-dir-new-in-v630). If the total disk size cannot be obtained, the default value is 50 GiB. When specifying `DISK_QUOTA` explicitly, make sure that the value does not exceed 80% of the disk space in the TiDB temporary directory. |
| `DISABLE_TIKV_IMPORT_MODE` | All file formats | Specifies whether to disable switching TiKV to import mode during the import process. By default, switching TiKV to import mode is not disabled. If there are ongoing read-write operations in the cluster, you can enable this option to avoid impact from the import process. |
| `THREAD=<number>` | All file formats and query results of `SELECT` | Specifies the concurrency for import. For `IMPORT INTO ... FROM FILE`, the default value of `THREAD` is 50% of the number of CPU cores on the TiDB node, the minimum value is `1`, and the maximum value is the number of CPU cores. For `IMPORT INTO ... FROM SELECT`, the default value of `THREAD` is `2`, the minimum value is `1`, and the maximum value is two times the number of CPU cores on the TiDB node. To import data into a new cluster without any data, it is recommended to increase this concurrency appropriately to improve import performance. If the target cluster is already used in a production environment, it is recommended to adjust this concurrency according to your application requirements. |
| `MAX_WRITE_SPEED='<string>'` | All file formats | Controls the write speed to a TiKV node. By default, there is no speed limit. For example, you can specify this option as `1MiB` to limit the write speed to 1 MiB/s. |
| `MAX_WRITE_SPEED='<string>'` | All file formats | Controls the write speed of each TiDB node to TiKV. By default, there is no speed limit. For example, if you have 10 TiDB nodes, specifying this option as `1MiB` limits the total write speed to 10 MiB/s. |
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
| `MAX_WRITE_SPEED='<string>'` | All file formats | Controls the write speed of each TiDB node to TiKV. By default, there is no speed limit. For example, if you have 10 TiDB nodes, specifying this option as `1MiB` limits the total write speed to 10 MiB/s. |
| `MAX_WRITE_SPEED='<string>'` | All file formats | Controls the write speed of each TiDB node to TiKV. By default, there is no speed limit. For example, if you have 10 TiDB nodes, specifying this option as `1MiB` limits the total write speed to some TiKV to be 10 MiB/s. |

| `CHECKSUM_TABLE='<string>'` | All file formats | Configures whether to perform a checksum check on the target table after the import to validate the import integrity. The supported values include `"required"` (default), `"optional"`, and `"off"`. `"required"` means performing a checksum check after the import. If the checksum check fails, TiDB will return an error and the import will exit. `"optional"` means performing a checksum check after the import. If an error occurs, TiDB will return a warning and ignore the error. `"off"` means not performing a checksum check after the import. |
| `DETACHED` | All file formats | Controls whether to execute `IMPORT INTO` asynchronously. When this option is enabled, executing `IMPORT INTO` immediately returns the information of the import job (such as the `Job_ID`), and the job is executed asynchronously in the backend. |
| `CLOUD_STORAGE_URI` | All file formats | Specifies the target address where encoded KV data for [Global Sort](/tidb-global-sort.md) is stored. When `CLOUD_STORAGE_URI` is not specified, `IMPORT INTO` determines whether to use Global Sort based on the value of the system variable [`tidb_cloud_storage_uri`](/system-variables.md#tidb_cloud_storage_uri-new-in-v740). If this system variable specifies a target storage address, `IMPORT INTO` uses this address for Global Sort. When `CLOUD_STORAGE_URI` is specified with a non-empty value, `IMPORT INTO` uses that value as the target storage address. When `CLOUD_STORAGE_URI` is specified with an empty value, local sorting is enforced. Currently, the target storage address only supports S3. For details about the URI configuration, see [Amazon S3 URI format](/external-storage-uri.md#amazon-s3-uri-format). When this feature is used, all TiDB nodes must have read and write access for the target S3 bucket, including at least these permissions: `s3:ListBucket`, `s3:GetObject`, `s3:DeleteObject`, `s3:PutObject`, `s3: AbortMultipartUpload`. |
Expand Down
3 changes: 2 additions & 1 deletion system-variables.md
Original file line number Diff line number Diff line change
Expand Up @@ -1741,11 +1741,12 @@ mysql> SELECT job_info FROM mysql.analyze_jobs ORDER BY end_time DESC LIMIT 1;
- Type: String
- Default value: `0`
- Range: `[0, 1PiB]`
- This variable limits the write bandwidth for each TiKV node and only takes effect when index creation acceleration is enabled (controlled by the [`tidb_ddl_enable_fast_reorg`](#tidb_ddl_enable_fast_reorg-new-in-v630) variable). When the data size in your cluster is quite large (such as billions of rows), limiting the write bandwidth for index creation can effectively reduce the impact on application workloads.
- This variable limits the write bandwidth of each TiDB node to TiKV and only takes effect when index creation acceleration is enabled (controlled by the [`tidb_ddl_enable_fast_reorg`](#tidb_ddl_enable_fast_reorg-new-in-v630) variable). When the data size in your cluster is quite large (such as billions of rows), limiting the write bandwidth for index creation can effectively reduce the impact on application workloads.
- The default value `0` means no write bandwidth limit.
- You can specify the value of this variable either with a unit or without a unit.
- When you specify the value without a unit, the default unit is bytes per second. For example, `67108864` represents `64MiB` per second.
- When you specify the value with a unit, supported units include KiB, MiB, GiB, and TiB. For example, `'1GiB`' represents 1 GiB per second, and `'256MiB'` represents 256 MiB per second.
- When the Distributed eXecution Framework (DXF) is enabled, this write limit applies to each TiDB node respectively. For example, if you add index using 4 TiDB nodes, setting this variable to `64MiB` means the maximum write speed to TiKV is `256MiB/s`.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
- When the Distributed eXecution Framework (DXF) is enabled, this write limit applies to each TiDB node respectively. For example, if you add index using 4 TiDB nodes, setting this variable to `64MiB` means the maximum write speed to TiKV is `256MiB/s`.
- When the Distributed eXecution Framework (DXF) is enabled, this write limit applies to each TiDB node respectively. For example, if you add index using 4 TiDB nodes, setting this variable to `64MiB` means the maximum write speed to one TiKV node is `256MiB/s`.


### tidb_ddl_reorg_worker_cnt

Expand Down
2 changes: 1 addition & 1 deletion tidb-lightning/tidb-lightning-configuration.md
Original file line number Diff line number Diff line change
Expand Up @@ -280,7 +280,7 @@ The `security` section specifies certificates and keys for TLS connections withi

#### `store-write-bwlimit`

- Limits the bandwidth in which TiDB Lightning writes data into each TiKV node in the physical import mode.
- Limits the per-table bandwidth to write data into TiKV for each TiDB Lightning instance in the physical import mode.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why add per-table?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

i remember all tables in one lightning instance shares the same limit

- Default value: `0`, which means no limit.

#### `disk-quota`
Expand Down
6 changes: 3 additions & 3 deletions tidb-lightning/tidb-lightning-physical-import-mode-usage.md
Original file line number Diff line number Diff line change
Expand Up @@ -57,8 +57,8 @@ duplicate-resolution = 'none'
# The directory of local KV sorting.
sorted-kv-dir = "./some-dir"

# Limits the bandwidth in which TiDB Lightning writes data into each TiKV
# node in the physical import mode. 0 by default, which means no limit.
# Limits the per-table bandwidth to write data into TiKV for each
# TiDB Lightning instance in the physical import mode.
# store-write-bwlimit = "128MiB"

# Specifies whether Physical Import Mode adds indexes via SQL. The default value is `false`, which means that TiDB Lightning will encode both row data and index data into KV pairs and import them into TiKV together. This mechanism is consistent with that of the historical versions. If you set it to `true`, it means that TiDB Lightning adds indexes via SQL after importing the row data.
Expand Down Expand Up @@ -206,7 +206,7 @@ By default, TiDB Lightning pauses the cluster scheduling for the minimum range p

```toml
[tikv-importer]
# Limits the bandwidth in which TiDB Lightning writes data into each TiKV node in the physical import mode.
# Limits the per-table bandwidth to write data into TiKV for each TiDB Lightning instance in the physical import mode.
store-write-bwlimit = "128MiB"

[tidb]
Expand Down