From cd08f5ed2ca4b0fe54353046fc7b515c2823fd53 Mon Sep 17 00:00:00 2001 From: Frank945946 Date: Wed, 6 Nov 2024 22:22:21 +0800 Subject: [PATCH 01/15] Update tidb-roadmap.md --- tidb-roadmap.md | 354 ++++++------------------------------------------ 1 file changed, 42 insertions(+), 312 deletions(-) diff --git a/tidb-roadmap.md b/tidb-roadmap.md index 41333efb2b866..490d074a0c386 100644 --- a/tidb-roadmap.md +++ b/tidb-roadmap.md @@ -9,320 +9,50 @@ This roadmap provides a look into the proposed future. This will be continually In the course of development, this roadmap is subject to change based on user needs and feedback. **DO NOT** schedule your release plans according to the content of the roadmap. If you have a feature request or want to prioritize a feature, please file an issue on [GitHub](https://github.com/pingcap/tidb/issues). +> **Note:** +> If not marked as GA, these features are experimental. + ## Rolling roadmap highlights - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
CategoryEnd of CY24 LTS releaseMid of CY25 LTS releaseFuture releases
- Scalability and Performance
Enhance horsepower - 		-
    -
  • - TiKV in-memory data cache
    - TiKV maintains recent versions of data in memory to reduce redundant MVCC scans, thus improving performance. -
    • -
    -
  • - Global indexing for partitioned tables
    -
    • -
    -
  • - Adaptive concurrency for statistics collection
    - TiDB automatically adjusts the parallelism and scan concurrency of statistics collection tasks based on the number of deployed nodes and hardware specifications, improving the speed of statistics collection. -
    • -
    -
  • - Rapid database recovery
    - Reduce the time required for full database recovery and point-in-time recovery (PITR). -
    • -
    -
  • - Unlimited-size transactions
    - The volume of data processed by uncommitted transactions is no longer limited by the available memory of TiDB nodes, thus improving the success rate of transactions and batch tasks. -
    • -
    -
  • - Load-based traffic routing by TiProxy
    - TiProxy forwards traffic based on the workload of the target TiDB node, maximizing hardware resource utilization. -
    • -
- 		-
    -
  • - Microservice for PD heartbeat
    - Heartbeat services in PD can be independently deployed and scaled, preventing PD from becoming a bottleneck for the cluster's resources. -
    • -
    -
  • - Less I/O consumption for statistics collection
    - Users can choose to scan only a portion of the data samples on TiKV during statistics collection, reducing time and resource consumption. -
    • -
    -
  • - Remove the limitation for pushing down Limit operator to TiKV
    -
    • -
    -
  • - Cascades optimizer framework
    - Introduce a more mature and powerful optimizer framework to expand the capabilities of the current optimizer. -
    • -
    -
  • - Single DM task reaches 150 MiB/s during full data migration
    -
    • -
    -
  • - Enhanced DDL execution framework
    - Provide a scalable parallel DDL execution framework to improve the performance and stability of DDL operations. -
    • -
- 		-
    -
  • - Table-level load balancing
    - PD determines data scheduling strategies based on the workload of each Region on every table. -
    • -
  • - Improve performance of handling system tables with large data volumes
    - Enhance query performance and reduce query overhead for system tables with large data volumes. -
    • -
-
- Reliability and Availability -
Enhance dependability - 		-
    -
  • - Limit memory consumption for backup tasks
    -
    • -
    -
  • - Limit memory consumption for statistics collection
    -
    • -
    -
  • - Manage massive SQL bindings
    - Improve the user experience of SQL binding, making it convenient for users to create and manage a large number of execution plans to stabilize database performance. -
    • -
    -
  • - Enhance resource group control over complex SQL
    - Regularly assess the Request Unit (RU) consumption of complex SQL before completion of SQL execution to prevent excessively large impacts on the entire system during execution. -
    • -
    -
  • - Automatically switch resource groups for runaway queries
    - When a query is identified as a runaway query, users can choose to switch it to a specific resource group and set an upper limit on resource consumption. -
    • -
- 		-
    -
  • - Limit memory consumption of schema metadata
    - Enhance the stability of large-scale clusters. -
    • -
    -
  • - Distributed statistics collection
    - Statistics collection supports parallel execution across multiple TiDB nodes to improve collection efficiency. -
    • -
    -
  • - Multi-version statistics
    - After statistics are updated, users can view the historical versions and choose to restore them to an earlier version. -
    • -
    -
  • - Reliable data backup
    - Reduce potential issues like insufficient memory during data backup and ensure the availability of backup data. -
    • -
    -
  • - Common operators support spilling to disk
    - Common operators such as HashAgg, Sort, TopN, HashJoin, WindowFunction, IndexJoin, and IndexHashJoin support spilling to disk, reducing the risk of out-of-memory (OOM). -
    • -
- 		-
    -
  • - Adaptive resource group
    - Resource groups automatically adjust their Request Unit (RU) settings based on past execution patterns. -
    • -
    -
  • - Enhanced memory protection
    - TiDB actively monitors the memory usage of all components and prevents memory operations that might impact system stability. -
    • -
    -
  • - Instance-level execution plan cache
    - All sessions within the same TiDB instance can share the execution plan cache, improving memory utilization. -
    • -
-
- Database Operations and Observability -
Enhance DB manageability and its ecosystem - 		-
    -
  • - Reliable query termination
    - Running SQL statements can be immediately terminated, and the corresponding resources are released from TiDB and TiKV. -
    • -
    -
  • - Permission control for switching resource groups -
    Only users with specific permissions can switch their resource groups, thus preventing resource abuse. -
    • -
    -
  • - Mapping tables or SQL with hot Regions -
    • -
    -
  • - Logical data import mode with IMPORT INTO -
    • -
    -
- 		-
    -
  • - Fine-grained customization of statistics collection -
    Users can modify the statistics collection strategy for specific tables, such as healthiness and parallelism. -
    • -
    -
  • - Workload Repository -
    TiDB persists workload information in memory to permanent volume, including cumulative and real-time statistic data, which aids in troubleshooting and analysis. -
    • -
    -
  • - Automatic index advisor -
    TiDB automatically analyzes SQL statements that can be optimized and recommends creating or dropping indexes. -
    • -
    -
  • - Support modifying column types for partitioned tables -
    Users can modify the data type of columns in partitioned tables, regardless of whether a column is a partitioning key. -
    • -
    -
  • - Conflict strategy for IMPORT INTO -
    Users can set the conflict resolution strategy when importing data, such as exiting with an error, ignoring, or replacing in case of conflicts. -
    • -
    -
  • - End-to-End monitoring -
    Track the time consumption of individual SQL statements throughout their entire lifecycle, including consumption on TiDB, TiKV, PD, and TiFlash components. -
    • -
- 		-
    -
  • - Workload analysis -
    Analyze historical workload data from the Workload Repository and provide optimization recommendations, such as SQL tuning and statistics collection. -
    • -
    -
  • - Revisable primary key -
    • -
    -
  • - Export data as SQL statements -
    • -
    -
-
- Security -
Enhance data safety and privacy - 		-
    -
  • - Google Cloud KMS -
    Enhance the key management mechanism for static encryption based on Google Cloud KMS, making it generally available (GA). -
    • -
    -
  • - Improved dynamic privilege -
    Improve the dynamic privilege design and limit the implementation of Super privilege. -
    • -
    -
  • - Marker-based log desensitization -
    Support marking sensitive information in the cluster log. Then, you can determine whether to desensitize it according to the usage scenario. -
    • -
    -
  • - FIPS -
    Encryption scenarios comply with FIPS. -
    • -
- 		-
    -
  • - IAM authentication for AWS -
    TiDB as AWS third-party ARN for AWS IAM access. -
    • -
    -
  • - Kerberos -
    Support Kerberos-based authentication. -
    • -
  • - MFA -
    Support the multi-factor authentication mechanism. -
    • -
- 		-
    -
  • - Label-based access control -
    Support data access control by configuring labels. -
    • -
    -
  • - Enhanced client-side encryption -
    Support client-side encryption of key fields to enhance data security. -
    • -
    -
  • - Dynamic desensitization of business data -
    Support desensitizing data based on different data application scenarios to ensure data security in important fields. -
    • -
-
+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
CategoryEnd of CY24 releaseMid of CY25 releaseFuture releases
Unmatched Scalability and Peak Performance
Deliver massive scalability and faster performance to support larger workloads, optimize resource utilization, and ensure superior responsiveness
In-memory caching in TiKV
TiKV maintains recent versions of data in memory to reduce redundant MVCC scans, thus improving performance.
Adaptive Parallelism for Stats Collection (GA)
Dynamically adjusts parallelism and concurrency based on hardware and node count, accelerating statistics collection.
Faster Database Restores
Reduces recovery time for full database and point-in-time restores (PITR).
Unlimited Transaction Size
Removes memory limits on uncommitted transactions, improving batch task success rates.
Load-Based Traffic Rebalancing in TiProxy(GA)
Routes traffic based on TiDB node workloads to optimize resource utilization.
PD Microservice Router Service
Enables independent deployment, stateless(no Strong Leader) and easy scaling of Router Service(region meta query/updates) in PD, preventing PD from becoming a bottleneck for cluster resources.
Reduced I/O for Statistics Collection (GA)
Allows users to scan only a portion of data samples on TiKV, reducing time and resource consumption for statistics collection.
Limit Operator Pushdown
Removes limitations on pushing down the Limit operator from TiDB to TiKV, enabling more efficient query processing directly at the storage layer.
Cascades Optimizer Framework
Introduces a mature, advanced optimizer framework, expanding the capabilities of the existing optimizer.
Scalable DDL Execution Framework
Provides a parallel DDL execution framework to improve the performance and stability of DDL operations.
Increased TiCDC Scalability
Updated TiCDC architecture delivers increased scalability and performance for change data capture use cases.
Table-Level Load Balancing
Optimizes data scheduling in PD based on workload distribution across Regions for each table.
System Table Performance Optimization
Enhances query speed and reduces overhead for system tables with large data volumes.
Enhance the Scability of Region Meta Storage
Migrate region meta storage from PD to TiKV, the storage layer can scale infinitely ans easily.
Advanced SQL Features and Flexibility
Cutting-edge SQL capabilities, improving compatibility, flexibility, and ease of use for complex queries and modern applications
Vector Search Support
Enables vector data types, indexing, and high-performance vector search, with support for mixed queries involving vector and relational data.
Foreign Keys(GA)
Providing robust relational integrity.
Global indexing on partitioned tables(GA)
Removes the unique key restriction on partition keys, boosting query performance for non-partitioned columns.		Modifiable Column Types in Partitioned Tables
Allows users to change column data types in partitioned tables, even if the column is a partitioning key.
Materialized Views Support
Enables materialized views to improve pre-computation, boost computational efficiency, and enhance data analysis performance.
Unbreakable Reliability and Always-On Availability
Near-zero downtime and enhanced fault tolerance to maintain uninterrupted operations and deliver a rock-solid user experience
Limit Memory for Backups
Limit Memory for Statistics Collection (GA)
Enhanced SQL Binding Management (GA)
Simplifies creating and managing large numbers of execution plans to stabilize performance.
Improved Resource Group Control for Complex SQL (GA)
Monitors RU usage of complex queries mid-execution to minimize system impact.
Automatic Resource Group Switching for Runaway Queries (GA)
Detects runaway queries and redirects them to designated resource groups with set limits.
Limit Memory Usage for Schema Metadata（GA）
Enhances stability in large-scale clusters by capping memory consumption for schema metadata.
Robust and Resilient Backup
Reduces memory-related issues during backup processes, ensuring dependable data protection and availability.
Optimized Memory Management with Disk Spilling
Allows operators like HashAgg, Sort, and Join to spill to disk, reducing memory load and preventing out-of-memory (OOM) issues.
Sharing Plan Cache across Sessions (GA)
Shares execution plan cache across sessions in the same TiDB instance, optimizing memory usage.
Resource Group Quota Management (GA)
Dynamically adjusts resource limits for Burstable resource groups, fully utilizing available resources without impacting other quotas.
Adaptive Resource Group
Automatically adjusts Request Unit (RU) settings in resource groups based on past execution patterns.
Enhanced Memory Protection
Monitors memory usage across all components to prevent operations that could impact system stability.
Automatic SQL Binding
Analyzes SQL performance metrics to automatically create bindings, stabilizing execution plans for transactional processing.
Multi-Versioned Statistics
Allows users to view and restore previous statistics versions after updates.
Distributed Statistics Collection
Enables parallel statistics collection across multiple TiDB nodes to boost efficiency.
Effortless Operations and Intelligent Observability
Simplify management with proactive monitoring and insights to optimize performance and ensure smooth operations
Reliable Query Termination（GA）
Instantly terminates running SQL statements and frees resources in TiDB and TiKV.
Permissioned Resource Group Switching (GA)
Restricts resource group switching to authorized users, preventing resource misuse.
CPU Time Observation for TiDB and TiKV (GA)
Adds CPU time metrics to logs, enabling quick identification of statements causing CPU spikes.
Customizable Statistics Collection (GA)
Allows tailored statistics strategies for specific tables, adjusting parameters like health and parallelism.
Workload Repository (GA)
Stores workload stats and real-time data for improved troubleshooting and analysis.
Automated Index Advisor (GA)
Automatically analyzes SQL statements to recommend index optimizations, including suggestions for creating or dropping indexes.
Standardized Time Model (GA)
Establishes a unified SQL execution time model to help identify database load sources through logs and cluster metrics, pinpointing problematic nodes and statements.
TiFlash CPU Time Monitoring (GA)
Adds TiFlash CPU time metrics to logs, enabling quick identification of statements that cause CPU spikes in TiFlash.		Workload Analysis
Analyzes historical data from the Workload Repository to provide optimization recommendations, including SQL tuning and statistics collection.
End-to-End SQL Monitoring
Tracks the entire lifecycle of SQL statements, measuring time spent across TiDB, TiKV, PD, and TiFlash for detailed performance insights.
Comprehensive Data Security and Privacy
Robust security measures to safeguard sensitive data, ensuring top-tier protection, encryption, and compliance with evolving privacy regulations		Google Cloud KMS(GA)
General availability for encryption-at-rest key management with Google Cloud KMS.
Azure Key Vault
Enhanced encryption-at-rest key management with Azure Key Vault integration.
Marker-Based Log Desensitization
Marks and selectively desensitizes sensitive data in cluster logs based on use case.
Column-Level Permission Management(GA)
Adds MySQL-compatible permissions at the column level for fine-grained access control.		AWS IAM Authentication
Supports AWS IAM third-party ARN integration for secure access control in TiDB.
Kerberos Authentication(GA)
Enables authentication using Kerberos for added security.
Multi-Factor Authentication (MFA)
Adds support for multi-factor authentication to enhance user verification the multi-factor authentication mechanism.
Enhanced TLS Security(GA)
Ensures encrypted connections between all components within the TiDB cluster.
Refined Dynamic Privileges
Improves dynamic privilege management, including limitations on Super privilege.
FIPS Compliance(GA)
Ensures encryption methods comply with FIPS standards for secure data handling.
Label-Based Access Control
Enables data access control through configurable labels.
Enhanced Client-Side Encryption
Supports encryption of key fields on the client side to strengthen data security.
Dynamic Data Desensitization
Allows data desensitization based on application scenarios, protecting sensitive business fields.
> **Note:** > From 7d7b98cc5c9911b067e528e7dc68ca07b5dbe923 Mon Sep 17 00:00:00 2001 From: Frank945946 Date: Mon, 11 Nov 2024 16:39:06 +0800 Subject: [PATCH 02/15] Update tidb-roadmap.md --- tidb-roadmap.md | 387 +++++++++++++++++++++++++++++++++++++++++++----- 1 file changed, 349 insertions(+), 38 deletions(-) diff --git a/tidb-roadmap.md b/tidb-roadmap.md index 490d074a0c386..bc01727bf59d7 100644 --- a/tidb-roadmap.md +++ b/tidb-roadmap.md @@ -10,49 +10,360 @@ This roadmap provides a look into the proposed future. This will be continually In the course of development, this roadmap is subject to change based on user needs and feedback. **DO NOT** schedule your release plans according to the content of the roadmap. If you have a feature request or want to prioritize a feature, please file an issue on [GitHub](https://github.com/pingcap/tidb/issues). > **Note:** +> > If not marked as GA, these features are experimental. ## Rolling roadmap highlights - - - - - - - +
CategoryEnd of CY24 releaseMid of CY25 releaseFuture releases
+ + + + + + + + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
+
Category
+		 +
End of CY24 release
+		 +
Mid of CY25 release
+		 +
Future releases
+
Unmatched Scalability and Peak Performance
Deliver massive scalability and faster performance to support larger workloads, optimize resource utilization, and ensure superior responsiveness
In-memory caching in TiKV
TiKV maintains recent versions of data in memory to reduce redundant MVCC scans, thus improving performance.
Adaptive Parallelism for Stats Collection (GA)
Dynamically adjusts parallelism and concurrency based on hardware and node count, accelerating statistics collection.
Faster Database Restores
Reduces recovery time for full database and point-in-time restores (PITR).
Unlimited Transaction Size
Removes memory limits on uncommitted transactions, improving batch task success rates.
Load-Based Traffic Rebalancing in TiProxy(GA)
Routes traffic based on TiDB node workloads to optimize resource utilization.
PD Microservice Router Service
Enables independent deployment, stateless(no Strong Leader) and easy scaling of Router Service(region meta query/updates) in PD, preventing PD from becoming a bottleneck for cluster resources.
Reduced I/O for Statistics Collection (GA)
Allows users to scan only a portion of data samples on TiKV, reducing time and resource consumption for statistics collection.
Limit Operator Pushdown
Removes limitations on pushing down the Limit operator from TiDB to TiKV, enabling more efficient query processing directly at the storage layer.
Cascades Optimizer Framework
Introduces a mature, advanced optimizer framework, expanding the capabilities of the existing optimizer.
Scalable DDL Execution Framework
Provides a parallel DDL execution framework to improve the performance and stability of DDL operations.
Increased TiCDC Scalability
Updated TiCDC architecture delivers increased scalability and performance for change data capture use cases.
Table-Level Load Balancing
Optimizes data scheduling in PD based on workload distribution across Regions for each table.
System Table Performance Optimization
Enhances query speed and reduces overhead for system tables with large data volumes.
Enhance the Scability of Region Meta Storage
Migrate region meta storage from PD to TiKV, the storage layer can scale infinitely ans easily.
Advanced SQL Features and Flexibility
Cutting-edge SQL capabilities, improving compatibility, flexibility, and ease of use for complex queries and modern applications
Vector Search Support
Enables vector data types, indexing, and high-performance vector search, with support for mixed queries involving vector and relational data.
Foreign Keys(GA)
Providing robust relational integrity.
Global indexing on partitioned tables(GA)
Removes the unique key restriction on partition keys, boosting query performance for non-partitioned columns.		Modifiable Column Types in Partitioned Tables
Allows users to change column data types in partitioned tables, even if the column is a partitioning key.
Materialized Views Support
Enables materialized views to improve pre-computation, boost computational efficiency, and enhance data analysis performance.
Unbreakable Reliability and Always-On Availability
Near-zero downtime and enhanced fault tolerance to maintain uninterrupted operations and deliver a rock-solid user experience
Limit Memory for Backups
Limit Memory for Statistics Collection (GA)
Enhanced SQL Binding Management (GA)
Simplifies creating and managing large numbers of execution plans to stabilize performance.
Improved Resource Group Control for Complex SQL (GA)
Monitors RU usage of complex queries mid-execution to minimize system impact.
Automatic Resource Group Switching for Runaway Queries (GA)
Detects runaway queries and redirects them to designated resource groups with set limits.
Limit Memory Usage for Schema Metadata（GA）
Enhances stability in large-scale clusters by capping memory consumption for schema metadata.
Robust and Resilient Backup
Reduces memory-related issues during backup processes, ensuring dependable data protection and availability.
Optimized Memory Management with Disk Spilling
Allows operators like HashAgg, Sort, and Join to spill to disk, reducing memory load and preventing out-of-memory (OOM) issues.
Sharing Plan Cache across Sessions (GA)
Shares execution plan cache across sessions in the same TiDB instance, optimizing memory usage.
Resource Group Quota Management (GA)
Dynamically adjusts resource limits for Burstable resource groups, fully utilizing available resources without impacting other quotas.
Adaptive Resource Group
Automatically adjusts Request Unit (RU) settings in resource groups based on past execution patterns.
Enhanced Memory Protection
Monitors memory usage across all components to prevent operations that could impact system stability.
Automatic SQL Binding
Analyzes SQL performance metrics to automatically create bindings, stabilizing execution plans for transactional processing.
Multi-Versioned Statistics
Allows users to view and restore previous statistics versions after updates.
Distributed Statistics Collection
Enables parallel statistics collection across multiple TiDB nodes to boost efficiency.
Effortless Operations and Intelligent Observability
Simplify management with proactive monitoring and insights to optimize performance and ensure smooth operations
Reliable Query Termination（GA）
Instantly terminates running SQL statements and frees resources in TiDB and TiKV.
Permissioned Resource Group Switching (GA)
Restricts resource group switching to authorized users, preventing resource misuse.
CPU Time Observation for TiDB and TiKV (GA)
Adds CPU time metrics to logs, enabling quick identification of statements causing CPU spikes.
Customizable Statistics Collection (GA)
Allows tailored statistics strategies for specific tables, adjusting parameters like health and parallelism.
Workload Repository (GA)
Stores workload stats and real-time data for improved troubleshooting and analysis.
Automated Index Advisor (GA)
Automatically analyzes SQL statements to recommend index optimizations, including suggestions for creating or dropping indexes.
Standardized Time Model (GA)
Establishes a unified SQL execution time model to help identify database load sources through logs and cluster metrics, pinpointing problematic nodes and statements.
TiFlash CPU Time Monitoring (GA)
Adds TiFlash CPU time metrics to logs, enabling quick identification of statements that cause CPU spikes in TiFlash.		Workload Analysis
Analyzes historical data from the Workload Repository to provide optimization recommendations, including SQL tuning and statistics collection.
End-to-End SQL Monitoring
Tracks the entire lifecycle of SQL statements, measuring time spent across TiDB, TiKV, PD, and TiFlash for detailed performance insights.
Comprehensive Data Security and Privacy
Robust security measures to safeguard sensitive data, ensuring top-tier protection, encryption, and compliance with evolving privacy regulations		Google Cloud KMS(GA)
General availability for encryption-at-rest key management with Google Cloud KMS.
Azure Key Vault
Enhanced encryption-at-rest key management with Azure Key Vault integration.
Marker-Based Log Desensitization
Marks and selectively desensitizes sensitive data in cluster logs based on use case.
Column-Level Permission Management(GA)
Adds MySQL-compatible permissions at the column level for fine-grained access control.		AWS IAM Authentication
Supports AWS IAM third-party ARN integration for secure access control in TiDB.
Kerberos Authentication(GA)
Enables authentication using Kerberos for added security.
Multi-Factor Authentication (MFA)
Adds support for multi-factor authentication to enhance user verification the multi-factor authentication mechanism.
Enhanced TLS Security(GA)
Ensures encrypted connections between all components within the TiDB cluster.
Refined Dynamic Privileges
Improves dynamic privilege management, including limitations on Super privilege.
FIPS Compliance(GA)
Ensures encryption methods comply with FIPS standards for secure data handling.
Label-Based Access Control
Enables data access control through configurable labels.
Enhanced Client-Side Encryption
Supports encryption of key fields on the client side to strengthen data security.
Dynamic Data Desensitization
Allows data desensitization based on application scenarios, protecting sensitive business fields.
+ + +
Unmatched Scalability and Peak Performance Deliver massive scalability and faster performance to support larger workloads, optimize resource utilization, and ensure superior responsiveness
+
 
+ + + +
 
+ + + +
 
+ + + +
 
+ + + + +
Advanced SQL Features and Flexibility
+
Cutting-edge SQL capabilities, improving compatibility, flexibility, and ease of use for complex queries and modern applications
+
 
+ + + + + + +
 
+ + + +
 
+ + + + +
Unbreakable Reliability and Always-On Availability Near-zero downtime and enhanced fault tolerance to maintain uninterrupted operations and deliver a rock-solid user experience
+
 
+ + + +
 
+ + + +
 
+ + + +
 
+ + + + +
Effortless Operations and Intelligent Observability Simplify management with proactive monitoring and insights to optimize performance and ensure smooth operations
+
 
+ + + +
 
+ + + + + + +
 
+ + + + +
Comprehensive Data Security and Privacy Robust security measures to safeguard sensitive data, ensuring top-tier protection, encryption, and compliance with evolving privacy regulations
+ + + +
 
+ + + +
 
+ + + +
 
+ + + + > **Note:** > From 2a5e2301ff892e6bc8bb19880943d44bd2c237bc Mon Sep 17 00:00:00 2001 From: houfaxin Date: Wed, 13 Nov 2024 11:00:08 +0800 Subject: [PATCH 03/15] Update tidb-roadmap.md --- tidb-roadmap.md | 683 +++++++++++++++++++++++------------------------- 1 file changed, 330 insertions(+), 353 deletions(-) diff --git a/tidb-roadmap.md b/tidb-roadmap.md index bc01727bf59d7..9819c26d1a2d9 100644 --- a/tidb-roadmap.md +++ b/tidb-roadmap.md @@ -11,360 +11,337 @@ In the course of development, this roadmap is subject to change based on user ne > **Note:** > -> If not marked as GA, these features are experimental. +> - If not marked as GA, these features are experimental. +> - These are non-exhaustive plans and are subject to change. +> - Features might differ per service subscriptions. ## Rolling roadmap highlights - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - +
-
Category
-		 -
End of CY24 release
-		 -
Mid of CY25 release
-		 -
Future releases
-
-
Unmatched Scalability and Peak Performance Deliver massive scalability and faster performance to support larger workloads, optimize resource utilization, and ensure superior responsiveness
-
 
-		 -
    -
  • -
    In-memory caching in TiKV
    -
    TiKV maintains recent versions of data in memory to reduce redundant MVCC scans, thus improving performance.
    -
    • -
  • -
    Adaptive Parallelism for Stats Collection (GA)
    -
    Dynamically adjusts parallelism and concurrency based on hardware and node count, accelerating statistics collection.
    -
    • -
  • -
    Faster Database Restores
    -
    Reduces recovery time for full database and point-in-time restores (PITR).
    -
    • -
  • -
    Unlimited Transaction Size
    -
    Removes memory limits on uncommitted transactions, improving batch task success rates.
    -
    • -
  • -
    Load-Based Traffic Rebalancing in TiProxy(GA)
    -
    Routes traffic based on TiDB node workloads to optimize resource utilization
    -
    • -
-
 
-		 -
    -
  • -
    PD Microservice Router Service
    -
    Enables independent deployment, stateless(no Strong Leader) and easy scaling of Router Service(region meta query/updates) in PD, preventing PD from becoming a bottleneck for cluster resources
    -
    • -
  • -
    Reduced I/O for Statistics Collection (GA)
    -
    Allows users to scan only a portion of data samples on TiKV, reducing time and resource consumption for statistics collection
    -
    • -
  • -
    Limit Operator Pushdown
    -
    Removes limitations on pushing down the Limit operator from TiDB to TiKV, enabling more efficient query processing directly at the storage layer
    -
    • -
  • -
    Cascades Optimizer Framework
    -
    Introduces a mature, advanced optimizer framework, expanding the capabilities of the existing optimizer.
    -
    • -
  • -
    Scalable DDL Execution Framework
    -
    Provides a parallel DDL execution framework to improve the performance and stability of DDL operations
    -
    • -
  • -
    Increased TiCDC Scalability
    -
    Updated TiCDC architecture delivers increased scalability and performance for change data capture use cases.
    -
    • -
-
 
-		 -
    -
  • -
    Table-Level Load Balancing
    -
    Optimizes data scheduling in PD based on workload distribution across Regions for each table
    -
    • -
  • -
    System Table Performance Optimization
    -
    Enhances query speed and reduces overhead for system tables with large data volumes
    -
    • -
  • -
    Enhance the Scability of Region Meta Storage
    -
    Migrate region meta storage from PD to TiKV, the storage layer can scale infinitely ans easily
    -
    • -
-
 
-
-
Advanced SQL Features and Flexibility
-
Cutting-edge SQL capabilities, improving compatibility, flexibility, and ease of use for complex queries and modern applications
-
 
-		 -
    -
  • -
    Vector Search Support
    -
    Enables vector data types, indexing, and high-performance vector search, with support for mixed queries involving vector and relational data.
    -
    • -
  • -
    Foreign Keys(GA)
    -
    Now generally available, providing robust relational integrity
    -
    • -
  • -
    Global indexing on partitioned tables(GA)
    -
    Removes the unique key restriction on partition keys, boosting query performance for non-partitioned columns.
    -
    • -
-		 -
    -
  • -
    Modifiable Column Types in Partitioned Tables
    -
    Allows users to change column data types in partitioned tables, even if the column is a partitioning key.
    -
    • -
-
 
-		 -
    -
  • -
    Materialized Views Support
    -
    Enables materialized views to improve pre-computation, boost computational efficiency, and enhance data analysis performance.
    -
    • -
-
 
-
-
Unbreakable Reliability and Always-On Availability Near-zero downtime and enhanced fault tolerance to maintain uninterrupted operations and deliver a rock-solid user experience
-
 
-		 -
    -
  • -
    Limit Memory for Backups
    -
    • -
  • -
    Limit Memory for Statistics Collection (GA)
    -
    • -
  • -
    Enhanced SQL Binding Management (GA)
    -
    Simplifies creating and managing large numbers of execution plans to stabilize performance.
    -
    • -
  • -
    Improved Resource Group Control for Complex SQL (GA)
    -
    Monitors RU usage of complex queries mid-execution to minimize system impact.
    -
    • -
  • -
    Automatic Resource Group Switching for Runaway Queries (GA)
    -
    Detects runaway queries and redirects them to designated resource groups with set limits.
    -
    • -
  • -
    Limit Memory Usage for Schema Metadata（GA）
    -
    Enhances stability in large-scale clusters by capping memory consumption for schema metadata.
    -
    • -
-
 
-		 -
    -
  • -
    Robust and Resilient Backup
    -
    Reduces memory-related issues during backup processes, ensuring dependable data protection and availability.
    -
    • -
  • -
    Optimized Memory Management with Disk Spilling
    -
    Allows operators like HashAgg, Sort, and Join to spill to disk, reducing memory load and preventing out-of-memory (OOM) issues.
    -
    • -
  • -
    Sharing Plan Cache across Sessions (GA)
    -
    Shares execution plan cache across sessions in the same TiDB instance, optimizing memory usage.
    -
    • -
  • -
    Resource Group Quota Management (GA)
    -
    Dynamically adjusts resource limits for Burstable resource groups, fully utilizing available resources without impacting other quotas
    -
    • -
-
 
-		 -
    -
  • -
    Adaptive Resource Group
    -
    Automatically adjusts Request Unit (RU) settings in resource groups based on past execution patterns
    -
    • -
  • -
    Enhanced Memory Protection
    -
    Monitors memory usage across all components to prevent operations that could impact system stability
    -
    • -
  • -
    Automatic SQL Binding
    -
    Analyzes SQL performance metrics to automatically create bindings, stabilizing execution plans for transactional processing
    -
    • -
  • -
    Multi-Versioned Statistics
    -
    Allows users to view and restore previous statistics versions after updates
    -
    • -
  • -
    Distributed Statistics Collection
    -
    Enables parallel statistics collection across multiple TiDB nodes to boost efficiency.
    -
    • -
-
 
-
-
Effortless Operations and Intelligent Observability Simplify management with proactive monitoring and insights to optimize performance and ensure smooth operations
-
 
-		 -
    -
  • -
    Reliable Query Termination（GA）
    -
    Instantly terminates running SQL statements and frees resources in TiDB and TiKV.
    -
    • -
  • -
    Permissioned Resource Group Switching (GA)
    -
    Restricts resource group switching to authorized users, preventing resource misuse.
    -
    • -
  • -
    CPU Time Observation for TiDB and TiKV (GA)
    -
    Adds CPU time metrics to logs, enabling quick identification of statements causing CPU spikes
    -
    • -
-
 
-		 -
    -
  • -
    Customizable Statistics Collection (GA)
    -
    Allows tailored statistics strategies for specific tables, adjusting parameters like health and parallelism.
    -
    • -
  • -
    Workload Repository (GA)
    -
    Stores workload stats and real-time data for improved troubleshooting and analysis.
    -
    • -
  • -
    Automated Index Advisor (GA)
    -
    Automatically analyzes SQL statements to recommend index optimizations, including suggestions for creating or dropping indexes
    -
    • -
  • -
    Standardized Time Model (GA)
    -
    Establishes a unified SQL execution time model to help identify database load sources through logs and cluster metrics, pinpointing problematic nodes and statements
    -
    • -
  • -
    TiFlash CPU Time Monitoring (GA)
    -
    Adds TiFlash CPU time metrics to logs, enabling quick identification of statements that cause CPU spikes in TiFlash
    -
    • -
-		 -
    -
  • -
    Workload Analysis
    -
    Analyzes historical data from the Workload Repository to provide optimization recommendations, including SQL tuning and statistics collection
    -
    • -
  • -
    End-to-End SQL Monitoring
    -
    Tracks the entire lifecycle of SQL statements, measuring time spent across TiDB, TiKV, PD, and TiFlash for detailed performance insights
    -
    • -
-
 
-
-
Comprehensive Data Security and Privacy Robust security measures to safeguard sensitive data, ensuring top-tier protection, encryption, and compliance with evolving privacy regulations
-		 -
    -
  • -
    Google Cloud KMS(GA)
    -
    General availability for encryption-at-rest key management with Google Cloud KMS.
    -
    • -
  • -
    Azure Key Vault
    -
    Enhanced encryption-at-rest key management with Azure Key Vault integration.
    -
    • -
  • -
    Marker-Based Log Desensitization
    -
    Marks and selectively desensitizes sensitive data in cluster logs based on use case.
    -
    • -
  • -
    Column-Level Permission Management(GA)
    -
    Adds MySQL-compatible permissions at the column level for fine-grained access control.
    -
    • -
-
 
-		 -
    -
  • -
    AWS IAM Authentication
    -
    Supports AWS IAM third-party ARN integration for secure access control in TiDB
    -
    • -
  • -
    Kerberos Authentication(GA)
    -
    Enables authentication using Kerberos for added security.
    -
    • -
  • -
    Multi-Factor Authentication (MFA)
    -
    Adds support for multi-factor authentication to enhance user verification the multi-factor authentication mechanism.
    -
    • -
  • -
    Enhanced TLS Security(GA)
    -
    Ensures encrypted connections between all components within the TiDB cluster
    -
    • -
  • -
    Refined Dynamic Privileges
    -
    Improves dynamic privilege management, including limitations on Super privilege
    -
    • -
  • -
    FIPS Compliance(GA)
    -
    Ensures encryption methods comply with FIPS standards for secure data handling.
    -
    • -
-
 
-		 -
    -
  • -
    Label-Based Access Control
    -
    Enables data access control through configurable labels.
    -
    • -
  • -
    Enhanced Client-Side Encryption
    -
    Supports encryption of key fields on the client side to strengthen data security
    -
    • -
  • -
    Dynamic Data Desensitization
    -
    Allows data desensitization based on application scenarios, protecting sensitive business fields
    -
    • -
-
 
-
+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
CategoryEnd of CY24 releaseMid of CY25 releaseFuture releases
+ Unmatched Scalability and Peak Performance +
Deliver massive scalability and faster performance to support larger workloads, optimize resource utilization, and ensure superior responsiveness. + 		+
    +
  • + In-memory caching in TiKV +
    TiKV maintains recent versions of data in memory to reduce redundant MVCC scans, thus improving performance. +
    • +
  • + Adaptive Parallelism for Stats Collection (GA) +
    Dynamically adjusts parallelism and concurrency based on hardware and node count, accelerating statistics collection. +
    • +
  • + Faster Database Restores +
    Reduces recovery time for full database and point-in-time restores (PITR). +
    • +
  • + Unlimited Transaction Size +
    Removes memory limits on uncommitted transactions, improving batch task success rates. +
    • +
  • + Load-Based Traffic Rebalancing in TiProxy (GA) +
    Routes traffic based on TiDB node workloads to optimize resource utilization. +
    • +
+ 		+
    +
  • + PD Microservice Router Service +
    Enables independent deployment, stateless (no Strong Leader) and easy scaling of Router Service (region meta query/updates) in PD, preventing PD from becoming a bottleneck for cluster resources. +
    • +
  • + Reduced I/O for Statistics Collection (GA) +
    Allows users to scan only a portion of data samples on TiKV, reducing time and resource consumption for statistics collection. +
    • +
  • + Limit Operator Pushdown +
    Removes limitations on pushing down the Limit operator from TiDB to TiKV, enabling more efficient query processing directly at the storage layer. +
    • +
  • + Cascades Optimizer Framework +
    Introduces a mature, advanced optimizer framework, expanding the capabilities of the existing optimizer. +
    • +
  • + Scalable DDL Execution Framework +
    Provides a parallel DDL execution framework to improve the performance and stability of DDL operations. +
    • +
  • + Increased TiCDC Scalability +
    Updated TiCDC architecture delivers increased scalability and performance for change data capture use cases. +
    • +
+ 		+
    +
  • + Table-Level Load Balancing +
    Optimizes data scheduling in PD based on workload distribution across Regions for each table. +
    • +
  • + System Table Performance Optimization +
    Enhances query speed and reduces overhead for system tables with large data volumes. +
    • +
  • + Enhance the Scalability of Region Meta Storage +
    Migrate region meta storage from PD to TiKV, allowing the storage layer to scale infinitely and easily. +
    • +
+
+ Advanced SQL Features and Flexibility + Cutting-edge SQL capabilities, improving compatibility, flexibility, and ease of use for complex queries and modern applications + +
    +
  • + Vector Search Support +
    Enables vector data types, indexing, and high-performance vector search, with support for mixed queries involving vector and relational data. +
    • +
  • + Foreign Keys (GA) +
    Now generally available, providing robust relational integrity. +
    • +
  • + Global indexing on partitioned tables (GA) +
    Removes the unique key restriction on partition keys, boosting query performance for non-partitioned columns. +
    • +
+ 		+
    +
  • + Modifiable Column Types in Partitioned Tables +
    Allows users to change column data types in partitioned tables, even if the column is a partitioning key. +
    • +
+ 		+
    +
  • + Materialized Views Support +
    Enables materialized views to improve pre-computation, boost computational efficiency, and enhance data analysis performance. +
    • +
+
+ Unbreakable Reliability and Always-On Availability + Near-zero downtime and enhanced fault tolerance to maintain uninterrupted operations and deliver a rock-solid user experience + +
    +
  • + Limit Memory for Backups +
    • +
  • + Limit Memory for Statistics Collection (GA) +
    • +
  • + Enhanced SQL Binding Management (GA) +
    Simplifies creating and managing large numbers of execution plans to stabilize performance. +
    • +
  • + Improved Resource Group Control for Complex SQL (GA) +
    Monitors RU usage of complex queries mid-execution to minimize system impact. +
    • +
  • + Automatic Resource Group Switching for Runaway Queries (GA) +
    Detects runaway queries and redirects them to designated resource groups with set limits. +
    • +
  • + Limit Memory Usage for Schema Metadata (GA) +
    Enhances stability in large-scale clusters by capping memory consumption for schema metadata. +
    • +
+ 		+
    +
  • + Robust and Resilient Backup +
    Reduces memory-related issues during backup processes, ensuring dependable data protection and availability. +
    • +
  • + Optimized Memory Management with Disk Spilling +
    Allows operators like HashAgg, Sort, and Join to spill to disk, reducing memory load and preventing out-of-memory (OOM) issues. +
    • +
  • + Sharing Plan Cache across Sessions (GA) +
    Shares execution plan cache across sessions in the same TiDB instance, optimizing memory usage. +
    • +
  • + Resource Group Quota Management (GA) +
    Dynamically adjusts resource limits for Burstable resource groups, fully utilizing available resources without impacting other quotas. +
    • +
+ 		+
    +
  • + Adaptive Resource Group +
    Automatically adjusts Request Unit (RU) settings in resource groups based on past execution patterns. +
    • +
  • + Enhanced Memory Protection +
    Monitors memory usage across all components to prevent operations that could impact system stability. +
    • +
  • + Automatic SQL Binding +
    Analyzes SQL performance metrics to automatically create bindings, stabilizing execution plans for transactional processing. +
    • +
  • + Multi-Versioned Statistics +
    Allows users to view and restore previous statistics versions after updates. +
    • +
  • + Distributed Statistics Collection +
    Enables parallel statistics collection across multiple TiDB nodes to boost efficiency. +
    • +
+
+ Effortless Operations and Intelligent Observability + Simplify management with proactive monitoring and insights to optimize performance and ensure smooth operations + +
    +
  • + Reliable Query Termination (GA) +
    Instantly terminates running SQL statements and frees resources in TiDB and TiKV. +
    • +
  • + Permissioned Resource Group Switching (GA) +
    Restricts resource group switching to authorized users, preventing resource misuse. +
    • +
  • + CPU Time Observation for TiDB and TiKV (GA) +
    Adds CPU time metrics to logs, enabling quick identification of statements causing CPU spikes. +
    • +
+ 		+
    +
  • + Customizable Statistics Collection (GA) +
    Allows tailored statistics strategies for specific tables, adjusting parameters like health and parallelism. +
    • +
  • + Workload Repository (GA) +
    Stores workload stats and real-time data for improved troubleshooting and analysis. +
    • +
  • + Automated Index Advisor (GA) +
    Automatically analyzes SQL statements to recommend index optimizations, including suggestions for creating or dropping indexes. +
    • +
  • + Standardized Time Model (GA) +
    Establishes a unified SQL execution time model to help identify database load sources through logs and cluster metrics, pinpointing problematic nodes and statements. +
    • +
  • + TiFlash CPU Time Monitoring (GA) +
    Adds TiFlash CPU time metrics to logs, enabling quick identification of statements that cause CPU spikes in TiFlash. +
    • +
+ 		+
    +
  • + Workload Analysis +
    Analyzes historical data from the Workload Repository to provide optimization recommendations, including SQL tuning and statistics collection. +
    • +
  • + End-to-End SQL Monitoring +
    Tracks the entire lifecycle of SQL statements, measuring time spent across TiDB, TiKV, PD, and TiFlash for detailed performance insights. +
    • +
+
+ Comprehensive Data Security and Privacy + Robust security measures to safeguard sensitive data, ensuring top-tier protection, encryption, and compliance with evolving privacy regulations + +
    +
  • + Google Cloud KMS (GA) +
    General availability for encryption-at-rest key management with Google Cloud KMS. +
    • +
  • + Azure Key Vault +
    Enhanced encryption-at-rest key management with Azure Key Vault integration. +
    • +
  • + Marker-Based Log Desensitization +
    Marks and selectively desensitizes sensitive data in cluster logs based on use case. +
    • +
  • + Column-Level Permission Management (GA) +
    Adds MySQL-compatible permissions at the column level for fine-grained access control. +
    • +
+ 		+
    +
  • + AWS IAM Authentication +
    Supports AWS IAM third-party ARN integration for secure access control in TiDB. +
    • +
  • + Kerberos Authentication (GA) +
    Enables authentication using Kerberos for added security. +
    • +
  • + Multi-Factor Authentication (MFA) +
    Adds support for multi-factor authentication to enhance user verification. +
    • +
  • + Enhanced TLS Security (GA) +
    Ensures encrypted connections between all components within the TiDB cluster. +
    • +
  • + Refined Dynamic Privileges +
    Improves dynamic privilege management, including limitations on Super privilege. +
    • +
  • + FIPS Compliance (GA) +
    Ensures encryption methods comply with FIPS standards for secure data handling. +
    • +
+ 		+
    +
  • + Label-Based Access Control +
    Enables data access control through configurable labels. +
    • +
  • + Enhanced Client-Side Encryption +
    Supports encryption of key fields on the client side to strengthen data security. +
    • +
  • + Dynamic Data Desensitization +
    Allows data desensitization based on application scenarios, protecting sensitive business fields. +
    • +
+
- -> **Note:** -> -> These are non-exhaustive plans and are subject to change. Features might differ per service subscriptions. \ No newline at end of file From 9bf5e1308c0d00c4af59efe010187faea7609a0e Mon Sep 17 00:00:00 2001 From: houfaxin Date: Wed, 13 Nov 2024 11:02:45 +0800 Subject: [PATCH 04/15] Update tidb-roadmap.md --- tidb-roadmap.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/tidb-roadmap.md b/tidb-roadmap.md index 9819c26d1a2d9..43fe9ff1e1368 100644 --- a/tidb-roadmap.md +++ b/tidb-roadmap.md @@ -64,7 +64,7 @@ In the course of development, this roadmap is subject to change based on user ne
  • Reduced I/O for Statistics Collection (GA) -
    Allows users to scan only a portion of data samples on TiKV, reducing time and resource consumption for statistics collection. +
    Allows you to scan only a portion of data samples on TiKV, reducing time and resource consumption for statistics collection.
  • Limit Operator Pushdown @@ -198,7 +198,7 @@ In the course of development, this roadmap is subject to change based on user ne
  • Enhanced Memory Protection -
    Monitors memory usage across all components to prevent operations that could impact system stability. +
    Monitors memory usage across all components to prevent operations that might impact system stability.
  • Automatic SQL Binding @@ -206,7 +206,7 @@ In the course of development, this roadmap is subject to change based on user ne
  • Multi-Versioned Statistics -
    Allows users to view and restore previous statistics versions after updates. +
    Allows you to view and restore previous statistics versions after updates.
  • Distributed Statistics Collection From d10a53a45bc017de2097cbf201f4b5fc6f8d7adc Mon Sep 17 00:00:00 2001 From: houfaxin Date: Wed, 13 Nov 2024 11:03:05 +0800 Subject: [PATCH 05/15] Update tidb-roadmap.md --- tidb-roadmap.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/tidb-roadmap.md b/tidb-roadmap.md index 43fe9ff1e1368..b35b47c17f0d1 100644 --- a/tidb-roadmap.md +++ b/tidb-roadmap.md @@ -126,7 +126,7 @@ In the course of development, this roadmap is subject to change based on user ne
    • Modifiable Column Types in Partitioned Tables -
      Allows users to change column data types in partitioned tables, even if the column is a partitioning key. +
      Allows you to change column data types in partitioned tables, even if the column is a partitioning key.
    From c31ce5d50b50ebf0965dc3dc9ec61273e8a1ae85 Mon Sep 17 00:00:00 2001 From: houfaxin Date: Wed, 13 Nov 2024 11:10:20 +0800 Subject: [PATCH 06/15] Update tidb-roadmap.md --- tidb-roadmap.md | 57 +++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 57 insertions(+) diff --git a/tidb-roadmap.md b/tidb-roadmap.md index b35b47c17f0d1..7a4c1160bfd8f 100644 --- a/tidb-roadmap.md +++ b/tidb-roadmap.md @@ -38,22 +38,27 @@ In the course of development, this roadmap is subject to change based on user ne In-memory caching in TiKV
    TiKV maintains recent versions of data in memory to reduce redundant MVCC scans, thus improving performance.
    • +
  • Adaptive Parallelism for Stats Collection (GA)
    Dynamically adjusts parallelism and concurrency based on hardware and node count, accelerating statistics collection.
    • +
  • Faster Database Restores
    Reduces recovery time for full database and point-in-time restores (PITR).
    • +
  • Unlimited Transaction Size
    Removes memory limits on uncommitted transactions, improving batch task success rates.
    • +
  • Load-Based Traffic Rebalancing in TiProxy (GA)
    Routes traffic based on TiDB node workloads to optimize resource utilization.
    • +
    @@ -62,26 +67,32 @@ In the course of development, this roadmap is subject to change based on user ne PD Microservice Router Service
    Enables independent deployment, stateless (no Strong Leader) and easy scaling of Router Service (region meta query/updates) in PD, preventing PD from becoming a bottleneck for cluster resources. +
  • Reduced I/O for Statistics Collection (GA)
    Allows you to scan only a portion of data samples on TiKV, reducing time and resource consumption for statistics collection.
    • +
  • Limit Operator Pushdown
    Removes limitations on pushing down the Limit operator from TiDB to TiKV, enabling more efficient query processing directly at the storage layer.
    • +
  • Cascades Optimizer Framework
    Introduces a mature, advanced optimizer framework, expanding the capabilities of the existing optimizer.
    • +
  • Scalable DDL Execution Framework
    Provides a parallel DDL execution framework to improve the performance and stability of DDL operations.
    • +
  • Increased TiCDC Scalability
    Updated TiCDC architecture delivers increased scalability and performance for change data capture use cases.
    • +
    @@ -90,14 +101,17 @@ In the course of development, this roadmap is subject to change based on user ne Table-Level Load Balancing
    Optimizes data scheduling in PD based on workload distribution across Regions for each table. +
  • System Table Performance Optimization
    Enhances query speed and reduces overhead for system tables with large data volumes.
    • +
  • Enhance the Scalability of Region Meta Storage
    Migrate region meta storage from PD to TiKV, allowing the storage layer to scale infinitely and easily.
    • +
    @@ -112,14 +126,17 @@ In the course of development, this roadmap is subject to change based on user ne Vector Search Support
    Enables vector data types, indexing, and high-performance vector search, with support for mixed queries involving vector and relational data. +
  • Foreign Keys (GA)
    Now generally available, providing robust relational integrity.
    • +
  • Global indexing on partitioned tables (GA)
    Removes the unique key restriction on partition keys, boosting query performance for non-partitioned columns.
    • +
    @@ -128,6 +145,7 @@ In the course of development, this roadmap is subject to change based on user ne Modifiable Column Types in Partitioned Tables
    Allows you to change column data types in partitioned tables, even if the column is a partitioning key. +
    @@ -136,6 +154,7 @@ In the course of development, this roadmap is subject to change based on user ne Materialized Views Support
    Enables materialized views to improve pre-computation, boost computational efficiency, and enhance data analysis performance. +
    @@ -149,25 +168,31 @@ In the course of development, this roadmap is subject to change based on user ne
  • Limit Memory for Backups
    • +
  • Limit Memory for Statistics Collection (GA)
    • +
  • Enhanced SQL Binding Management (GA)
    Simplifies creating and managing large numbers of execution plans to stabilize performance.
    • +
  • Improved Resource Group Control for Complex SQL (GA)
    Monitors RU usage of complex queries mid-execution to minimize system impact.
    • +
  • Automatic Resource Group Switching for Runaway Queries (GA)
    Detects runaway queries and redirects them to designated resource groups with set limits.
    • +
  • Limit Memory Usage for Schema Metadata (GA)
    Enhances stability in large-scale clusters by capping memory consumption for schema metadata.
    • +
    @@ -176,18 +201,22 @@ In the course of development, this roadmap is subject to change based on user ne Robust and Resilient Backup
    Reduces memory-related issues during backup processes, ensuring dependable data protection and availability. +
  • Optimized Memory Management with Disk Spilling
    Allows operators like HashAgg, Sort, and Join to spill to disk, reducing memory load and preventing out-of-memory (OOM) issues.
    • +
  • Sharing Plan Cache across Sessions (GA)
    Shares execution plan cache across sessions in the same TiDB instance, optimizing memory usage.
    • +
  • Resource Group Quota Management (GA)
    Dynamically adjusts resource limits for Burstable resource groups, fully utilizing available resources without impacting other quotas.
    • +
    @@ -196,22 +225,27 @@ In the course of development, this roadmap is subject to change based on user ne Adaptive Resource Group
    Automatically adjusts Request Unit (RU) settings in resource groups based on past execution patterns. +
  • Enhanced Memory Protection
    Monitors memory usage across all components to prevent operations that might impact system stability.
    • +
  • Automatic SQL Binding
    Analyzes SQL performance metrics to automatically create bindings, stabilizing execution plans for transactional processing.
    • +
  • Multi-Versioned Statistics
    Allows you to view and restore previous statistics versions after updates.
    • +
  • Distributed Statistics Collection
    Enables parallel statistics collection across multiple TiDB nodes to boost efficiency.
    • +
    @@ -226,14 +260,17 @@ In the course of development, this roadmap is subject to change based on user ne Reliable Query Termination (GA)
    Instantly terminates running SQL statements and frees resources in TiDB and TiKV. +
  • Permissioned Resource Group Switching (GA)
    Restricts resource group switching to authorized users, preventing resource misuse.
    • +
  • CPU Time Observation for TiDB and TiKV (GA)
    Adds CPU time metrics to logs, enabling quick identification of statements causing CPU spikes.
    • +
    @@ -242,22 +279,27 @@ In the course of development, this roadmap is subject to change based on user ne Customizable Statistics Collection (GA)
    Allows tailored statistics strategies for specific tables, adjusting parameters like health and parallelism. +
  • Workload Repository (GA)
    Stores workload stats and real-time data for improved troubleshooting and analysis.
    • +
  • Automated Index Advisor (GA)
    Automatically analyzes SQL statements to recommend index optimizations, including suggestions for creating or dropping indexes.
    • +
  • Standardized Time Model (GA)
    Establishes a unified SQL execution time model to help identify database load sources through logs and cluster metrics, pinpointing problematic nodes and statements.
    • +
  • TiFlash CPU Time Monitoring (GA)
    Adds TiFlash CPU time metrics to logs, enabling quick identification of statements that cause CPU spikes in TiFlash.
    • +
    @@ -266,10 +308,12 @@ In the course of development, this roadmap is subject to change based on user ne Workload Analysis
    Analyzes historical data from the Workload Repository to provide optimization recommendations, including SQL tuning and statistics collection. +
  • End-to-End SQL Monitoring
    Tracks the entire lifecycle of SQL statements, measuring time spent across TiDB, TiKV, PD, and TiFlash for detailed performance insights.
    • +
    @@ -284,18 +328,22 @@ In the course of development, this roadmap is subject to change based on user ne Google Cloud KMS (GA)
    General availability for encryption-at-rest key management with Google Cloud KMS. +
  • Azure Key Vault
    Enhanced encryption-at-rest key management with Azure Key Vault integration.
    • +
  • Marker-Based Log Desensitization
    Marks and selectively desensitizes sensitive data in cluster logs based on use case.
    • +
  • Column-Level Permission Management (GA)
    Adds MySQL-compatible permissions at the column level for fine-grained access control.
    • +
    @@ -304,26 +352,32 @@ In the course of development, this roadmap is subject to change based on user ne AWS IAM Authentication
    Supports AWS IAM third-party ARN integration for secure access control in TiDB. +
  • Kerberos Authentication (GA)
    Enables authentication using Kerberos for added security.
    • +
  • Multi-Factor Authentication (MFA)
    Adds support for multi-factor authentication to enhance user verification.
    • +
  • Enhanced TLS Security (GA)
    Ensures encrypted connections between all components within the TiDB cluster.
    • +
  • Refined Dynamic Privileges
    Improves dynamic privilege management, including limitations on Super privilege.
    • +
  • FIPS Compliance (GA)
    Ensures encryption methods comply with FIPS standards for secure data handling.
    • +
    @@ -332,14 +386,17 @@ In the course of development, this roadmap is subject to change based on user ne Label-Based Access Control
    Enables data access control through configurable labels. +
  • Enhanced Client-Side Encryption
    Supports encryption of key fields on the client side to strengthen data security.
    • +
  • Dynamic Data Desensitization
    Allows data desensitization based on application scenarios, protecting sensitive business fields.
    • +
    From 20cf6967cc062f249cf64e4cc031bde8c0d16c86 Mon Sep 17 00:00:00 2001 From: houfaxin Date: Thu, 14 Nov 2024 14:40:43 +0800 Subject: [PATCH 07/15] Update tidb-roadmap.md --- tidb-roadmap.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/tidb-roadmap.md b/tidb-roadmap.md index 7a4c1160bfd8f..925c44552ee09 100644 --- a/tidb-roadmap.md +++ b/tidb-roadmap.md @@ -118,7 +118,7 @@ In the course of development, this roadmap is subject to change based on user ne Advanced SQL Features and Flexibility - Cutting-edge SQL capabilities, improving compatibility, flexibility, and ease of use for complex queries and modern applications +
    Cutting-edge SQL capabilities, improving compatibility, flexibility, and ease of use for complex queries and modern applications
      @@ -161,7 +161,7 @@ In the course of development, this roadmap is subject to change based on user ne Unbreakable Reliability and Always-On Availability - Near-zero downtime and enhanced fault tolerance to maintain uninterrupted operations and deliver a rock-solid user experience +
      Near-zero downtime and enhanced fault tolerance to maintain uninterrupted operations and deliver a rock-solid user experience
        @@ -252,7 +252,7 @@ In the course of development, this roadmap is subject to change based on user ne Effortless Operations and Intelligent Observability - Simplify management with proactive monitoring and insights to optimize performance and ensure smooth operations +
        Simplify management with proactive monitoring and insights to optimize performance and ensure smooth operations
          @@ -320,7 +320,7 @@ In the course of development, this roadmap is subject to change based on user ne Comprehensive Data Security and Privacy - Robust security measures to safeguard sensitive data, ensuring top-tier protection, encryption, and compliance with evolving privacy regulations +
          Robust security measures to safeguard sensitive data, ensuring top-tier protection, encryption, and compliance with evolving privacy regulations
            From fcfc6c4aeb624a96765fe2349753a9a263fa6e5a Mon Sep 17 00:00:00 2001 From: xixirangrang Date: Thu, 14 Nov 2024 14:41:46 +0800 Subject: [PATCH 08/15] Apply suggestions from code review --- tidb-roadmap.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/tidb-roadmap.md b/tidb-roadmap.md index 925c44552ee09..cbac276d952b2 100644 --- a/tidb-roadmap.md +++ b/tidb-roadmap.md @@ -46,7 +46,7 @@ In the course of development, this roadmap is subject to change based on user ne
          • Faster Database Restores -
            Reduces recovery time for full database and point-in-time restores (PITR). +
            Reduces recovery time for full database and Point in Time Recovery (PITR).

          • From 334b34fd45654422885ca46b17e8a5b239aede02 Mon Sep 17 00:00:00 2001 From: houfaxin Date: Thu, 14 Nov 2024 15:32:37 +0800 Subject: [PATCH 09/15] Update tidb-roadmap.md --- tidb-roadmap.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/tidb-roadmap.md b/tidb-roadmap.md index 925c44552ee09..e2fbbddab2712 100644 --- a/tidb-roadmap.md +++ b/tidb-roadmap.md @@ -123,7 +123,7 @@ In the course of development, this roadmap is subject to change based on user ne
            • - Vector Search Support + Vector Search
              Enables vector data types, indexing, and high-performance vector search, with support for mixed queries involving vector and relational data.

              • @@ -151,7 +151,7 @@ In the course of development, this roadmap is subject to change based on user ne
              • - Materialized Views Support + Materialized Views
                Enables materialized views to improve pre-computation, boost computational efficiency, and enhance data analysis performance.

                • @@ -268,7 +268,7 @@ In the course of development, this roadmap is subject to change based on user ne
              • CPU Time Observation for TiDB and TiKV (GA) -
                Adds CPU time metrics to logs, enabling quick identification of statements causing CPU spikes. +
                Adds TiDB and TiKV CPU time metrics to statements and slow logs, enabling quick identification of statements causing TiDB or TiKV CPU spikes.

              @@ -297,7 +297,7 @@ In the course of development, this roadmap is subject to change based on user ne
            • TiFlash CPU Time Monitoring (GA) -
              Adds TiFlash CPU time metrics to logs, enabling quick identification of statements that cause CPU spikes in TiFlash. +
              Adds TiFlash CPU time metrics to statements and slow logs, enabling quick identification of statements that cause CPU spikes in TiFlash.

            From 0ca108becdcf970cb114b8b945d5d8d4de59fd7f Mon Sep 17 00:00:00 2001 From: houfaxin Date: Thu, 14 Nov 2024 15:35:51 +0800 Subject: [PATCH 10/15] Update tidb-roadmap.md --- tidb-roadmap.md | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/tidb-roadmap.md b/tidb-roadmap.md index 9947860b4d611..7bf1a8361f77d 100644 --- a/tidb-roadmap.md +++ b/tidb-roadmap.md @@ -128,8 +128,7 @@ In the course of development, this roadmap is subject to change based on user ne

          • - Foreign Keys (GA) -
            Now generally available, providing robust relational integrity. + Foreign Keys is generally available (GA)

          • From d5b718869a48f40d7abf817f60d2b6109cf384c0 Mon Sep 17 00:00:00 2001 From: Frank945946 <108602632+Frank945946@users.noreply.github.com> Date: Mon, 25 Nov 2024 15:59:57 +0800 Subject: [PATCH 11/15] Update tidb-roadmap.md Co-authored-by: Grace Cai --- tidb-roadmap.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/tidb-roadmap.md b/tidb-roadmap.md index 7bf1a8361f77d..968f5de2ae80e 100644 --- a/tidb-roadmap.md +++ b/tidb-roadmap.md @@ -40,7 +40,7 @@ In the course of development, this roadmap is subject to change based on user ne

          • - Adaptive Parallelism for Stats Collection (GA) + Adaptive Parallelism for Statistics Collection (GA)
            Dynamically adjusts parallelism and concurrency based on hardware and node count, accelerating statistics collection.

            • From 8aa06202027b65cffd3144d6ff52027a3ff45761 Mon Sep 17 00:00:00 2001 From: Frank945946 <108602632+Frank945946@users.noreply.github.com> Date: Mon, 25 Nov 2024 16:01:18 +0800 Subject: [PATCH 12/15] Update tidb-roadmap.md Co-authored-by: Grace Cai --- tidb-roadmap.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/tidb-roadmap.md b/tidb-roadmap.md index 968f5de2ae80e..1ad676e857149 100644 --- a/tidb-roadmap.md +++ b/tidb-roadmap.md @@ -51,7 +51,7 @@ In the course of development, this roadmap is subject to change based on user ne
          • Unlimited Transaction Size -
            Removes memory limits on uncommitted transactions, improving batch task success rates. +
            Removes memory limits on uncommitted transactions, improving success rates of transactions and batch tasks.

          • From fd573401b9f622e52ae71f8c208d1413614e33f9 Mon Sep 17 00:00:00 2001 From: Frank945946 <108602632+Frank945946@users.noreply.github.com> Date: Mon, 25 Nov 2024 16:52:06 +0800 Subject: [PATCH 13/15] Apply suggestions from code review Co-authored-by: Grace Cai --- tidb-roadmap.md | 40 ++++++++++++++++++++-------------------- 1 file changed, 20 insertions(+), 20 deletions(-) diff --git a/tidb-roadmap.md b/tidb-roadmap.md index 1ad676e857149..680fd5e882136 100644 --- a/tidb-roadmap.md +++ b/tidb-roadmap.md @@ -65,7 +65,7 @@ In the course of development, this roadmap is subject to change based on user ne
            • PD Microservice Router Service -
              Enables independent deployment, stateless (no Strong Leader) and easy scaling of Router Service (region meta query/updates) in PD, preventing PD from becoming a bottleneck for cluster resources. +
              Enables independent deployment, stateless operation (without a strong leader), and easy scaling of the PD router service (region metadata queries and updates), preventing PD from becoming a bottleneck for cluster resources.

            • @@ -74,8 +74,8 @@ In the course of development, this roadmap is subject to change based on user ne

            • - Limit Operator Pushdown -
              Removes limitations on pushing down the Limit operator from TiDB to TiKV, enabling more efficient query processing directly at the storage layer. + Limit Operator Pushdown +
              Removes limitations on pushing down the Limit operator from TiDB to TiKV, enabling more efficient query processing directly at the storage layer.

            • @@ -85,12 +85,12 @@ In the course of development, this roadmap is subject to change based on user ne
            • Scalable DDL Execution Framework -
              Provides a parallel DDL execution framework to improve the performance and stability of DDL operations. +
              Provides a scalable and parallel DDL execution framework to improve the performance and stability of DDL operations.

            • Increased TiCDC Scalability -
              Updated TiCDC architecture delivers increased scalability and performance for change data capture use cases. +
              Introduces a new TiCDC architecture that enhances scalability and performance for change data capture scenarios.

            @@ -108,7 +108,7 @@ In the course of development, this roadmap is subject to change based on user ne

          • - Enhance the Scalability of Region Meta Storage + Enhance the Scalability of Region Metadata Storage
            Migrate region meta storage from PD to TiKV, allowing the storage layer to scale infinitely and easily.

            • @@ -124,7 +124,7 @@ In the course of development, this roadmap is subject to change based on user ne
            • Vector Search -
              Enables vector data types, indexing, and high-performance vector search, with support for mixed queries involving vector and relational data. +
              Enables vector data types, indexing, and high-performance vector search, supporting mixed queries involving vector and relational data.

            • @@ -165,11 +165,11 @@ In the course of development, this roadmap is subject to change based on user ne
              • - Limit Memory for Backups + Limit Memory Usage for Backups

              • - Limit Memory for Statistics Collection (GA) + Limit Memory Usage for Statistics Collection (GA)

              • @@ -179,17 +179,17 @@ In the course of development, this roadmap is subject to change based on user ne
              • Improved Resource Group Control for Complex SQL (GA) -
                Monitors RU usage of complex queries mid-execution to minimize system impact. +
                Monitors RU usage during complex query execution to minimize system impact.

              • Automatic Resource Group Switching for Runaway Queries (GA) -
                Detects runaway queries and redirects them to designated resource groups with set limits. +
                Detects runaway queries and redirects them to designated resource groups with predefined limits.

              • Limit Memory Usage for Schema Metadata (GA) -
                Enhances stability in large-scale clusters by capping memory consumption for schema metadata. +
                Enhances stability in large-scale clusters by reducing memory consumption for schema metadata.

              @@ -203,7 +203,7 @@ In the course of development, this roadmap is subject to change based on user ne
            • Optimized Memory Management with Disk Spilling -
              Allows operators like HashAgg, Sort, and Join to spill to disk, reducing memory load and preventing out-of-memory (OOM) issues. +
              Allows operators such as HashAgg, Sort, and Join to spill to disk, reducing memory load and preventing out-of-memory (OOM) issues.

            • @@ -227,7 +227,7 @@ In the course of development, this roadmap is subject to change based on user ne
            • Enhanced Memory Protection -
              Monitors memory usage across all components to prevent operations that might impact system stability. +
              Monitors memory usage across all components to prevent operations that might affect system stability.

            • @@ -261,7 +261,7 @@ In the course of development, this roadmap is subject to change based on user ne

            • - Permissioned Resource Group Switching (GA) + Access Control for Resource Group Switching (GA)
              Restricts resource group switching to authorized users, preventing resource misuse.

              • @@ -276,7 +276,7 @@ In the course of development, this roadmap is subject to change based on user ne
              • Customizable Statistics Collection (GA) -
                Allows tailored statistics strategies for specific tables, adjusting parameters like health and parallelism. +
                Allows tailored statistics strategies for specific tables, adjusting parameters such as health and concurrency.

              • @@ -291,12 +291,12 @@ In the course of development, this roadmap is subject to change based on user ne
              • Standardized Time Model (GA) -
                Establishes a unified SQL execution time model to help identify database load sources through logs and cluster metrics, pinpointing problematic nodes and statements. +
                Establishes a unified SQL execution time model to help identify database load sources through statement summary tables, logs, and cluster metrics, pinpointing problematic nodes and statements.

              • TiFlash CPU Time Monitoring (GA) -
                Adds TiFlash CPU time metrics to statements and slow logs, enabling quick identification of statements that cause CPU spikes in TiFlash. +
                Adds TiFlash CPU time metrics to statement summary tables and slow logs, enabling quick identification of statements that cause CPU spikes in TiFlash.

              @@ -354,12 +354,12 @@ In the course of development, this roadmap is subject to change based on user ne
            • Kerberos Authentication (GA) -
              Enables authentication using Kerberos for added security. +
              Enables authentication using Kerberos for enhanced security.

            • Multi-Factor Authentication (MFA) -
              Adds support for multi-factor authentication to enhance user verification. +
              Supports multi-factor authentication to enhance user verification.

            • From 3c6e1a372ffabe7425f44f1cb6481a13cc9b1439 Mon Sep 17 00:00:00 2001 From: xixirangrang Date: Mon, 25 Nov 2024 23:00:00 +0800 Subject: [PATCH 14/15] Update tidb-roadmap.md --- tidb-roadmap.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/tidb-roadmap.md b/tidb-roadmap.md index 680fd5e882136..09106b7243c49 100644 --- a/tidb-roadmap.md +++ b/tidb-roadmap.md @@ -109,7 +109,7 @@ In the course of development, this roadmap is subject to change based on user ne
            • Enhance the Scalability of Region Metadata Storage -
              Migrate region meta storage from PD to TiKV, allowing the storage layer to scale infinitely and easily. +
              Splits a dedicated stateless Router Service (Read/Write of region metadata), and migrates region metadata storage from PD to TiKV. The metadata storage layer will scale limitless and easily.

            From ffd44b26cd6730244276248d8ddd421ddacf12dc Mon Sep 17 00:00:00 2001 From: xixirangrang Date: Tue, 26 Nov 2024 09:44:57 +0800 Subject: [PATCH 15/15] Update tidb-roadmap.md Co-authored-by: Grace Cai --- tidb-roadmap.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/tidb-roadmap.md b/tidb-roadmap.md index 09106b7243c49..9674eec35185c 100644 --- a/tidb-roadmap.md +++ b/tidb-roadmap.md @@ -109,7 +109,7 @@ In the course of development, this roadmap is subject to change based on user ne
          • Enhance the Scalability of Region Metadata Storage -
            Splits a dedicated stateless Router Service (Read/Write of region metadata), and migrates region metadata storage from PD to TiKV. The metadata storage layer will scale limitless and easily. +
            Splits a dedicated stateless router service (read/write of region metadata) and migrates region metadata storage from PD to TiKV. The metadata storage layer will scale limitlessly and easily.