Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix:when the shard is created, disk iops is high, causing http write … #24584

Open
wants to merge 1 commit into
base: master-1.x
Choose a base branch
from

Conversation

lzle
Copy link

@lzle lzle commented Jan 19, 2024

I have identified an issue with the InfluxDB service that occurs daily at midnight. During the creation of shards each day (shardGroupDuration=24h), the disk IO usage spikes to 100%, the number of goroutines increases to 200k, and occasionally leads to system OOM. This results in HTTP write requests failing due to timeouts, significantly impacting the service's usability.

Upon analysis and tracing, I have pinpointed the root cause of the issue. In the log_file.AddSeriesList function, if the seriesID of a point is not present in the seriesSet, it triggers the f.FlushAndSync() operation, causing a significant increase in disk IOPS. I suggest increasing the probability of performing this operation from the current 10%, allowing more flush operations to be handled by the operating system to alleviate disk pressure.

System: CentOS Linux release 7.9.2009 (Core)
Version: InfluxDB1.8.6

Please help me review the code and give suggestions!

Closes #

Describe your proposed changes here.

  • I've read the contributing section of the project README.
  • Signed CLA (if not already signed).

@AlbertasB
Copy link

Could you add the same change to 2.x? Since this sounds awfully familiar issue that is happening in 2.x and none one can fix it so far. #23956

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants