Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Issue 2854: add spliting read request in mirrror partition by device borders #2943

Open
wants to merge 5 commits into
base: main
Choose a base branch
from
Open

Issue 2854: add spliting read request in mirrror partition by device borders #2943

wants to merge 5 commits into from

Conversation

vladstepanyuk
Copy link
Contributor

@vladstepanyuk vladstepanyuk commented Jan 29, 2025 •
edited
Loading

#2854
Пришлось в тесте TMirrorPartitionTest::ShouldTryToSplitReadRequest
и в лоад тестах

nbs/cloud/blockstore/tests/loadtest/local-mirror/test.py

Lines 87 to 103 in 9686d14

# There are non-intuitive replica configuration permutations during migration,
# that's why we should run tests with different fresh device topologies.
# See issue-2854
TestCase(
"mirror2-split-read-cross",
"cloud/blockstore/tests/loadtest/local-mirror/local-mirror2-split-read-cross.txt",
agent_count=6,
dump_block_digests=True,
max_migration_bandwidth=1
),
TestCase(
"mirror2-split-read-column",
"cloud/blockstore/tests/loadtest/local-mirror/local-mirror2-split-read-column.txt",
agent_count=6,
dump_block_digests=True,
max_migration_bandwidth=1
),

Перебрать разные топологии свежих девайсов из-за изменений в ReplicasInfo, которые происходят при налитии свежих девайсов в этой функции

nbs/cloud/blockstore/libs/storage/partition_nonrepl/part_mirror_state.cpp

Line 129 in 8dd2375

bool TMirrorPartitionState::PrepareMigrationConfigForFreshDevices()

@github-actions GitHub Actions
Copy link
Contributor

Hi! Thank you for contributing!
The tests on this PR will run after a maintainer adds an ok-to-test label to this PR manually. Thank you for your patience!

vladstepanyuk
vladstepanyuk commented Jan 29, 2025
View reviewed changes
cloud/blockstore/libs/storage/partition_nonrepl/part_mirror_split_request_helpers.cpp Outdated
for (size_t i = 0; i < blockRangeSplittedByDeviceBorders.size(); ++i) {
TSgList newSglist;
auto rangeSizeLeft = blockRangeSplittedByDeviceBorders[i].Size() *
originalRequest.BlockSize;
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@komarevtsev-d стоит может брать BlockSize из конфига, а не реквеста?

@komarevtsev-d komarevtsev-d added blockstore Add this label to run only cloud/blockstore build and tests on PR large-tests Launch large tests for PR ok-to-test Label to approve test launch for external members labels Jan 29, 2025
@github-actions github-actions bot removed the ok-to-test Label to approve test launch for external members label Jan 29, 2025
@komarevtsev-d komarevtsev-d added the ok-to-test Label to approve test launch for external members label Jan 29, 2025
@github-actions github-actions bot removed the ok-to-test Label to approve test launch for external members label Jan 29, 2025
@komarevtsev-d komarevtsev-d added the ok-to-test Label to approve test launch for external members label Jan 29, 2025
@github-actions github-actions bot removed the ok-to-test Label to approve test launch for external members label Jan 29, 2025
@vladstepanyuk vladstepanyuk changed the title Issue 2854: split read request if we can''t read it as one because of fresh devices Issue 2854: split read request in mirrror partition Jan 29, 2025
@vladstepanyuk vladstepanyuk changed the title Issue 2854: split read request in mirrror partition Issue 2854: add spliting read request in mirrror partition by device borders Jan 29, 2025
@komarevtsev-d komarevtsev-d added the ok-to-test Label to approve test launch for external members label Jan 29, 2025
@github-actions github-actions bot removed the ok-to-test Label to approve test launch for external members label Jan 29, 2025
@github-actions GitHub Actions
Copy link
Contributor

Note

This is an automated comment that will be appended during run.

🟢 linux-x86_64-relwithdebinfo: all tests PASSED for commit f159c02.

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
3797 3797 0 0 0 0

@komarevtsev-d komarevtsev-d added the ok-to-test Label to approve test launch for external members label Jan 30, 2025
@github-actions github-actions bot removed the ok-to-test Label to approve test launch for external members label Jan 30, 2025
@github-actions GitHub Actions
Copy link
Contributor

Note

This is an automated comment that will be appended during run.

🟢 linux-x86_64-relwithdebinfo: all tests PASSED for commit c9baed2.

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
3797 3797 0 0 0 0

@komarevtsev-d komarevtsev-d self-requested a review February 4, 2025 17:02
komarevtsev-d
komarevtsev-d reviewed Feb 6, 2025
View reviewed changes
cloud/blockstore/tools/testing/loadtest/protos/loadtest.proto

message TDeviceCoordinate {
uint64 ReplicaIndex = 1;
uint64 DeviceIndex = 2;
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Сделай комент какой-нибудь что за индексы.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

порядковый номер реплики, порядковый номер девайса?

cloud/blockstore/libs/storage/partition_nonrepl/part_mirror_actor_readblocks.cpp Outdated Show resolved Hide resolved
cloud/blockstore/libs/storage/partition_nonrepl/part_mirror_state.cpp Outdated Show resolved Hide resolved
cloud/blockstore/libs/storage/partition_nonrepl/part_mirror_actor_readblocks.cpp Outdated
auto splittedRequest = SplitRequest<TMethod>(
record,
blockRangeSplittedByDeviceBorders,
actorIdsForRequests);
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Есть предложение: чтобы тут не наводить шаблонизацию, можно забить и делать запрос ReadBlocks всегда, в независимости какой запрос пришел реально.

Copy link
Contributor Author

@vladstepanyuk vladstepanyuk Feb 6, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Тут и так есть шаблонизация в методе ReadBlocks и TRequestActor, плюс, если из любого запроса делать readblocks, то, учитывая, что методы ReadBlocks и RequestActor и так шаблонные, придется все равно же в зависимости от метода выполнять различную логику — для локального метода прикапывать буферы в акторе и отдельный метод, который их заполнит, только, как мне кажется, это будет выглядеть даже хуже. Как будто лучше, когда у нас актор TSplittedActor с одинаковой логикой для всех методов и всего 2 шаблонные функции SplitRequest/UnifyResponses, которые уже в зависимости от метода по-своему распиливают реквесты/объединяют респонсы.

cloud/blockstore/libs/storage/partition_nonrepl/part_mirror_split_request_helpers.cpp Show resolved Hide resolved
@komarevtsev-d komarevtsev-d added the ok-to-test Label to approve test launch for external members label Feb 6, 2025
@github-actions github-actions bot removed the ok-to-test Label to approve test launch for external members label Feb 6, 2025
@github-actions GitHub Actions
Copy link
Contributor

github-actions bot commented Feb 6, 2025

Note

This is an automated comment that will be appended during run.

🔴 linux-x86_64-relwithdebinfo: some tests FAILED for commit 29c4671.

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
3780 3743 0 37 0 0

@sharpeye sharpeye self-requested a review February 6, 2025 19:54
@komarevtsev-d komarevtsev-d added the ok-to-test Label to approve test launch for external members label Feb 7, 2025
@github-actions github-actions bot removed the ok-to-test Label to approve test launch for external members label Feb 7, 2025
@github-actions GitHub Actions
Copy link
Contributor

github-actions bot commented Feb 7, 2025

Note

This is an automated comment that will be appended during run.

🔴 linux-x86_64-relwithdebinfo: some tests FAILED for commit 474253d.

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
3797 3796 0 1 0 0

@komarevtsev-d komarevtsev-d added the ok-to-test Label to approve test launch for external members label Feb 10, 2025
@github-actions github-actions bot removed the ok-to-test Label to approve test launch for external members label Feb 10, 2025
@github-actions GitHub Actions
Copy link
Contributor

Note

This is an automated comment that will be appended during run.

🔴 linux-x86_64-relwithdebinfo: some tests FAILED for commit bf4182b.

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
3829 3826 0 3 0 0

@komarevtsev-d komarevtsev-d added the ok-to-test Label to approve test launch for external members label Feb 10, 2025
@github-actions github-actions bot removed the ok-to-test Label to approve test launch for external members label Feb 10, 2025
@github-actions GitHub Actions
Copy link
Contributor

Note

This is an automated comment that will be appended during run.

🔴 linux-x86_64-relwithdebinfo: some tests FAILED for commit fac4611.

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
3829 3828 0 1 0 0

sharpeye
sharpeye reviewed Feb 10, 2025
View reviewed changes
cloud/blockstore/libs/storage/partition_nonrepl/part_mirror_split_request_helpers.h
Comment on lines +44 to +62
template <typename TMethod>
std::optional<TSplittedRequest<TMethod>> SplitRequest(
const TRequestRecordType<TMethod>& originalRequest,
std::span<const TBlockRange64> blockRangeSplittedByDeviceBorders)
{
if constexpr (std::is_same_v<TMethod, TEvService::TReadBlocksMethod>) {
return SplitRequestRead(
originalRequest,
blockRangeSplittedByDeviceBorders);
} else if constexpr (
std::is_same_v<TMethod, TEvService::TReadBlocksLocalMethod>)
{
return SplitRequestReadLocal(
originalRequest,
blockRangeSplittedByDeviceBorders);
} else {
return {};
}
}
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Достаточно двух функций, перегрузка всё сделает:

auto SplitReadBlocksRequest(
    const NProto::TReadBlocksRequest& request,
    std::span<TBlockRange64> ranges)
    -> TSplittedRequest<TEvService::TReadBlocksMethod>;

auto SplitReadBlocksRequest(
    const NProto::TReadBlocksLocalRequest& request,
    std::span<TBlockRange64> ranges)
    -> TSplittedRequest<TEvService::TReadBlocksLocalMethod>;

cloud/blockstore/libs/storage/partition_nonrepl/part_mirror_split_request_actor.h
Comment on lines +44 to +62
, Requests(std::move(requests))
, ParentActorId(parentActorId)
, BlockSize(blockSize)
, RequestIdentityKey(requestIdentityKey)
{}

void Bootstrap(const NActors::TActorContext& ctx)
{
Responses.resize(Requests.size());
for (size_t i = 0; i < Requests.size(); ++i) {
Responses[i].BlocksCountRequested =
Requests[i].BlockRangeForRequest.Size();
auto req = std::make_unique<typename TMethod::TRequest>();
req->Record = std::move(Requests[i].Request);
NCloud::Send(
ctx,
ParentActorId,
std::move(req),
RequestInfo->Cookie + i + 1);
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Почему так?

cloud/blockstore/libs/storage/partition_nonrepl/part_mirror_actor.h
Comment on lines +71 to +72
TRequestsInProgress<ui64, TBlockRange64> ReadRequestsInProgress{
EAllowedRequests::ReadOnly};
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Это изменение само по себе не достойно отдельного PR?

cloud/blockstore/libs/storage/partition_nonrepl/part_mirror_actor_readblocks.cpp
@@ -472,31 +478,85 @@ void TMirrorPartitionActor::ReadBlocks(
const auto replicaIndex = record.GetHeaders().GetReplicaIndex();
auto [replicaActorIds, error] =
SelectReplicasToReadFrom(replicaIndex, blockRange, TMethod::Name);
if (!HasError(error)) {
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Лучше оставить старый порядок:

if (HasError(error)) { ... распиливаем запрос ... }
... обычный запрос ...

cloud/blockstore/libs/storage/partition_nonrepl/part_mirror_split_request_actor.h
ui32 PendingRequests = 0;
TVector<TUnifyResponsesContext<TMethod>> Responses;

using TResponseProto = typename TMethod::TResponse::ProtoRecordType;
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

TRequestRecordType ?

cloud/blockstore/libs/storage/partition_nonrepl/part_mirror_split_request_actor.h
private:
STFUNC(StateWork)
{
TRequestScope timer(*RequestInfo);
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Зачем тут таймер?

cloud/blockstore/libs/storage/partition_nonrepl/part_mirror_split_request_actor.h
namespace NCloud::NBlockStore::NStorage::NSplitRequest {

template <typename TMethod>
class TSplittedRequestActor final
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

TSplitReadBlocksActor

cloud/blockstore/libs/storage/partition_nonrepl/part_mirror_actor_readblocks.cpp
return;
}

auto blockRangeSplittedByDeviceBorders =
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

blockRangeSplitByDeviceBorders

cloud/blockstore/libs/storage/partition_nonrepl/part_mirror_actor_readblocks.cpp
}
}

auto splittedRequest = SplitRequest<TMethod>(
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

splitRequests

cloud/blockstore/libs/storage/partition_nonrepl/part_mirror_split_request_helpers.cpp
Comment on lines +67 to +70
if (blocks.BuffersSize() == 0) {
for (size_t i = 0; i < blocksCountRequested; ++i) {
result.MutableBlocks()->AddBuffers(TString(blockSize, '\0'));
}
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Это когда в одном из ответов allZeros, поэтому прихрдится добавлять на место этого запроса нули в итоговом ответе?

@sharpeye
Copy link
Collaborator

Нужно описание, что происходит в этом PR

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@sharpeye sharpeye sharpeye left review comments

@komarevtsev-d komarevtsev-d Awaiting requested review from komarevtsev-d

At least 2 approving reviews are required to merge this pull request.

Assignees
No one assigned
Labels
blockstore Add this label to run only cloud/blockstore build and tests on PR large-tests Launch large tests for PR
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@vladstepanyuk @sharpeye @komarevtsev-d