-
Notifications
You must be signed in to change notification settings - Fork 1.1k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Smithy Identity auth refactor for Rest XML S3 client #3262
base: main
Are you sure you want to change the base?
Conversation
# This is the 1st commit message: s3 code gen updates # This is the commit message #2: fixes # This is the commit message #3: updates # This is the commit message #4: updates # This is the commit message #5: compiling version # This is the commit message #6: updates # This is the commit message #7: test changes # This is the commit message #8: updates # This is the commit message #9: temp changes in s3client # This is the commit message #10: for debug # This is the commit message #11: Update API model # This is the commit message #12: corrects the dual-stack endpoint configuration for cognitoidp Added DeleteContactFlowVersion API and the CAMPAIGN flow type AWS IoT SiteWise now supports ingestion and querying of Null (all data types) and NaN (double type) values of bad or uncertain data quality. New partial error handling prevents data loss during ingestion. Enabled by default for new customers; existing customers can opt-in. Documentation-only update to address doc errors Documentation-only update: clarified the description of the shareDecaySeconds parameter of the FairsharePolicy data type, clarified the description of the priority parameter of the JobQueueDetail data type. Added `DigitGroupingStyle` in ThousandsSeparator to allow grouping by `LAKH`( Indian Grouping system ) currency. Support LAKH and `CRORE` currency types in Column Formatting. This release adds support for the topic attribute FifoThroughputScope for SNS FIFO topics. For details, see the documentation history in the Amazon Simple Notification Service Developer Guide. Increasing entryPoint in SparkSubmit to accept longer script paths. New limit is 4kb. # This is the commit message #13: upgrade smithy version (#3263) # This is the commit message #14: Refactor and simplify logic in sdksCommon cmake script to not rely on c2j model (#3251) # This is the commit message #15: Update API model # This is the commit message #16: Adds multi-turn input support for an Agent node in an Amazon Bedrock Flow Docs Update for timeout changes Rename WorkSpaces Web to WorkSpaces Secure Browser AWS Elemental MediaLive adds a new feature, ID3 segment tagging, in CMAF Ingest output groups. It allows customers to insert ID3 tags into every output segment, controlled by a newly added channel schedule action Id3SegmentTagging. # This is the commit message #17: Fix serialization for query xml # This is the commit message #18: Update API model # This is the commit message #19: Added "future" allocation type for future dated capacity reservation # This is the commit message #20: add stable order for smoke tests + restoring client codegen (#3265) # This is the commit message #21: add protocol test models (clients+tests) to the repo
@@ -90,4 +90,5 @@ public Operation getOperationForRequestShapeName(String requestShapeName) { | |||
ClientContextParams clientContextParams; | |||
boolean useSmithyClient; | |||
List<String> authSchemes; | |||
String rawEndpointRules; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This is needed to parse at the service model level since otherwise only shortened rules get stored
@@ -836,4 +842,105 @@ private void addRequestlessRequestObjectS(final ServiceModel serviceModel) { | |||
requestlessOperations.add(operation.getName()); | |||
}); | |||
} | |||
|
|||
//dfs |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Since this is the base class , added here so that it can be extended to any client code generator which needs multi auth
serviceModel.setAuthSchemes(authschemes); | ||
} | ||
//auth schemes can be named differently in endpoints/operations, this is a mapping | ||
private static final Map<String, String> AuthSchemeNameMapping = ImmutableMap.of( |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
mapping is needed to simplify auth scheme lookup
m_httpClient->EnableRequestProcessing(); | ||
} | ||
|
||
StreamOutcome AwsSmithyClientBase::MakeRequestWithUnparsedResponse(Aws::AmazonWebServiceRequest const * const request, |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Different outcome handling as legacy but reusing smithy functions
@@ -305,7 +305,7 @@ TEST_F(SmithyClientTest, testSigV4a) { | |||
|
|||
EXPECT_EQ(res2.IsSuccess(), true); | |||
|
|||
EXPECT_TRUE(res2.GetResult()->GetSigningAccessKey().empty()); | |||
EXPECT_TRUE(!res2.GetResult()->GetSigningAccessKey().empty()); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
previously, this was missing support which legacy had. Now added and tested
@@ -150,13 +150,37 @@ namespace | |||
long CalculateDelayBeforeNextRetry(const AWSError<CoreErrors>&, long) const override { return 0; } | |||
}; | |||
|
|||
class S3TestClient : public S3Client |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This wrapper client is used instead because of use of protected methods in smithy client. Else will need to be exposed.
|
||
AwsSmithyClientT& operator=(AwsSmithyClientT&&) = default; | ||
AwsSmithyClientT& operator=(AwsSmithyClientT&& other) { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This is exposed by a test case in s3control and is needed due to the provided inline comments.
{ | ||
} | ||
|
||
SigningFutureOutcome sign(std::shared_ptr<Aws::Http::HttpRequest> httpRequest, const AwsCredentialIdentityBase& identity, SigningProperties properties) override |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This file no longer needed to be auto generated as all the params are provided for signer resolution in the new smithy way that an handle v4a as well as v4 flavor of instantiation
@@ -196,6 +196,7 @@ public ServiceModel convert() { | |||
shortenedRules += "\0"; | |||
serviceModel.setEndpointRules(shortenedRules); | |||
} | |||
serviceModel.setRawEndpointRules(c2jServiceModel.getEndpointRules() ); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
necessary because need raw endpoint rules to be parsed at service model level (which doesn't have hook to top level model)
@@ -494,7 +550,7 @@ void AwsSmithyClientBase::HandleAsyncReply(std::shared_ptr<AwsSmithyClientAsyncR | |||
if (retryWithCorrectRegion) | |||
{ | |||
Aws::String newEndpoint = m_errorMarshaller->ExtractEndpoint(outcome.GetError()); | |||
if (newEndpoint.empty()) { | |||
if (!newEndpoint.empty()) { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
bug fix. Typo error
{ | ||
//get signer Name from end point and pass this info | ||
if (endpoint.GetAttributes()) { | ||
auto authschemeName = endpoint.GetAttributes()->authScheme.GetName(); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Known properties needed as per original implementation. This is used in signing and credential resolution appropriately. The properties are maintained in a separate file for clarity
} | ||
|
||
void AwsSmithyClientBase::AppendToUserAgent(const Aws::String& valueToAppend) | ||
{ |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
without this legacy test cases fail in s3 encryption
return; | ||
} | ||
|
||
List<String> authschemes = new ArrayList<>(serviceModel.getAuthSchemes()); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Needed to use a copy of list , otherwise it crashes for some services.
@@ -58,10 +57,6 @@ namespace Aws | |||
bool useArnRegion = false; | |||
Client::AWSAuthV4Signer::PayloadSigningPolicy payloadSigningPolicy = Client::AWSAuthV4Signer::PayloadSigningPolicy::RequestDependent; | |||
bool disableS3ExpressAuth = false; | |||
using IdentityProviderSupplier = std::function<std::shared_ptr<S3ExpressIdentityProvider> (const S3Client &)>; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
No longer needed here as the provider is passed in the smithy constructor
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
you cant remove this because it is part of the public interface i.e.
auto main() -> int {
SDKOptions options{};
InitAPI(options);
{
S3ClientConfiguration configuration{};
configuration.identityProviderSupplier =
[](const S3Client &client) -> std::shared_ptr<S3ExpressIdentityProvider> {
//...
}
S3Client client{configuration};
}
ShutdownAPI(options);
return 0;
}
that is valid code and our customers have been using that to override the identity provider.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
ok, will restore this and reuse it internally
@@ -135,6 +136,13 @@ namespace client | |||
Aws::Http::HttpMethod method, | |||
EndpointUpdateCallback&& endpointCallback) const; | |||
|
|||
StreamOutcome MakeRequestWithUnparsedResponse(Aws::AmazonWebServiceRequest const * const request, |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
needed for some operations which dont parse the response. Check s3 client for usage
@@ -73,10 +84,11 @@ void AwsSmithyClientBase::baseCopyInit() { | |||
createFromFactoriesIfPresent(m_clientConfig->readRateLimiter, m_clientConfig->configFactories.readRateLimiterCreateFn); | |||
createFromFactoriesIfPresent(m_clientConfig->telemetryProvider, m_clientConfig->configFactories.telemetryProviderCreateFn); | |||
if (m_clientConfig && m_clientConfig->retryStrategy) { | |||
m_interceptors.emplace_back(Aws::MakeShared<UserAgentInterceptor>(AWS_SMITHY_CLIENT_LOG, | |||
m_userAgentInterceptor = Aws::MakeShared<UserAgentInterceptor>(AWS_SMITHY_CLIENT_LOG, |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
non smithy client has this. Smithy client has vector of interceptors but we need the exact instance of derived class type useragent interceptor. looping to find out which one is the first useragent interceptor will involve casting. One simple approach is to just using one allocated resource and 2 shared ptrs pointing to it in the same object.
An alternative considered was just to store the index , but even that approach will need dynamic casting to child type. hence discarded.
{ | ||
AWS_LOGSTREAM_DEBUG(AWS_SMITHY_CLIENT_LOG, "Request returned error. Attempting to generate appropriate error codes from response"); | ||
assert(m_errorMarshaller); | ||
auto error = m_errorMarshaller->BuildAWSError(httpResponse); | ||
return HttpResponseOutcome(std::move(error)); | ||
} | ||
else if (hasEmbeddedError()) { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Follows legacy design. Without this face error marshalling test errors. Embedded error is handled differently that other errors. This invokes client overriden implementation which changes the underlying iostream ptrs as a result of which one fetching the response body more than once sets the fail bit BuildAWSError causes incorrect parsing of the error
#end | ||
}) | ||
{} | ||
|
||
/* Legacy constructors due deprecation */ | ||
#if($serviceModel.metadata.serviceId == "S3") |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
s3 has specific signature of legacy, hence need this
@@ -4,14 +4,22 @@ | |||
*/ | |||
|
|||
#pragma once | |||
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
added for readability
PropertyBag virtual signerProperties() const { return PropertyBag{}; }; | ||
PropertyBag virtual identityProperties() const { return m_identityProperties; }; | ||
PropertyBag virtual signerProperties() const { return m_signerProperties; }; | ||
void putIdentityProperty( const Aws::String& key,const Aws::Crt::Variant<Aws::String, bool>& value) { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Need this now as the resolution interfaces accepts properties which will need all the parameters that was used previously but now simply respective property bag for identity or signer
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
high level feedback:
- there are some backwards incompatible changes that we cant do
- we shouldnt care about signer in identity provider, we should remove all references to signer in identity provider.
- theres now s3-crt in here, we likely want to do that at the same time.
- s3 components should be scoped to the s3 library and shouldnt leak into core
@@ -58,10 +57,6 @@ namespace Aws | |||
bool useArnRegion = false; | |||
Client::AWSAuthV4Signer::PayloadSigningPolicy payloadSigningPolicy = Client::AWSAuthV4Signer::PayloadSigningPolicy::RequestDependent; | |||
bool disableS3ExpressAuth = false; | |||
using IdentityProviderSupplier = std::function<std::shared_ptr<S3ExpressIdentityProvider> (const S3Client &)>; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
you cant remove this because it is part of the public interface i.e.
auto main() -> int {
SDKOptions options{};
InitAPI(options);
{
S3ClientConfiguration configuration{};
configuration.identityProviderSupplier =
[](const S3Client &client) -> std::shared_ptr<S3ExpressIdentityProvider> {
//...
}
S3Client client{configuration};
}
ShutdownAPI(options);
return 0;
}
that is valid code and our customers have been using that to override the identity provider.
@@ -1,71 +0,0 @@ | |||
/** |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
what if someone is including this header in their project right now? that would break them, i dont think you can simply remove these. i.e.
#include <aws/core/Aws.h>
#include <aws/s3/S3ExpressIdentity.h>
using namespace Aws;
using namespace Aws::S3;
auto main() -> int {
SDKOptions options{};
InitAPI(options);
{
S3ExpressIdentity indentity{};
}
ShutdownAPI(options);
return 0;
}
return ResolveIdentityFutureOutcome(); | ||
} | ||
auto creds = m_credsProvider->GetAWSCredentials(); | ||
return ResolveIdentityFutureOutcome(Aws::MakeUnique<AwsCredentialIdentity>("DefaultAwsCredentialIdentityResolver", AwsCredentialIdentity{creds.GetAWSAccessKeyId(), creds.GetAWSSecretKey(), creds.GetSessionToken(), creds.GetExpiration()})); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
nit: why are you using DefaultAwsCredentialIdentityResolver
instead of S3_EXPRESS_IDENTITY_PROVIDER
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
COpy paste error. Will fix
} | ||
|
||
//if signer name is not s3 express as set in signer properties, get from credential provider | ||
auto signerName = params->parameterMap.find(smithy::AUTH_SCHEME_PROPERTY); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
i think search for the signer here isnt the intention of the SRA the S3ExpressIdentityResolver should not be interacting with signer at all. the s3 express identity provider should only be responsible for providing the identity for s3 express, i.e. from calling the connect API. before going into this rather long winded explanation you actually did get the general flow down, but i think we drifted from the goal.
at a very very very very high level the call flow is supposed to follow
struct Client {
public:
http_response MakeRequest(http_request& request) {
const auto schemes = auth_scheme_resolver_->resolveAuthScheme();
const auto option = SelectFirstKnownAuthOption(schemes, auth_schemes_);
auth_scheme_variant authScheme = auth_schemes_[option.schemeId];
auto signOutcome = sign_request(request, authOption);
auto http_response = http_client.send_request(signOutcome.http_request);
return http_response;
}
private:
signing_outcome sign_request(http_request& request, const auth_option& option)
{
//...visit the variant
}
std::map<std::string, AuthOption> auth_schemes_;
AuthSchemeResolverBase<>* auth_scheme_resolver_;
};
so in this case the two different variants of Identity that are returned are either AwsIdentity or S3 Express Identity. the signer does not need to know which one it is getting, it will just visit variants. the identity lives by itself.
at a very high level the calls are supposed to look like this
so translating this to our use. we dont care about signer in idenitity provider, we just provide identity.
i think this boils down to the idea of AuthSchemeOption
and the "Choose Auth Scheme" where we declare a auth scheme as such
class AuthScheme
{
public:
virtual std::shared_ptr<IdentityResolverBase<IdentityT>> identityResolver() = 0;
virtual std::shared_ptr<AwsSignerBase<IdentityT>> signer() = 0;
};
}
ideally we should have something like
class S3ExpressIdentityProvider: public AuthScheme {
public:
std::shared_ptr<IdentityResolverBase<IdentityT>> identityResolver() override {
// call s3 express identity provider
}
std::shared_ptr<AwsSignerBase<IdentityT>> signer() override {
// call s3 express signer
}
};
};
which you did add
the when we "Choose Auth Scheme", i.e. here in the the smithy client we choose auth sceme option you correctly map it to different auth scheme options.
so why do we care about the signer here, the smithy client chooses the auth scheme, the identity provider just returns the identity.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
ok, made the change in the next commit
@@ -98,9 +99,9 @@ namespace client | |||
return *this; | |||
} | |||
|
|||
AwsSmithyClientT (AwsSmithyClientT&&) = default; | |||
AwsSmithyClientT (AwsSmithyClientT&& other) = default; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
nit why do we need to add "other" unused param for default?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
artifact of latest add and remove. Will restore
|
||
//Ensuring S3 Express Signer can use Sigv4 or Sigv4a signing algorithm | ||
template <typename BASECLASS> | ||
class S3ExpressSigner : public std::enable_if<IsValidS3ExpressSigner<BASECLASS>::value, BASECLASS>::type |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
this should likely exist in the S3 library, since only the S3ExpressAuthScheme uses it, and its created in the constructor there, should live there, and only reference as we visit the auth scheme in the smithy client.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Sure, since s3 name in it , I will relocate this back to where it was
namespace smithy { | ||
constexpr char SIGV4_EXPRESS[] = "sigv4-s3express"; | ||
|
||
class S3ExpressSigV4AuthScheme : public AuthScheme<AwsCredentialIdentityBase> |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
this should likely be in s3 and not in core. nothing in core directly calls this, it is all populated from constructor calls.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
All auth schemes are in core already. Putting this in S3 is fine since s3 name in it
|
||
namespace smithy { | ||
template<typename ServiceAuthSchemeParametersT = DefaultAuthSchemeResolverParameters> | ||
class SigV4MultiAuthSchemeResolver : public AuthSchemeResolverBase<ServiceAuthSchemeParametersT> |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This should be codegenerated in S3 and called S3AuthResolver
and passed in to the client at construction time.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This is actually the geenric multi auth resolver. What i can do i I can add a s3 class which inherits from this for the s3 additional part. But this is needed in the core as the other generic resolvers are there.
@@ -209,7 +210,8 @@ bool AWSAuthV4Signer::SignRequestWithCreds(Aws::Http::HttpRequest& request, cons | |||
0 /* expirationTimeInSeconds doesn't matter for HttpRequestViaHeaders */, Aws::Crt::Auth::SignatureType::HttpRequestViaHeaders); | |||
} | |||
|
|||
if (!credentials.GetSessionToken().empty()) | |||
//additional check is needed for s3 header since this is invoked by s3 signer | |||
if (!credentials.GetSessionToken().empty() && !request.HasHeader(S3_EXPRESS_HEADER)) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Sigv4 signer should be unaware of the s3 express, when we choose a auth scheme, we choose the signer then, and if we do use s3 express signer, we just use it. It is the sigv4 signer with an extra header, so it should be the same as before
@@ -192,13 +206,20 @@ void AwsSmithyClientBase::MakeRequestAsync(Aws::AmazonWebServiceRequest const* c | |||
auto epResolutionOutcome = this->ResolveEndpoint(std::move(epParams), std::move(endpointCallback)); | |||
if (!epResolutionOutcome.IsSuccess()) | |||
{ | |||
pExecutor->Submit([epResolutionOutcome, responseHandler]() mutable | |||
auto epOutcome = ResolveEndpointOutcome(Aws::Client::AWSError<Aws::Client::CoreErrors>{ |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
dont think we need to add this variable, looks like nothing was changed except adding a intermediate variable.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This is needed as it was missing from smithy implementation which our test cases catch . This restores this error validation on endpoint resolution failure
Issue #, if available:
Description of changes:
Check all that applies:
Check which platforms you have built SDK on to verify the correctness of this PR.
By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.