Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Smithy Identity auth refactor for Rest XML S3 client #3262

Open
wants to merge 31 commits into
base: main
Choose a base branch
from
Open

Conversation

sbera87
Copy link
Contributor

@sbera87 sbera87 commented Jan 20, 2025

Issue #, if available:

Description of changes:

Check all that applies:

  • Did a review by yourself.
  • Added proper tests to cover this PR. (If tests are not applicable, explain.)
  • Checked if this PR is a breaking (APIs have been changed) change.
  • Checked if this PR will not introduce cross-platform inconsistent behavior.
  • Checked if this PR would require a ReadMe/Wiki update.

Check which platforms you have built SDK on to verify the correctness of this PR.

  • Linux
  • Windows
  • Android
  • MacOS
  • IOS
  • Other Platforms

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

@sbera87 sbera87 marked this pull request as draft January 23, 2025 17:41
@sbera87 sbera87 changed the title Smithy Identity auth refactor for Rest XML clients [Work in Progress] Smithy Identity auth refactor for Rest XML client S3 [Work in Progress] Jan 30, 2025
# This is the 1st commit message:

s3 code gen updates

# This is the commit message #2:

fixes

# This is the commit message #3:

updates

# This is the commit message #4:

updates

# This is the commit message #5:

compiling version

# This is the commit message #6:

updates

# This is the commit message #7:

test changes

# This is the commit message #8:

updates

# This is the commit message #9:

temp changes in s3client

# This is the commit message #10:

for debug

# This is the commit message #11:

Update API model

# This is the commit message #12:

corrects the dual-stack endpoint configuration for cognitoidp
Added DeleteContactFlowVersion API and the CAMPAIGN flow type
AWS IoT SiteWise now supports ingestion and querying of Null (all data types) and NaN (double type) values of bad or uncertain data quality. New partial error handling prevents data loss during ingestion. Enabled by default for new customers; existing customers can opt-in.
Documentation-only update to address doc errors
Documentation-only update: clarified the description of the shareDecaySeconds parameter of the FairsharePolicy data type, clarified the description of the priority parameter of the JobQueueDetail data type.
Added `DigitGroupingStyle` in ThousandsSeparator to allow grouping by `LAKH`( Indian Grouping system ) currency. Support LAKH and `CRORE` currency types in Column Formatting.
This release adds support for the topic attribute FifoThroughputScope for SNS FIFO topics. For details, see the documentation history in the Amazon Simple Notification Service Developer Guide.
Increasing entryPoint in SparkSubmit to accept longer script paths. New limit is 4kb.

# This is the commit message #13:

upgrade smithy version (#3263)


# This is the commit message #14:

Refactor and simplify logic in sdksCommon cmake script to not rely on c2j model (#3251)


# This is the commit message #15:

Update API model

# This is the commit message #16:

Adds multi-turn input support for an Agent node in an Amazon Bedrock Flow
Docs Update for timeout changes
Rename WorkSpaces Web to WorkSpaces Secure Browser
AWS Elemental MediaLive adds a new feature, ID3 segment tagging, in CMAF Ingest output groups. It allows customers to insert ID3 tags into every output segment, controlled by a newly added channel schedule action Id3SegmentTagging.

# This is the commit message #17:

Fix serialization for query xml

# This is the commit message #18:

Update API model

# This is the commit message #19:

Added "future" allocation type for future dated capacity reservation

# This is the commit message #20:

add stable order for smoke tests + restoring client codegen (#3265)


# This is the commit message #21:

add protocol test models (clients+tests) to the repo
@@ -90,4 +90,5 @@ public Operation getOperationForRequestShapeName(String requestShapeName) {
ClientContextParams clientContextParams;
boolean useSmithyClient;
List<String> authSchemes;
String rawEndpointRules;
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is needed to parse at the service model level since otherwise only shortened rules get stored

@@ -836,4 +842,105 @@ private void addRequestlessRequestObjectS(final ServiceModel serviceModel) {
requestlessOperations.add(operation.getName());
});
}

//dfs
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Since this is the base class , added here so that it can be extended to any client code generator which needs multi auth

serviceModel.setAuthSchemes(authschemes);
}
//auth schemes can be named differently in endpoints/operations, this is a mapping
private static final Map<String, String> AuthSchemeNameMapping = ImmutableMap.of(
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

mapping is needed to simplify auth scheme lookup

m_httpClient->EnableRequestProcessing();
}

StreamOutcome AwsSmithyClientBase::MakeRequestWithUnparsedResponse(Aws::AmazonWebServiceRequest const * const request,
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Different outcome handling as legacy but reusing smithy functions

@@ -305,7 +305,7 @@ TEST_F(SmithyClientTest, testSigV4a) {

EXPECT_EQ(res2.IsSuccess(), true);

EXPECT_TRUE(res2.GetResult()->GetSigningAccessKey().empty());
EXPECT_TRUE(!res2.GetResult()->GetSigningAccessKey().empty());
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

previously, this was missing support which legacy had. Now added and tested

@@ -150,13 +150,37 @@ namespace
long CalculateDelayBeforeNextRetry(const AWSError<CoreErrors>&, long) const override { return 0; }
};

class S3TestClient : public S3Client
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This wrapper client is used instead because of use of protected methods in smithy client. Else will need to be exposed.

@sbera87 sbera87 marked this pull request as ready for review February 4, 2025 16:03
@sbera87 sbera87 changed the title Smithy Identity auth refactor for Rest XML client S3 [Work in Progress] Smithy Identity auth refactor for Rest XML client S3 Feb 4, 2025

AwsSmithyClientT& operator=(AwsSmithyClientT&&) = default;
AwsSmithyClientT& operator=(AwsSmithyClientT&& other) {
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is exposed by a test case in s3control and is needed due to the provided inline comments.

{
}

SigningFutureOutcome sign(std::shared_ptr<Aws::Http::HttpRequest> httpRequest, const AwsCredentialIdentityBase& identity, SigningProperties properties) override
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This file no longer needed to be auto generated as all the params are provided for signer resolution in the new smithy way that an handle v4a as well as v4 flavor of instantiation

@@ -196,6 +196,7 @@ public ServiceModel convert() {
shortenedRules += "\0";
serviceModel.setEndpointRules(shortenedRules);
}
serviceModel.setRawEndpointRules(c2jServiceModel.getEndpointRules() );
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

necessary because need raw endpoint rules to be parsed at service model level (which doesn't have hook to top level model)

@sbera87 sbera87 changed the title Smithy Identity auth refactor for Rest XML client S3 Smithy Identity auth refactor for Rest XML S3 client Feb 4, 2025
@@ -494,7 +550,7 @@ void AwsSmithyClientBase::HandleAsyncReply(std::shared_ptr<AwsSmithyClientAsyncR
if (retryWithCorrectRegion)
{
Aws::String newEndpoint = m_errorMarshaller->ExtractEndpoint(outcome.GetError());
if (newEndpoint.empty()) {
if (!newEndpoint.empty()) {
Copy link
Contributor Author

@sbera87 sbera87 Feb 4, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

bug fix. Typo error

{
//get signer Name from end point and pass this info
if (endpoint.GetAttributes()) {
auto authschemeName = endpoint.GetAttributes()->authScheme.GetName();
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Known properties needed as per original implementation. This is used in signing and credential resolution appropriately. The properties are maintained in a separate file for clarity

}

void AwsSmithyClientBase::AppendToUserAgent(const Aws::String& valueToAppend)
{
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

without this legacy test cases fail in s3 encryption

return;
}

List<String> authschemes = new ArrayList<>(serviceModel.getAuthSchemes());
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Needed to use a copy of list , otherwise it crashes for some services.

@@ -58,10 +57,6 @@ namespace Aws
bool useArnRegion = false;
Client::AWSAuthV4Signer::PayloadSigningPolicy payloadSigningPolicy = Client::AWSAuthV4Signer::PayloadSigningPolicy::RequestDependent;
bool disableS3ExpressAuth = false;
using IdentityProviderSupplier = std::function<std::shared_ptr<S3ExpressIdentityProvider> (const S3Client &)>;
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No longer needed here as the provider is passed in the smithy constructor

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

you cant remove this because it is part of the public interface i.e.

auto main() -> int {
  SDKOptions options{};
  InitAPI(options);
  {
    S3ClientConfiguration configuration{};
    configuration.identityProviderSupplier = 
      [](const S3Client &client) -> std::shared_ptr<S3ExpressIdentityProvider>  {
         //...
      }
    S3Client client{configuration};
  }
  ShutdownAPI(options);
  return 0;
}

that is valid code and our customers have been using that to override the identity provider.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ok, will restore this and reuse it internally

@@ -135,6 +136,13 @@ namespace client
Aws::Http::HttpMethod method,
EndpointUpdateCallback&& endpointCallback) const;

StreamOutcome MakeRequestWithUnparsedResponse(Aws::AmazonWebServiceRequest const * const request,
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

needed for some operations which dont parse the response. Check s3 client for usage

@@ -73,10 +84,11 @@ void AwsSmithyClientBase::baseCopyInit() {
createFromFactoriesIfPresent(m_clientConfig->readRateLimiter, m_clientConfig->configFactories.readRateLimiterCreateFn);
createFromFactoriesIfPresent(m_clientConfig->telemetryProvider, m_clientConfig->configFactories.telemetryProviderCreateFn);
if (m_clientConfig && m_clientConfig->retryStrategy) {
m_interceptors.emplace_back(Aws::MakeShared<UserAgentInterceptor>(AWS_SMITHY_CLIENT_LOG,
m_userAgentInterceptor = Aws::MakeShared<UserAgentInterceptor>(AWS_SMITHY_CLIENT_LOG,
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

non smithy client has this. Smithy client has vector of interceptors but we need the exact instance of derived class type useragent interceptor. looping to find out which one is the first useragent interceptor will involve casting. One simple approach is to just using one allocated resource and 2 shared ptrs pointing to it in the same object.
An alternative considered was just to store the index , but even that approach will need dynamic casting to child type. hence discarded.

{
AWS_LOGSTREAM_DEBUG(AWS_SMITHY_CLIENT_LOG, "Request returned error. Attempting to generate appropriate error codes from response");
assert(m_errorMarshaller);
auto error = m_errorMarshaller->BuildAWSError(httpResponse);
return HttpResponseOutcome(std::move(error));
}
else if (hasEmbeddedError()) {
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Follows legacy design. Without this face error marshalling test errors. Embedded error is handled differently that other errors. This invokes client overriden implementation which changes the underlying iostream ptrs as a result of which one fetching the response body more than once sets the fail bit BuildAWSError causes incorrect parsing of the error

#end
})
{}

/* Legacy constructors due deprecation */
#if($serviceModel.metadata.serviceId == "S3")
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

s3 has specific signature of legacy, hence need this

@@ -4,14 +4,22 @@
*/

#pragma once

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

added for readability

PropertyBag virtual signerProperties() const { return PropertyBag{}; };
PropertyBag virtual identityProperties() const { return m_identityProperties; };
PropertyBag virtual signerProperties() const { return m_signerProperties; };
void putIdentityProperty( const Aws::String& key,const Aws::Crt::Variant<Aws::String, bool>& value) {
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Need this now as the resolution interfaces accepts properties which will need all the parameters that was used previously but now simply respective property bag for identity or signer

Copy link
Contributor

@sbiscigl sbiscigl left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

high level feedback:

  • there are some backwards incompatible changes that we cant do
  • we shouldnt care about signer in identity provider, we should remove all references to signer in identity provider.
  • theres now s3-crt in here, we likely want to do that at the same time.
  • s3 components should be scoped to the s3 library and shouldnt leak into core

@@ -58,10 +57,6 @@ namespace Aws
bool useArnRegion = false;
Client::AWSAuthV4Signer::PayloadSigningPolicy payloadSigningPolicy = Client::AWSAuthV4Signer::PayloadSigningPolicy::RequestDependent;
bool disableS3ExpressAuth = false;
using IdentityProviderSupplier = std::function<std::shared_ptr<S3ExpressIdentityProvider> (const S3Client &)>;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

you cant remove this because it is part of the public interface i.e.

auto main() -> int {
  SDKOptions options{};
  InitAPI(options);
  {
    S3ClientConfiguration configuration{};
    configuration.identityProviderSupplier = 
      [](const S3Client &client) -> std::shared_ptr<S3ExpressIdentityProvider>  {
         //...
      }
    S3Client client{configuration};
  }
  ShutdownAPI(options);
  return 0;
}

that is valid code and our customers have been using that to override the identity provider.

@@ -1,71 +0,0 @@
/**
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

what if someone is including this header in their project right now? that would break them, i dont think you can simply remove these. i.e.

#include <aws/core/Aws.h>
#include <aws/s3/S3ExpressIdentity.h>

using namespace Aws;
using namespace Aws::S3;

auto main() -> int {
  SDKOptions options{};
  InitAPI(options);
  {
    S3ExpressIdentity indentity{};
  }
  ShutdownAPI(options);
  return 0;
}

return ResolveIdentityFutureOutcome();
}
auto creds = m_credsProvider->GetAWSCredentials();
return ResolveIdentityFutureOutcome(Aws::MakeUnique<AwsCredentialIdentity>("DefaultAwsCredentialIdentityResolver", AwsCredentialIdentity{creds.GetAWSAccessKeyId(), creds.GetAWSSecretKey(), creds.GetSessionToken(), creds.GetExpiration()}));
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: why are you using DefaultAwsCredentialIdentityResolver instead of S3_EXPRESS_IDENTITY_PROVIDER

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

COpy paste error. Will fix

}

//if signer name is not s3 express as set in signer properties, get from credential provider
auto signerName = params->parameterMap.find(smithy::AUTH_SCHEME_PROPERTY);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

i think search for the signer here isnt the intention of the SRA the S3ExpressIdentityResolver should not be interacting with signer at all. the s3 express identity provider should only be responsible for providing the identity for s3 express, i.e. from calling the connect API. before going into this rather long winded explanation you actually did get the general flow down, but i think we drifted from the goal.

at a very very very very high level the call flow is supposed to follow

struct Client {
public:
  http_response MakeRequest(http_request& request) {
      const auto schemes = auth_scheme_resolver_->resolveAuthScheme();
      const auto option = SelectFirstKnownAuthOption(schemes, auth_schemes_);
      auth_scheme_variant authScheme = auth_schemes_[option.schemeId];
      auto signOutcome = sign_request(request, authOption);
      auto http_response = http_client.send_request(signOutcome.http_request);
      return http_response;
  }
private:
  signing_outcome sign_request(http_request& request, const auth_option& option)
  {
      //...visit the variant
  }

  std::map<std::string, AuthOption> auth_schemes_;
  AuthSchemeResolverBase<>* auth_scheme_resolver_;
};

so in this case the two different variants of Identity that are returned are either AwsIdentity or S3 Express Identity. the signer does not need to know which one it is getting, it will just visit variants. the identity lives by itself.

at a very high level the calls are supposed to look like this

Screenshot 2025-02-04 at 3 56 12 PM

so translating this to our use. we dont care about signer in idenitity provider, we just provide identity.

i think this boils down to the idea of AuthSchemeOption and the "Choose Auth Scheme" where we declare a auth scheme as such

class AuthScheme
{
  public:
    virtual std::shared_ptr<IdentityResolverBase<IdentityT>> identityResolver() = 0;
    virtual std::shared_ptr<AwsSignerBase<IdentityT>> signer() = 0;
  };
}

ideally we should have something like

class S3ExpressIdentityProvider: public AuthScheme {
  public:
    std::shared_ptr<IdentityResolverBase<IdentityT>> identityResolver() override {
       // call s3 express identity provider
    }
    std::shared_ptr<AwsSignerBase<IdentityT>> signer() override {
       // call s3 express signer
     }
  };
};

which you did add

the when we "Choose Auth Scheme", i.e. here in the the smithy client we choose auth sceme option you correctly map it to different auth scheme options.

so why do we care about the signer here, the smithy client chooses the auth scheme, the identity provider just returns the identity.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ok, made the change in the next commit

@@ -98,9 +99,9 @@ namespace client
return *this;
}

AwsSmithyClientT (AwsSmithyClientT&&) = default;
AwsSmithyClientT (AwsSmithyClientT&& other) = default;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit why do we need to add "other" unused param for default?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

artifact of latest add and remove. Will restore


//Ensuring S3 Express Signer can use Sigv4 or Sigv4a signing algorithm
template <typename BASECLASS>
class S3ExpressSigner : public std::enable_if<IsValidS3ExpressSigner<BASECLASS>::value, BASECLASS>::type
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this should likely exist in the S3 library, since only the S3ExpressAuthScheme uses it, and its created in the constructor there, should live there, and only reference as we visit the auth scheme in the smithy client.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sure, since s3 name in it , I will relocate this back to where it was

namespace smithy {
constexpr char SIGV4_EXPRESS[] = "sigv4-s3express";

class S3ExpressSigV4AuthScheme : public AuthScheme<AwsCredentialIdentityBase>
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this should likely be in s3 and not in core. nothing in core directly calls this, it is all populated from constructor calls.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All auth schemes are in core already. Putting this in S3 is fine since s3 name in it


namespace smithy {
template<typename ServiceAuthSchemeParametersT = DefaultAuthSchemeResolverParameters>
class SigV4MultiAuthSchemeResolver : public AuthSchemeResolverBase<ServiceAuthSchemeParametersT>
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This should be codegenerated in S3 and called S3AuthResolver and passed in to the client at construction time.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is actually the geenric multi auth resolver. What i can do i I can add a s3 class which inherits from this for the s3 additional part. But this is needed in the core as the other generic resolvers are there.

@@ -209,7 +210,8 @@ bool AWSAuthV4Signer::SignRequestWithCreds(Aws::Http::HttpRequest& request, cons
0 /* expirationTimeInSeconds doesn't matter for HttpRequestViaHeaders */, Aws::Crt::Auth::SignatureType::HttpRequestViaHeaders);
}

if (!credentials.GetSessionToken().empty())
//additional check is needed for s3 header since this is invoked by s3 signer
if (!credentials.GetSessionToken().empty() && !request.HasHeader(S3_EXPRESS_HEADER))
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sigv4 signer should be unaware of the s3 express, when we choose a auth scheme, we choose the signer then, and if we do use s3 express signer, we just use it. It is the sigv4 signer with an extra header, so it should be the same as before

@@ -192,13 +206,20 @@ void AwsSmithyClientBase::MakeRequestAsync(Aws::AmazonWebServiceRequest const* c
auto epResolutionOutcome = this->ResolveEndpoint(std::move(epParams), std::move(endpointCallback));
if (!epResolutionOutcome.IsSuccess())
{
pExecutor->Submit([epResolutionOutcome, responseHandler]() mutable
auto epOutcome = ResolveEndpointOutcome(Aws::Client::AWSError<Aws::Client::CoreErrors>{
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

dont think we need to add this variable, looks like nothing was changed except adding a intermediate variable.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is needed as it was missing from smithy implementation which our test cases catch . This restores this error validation on endpoint resolution failure

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants