Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Adds raw_header to MockRawResponse #6228

Closed
wants to merge 1 commit into from

Conversation

akshara08
Copy link

@akshara08 akshara08 commented Apr 18, 2023

Issue

The primary problem with moto integration with aiobotocore is this method. There are 2 main issues with that method

  1. It expects raw to have raw_headers just to convert the headers to lowercase.
  2. Moto passes MockRawResponse which is not awaitable causing the breaks with lines containing await.

How to reproduce the error

# test_response.py

@pytest.fixture
def aws_credentials():
    """Mocked AWS Credentials for moto."""
    os.environ["AWS_ACCESS_KEY_ID"] = "testing"
    os.environ["AWS_SECRET_ACCESS_KEY"] = "testing"
    os.environ["AWS_SECURITY_TOKEN"] = "testing"
    os.environ["AWS_SESSION_TOKEN"] = "testing"


@pytest.fixture
def s3(aws_credentials):
    with mock_s3():
        conn = boto3.resource("s3")
        conn.create_bucket(Bucket='testbucket')
        yield conn


def test_moto(s3):
    for i in range(3):
        s3.Bucket('testbucket').put_object(
            Key=f'glob_{i}.txt', Body=f"test glob file {i}"
        )
    path = 's3://testbucket/glob_*.txt'
    fs = s3fs.S3FileSystem()
    files = fs.glob(path) # Fails 
    assert len(files) == 3

Resolution

It's a two part solution

  1. The first part can be a addressed by adding raw_headers to MockRawResponse
  2. The second part of the problem can be addressed in this repo by only awaiting for awaitable objects. (PR)

I haven't written the tests because this needs to be merged for tests to pass. I was able to test locally and it looks fine

All the moto tests are otherwise passing and I'll add tests once the other PR is merged

@bblommers
Copy link
Collaborator

Hi @akshara08!

I don't want to make changes to Moto just because another open source project needs it for their code to work. That is frankly not our problem.

If you can make an argument that Moto's usage of AWSResponse does not follow the AWS spec/behaviour, a change is more likely to be accepted. What kind of class/argument does AWS pass to AWSResponse, instead of the MockRawResponse? Can we subclass it? Are there any other parameters/methods that Moto should consider?

@akshara08 akshara08 closed this Apr 18, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants