aiobotocore expects raw_headers - missing from batch job description response

[dazza-codes]

This might be too difficult to replicate. I'm basically asking for clarification about whether or not the raw_headers should exist in the moto response and when/why it might be missing. This might be better explained or replicated by using code noted in

• [test infra] switch to pytest-moto aio-libs/aiobotocore#665
• support moto client wrappers aio-libs/aiobotocore#755

When using moto mocks to test batch jobs with aiobotocore, ran into an exception like the following. This issue might belong on aiobotocore or here.

_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
/opt/conda/envs/gis-dataprocessing/lib/python3.7/asyncio/base_events.py:583: in run_until_complete
    return future.result()
jupiter/gis/async_aws_batch.py:147: in async_batch_job_status
    return await batch_client.describe_jobs(jobs=jobs)
/opt/conda/envs/gis-dataprocessing/lib/python3.7/site-packages/aiobotocore/client.py:89: in _make_api_call
    operation_model, request_dict, request_context)
/opt/conda/envs/gis-dataprocessing/lib/python3.7/site-packages/aiobotocore/client.py:110: in _make_request
    request_dict)
/opt/conda/envs/gis-dataprocessing/lib/python3.7/site-packages/aiobotocore/endpoint.py:73: in _send_request
    request, operation_model, context)
/opt/conda/envs/gis-dataprocessing/lib/python3.7/site-packages/aiobotocore/endpoint.py:106: in _get_response
    request, operation_model)
/opt/conda/envs/gis-dataprocessing/lib/python3.7/site-packages/aiobotocore/endpoint.py:154: in _do_get_response
    operation_model)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

http_response = <botocore.awsrequest.AWSResponse object at 0x7fce4d5f0150>, operation_model = OperationModel(name=DescribeJobs)

    async def convert_to_response_dict(http_response, operation_model):
        """Convert an HTTP response object to a request dict.
    
        This converts the requests library's HTTP response object to
        a dictionary.
    
        :type http_response: botocore.vendored.requests.model.Response
        :param http_response: The HTTP response from an AWS service request.
    
        :rtype: dict
        :return: A response dictionary which will contain the following keys:
            * headers (dict)
            * status_code (int)
            * body (string or file-like object)
    
        """
        response_dict = {
            # botocore converts keys to str, so make sure that they are in
            # the expected case. See detailed discussion here:
            # https://github.com/aio-libs/aiobotocore/pull/116
            # aiohttp's CIMultiDict camel cases the headers :(
            'headers': HTTPHeaderDict(
                {k.decode('utf-8').lower(): v.decode('utf-8')
>                for k, v in http_response.raw_headers}),
            'status_code': http_response.status_code,
            'context': {
                'operation_name': operation_model.name,
            }
        }
E       AttributeError: 'AWSResponse' object has no attribute 'raw_headers'

/opt/conda/envs/gis-dataprocessing/lib/python3.7/site-packages/aiobotocore/endpoint.py:43: AttributeError

[thehesiod]

moto should behave like the production AWS response. To properly debug this, run botocore against the production AWS system to see what happens in the same part of the code in botocore. I can look if I have some time. The raw_headers thing I think is supposed to come from aiohttp. This is probably an aiobotocore bug

[spulec]

@thehesiod should this be closed since aio-libs/aiobotocore#755 was closed or do you think this should still be open?

[dazza-codes]

This can be closed; it's specific to aiobotocore.