Querying with resource token and the first level of a 2-level hierarchical partition key results in a 403 Forbidden

[wygoda]

This issue was first described here AzureCosmosDB/HierarchicalPartitionKeysFeedbackGroup#14 by @idg10 but the feedback group looks abandoned.

Description

Querying data on a container having a 2-level hierarchical partition key (like /customerName/documentId) with a CosmosClient
created with a resource token based on a single level permission (/customerName) results with a 403 Forbidden.

Repro

Create a container with a 2-level hierarchical partition key.
Insert test data using Azure Portal UI.

{
    "documentId": "1",
    "customerName": "unity",
    "name": "sample-project-1"
}
{
    "documentId": "2",
    "customerName": "unity",
    "name": "sample-project-2"
}
{
    "documentId": "3",
    "customerName": "notunity",
    "name": "sample-project-3"
}

Create a user and a permission for the first level of the hierarchical partition key.

var user = (await database.CreateUserAsync("unityuser")).User;
var response = await user.CreatePermissionAsync(
    new PermissionProperties(
        id: "unity-permission",
        permissionMode: PermissionMode.All,
        container: container,
        resourcePartitionKey: new PartitionKeyBuilder().Add("unity").Build()
    ));

Instantiate a CosmosClient using the permission.

var permission = await database.GetUser("unityuser").GetPermission("unity-permission").ReadAsync();
cosmosClient = new CosmosClient(endpoint, permission.Resource.Token);

Query the data using the CosmosClient.

var query = new QueryDefinition(
        "SELECT * FROM i WHERE i.customerName = @customerName")
    .WithParameter("@customerName", "unity");
var container = cosmosClient.GetContainer(dbid, containerId);
using FeedIterator<object> feed = container.GetItemQueryIterator<object>
    (query, null, new QueryRequestOptions { PartitionKey = new PartitionKeyBuilder().Add("unity").Build() });
while (feed.HasMoreResults)
{
    foreach (var item in await feed.ReadNextAsync())
    {
        Console.WriteLine(item);
    }
}

Expected behaviour

Documents having customerName equal to "unity" are logged to the console.

Actual behaviour

feed.ReadNextAsync() throws a CosmosException with message

Response status code does not indicate success: Forbidden (403); Substatus: 0; ActivityId: 9eed650a-5085-4f45-874f-aa66c6abc153; Reason: (Message: {"Errors":["Request is blocked. Please check your authorization token and Cosmos DB account firewall settings. Learn more: https:\/\/aka.ms\/cosmosdb-tsg-forbidden"]}
ActivityId: 9eed650a-5085-4f45-874f-aa66c6abc153, Request URI: /apps/bcd538a6-63b2-4d45-863d-545704915a4a/services/c66ab883-e7fb-45ba-8317-aee151873cd9/partitions/54b91e19-c6d1-40a9-bc0e-5117c841d0bf/replicas/133396737087664403s/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.19045 cosmos-netstandard-sdk/3.31.4);

Stack trace below.
I noticed I get the same result when I don't provide QueryRequestOptions at all or when I try with a LINQ + .ToFeedIterator().

Environment

SDK Version: 3.35.4

Stack trace

   at Microsoft.Azure.Documents.StoreResult.ToResponse(RequestChargeTracker requestChargeTracker)
   at Microsoft.Azure.Documents.ConsistencyReader.ReadSessionAsync(DocumentServiceRequest entity, ReadMode readMode)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.ExecuteRetryAsync[TParam,TPolicy](Func`1 callbackMethod, Func`3 callbackMethodWithParam, Func`2 callbackMethodWithPolicy, TParam param, IRetryPolicy retryPolicy, IRetryPolicy`1 retryPolicyWithArg, Func`1 inBackoffAlternateCallbackMethod, Func`2 inBackoffAlternateCallbackMethodWithPolicy, TimeSpan minBackoffForInBackoffCallback, CancellationToken cancellationToken, Action`1 preRetryCallback)
   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.ExecuteRetryAsync[TParam,TPolicy](Func`1 callbackMethod, Func`3 callbackMethodWithParam, Func`2 callbackMethodWithPolicy, TParam param, IRetryPolicy retryPolicy, IRetryPolicy`1 retryPolicyWithArg, Func`1 inBackoffAlternateCallbackMethod, Func`2 inBackoffAlternateCallbackMethodWithPolicy, TimeSpan minBackoffForInBackoffCallback, CancellationToken cancellationToken, Action`1 preRetryCallback)
   at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass31_0.<<InvokeAsync>b__0>d.MoveNext()
--- End of stack trace from previous location ---
   at Microsoft.Azure.Documents.RequestRetryUtility.ProcessRequestAsync[TRequest,IRetriableResponse](Func`1 executeAsync, Func`1 prepareRequest, IRequestRetryPolicy`2 policy, CancellationToken cancellationToken, Func`1 inBackoffAlternateCallbackMethod, Nullable`1 minBackoffForInBackoffCallback)
   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
   at Microsoft.Azure.Documents.RequestRetryUtility.ProcessRequestAsync[TRequest,IRetriableResponse](Func`1 executeAsync, Func`1 prepareRequest, IRequestRetryPolicy`2 policy, CancellationToken cancellationToken, Func`1 inBackoffAlternateCallbackMethod, Nullable`1 minBackoffForInBackoffCallback)
   at Microsoft.Azure.Documents.StoreClient.ProcessMessageAsync(DocumentServiceRequest request, CancellationToken cancellationToken, IRetryPolicy retryPolicy)
   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.ProcessMessageAsync(RequestMessage request, CancellationToken cancellationToken)
   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.SendAsync(RequestMessage request, CancellationToken cancellationToken)

[ramarag]

Thanks @wygoda for reporting this issue. The fix is on the service side, we have added to this our backlog. We have no ETA for the fix at this time and will update this issue when we have a fix.

[lnajaroen]

@wygoda As a workaround until the fix has been released, please specify the partition key at all levels to access the data.

//Create a user and a permission for the first level of the hierarchical partition key.
var user = (await database.CreateUserAsync("User")).User;
var response = await user.CreatePermissionAsync(
    new PermissionProperties(
        id: "1-1-permission",
        permissionMode: PermissionMode.All,
        container: container,
        resourcePartitionKey: new PartitionKeyBuilder().Add("1").Add("1").Build()
    ));

//Instantiate a CosmosClient using the permission.
var permission = await database.GetUser("firstLevelUser").GetPermission("1-1-permission").ReadAsync();
var cosmosClient2 = new CosmosClient(EndpointUri, permission.Resource.Token);

//Query the data using the CosmosClient2
queryDefinition = new QueryDefinition("SELECT * FROM i WHERE i.id = @id").WithParameter("@id", "1");
container = cosmosClient2.GetContainer(DatabaseName, ContainerName);
using FeedIterator<object> feed = container.GetItemQueryIterator<object>
    (queryDefinition, null, new QueryRequestOptions { PartitionKey = new PartitionKeyBuilder().Add("1").Add("1").Build() });
while (feed.HasMoreResults)
{
    foreach (var item in await feed.ReadNextAsync())
    {
        Console.WriteLine(item);
    }
}

[wygoda]

@wygoda As a workaround until the fix has been released, please specify the partition key at all levels to access the data.

@lnajaroen Thanks but this doesn't help much as it requires me to create permissions for every single document that I'm interested in and I'll have hundreds of new documents every day. Additionally it requires me to query by the id which I don't know at the time of querying.

[calebherne]

I have the same issue, and it seems like it will be a common one for any multi-tenant applications. The ability to partition by more than just the tenant, while also providing access at the tenant level would be extremely useful.