Getting 403 for get properties

[stasasekulic]

Describe the bug
Setup:
Storage account, type Datalake with hierarchical namespace ON
Two private endpoints, one with sub resource type BLOB, other DFS
Using client secret authentication
Storage Blob Data Contributor role added in IAM
Test:
Try to get properties of blob using DataLakeFileSystemClient
Result:
Works fine

Setup:
Storage account, type Datalake with hierarchical namespace ON
One private endpoints, BLOB or DFS
Using client secret authentication
Storage Blob Data Contributor role added in IAM
Test:
Try to get properties of blob using DataLakeFileSystemClient
Result:
403
Exception or Stack Trace
Add the exception log and stack trace if available

To Reproduce
Steps to reproduce the behavior:
Create storage account Data Lake Storage with hierarchical namespace Enabled
Disable public access
Add two private endpoints
Try getting properties using DataLakeFileSystemClient
Remove either endpoint

Code Snippet
`// Copyright (c) Microsoft Corporation.
// Licensed under the MIT License.

#include <azure/identity/client_secret_credential.hpp>
#include <azure/storage/files/datalake.hpp>

#include
#include
#include
#include <unordered_map>

// Function to load configuration from a file
std::unordered_map<std::string, std::string> LoadConfig(const std::string& filename)
{
std::unordered_map<std::string, std::string> config;
std::ifstream file(filename);
if (!file)
{
throw std::runtime_error("Failed to open config file: " + filename);
}

std::string line;
while (std::getline(file, line))
{
    auto pos = line.find('=');
    if (pos != std::string::npos)
    {
        std::string key = line.substr(0, pos);
        std::string value = line.substr(pos + 1);
        config[key] = value;
    }
}
return config;

}

// Function to read directory contents from Azure Data Lake Storage
void ReadDataLakeDirectory(
const Azure::Storage::Files::DataLake::DataLakeFileSystemClient& dataLakeClient,
const std::string& directoryName)
{
try
{
auto directoryClient = dataLakeClient.GetDirectoryClient(directoryName);
auto listDFSOptions = Azure::Storage::Files::DataLake::ListPathsOptions();
auto paths = directoryClient.ListPaths(true, listDFSOptions);

    std::cout << "DFS Reading" << std::endl;
    for (const auto& path : paths.Paths)
    {
        std::cout << "******************" << std::endl;
        std::cout << path.Name << std::endl;
    }
}
catch (const Azure::Storage::StorageException& e)
{
    std::cerr << "Error reading Data Lake directory: " << e.what() << std::endl;
}

}

// Function to fetch file properties from Azure Data Lake Storage
void ReadDataLakeProperties(
const Azure::Storage::Files::DataLake::DataLakeFileSystemClient& dataLakeClient,
const std::string& directoryName)
{
try
{
std::cout << "Fetching DFS Properties..." << std::endl;
auto fileClient = dataLakeClient.GetFileClient(directoryName);
auto props = fileClient.GetProperties();
std::cout << "Type: " << (props.Value.IsDirectory ? "Directory" : "File") << std::endl;
std::cout << "DFS Properties Retrieved" << std::endl;
}
catch (const Azure::Storage::StorageException& e)
{
std::cerr << "Error fetching Data Lake properties: " << e.what() << std::endl;
}
}

// Function to read directory contents from Azure Blob Storage
void ReadBlobDirectory(
const Azure::Storage::Blobs::BlobContainerClient& containerClient)
{
try
{
auto listBlobOptions = Azure::Storage::Blobs::ListBlobsOptions();
auto blobPages = containerClient.ListBlobsByHierarchy("", listBlobOptions);

    std::cout << "Blob Reading" << std::endl;
    for (const auto& blobItem : blobPages.Blobs)
    {
        std::cout << "******************" << std::endl;
        std::cout << blobItem.Name << std::endl;
    }
}
catch (const Azure::Storage::StorageException& e)
{
    std::cerr << "Error reading Blob directory: " << e.what() << std::endl;
}

}

// Function to fetch file properties from Azure Blob Storage
void ReadBlobProperties(
const Azure::Storage::Blobs::BlobContainerClient& containerClient,
const std::string& directoryName)
{
try
{
std::cout << "Fetching Blob Properties..." << std::endl;
auto blobClient = containerClient.GetBlobClient(directoryName);
auto props = blobClient.GetProperties();
std::cout << "Blob size"+ props.Value.BlobSize << std::endl;
std::cout << "Blob Properties Retrieved" << std::endl;
}
catch (const Azure::Storage::StorageException& e)
{
std::cerr << "Error fetching Blob properties: " << e.what() << std::endl;
}
}

int main()
{
using namespace Azure::Storage::Files::DataLake;
using namespace Azure::Storage::Blobs;

// Load configuration
auto config = LoadConfig("../config");

const std::string containerName = config["containerName"];
const std::string directoryName = config["directoryName"];
const std::string tenantId = config["tenantId"];
const std::string clientId = config["clientId"];
const std::string clientSecret = config["clientSecret"];
const std::string storageAccountUrlDFS = config["storageAccountUrlDFS"];
const std::string storageAccountUrlBlob = config["storageAccountUrlBlob"];

auto clientSecretCredential = std::make_shared<Azure::Identity::ClientSecretCredential>(tenantId, clientId, clientSecret);

try
{
    auto datalakeOptions = DataLakeClientOptions();
    auto dataLakeClient = DataLakeFileSystemClient(storageAccountUrlDFS + containerName, clientSecretCredential, datalakeOptions);
    
    ReadDataLakeDirectory(dataLakeClient, directoryName);
    ReadDataLakeProperties(dataLakeClient, directoryName);
}
catch (const Azure::Storage::StorageException& e)
{
    std::cerr << "Error with Data Lake operations: " << e.what() << std::endl;
}

try
{
    auto blobClientOptions = BlobClientOptions();
    auto blobServiceClient = BlobServiceClient(storageAccountUrlBlob, clientSecretCredential, blobClientOptions);
    auto containerClient = blobServiceClient.GetBlobContainerClient(containerName);

    ReadBlobProperties(containerClient, directoryName);
    ReadBlobDirectory(containerClient);
}
catch (const Azure::Storage::StorageException& e)
{
    std::cerr << "Error with Blob operations: " << e.what() << std::endl;
}

return 0;

}
`

Expected behavior
Get properties works when user uses DataLakeFileSystemClient and have DFS endpoint
Screenshots
If applicable, add screenshots to help explain your problem.

Setup (please complete the following information):

• OS: Linux
• IDE : VS Code
• Version of the Library used

Additional context
Add any other context about the problem here.

Information Checklist
Kindly make sure that you have added all the following information above and checkoff the required fields otherwise we will treat the issuer as an incomplete report

• Bug Description Added
• Repro Steps Added
• Setup information Added

[github-actions]

Thanks for the feedback! We are routing this to the appropriate team for follow-up. cc @Jinming-Hu @microzchang @vinjiang.

[microzchang]

Hi, @stasasekulic

403 is usually because you don't have the correct permission for the specific operation. It's not a sdk issue.

From your code, I saw you were using token based authorization to access, so you can refer to this doc and see if you have assigned the correct permissions:

https://learn.microsoft.com/en-us/rest/api/storageservices/authorize-with-azure-active-directory

And if you think the permission is correct, please review if the network rule is correct, the network failure can also lead to 403.

For further investigation, it's better to provide your failure request-id in the error message and timestamp and storage account name.

Thanks.

[stasasekulic]

And if you think the permission is correct, please review if the network rule is correct, the network failure can also lead to 403.
Could you please explain to me how to check if network rule is correct?

I have storage blob data contributor role assigned to the storage account.

I am convinced that props are dependent on the private endpoint BLOB type somehow, or however is that done under the hood.
If you create private endpoint with type DFS and try to get properties, it will fail.
But if you create private endpoint with type BLOB and try to get properties, same code will work.

       auto clientSecretCredential = std::make_shared<Azure::Identity::ClientSecretCredential>(tenantId, clientId, clientSecret);
       auto datalakeOptions = DataLakeClientOptions();
       auto dataLakeClient = DataLakeFileSystemClient("https://lalalala.dfs.core.windows.net/test-container", clientSecretCredential, datalakeOptions);
        auto fileClient = dataLakeClient.GetFileClient(directoryName);
        auto props = fileClient.GetProperties();

[microzchang]

And if you think the permission is correct, please review if the network rule is correct, the network failure can also lead to 403. Could you please explain to me how to check if network rule is correct?

I have storage blob data contributor role assigned to the storage account.

I am convinced that props are dependent on the private endpoint BLOB type somehow, or however is that done under the hood. If you create private endpoint with type DFS and try to get properties, it will fail. But if you create private endpoint with type BLOB and try to get properties, same code will work.

       auto clientSecretCredential = std::make_shared<Azure::Identity::ClientSecretCredential>(tenantId, clientId, clientSecret);
       auto datalakeOptions = DataLakeClientOptions();
       auto dataLakeClient = DataLakeFileSystemClient("https://lalalala.dfs.core.windows.net/test-container", clientSecretCredential, datalakeOptions);
        auto fileClient = dataLakeClient.GetFileClient(directoryName);
        auto props = fileClient.GetProperties();

Hi, @stasasekulic

I'm not the expert of the network area so I can't directly tell you the network rules, but you can refer to the doc:
Configure Azure Storage firewalls and virtual networks and Use private endpoints for Azure Storage

And only for 403, I can't know what's the failure reason of your request. Can you help provide the x-ms-request-id in response header of your 403 request so that I can get the detail server logs and detect the failure reason.

[stasasekulic]

I think it would be better if you would reproduce the issue, it would give you better insight what is the problem.

1. Create Azure VM

2. Create storage account

3. Disable public access on the storage account

4. Add one private endpoint of sub resource type DFS

5. Run the functions from dummy azure app that is on Azure VM from step 1.
   auto clientSecretCredential = std::make_shared<Azure::Identity::ClientSecretCredential>(tenantId, clientId, clientSecret); auto datalakeOptions = DataLakeClientOptions(); auto dataLakeClient = DataLakeFileSystemClient("https://lalalala.dfs.core.windows.net/test-container", clientSecretCredential, datalakeOptions); auto fileClient = dataLakeClient.GetFileClient(directoryName); auto props = fileClient.GetProperties();

6. They will fail, and they should work....

If you add another private endpoint of sub resource type BLOB, everything will work, which means that DFS endpoints are dependable on BLOB permissions...