[HiveConnector] How to support FileSystemConfiguration at run-time?

[skadilover]

HiveConnector::HiveConnector(
    const std::string& id,
    std::shared_ptr<const Config> config,
    folly::Executor* executor)
    : Connector(id),
      hiveConfig_(std::make_shared<HiveConfig>(config)),
      fileHandleFactory_(
          hiveConfig_->isFileHandleCacheEnabled()
              ? std::make_unique<
                    SimpleLRUCache<std::string, std::shared_ptr<FileHandle>>>(
                    hiveConfig_->numCacheFileHandles())
              : nullptr,
          std::make_unique<FileHandleGenerator>(config))

At present, Config is bound to the Connector layer. In some scenarios, we need to change the configuration at the query layer. For example, the user changes the id & key of the S3 table, or for hdfs table users want some tables to support HA configuration, some tables not.

Can anyone help with this issue and see how the velox community plans for the long term?

@mbasmanova @Yuhta

[mbasmanova]

@skadilover One way to make progress is to describe a few use cases you'd like to enable and see which connector APIs you'd need to change.

the user changes the id & key of the S3 table, or for hdfs table users want some tables to support HA configuration, some tables not.

Looks like these can be table-level properties included in HiveTableHandle.

[skadilover]

Looks like these can be table-level properties included in HiveTableHandle.

@mbasmanova Yes it`s table layer, assume that our user has 2 table:
tableA
tableproperty {
"hive.s3.aws-access-key":"xxx",
"hive.s3.aws-secret-key"":"aaa",
"hive.s3.endpoint":"host1"
}
tableB
tableproperty {
"hive.s3.aws-access-key":"yyy",
"hive.s3.aws-secret-key"":"bbb",
"hive.s3.endpoint":"host2"
}

···
std::make_unique<FileHandleGenerator>(config))
···
The current Hive Connector (only one instance per node) has only one Generator object (holding properties), which means it cannot support the above case.

[mbasmanova]

CC: @majetideepak

[majetideepak]

@skadilover We can achieve your use case by creating two HiveConnector instances. In Presto terms, this would map to two catalogs.
For use cases with expiring keys, we have a proposal here #8071

[skadilover]

We can achieve your use case by creating two HiveConnector instances
@majetideepak
Maybe not make sense for we have 10w+ tables in one instance in practice.