Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support of hadoop delegation tokens with HADOOP_TOKEN_FILE_LOCATION env #335

Open
mutexed opened this issue Jan 30, 2024 · 0 comments
Open

Comments

@mutexed
Copy link

mutexed commented Jan 30, 2024

Hello, I see you have support of KRB ticket cache file, but in hadoop there is more lightweight type of authenticate cache. Here it is described: https://blog.cloudera.com/hadoop-delegation-tokens-explained/

The motivation is when you are using hdfs client in some big mapreduce operations (or simple map operations for transfering data from hadoop to somewhere else), each job will reauthenticate in kerberos using ticket cache, so KDC becomes a bottleneck. Moreover, while using hadoop delegation token you don't need even network connection to KDC server.

Default apache hdfs binary looks into env for HADOOP_TOKEN_FILE_LOCATION and use it to communicate with namenode if presented, is it supported in your client? If not, is it in plans to support it?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant