ORC-1458: Reduce HDFS NameNode getFileInfo RPC

[cxzl25]

What changes were proposed in this pull request?

If the file in HDFS is in a completed state, avoid calling the HDFS getFileInfo RPC.

Provide orc.file.length.fast configuration to enable this behavior.

Why are the changes needed?

Now reading an ORC file in HDFS will generate at least one open and getFileInfo RPC. This optimization can remove getFileInfo RPC as much as possible, improve ORC reading efficiency, and reduce the number of HDFS RPCs.

How was this patch tested?

The production environment has been running stably for several months

Was this patch authored or co-authored using generative AI tooling?

No

[dongjoon-hyun]

Why do we reflect this when Apache ORC can access this directly?

$ javap -cp hadoop-client-api-3.3.6.jar org.apache.hadoop.hdfs.DFSInputStream | head -n3
Compiled from "DFSInputStream.java"
public class org.apache.hadoop.hdfs.DFSInputStream extends org.apache.hadoop.fs.FSInputStream implements org.apache.hadoop.fs.ByteBufferReadable,org.apache.hadoop.fs.CanSetDropBehind,org.apache.hadoop.fs.CanSetReadahead,org.apache.hadoop.fs.HasEnhancedByteBufferAccess,org.apache.hadoop.fs.CanUnbuffer,org.apache.hadoop.fs.StreamCapabilities,org.apache.hadoop.fs.ByteBufferPositionedReadable {
  public static boolean tcpReadsDisabledForTesting;

[cxzl25]

org.apache.hadoop.hdfs.DFSInputStream.shortCircuitForbidden Method is not public.

[cxzl25]

Before ORC-1430, we could not access hdfs-related classes in the core module, so I used reflection.
Now I have done some refactoring, but the method of shortCircuitForbidden is not public, so I still need to use reflection.

[dongjoon-hyun]

This is a high-risky way which Apache Hadoop community is also not recommended.

May I ask if you have any reference in ASF communities which adopts this hack, @cxzl25 ?

[dongjoon-hyun]

Gentle ping, @cxzl25 .

[cxzl25]

This is a high-risky way which Apache Hadoop community is also not recommended.

May I ask if you have any reference in ASF communities which adopts this hack

@Hexiaoqiao Can you help review this PR?

[Hexiaoqiao]

Thanks involve me here. I have left my concerns here, I would like to reassert: I believe this could reduce load to NameNode when using ORC on HDFS but I am worried it involve any potential issues from HDFS view.

Back to this PR, IIUC this is used only for extract footer, right? Technically, for one immutable file, the result from DFSInputStream#getFileLength() and FileStatus#getLen() will be same, but it is not one recommended usage as @dongjoon-hyun mentioned above.

Thanks again.

[dongjoon-hyun]

Happy New Year. Do you still want to introduce this feature in Apache ORC for your use case, @cxzl25 ?

[cxzl25]

Happy New Year. @dongjoon-hyun

Do you still want to introduce this feature in Apache ORC for your use case

Let me close this PR first, this is a bit tricky, BTW S3 has a similar problem.

[dongjoon-hyun]

Thank you for closing, @cxzl25 .

For the following, could you file a JIRA Issue too? If we are aware of issues, the community can make alternative solution .

BTW S3 has a similar problem.

[cxzl25]

For the following, could you file a JIRA Issue too? If we are aware of issues, the community can make alternative solution

Sorry for the late reply.

Created https://issues.apache.org/jira/browse/ORC-1845

[dongjoon-hyun]

Thank you, @cxzl25 .