Proposal for #267 - configurable option to disable lzo index lookup

[EugenCepoi]

Making the reading of lzo indexes optional. This should reduce job start time when the input doesn't have any indexes.

[pkallos]

👍

[EugenCepoi]

Please, can one of the commiters take a look at this PR? When the users knows in advance that there are no indexes for the lzo files then this improves greatly the start time.

This, combined with using list status in parallel (mapreduce.input.fileinputformat.list-status.num-threads) can be an alternative solution to issue #426.

Another improvement when we do want to read indexes would be to read the indexes in parallel like FileInputFormat does in listStatus.

[gerashegalov]

I think getting index files and lzo themselves in a single fetch/listStatus RPC with an appropriate path filter would be most elegant and effective improvement. Then you can just hash index files into a map for quick lookups.

Disclaimer: I am not a committer

[pkallos]

I think getting index files and lzo themselves in a single fetch/listStatus RPC with an appropriate path filter would be most elegant and effective improvement. Then you can just hash index files into a map for quick lookups.

that sounds like a changeset with a much higher blast radius :)

[EugenCepoi]

@gerashegalov yes it would be a more general solution improving indexes lookup. But here the goal is really to just disable indexes lookup/reading when we know in advance that there are no indexes.

[gerashegalov]

@EugenCepoi If we list *.{lzo,lzo.index} in a single call there will be no performance advantage of disabling index lookups. So we can save unnecessary tuning key. The case when there are index files gets faster as well.

[gerashegalov]

I implemented my suggestion in #434