allow configuring larger max layer size in build time

[2rigor]

Closes #311

E.g., for 10Gb (10*10^9 bytes) append to go build command:
-ldflags "-w -s -X github.com/anchore/stereoscope/pkg/file.perFileReadLimitStr=10000000000"

[2rigor]

@wagoodman I don't know how it works. I see you were involved in many PRs. Will you guys automatically have a look at new PR or I should assign it (to whom)? Thanks!

[kzantow]

Hey @2rigor -- thanks for the contribution. We might like to take a different approach to allow for modifying the per-file read limit. Specifying this at build time doesn't allow configuration at runtime, which is probably the most important spot.

What if instead of making the change as you've implemented it, we just change the perFileReadLimit to a var, and add an exported function to set it; e.g.:

var perFileReadLimit = 2 * GB

func SetPerFileReadLimit(maxBytes int64) {
  // some reasonable validation like maxBytes > 0?
  perFileReadLimit = maxBytes
}

... this allows library users including Syft to configure this value via configuration files at runtime.

[2rigor]

Hey @2rigor -- thanks for the contribution. We might like to take a different approach to allow for modifying the per-file read limit. Specifying this at build time doesn't allow configuration at runtime, which is probably the most important spot.

What if instead of making the change as you've implemented it, we just change the perFileReadLimit to a var, and add an exported function to set it; e.g.:

var perFileReadLimit = 2 * GB

func SetPerFileReadLimit(maxBytes int64) {
  // some reasonable validation like maxBytes > 0?
  perFileReadLimit = maxBytes
}

... this allows library users including Syft to configure this value via configuration files at runtime.

Hi @kzantow.
Thanks for the response and the suggestion. I apologize for the delay
Indeed your suggestion sounds good.
I suggest both leaving the build time option and adding runtime one. I know it's a bit more complicated, but it will allow both configuration ways. And eventually it will be very short.
What do you think?

There is a minor disadvantage of providing an ability to update during runtime - since this variable can be read from multiple goroutines (even if today it is not). I think I saw PR in syft about handling layers in parallel. Practically setting the variable would happen (should happen) before starting the traversal over the image, but having such a function provides an option to change the variable at any time. I still think it's acceptable. Just wanted to share my thoughts.

[kzantow]

but having such a function provides an option to change the variable at any time. I still think it's acceptable. Just wanted to share my thoughts.

Yes, I don't think this is a problem, though -- if the value gets changed, it gets changed for the next reader; I don't think there's a concurrency issue with this. You can set this at initialization time by creating an init() function in your code.

[2rigor]

but having such a function provides an option to change the variable at any time. I still think it's acceptable. Just wanted to share my thoughts.

Yes, I don't think this is a problem, though -- if the value gets changed, it gets changed for the next reader; I don't think there's a concurrency issue with this. You can set this at initialization time by creating an init() function in your code.

Got you, I agree it's better. Implemented as you originally suggested

[kzantow]

LGTM -- thanks for the contribution @2rigor !

[tomersein]

how can i set this value via syft? @kzantow

[kzantow]

@tomersein -- there is no way to configure this in Syft at the moment, but you could add a configuration to one of the options that in a PostLoad() hook, reads a value and sets this using the stereoscope hook that was added.

[tomersein]

hi, the 1st part I think I understand, maybe under catalogers field (because it is related to scan)
but didn't understand the 2nd part, where is the hook of stereoscope and how can i add it? :) @kzantow

[kzantow]

@tomersein responded on Discourse