[Feat] allow optional searcher.terminationGracePeriodSeconds

[katlim-br]

I propose to add this optional terminationGracePeriodSeconds to the searcher (similar to the indexer).

In our use case, our customers directly use the QW instances. Therefore, we want to wait the longest possible so most queries return without breaking the UX.

Probable values to use: 60s, 90s.

[katlim-br]

This will wait for this other more urgent PR to be merged first (#128).

That is why I'm setting here to version 0.7.14.

[guilload]

I'm sorry, I don't understand the use case. Indexers are stateful and we want to wait for the data in-flight to be indexed before shutting down the pods. On the hand, searchers are stateless.

[katlim-br]

I'm sorry, I don't understand the use case. Indexers are stateful and we want to wait for the data in-flight to be indexed before shutting down the pods. On the hand, searchers are stateless.

Sure. We want to minimize the search queries getting cut off if a deploy is triggered.
Some of our queries take more than 30 seconds, so we want to tune this.

I agree that the solution is to handle the error and retry from the client, but we want to minimize the need to do it.

In our use case, our customers run the queríes for themselves, so we want to be less disruptive as possible.

As mentioned, our target will probably be 60s.

[guilload]

Ok that makes sense. However, I'm not entirely sure Quickwit searchers do the right thing when receiving a shutdown signal: i.e. they should stop accepting new requests and wait for the in-flight requests to be drained, but I don't know if currently this is implemented correctly.

[katlim-br]

Ok that makes sense. However, I'm not entirely sure Quickwit searchers do the right thing when receiving a shutdown signal: i.e. they should stop accepting new requests and wait for the in-flight requests to be drained, but I don't know if currently this is implemented correctly.

I can make a compromise on that. At least this will allow us to tune it for now.

If this is accepted, please merge this other PR first, so the versions are upgraded in order.

[guilload]

You must rebase first.

[katlim-br]

You must rebase first.

done