Gluster "pass-through" tuning options explanations?

[edrock200]

So, I'm having a difficult time finding documentation on the pass-through tuning options in gluster. running gluster volume set help, and all the descriptions say is "Enable Disable item translator".

So I googled gluster translators, and found a few articles, the most descriptive was this one:
https://glusterdocs-beta.readthedocs.io/en/latest/overview-concepts/translators.html

But it doesn't mention any of the specific tuning options. Another article spoke about writing your own translator, so then I thought perhaps these are like enabling "API" like interfaces into gluster.

However, one tuning option, like global threading, recommends enabling performance.iot-pass-through. So I enabled global threading and this pass through option, and restarted glusterd services.

The global threading was a bust, when I remounted the clients with global threading enabled, they would randomly dismount after a few hours. so I changed the mount back to not use global threading but left performance.iot-pass-through on. To my surprise, the gluster nodes went from 80+% CPU consumption to ~30-50%.

I don't know if this was a side effect of restarting gluster across or if this pass-through option somehow made gluster more efficient. But now I'm curious about all the pass-through options and if they serve any benefit to fuse mounted clients.

Can anyone explain with a bit more detail what these options do?
Thanks in advance.

[edrock200]

Anyone have any insight into these tunes?

[edrock200]

Even a link pointer would be appreciated. I cannot find any in depth details on these tunes.

[mohit84]

As the name suggest pass-through the option is useful to disable a xlator at runtime basis. While this option is enabled for specific xlator it means the xlator is disabled functionality wise the xlator will not process any fops even xlator is loaded in volfile. It will work only while a specific xlator has this option in that option table at source code level, currently not every xlator provides this option.

For ex , As you mentioned a tuning option recommend set performance.iot-pass-through while enable global-threading. By default
io-thread is enabled for gluster, io-thread is the xlator that winds a operation parallel, the xlator is loaded on both sides(client as well as on server). The global-thread is a framework to provide parallelism, it is not a xlator and the purpose of this framework is to reduce the lock contention, sometime a user observe high lock contention while using io-thread.At a time the user should enable only one option either global-thread or io-thread. To know more about global-threading please follow this link (https://review.gluster.org/#/c/glusterfs/+/20636/). Let me know if you have any other question about pass-through.

[edrock200]

As the name suggest pass-through the option is useful to disable a xlator at runtime basis. While this option is enabled for specific xlator it means the xlator is disabled functionality wise the xlator will not process any fops even xlator is loaded in volfile. It will work only while a specific xlator has this option in that option table at source code level, currently not every xlator provides this option.

For ex , As you mentioned a tuning option recommend set performance.iot-pass-through while enable global-threading. By default io-thread is enabled for gluster, io-thread is the xlator that winds a operation parallel, the xlator is loaded on both sides(client as well as on server). The global-thread is a framework to provide parallelism, it is not a xlator and the purpose of this framework is to reduce the lock contention, sometime a user observe high lock contention while using io-thread.At a time the user should enable only one option either global-thread or io-thread. To know more about global-threading please follow this link (https://review.gluster.org/#/c/glusterfs/+/20636/). Let me know if you have any other question about pass-through.

Thank you so much for your reply! My biggest question right now is, I have global threading disabled, but performance.iot-pass-through enabled. With this setting enabled, my gluster nodes go from 80+% CPU down to 40-50% cpu consumption, but tasks like readdirp on clients take longer. So I'm just trying to understand what this setting does specifically when not using global threading that is freeing up so much CPU, but also slowing down readdirp operations.

[mohit84]

Can you please share "gluster v info" output from your environment?
Can you share how long is taking readdirp in your environment?

[edrock200]

volinfo.txt
Attached the vol info.

readdirp is inconsistent with regards to timing depending on load, but for example a dir with 8000 objects under heavy load will take 5m with iot pass through off, and ~9m with it on. However, if I toggle iot passthrough off, within seconds, I can see the gluster nodes jump 30-40% CPU utilization from ~50's to high 80's. Turning it back on drops the CPU utilization by the same percentage across all nodes. So I'm not sure if it's simply processing things slower and therefore less CPU is used, or if its actually that much more efficient, less the readdirp issue.

[mohit84]

As per current volume configuration i think performance reduction is expected. Currrently performance.iot-pass-through is enabled(it means io-thread is disabled) and global-threading is disabled. It means the brick process wind the fop in event threads. You should enable anyone option either io-thread or global-thread , here in this case both option are disabled.
The second configuration is specific to readdirp, It seems you have disabled readdirp. It means currently readdir is enabled.
Any specific reason why you did disable readdirp. If you want to get the best performance you should use readdirp(by default readdirp is enabled). You can enable readdir-ahead also to improve the performance more. As you enabled readdir-ahead the xlator(readdir-ahead) fetch the entries in background during opendir and keep the data in cache unless fd is closed, once fd is closed all entries are cleanup.

[edrock200]

Are you referring to dht.force-readdirp or performance.force-readdirp or something else? When you say enable io-thread do you mean by disabling performance.-iot-pass-through?

When I tried enabling global threading and remounting clients with global threading, those mounts would just randomly drop which is why I disabled global threading. With regards to read ahead, I did try enabling it, however in my time tests of recursive directory listings, it actually increased times significantly.

[mohit84]

I am assuming if you have disabled readdirp at dht/md-cache level so it means you have disabled
readdirp at fuse level also. What are the arguments you are using to mount a volume? Yes when
i have said enable io-thread means disable peformance.iot-pass-through. There is no compulsion
use the global-thread for both (client and server), you can enable only for server and anaylyze
the improvement or you can skip for both and go with default option.
Can you run the below test case and share the data

1. Enable io-thread
2. Run the below command
   ps -T -p <any_brick_pid>
3. Generate a statedump of client to cleanup existing profile data
   kill -SIGUSR1 <client_pid>, it will generate the statedump in /var/run/gluster directory
4. Run the test case(readdirp)
5. Generate a statedump again same as step3 for the same client

[edrock200]

mount statement for almost all clients is:
node:/volume /mnt/gluster glusterfs defaults,ro,_netdev 0 0

The 2 read/write nodes have the "ro" removed.

I will give this a shot. I just turned global-threading on and am running some tests. My understanding is having "force-readdirp" options disabled doesn't disable readdirp altogether, but doesn't force every call to be a readdirp call. If this is not accurate I can roll these two settings back.

Just out of curiosity, if I have performance.iot-pass-through on, and global threading on, is my understanding correct that the servers will use performance.io-thread-count and, if the clients aren't mounted with global threading, they will use client.event-threads or config.client-threads?

[edrock200]

I should have noted I will run those tests as well. Thank you for your help by the way @mohit84 ! 😁