How to limit number of threads so that catfs can be used in multi-tenant environments? #63

Activity

1. HenrikBengtsson commented on Jan 19, 2022

   

   HenrikBengtsson

   on Jan 19, 2022

   Last edited by HenrikBengtsson

   Author

   More actions

   From some more ps inspections, it looks like the number of threads is fixed at some large number (107?). Using two catfs mounts as an example, I see 107 threads for each catfs process:

   $ for pid in $(pgrep catfs); do echo "$pid:"; pstree "$pid"; done                                                                       
   25800:
   catfs───107*[{catfs}]
   25962:
   catfs───107*[{catfs}]

   I see the same numbers on both an 8-core machine, and a 32-core machine.
2. gaul commented on Jan 19, 2022

   

   gaul

   on Jan 19, 2022

   Collaborator

   More actions

   Can you limit CPU usage via cgroups?

   https://stackoverflow.com/questions/28814002/using-cgroups-to-limit-cpu-usage
3. HenrikBengtsson commented on Jan 19, 2022

   

   HenrikBengtsson

   on Jan 19, 2022

   Last edited by HenrikBengtsson

   Author

   More actions

   Thanks for the quick reply.

   Unfortunately, cgroups is out of reach. It'll require sysadms to step in to configure the hosts, which might not be possible or an option for all catfs users. The beauty of catfs itself is that it requires no privileges.

   On some systems, a non-privileged user might be able to limit the CPU usage to, say 150%, using:

   systemd-run --user --scope -p CPUWeight=150 catfs ...

   However, that is not supported on all systems by default.

   Either way, I think using cgroups to limit the behavior of catfs will be a bit like running a Formula 1 car at full throttle all the time and controlling it's speed by breaking it from behind using some heavy tractor. I'd imagine running 100+ threads with limited CPU affinity would be inefficient for catfs itself.

   More importantly, software that assumes and behaves as they have exclusive access to all CPU resources on a machine is often frowned upon by sysadms, especially so on multi-tentant systems like HPC environments, because such software tend to clog up machines. It's not unheard of that end-users suffer in the crossfire without even knowing they're using the system in a bad way.

   Looking at the code, I found two instances of ThreadPool::new(<nthreads>) and both specify a hard-coded number of threads;

   catfs/src/catfs/mod.rs

   Line 126 in cbd7ab7

   tp: Mutex::new(ThreadPool::new(5)),

   catfs/src/pcatfs/mod.rs

   Line 33 in cbd7ab7

   tp: ThreadPool::new(100),

   I have no understanding what those two thread pools are doing, but if I replace (5,100) with, say, (3,10), recompile, I get:

   $ for pid in $(pgrep catfs); do echo "$pid:"; pstree "$pid"; done
   20355:
   catfs───15*[{catfs}]
   32486:
   catfs───15*[{catfs}]

   so I'm pretty certain those lines of code controls then number of threads used by catfs.

   My wish/feature request, would be to make it, at a minimum, possible to control those via a command-line option, e.g. --threads=3,10.
4. jpedrick

   added a commit that references this issue on Mar 16, 2023

   kahing#63 threadpool configuration for command line

   06ef879