Improve PR2 ethercat realtime performance

[HiroIshida]

Most of the interrupts concentrates to CPU0

applications@pr1040:~$ cat /proc/interrupts 
            CPU0       CPU1       CPU2       CPU3       CPU4       CPU5       CPU6       CPU7       
   0:         21          0          0          0          0          0          0          0  IR-IO-APIC    2-edge      timer
   8:          0          0          0          0          0          0          0          0  IR-IO-APIC    8-edge      rtc0
   ....
   9:          0          1          0          0          0          0          0          0  IR-IO-APIC    9-fasteoi   acpi194308-edge      lan2-TxRx-3
 135:          0          0         75          0         31         20  452257865         57  IR-PCI-MSI 1572864-edge      lan1
 136:         48          0          0          0          0          0          0  339699073  IR-PCI-MSI 1574912-edge      lan0
 137:    5470193      39908      30405   60331702    1190443   47189737          0          0  IR-PCI-MSI 2097152-edge      wan0
 138:  211172112          0         29          0          7          0      10651          0  IR-PCI-MSI 2099200-edge      ecat0

And, actually the IRQ affinity is binds to CPU0

applications@pr1040:~$ cat /proc/irq/138/smp_affinity
01

もし, c++コード内部でaffinityが設定されているなら以下が呼ばれているはず.
sched_setaffinity()
だけど objdump -C -s eml.so | grep ... してもない.

[HiroIshida]

pr2_ethercat プロセスが112%くらいになっている. 何がおきているのか調べるためperfを使う

[HiroIshida]

futexシステムコールに多大な時間を割いているｊ

pr2admin@pr1040:~/tmp$ sudo strace -c -p $(pgrep -f pr2_ether)
strace: Process 13156 attached
^Cstrace: Process 13156 detached
% time     seconds  usecs/call     calls    errors syscall
------ ----------- ----------- --------- --------- ----------------
 98.24   12.288435        1489      8251       239 futex
  0.85    0.106732          54      1974           write
  0.41    0.051252        4271        12           mprotect
  0.19    0.023239       23239         1         1 restart_syscall
  0.18    0.023035        1919        12           munmap
  0.05    0.006274         116        54           clock_nanosleep
  0.04    0.005041         168        30           clone
  0.02    0.002844         123        23           epoll_ctl
  0.01    0.000962          80        12           sendto
  0.01    0.000666          60        11           shutdown
  0.00    0.000311          28        11           close
  0.00    0.000251          20        12           mmap
------ ----------- ----------- --------- --------- ----------------
100.00   12.509042                 10403       240 total

[HiroIshida]

applications@pr1040:~$ ls /proc/$(pgrep -f pr2_ether)/task | wc -l
88

めちゃくちゃ多くのスレッドを作っている. ちなみにrobot stop, jsk start で立ち上げなおすと

pr2admin@pr1040:~/tmp$ python3 monitor.py 
Starting monitoring of pr2_ether
[2024-12-18 07:56:10] Thread count: 9
[2024-12-18 07:56:30] Thread count: 77
[2024-12-18 07:56:50] Thread count: 87
[2024-12-18 07:57:10] Thread count: 87
[2024-12-18 07:57:30] Thread count: 87

pr2admin@pr1040:~/tmp$ sudo strace -c -p $(pgrep -f pr2_ether)
strace: Process 95376 attached
^Cstrace: Process 95376 detached
% time     seconds  usecs/call     calls    errors syscall
------ ----------- ----------- --------- --------- ----------------
 98.46   13.779024        1269     10852        36 futex
  0.62    0.086224          45      1885           write
  0.48    0.066550        5545        12           mprotect
  0.26    0.035837        2986        12           munmap
  0.07    0.010449         222        47           clock_nanosleep
  0.06    0.007837         261        30           clone
  0.03    0.003800        3800         1           restart_syscall
  0.01    0.002037         169        12           sendto
  0.01    0.001021          44        23           epoll_ctl
  0.01    0.000998          90        11           shutdown
  0.00    0.000487          44        11           close
  0.00    0.000283          23        12           mmap
------ ----------- ----------- --------- --------- ----------------
100.00   13.994547                 12908        36 total

futex errorが少ないことがわかる.

[HiroIshida]

april tag rosを taskset -c 0 で cpu0にバインドしたけど、特に遅くなりはしなかった..

[HiroIshida]

futex error rateが大きくなっていることはわかるが、原因は別のところにあるのだろう