e-server / gdb: thread-specific breakpoint issues

[olajep]

Stop and resume of threads seem to work reliably, normal breakpoints too.
But setting a thread specific breakpoint (b foo thread 8) breaks things.

@palves: could you please take a look at this, it's a really useful feature to have IMO.

Test setup:

1. Start e-server --multiprocess in one terminal
2. Start epiphany-examples/apps/eprime example in a second terminal
3. Attach with epiphany-elf-gdb in a third terminal:

A try with "c -a &":
gdblog2.txt

parallella@parallella:~/epiphany-examples/apps/eprime$ epiphany-elf-gdb e_prime.elf -q -ex "set non-stop" -ex "tar extended-remote localhost:51000" -ex "attach 1"
Reading symbols from e_prime.elf...done.
Remote debugging using localhost:51000
Attaching to program: /home/parallella/epiphany-examples/apps/eprime/e_prime.elf, process 1
[New Thread 1.101]
[New Thread 1.102]
[New Thread 1.103]
[New Thread 1.104]
[New Thread 1.201]
[New Thread 1.202]
[New Thread 1.203]
[New Thread 1.204]
[New Thread 1.301]
[New Thread 1.302]
[New Thread 1.303]
[New Thread 1.304]
[New Thread 1.401]
[New Thread 1.402]
[New Thread 1.403]
[New Thread 1.404]
__ieee754_sqrt (x=<optimized out>) at /home/esim/esdk-submodule/unisrc-master/newlib/libm/math/e_sqrt.c:158
158     /home/esim/esdk-submodule/unisrc-master/newlib/libm/math/e_sqrt.c: No such file or directory.
(gdb)
Thread 2 stopped.
0x00001b46 in __fpcmp_parts_d (a=0x7f58, b=0x7f70) at /home/esim/esdk-submodule/unisrc-master/libgcc/fp-bit.c:1133
1133    /home/esim/esdk-submodule/unisrc-master/libgcc/fp-bit.c: No such file or directory.

Thread 3 stopped.
0x000021c4 in __umodsi3 ()

Thread 4 stopped.
0x0000220a in __umodsi3 ()

Thread 5 stopped.
main () at src/e_prime.c:64
64                      number += (2*16);

Thread 6 stopped.
0x000021e4 in __umodsi3 ()

Thread 7 stopped.
0x000021ec in __umodsi3 ()

Thread 8 stopped.
0x000020b0 in is_prime (number=4184881) at src/isprime.c:26
26              for(i=3;i<=s;i+=2)

Thread 9 stopped.
0x000021f0 in __umodsi3 ()

Thread 10 stopped.
0x00000300 in __ieee754_sqrt (x=<optimized out>)
    at /home/esim/esdk-submodule/unisrc-master/newlib/libm/math/e_sqrt.c:169
169     /home/esim/esdk-submodule/unisrc-master/newlib/libm/math/e_sqrt.c: No such file or directory.

Thread 11 stopped.
0x00002142 in __umodsi3 ()

Thread 12 stopped.
0x000002c0 in __ieee754_sqrt (x=<optimized out>)
    at /home/esim/esdk-submodule/unisrc-master/newlib/libm/math/e_sqrt.c:145
145     in /home/esim/esdk-submodule/unisrc-master/newlib/libm/math/e_sqrt.c

Thread 13 stopped.
0x00002212 in __umodsi3 ()

Thread 14 stopped.
0x0000222c in __umodsi3 ()

Thread 15 stopped.
__ieee754_sqrt (x=<optimized out>) at /home/esim/esdk-submodule/unisrc-master/newlib/libm/math/e_sqrt.c:169
169     in /home/esim/esdk-submodule/unisrc-master/newlib/libm/math/e_sqrt.c

Thread 16 stopped.
0x000021e2 in __umodsi3 ()
l main
31      volatile unsigned long long *max_tests = (void *)0x7020;
32
33      int is_prime(unsigned long number);
34
35      int main(void)
36      {
37              e_coreid_t id = e_get_coreid();
38              unsigned row, col;
39
40              unsigned long number;
(gdb)
41
42              // Get the row, column coordinates of this core
43              e_coords_from_coreid(id, &row, &col);
44
45              // Assign the starting value for each core (3,5,7,..33)
46              // Each core starts with a unique odd number (2 is the only even prime number)
47              number = 2 + ((2*row*4) + (2*col+1));
48
49              // Initialize this core iteration count for stats
50              (*count) = 0;
(gdb)
51
52              // Initialize the number of found primes counter
53              (*primes) = 0;
54
55              while(*count < *max_tests)
56              {
57                      if(is_prime(number))
58                              (*primes)++;
59
60                      // Skip to the next odd number for this core to test, assuming total of 16 cores
(gdb) b 59 thread 8
Breakpoint 1 at 0x1d94: file src/e_prime.c, line 59.
(gdb) c -a &
Continuing.
(gdb)
Thread 8 hit Breakpoint 1, main () at src/e_prime.c:64
64                      number += (2*16);
c -a &
Continuing.
(gdb) interrupt -a
(gdb) /home/esim/esdk-submodule/unisrc-master/gdb/inline-frame.c:343: internal-error: void skip_inline_frames(ptid_t): Assertion `find_inline_frame_state (ptid) == NULL' failed.
A problem internal to GDB has been detected,
further debugging may prove unreliable.
Quit this debugging session? (y or n) n

This is a bug, please report it.  For instructions, see:
<[email protected]>.

Another try with "c -a", same setup. Note that a different assertion fails here:
gdblog3.txt

esim@adapteva-dev:~/esdk-submodule/builds/bd-x86_64-master$ ./gdb/gdb ./e_prime.elf -ex "set non-stop" -ex "target extended-remote 192.168.0.110:51000" -ex "attach 1"
GNU gdb (Epiphany toolchain master) 7.12
Copyright (C) 2016 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "--host=x86_64-unknown-linux-gnu --target=epiphany-elf".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<[email protected]>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from ./e_prime.elf...done.
Remote debugging using 192.168.0.110:51000
Attaching to program: /home/esim/esdk-submodule/builds/bd-x86_64-master/e_prime.elf, process 1
[New Thread 1.101]
[New Thread 1.102]
[New Thread 1.103]
[New Thread 1.104]
[New Thread 1.201]
[New Thread 1.202]
[New Thread 1.203]
[New Thread 1.204]
[New Thread 1.301]
[New Thread 1.302]
[New Thread 1.303]
[New Thread 1.304]
[New Thread 1.401]
[New Thread 1.402]
[New Thread 1.403]
[New Thread 1.404]
0x000021dc in __umodsi3 ()
(gdb) 
Thread 2 stopped.
0x000020a6 in is_prime (number=27818789) at src/isprime.c:28
28      src/isprime.c: No such file or directory.

Thread 3 stopped.
0x000021ca in __umodsi3 ()

Thread 4 stopped.
0x0000223a in __umodsi3 ()

Thread 5 stopped.
0x000020a2 in is_prime (number=27819947) at src/isprime.c:28
28      in src/isprime.c

Thread 6 stopped.
0x00002110 in __umodsi3 ()

Thread 7 stopped.
0x00002214 in __umodsi3 ()

Thread 8 stopped.
__ieee754_sqrt (x=<optimized out>) at /home/esim/esdk-submodule/unisrc-master/newlib/libm/math/e_sqrt.c:152
152                 ix0 += ix0 + ((ix1&sign)>>31);

Thread 9 stopped.
0x00002234 in __umodsi3 ()

Thread 10 stopped.
0x00002214 in __umodsi3 ()

Thread 11 stopped.
0x00002212 in __umodsi3 ()

Thread 12 stopped.
0x00002214 in __umodsi3 ()

Thread 13 stopped.
0x00002140 in __umodsi3 ()

Thread 14 stopped.
0x00002232 in __umodsi3 ()

Thread 15 stopped.
0x00002140 in __umodsi3 ()

Thread 16 stopped.
__ieee754_sqrt (x=<optimized out>) at /home/esim/esdk-submodule/unisrc-master/newlib/libm/math/e_sqrt.c:146
146                 t = s0+r; 
(gdb) b 59
Breakpoint 1 at 0x1d94: file src/e_prime.c, line 59.
(gdb) delete breakpoints 
Delete all breakpoints? (y or n) y
(gdb) b 59 thread 8
Breakpoint 2 at 0x1d94: file src/e_prime.c, line 59.
(gdb) c -a
Continuing.

Thread 8 hit Breakpoint 2, main () at src/e_prime.c:64
64      in src/e_prime.c
(gdb) info threads
  Id   Target Id         Frame 
* 1    Thread 1.101 (Core: 0000: halted, interruptible) (running)
  2    Thread 1.102 (Core: 0001: halted, interruptible) (running)
  3    Thread 1.103 (Core: 0002: halted, interruptible) (running)
  4    Thread 1.104 (Core: 0003: halted, interruptible) (running)
  5    Thread 1.201 (Core: 0100: halted, interruptible) (running)
  6    Thread 1.202 (Core: 0101: halted, interruptible) (running)
  7    Thread 1.203 (Core: 0102: halted, interruptible) (running)
  8    Thread 1.204 (Core: 0103: halted, interruptible) main () at src/e_prime.c:64
  9    Thread 1.301 (Core: 0200: halted, interruptible) (running)
  10   Thread 1.302 (Core: 0201: halted, interruptible) (running)
  11   Thread 1.303 (Core: 0202: halted, interruptible) (running)
  12   Thread 1.304 (Core: 0203: halted, interruptible) (running)
  13   Thread 1.401 (Core: 0300: halted, interruptible) (running)
  14   Thread 1.402 (Core: 0301: halted, interruptible) (running)
  15   Thread 1.403 (Core: 0302: halted, interruptible) (running)
  16   Thread 1.404 (Core: 0303: halted, interruptible) (running)
(gdb) interrupt -a
/home/esim/esdk-submodule/unisrc-master/gdb/infrun.c:5565: internal-error: int finish_step_over(execution_control_state*): Assertion `ecs->event_thread->control.trap_expected' failed.
A problem internal to GDB has been detected,
further debugging may prove unreliable.
Quit this debugging session? (y or n) 

[olajep]

Things improve with this patch:
adapteva/epiphany-binutils-gdb@3bcc59a

I can now set a breakpoint on a thread, continue, wait for the breakpoint to hit, continue... that works.

But when I issue "interrupt -a" when a thread with a breakpoint is running this assertion fails
gdb/gdb/infrun.c:2402: internal-error: void resume(gdb_signal): Assertion `!tp->stop_requested' failed.

gdblog4.txt

$ ./gdb/gdb ./e_prime.elf -ex "set non-stop on" -ex "set target-async on" -ex "target extended-remote 192.168.0.110:51000" -ex "attach 1 &"
Reading symbols from ./e_prime.elf...done.
Remote debugging using 192.168.0.110:51000
Attaching to program: /tmp/gdb/e_prime.elf, process 1
[New Thread 1.101]
[New Thread 1.102]
[New Thread 1.103]
[New Thread 1.104]
[New Thread 1.201]
[New Thread 1.202]
[New Thread 1.203]
[New Thread 1.204]
[New Thread 1.301]
[New Thread 1.302]
[New Thread 1.303]
[New Thread 1.304]
[New Thread 1.401]
[New Thread 1.402]
[New Thread 1.403]
[New Thread 1.404]
(gdb) b sqrt thread 8
Breakpoint 1 at 0x11c
(gdb) info threads
  Id   Target Id         Frame 
* 1    Thread 1.101 (Core: 0000: halted, interruptible) (running)
  2    Thread 1.102 (Core: 0001: halted, interruptible) (running)
  3    Thread 1.103 (Core: 0002: halted, interruptible) (running)
  4    Thread 1.104 (Core: 0003: halted, interruptible) (running)
  5    Thread 1.201 (Core: 0100: halted, interruptible) (running)
  6    Thread 1.202 (Core: 0101: halted, interruptible) (running)
  7    Thread 1.203 (Core: 0102: halted, interruptible) (running)
  8    Thread 1.204 (Core: 0103: halted, interruptible) (running)
  9    Thread 1.301 (Core: 0200: halted, interruptible) (running)
  10   Thread 1.302 (Core: 0201: halted, interruptible) (running)
  11   Thread 1.303 (Core: 0202: halted, interruptible) (running)
  12   Thread 1.304 (Core: 0203: halted, interruptible) (running)
  13   Thread 1.401 (Core: 0300: halted, interruptible) (running)
  14   Thread 1.402 (Core: 0301: halted, interruptible) (running)
  15   Thread 1.403 (Core: 0302: halted, interruptible) (running)
  16   Thread 1.404 (Core: 0303: halted, interruptible) (running)
(gdb) 
Thread 8 hit Breakpoint 1, 0x0000011c in sqrt ()
interrupt -a
(gdb) 
Thread 4 stopped.
0x00000124 in sqrt ()

Thread 12 stopped.
0x0000011c in sqrt ()

Thread 11 stopped.
0x00000120 in sqrt ()

Thread 10 stopped.
0x00000120 in sqrt ()

Thread 13 stopped.
0x0000011c in sqrt ()

Thread 9 stopped.
0x0000011c in sqrt ()

Thread 7 stopped.
0x00000224 in __ieee754_sqrt ()

Thread 14 stopped.
0x00000224 in __ieee754_sqrt ()

Thread 6 stopped.
0x00000122 in sqrt ()

Thread 5 stopped.
0x0000011c in sqrt ()

Thread 15 stopped.
0x0000011c in sqrt ()

Thread 3 stopped.
0x00000124 in sqrt ()

Thread 2 stopped.
0x0000011c in sqrt ()

Thread 16 stopped.
0x0000011c in sqrt ()

Thread 1 stopped.
0x00000120 in sqrt ()
c -a &
Continuing.
(gdb) info threads
  Id   Target Id         Frame 
* 1    Thread 1.101 (Core: 0000: halted, interruptible) (running)
  2    Thread 1.102 (Core: 0001: halted, interruptible) (running)
  3    Thread 1.103 (Core: 0002: halted, interruptible) (running)
  4    Thread 1.104 (Core: 0003: halted, interruptible) (running)
  5    Thread 1.201 (Core: 0100: halted, interruptible) (running)
  6    Thread 1.202 (Core: 0101: halted, interruptible) (running)
  7    Thread 1.203 (Core: 0102: halted, interruptible) (running)
  8    Thread 1.204 (Core: 0103: halted, interruptible) (running)
  9    Thread 1.301 (Core: 0200: halted, interruptible) (running)
  10   Thread 1.302 (Core: 0201: halted, interruptible) (running)
  11   Thread 1.303 (Core: 0202: halted, interruptible) (running)
  12   Thread 1.304 (Core: 0203: halted, interruptible) (running)
  13   Thread 1.401 (Core: 0300: halted, interruptible) (running)
  14   Thread 1.402 (Core: 0301: halted, interruptible) (running)
  15   Thread 1.403 (Core: 0302: halted, interruptible) (running)
  16   Thread 1.404 (Core: 0303: halted, interruptible) (running)
(gdb) 
Thread 8 hit Breakpoint 1, 0x0000011c in sqrt ()
c -a &
Continuing.
[this interrupt issued immediately after continue ]
(gdb) interrupt -a
(gdb) 
Thread 14 stopped.
0x0000024e in __ieee754_sqrt ()

Thread 13 stopped.
0x0000023c in __ieee754_sqrt ()
/home/esim/esdk-submodule/gdb/gdb/infrun.c:2402: internal-error: void resume(gdb_signal): Assertion `!tp->stop_requested' failed.
A problem internal to GDB has been detected,
further debugging may prove unreliable.
Quit this debugging session? (y or n)