Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

rtpengine mr10.5 crashed in __dtx_send_later #1817

Open
andingv opened this issue Apr 23, 2024 · 13 comments
Open

rtpengine mr10.5 crashed in __dtx_send_later #1817

andingv opened this issue Apr 23, 2024 · 13 comments
Labels
bug

Comments

@andingv
Copy link
Contributor

andingv commented Apr 23, 2024 •
edited
Loading

rtpengine version the issue has been seen with

mr10.5.1

Used distribution and its version

Ubuntu 18.04.6 LTS

Linux kernel version used

No response

CPU architecture issue was seen on (see uname -m)

x86_64

Expected behaviour you didn't see

No response

Unexpected behaviour you saw

I am seeing a seldom crash, last week two hosts crashed independently.

Steps to reproduce the problem

Not sure, but there's one idea . Am I seeing it right that some endpoint is sending the RTP packet with huge ts causing this behavior?
Program terminated with signal SIGSEGV, Segmentation fault.
#0 0x000055bed2c174a5 in decoder_input_data_ptime (dec=0x7fc6ec17e690, data=0xc0089a793fe274df, ts=4715930405566051902, ptime=ptime@entry=0x7fc7bb5f9288,
callback=0x55bed2bebbe0 <packet_decoded_fifo>, u1=0x7fc73c089d90, u2=0x7fc7bb5f91c0) at codeclib.c:936

Additional program output to the terminal or logs illustrating the issue

Here is the backtrace:

(gdb) 
#0  0x000055bed2c174a5 in decoder_input_data_ptime (dec=0x7fc6ec17e690, data=0xc0089a793fe274df, ts=4715930405566051902, ptime=ptime@entry=0x7fc7bb5f9288, 
    callback=0x55bed2bebbe0 <packet_decoded_fifo>, u1=0x7fc73c089d90, u2=0x7fc7bb5f91c0) at codeclib.c:936
No locals.
#1  0x000055bed2bef42f in __rtp_decode (ch=<optimized out>, input_ch=0x7fc73c089d90, packet=<optimized out>, mp=0x7fc7bb5f91c0) at codec.c:3682
        ret = 0
#2  0x000055bed2bec9e7 in __dtx_send_later (ct=0x7fc6a401d5f0) at codec.c:3072
        dtxb = 0x7fc6a401d5f0
        mp_copy = {raw = {s = 0x0, len = 0}, fsin = {address = {family = 0x55bed2e51b80 <__socket_families>, u = {ipv4 = {s_addr = 113162329}, ipv6 = {__in6_u = {
                    __u6_addr8 = "Y\270\276\006", '\000' <repeats 11 times>, __u6_addr16 = {47193, 1726, 0, 0, 0, 0, 0, 0}, __u6_addr32 = {113162329, 0, 0, 0}}}}}, port = 26856}, tv = {
            tv_sec = 1713279718, tv_usec = 40418}, sfd = 0x7fc630056450, call = 0x7fc6fc0105a0, stream = 0x7fc71c0900d0, media = 0x7fc66c0d0810, media_out = 0x7fc62c0db7b0, sink = {
            sink = 0x7fc6cc111080, handler = 0x55bed2e486d0 <__sh_noop_rtp>, kernel_output_idx = -1, rtcp_only = 0, transcoding = 1}, rtp = 0x7fc48c038080, rtcp = 0x0, ssrc_in = 0x7fc69814a570, 
          ssrc_out = 0x7fc69814cbf0, payload = {s = 0x0, len = 0}, packets_out = {head = 0x0, tail = 0x0, length = 0}, ptime = -1}
        ret = 0
        ts = 4715930405566051902
        p_left = 7
        tv_diff = 130003
        ts_diff = 4715930405563795742
        dtxp = <optimized out>
        call = 0x7fc6fc0105a0
        ch = 0x7fc73c089d90
        ps = <optimized out>
        input_ch = 0x7fc73c089d90
        ptime = 20
#3  0x000055bed2bfb0c8 in timerthread_run (p=0x55bed2e53bc0 <codec_timers_thread>) at timerthread.c:62
        tt_obj = <optimized out>
        sleeptime = <optimized out>
        tv = <optimized out>
        tt = <optimized out>
        waker = {lock = 0x55bed2e53bc8 <codec_timers_thread+8>, cond = 0x55bed2e53bf0 <codec_timers_thread+48>}
#4  0x000055bed2b79f99 in thread_detach_func (d=0x55bed4f94e60) at aux.c:250
        __cancel_buf = {__cancel_jmp_buf = {{__cancel_jmp_buf = {94278105255520, -5219573935894328112, 140495818822912, 0, 140482340461232, 140720364687680, 5196924824070491344, 
                2084599588019342544}, __mask_was_saved = 0}}, __pad = {0x7fc7bb5f9450, 0x0, 0x0, 0x0}}
        __cancel_routine = 0x55bed2b79e00 <thread_detach_cleanup>
        __cancel_arg = 0x55bed4f94e60
        __not_first_call = <optimized out>
        dt = 0x55bed4f94e60
        t = <optimized out>
        scheduler = <optimized out>
#5  0x00007fc8104596db in start_thread (arg=0x7fc7bb5fa700) at pthread_create.c:463
        pd = 0x7fc7bb5fa700
        now = <optimized out>
        unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140495818827520, -5219573935894328112, 140495818822912, 0, 94278105255520, 140720364687680, 5196924823827221712, 5188280515029208272}, 
              mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
        not_first_call = <optimized out>
#6  0x00007fc80fcf061f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

something more:

(gdb) fr 2
#2  0x000055bed2bec9e7 in __dtx_send_later (ct=0x7fc6a401d5f0) at codec.c:3072
3072	in codec.c
(gdb) p *enc
No symbol "enc" in current context.
(gdb) p *ch
$1 = {h = {obj = {ref = 21, free_func = 0x55bed2be91c0 <__free_ssrc_handler>, size = 376}, lock = {__data = {__lock = 0, __count = 0, __owner = 0, __nusers = 0, __kind = 0, __spins = 0, 
        __elision = 0, __list = {__prev = 0x0, __next = 0x0}}, __size = '\000' <repeats 39 times>, __align = 0}, ssrc = 3270805003, last_used = 1713279718}, handler = 0x7fc6bc0fb070, 
  decoder = 0x7fc6ec17e690, encoder = 0x7fc6b8110230, encoder_format = {clockrate = 16000, channels = 1, format = 1}, bitrate = 14250, ptime = 20, bytes_per_packet = 80, first_ts = 4800, 
  last_ts = 2256640, first_send = {tv_sec = 1713279436, tv_usec = 750421}, first_send_ts = 4160, output_skew = 0, sample_buffer = 0x7fc728157920, dtx_buffer = 0x7fc6a401d5f0, dtmf_dsp = 0x0, 
  dtmf_resampler = {swresample = 0x0, no_filter = false}, dtmf_format = {clockrate = 0, channels = 0, format = 0}, dtmf_ts = 0, last_dtmf_event_ts = 0, dtmf_events = {head = 0x0, tail = 0x0, 
    length = 0}, dtmf_event = {code = 0, volume = 0, ts = 0, rand_code = 0, block_dtmf = BLOCK_DTMF_OFF}, dtmf_state = {code = 0, volume = 0, ts = 0, rand_code = 0, block_dtmf = BLOCK_DTMF_OFF}, 
  silence_events = {head = 0x0, tail = 0x0, length = 0}, dtmf_start_ts = 0, dtmf_first_duration = 0, skip_pts = 0, rtp_mark = 0}



### Anything else?

The machines are doing transcoding and AMR DTX so the load is quite high.
We have manually picked the fix for #1591 to this locally built version.
@andingv andingv added the bug label Apr 23, 2024
@rfuchs
Copy link
Member

rfuchs commented Apr 25, 2024

A large timestamp shouldn't be a problem, but the data pointer looks corrupted, which probably means ts was corrupted as well. Possibly a double-free of the packet object or something like it. What are the contents of *dtxb in frame 2?

@andingv
Copy link
Contributor Author

andingv commented Apr 26, 2024

ok thank you, here you go:

(gdb) fr 2
#2  0x000055bed2bec9e7 in __dtx_send_later (ct=0x7fc6a401d5f0) at codec.c:3072
3072	codec.c: No such file or directory.
(gdb) p *dtxb
$1 = {ct = {tt_obj = {obj = {ref = 2, free_func = 0x55bed2be9390 <__dtx_free>, size = 416}, tt = 0x55bed2e53bc0 <codec_timers_thread>, next_check = {tv_sec = 0, tv_usec = 0}, last_run = {
        tv_sec = 1713279718, tv_usec = 170421}}, next = {tv_sec = 1713279718, tv_usec = 170421}, timer_func = 0x55bed2bec4c0 <__dtx_send_later>}, lock = {__data = {__lock = 0, __count = 0, 
      __owner = 0, __nusers = 0, __kind = 0, __spins = 0, __elision = 0, __list = {__prev = 0x0, __next = 0x0}}, __size = '\000' <repeats 39 times>, __align = 0}, csh = 0x7fc73c089d90, ptime = 20, 
  tspp = 160, clockrate = 8000, call = 0x7fc6fc0105a0, packets = {head = 0x7fbfd406f820, tail = 0x7fbfeffbbd00, length = 7}, last_mp = {raw = {s = 0x0, len = 0}, fsin = {address = {
        family = 0x55bed2e51b80 <__socket_families>, u = {ipv4 = {s_addr = 113162329}, ipv6 = {__in6_u = {__u6_addr8 = "Y\270\276\006", '\000' <repeats 11 times>, __u6_addr16 = {47193, 1726, 0, 0, 
                0, 0, 0, 0}, __u6_addr32 = {113162329, 0, 0, 0}}}}}, port = 26856}, tv = {tv_sec = 1713279718, tv_usec = 40418}, sfd = 0x7fc630056450, call = 0x7fc6fc0105a0, stream = 0x7fc71c0900d0, 
    media = 0x7fc66c0d0810, media_out = 0x7fc62c0db7b0, sink = {sink = 0x7fc6cc111080, handler = 0x55bed2e486d0 <__sh_noop_rtp>, kernel_output_idx = -1, rtcp_only = 0, transcoding = 1}, 
    rtp = 0x7fc48c0df360, rtcp = 0x0, ssrc_in = 0x7fc69814a570, ssrc_out = 0x7fc69814cbf0, payload = {s = 0x0, len = 0}, packets_out = {head = 0x0, tail = 0x0, length = 0}, ptime = 0}, 
  head_ts = 4715930405566051902, ssrc = 3270805003, start = 1713279718}

@andingv
Copy link
Contributor Author

andingv commented Nov 11, 2024 •
edited
Loading

There are many more occurrences that are happening in AMR transcoding scenario (branch mr11.5) due to CN dtx handling.

(gdb) bt full
#0  0x00007f6d6f7eb99a in ?? () from /usr/lib/libavcodec.so.58
No symbol table info available.
#1  0x00007f6d6f9f61bb in ?? () from /usr/lib/libavcodec.so.58
No symbol table info available.
#2  0x00007f6d6f9f7678 in avcodec_send_packet () from /usr/lib/libavcodec.so.58
No symbol table info available.
#3  0x0000557e5873b5e7 in avc_decoder_input (dec=0x7f6d689fe7e0, data=0x557e5878c430 <rtpe_config+1040>, out=0x7f6d6314ed40) at ../lib/codeclib.c:1004
        got_frame = 0
        err = 0x557e58770873 "failed to alloc av frame"
        av_ret = 0
        frame = 0x7f6d603eda00
        keep_going = 0
#4  0x0000557e5874c5b9 in cn_decoder_input (dec=0x7f6d689fe7e0, data=0x557e5878c430 <rtpe_config+1040>, out=0x7f6d6314ed40) at ../lib/codeclib.c:3439
        ptime = 1
        samples = 8
        ret = 82601444
#5  0x0000557e5873c2aa in __decoder_input_data (dec=0x7f6d689fe7e0, data=0x557e5878c430 <rtpe_config+1040>, ts=125288, ptime=0x0, callback=0x557e5874b032 <cn_append_frame>, u1=0x7f6d6314eea0, u2=0x0) at ../lib/codeclib.c:1119
        frames = {head = 0x0, tail = 0x0, length = 0}
        frame = 0x4ec65e4ffffffff
        ret = 0
        samples = 140107790480784
#6  0x0000557e5873c5b9 in decoder_input_data (dec=0x7f6d689fe7e0, data=0x557e5878c430 <rtpe_config+1040>, ts=125288, callback=0x557e5874b032 <cn_append_frame>, u1=0x7f6d6314eea0, u2=0x0) at ../lib/codeclib.c:1151
No locals.
#7  0x0000557e5874b0d0 in generic_cn_dtx (dec=0x7f6d6031d540, out=0x7f6d6314eea0, ptime=1) at ../lib/codeclib.c:3174
No locals.
#8  0x0000557e5873c2d7 in __decoder_input_data (dec=0x7f6d6031d540, data=0x0, ts=125288, ptime=0x7f6d6314eefc, callback=0x557e586f4621 <packet_decoded_fifo>, u1=0x7f6d696b4af0, u2=0x7f6d6314f1d0) at ../lib/codeclib.c:1121
        frames = {head = 0x0, tail = 0x0, length = 0}
        frame = 0x7f6dffffffff
        ret = 0
        samples = 140107933425952
#9  0x0000557e5873c686 in decoder_dtx (dec=0x7f6d6031d540, ts=125288, ptime=1, callback=0x557e586f4621 <packet_decoded_fifo>, u1=0x7f6d696b4af0, u2=0x7f6d6314f1d0) at ../lib/codeclib.c:1163
No locals.
#10 0x0000557e586efefc in __dtx_send_later (ct=0x7f6d698186e0) at codec.c:3417
        diff = 0
        dtxb = 0x7f6d698186e0
        mp_copy = {raw = {s = 0x0, len = 0}, fsin = {address = {family = 0x557e58792c08 <__socket_families+264>, {ipv4 = {s_addr = 1414414845}, ipv6 = {__in6_union = {__s6_addr = "\375ANTRH \016\000\000\000\000\000\000\001s", 
                    __s6_addr16 = {16893, 21582, 18514, 3616, 0, 0, 0, 29441}, __s6_addr32 = {1414414845, 236996690, 0, 1929445376}}}}}, port = 21482}, tv = {tv_sec = 1728979775, tv_usec = 58635}, sfd = 0x7f6d60063930, 
          call = 0x7f6d696bb080, stream = 0x7f6d69891b60, media = 0x7f6d600f0320, media_out = 0x7f6d600ef6c0, sink = {sink = 0x7f6d696a2de0, handler = 0x557e587879d0 <__sh_noop_rtp>, kernel_output_idx = -1, attrs = {block_media = false, 
              silence_media = false, offer_answer = 1, rtcp_only = 0, transcoding = 1, egress = 0}}, cache_entry = 0x0, rtp = 0x7f6d603c4bd0, rtcp = 0x0, ssrc_in = 0x7f6d603ad3e0, ssrc_out = 0x7f6d603adc80, payload = {s = 0x0, len = 0}, 
          packets_out = {head = 0x0, tail = 0x0, length = 0}, ptime = 0}
        ret = 0
        ts = 125288
        p_left = 3
        tv_diff = -1
        ts_diff = 160
        dtxp = 0x0
        call = 0x7f6d696bb080
        ch = 0x7f6d696b4af0
        ps = 0x7f6d69891b60
        input_ch = 0x0
        ptime = 1
        dtxb_start = 1728979775
#11 0x0000557e586fe7f9 in codec_timers_run (p=0x7f6d698186e0) at codec.c:5513
        ct = 0x7f6d698186e0
#12 0x0000557e587031f7 in timerthread_run (p=0x557e587987a0 <codec_timers_thread>) at timerthread.c:62
        tt_obj = 0x7f6d698186e0
        sleeptime = -61
        tv = {tv_sec = 1728979775, tv_usec = 131761}
        tt = 0x557e587987a0 <codec_timers_thread>
        waker = {lock = 0x557e587987a8 <codec_timers_thread+8>, cond = 0x557e587987d0 <codec_timers_thread+48>}
#13 0x0000557e586fe84d in codec_timers_loop (p=0x0) at codec.c:5523
No locals.
#14 0x0000557e585c5094 in thread_detach_func (d=0x7f6d68a2ecc0) at helpers.c:258
        __cb = {__f = 0x557e585c469e <thread_detach_cleanup>, __x = 0x7f6d68a2ecc0, __next = 0x0}
        dt = 0x7f6d68a2ecc0
--Type <RET> for more, q to quit, c to continue without paging--
        t = 0x7f6d68a4e820
        scheduler = 0x0
#15 0x00007f6d7123d1d9 in start (p=0x7f6d6314f590) at src/thread/pthread_create.c:203
        args = 0x7f6d6314f590
        state = <optimized out>
#16 0x00007f6d7123f454 in __clone () at src/thread/x86_64/clone.s:22

(gdb) fr 10
#10 0x0000557e586efefc in __dtx_send_later (ct=0x7f6d698186e0) at codec.c:3417
3417	codec.c: No such file or directory.
(gdb) p *dtxb
$2 = {ct = {tt_obj = {obj = {ref = 2, free_func = 0x557e586f0eb4 <__dtx_free>, size = 424}, 
      tt = 0x557e587987a0 <codec_timers_thread>, next_check = {tv_sec = 0, tv_usec = 0}, last_run = {tv_sec = 1728979775, 
        tv_usec = 131761}}, next = {tv_sec = 1728979775, tv_usec = 131761}, timer_func = 0x557e586ee5d0 <__dtx_send_later>}, 
  lock = {__u = {__i = {0, 0, 0, 0, 0, 0, 0, 0, 0, 0}, __vi = {0, 0, 0, 0, 0, 0, 0, 0, 0, 0}, __p = {0x0, 0x0, 0x0, 0x0, 0x0}}}, 
  csh = 0x7f6d696b4af0, ptime = 1, tspp = 8, clockrate = 8000, call = 0x7f6d696bb080, packets = {head = 0x7f6d6b3c2d80, 
    tail = 0x7f6d6b8c4c40, length = 3}, last_mp = {raw = {s = 0x0, len = 0}, fsin = {address = {
        family = 0x557e58792c08 <__socket_families+264>, {ipv4 = {s_addr = 1414414845}, ipv6 = {__in6_union = {
              __s6_addr = "\375ANTRH \016\000\000\000\000\000\000\001s", __s6_addr16 = {16893, 21582, 18514, 3616, 0, 0, 0, 
                29441}, __s6_addr32 = {1414414845, 236996690, 0, 1929445376}}}}}, port = 21482}, tv = {tv_sec = 1728979775, 
      tv_usec = 58635}, sfd = 0x7f6d60063930, call = 0x7f6d696bb080, stream = 0x7f6d69891b60, media = 0x7f6d600f0320, 
    media_out = 0x7f6d600ef6c0, sink = {sink = 0x7f6d696a2de0, handler = 0x557e587879d0 <__sh_noop_rtp>, kernel_output_idx = -1, 
      attrs = {block_media = false, silence_media = false, offer_answer = 1, rtcp_only = 0, transcoding = 1, egress = 0}}, 
    cache_entry = 0x0, rtp = 0x7f6d603c4b50, rtcp = 0x0, ssrc_in = 0x7f6d603ad3e0, ssrc_out = 0x7f6d603adc80, payload = {s = 0x0, 
      len = 0}, packets_out = {head = 0x0, tail = 0x0, length = 0}, ptime = 0}, head_ts = 125288, ssrc = 2199246979, 
  start = 1728979775}
(gdb) p *ch
$3 = {h = {obj = {ref = 10, free_func = 0x557e586f2d95 <__free_ssrc_handler>, size = 400}, lock = {__u = {__i = {0, 0, 0, 0, 0, 
          0, 0, 0, 0, 0}, __vi = {0, 0, 0, 0, 0, 0, 0, 0, 0, 0}, __p = {0x0, 0x0, 0x0, 0x0, 0x0}}}, ssrc = 2199246979, 
    last_used = 1728979775}, handler = 0x7f6d68a064e0, decoder = 0x7f6d6031d540, encoder = 0x7f6d600744c0, chain = 0x0, 
  encoder_format = {clockrate = 8000, channels = 1, format = 1}, bitrate = 6700, ptime = 20, bytes_per_packet = 40, csch = {
    first_ts = 18880, last_ts = 125760, first_send = {tv_sec = 1728979761, tv_usec = 838761}, first_send_ts = 18432, 
    output_skew = 3985}, sample_buffer = 0x7f6d6b5e9560, dtx_buffer = 0x7f6d698186e0, dtmf_dsp = 0x0, dtmf_resampler = {
    swresample = 0x0, no_filter = false}, dtmf_format = {clockrate = 0, channels = 0, format = 0}, dtmf_ts = 0, 
  last_dtmf_event_ts = 0, dtmf_events = {head = 0x0, tail = 0x0, length = 0}, dtmf_event = {code = 0, volume = 0, ts = 0, 
    rand_code = 0, index = 0, block_dtmf = BLOCK_DTMF_OFF}, dtmf_state = {code = 0, volume = 0, ts = 0, rand_code = 0, index = 0, 
    block_dtmf = BLOCK_DTMF_OFF}, silence_events = {head = 0x0, tail = 0x0, length = 0}, dtmf_start_ts = 0, 
  dtmf_first_duration = 0, skip_pts = 0, rtp_mark = 0}

I will try amr-dtx = native but not sure that this solves it.
Looking for some clues about this problem.

@rfuchs
Copy link
Member

rfuchs commented Nov 19, 2024

There are many more occurrences that are happening in AMR transcoding scenario (branch mr11.5) due to CN dtx handling.

(gdb) bt full
#0  0x00007f6d6f7eb99a in ?? () from /usr/lib/libavcodec.so.58
No symbol table info available.
#1  0x00007f6d6f9f61bb in ?? () from /usr/lib/libavcodec.so.58
No symbol table info available.
#2  0x00007f6d6f9f7678 in avcodec_send_packet () from /usr/lib/libavcodec.so.58
No symbol table info available.
#3  0x0000557e5873b5e7 in avc_decoder_input (dec=0x7f6d689fe7e0, data=0x557e5878c430 <rtpe_config+1040>, out=0x7f6d6314ed40) at ../lib/codeclib.c:1004
        got_frame = 0
        err = 0x557e58770873 "failed to alloc av frame"
        av_ret = 0
        frame = 0x7f6d603eda00
        keep_going = 0
#4  0x0000557e5874c5b9 in cn_decoder_input (dec=0x7f6d689fe7e0, data=0x557e5878c430 <rtpe_config+1040>, out=0x7f6d6314ed40) at ../lib/codeclib.c:3439
        ptime = 1
        samples = 8
        ret = 82601444
#5  0x0000557e5873c2aa in __decoder_input_data (dec=0x7f6d689fe7e0, data=0x557e5878c430 <rtpe_config+1040>, ts=125288, ptime=0x0, callback=0x557e5874b032 <cn_append_frame>, u1=0x7f6d6314eea0, u2=0x0) at ../lib/codeclib.c:1119
        frames = {head = 0x0, tail = 0x0, length = 0}
        frame = 0x4ec65e4ffffffff
        ret = 0
        samples = 140107790480784
#6  0x0000557e5873c5b9 in decoder_input_data (dec=0x7f6d689fe7e0, data=0x557e5878c430 <rtpe_config+1040>, ts=125288, callback=0x557e5874b032 <cn_append_frame>, u1=0x7f6d6314eea0, u2=0x0) at ../lib/codeclib.c:1151
No locals.
#7  0x0000557e5874b0d0 in generic_cn_dtx (dec=0x7f6d6031d540, out=0x7f6d6314eea0, ptime=1) at ../lib/codeclib.c:3174
No locals.
#8  0x0000557e5873c2d7 in __decoder_input_data (dec=0x7f6d6031d540, data=0x0, ts=125288, ptime=0x7f6d6314eefc, callback=0x557e586f4621 <packet_decoded_fifo>, u1=0x7f6d696b4af0, u2=0x7f6d6314f1d0) at ../lib/codeclib.c:1121
        frames = {head = 0x0, tail = 0x0, length = 0}
        frame = 0x7f6dffffffff
        ret = 0
        samples = 140107933425952
#9  0x0000557e5873c686 in decoder_dtx (dec=0x7f6d6031d540, ts=125288, ptime=1, callback=0x557e586f4621 <packet_decoded_fifo>, u1=0x7f6d696b4af0, u2=0x7f6d6314f1d0) at ../lib/codeclib.c:1163
No locals.
#10 0x0000557e586efefc in __dtx_send_later (ct=0x7f6d698186e0) at codec.c:3417
        diff = 0
        dtxb = 0x7f6d698186e0
        mp_copy = {raw = {s = 0x0, len = 0}, fsin = {address = {family = 0x557e58792c08 <__socket_families+264>, {ipv4 = {s_addr = 1414414845}, ipv6 = {__in6_union = {__s6_addr = "\375ANTRH \016\000\000\000\000\000\000\001s", 
                    __s6_addr16 = {16893, 21582, 18514, 3616, 0, 0, 0, 29441}, __s6_addr32 = {1414414845, 236996690, 0, 1929445376}}}}}, port = 21482}, tv = {tv_sec = 1728979775, tv_usec = 58635}, sfd = 0x7f6d60063930, 
          call = 0x7f6d696bb080, stream = 0x7f6d69891b60, media = 0x7f6d600f0320, media_out = 0x7f6d600ef6c0, sink = {sink = 0x7f6d696a2de0, handler = 0x557e587879d0 <__sh_noop_rtp>, kernel_output_idx = -1, attrs = {block_media = false, 
              silence_media = false, offer_answer = 1, rtcp_only = 0, transcoding = 1, egress = 0}}, cache_entry = 0x0, rtp = 0x7f6d603c4bd0, rtcp = 0x0, ssrc_in = 0x7f6d603ad3e0, ssrc_out = 0x7f6d603adc80, payload = {s = 0x0, len = 0}, 
          packets_out = {head = 0x0, tail = 0x0, length = 0}, ptime = 0}
        ret = 0
        ts = 125288
        p_left = 3
        tv_diff = -1
        ts_diff = 160
        dtxp = 0x0
        call = 0x7f6d696bb080
        ch = 0x7f6d696b4af0
        ps = 0x7f6d69891b60
        input_ch = 0x0
        ptime = 1
        dtxb_start = 1728979775
#11 0x0000557e586fe7f9 in codec_timers_run (p=0x7f6d698186e0) at codec.c:5513
        ct = 0x7f6d698186e0
#12 0x0000557e587031f7 in timerthread_run (p=0x557e587987a0 <codec_timers_thread>) at timerthread.c:62
        tt_obj = 0x7f6d698186e0
        sleeptime = -61
        tv = {tv_sec = 1728979775, tv_usec = 131761}
        tt = 0x557e587987a0 <codec_timers_thread>
        waker = {lock = 0x557e587987a8 <codec_timers_thread+8>, cond = 0x557e587987d0 <codec_timers_thread+48>}
#13 0x0000557e586fe84d in codec_timers_loop (p=0x0) at codec.c:5523
No locals.
#14 0x0000557e585c5094 in thread_detach_func (d=0x7f6d68a2ecc0) at helpers.c:258
        __cb = {__f = 0x557e585c469e <thread_detach_cleanup>, __x = 0x7f6d68a2ecc0, __next = 0x0}
        dt = 0x7f6d68a2ecc0
--Type <RET> for more, q to quit, c to continue without paging--
        t = 0x7f6d68a4e820
        scheduler = 0x0
#15 0x00007f6d7123d1d9 in start (p=0x7f6d6314f590) at src/thread/pthread_create.c:203
        args = 0x7f6d6314f590
        state = <optimized out>
#16 0x00007f6d7123f454 in __clone () at src/thread/x86_64/clone.s:22

(gdb) fr 10
#10 0x0000557e586efefc in __dtx_send_later (ct=0x7f6d698186e0) at codec.c:3417
3417	codec.c: No such file or directory.
(gdb) p *dtxb
$2 = {ct = {tt_obj = {obj = {ref = 2, free_func = 0x557e586f0eb4 <__dtx_free>, size = 424}, 
      tt = 0x557e587987a0 <codec_timers_thread>, next_check = {tv_sec = 0, tv_usec = 0}, last_run = {tv_sec = 1728979775, 
        tv_usec = 131761}}, next = {tv_sec = 1728979775, tv_usec = 131761}, timer_func = 0x557e586ee5d0 <__dtx_send_later>}, 
  lock = {__u = {__i = {0, 0, 0, 0, 0, 0, 0, 0, 0, 0}, __vi = {0, 0, 0, 0, 0, 0, 0, 0, 0, 0}, __p = {0x0, 0x0, 0x0, 0x0, 0x0}}}, 
  csh = 0x7f6d696b4af0, ptime = 1, tspp = 8, clockrate = 8000, call = 0x7f6d696bb080, packets = {head = 0x7f6d6b3c2d80, 
    tail = 0x7f6d6b8c4c40, length = 3}, last_mp = {raw = {s = 0x0, len = 0}, fsin = {address = {
        family = 0x557e58792c08 <__socket_families+264>, {ipv4 = {s_addr = 1414414845}, ipv6 = {__in6_union = {
              __s6_addr = "\375ANTRH \016\000\000\000\000\000\000\001s", __s6_addr16 = {16893, 21582, 18514, 3616, 0, 0, 0, 
                29441}, __s6_addr32 = {1414414845, 236996690, 0, 1929445376}}}}}, port = 21482}, tv = {tv_sec = 1728979775, 
      tv_usec = 58635}, sfd = 0x7f6d60063930, call = 0x7f6d696bb080, stream = 0x7f6d69891b60, media = 0x7f6d600f0320, 
    media_out = 0x7f6d600ef6c0, sink = {sink = 0x7f6d696a2de0, handler = 0x557e587879d0 <__sh_noop_rtp>, kernel_output_idx = -1, 
      attrs = {block_media = false, silence_media = false, offer_answer = 1, rtcp_only = 0, transcoding = 1, egress = 0}}, 
    cache_entry = 0x0, rtp = 0x7f6d603c4b50, rtcp = 0x0, ssrc_in = 0x7f6d603ad3e0, ssrc_out = 0x7f6d603adc80, payload = {s = 0x0, 
      len = 0}, packets_out = {head = 0x0, tail = 0x0, length = 0}, ptime = 0}, head_ts = 125288, ssrc = 2199246979, 
  start = 1728979775}
(gdb) p *ch
$3 = {h = {obj = {ref = 10, free_func = 0x557e586f2d95 <__free_ssrc_handler>, size = 400}, lock = {__u = {__i = {0, 0, 0, 0, 0, 
          0, 0, 0, 0, 0}, __vi = {0, 0, 0, 0, 0, 0, 0, 0, 0, 0}, __p = {0x0, 0x0, 0x0, 0x0, 0x0}}}, ssrc = 2199246979, 
    last_used = 1728979775}, handler = 0x7f6d68a064e0, decoder = 0x7f6d6031d540, encoder = 0x7f6d600744c0, chain = 0x0, 
  encoder_format = {clockrate = 8000, channels = 1, format = 1}, bitrate = 6700, ptime = 20, bytes_per_packet = 40, csch = {
    first_ts = 18880, last_ts = 125760, first_send = {tv_sec = 1728979761, tv_usec = 838761}, first_send_ts = 18432, 
    output_skew = 3985}, sample_buffer = 0x7f6d6b5e9560, dtx_buffer = 0x7f6d698186e0, dtmf_dsp = 0x0, dtmf_resampler = {
    swresample = 0x0, no_filter = false}, dtmf_format = {clockrate = 0, channels = 0, format = 0}, dtmf_ts = 0, 
  last_dtmf_event_ts = 0, dtmf_events = {head = 0x0, tail = 0x0, length = 0}, dtmf_event = {code = 0, volume = 0, ts = 0, 
    rand_code = 0, index = 0, block_dtmf = BLOCK_DTMF_OFF}, dtmf_state = {code = 0, volume = 0, ts = 0, rand_code = 0, index = 0, 
    block_dtmf = BLOCK_DTMF_OFF}, silence_events = {head = 0x0, tail = 0x0, length = 0}, dtmf_start_ts = 0, 
  dtmf_first_duration = 0, skip_pts = 0, rtp_mark = 0}

I will try amr-dtx = native but not sure that this solves it. Looking for some clues about this problem.

What does frame 3 and print *dec produce?

@andingv
Copy link
Contributor Author

andingv commented Nov 19, 2024

hi,

(gdb) p *dec
$1 = {def = 0x558bcada0528 <__codec_defs+5704>, clockrate_fact = {mult = 1, div = 1}, codec_options = {amr = {bits_per_frame = 0x0, bitrates = 0x0, mode_change_interval = 0, cmr_interval = 0}, evs = {max_bw = EVS_BW_NB}}, 
  format_options = {amr = {interleaving = 0, mode_set = 0, mode_change_period = 0, octet_aligned = 0, crc = 0, robust_sorting = 0, mode_change_neighbor = 0}, ilbc = {mode = 0}, evs = {min_br = 0, max_br = 0, min_br_send = 0, 
      max_br_send = 0, min_br_recv = 0, max_br_recv = 0, min_bw = EVS_BW_NB, max_bw = EVS_BW_NB, min_bw_send = EVS_BW_NB, max_bw_send = EVS_BW_NB, min_bw_recv = EVS_BW_NB, max_bw_recv = EVS_BW_NB, mode_set = 0, mode_change_period = 0, 
      hf_only = 0, amr_io = 0, no_dtx = 0, no_dtx_recv = 0, cmr = 0, mode_change_neighbor = 0}, opus = {stereo_recv = 0, stereo_send = 0, fec_recv = 0, fec_send = 0, cbr = 0, usedtx = 0, maxplaybackrate = 0, sprop_maxcapturerate = 0, 
      maxaveragebitrate = 0, minptime = 0}}, dtx = {method_id = DTX_NATIVE, init = 0x0, cleanup = 0x0, do_dtx = 0x0, {cn = {cn_dec = 0x0, cn_payload = 0x0}}}, in_format = {clockrate = 8000, channels = 1, format = -1}, dec_out_format = {
    clockrate = 8000, channels = 1, format = 1}, dest_format = {clockrate = 8000, channels = 1, format = 1}, resampler = {swresample = 0x0, no_filter = true}, {avc = {avcctx = 0x7f9e333e6b40, avpkt = 0x7f9e2875d700, {amr = {
          bitrate_tracker = {0 <repeats 14 times>}, tracker_end = {tv_sec = 0, tv_usec = 0}, last_cmr = {tv_sec = 0, tv_usec = 0}}}}, bcg729 = 0x7f9e333e6b40, dtmf = {start_ts = 140317441289024, event = 678811392, duration = 0}, 
    evs = 0x7f9e333e6b40, opus = 0x7f9e333e6b40}, rtp_ts = 126088, pts = 65920, ptime = 1, event_func = 0x0, event_data = 0x0}

attached completed output.
GH1817.txt

@rfuchs
Copy link
Member

rfuchs commented Nov 20, 2024

How about

frame 10
p *mp_copy.ssrc_in
p *mp_copy.ssrc_out
p *mp_copy.ssrc_in->parent
p *mp_copy.ssrc_out->parent
p *ch

@andingv
Copy link
Contributor Author

andingv commented Nov 21, 2024

here you go

(gdb) p *mp_copy.ssrc_in
$2 = {parent = 0x7f9e31001d00, tracker = {lock = {__u = {__i = {0, 0, 0, 0, 0, 0, 0, 0, 0, 0}, __vi = {0, 0, 0, 0, 0, 0, 0, 0, 0, 0}, __p = {0x0, 0x0, 0x0, 0x0, 0x0}}}, last = '\b' <repeats 32 times>, last_idx = 1, 
    count = "\000\000\000\000\000\000\000\000 ", '\000' <repeats 118 times>, idx = "\377\377\377\377\377\377\377\377\000", '\377' <repeats 119 times>, most = "\b", '\377' <repeats 127 times>, most_len = 1}, ref = 0x7f9e31c36810, 
  srtp_index = 0, srtcp_index = 0, ssrc_map_out = 794772674, packets = {p = 0x582}, octets = {p = 0x39066}, last_seq = {p = 0x6f0f}, last_ts = {p = 0x1ee60}, last_sample = {p = 0x67022d4b96c9b}, sample_packets = {p = 0x0}, 
  sample_octets = {p = 0x0}, sample_packets_lost = {p = 0x0}, sample_duplicates = {p = 0x0}, next_rtcp = {tv_sec = 0, tv_usec = 0}}
(gdb) p *mp_copy.ssrc_out
$3 = {parent = 0x7f9e31dde050, tracker = {lock = {__u = {__i = {0, 0, 0, 0, 0, 0, 0, 0, 0, 0}, __vi = {0, 0, 0, 0, 0, 0, 0, 0, 0, 0}, __p = {0x0, 0x0, 0x0, 0x0, 0x0}}}, last = 'h' <repeats 32 times>, last_idx = 24, 
    count = '\000' <repeats 104 times>, " ", '\000' <repeats 22 times>, idx = '\377' <repeats 104 times>, "\000", '\377' <repeats 23 times>, most = "h", '\377' <repeats 127 times>, most_len = 1}, ref = 0x7f9e31c369e0, srtp_index = 0, 
  srtcp_index = 0, ssrc_map_out = 794772674, packets = {p = 0x2fc}, octets = {p = 0x923c}, last_seq = {p = 0x0}, last_ts = {p = 0x3c820}, last_sample = {p = 0x67022d4b96c9b}, sample_packets = {p = 0x0}, sample_octets = {p = 0x0}, 
  sample_packets_lost = {p = 0x0}, sample_duplicates = {p = 0x0}, next_rtcp = {tv_sec = 0, tv_usec = 0}}
(gdb) p *mp_copy.ssrc_in->parent
$4 = {h = {obj = {ref = 9, free_func = 0x558bcacdeed3 <__free_ssrc_entry_call>, size = 1464}, lock = {__u = {__i = {0, 16, 0, 0, 0, 0, 0, 0, 0, 0}, __vi = {0, 16, 0, 0, 0, 0, 0, 0, 0, 0}, __p = {0x1000000000, 0x0, 0x0, 0x0, 0x0}}}, 
    ssrc = 794772674, last_used = 1728195927}, input_ctx = {parent = 0x7f9e31001d00, tracker = {lock = {__u = {__i = {0, 0, 0, 0, 0, 0, 0, 0, 0, 0}, __vi = {0, 0, 0, 0, 0, 0, 0, 0, 0, 0}, __p = {0x0, 0x0, 0x0, 0x0, 0x0}}}, 
      last = '\b' <repeats 32 times>, last_idx = 1, count = "\000\000\000\000\000\000\000\000 ", '\000' <repeats 118 times>, idx = "\377\377\377\377\377\377\377\377\000", '\377' <repeats 119 times>, 
      most = "\b", '\377' <repeats 127 times>, most_len = 1}, ref = 0x7f9e31c36810, srtp_index = 0, srtcp_index = 0, ssrc_map_out = 794772674, packets = {p = 0x582}, octets = {p = 0x39066}, last_seq = {p = 0x6f0f}, last_ts = {
      p = 0x1ee60}, last_sample = {p = 0x67022d4b96c9b}, sample_packets = {p = 0x0}, sample_octets = {p = 0x0}, sample_packets_lost = {p = 0x0}, sample_duplicates = {p = 0x0}, next_rtcp = {tv_sec = 0, tv_usec = 0}}, output_ctx = {
    parent = 0x7f9e31001d00, tracker = {lock = {__u = {__i = {0, 0, 0, 0, 0, 0, 0, 0, 0, 0}, __vi = {0, 0, 0, 0, 0, 0, 0, 0, 0, 0}, __p = {0x0, 0x0, 0x0, 0x0, 0x0}}}, last = '\377' <repeats 32 times>, last_idx = 0, 
      count = '\000' <repeats 127 times>, idx = '\377' <repeats 128 times>, most = '\377' <repeats 128 times>, most_len = 0}, ref = 0x0, srtp_index = 0, srtcp_index = 0, ssrc_map_out = 1417907665, packets = {p = 0x0}, octets = {
      p = 0x0}, last_seq = {p = 0x0}, last_ts = {p = 0x0}, last_sample = {p = 0x67022d4b96c9b}, sample_packets = {p = 0x0}, sample_octets = {p = 0x0}, sample_packets_lost = {p = 0x0}, sample_duplicates = {p = 0x0}, next_rtcp = {
      tv_sec = 0, tv_usec = 0}}, sender_reports = {head = 0x7f9e31cea860, tail = 0x7f9e31cec4e0, length = 5}, rr_time_reports = {head = 0x0, tail = 0x0, length = 0}, stats_blocks = {head = 0x7f9e287f5c00, tail = 0x7f9e286a3d40, 
    length = 3}, lowest_mos = 0x7f9e2884ca80, highest_mos = 0x7f9e2887ea00, average_mos = {reported = {tv_sec = 0, tv_usec = 0}, jitter = 4, rtt = 18571, rtt_leg = 18553, packetloss = 0, mos = 131}, no_mos_count = 0, last_rtt = 6, 
  last_rtt_xr = 0, last_seq_tracked = 0, lost_bits = 4294967295, packets_lost = 0, duplicates = 0, sequencers = 0x7f9e28992240, jitter = 7, transit = 67595424, seq_diff = 0}
(gdb) p *mp_copy.ssrc_out->parent
$5 = {h = {obj = {ref = 9, free_func = 0x558bcacdeed3 <__free_ssrc_entry_call>, size = 1464}, lock = {__u = {__i = {0, 16, 0, 0, 0, 0, 0, 0, 0, 0}, __vi = {0, 16, 0, 0, 0, 0, 0, 0, 0, 0}, __p = {0x1000000000, 0x0, 0x0, 0x0, 0x0}}}, 
    ssrc = 794772674, last_used = 1728195925}, input_ctx = {parent = 0x7f9e31dde050, tracker = {lock = {__u = {__i = {0, 0, 0, 0, 0, 0, 0, 0, 0, 0}, __vi = {0, 0, 0, 0, 0, 0, 0, 0, 0, 0}, __p = {0x0, 0x0, 0x0, 0x0, 0x0}}}, 
      last = '\377' <repeats 32 times>, last_idx = 0, count = '\000' <repeats 127 times>, idx = '\377' <repeats 128 times>, most = '\377' <repeats 128 times>, most_len = 0}, ref = 0x0, srtp_index = 0, srtcp_index = 0, 
    ssrc_map_out = 3598747725, packets = {p = 0x0}, octets = {p = 0x0}, last_seq = {p = 0x0}, last_ts = {p = 0x0}, last_sample = {p = 0x67022d4b96c9b}, sample_packets = {p = 0x0}, sample_octets = {p = 0x0}, sample_packets_lost = {
      p = 0x0}, sample_duplicates = {p = 0x0}, next_rtcp = {tv_sec = 0, tv_usec = 0}}, output_ctx = {parent = 0x7f9e31dde050, tracker = {lock = {__u = {__i = {0, 0, 0, 0, 0, 0, 0, 0, 0, 0}, __vi = {0, 0, 0, 0, 0, 0, 0, 0, 0, 0}, __p = {
            0x0, 0x0, 0x0, 0x0, 0x0}}}, last = 'h' <repeats 32 times>, last_idx = 24, count = '\000' <repeats 104 times>, " ", '\000' <repeats 22 times>, idx = '\377' <repeats 104 times>, "\000", '\377' <repeats 23 times>, 
      most = "h", '\377' <repeats 127 times>, most_len = 1}, ref = 0x7f9e31c369e0, srtp_index = 0, srtcp_index = 0, ssrc_map_out = 794772674, packets = {p = 0x2fc}, octets = {p = 0x923c}, last_seq = {p = 0x0}, last_ts = {p = 0x3c820}, 
    last_sample = {p = 0x67022d4b96c9b}, sample_packets = {p = 0x0}, sample_octets = {p = 0x0}, sample_packets_lost = {p = 0x0}, sample_duplicates = {p = 0x0}, next_rtcp = {tv_sec = 0, tv_usec = 0}}, sender_reports = {head = 0x0, 
    tail = 0x0, length = 0}, rr_time_reports = {head = 0x0, tail = 0x0, length = 0}, stats_blocks = {head = 0x0, tail = 0x0, length = 0}, lowest_mos = 0x0, highest_mos = 0x0, average_mos = {reported = {tv_sec = 0, tv_usec = 0}, 
    jitter = 0, rtt = 0, rtt_leg = 0, packetloss = 0, mos = 0}, no_mos_count = 0, last_rtt = 0, last_rtt_xr = 0, last_seq_tracked = 0, lost_bits = 4294967295, packets_lost = 0, duplicates = 0, sequencers = 0x0, jitter = 7, 
  transit = 135196160, seq_diff = 62}
(gdb) p *ch
$6 = {h = {obj = {ref = 10, free_func = 0x558bcacfed95 <__free_ssrc_handler>, size = 400}, lock = {__u = {__i = {0, 0, 0, 0, 0, 0, 0, 0, 0, 0}, __vi = {0, 0, 0, 0, 0, 0, 0, 0, 0, 0}, __p = {0x0, 0x0, 0x0, 0x0, 0x0}}}, ssrc = 794772674, 
    last_used = 1728195930}, handler = 0x7f9e286da010, decoder = 0x7f9e31ce4a80, encoder = 0x7f9e28948e10, chain = 0x0, encoder_format = {clockrate = 16000, channels = 1, format = 1}, bitrate = 14250, ptime = 20, bytes_per_packet = 80, 
  csch = {first_ts = 3680, last_ts = 126560, first_send = {tv_sec = 1728195915, tv_usec = 717058}, first_send_ts = 3664, output_skew = 6977}, sample_buffer = 0x7f9e30fd9420, dtx_buffer = 0x7f9e31cd03d0, dtmf_dsp = 0x0, dtmf_resampler = {
    swresample = 0x0, no_filter = false}, dtmf_format = {clockrate = 0, channels = 0, format = 0}, dtmf_ts = 0, last_dtmf_event_ts = 0, dtmf_events = {head = 0x0, tail = 0x0, length = 0}, dtmf_event = {code = 0, volume = 0, ts = 0, 
    rand_code = 0, index = 0, block_dtmf = BLOCK_DTMF_OFF}, dtmf_state = {code = 0, volume = 0, ts = 0, rand_code = 0, index = 0, block_dtmf = BLOCK_DTMF_OFF}, silence_events = {head = 0x0, tail = 0x0, length = 0}, dtmf_start_ts = 0, 
  dtmf_first_duration = 0, skip_pts = 0, rtp_mark = 0}

@rfuchs
Copy link
Member

rfuchs commented Nov 22, 2024

Ok, one more.

frame 7
p *dec

@andingv
Copy link
Contributor Author

andingv commented Nov 22, 2024

sure

(gdb) fr 7
#7  0x0000558bcad570d0 in generic_cn_dtx (dec=0x7f9e31ce4a80, out=0x7f9e29eedea0, ptime=1) at ../lib/codeclib.c:3174
3174	../lib/codeclib.c: No such file or directory.
(gdb) p *dec
$7 = {def = 0x558bcad9eee0 <__codec_defs>, clockrate_fact = {mult = 1, div = 1}, codec_options = {amr = {bits_per_frame = 0x0, bitrates = 0x0, mode_change_interval = 0, cmr_interval = 0}, evs = {max_bw = EVS_BW_NB}}, format_options = {
    amr = {interleaving = 0, mode_set = 0, mode_change_period = 0, octet_aligned = 0, crc = 0, robust_sorting = 0, mode_change_neighbor = 0}, ilbc = {mode = 0}, evs = {min_br = 0, max_br = 0, min_br_send = 0, max_br_send = 0, 
      min_br_recv = 0, max_br_recv = 0, min_bw = EVS_BW_NB, max_bw = EVS_BW_NB, min_bw_send = EVS_BW_NB, max_bw_send = EVS_BW_NB, min_bw_recv = EVS_BW_NB, max_bw_recv = EVS_BW_NB, mode_set = 0, mode_change_period = 0, hf_only = 0, 
      amr_io = 0, no_dtx = 0, no_dtx_recv = 0, cmr = 0, mode_change_neighbor = 0}, opus = {stereo_recv = 0, stereo_send = 0, fec_recv = 0, fec_send = 0, cbr = 0, usedtx = 0, maxplaybackrate = 0, sprop_maxcapturerate = 0, 
      maxaveragebitrate = 0, minptime = 0}}, dtx = {method_id = DTX_CN, init = 0x558bcad570d2 <generic_cn_dtx_init>, cleanup = 0x558bcad57171 <generic_cn_dtx_cleanup>, do_dtx = 0x558bcad5706c <generic_cn_dtx>, {cn = {
        cn_dec = 0x7f9e31ce4e70, cn_payload = 0x558bcad98430 <rtpe_config+1040>}}}, in_format = {clockrate = 8000, channels = 1, format = -1}, dec_out_format = {clockrate = 8000, channels = 1, format = 1}, dest_format = {
    clockrate = 16000, channels = 1, format = 1}, resampler = {swresample = 0x7f9e288fe040, no_filter = false}, {avc = {avcctx = 0x7f9e333e6600, avpkt = 0x7f9e2875af00, {amr = {bitrate_tracker = {0 <repeats 14 times>}, tracker_end = {
            tv_sec = 0, tv_usec = 0}, last_cmr = {tv_sec = 0, tv_usec = 0}}}}, bcg729 = 0x7f9e333e6600, dtmf = {start_ts = 140317441287680, event = 678801152, duration = 0}, evs = 0x7f9e333e6600, opus = 0x7f9e333e6600}, rtp_ts = 126088, 
  pts = 122408, ptime = 20, event_func = 0x558bcacf7434 <codec_decoder_event>, event_data = 0x7f9e33bbd3f0}
(gdb)

@rfuchs
Copy link
Member

rfuchs commented Nov 26, 2024 •
edited
Loading

Do you have debug symbols for libavcodec.so.58? There isn't anything obvious wrong on the rtpengine side, so would be interesting to know where exactly ffmpeg crashed. (Also the exact source/version would be important to know)

@andingv
Copy link
Contributor Author

andingv commented Nov 26, 2024

Hi thanks,
ok I guessed it could be in the library. On that system we use ffmpeg for alpine with custom build flags

$ tail APKGBUILD.patch
 		--enable-vdpau \
 		--enable-vulkan \
 		--enable-libdrm \
+		--enable-version3 \
+		--enable-libvo-amrwbenc \
+		--enable-libopencore-amrwb \
+		--enable-libopencore-amrnb \
 		$_asm $_dbg
 	make
 }

Dockerfile:

...
COPY APKBUILD.patch /tmp/build/aports/community/ffmpeg4/

RUN cd /tmp/build/aports/community/ffmpeg4/ \
    && git apply APKBUILD.patch \
    && abuild -F -r

ARG OS_IMAGE_TAG=3.16.1
# build rtp-engine
FROM $OS_IMAGE:$OS_IMAGE_TAG

COPY --from=ffmpeg /root/.abuild/*.pub /etc/apk/keys/
COPY --from=ffmpeg /root/packages/community /root/packages/community/

RUN echo "@testing https://dl-cdn.alpinelinux.org/alpine/edge/testing" >> /etc/apk/repositories \
    && echo "@local /root/packages/community/" >> /etc/apk/repositories \
    && apk --no-cache add \
        bcg729-dev@testing \
        build-base \
        ffmpeg4-dev@local \
...

I'll add the '-g' option when building it next time. meanwhile it hasn't crashed any more after disabling dtx.
This means I have to try to reproduce the crash locally, which I am not sure of, otherwise as per plan i am moving the container image to debian 12 anyway so either crash problem goes away or we'll be able to get the proper debug output.

@andingv
Copy link
Contributor Author

andingv commented Nov 26, 2024

The version is ffmpeg4 - 4.4.5, in the so filename i can see libavcodec.so.58.134.100
ok I have found the base OS and port version like on the host so I am adding there options "!strip debug" to try something quick now.

@andingv
Copy link
Contributor Author

andingv commented Nov 26, 2024

It was a 4.4.4 actually but that didn't work
warning: .dynamic section for "/usr/lib/libavformat.so.58" is not at the expected address (wrong library or version mismatch?)
ok i'll see if it's reproducible elsewhere so that i can obtain a new core file or otherwise hold on while updating to debian 12..will close this issue in that case.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@rfuchs @andingv