triggering MEXT irq. is hanging freeRTOS demo

[jaaaaaaaaan]

I need help again please. I can't figure out why, but triggering MEXT irq. causes freeRTOS demo to hang after a while. I've changed only the uart baud to 115200 (but I admit that running it on 19200 makes it crash a little bit later), otherwise it is in original state, but I've tried to change stack size (to more) and it did not changed anything. My VHDL top looks like this:

use ieee.std_logic_1164.all;
use ieee.numeric_std.all;

library neorv32;
use neorv32.neorv32_package.all;

library Wishbone;
use Wishbone.Wishbone_pkg.all;


entity neorv32_ProcessorTop_Test is
  generic ( WITH_BOOTLOADER : boolean := true );
  port (
    -- Global control --
    clk_i       : in  std_ulogic := '0'; -- global clock, rising edge
    rstn_i      : in  std_ulogic := '0'; -- global reset, low-active, async
    
    -- UART0 --
    uart0_txd_o : out std_ulogic; -- UART0 send data
    uart0_rxd_i : in  std_ulogic := '0'; -- UART0 receive data
    -- leds
    led_o      : out std_ulogic_vector(3 downto 0);
    -- RGB leds --
    rled_o : out std_ulogic_vector(3 downto 0);
    gled_o : out std_ulogic_vector(3 downto 0);
    bled_o : out std_ulogic_vector(3 downto 0);
    -- motors --
    encoder_ab1_i : in std_ulogic_vector(1 downto 0);
    motor1_dir_o : out std_ulogic;
    motor1_pwm_o : out std_ulogic;
    encoder_ab2_i : in std_ulogic_vector(1 downto 0);
    motor2_dir_o : out std_ulogic;
    motor2_pwm_o : out std_ulogic;
    -- ADC signals --
    vauxp15: in  STD_LOGIC:= '0';                         -- Auxiliary Channel 1
    vauxn15: in  STD_LOGIC:= '0';
    vauxp7: in  STD_LOGIC:= '0';                         -- Auxiliary Channel 2
    vauxn7: in  STD_LOGIC:= '0';
    vauxp4 : in  STD_LOGIC:= '0';                         -- Auxiliary Channel 4
    vauxn4 : in  STD_LOGIC:= '0';
    vauxp5 : in  STD_LOGIC:= '0';                         -- Auxiliary Channel 5
    vauxn5 : in  STD_LOGIC:= '0';
    vauxp6 : in  STD_LOGIC:= '0';                         -- Auxiliary Channel 6
    vauxn6 : in  STD_LOGIC:= '0';
    -- IR LED's mux --
    IR_LED_MUX_O : out STD_ULOGIC_VECTOR (4 downto 0) );
end entity;

architecture neorv32_ProcessorTop_rtl of neorv32_ProcessorTop_Test is
  --Addresses of peripherals--
  constant led_addr : STD_ULOGIC_VECTOR(31 downto 0) := x"A0000000";
  constant motor1_addr : STD_ULOGIC_VECTOR (31 downto 0) := x"A0000004";
  constant motor2_addr : STD_ULOGIC_VECTOR (31 downto 0) := x"A0000008";
  constant sensor1_addr : STD_ULOGIC_VECTOR (31 downto 0) := x"A000000C";
  constant sensor2_addr : STD_ULOGIC_VECTOR (31 downto 0) := x"A0000010";
  constant sensor3_addr : STD_ULOGIC_VECTOR (31 downto 0) := x"A0000014";
  constant sensor4_addr : STD_ULOGIC_VECTOR (31 downto 0) := x"A0000018";
  constant sensor5_addr : STD_ULOGIC_VECTOR (31 downto 0) := x"A000001C";
  constant irq_cnt_cause_addr : STD_ULOGIC_VECTOR (31 downto 0) := x"F0000000";
  constant irq_cnt_cfg_addr : STD_ULOGIC_VECTOR (31 downto 0) := x"F0000004";
  
  -- gpio output --
  signal gpio_out : std_ulogic_vector(63 downto 0) := (others => '0');
  -- Wishbone bus interface --
  signal wb_adr_o : std_ulogic_vector (31 downto 0) := (others => '0');
  signal wb_dat_i : std_ulogic_vector (31 downto 0) := (others => '0');
  signal wb_dat_o : std_ulogic_vector (31 downto 0) := (others => '0');
  signal wb_we_o  : std_ulogic := '0'; 
  signal wb_sel_o : std_ulogic_vector (3 downto 0) := (others => '0');
  signal wb_stb_o : std_ulogic := '0';    
  signal wb_cyc_o : std_ulogic := '0';    
  signal wb_ack_i : std_ulogic := '0';
  -- Peripherals --
   --leds
  signal reg_leds : std_ulogic_vector(31 downto 0) := (others => '0');
  signal wb_leds_d_o : std_ulogic_vector(31 downto 0) := (others => '0');
  signal wb_leds_ack_o : std_ulogic := '0';
   --motors and encoders
  signal wb_motor1_d_o : std_ulogic_vector(31 downto 0) := (others => '0');
  signal wb_motor1_ack_o : std_ulogic := '0';
  signal motor1_irq_o : std_ulogic := '0';
  signal wb_motor2_d_o : std_ulogic_vector(31 downto 0) := (others => '0');
  signal wb_motor2_ack_o : std_ulogic := '0';
  signal motor2_irq_o : std_ulogic := '0';
  --sensors
  signal wb_sensor1_o : std_ulogic_vector (31 downto 0) := (others => '0');
  signal wb_sensor1_ack_o : std_ulogic := '0';
  signal wb_sensor2_o : std_ulogic_vector (31 downto 0) := (others => '0');
  signal wb_sensor2_ack_o : std_ulogic := '0';
  signal wb_sensor3_o : std_ulogic_vector (31 downto 0) := (others => '0');
  signal wb_sensor3_ack_o : std_ulogic := '0';
  signal wb_sensor4_o : std_ulogic_vector (31 downto 0) := (others => '0');
  signal wb_sensor4_ack_o : std_ulogic := '0';
  signal wb_sensor5_o : std_ulogic_vector (31 downto 0) := (others => '0');
  signal wb_sensor5_ack_o : std_ulogic := '0';
  signal sensor_mux_irq_o : std_ulogic := '0';
  --IRQ controller
  signal wb_irq_cnt_cause_o : std_ulogic_vector (31 downto 0) := (others => '0');
  signal wb_irq_cnt_cause_ack_o : std_ulogic := '0'; 
  signal wb_irq_cnt_cfg_o : std_ulogic_vector (31 downto 0) := (others => '0');
  signal wb_irq_cnt_cfg_ack_o : std_ulogic := '0'; 
  signal mext_irq : std_ulogic := '0';
  signal irq_lines : std_ulogic_vector(2 downto 0) := (others => '0');
  -- others -- 
  signal rgb_led_pwm : std_ulogic := '0';
  
  signal mtime : std_ulogic_vector (63 downto 0) := (others => '0');

  signal delet : std_ulogic_vector(3 downto 0) := (others => '0');
 
begin

  -- The Core Of The Problem ----------------------------------------------------------------
  -- -------------------------------------------------------------------------------------------
  neorv32_top_inst: neorv32_top
  generic map (
    -- General --
    CLOCK_FREQUENCY              => 100000000,   -- clock frequency of clk_i in Hz
    INT_BOOTLOADER_EN            => WITH_BOOTLOADER,        -- boot configuration: true = boot explicit bootloader; false = boot from int/ext (I)MEM
    USER_CODE                    => x"00000000", -- custom user code
    HW_THREAD_ID                 => 0,           -- hardware thread id (hartid)
    -- On-Chip Debugger (OCD) --
    ON_CHIP_DEBUGGER_EN          => false,       -- implement on-chip debugger
    -- RISC-V CPU Extensions --
    CPU_EXTENSION_RISCV_A        => true,        -- implement atomic extension?
    CPU_EXTENSION_RISCV_C        => true,        -- implement compressed extension?
    CPU_EXTENSION_RISCV_E        => false,       -- implement embedded RF extension?
    CPU_EXTENSION_RISCV_M        => true,        -- implement muld/div extension?
    CPU_EXTENSION_RISCV_U        => false,        -- implement user mode extension?
    CPU_EXTENSION_RISCV_Zfinx    => false,       -- implement 32-bit floating-point extension (using INT reg!)
    CPU_EXTENSION_RISCV_Zicsr    => true,        -- implement CSR system?
    CPU_EXTENSION_RISCV_Zifencei => false,       -- implement instruction stream sync.?
    -- Extension Options --
    FAST_MUL_EN                  => true,       -- use DSPs for M extension's multiplier
    FAST_SHIFT_EN                => true,       -- use barrel shifter for shift operations
    CPU_CNT_WIDTH                => 64,          -- total width of CPU cycle and instret counters (0..64)
    -- Physical Memory Protection (PMP) --
    PMP_NUM_REGIONS              => 0,           -- number of regions (0..64)
    PMP_MIN_GRANULARITY          => 64*1024,     -- minimal region granularity in bytes, has to be a power of 2, min 8 bytes
    -- Hardware Performance Monitors (HPM) --
    HPM_NUM_CNTS                 => 4,           -- number of implemented HPM counters (0..29)
    HPM_CNT_WIDTH                => 40,          -- total size of HPM counters (0..64)
    -- Internal Instruction memory --
    MEM_INT_IMEM_EN              => true,        -- implement processor-internal instruction memory
    MEM_INT_IMEM_SIZE            => 32*1024,     -- size of processor-internal instruction memory in bytes
    -- Internal Data memory --
    MEM_INT_DMEM_EN              => true,        -- implement processor-internal data memory
    MEM_INT_DMEM_SIZE            => 16*1024,      -- size of processor-internal data memory in bytes
    -- Internal Cache memory --
    ICACHE_EN                    => false,       -- implement instruction cache
    ICACHE_NUM_BLOCKS            => 4,           -- i-cache: number of blocks (min 1), has to be a power of 2
    ICACHE_BLOCK_SIZE            => 64,          -- i-cache: block size in bytes (min 4), has to be a power of 2
    ICACHE_ASSOCIATIVITY         => 1,           -- i-cache: associativity / number of sets (1=direct_mapped), has to be a power of 2
    -- External memory interface --
    MEM_EXT_EN                   => true,       -- implement external memory bus interface?
    MEM_EXT_TIMEOUT              => 10,           -- cycles after a pending bus access auto-terminates (0 = disabled)
    -- Processor peripherals --
    IO_GPIO_EN                   => true,        -- implement general purpose input/output port unit (GPIO)?
    IO_MTIME_EN                  => true,        -- implement machine system timer (MTIME)?
    IO_UART0_EN                  => true,        -- implement primary universal asynchronous receiver/transmitter (UART0)?
    IO_UART1_EN                  => false,       -- implement secondary universal asynchronous receiver/transmitter (UART1)?
    IO_SPI_EN                    => false,       -- implement serial peripheral interface (SPI)?
    IO_TWI_EN                    => false,       -- implement two-wire interface (TWI)?
    IO_PWM_NUM_CH                => 0,           -- number of PWM channels to implement (0..60); 0 = disabled
    IO_WDT_EN                    => true,        -- implement watch dog timer (WDT)?
    IO_TRNG_EN                   => false,       -- implement true random number generator (TRNG)?
    IO_CFS_EN                    => false,       -- implement custom functions subsystem (CFS)?
    IO_CFS_CONFIG                => x"00000000", -- custom CFS configuration generic
    IO_CFS_IN_SIZE               => 32,          -- size of CFS input conduit in bits
    IO_CFS_OUT_SIZE              => 32,          -- size of CFS output conduit in bits
    IO_NEOLED_EN                 => false,        -- implement NeoPixel-compatible smart LED interface (NEOLED)?
    --- interrupts --
    XIRQ_NUM_CH                  => 0,      -- number of external IRQ channels (0..32)
    XIRQ_TRIGGER_TYPE            => (others => '0') -- trigger type: 0=level, 1=edge

  )
  port map (
    -- Global control --
    clk_i       => clk_i,           -- global clock, rising edge
    rstn_i      => rstn_i,          -- global reset, low-active, async
    -- JTAG on-chip debugger interface (available if ON_CHIP_DEBUGGER_EN = true) --
    jtag_trst_i => '0',             -- low-active TAP reset (optional)
    jtag_tck_i  => '0',             -- serial clock
    jtag_tdi_i  => '0',             -- serial data input
    jtag_tdo_o  => open,            -- serial data output
    jtag_tms_i  => '0',             -- mode select
    -- Wishbone bus interface (available if MEM_EXT_EN = true) --
    wb_tag_o    => open,            -- tag
    wb_adr_o    => wb_adr_o,            -- address
    wb_dat_i    => wb_dat_i, -- read data
    wb_dat_o    => wb_dat_o,            -- write data
    wb_we_o     => wb_we_o,            -- read/write
    wb_sel_o    => wb_sel_o,            -- byte enable
    wb_stb_o    => wb_stb_o,            -- strobe
    wb_cyc_o    => wb_cyc_o,            -- valid cycle
    wb_lock_o   => open,            -- exclusive access request
    wb_ack_i    => wb_ack_i,             -- transfer acknowledge
    wb_err_i    => '0',             -- transfer error
    -- Advanced memory control signals (available if MEM_EXT_EN = true) --
    fence_o     => open,            -- indicates an executed FENCE operation
    fencei_o    => open,            -- indicates an executed FENCEI operation
    -- GPIO (available if IO_GPIO_EN = true) --
    gpio_o      => gpio_out,        -- parallel output
    gpio_i      => (others => '0'), -- parallel input
    -- primary UART0 (available if IO_UART0_EN = true) --
    uart0_txd_o => uart0_txd_o,     -- UART0 send data
    uart0_rxd_i => uart0_rxd_i,     -- UART0 receive data
    uart0_rts_o => open,            -- hw flow control: UART0.RX ready to receive ("RTR"), low-active, optional
    uart0_cts_i => '0',             -- hw flow control: UART0.TX allowed to transmit, low-active, optional
    -- secondary UART1 (available if IO_UART1_EN = true) --
    uart1_txd_o => open,            -- UART1 send data
    uart1_rxd_i => '0',             -- UART1 receive data
    uart1_rts_o => open,            -- hw flow control: UART1.RX ready to receive ("RTR"), low-active, optional
    uart1_cts_i => '0',             -- hw flow control: UART1.TX allowed to transmit, low-active, optional
    -- SPI (available if IO_SPI_EN = true) --
    spi_sck_o   => open,            -- SPI serial clock
    spi_sdo_o   => open,            -- controller data out, peripheral data in
    spi_sdi_i   => '0',             -- controller data in, peripheral data out
    spi_csn_o   => open,            -- SPI CS
    -- TWI (available if IO_TWI_EN = true) --
    twi_sda_io  => open,            -- twi serial data line
    twi_scl_io  => open,            -- twi serial clock line
    -- PWM (available if IO_PWM_NUM_CH > 0) --
    pwm_o       => open,            -- pwm channels
    -- Custom Functions Subsystem IO --
    cfs_in_i    => (others => '0'), -- custom inputs
    cfs_out_o   => open,            -- custom outputs
    -- NeoPixel-compatible smart LED interface (available if IO_NEOLED_EN = true) --
    neoled_o    => open,            -- async serial data line
    -- System time --
    mtime_i     => (others => '0'), -- current system time from ext. MTIME (if IO_MTIME_EN = false)
    mtime_o     => mtime,            -- current system time from int. MTIME (if IO_MTIME_EN = true)
    -- Interrupts --
    nm_irq_i    => '0',             -- non-maskable interrupt
    mtime_irq_i => '0',             -- machine timer interrupt, available if IO_MTIME_EN = false
    msw_irq_i   => '0',             -- machine software interrupt
    mext_irq_i  => mext_irq         -- machine external interrupt
    );

  --irq_lines(2) <= motor2_irq_o;
  --irq_lines(1) <= motor1_irq_o;
  --irq_lines(0) <= sensor_mux_irq_o;
  
   --irq_lines <= (0 => delet(0), others => '0');
  mext_irq <= delet(0);
  process (clk_i)
    variable pwm_counter : integer range 0 to 665535 := 0;
  begin
    if rising_edge(clk_i) then
      if pwm_counter < 665535 then
        pwm_counter := pwm_counter + 1;
      else
        pwm_counter := 0;
      end if;
      if pwm_counter = 665535/8 then
        delet(0) <=  '1';
      else
        delet(0) <= '0';
      end if;
    end if;
  end process;
  wb_iconn : component wb_interconnect
    generic map ( ADDR0 => led_addr,
                  ADDR1 => motor1_addr,
                  ADDR2 => motor2_addr,
                  ADDR3 => sensor1_addr,
                  ADDR4 => sensor2_addr,
                  ADDR5 => sensor3_addr,
                  ADDR6 => sensor4_addr,
                  ADDR7 => sensor5_addr,
                  ADDR8 => irq_cnt_cause_addr,
                  ADDR9 => irq_cnt_cfg_addr)
    port map ( D_OUT => wb_dat_i,
               D_IN0 => wb_leds_d_o,
               D_IN1 => wb_motor1_d_o,
               D_IN2 => wb_motor2_d_o,
               D_IN3 => wb_sensor1_o,
               D_IN4 => wb_sensor2_o,
               D_IN5 => wb_sensor3_o,
               D_IN6 => wb_sensor4_o,
               D_IN7 => wb_sensor5_o,
               D_IN8 => wb_irq_cnt_cause_o,
               D_IN9 => wb_irq_cnt_cfg_o,
              
               ACK_OUT => wb_ack_i,
               ACK_IN0 => wb_leds_ack_o,
               ACK_IN1 => wb_motor1_ack_o,
               ACK_IN2 => wb_motor2_ack_o,
               ACK_IN3 => wb_sensor1_ack_o,
               ACK_IN4 => wb_sensor2_ack_o,    
               ACK_IN5 => wb_sensor3_ack_o,
               ACK_IN6 => wb_sensor4_ack_o,
               ACK_IN7 => wb_sensor5_ack_o,
               ACK_IN8 => wb_irq_cnt_cause_ack_o,
               ACK_IN9 => wb_irq_cnt_cfg_ack_o,
               ADDR_IN => wb_adr_o );
  
  wb_leds : component wb_slave_node
    generic map( ADDR => led_addr )
    port map( RST_I => rstn_i,
              CLK_I => clk_i,
              WE_I => wb_we_o,
              STB_I => wb_stb_o,
              CYC_I => wb_cyc_o,
              ACK_O => wb_leds_ack_o,
              SEL_I => wb_sel_o,
              ADDR_I => wb_adr_o,
              D_I => wb_dat_o,
              D_O => wb_leds_d_o,
              REG_O => reg_leds,
              REG_I => reg_leds );

  rled_o <= reg_leds(15 downto 12) when rgb_led_pwm = '1' else (others => '0');
  gled_o <= reg_leds(11 downto 8) when rgb_led_pwm = '1' else (others => '0');
  bled_o <= reg_leds(7 downto 4) when rgb_led_pwm = '1' else (others => '0');
  led_o(3 downto 0) <= reg_leds(3 downto 0);

  rgb_led_dimmer : process (clk_i)
    variable pwm_counter : integer range 0 to 65535 := 0;
  begin
    if rising_edge(clk_i) then
      if pwm_counter < 65535 then
        pwm_counter := pwm_counter + 1;
      else
        pwm_counter := 0;
      end if;
      if pwm_counter < 65535/8 then
        rgb_led_pwm <= '1';
      else
        rgb_led_pwm <= '0';
      end if;
    end if;
    
  end process rgb_led_dimmer;

end architecture;

I've tried to comment out everything in SystemIrqHandler too but it will crash eventually too. What can be the cause of this? Is it lacking context save/restore or something?
I'm sorry for flooding discussions section, but again I really have no clue what went wrong this time too.

[stnolting]

The problem with FreeRTOS and external interrupts is that there is basically no unified interface for this. Many RISC-V cores have a core-local or platform-local interrupt controller connected to the "RISC-V external machine interrupt". These controller provide a more sophisticated interface for acknowledging interrupts.

Furthermore, there is no "default" option in FreeRTOS that could be used to support NEORV32-specific interrupts like the FIRQs. I am working on a "plug-in" for the kernel, but somehow that got shelved... 🙈

Anyway, the NEORV32 mext_irq_i interrupt is high-level triggered. So if you interrupt source drives this signal high for more than one clock cycle, several interrupt requests might be generated. Since the mext_irq_i interrupt has higher priority than the timer tick interrupt (via processor-internal MTIME timer) a permanent trigger on mext_irq_i will block the system. You should add an edge detector to your interrupt signal so that only a rising-edge triggers the external machine interrupt (setting it high for only one cycle). Maybe I should add default edge detector to that interrupt inputs... 🤔

I have tested the "blinky" RTOS demo with the external interrupt. In my test scenario mext_irq_i is driven by gpio_o(0) via an edge detector. This setup works stable so far.

In theory, even a permanent external interrupt should not crash the system - it should only stall it. However, it looks like a permanent external interrupt is causing a stack or heap overflow that finally crashes the kernel (tested with mext_irq_i <= '1'). I will do some more tests there.

[jaaaaaaaaan]

The problem with FreeRTOS and external interrupts is that there is basically no unified interface for this. Many RISC-V cores have a core-local or platform-local interrupt controller connected to the "RISC-V external machine interrupt". These controller provide a more sophisticated interface for acknowledging interrupts.

Yes I have wrote one myself, I did not included it in this question to "remove it from equation", and be certain that this problem is not related to it.

So if you interrupt source drives this signal high for more than one clock cycle, several interrupt requests might be generated (...) You should add an edge detector to your interrupt signal so that only a rising-edge triggers the external machine interrupt (setting it high for only one cycle).

I have tested the "blinky" RTOS demo with the external interrupt. In my test scenario mext_irq_i is driven by gpio_o(0) via an edge detector. This setup works stable so far.

I have triggered it with single pulse for one clk period, but it didn't help. It work for some time but it will eventually crash (at least when trigger occurs periodically). However your test case triggers an IRQ "in sync" with code (depends strictly on it), and mine does not, as it is just an output from some logic. Can it be that IRQ epilogue gets interrupted ant that's why it crashes?
I'll go examine it myself too, but I'm rather newbie in this, as you probably noticed.

[stnolting]

Yes I have wrote one myself, I did not included it in this question to "remove it from equation", and be certain that this problem is not related to it.

The "problem" with the external interrupt / core-local interrupt controller concept is that the RISC-V spec. states something like this:

• The external interrupt is triggered once (so one cycle high)
• The interrupt handler does its thing; at the end it has to acknowledge the external interrupt in some way
• The external interrupt can only fire again if it has been acknowledged

However, there should be no problem when there is a permanent interrupt at any interrupt channel. The CPU control ensure that at least one instruction of the interrupted program is executed before another interrupt is serviced. Hence, permanent interrupts do not stall the actual program - they will just slow down execution speed.

I am doing some tests right now.
Permanent interrupts seem to be ok - the interrupted program does still advance. But something isn't right there. Seems like there is a problem with global variables... At least gdb does not show any progress (changing values) in the interrupted program's global variables... 🤔

I will further investigate. I think you might have found a major bug! 😨 👍 😅

[stnolting]

There is a bug in the CPU interrupt system that crashes memory accesses. I have fixed that in version 1.5.8.4. I have tested the freeRTOS blinky example with a permanent external interrupt - now freeRTOS does not stall anymore 🎉

Thanks for identifying this bug!! 👍