Memory problem

[howeWang97]

Description

Hi Team, I encountered a problem when sending a large message (130M). During the sending process, the memory usage rose to 400M and was not reclaimed after the method ended. Even after I manually triggered a garbage collection, the memory did not drop back to within 20M. When receiving the message, the memory suddenly spiked to 1G. After I manually triggered a garbage collection, it dropped back to 900M.

Reproduction Steps

1.Send a big message（170M）：
byte[] byteArray = null;
using (FileStream fs = new FileStream(filePath, FileMode.Open, FileAccess.Read))
{
byteArray = new byte[fs.Length];
fs.Read(byteArray, 0, byteArray.Length);
}
var messageIdT = await producer.NewMessage().Send(byteArray);
2.The memory will increase by 390M after Send()
3.Receive message
await foreach (var message in consumer.Messages())
{
await consumer.Acknowledge(message);
}
4.When this message is received, the memory usage will start to rise, reaching one gigabyte, and after completing this cycle, the memory is not reclaimed.

Expected behavior

Want memory to be reclaimed

Actual behavior

I observed that when DotPulsar sends large messages, it takes some time. During this period, the memory usage starts to rise, and even after the sending is completed, the memory is not reclaimed. When receiving messages, the memory usage also rises sharply. And because it's in a foreach loop, I'm not sure if I can manually reclaim the memory.

Regression?

No response

Known Workarounds

No response

Configuration

1. .net Framework 4.6.2
2. DotPulsar 3.6.0
3. Windows 10
4. Any CPU

Other information

I'm sorry. I don't know

[blankensteiner]

Hi @howeWang97
What version of Pulsar are you using (broker side) and have you configured the broker to accept messages larger than the default 5 mb, since you are able to send so large messages?

[howeWang97]

Hi @howeWang97 What version of Pulsar are you using (broker side) and have you configured the broker to accept messages larger than the default 5 mb, since you are able to send so large messages?
Hi, Sir
I'm using Pulsar version 4.0.0. I have already configured the maxMessageSize and dispatcherMaxReadSizeBytes in broker.conf, and set nettyMaxFrameSizeBytes in bookkeeper.conf to 500M.

[blankensteiner]

This might come down to poor LOH handling in .NET Framework 4.6.2. Can you try using .NET 9 and see if the memory is reclaimed over time?

[howeWang97]

This might come down to poor LOH handling in .NET Framework 4.6.2. Can you try using .NET 9 and see if the memory is reclaimed over time?

Sir, I'm upgrading my project to. Net Framework 4.8 Memory still not reclaimed. As much as I'd like to upgrade the project to. Net 9 but my manager wouldn't allow the overall project to be upgraded to. Net Core. I will try to use it myself. Net 9 See if you have memory problems

[howeWang97]

Hi Sir @blankensteiner , I have tried using.NET 9 and memory has not been reclaimed. What should I do if I want to reclaim the message variable in "for each (v a r message in consumer . Messages()){}" before the end of a loop? Does the message variable have a method similar to Dispose? Is it possible for me to free up memory occupied by message if I use Marshal