Issue with JetStream DLQ Functionality and Message Removal

[Atanusaha143]

Description:
I am implementing Dead Letter Queue (DLQ) functionality with JetStream and encountered an issue regarding message persistence in the stream after maximum retry attempts. Below is the setup and the observed behavior.

Streams Report

+-------------------------------------------------------------------------------------------+
|                                       Stream Report                                       |
+----------------------+---------+-----------+----------+-------+------+---------+----------+
| Stream               | Storage | Consumers | Messages | Bytes | Lost | Deleted | Replicas |
+----------------------+---------+-----------+----------+-------+------+---------+----------+
| test_pull_dlq_stream | File    | 0         | 0        | 0 B   | 0    | 0       |          |
| test_pull_stream     | File    | 1         | 0        | 0 B   | 0    | 0       |          |
| test_push_stream     | File    | 1         | 0        | 0 B   | 0    | 0       |          |
+----------------------+---------+-----------+----------+-------+------+---------+----------+

Stream Configuration:

# test_pull_stream configuration:

Information for Stream test_pull_stream created 2024-11-06T12:53:37Z

Configuration:

             Subjects: test_pull_subject
     Acknowledgements: true
            Retention: File - WorkQueue
             Replicas: 1
       Discard Policy: Old
     Duplicate Window: 2m0s
     Maximum Messages: unlimited
        Maximum Bytes: unlimited
          Maximum Age: 0.00s
 Maximum Message Size: unlimited
    Maximum Consumers: unlimited


State:

             Messages: 0
                Bytes: 0 B
             FirstSeq: 22
              LastSeq: 21 @ 2024-11-07T10:59:49 UTC
     Active Consumers: 1

Consumer Configuration:

# test_pull_con configuration:

Information for Consumer test_pull_stream > test_pull_con created 2024-11-06T12:54:14Z

Configuration:

        Durable Name: test_pull_con
           Pull Mode: true
      Filter Subject: test_pull_subject
         Deliver All: true
          Ack Policy: Explicit
            Ack Wait: 30s
       Replay Policy: Instant
  Maximum Deliveries: 5
     Max Ack Pending: 1,000

State:

   Last Delivered Message: Consumer sequence: 69 Stream sequence: 21
     Acknowledgment floor: Consumer sequence: 69 Stream sequence: 21
         Outstanding Acks: 0 out of maximum 1000
     Redelivered Messages: 0
     Unprocessed Messages: 0

Now, let's imagine a scenario where I try to publish a message by executing the following line:

# Publisher:

await test_publisher.publish(stream="test_pull_stream", subject="test_pull_subject", message={"message": message})

And I have the below subscriber to listen on:

# Subscriber:

@broker.subscriber(subject="test_pull_subject",
                   stream=test_pull_stream,
                   durable="test_pull_con",
                   pull_sub=PullSub(),
                   no_ack=True,
                   retry=True)
async def handle_pull_test(data: dict, msg: NatsMessage):
    try:
        div_result = 5 / 0   # Throws ZeroDivisionError
        await msg.ack()
    except Exception as e:
        await msg.nack()
        print(msg)

When a ZeroDivisionError occurs, the consumer test_pull_con retries processing the message up to Maximum Deliveries: 5 times. Since Acknowledgements are enabled in the stream configuration, the message remains in the stream until acknowledged, and a copy of the message is sent to test_pull_dlq_stream. Implemented with "Dead Letter Queues" type functionality

Observed Behavior:
The message continues to exist in the stream after the retries are exhausted. Since Maximum Deliveries have been reached, the message is no longer processed but still remains in the test_pull_stream. And as intended test_pull_dlq_stream gets a copy of it.

Question:
How can I remove the message from the test_pull_stream once Maximum Deliveries is exhausted, as it has no further retries left and I already have it on test_pull_dlq_stream without any manual action?

NOTE
It is mentioned on "Dead Letter Queues" type functionality that If a message reaches its maximum level of delivery attempts, it will still stay in the stream until it is manually deleted or manually acknowledged.