Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

CVPN-1818 Inside Packet Codec #161

Open
wants to merge 8 commits into
base: main
Choose a base branch
from
Open

CVPN-1818 Inside Packet Codec #161

wants to merge 8 commits into from

Conversation

kp-max-li
Copy link
Contributor

@kp-max-li kp-max-li commented Feb 25, 2025

Description

Added inside packet codec interface for Lightway UDP:

  • Lightway
    • Interface to include an optional inside packet codec
    • inside packets can be accumulated, and sent at once as encoded packets (vice versa).
    • Support for toggling the codec at runtime without disrupting the connection
      • Simultaneously processing encoded and un-encoded packets by introducing new inside data frame types EncodedData and EncodedDataFrag
      • New inside wire frame type ToggleEncoding that allows the client to enable encoding on both sides
  • Lightway Client's CLI
    • -e flag in CLI args to automatically send a ToggleEncoding packet to the server once lightway-core changes state to Connected
    • mpsc channel for toggling encoding at runtime (TODO: an appropriate interface to send toggling signals via CLI)

Motivation and Context

FEC code can be integrated to inside packets. Throughput and latency can be reduced under high packet loss network conditions.

How Has This Been Tested?

  1. E2E test with network namespace bash scripts.
    • iperf3 and ping 8.8.8.8

Integration with C servers

With the network namespace scripts, running the latest version of lightway C UDP server: Verified that sending wire frames with the new ids does not break the connection with the server.

Types of changes

  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to change)

Checklist:

  • My change requires a change to the documentation.
  • I have updated the documentation accordingly.
  • The correct base branch is being used, if not main

Copy link

github-actions bot commented Feb 25, 2025

Code coverage summary for 69482af:

Filename                                                     Regions    Missed Regions     Cover   Functions  Missed Functions  Executed       Lines      Missed Lines     Cover    Branches   Missed Branches     Cover
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
lightway-app-utils/src/args/cipher.rs                              4                 4     0.00%           1                 1     0.00%           5                 5     0.00%           0                 0         -
lightway-app-utils/src/args/connection_type.rs                     4                 4     0.00%           1                 1     0.00%           5                 5     0.00%           0                 0         -
lightway-app-utils/src/args/duration.rs                            7                 7     0.00%           5                 5     0.00%          13                13     0.00%           0                 0         -
lightway-app-utils/src/args/ip_map.rs                             11                11     0.00%           3                 3     0.00%          15                15     0.00%           0                 0         -
lightway-app-utils/src/args/logging.rs                            14                14     0.00%           2                 2     0.00%          16                16     0.00%           0                 0         -
lightway-app-utils/src/connection_ticker.rs                       39                 4    89.74%          17                 2    88.24%         156                 5    96.79%           0                 0         -
lightway-app-utils/src/dplpmtud_timer.rs                          63                 7    88.89%          22                 4    81.82%         282                11    96.10%           0                 0         -
lightway-app-utils/src/event_stream.rs                             5                 0   100.00%           3                 0   100.00%          13                 0   100.00%           0                 0         -
lightway-app-utils/src/iouring.rs                                157               142     9.55%          23                17    26.09%         304               281     7.57%           0                 0         -
lightway-app-utils/src/metrics.rs                                  2                 2     0.00%           2                 2     0.00%           4                 4     0.00%           0                 0         -
lightway-app-utils/src/sockopt/ip_mtu_discover.rs                 23                23     0.00%           4                 4     0.00%          93                93     0.00%           0                 0         -
lightway-app-utils/src/sockopt/ip_pktinfo.rs                       4                 4     0.00%           1                 1     0.00%          16                16     0.00%           0                 0         -
lightway-app-utils/src/tun.rs                                     76                76     0.00%          22                22     0.00%          94                94     0.00%           0                 0         -
lightway-app-utils/src/utils.rs                                   13                13     0.00%           1                 1     0.00%          11                11     0.00%           0                 0         -
lightway-client/src/args.rs                                       34                34     0.00%          30                30     0.00%          37                37     0.00%           0                 0         -
lightway-client/src/io/inside/tun.rs                              39                39     0.00%           7                 7     0.00%          48                48     0.00%           0                 0         -
lightway-client/src/io/outside/tcp.rs                             48                48     0.00%          10                10     0.00%          45                45     0.00%           0                 0         -
lightway-client/src/io/outside/udp.rs                             62                62     0.00%          12                12     0.00%          62                62     0.00%           0                 0         -
lightway-client/src/keepalive.rs                                 245                33    86.53%          56                 8    85.71%         372                26    93.01%           0                 0         -
lightway-client/src/lib.rs                                       265               265     0.00%          26                26     0.00%         396               396     0.00%           0                 0         -
lightway-client/src/main.rs                                       48                48     0.00%           7                 7     0.00%         181               181     0.00%           0                 0         -
lightway-core/src/borrowed_bytesmut.rs                            86                 1    98.84%          24                 0   100.00%         196                 1    99.49%           0                 0         -
lightway-core/src/builder_predicates.rs                           20                10    50.00%           4                 2    50.00%          28                14    50.00%           0                 0         -
lightway-core/src/cipher.rs                                        8                 0   100.00%           3                 0   100.00%          13                 0   100.00%           0                 0         -
lightway-core/src/connection.rs                                  629               367    41.65%          61                30    50.82%         832               423    49.16%           0                 0         -
lightway-core/src/connection/builders.rs                          67                25    62.69%          22                11    50.00%         260                56    78.46%           0                 0         -
lightway-core/src/connection/dplpmtud.rs                         761                90    88.17%          66                 1    98.48%         947                11    98.84%           0                 0         -
lightway-core/src/connection/fragment_map.rs                     132                11    91.67%          27                 0   100.00%         290                 7    97.59%           0                 0         -
lightway-core/src/connection/io_adapter.rs                       137                17    87.59%          36                 5    86.11%         313                23    92.65%           0                 0         -
lightway-core/src/connection/key_update.rs                        23                 7    69.57%           5                 0   100.00%          38                19    50.00%           0                 0         -
lightway-core/src/context.rs                                      91                27    70.33%          26                 9    65.38%         213                52    75.59%           0                 0         -
lightway-core/src/context/ip_pool.rs                               7                 3    57.14%           1                 0   100.00%           6                 0   100.00%           0                 0         -
lightway-core/src/context/server_auth.rs                          14                11    21.43%           4                 3    25.00%          24                20    16.67%           0                 0         -
lightway-core/src/io.rs                                            3                 3     0.00%           3                 3     0.00%          10                10     0.00%           0                 0         -
lightway-core/src/lib.rs                                           9                 6    33.33%           4                 1    75.00%          18                 9    50.00%           0                 0         -
lightway-core/src/metrics.rs                                      11                11     0.00%           9                 9     0.00%          21                21     0.00%           0                 0         -
lightway-core/src/packet.rs                                       27                 7    74.07%           4                 1    75.00%          30                 6    80.00%           0                 0         -
lightway-core/src/plugin.rs                                       66                 8    87.88%          23                 5    78.26%         160                 9    94.38%           0                 0         -
lightway-core/src/utils.rs                                       101                21    79.21%          22                 2    90.91%         184                17    90.76%           0                 0         -
lightway-core/src/version.rs                                      37                 0   100.00%          19                 0   100.00%          89                 0   100.00%           0                 0         -
lightway-core/src/wire.rs                                        152                28    81.58%          32                 0   100.00%         265                 8    96.98%           0                 0         -
lightway-core/src/wire/auth_failure.rs                             9                 1    88.89%           3                 0   100.00%          19                 0   100.00%           0                 0         -
lightway-core/src/wire/auth_request.rs                           147                12    91.84%          34                 0   100.00%         287                 0   100.00%           0                 0         -
lightway-core/src/wire/auth_success_with_config_ipv4.rs           69                 4    94.20%          13                 0   100.00%         145                 0   100.00%           0                 0         -
lightway-core/src/wire/data.rs                                    20                 1    95.00%           7                 0   100.00%          43                 0   100.00%           0                 0         -
lightway-core/src/wire/data_frag.rs                               41                 1    97.56%          17                 0   100.00%          92                 0   100.00%           0                 0         -
lightway-core/src/wire/ping.rs                                    22                 3    86.36%           7                 0   100.00%          68                 0   100.00%           0                 0         -
lightway-core/src/wire/pong.rs                                    15                 2    86.67%           5                 0   100.00%          34                 0   100.00%           0                 0         -
lightway-core/src/wire/server_config.rs                           20                 2    90.00%           5                 0   100.00%          44                 0   100.00%           0                 0         -
lightway-core/src/wire/toggle_encoding.rs                         22                 2    90.91%           6                 0   100.00%          48                 1    97.92%           0                 0         -
lightway-server/src/args.rs                                       29                29     0.00%          29                29     0.00%          29                29     0.00%           0                 0         -
lightway-server/src/auth.rs                                      104                36    65.38%          19                 5    73.68%         148                28    81.08%           0                 0         -
lightway-server/src/connection.rs                                 58                58     0.00%          27                27     0.00%         125               125     0.00%           0                 0         -
lightway-server/src/connection_manager.rs                         87                87     0.00%          25                25     0.00%         226               226     0.00%           0                 0         -
lightway-server/src/connection_manager/connection_map.rs          87                 9    89.66%          26                 2    92.31%         273                 9    96.70%           0                 0         -
lightway-server/src/io/inside/tun.rs                              25                25     0.00%           8                 8     0.00%          34                34     0.00%           0                 0         -
lightway-server/src/io/outside/tcp.rs                             80                80     0.00%           8                 8     0.00%          99                99     0.00%           0                 0         -
lightway-server/src/io/outside/udp.rs                            144               144     0.00%          18                18     0.00%         263               263     0.00%           0                 0         -
lightway-server/src/io/outside/udp/cmsg.rs                        47                16    65.96%          14                 5    64.29%         237                56    76.37%           0                 0         -
lightway-server/src/ip_manager.rs                                 82                25    69.51%          19                 4    78.95%         254                28    88.98%           0                 0         -
lightway-server/src/ip_manager/ip_pool.rs                        101                 1    99.01%          32                 0   100.00%         305                 0   100.00%           0                 0         -
lightway-server/src/lib.rs                                       132               132     0.00%          17                17     0.00%         213               213     0.00%           0                 0         -
lightway-server/src/main.rs                                       65                65     0.00%           9                 9     0.00%         181               181     0.00%           0                 0         -
lightway-server/src/metrics.rs                                    91                89     2.20%          80                78     2.50%         216               212     1.85%           0                 0         -
lightway-server/src/statistics.rs                                 53                23    56.60%           8                 4    50.00%         103                42    59.22%           0                 0         -
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
TOTAL                                                           5094              2340    54.06%        1086               486    55.25%        9591              3616    62.30%           0                 0         -

✅ Region coverage 54% passes
✅ Line coverage 62% passes

@kp-max-li kp-max-li force-pushed the cvpn-1818 branch 16 times, most recently from 958b759 to 3de9228 Compare March 4, 2025 10:22
@kp-max-li kp-max-li force-pushed the cvpn-1818 branch 6 times, most recently from aaf7e86 to eb7088e Compare March 5, 2025 09:35
// Ignore invalid inside packet
}
Err(err) => {
// Propagate fatal error up
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This will drop all the remaining packets in the current vector.
We will need a metric to monitor this usage

@kp-max-li kp-max-li force-pushed the cvpn-1818 branch 2 times, most recently from 27a2532 to 8231bb6 Compare March 10, 2025 09:33
@kp-max-li kp-max-li force-pushed the cvpn-1818 branch 5 times, most recently from 6e84723 to e499d14 Compare March 11, 2025 04:03
@kp-max-li kp-max-li force-pushed the cvpn-1818 branch 2 times, most recently from 7d183f5 to f5bc40a Compare March 11, 2025 07:01
@kp-max-li kp-max-li changed the title [DO NOT MERGE] CVPN-1818 Packet Accumulator POC [DO NOT MERGE] CVPN-1818 Packet Codec POC Mar 11, 2025
@kp-max-li kp-max-li force-pushed the cvpn-1818 branch 3 times, most recently from b169417 to 2fe9714 Compare March 14, 2025 05:57
@kp-max-li kp-max-li changed the title [DO NOT MERGE] CVPN-1818 Packet Codec POC CVPN-1818 Inside Packet Codec and Raptor Mar 14, 2025
@kp-max-li kp-max-li changed the title CVPN-1818 Inside Packet Codec and Raptor CVPN-1818 Inside Packet Codec and RaptorQ FEC Mar 14, 2025
@kp-max-li kp-max-li changed the title CVPN-1818 Inside Packet Codec and RaptorQ FEC CVPN-1818 Inside Packet Codec Mar 14, 2025
Interface for encoding and decoding inside packets in lightway-core.

Inside packets can be accumulated and encoded, which allows  error
correction code to be integrated to lightway-core.
These new types of wire frame works like Data and DataFrag.
The sole purpose is to determines whether the Data payload is
encoded or not.
Connection optionally holds an encoder and an decoder for inside packets.

Encoder can either accumulates the inside packets and encode it before it is sent to WolfSSL,
or skip the packet and Connection will directly pass it to WoflSSL.

Decoder accumulates all encoded packets and decode it before it is sent to TUN.
Packet Codec is supplied by lw-server to lw-core.

The packets in the encoder might have sit in the encoder for too long if
inside packets are not frequently received. To avoid having a high
packet delay,  lw-server triggers a flush when the encoder is ready.

There could be stale states in the decoder. The decoder's stale state is
cleaned periodically.

lw-server tracks the list of encoders and decoders for each connection.
Instead of accesing the codec via locking the connection, codec is access directly instead.
In this way, the number of locking to Connection is minimized.
Packet Codec is supplied by lw-client to lw-core.

The packets in the encoder might have sit in the encoder for too long if inside packets are not frequently received. To avoid having a high packet delay, lw-server triggers a flush when the encoder is ready.

There could be stale states in the decoder. The decoder's stale state is cleaned periodically.
By default, inside packet encoding is disabled.
The client now can toggle encoding in both directions in the following work-flow:
1. Client sends a ToggleEncoding packet to the server
2. Server toggles encoding, and reply with the same packet
3. Client receives the packet and toggles encoding
TODO: under heavy packet loss, the above packets could be lost.
We may need to add a re-transmission mechanism to make this
more robust.
Client now has two ways to toggle encoding:
1. -e flag in CLI: automatically sends a toggle encoding packet to the server
   when lightway-core's state changes to Connected.
2. send a signal to the toggle_encode_tx in main.rs to toggle encoding at runtime.
   The client may enable/disable in runtime without disrupting the connection.

TODO: create an appropriate interface to send a toggle encoding signal via CLI?
When flushing the encoder, more than one packets are sent to WolfSSL.
If any of the send operation returns a fatal error, all subsequent inside
packets are dropped.
Added a metric to track the number of such packets.
@kp-max-li kp-max-li marked this pull request as ready for review March 14, 2025 09:29
@kp-max-li kp-max-li requested a review from a team as a code owner March 14, 2025 09:29
///
/// Returns a [`CodecStatus`].
/// If the status is [`CodecStatus::ReadyToFlush`], encoded packets are ready to be retrieved.
/// If the status is [`CodecStatus::SkipPacket`], the packet should not be added to the encoder.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does not this mean `the packet is skipped by the encoder and lightway can send it as normal Wire::Data packet.

fn cleanup_stale_states(&mut self);
}

/// Indicates whether the [`PacketEncoder`] or [`PacketDecoder`] is ready to be flushed or not
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Indicates the status of [PacketEncoder] or [PacketDecoder] after storing the current packet

@@ -137,6 +138,9 @@ async fn main() -> Result<()> {
key_update_interval: config.key_update_interval.into(),
inside_plugins: Default::default(),
outside_plugins: Default::default(),
inside_pkt_codec: None,
pkt_encoder_flush_interval: Duration::from_secs_f64(0.0001),
pkt_decoder_clean_up_interval: Duration::from_secs_f64(0.5),
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we make this values as cli args ?

} else {
metrics::tun_rejected_packet_no_connection();
}
}
});

tokio::spawn(pkt_encoder_flush(
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This two tasks should be created only if config.inside_pkt_codec is not None, should not this ?

@@ -97,6 +99,9 @@ async fn main() -> Result<()> {
server: config.server,
inside_plugins: Default::default(),
outside_plugins: Default::default(),
inside_pkt_codec: None,
pkt_encoder_flush_interval: Duration::from_secs_f64(0.0001),
pkt_decoder_clean_up_interval: Duration::from_secs_f64(0.5),
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same as server, better this be an cli config

// encoder is not set. Awaits forever...
// Not returning immediately to avoid triggering an exit of the
// entire client program.
tracing::debug!("Pkt encoder flush task is freezed");
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

FYI, tokio::select supports enabling the branch only on valid condition (valid packet codec)
You might use it to disable this task check instead of this.


/// Signal for toggling inside packet encoding
#[educe(Debug(ignore))]
pub toggle_encoding_signal: tokio::sync::mpsc::Receiver<bool>,
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We should not force apps to provide this value. Better to make this as option similar to inside_pkt_codec

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Or should we create a new PacketEncoderConfig and add all these variables inside it and make it as an option.
It reduces the burden for apps to skip this, if codec is not needed

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants