Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add netpacket interface (request for comments) #15413

Merged
merged 11 commits into from
Jun 29, 2023
105 changes: 105 additions & 0 deletions libretro-common/include/libretro.h
Original file line number Diff line number Diff line change
Expand Up @@ -1809,6 +1809,28 @@ enum retro_mod
* even before the microphone driver is ready.
*/

#define RETRO_ENVIRONMENT_SET_NETPACKET_INTERFACE 76
/* const struct retro_netpacket_callback * --
* When set, a core gets control over network packets sent and
* received during a multiplayer session. This can be used to emulate
* multiplayer games that were originally played on 2 or more separate
* consoles or computers connected together.
*
* The frontend will take care of connecting players together.
* The core only needs to send the actual data as needed for the
* emulation while handshake and connection management happens in
* the background.
*
* When 2 or more players are connected and this interface has been
* set, time manipulation features (pausing, slow motion, fast forward,
* rewinding, save state loading, etc.) are disabled to not interrupt
* communication.
*
* When not set, a frontend may use state serialization based
* multiplayer where a deterministic core supporting multiple
* input devices does not need to do anything on its own.
*/

/* VFS functionality */

/* File paths:
Expand Down Expand Up @@ -3030,6 +3052,89 @@ struct retro_disk_control_ext_callback
retro_get_image_label_t get_image_label; /* Optional - may be NULL */
};

/* Callbacks for RETRO_ENVIRONMENT_SET_NETPACKET_INTERFACE.
* A core can set it if sending and receiving custom network packets
* during a multiplayer session is desired.
*/

/* Used by the core to send a packet to one or more connected players.
* A single packet sent via this interface can contain up to 64kb of data.
*
* If the ready callback has indicated the local player to be the host:
* - The broadcast flag can be set to true to send to multiple connected clients
* - On a broadcast, the client_id argument indicates 1 client NOT to send the packet to
* - Otherwise, the client_id argument indicates a single client to send the packet to
* If the local player is a client connected to a host:
* - The broadcast flag is ignored
* - The client_id argument must be set to 0
*
* This function is not guaranteed to be thread-safe and must be called during
* retro_run or any of the netpacket callbacks passed with this interface.
*/
typedef void (RETRO_CALLCONV *retro_netpacket_send_t)(const void* buf, size_t len, uint16_t client_id, bool broadcast);
Copy link
Contributor

@JesseTG JesseTG Jun 20, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I can think of a use case for all clients being able to talk to each other (rather than just the host). This thread on Discord has the details, but in a nutshell:

  • Handheld consoles like the Nintendo DS or Game Boy support local multiplayer via link cables or custom wireless protocols.
  • These protocols usually have extremely tight latency requirements that can't be met over the Internet.
  • The best general solution (currently adopted by at least one Game Boy core) is for all clients to emulate all consoles, and synchronize the results via rollback.
  • Not to be confused with true Internet connectivity, like the Wi-fi services offered by the Wii and DS. The original hardware had to account for Wi-fi latency, so Wi-fi can be emulated without rollback tricks.

Given this, a practical use case for P2P communication would be to send savestates of all consoles to all other participants.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, I think it would be feasible to allow clients talking to clients via this directly.
In RetroArch netplay this always will have to relayed via the host (i.e. CLIENTA -> HOST -> CLIENTB) but perhaps in another frontend it could be a different kind of network setup.
Currently in the implementation I did in my DOS core I have such a packet relaying done inside the core, which is aware if it acts as the host and it does the packet relaying there. But we could move that into the frontend and change the API to allow a client talking to another client. I'll see if that can be done.

As for using this in a tight latency requirements, an easy first try might be to do lockstep without rollback. This new interface has this specified:

* When 2 or more players are connected and this interface has been
* set, time manipulation features (pausing, slow motion, fast forward,
* rewinding, save state loading, etc.) are disabled to not interrupt
* communication.

So when this is set up and cores can talk to each other, the frontend will not mess with timing. Thus a core could take over and only advance emulation when the data over the emulated local wi-fi or link cable has arrived.
Once that works, rollback could be implemented on top of that. Don't lockstep anymore but roll back when wifi/cable data from the past arrives.

As for sending whole savestates over this, the limitation of 64kb of data per packet might be annoying. Though splitting it up into multiple packets wouldn't be too hard. The 64kb limit I chose pretty arbitrarily, we could even remove it and have the frontend realloc the buffer it uses for this once a core wants to send a bigger packet at once.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The new commit c0f85f8 now adds the option for clients to send packets directly to other clients.

This means all peers are equal in regards to what retro_netpacket_send_t can do. Anyone can send a packet to any other peer as well as send broadcasts to everyone connected.

Because in RetroArch netplay clients don't connect to each other (they only connect to the host) it is implemented by having the host relay packets as needed. If the core on client A sends a packet addressed to client B, the netplay host will relay that packet but it won't call retro_netpacket_receive_t locally. Another frontend could have a mesh connection where clients can directly talk to each other, resulting in the core experiencing the same behavior.


/* Called by the frontend to signify that a multiplayer session has started.
* If client_id is 0 the local player is the host of the session and at this
* point no other player has connected yet.
*
* If client_id is > 0 the local player is a client connected to a host and
* at this point is already fully connected to the host.
*
* The core will have to store the retro_netpacket_send_t function pointer
* passed here and use it whenever it wants to send a packet. That send
* function pointer is valid until the frontend calls retro_netpacket_stop_t.
*/
typedef void (RETRO_CALLCONV *retro_netpacket_start_t)(uint16_t client_id, retro_netpacket_send_t send_fn);

/* Called by the frontend when a new packet arrives which has been sent from
* a connected client or the host with retro_netpacket_send_t.
* The client_id argument indicates who has sent the packet. On the host side
* this will always be > 0 (coming from a connected client).
* On a client connected to the host it is always 0 (coming from the host).
* Packets sent with this interface arrive at this callback in a reliable
* manner, meaning in the same order they were sent and without packet loss.
*/
typedef void (RETRO_CALLCONV *retro_netpacket_receive_t)(const void* buf, size_t len, uint16_t client_id);
Copy link
Contributor

@JesseTG JesseTG Jun 20, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Shouldn't you specify requirements for the transport layer somewhere? Cores don't need to care, but frontends will so they can connect to other frontends. "Reliable" could mean TCP, WebSockets, something custom on top of UDP...

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I agree, ideally retro_netpacket_send_t would have something like a enum retro_netpacket_flag flag argument which could be set to various modes of sending (i.e. reliable, unreliable, unsequenced). But at least in the implementation I can contribute it will be limited to what RetroArch netplay currently can do, which is only reliable as it goes over TCP.

Maybe we add the flag for now for future expansion but only have reliable as an option? Or nothing and have reliable be the default.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe we add the flag for now for future expansion but only have reliable as an option?

I like this idea, but I still think you should explicitly document how the protocol (at the application level) works. You might say "TCP", and that's fine, but what can other frontends expect in the payload? Just the buffer? Some preceding metadata?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As the interface stands now it is all transparent to the core. The core sends 10 bytes with retro_netpacket_send_t, the other side will receive that 10 bytes with retro_netpacket_receive_t. By design the core shouldn't care what communication was used to send the packet. It could be pigeon post :-)
This is what I meant with this part of the spec:

* The frontend will take care of connecting players together.
* The core only needs to send the actual data as needed for the
* emulation while handshake and connection management happens in
* the background.

A frontend other than RetroArch could use websockets to transmit the data, or basing it on a library like ENET. In the end it doesn't matter if it's TCP or UDP or something else. This is not meant to connect frontends together. If another frontend wants to support netplay with RetroArch, then it needs to follow the specs of netplay in RetroArch - which are not part of libretro.h.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In the new commit c0f85f8 I added a flags argument to retro_netpacket_send_t with the following options:

/* Netpacket flags for retro_netpacket_send_t */
#define RETRO_NETPACKET_UNRELIABLE  0        /* Packet to be sent unreliable, depending on network quality it might not arrive. */
#define RETRO_NETPACKET_RELIABLE    (1 << 0) /* Reliable packets are guaranteed to arrive at the target in the order they were send. */
#define RETRO_NETPACKET_UNSEQUENCED (1 << 1) /* Packet will not be sequenced with other packets and may arrive out of order. Cannot be set on reliable packets. */

Also I extended the comment for retro_netpacket_send_t with this:

 * A frontend must support sending of reliable packets (RETRO_NETPACKET_RELIABLE).
 * Unreliable packets might not be supported by the frontend but the flags can
 * still be specified, reliable transmission will be used instead.

Because NetPlay in RetroArch can only send packets reliable. Other frontends may use communication over something like UDP which would enable the other reliability modes.


/* Called by the frontend when the multiplayer session has ended.
* Once this gets called the retro_netpacket_send_t function pointer passed
* to retro_netpacket_start_t will not be valid anymore.
*/
typedef void (RETRO_CALLCONV *retro_netpacket_stop_t)(void);

/* Called by the frontend every frame (between calls to retro_run while
* updating the state of the multiplayer session.
* This is a good place for the core to call retro_netpacket_send_t from.
*/
typedef void (RETRO_CALLCONV *retro_netpacket_poll_t)(void);

/* Called by the frontend when a new player connects to the hosted session.
* This is only called on the host side, not for clients connected to the host.
*/
typedef void (RETRO_CALLCONV *retro_netpacket_connected_t)(uint16_t client_id);

/* Called by the frontend when a player leaves or disconnects from the hosted session.
* This is only called on the host side, not for clients connected to the host.
*/
typedef void (RETRO_CALLCONV *retro_netpacket_disconnected_t)(uint16_t client_id);

/**
* A callback interface for giving a core the ability to send and receive custom
* network packets during a multiplayer session between two or more instances
* of a libretro frontend.
*
* @see RETRO_ENVIRONMENT_SET_NETPACKET_INTERFACE
*/
struct retro_netpacket_callback
{
retro_netpacket_start_t start;
retro_netpacket_receive_t receive;
retro_netpacket_stop_t stop; /* Optional - may be NULL */
retro_netpacket_poll_t poll; /* Optional - may be NULL */
retro_netpacket_connected_t connected; /* Optional - may be NULL */
retro_netpacket_disconnected_t disconnected; /* Optional - may be NULL */
};

enum retro_pixel_format
{
/* 0RGB1555, native endian.
Expand Down
1 change: 1 addition & 0 deletions network/netplay/netplay.h
Original file line number Diff line number Diff line change
Expand Up @@ -147,6 +147,7 @@ typedef struct
struct netplay_room host_room;
struct netplay_room *room_list;
struct netplay_rooms *rooms_data;
struct retro_netpacket_callback *core_netpacket_interface;
/* Used while Netplay is running */
netplay_t *data;
netplay_client_info_t *client_info;
Expand Down
5 changes: 4 additions & 1 deletion network/netplay/netplay_defines.h
Original file line number Diff line number Diff line change
Expand Up @@ -74,7 +74,10 @@ enum rarch_netplay_ctl_state
RARCH_NETPLAY_CTL_DESYNC_PUSH,
RARCH_NETPLAY_CTL_DESYNC_POP,
RARCH_NETPLAY_CTL_KICK_CLIENT,
RARCH_NETPLAY_CTL_BAN_CLIENT
RARCH_NETPLAY_CTL_BAN_CLIENT,
RARCH_NETPLAY_CTL_SET_CORE_PACKET_INTERFACE,
RARCH_NETPLAY_CTL_SKIP_NETPLAY_CALLBACKS,
RARCH_NETPLAY_CTL_ALLOW_TIMESKIP
};

/* The current status of a connection */
Expand Down
Loading