Content location enhancement

[Xevrac]

Is your feature request related to a problem? Please describe.
A source server might be an ideal location for a server, but not content fetching. A network might not be designed to handle download operations however is suitable for network traffic on a server. Using a CDN or S3 bucket would be more efficient in most cases for users requesting modded content on servers.

Describe the solution you'd like
BeamMP should look at allowing content to be stored in a remote storage location. This could be using S3, B2 or other CDN storage technologies. By default, the server looks for the content in its own directory unless otherwise specified.

Describe alternatives you've considered
Storing mods with the server on a larger instance is always the obvious practicality; as is a larger network pipe. But for others without this ability, having remote storage as an option would boost the player's experience as the download times would be significantly faster.

[carsakiller]

Fetching mods from a CDN would be very nice!

[Starystars67]

Fetching mods from a CDN would be very nice!

This is something we have discussed in various capacities many times over the years.

Unfortunately there are a number of issues that arise with this.
The first and foremost comes with ownership and accountability.

Lets say we enabled the ability for you to list mods from the BeamNG repo, this would put an increase on the load of that service for what is essentially a third party purpose and I cannot imagine BeamNG being best pleased with that.

So, if we look at other options we would have the following:

1. BeamMP create their own mod repo of sorts for content to be hosted and served from. The issues here are as follows

• Cost (Both financially to run and in time to build & support such a system).
• Accountability (Who is responsible to verify that the content being uploaded is safe and legitimate).
• Ownership (Every uploaded item would need to be checked and at least some attempt to verify that the person uploading actually has the rights to do so).

Finally the last option would be to give BeamMP the ability to download from multiple sources which could introduce any number untold of issues.

Just some food for thought ultimately.

[Xevrac]

Fetching mods from a CDN would be very nice!

This is something we have discussed in various capacities many times over the years.

Unfortunately there are a number of issues that arise with this. The first and foremost comes with ownership and accountability.

Lets say we enabled the ability for you to list mods from the BeamNG repo, this would put an increase on the load of that service for what is essentially a third party purpose and I cannot imagine BeamNG being best pleased with that.

So, if we look at other options we would have the following:

1. BeamMP create their own mod repo of sorts for content to be hosted and served from. The issues here are as follows


* Cost (Both financially to run and in time to build & support such a system).

* Accountability (Who is responsible to verify that the content being uploaded is safe and legitimate).

* Ownership (Every uploaded item would need to be checked and at least some attempt to verify that the person uploading actually has the rights to do so).

Finally the last option would be to give BeamMP the ability to download from multiple sources which could introduce any number untold of issues.

Just some food for thought ultimately.

@Starystars67, I agree with you, but I am not suggesting we use 1st party repos or even BeamMP guys hosting your own.

Right now, BeamMP server operators use their own decentralised servers to host the content anyway, but directly on their instance.

There's no difference in context I can see here if people were given the ability to use an s3 protocol to get the content from their own s3 bucket.

I wasn't suggesting a centralised CDN.

But a private one.

I want to host my servers and the players fetch their content from MY S3 bucket.

If players open-source their own S3 bucket, that's on their own cost and responsibility.

[Starystars67]

I agree with you, but I am not suggesting we use 1st party repos or even you guys hosting your own.

Right now, BeamMP server operators use their own decentralised servers to host the content anyway, but directly on their instance.

There's no difference in context I can see here if people were given the ability to use an s3 protocol to get the content from their own s3 bucket.

I wasn't suggesting a centralised CDN.

But a private one.

I want to host my servers and the players fetch their content from MY S3 bucket.

If players open-source their own S3 bucket, that's on their own cost and responsibility.

I see, in that case the only new issue I see which does kinda play into what I was saying above is that most likely kids will copy the download addresses of content so they can put them on their own server or people will start creating free links that people can use where the content being served at the end could literally be anything at all.

For the moment with the setup that we have it is required that all content being served is at least zipped up for it to be served in the first place. I am not saying that this is a catch all for security or anything but it does reduce the potential risks a little bit.

[Xevrac]

I agree with you, but I am not suggesting we use 1st party repos or even you guys hosting your own.
Right now, BeamMP server operators use their own decentralised servers to host the content anyway, but directly on their instance.
There's no difference in context I can see here if people were given the ability to use an s3 protocol to get the content from their own s3 bucket.
I wasn't suggesting a centralised CDN.
But a private one.
I want to host my servers and the players fetch their content from MY S3 bucket.
If players open-source their own S3 bucket, that's on their own cost and responsibility.

I see, in that case the only new issue I see which does kinda play into what I was saying above is that most likely kids will copy the download addresses of content so they can put them on their own server or people will start creating free links that people can use where the content being served at the end could literally be anything at all.

For the moment with the setup that we have it is required that all content being served is at least zipped up for it to be served in the first place. I am not saying that this is a catch all for security or anything but it does reduce the potential risks a little bit.

I disagree, if the s3 URI is in the server configuration and the method is protected with middleware or something then it can't be sniffed. S3 also supports encryption and credentials.

I do not see any security risks if the solution was developed appropriately.

https://docs.aws.amazon.com/AmazonS3/latest/userguide/security-best-practices.html

[Starystars67]

So just to confirm, are you thinking that the BeamMP server would get the mod from the S3 bucket and then serve it to the client?

[Xevrac]

Something like that, bypassing the load on the network the actual BeamMP server instance is hosted on. The BeamMP clientside might need some kind of implementation to stream that data directly, it would remove the middleman if possible- I've not explored that in its entirety so I can't tell you how complex or difficult it might be

[Starystars67]

Sorry, if it is to bypass the network load on the server then that would mean it is redirecting the user to the S3 bucket no?

Which would mean they would know where it is residing?

I might be missing something though, it is 9am :D

[Xevrac]

I see what you're saying, think of it in two parts.

Server-side:
Handles the S3 definition, settings, URI, token etc.

{
  "modStorage": {
    "type": "s3",
    "bucketUri": "https://example-bucket.s3.amazonaws.com/",
    "accessKey": "your-access-key",
    "secretKey": "your-secret-key"
  }
}

Client-side:
Some kind of datastream (I'm not sure what to call this, not experienced in this specific method) that fetches this server-side information securely, then the client downloads the files from the Uri fetched from the server upon joining because the server has told it where to look.

Content validation would also be needed on top of the S3 best practices (HTTPS, ACLs, Tokens etc.)

I understand that this adds some client-side complexity, but it would speed up the user experience, in addition to the other issue I raised (network packet prioritization or content download shaping) would really help improve the user experience.

[carsakiller]

So, let's explain the goal some more. The goal of this would be to remove the network load of content delivery from the BeamMP Server, also allowing content to be served from an optimized server, geographically closer to the user, potentially for cheaper.

What this won't provide is protection of assets/content. Prevention of hotlinking is important, but making it so only the intended user can retrieve the asset is difficult and likely not worth it. At some point, we will need to redirect the user to the URL where the asset is, and that URL can be shared (more on this below). At the end of the day, any user can just join the server and share the ZIPs after download. With that said, this feature can still be useful for centralized content delivery.

---

Talking specifically about Amazon S3 (but it should also be possible with other CDN solutions), it is possible to generate pre-signed URLs for a private bucket for the user, which could expire after a limited time (say 10 seconds). The user's client can then download directly from the S3 bucket.

Here is a very rough example:

This should help prevent sharing the URL to the asset as the user would first need to sniff out the URL, which while using HTTPS would be difficult, and even then the URL expires very quickly. If done successfully, then someone else could potentially tell their BeamMP server to use your BeamMP server as their CDN using the links from step 2. To help prevent this, there are many steps to take, perhaps:

• Reject when the User-Agent header matches the BeamMP Server's (spoofable, easy, likely adequate for most users)
• Generate a requestID in step 1 and ensure it matches in step 3
• Rate limit step 1 to prevent generating many requestIDs for hotlinking

This should help prevent hotlinking while taking almost all the load for content delivery off of the BeamMP server.

---

As for providing the URLs to the S3 bucket; the BeamMP server could handle the communication with AWS, taking in just the object IDs, or it could be exposed to plugins to leave it up to the server owner to provide the link to the mod (removing the need for it to be AWS). Something like this maybe:

local MODS_BUCKET = "my-mods-bucket"

-- Generate a presigned URL for an S3 bucket object using AWS CLI (must be installed locally on server and available in PATH)
local function generate_presigned_url(object_key, expires_in)
    local cmd = string.format(
        'aws s3 presign s3://%s/%s --expires-in %d',
        MODS_BUCKET,
        object_key,
        expires_in or 3600
    )
    local handle = io.popen(cmd)
    local result = handle:read("*a")
    handle:close()
    return result
end

function onRequestMod(beammp_id, ip, modID)
    return generate_presigned_url(modID, 10)
end

MP.RegisterEvent("onRequestMod", "onRequestMod")

[Xevrac]

So, let's explain the goal some more. The goal of this would be to remove the network load of content delivery from the BeamMP Server, also allowing content to be served from an optimized server, geographically closer to the user, potentially for cheaper.

What this won't provide is protection of assets/content. Prevention of hotlinking is important, but making it so only the intended user can retrieve the asset is difficult and likely not worth it. At some point, we will need to redirect the user to the URL where the asset is, and that URL can be shared (more on this below). At the end of the day, any user can just join the server and share the ZIPs after download. With that said, this feature can still be useful for centralized content delivery.

Talking specifically about Amazon S3 (but it should also be possible with other CDN solutions), it is possible to generate pre-signed URLs for a private bucket for the user, which could expire after a limited time (say 10 seconds). The user's client can then download directly from the S3 bucket.

Here is a very rough example:

This should help prevent sharing the URL to the asset as the user would first need to sniff out the URL, which while using HTTPS would be difficult, and even then the URL expires very quickly. If done successfully, then someone else could potentially tell their BeamMP server to use your BeamMP server as their CDN using the links from step 2. To help prevent this, there are many steps to take, perhaps:

* Reject when the `User-Agent` header matches the BeamMP Server's (spoofable, easy, likely adequate for most users)

* Generate a `requestID` in step 1 and ensure it matches in step 3

* Rate limit step 1 to prevent generating many `requestID`s for hotlinking

This should help prevent hotlinking while taking almost all the load for content delivery off of the BeamMP server.

As for providing the URLs to the S3 bucket; the BeamMP server could handle the communication with AWS, taking in just the object IDs, or it could be exposed to plugins to leave it up to the server owner to provide the link to the mod (removing the need for it to be AWS). Something like this maybe:

local MODS_BUCKET = "my-mods-bucket"

-- Generate a presigned URL for an S3 bucket object using AWS CLI (must be installed locally on server and available in PATH)
local function generate_presigned_url(object_key, expires_in)
    local cmd = string.format(
        'aws s3 presign s3://%s/%s --expires-in %d',
        MODS_BUCKET,
        object_key,
        expires_in or 3600
    )
    local handle = io.popen(cmd)
    local result = handle:read("*a")
    handle:close()
    return result
end

function onRequestMod(beammp_id, ip, modID)
    return generate_presigned_url(modID, 10)
end

MP.RegisterEvent("onRequestMod", "onRequestMod")

Awesome. I can't see any flaws logically with this detail, I like the idea of a mod bucket array. If we can remove a S3 or AWS requirement? Let's say it could be any URL due to it just using GET -> 302; that would be extremely flexible for other server administrators.