Suggestion: Use maximum Gzip compression level (level 9) for uploading jsons to the server.

[touhidurrr]

Before creating

• This is NOT a gameplay feature from Civ VI, BNW, or outside - see Roadmap
• This is NOT a gameplay feature from Vanilla Civ V or from G&K - If so, it should be a comment in Missing features from Civ V - G&K #4697

Problem Description

The assumption is CPU time required for compression / decompression is much lower than network time required to fetch the files larger files and the tradeoff is generally worth it. Using maximum compression level will most likely not change much. However, I suggest the idea of using this setting. This decrease badwidth and storage usage somewhat.

Related Issue Links

No response

Desired Solution

use Gzip compression level 9

Alternative Approaches

use something in the middle

Additional Context

This mostly increases CPU time on client side. As server only needs to decompress these files on PUT requests. Any bandwidth decrease is appreciated.

[touhidurrr]

And so, I did some tests with Bun.gzipSync() which has 2 library options zlib and libdeflate for some reason. Here are the results:

Input File: test.json (421.27KB)

zlib (default 6)
zlib 0 [1.59ms] (421.35KB)
zlib 1 [2.23ms] (63.58KB) {0.00% 1.00x}
zlib 2 [2.86ms] (60.48KB) {-4.88% 1.28x}
zlib 3 [3.40ms] (56.22KB) {-11.57% 1.52x}
zlib 4 [4.20ms] (51.75KB) {-18.60% 1.88x}
zlib 5 [4.65ms] (46.69KB) {-26.56% 2.09x}
zlib 6 [6.25ms] (43KB) {-32.36% 2.80x}
zlib 7 [9.40ms] (42.38KB) {-33.35% 4.21x}
zlib 8 [13.24ms] (41.74KB) {-34.34% 5.93x}
zlib 9 [12.43ms] (41.74KB) {-34.35% 5.57x}

libdeflate (default unknown)
libdeflate 0 [0.60ms] (421.32KB)
libdeflate 1 [1.92ms] (52.87KB) {0.00% 1.00x}
libdeflate 2 [1.93ms] (52.77KB) {-0.18% 1.01x}      
libdeflate 3 [2.57ms] (51.1KB) {-3.35% 1.34x}       
libdeflate 4 [2.54ms] (46.89KB) {-11.30% 1.32x}     
libdeflate 5 [2.73ms] (45.22KB) {-14.46% 1.42x}     
libdeflate 6 [3.41ms] (42.37KB) {-19.87% 1.78x}
libdeflate 7 [4.73ms] (39.35KB) {-25.57% 2.46x}     
libdeflate 8 [9.03ms] (38.85KB) {-26.52% 4.71x}
libdeflate 9 [12.97ms] (38.71KB) {-26.79% 6.76x}    
libdeflate 10 [50.03ms] (38.1KB) {-27.93% 26.07x}
libdeflate 11 [105.08ms] (37.02KB) {-29.98% 54.75x}
libdeflate 12 [153.17ms] (37KB) {-30.01% 79.81x}

[touhidurrr]

So, we are talking about ~50.28% increase in time and ~2% size decrease here huh (zlib 6 vs 9)
Considering libdeflate level 10 to 12 time increase are insane libdeflate 9 gives ~7.26% size decrease vs zlib 9.
huh.

[touhidurrr]

And so I did Kotlin perf tests also. Thanks to Unciv, StackOverflow and ChatGPT.

Benchmark Code

package touhidurrr.gzipleveltest

import java.io.BufferedReader
import java.io.ByteArrayInputStream
import java.io.ByteArrayOutputStream
import java.io.InputStreamReader
import java.net.URI
import java.util.zip.Deflater
import java.util.zip.GZIPInputStream
import java.util.zip.GZIPOutputStream
import kotlin.system.measureNanoTime

object Gzip {
    fun compress(data: String, level: Int = Deflater.DEFAULT_COMPRESSION): ByteArray {
        val bos = ByteArrayOutputStream(data.length)
        val gzip = object : GZIPOutputStream(bos) {
            init {
                def.setLevel(level)
            }
        }
        gzip.write(data.toByteArray())
        gzip.close()
        val compressed = bos.toByteArray()
        bos.close()
        return compressed
    }

    fun decompress(compressed: ByteArray): String {
        val bis = ByteArrayInputStream(compressed)
        val gis = GZIPInputStream(bis)
        val br = BufferedReader(InputStreamReader(gis, Charsets.UTF_8))
        val sb = StringBuilder()
        var line: String? = br.readLine()
        while (line != null) {
            sb.append(line)
            line = br.readLine()
        }
        br.close()
        gis.close()
        bis.close()
        return sb.toString()
    }
}

fun fetchGameJSON(gameId: String): String {
    val url = URI("https://uncivserver.xyz/jsons/$gameId").toURL()
    return url.readText(Charsets.UTF_8)
}

fun main() {
    var levelOneSz = 0f
    val iterations = 1_000
    val testData = fetchGameJSON("b78948eb-452f-42ea-8425-93dab002bcdc")

    println("Iterations: $iterations")
    println("Level\tCompress\tDecompress\tRatio\t\tTime vs Level 1")
    for (level in -1..9) {
        // warm up
        var compressed = Gzip.compress(testData, level)
        var decompressed = Gzip.decompress(compressed)

        // Measure compression time
        val compressTimeMs = measureNanoTime {
            repeat(iterations) {
                compressed = Gzip.compress(testData, level)
            }
        } / 1_000_000F / iterations

        // Measure decompression time
        val decompressTimeMs = measureNanoTime {
            repeat(iterations) {
                decompressed = Gzip.decompress(compressed)
            }
        } / 1_000_000F / iterations

        if (level == 1) levelOneSz = compressed.size.toFloat()

        println(
            "%2d\t\t%.2fms\t\t%.2fms\t\t%+.2f%%\t\t${
                if (level > 0) "%+.2f%%%%".format((100 * (compressed.size - levelOneSz)) / levelOneSz) else ""
            }"
                .format(
                    level,
                    compressTimeMs,
                    decompressTimeMs,
                    100 * (compressed.size.toFloat() - testData.length) / testData.length
                )
        )
    }
}

Results:

Iterations: 1000
Level	Compress	Decompress	Ratio		Time vs Level 1
-1	6.97ms		1.31ms		-90.03%		
 0	0.59ms		0.89ms		+0.02%		
 1	2.36ms		1.43ms		-85.23%		+0.00%
 2	2.49ms		1.39ms		-86.09%		-5.83%
 3	2.83ms		1.34ms		-87.24%		-13.57%
 4	4.02ms		1.34ms		-87.62%		-16.18%
 5	5.10ms		1.29ms		-88.90%		-24.83%
 6	6.96ms		1.24ms		-90.03%		-32.51%
 7	9.04ms		1.27ms		-90.28%		-34.17%
 8	16.10ms		1.24ms		-90.52%		-35.78%
 9	17.09ms		1.24ms		-90.52%		-35.80%

[touhidurrr]

So, Level 6 vs Level 9 is ~3.23% size decrease for ~2.56x time increase, huh. And decompression is negligible. (And actually decrease with compression level)

UPDATE December 23, 2024

Updated tests with 1,000 iterations ~3.23% size decrease for ~146% time increase between Level 6 & 9. Not much difference.

So,

1. Loading files takes a little less time.
2. Uploading files takes a little hit. (Since ~10ms is still less compared to other times like serialization and network)

Verdict

Not sure. I will open a pr, add it in your discretion.

[yairm210]

Wow, yeah then my verdict is definitely to NOT do this