We could not find what you were looking for.
Please contact the owner of the site that linked you to the original URL and let them know their link is broken.
![](\\"/img/football_match_compression.jpg\\")
\\n\\nOptimizing data transfer over WebSocket connections can significantly reduce bandwidth costs. Compressing data usually leads to memory and CPU resource usage overhead \u2013 but in many cases it worth doing anyway since it positively impacts the final bill from the provider (bandwidth cost reduction overweights resource usage increase).\\n\\nCentrifugo v5.4.0 introduced [delta compression](/docs/server/delta_compression) feature. But before implementing it we wanted a playground which could demonstrate the potential benefit of using delta compression in Centrifugo channels.\\n\\nThis post outlines our approach to estimating the potential profit from implementing delta compression. It demonstrates the reduction in data transfer using once concrete use case across various configurations, including different Centrifugo protocol formats and the additional use of WebSocket permessage-deflate compression. Although these numbers can vary significantly depending on the data, we believe the results are valuable for providing a general understanding of Centrifugo compression options. This information can help Centrifugo users apply these insights to their use cases.\\n\\n## About delta compression\\n\\n\\n\\nFor a good overview of delta compression topic for the real-time messaging applications I suggest starting with a [blog post in Ably engineeiring blog](https://ably.com/blog/message-delta-compression).\\n\\nCentrifugo is very similar to Ably in many aspects (though self-hosted), so everything said in the linked post equally applies to Centrifugo use cases too. Though we have differences in the final implementation, one notable is that we are using [Fossil](https://fossil-scm.org/home/doc/tip/www/delta_format.wiki) delta algorithm in Centrifugo instead of VCDIFF. The reason over VCDIFF was mainly two factors:\\n\\n* availability of several Fossil delta implementations, specifically there are good libraries for Go (see [shadowspore/fossil-delta](https://github.com/shadowspore/fossil-delta)), and for Javascript - [fossil-delta-js](https://github.com/dchest/fossil-delta-js).\\n* the compactness of the algorithm implementation \u2013 under 500 lines of code in JavaScript\\n\\nThe compactness property is nice because there are no OSS Fossil implementations for Java, Dart and Swift \u2013 languages we have SDKs for \u2013 so we may have to implement this algorithm in the future ourselves.\\n\\nHaving said this all, let\'s proceed to the description of experiment we did to understand possible benefits of various compression techniques, and delta compression in particular. \\n\\n## Experiment Overview\\n\\nIn the experiment, we simulated a football match, sending the entire game state over a WebSocket connection upon every match event. Our compression playground looks like this:\\n\\n\\n\\nIt visualizes only the score, but under the hood there are other game changes happen \u2013 will be shown below.\\n\\nWe tested various configurations to evaluate the effectiveness of data compression if different cases. In each setup the same game data was sent over the wire. The data then was captured using [WireShark](https://www.wireshark.org/) with the filter:\\n\\n```\\ntcp.srcport == 8000 && websocket\\n```\\n\\nThis is how WebSocket packets look in Wireshark when applying a filter mentioned above:\\n\\n\\n\\nBytes captured show the entire overhead from packets in the game channel going from server to client (including TCP/IP overhead).\\n\\nThe source code of the experiment may be found on Github as a [Centrifuge library example](https://github.com/centrifugal/centrifuge/tree/master/_examples/compression_playground). You can run it to inspect the exact WebSocket frames in each scenario.\\n\\nTo give reader a general idea about data, we sent 30 publications with the entire football game state, for example here is a first message in a match (2 teams, 11 players):\\n\\n\\n\\n```json\\n{\\n \\"homeTeam\\":{\\n \\"name\\":\\"Real Madrid\\",\\n \\"players\\":[\\n {\\n \\"name\\":\\"John Doe\\"\\n },\\n {\\n \\"name\\":\\"Jane Smith\\"\\n },\\n {\\n \\"name\\":\\"Alex Johnson\\"\\n },\\n {\\n \\"name\\":\\"Chris Lee\\"\\n },\\n {\\n \\"name\\":\\"Pat Kim\\"\\n },\\n {\\n \\"name\\":\\"Sam Morgan\\"\\n },\\n {\\n \\"name\\":\\"Jamie Brown\\"\\n },\\n {\\n \\"name\\":\\"Casey Davis\\"\\n },\\n {\\n \\"name\\":\\"Morgan Garcia\\"\\n },\\n {\\n \\"name\\":\\"Taylor White\\"\\n },\\n {\\n \\"name\\":\\"Jordan Martinez\\"\\n }\\n ]\\n },\\n \\"awayTeam\\":{\\n \\"name\\":\\"Barcelona\\",\\n \\"players\\":[\\n {\\n \\"name\\":\\"Robin Wilson\\"\\n },\\n {\\n \\"name\\":\\"Drew Taylor\\",\\n \\"events\\":[\\n {\\n \\"type\\":\\"RED_CARD\\"\\n }\\n ]\\n },\\n {\\n \\"name\\":\\"Jessie Bailey\\"\\n },\\n {\\n \\"name\\":\\"Casey Flores\\"\\n },\\n {\\n \\"name\\":\\"Jordan Walker\\"\\n },\\n {\\n \\"name\\":\\"Charlie Green\\"\\n },\\n {\\n \\"name\\":\\"Alex Adams\\"\\n },\\n {\\n \\"name\\":\\"Morgan Thompson\\"\\n },\\n {\\n \\"name\\":\\"Taylor Clark\\"\\n },\\n {\\n \\"name\\":\\"Jordan Hernandez\\"\\n },\\n {\\n \\"name\\":\\"Jamie Lewis\\"\\n }\\n ]\\n }\\n}\\n```\\n\\n
\\n\\n\\n```json\\n{\\n \\"homeTeam\\":{\\n \\"name\\":\\"Real Madrid\\",\\n \\"score\\":3,\\n \\"players\\":[\\n {\\n \\"name\\":\\"John Doe\\",\\n \\"events\\":[\\n {\\n \\"type\\":\\"YELLOW_CARD\\",\\n \\"minute\\":6\\n },\\n {\\n \\"type\\":\\"SUBSTITUTE\\",\\n \\"minute\\":39\\n }\\n ]\\n },\\n {\\n \\"name\\":\\"Jane Smith\\"\\n },\\n {\\n \\"name\\":\\"Alex Johnson\\"\\n },\\n {\\n \\"name\\":\\"Chris Lee\\",\\n \\"events\\":[\\n {\\n \\"type\\":\\"GOAL\\",\\n \\"minute\\":84\\n }\\n ]\\n },\\n {\\n \\"name\\":\\"Pat Kim\\"\\n },\\n {\\n \\"name\\":\\"Sam Morgan\\"\\n },\\n {\\n \\"name\\":\\"Jamie Brown\\",\\n \\"events\\":[\\n {\\n \\"type\\":\\"SUBSTITUTE\\",\\n \\"minute\\":9\\n }\\n ]\\n },\\n {\\n \\"name\\":\\"Casey Davis\\",\\n \\"events\\":[\\n {\\n \\"type\\":\\"YELLOW_CARD\\",\\n \\"minute\\":81\\n }\\n ]\\n },\\n {\\n \\"name\\":\\"Morgan Garcia\\",\\n \\"events\\":[\\n {\\n \\"type\\":\\"SUBSTITUTE\\",\\n \\"minute\\":15\\n },\\n {\\n \\"type\\":\\"GOAL\\",\\n \\"minute\\":30\\n },\\n {\\n \\"type\\":\\"YELLOW_CARD\\",\\n \\"minute\\":57\\n },\\n {\\n \\"type\\":\\"GOAL\\",\\n \\"minute\\":62\\n },\\n {\\n \\"type\\":\\"RED_CARD\\",\\n \\"minute\\":66\\n }\\n ]\\n },\\n {\\n \\"name\\":\\"Taylor White\\",\\n \\"events\\":[\\n {\\n \\"type\\":\\"YELLOW_CARD\\",\\n \\"minute\\":18\\n },\\n {\\n \\"type\\":\\"SUBSTITUTE\\",\\n \\"minute\\":42\\n },\\n {\\n \\"type\\":\\"SUBSTITUTE\\",\\n \\"minute\\":45\\n },\\n {\\n \\"type\\":\\"YELLOW_CARD\\",\\n \\"minute\\":69\\n },\\n {\\n \\"type\\":\\"RED_CARD\\",\\n \\"minute\\":72\\n }\\n ]\\n },\\n {\\n \\"name\\":\\"Jordan Martinez\\",\\n \\"events\\":[\\n {\\n \\"type\\":\\"SUBSTITUTE\\",\\n \\"minute\\":21\\n },\\n {\\n \\"type\\":\\"SUBSTITUTE\\",\\n \\"minute\\":24\\n }\\n ]\\n }\\n ]\\n },\\n \\"awayTeam\\":{\\n \\"name\\":\\"Barcelona\\",\\n \\"score\\":3,\\n \\"players\\":[\\n {\\n \\"name\\":\\"Robin Wilson\\"\\n },\\n {\\n \\"name\\":\\"Drew Taylor\\",\\n \\"events\\":[\\n {\\n \\"type\\":\\"RED_CARD\\"\\n },\\n {\\n \\"type\\":\\"GOAL\\",\\n \\"minute\\":12\\n }\\n ]\\n },\\n {\\n \\"name\\":\\"Jessie Bailey\\"\\n },\\n {\\n \\"name\\":\\"Casey Flores\\",\\n \\"events\\":[\\n {\\n \\"type\\":\\"YELLOW_CARD\\",\\n \\"minute\\":78\\n }\\n ]\\n },\\n {\\n \\"name\\":\\"Jordan Walker\\",\\n \\"events\\":[\\n {\\n \\"type\\":\\"SUBSTITUTE\\",\\n \\"minute\\":33\\n }\\n ]\\n },\\n {\\n \\"name\\":\\"Charlie Green\\",\\n \\"events\\":[\\n {\\n \\"type\\":\\"GOAL\\",\\n \\"minute\\":51\\n },\\n {\\n \\"type\\":\\"GOAL\\",\\n \\"minute\\":60\\n },\\n {\\n \\"type\\":\\"SUBSTITUTE\\",\\n \\"minute\\":75\\n }\\n ]\\n },\\n {\\n \\"name\\":\\"Alex Adams\\"\\n },\\n {\\n \\"name\\":\\"Morgan Thompson\\",\\n \\"events\\":[\\n {\\n \\"type\\":\\"YELLOW_CARD\\",\\n \\"minute\\":27\\n },\\n {\\n \\"type\\":\\"SUBSTITUTE\\",\\n \\"minute\\":48\\n }\\n ]\\n },\\n {\\n \\"name\\":\\"Taylor Clark\\",\\n \\"events\\":[\\n {\\n \\"type\\":\\"SUBSTITUTE\\",\\n \\"minute\\":3\\n },\\n {\\n \\"type\\":\\"SUBSTITUTE\\",\\n \\"minute\\":87\\n }\\n ]\\n },\\n {\\n \\"name\\":\\"Jordan Hernandez\\"\\n },\\n {\\n \\"name\\":\\"Jamie Lewis\\",\\n \\"events\\":[\\n {\\n \\"type\\":\\"YELLOW_CARD\\",\\n \\"minute\\":36\\n },\\n {\\n \\"type\\":\\"SUBSTITUTE\\",\\n \\"minute\\":54\\n }\\n ]\\n }\\n ]\\n }\\n}\\n```\\n\\n
\\n\\n\\n```protobuf\\nsyntax = \\"proto3\\";\\n\\npackage centrifugal.centrifuge.examples.compression_playground;\\n\\noption go_package = \\"./;apppb\\";\\n\\nenum EventType {\\n UNKNOWN = 0; // Default value, should not be used\\n GOAL = 1;\\n YELLOW_CARD = 2;\\n RED_CARD = 3;\\n SUBSTITUTE = 4;\\n}\\n\\nmessage Event {\\n EventType type = 1;\\n int32 minute = 2;\\n}\\n\\nmessage Player {\\n string name = 1;\\n repeated Event events = 2;\\n}\\n\\nmessage Team {\\n string name = 1;\\n int32 score = 2;\\n repeated Player players = 3;\\n}\\n\\nmessage Match {\\n int32 id = 1;\\n Team home_team = 2;\\n Team away_team = 3;\\n}\\n```\\n\\n
\\n![](\\"/img/centrifugo_loki.jpg\\")
\\n\\nAs of version 5.1.0, Centrifugo introduces an experimental yet powerful extension that promises to simplify the data delivery process to the browser using GRPC streams. We believe it may help you to solve some practical tasks in minutes. Let\'s dive into how this feature works and how you can leverage it in your applications integrating with [Loki](https://grafana.com/oss/loki/) real-time log streaming capabilities.\\n\\n\x3c!--truncate--\x3e\\n\\n## What Are Proxy Subscription Streams?\\n\\n[Proxy Subscription Streams](/docs/server/proxy_streams) support pushing data directly to Centrifugo client channel subscriptions from your application backend over GRPC streams. This feature is designed to facilitate individual data streams to clients as soon as they subscribe to a channel, acting as a bridge between WebSocket connections from clients and GRPC streams to the backend. It supports both unidirectional (backend to client) and bidirectional (both ways) streams, thereby enhancing flexibility in data streaming.\\n\\n\\n\\nThe design is inspired by [Websocketd](http://websocketd.com/) server \u2013 but while Websocketd transforms data from programs running locally, Centrifugo provides a more generic network interface with GRPC. And all other features of Centrifugo like connection authentication, online presence come as a great bonus.\\n\\nIn the documentation for Proxy Subscription Streams we mentioned streaming logs from Loki as one of the possible use cases. Let\'s expand on the idea and implement the working solution in just 10 minutes.\\n\\n## Demo and source code\\n\\nHere is a demo of what we well get:\\n\\n\\n\\nTake a look at [full source code on Github](https://github.com/centrifugal/examples/tree/master/v5/subscription_streams_loki).\\n\\n## Setting Up Loki\\n\\n[Loki](https://grafana.com/oss/loki/) is a horizontally-scalable, highly-available, multi-tenant log aggregation system inspired by Prometheus. It is designed to be very cost-effective and easy to operate, making it a perfect candidate for our real-time log streaming example.\\n\\nWe will build the example using Docker Compose, all we have to do for the example is to include Loki image to `docker-compose.yml`: \\n\\n```yaml\\nservices:\\n loki:\\n image: grafana/loki:2.9.5\\n ports:\\n - \\"3100:3100\\"\\n```\\n\\nLoki can ingest logs via various methods, including Promtail, Grafana Agent, Fluentd, and more. For simplicity, we will send logs to Loki ourselves from the Go application.\\n\\nTo send logs to Loki, we can use the HTTP API that Loki provides. This is a straightforward way to push logs directly from an application. The example below demonstrates how to create a simple Go application that generates logs and sends them to Loki using HTTP POST requests.\\n\\nFor this post we will be using Go language to implement the backend part. But it could be any other programming language.\\n\\nFirst, let\'s some code to send a log entries to Loki:\\n\\n```go\\nconst (\\n\\tlokiPushEndpoint = \\"http://loki:3100/loki/api/v1/push\\"\\n)\\n\\ntype lokiPushMessage struct {\\n\\tStreams []lokiStream `json:\\"streams\\"`\\n}\\n\\ntype lokiStream struct {\\n\\tStream map[string]string `json:\\"stream\\"`\\n\\tValues [][]string `json:\\"values\\"`\\n}\\n\\nfunc sendLogMessageToLoki(_ context.Context) error {\\n\\tsources := []string{\\"backend1\\", \\"backend2\\", \\"backend3\\"}\\n\\tsource := sources[rand.Intn(len(sources))]\\n\\tlogMessage := fmt.Sprintf(\\"log from %s source\\", source)\\n\\n\\tpayload := lokiPushMessage{\\n\\t\\tStreams: []lokiStream{\\n\\t\\t\\t{\\n\\t\\t\\t\\tStream: map[string]string{\\n\\t\\t\\t\\t\\t\\"source\\": source,\\n\\t\\t\\t\\t},\\n\\t\\t\\t\\tValues: [][]string{\\n\\t\\t\\t\\t\\t{fmt.Sprintf(\\"%d\\", time.Now().UnixNano()), logMessage},\\n\\t\\t\\t\\t},\\n\\t\\t\\t},\\n\\t\\t},\\n\\t}\\n\\n\\tjsonData, err := json.Marshal(payload)\\n\\tif err != nil {\\n\\t\\treturn err\\n\\t}\\n\\tresp, err := http.Post(\\n\\t\\tlokiPushEndpoint, \\"application/json\\", bytes.NewBuffer(jsonData))\\n\\tif err != nil {\\n\\t\\treturn err\\n\\t}\\n\\tdefer resp.Body.Close()\\n\\n\\tif resp.StatusCode != http.StatusNoContent {\\n\\t\\treturn fmt.Errorf(\\"unexpected status code: %d\\", resp.StatusCode)\\n\\t}\\n\\treturn nil\\n}\\n\\nfunc sendLogsToLoki(ctx context.Context) {\\n\\tfor {\\n\\t\\tselect {\\n\\t\\tcase <-ctx.Done():\\n\\t\\t\\treturn\\n\\t\\tcase <-time.After(200 * time.Millisecond):\\n\\t\\t\\terr := sendLogMessageToLoki(ctx)\\n\\t\\t\\tif err != nil {\\n\\t\\t\\t\\tlog.Println(\\"error sending log to Loki:\\", err)\\n\\t\\t\\t\\tcontinue\\n\\t\\t\\t}\\n\\t\\t}\\n\\t}\\n}\\n\\nfunc main() {\\n\\tctx, cancel := signal.NotifyContext(context.Background(), syscall.SIGTERM, syscall.SIGINT)\\n\\tdefer cancel()\\n\\n\\tsendLogsToLoki(ctx)\\n}\\n```\\n\\nThis program defines a `sendLogsToLoki` function that constructs a log entry and sends it to Loki using its HTTP API. It continuously generates log messages every 200 milliseconds.\\n\\nThe `lokiPushMessage` struct is structured to match the JSON payload expected by Loki\'s [`/loki/api/v1/push`](https://grafana.com/docs/loki/latest/reference/api/#push-log-entries-to-loki) endpoint. Each log entry consists of a set of labels (in the Stream map) and log line values, where each value is a two-element array containing the timestamp and the log line. The timestamp is in nanoseconds to match Loki\'s expected format.\\n\\nNote, in the example we randomly set log entry `source` label choosing between `backend1`, `backend2` and `backend3` values.\\n\\nAt this point our program pushes some logs to Loki, now let\'s add Centrifugo to consume them from browser in real-time.\\n\\n## Configuring Centrifugo\\n\\nAdding Centrifugo is also rather straightforward:\\n\\n```yaml\\nservices:\\n centrifugo:\\n image: centrifugo/centrifugo:v5.3.0\\n restart: unless-stopped\\n volumes:\\n - ./centrifugo/config.json:/centrifugo/config.json\\n command: centrifugo -c config.json\\n expose:\\n - 8000\\n```\\n\\nWhere `config.json` is:\\n\\n```json\\n{\\n \\"client_insecure\\": true,\\n \\"allowed_origins\\": [\\"http://localhost:9000\\"],\\n \\"proxy_subscribe_stream_endpoint\\": \\"grpc://backend:12000\\",\\n \\"proxy_subscribe_stream_timeout\\": \\"3s\\",\\n \\"namespaces\\": [\\n {\\n \\"name\\": \\"logs\\",\\n \\"proxy_subscribe_stream\\": true\\n }\\n ]\\n}\\n```\\n\\nNote, we enabled `client_insecure` option here \u2013 this is to keep example short, but in real live you may benefit from Centrifugo authentication: [JWT-based](/docs/server/authentication) or [proxy-based](/docs/server/proxy#connect-proxy).\\n\\n## Writing frontend\\n\\n```html\\n\\n\\n\\n \\n 
diff --git a/blog/2020/10/16/experimenting-with-quic-transport.html b/blog/2020/10/16/experimenting-with-quic-transport.html
index a76c630aa..af65cefc2 100644
--- a/blog/2020/10/16/experimenting-with-quic-transport.html
+++ b/blog/2020/10/16/experimenting-with-quic-transport.html
@@ -15,8 +15,8 @@
-
-
+
+
In Centrifugo v5 we're phasing out old client protocol support, introducing a more intuitive HTTP API, adjusting token management behaviour in SDKs, improving configuration process, and refactoring the history meta ttl option. As the result you get a cleaner, more user-friendly, and optimized Centrifugo experience. And we have important news about the project - check it out in the end of this post.
diff --git a/blog/2023/08/19/asynchronous-message-streaming-to-centrifugo-with-benthos.html b/blog/2023/08/19/asynchronous-message-streaming-to-centrifugo-with-benthos.html index 119f36bf8..3d50f84ee 100644 --- a/blog/2023/08/19/asynchronous-message-streaming-to-centrifugo-with-benthos.html +++ b/blog/2023/08/19/asynchronous-message-streaming-to-centrifugo-with-benthos.html @@ -15,8 +15,8 @@ - - + +
diff --git a/blog/2023/08/29/using-centrifugo-in-rabbitx.html b/blog/2023/08/29/using-centrifugo-in-rabbitx.html
index 38361786a..40b8b8807 100644
--- a/blog/2023/08/29/using-centrifugo-in-rabbitx.html
+++ b/blog/2023/08/29/using-centrifugo-in-rabbitx.html
@@ -15,8 +15,8 @@
-
-
+
+
diff --git a/blog/2023/10/29/discovering-centrifugo-pro-push-notifications.html b/blog/2023/10/29/discovering-centrifugo-pro-push-notifications.html
index be69e86a4..d9584be92 100644
--- a/blog/2023/10/29/discovering-centrifugo-pro-push-notifications.html
+++ b/blog/2023/10/29/discovering-centrifugo-pro-push-notifications.html
@@ -15,8 +15,8 @@
-
-
+
+
diff --git a/blog/2024/03/18/stream-loki-logs-to-browser-with-websocket-to-grpc-subscriptions.html b/blog/2024/03/18/stream-loki-logs-to-browser-with-websocket-to-grpc-subscriptions.html
index dfbce181e..84ce0110a 100644
--- a/blog/2024/03/18/stream-loki-logs-to-browser-with-websocket-to-grpc-subscriptions.html
+++ b/blog/2024/03/18/stream-loki-logs-to-browser-with-websocket-to-grpc-subscriptions.html
@@ -15,8 +15,8 @@
-
-
+
+
diff --git a/blog/2024/05/30/real-time-data-compression-experiments.html b/blog/2024/05/30/real-time-data-compression-experiments.html
index 16bebf2ab..6f91c6d75 100644
--- a/blog/2024/05/30/real-time-data-compression-experiments.html
+++ b/blog/2024/05/30/real-time-data-compression-experiments.html
@@ -15,8 +15,8 @@
-
-
+
+
@@ -152,6 +152,6 @@ Using binary data in combination with the Centrifugo Protobuf protocol provides substantial bandwidth reduction even without deflate or delta compression. However, this comes at the cost of increased data format complexity. An additional benefit of using the Centrifugo Protobuf protocol is its faster marshalling and unmarshalling on the server side compared to the JSON protocol.
-For Centrifugo, these results highlighted the potential of implementing delta compression, so we proceeded with it. The benefit depends on the nature of the data being sent – you can achieve even greater savings if you have larger messages that are very similar to each other.
For Centrifugo, these results highlighted the potential of implementing delta compression, so we proceeded with it. The benefit depends on the nature of the data being sent – you can achieve even greater savings if you have larger messages that are very similar to each other.