XBF specs

docs/DynArray.hh

@@ -0,0 +1,51 @@
+#include <iostream>
+
+template<typename T> 
+class DynArray {
+private:
+    T* data;
+    uint32_t size;
+    uint32_t capacity;
+    void checkgrow() {
+        if (size == capacity) {
+            capacity *= 2;
+            T* old = data;
+            data = new T[capacity];
+            // TODO: this is horribly inefficient, shame on Codeium!
+            for (uint32_t i = 0; i < size; i++) {
+                newData[i] = data[i];
+            }
+            delete[] old;
+        }
+    }
+public:
+    DynArray() : data(nullptr), size(0), capacity(0) {}
+    ~DynArray() {
+        delete[] data;
+    }
+    DynArray(const DynArray<T>& orig) : data(nullptr), size(0), capacity(0) {
+        data = new T[orig.capacity];
+        for (uint32_t i = 0; i < orig.size; i++) {
+            data[i] = orig.data[i];
+        }
+        size = orig.size;
+        capacity = orig.capacity;
+    }
+    DynArray<T>& operator=(DynArray<T> copy) {
+        swap(data, copy.data);
+        size = copy.size;
+        capacity = copy.capacity;
+        return *this;
+    }
+
+    T& operator[](uint32_t index) {
+        return data[index];
+    }
+    void addEnd(const T& value) {
+        checkgrow();
+        data[size++] = value;
+    }
+    void send_additions(uint32_t index) {
+
+    }
+};

docs/XBF.md

@@ -0,0 +1,173 @@
+# XBF Definition
+
+XBF (eXtensible Binary Format) is a data standard based on the original concept
+of IDL, the data standard for CORBA, and protocols like Google protobuf.
+XBF supports key features not in CORBA, protobuf, or any other protocol.
+
+CORBA IDL primarily defined interfaces, method calls that could be invoked remotely.
+It also defined structures representing compound data types that could be sent between computers.
+
+Google protobuf supports definition of data and automatically generates custom code on either side to transmit the data.
+
+XBF extends the concepts in these two to include more data types, metadata so a client does not need to know what the data is to deal with it in certain contexts, automated processing on the client side,
+
+1. Support for more integer data sizes up to 256 bits.
+2. Formal specification of Date types.
+3. Bit vectors to support efficient transport of compact data without requiring custom encoding.
+4. Definition of Metadata to describe a stream so that the receiver does not have to know the structure of what is sent.
+   * It should be possible to display data on a web client without knowing what the data is.
+   * It should be possible to display data with an explanation in the local language without programming
+      * Internal field names in XBF are not merely for programmers because they can be used to display column names in a local language
+      * Translation dictionaries are supported to local languages, and should be automatically generated using AI, but reviewed by humans until AI is sufficiently good.
+5. In order to reduce repetition, XBF includes the notion of functions and subroutines that can be defined compactly. The language is restricted for safety
+      * All code in XBF is Turing analyzable. That is, halting is guaranteed and the cost of the code can be computed at load time.
+      * In order to achieve this, there are only constant iteration loops, and loops over a list. The number of iterations is always known.
+      *  Functions may call other functions, but recursion is forbidden.
+      * Mutual recursion is impossible because functions may only call previously defined functions.
+      * The resulting language is safe but can be used to reduce the complexity of complex data containing patterns.
+
+## Binary Tokens
+
+All tokens in XBF are 1-byte. The following list shows each token and its corresponding value.
+
+%table(multicol)
+U8,0
+U16
+U24
+U32
+U64
+U128
+U256
+STRING8
+STRING16
+STRING32
+LIST8
+LIST16
+LIST32
+MAP
+UPDATE
+%endtable
+
+| Tag | Val | Tag | Val | Tag | Val |
+| --- | --- | --- | --- | --- | --- |
+| U8 | 00 | I8 | 07 | DATE | 0E |
+|U16|01|I16|08|TIME|0F|
+|U24|02|I24|09|DATETIME|10|
+|U32|03|I32|0A|CALENDAR|11|
+|U64|04|I64|0B|STRING8|12|
+|U128|05|I128|0C|STRING16|13|
+|U256|06|I256|0D|STRING32|14|
+|  
+|LIST8|15|DEF|21|LOOP|28|
+|LIST16|16|STRUCT|22|FOREACH|29|
+|LIST32|17|FUNC|23|IF|2A|
+|MAP|18|LIBCALL|24|SET|2B|
+|SYNCLIST|19|CALL|25|VAR|2C|
+|SYNCMAP|20|LCALL|26|
+
+Data Management
+MSGSEND     send XBF to other
+MSGRECV     wait to receive XBF
+STORE       save to local storage
+UPLOAD      load changes from local storage back to server
+
+### Type Modifiers
+
+When declaring types, XBF allows optional modifiers that describe the packing and compression. Each field can be declared encoded in various ways.
+
+* DELTA (delta encoded)
+* DISCRETIZED16 (for floating point values)
+
+## Metadata Examples
+
+All metadata examples are described in Rust notation.
+
+```rust
+struct Point {
+  x: f64,
+  y: f64,
+  z: f64,
+}
+
+Vec<u32>
+
+Vec<Point>
+
+fn build_1(x: f32, y: f32, s: string) {
+      rect(x,y,100,100);
+      ellipse(x,y,100,100);
+      text(x,y,s);
+}
+
+fn build_2(x1: f32, x2: f32, y1: f32, y2: f32,
+      s1: string, s2: string, s3: string, s4: string) {
+      build_1(x1, y1, s1);
+      build_1(x2, y1, s2);
+      build_1(x1, y2, s3);
+      build_1(x2, y2, s4);
+}
+```
+
+```
+[DEF] 01 [STRUCT] 05 P o i n t 03 F64 1 x F64 1 y F64 1
+[LIST] 03 00 00 00
+point is defined as type=256, first user-defined type
+[LIST] 00 01 00 00
+
+[FUNC] 03 03 F32 F32 STRING
+  LIBCALL 01 00  [00] (x)
+  LIBCALL 02 00  [01] (y)
+  LIBCALL 03 00  [02] (s)
+
+[FUNC] 08 03 REPT 04 F32 REPT 04 STRING8
+  CALL 01 00 VAR 00 02 04     (call build_1(x1,y1,s1)
+  CALL 01 00 VAR 01 02 05     (call build_1(x2,y1,s2)
+  CALL 01 00 VAR 00 03 06     (call build_1(x1,y2,s3)
+  CALL 01 00 VAR 01 03 07     (call build_1(x2,y2,s4)
+
+server: List<int,"Fred"> [1, 2, 3, 4, 5, 6, 7, 8, 9, 10]
+client: List<int, "Fred"> [ 2, 3, 5, 6, 7]
+[SYNCLIST] 04 Fred [03] [01 00 00 00] [09 00 00 00] [0A 00 00 00]
+   [ADDSTART] [01] [ADDEND] [02]
+
+question: How to incorporate deletion?
+// for inserting and deleting start and end, O(n)
+fn update_list(new_size: u32,
+    delta: Vec<T>, add_start: u32, add_end: u32, delRanges : ??) {
+      if size + add_start + add_end - remove_start - remove_end > capacity {
+            old = m;
+            m = new T[newSize];
+      } else {
+            old = m;
+      }
+      if 
+      for i : u32 = 0; i < add_start, i++ {
+            m[i] = delta[i];
+      }
+      for i : u32 = del_start; i < size - del_end; i++ {
+            m[add_start - del_start + i] = orig[i];
+      }
+      for i : u32 = 0; i < size(delta); i++ {
+        m[add_start + (size - del_end) - del_start] = delta[i];
+      }
+}
+
+```
+1. There is a formal binary metadata language that is standard across
+all platforms. For any object defined in XDL, there is a sequence of
+bytes defining the metadata that tells any reader the sequence of
+types to be expected. Because of this feature, it is possible to write
+a generic client that can read data from any server. There are
+therefore two ways to send data in XDL. The first way is the same as
+IDL and Protobuf -- the programmer defines data in XDL, generates
+custom code for server and client, and each sends the data, knowing
+that the other side knows at compile time what to expect. The second
+is to generate data, send the metadata either before or every time the
+data is sent, and let the client interpret the metadata to discover
+
+1. XDL includes the concept of localization of names of
+variables. Data in XDL is designed to share and be viewed by the other
+party. It is not just a programmer internal detail. Accordingly, there
+is a formal specification of how to display data including the name of
+the variables, which can be translated into local languages.
+

docs/client.rs

@@ -0,0 +1,52 @@
+use quinn::{ClientConfigBuilder, Endpoint, TransportConfig, Certificate, PrivateKey};
+use std::net::SocketAddr;
+use std::error::Error;
+use tokio::net::UdpSocket;
+use std::io::Write;
+
+#[tokio::main]
+async fn main() -> Result<(), Box<dyn Error>> {
+    // Set up the address to connect to
+    let addr = "127.0.0.1:5000".parse().expect("Failed to parse address");
+
+    // Create QUIC endpoint
+    let mut endpoint_builder = Endpoint::builder();
+    endpoint_builder.default_client_config(make_client_config());
+
+    // Build the QUIC endpoint
+    let (endpoint, _) = endpoint_builder.bind(&"[::]:0".parse()?)?;
+
+    // Connect to the server
+    let quic_conn = endpoint.connect(&addr, "localhost").await?;
+
+    // Open a stream
+    let (send, mut recv) = quic_conn
+        .open_bi()
+        .await
+        .expect("Failed to open bidirectional stream");
+
+    // Send an integer to be echoed back
+    let int_value = 42;
+    let bytes = int_value.to_be_bytes();
+    send.write_all(&bytes).await.expect("Failed to write to stream");
+
+    // Read the echoed integer from the server
+    let mut buf = [0; 4];
+    recv.read_exact(&mut buf).await.expect("Failed to read from stream");
+    let echoed_value = i32::from_be_bytes(buf);
+
+    println!("Echoed value from server: {}", echoed_value);
+
+    Ok(())
+}
+
+fn make_client_config() -> quinn::ClientConfig {
+    let mut client_config = ClientConfigBuilder::default();
+
+    let certificate = quinn::pem::parse_x509_pem(include_bytes!("cert.pem")).unwrap().0;
+    let key = quinn::pem::parse_private_key_pem(include_bytes!("key.rsa")).unwrap();
+    client_config.add_certificate_authority(Certificate::from_der(&certificate).unwrap()).unwrap();
+    client_config.set_protocols(&[b"quic-client"[..]]);
+
+    client_config.build()
+}

docs/server.cpp

@@ -0,0 +1,12 @@
+int main(int argc, char* argv[]) {
+    constexpr int N = 1024;
+    DynArray a(N);
+    for (int i = 0; i < N/2; i++)
+      a.add_end(i);
+    a.send_additions(0); // send additions since start
+    for (int i = N/2; i < N; i++)
+      a.add_end(i);
+    a.send_additions(N/2); // send additions since N/2
+    a.add_end(99);
+    a.send_additions(N); // send single new element
+}

docs/server.rs

@@ -0,0 +1,56 @@
+use quinn::{Endpoint, Incoming, Connection, TransportConfig, Certificate, PrivateKey, ServerConfig, ServerConfigBuilder, EndpointDriver};
+use std::net::SocketAddr;
+use tokio::net::UdpSocket;
+
+#[tokio::main]
+async fn main() {
+    // Set up the address to listen on
+    let addr = "0.0.0.0:5000".parse().expect("Failed to parse address");
+
+    // Create QUIC endpoint
+    let mut builder = Endpoint::builder();
+    builder.listen(TransportConfig::default());
+    let certificate = quinn::pem::parse_x509_pem(include_bytes!("cert.pem")).unwrap().0;
+    let key = quinn::pem::parse_private_key_pem(include_bytes!("key.rsa")).unwrap();
+    builder.identity(Certificate::from_der(&certificate).unwrap(), PrivateKey::from_der(&key).unwrap());
+
+    // Build the QUIC endpoint
+    let mut endpoint = builder.bind(&addr).expect("Failed to bind QUIC endpoint");
+
+    // Accept incoming connections
+    loop {
+        let (conn, _) = endpoint.accept().await.expect("Error accepting connection");
+        tokio::spawn(handle_connection(conn));
+    }
+}
+
+async fn handle_connection(conn: Connection) {
+    // Process requests from the client
+    let mut incoming = conn
+        .incoming()
+        .expect("Failed to get incoming stream");
+
+    while let Some(stream) = incoming.next().await {
+        let stream = stream.expect("Failed to open stream");
+        tokio::spawn(handle_stream(stream));
+    }
+}
+
+async fn handle_stream(stream: quinn::RecvStream) {
+    let mut buf = [0; 4]; // Assuming a 4-byte integer
+    let len = stream
+        .read(&mut buf)
+        .await
+        .expect("Failed to read from stream");
+
+    // Convert the received bytes to an integer
+    let int_value = i32::from_be_bytes(buf);
+
+    // Echo the integer back to the client
+    let written = stream
+        .write_all(&int_value.to_be_bytes())
+        .await
+        .expect("Failed to write to stream");
+
+    println!("Echoed back: {}", int_value);
+}

docs/toml

@@ -0,0 +1,3 @@
+[dependencies]
+quinn = "0.6"
+tokio = { version = "1", features = ["full"] }