Re-enable old metadata format under a specific flag.

  Turns out some people liked it (looking at you Andrew). I don't think it makes sense to keep it as a default however, since many people got surprised by it. But having it enabled on demand via a flag sounds like a good compromise.

architectural-decisions/accepted/017-api-version-6-major-rewrite.md

@@ -754,7 +754,7 @@ The representation of `Value` has been changed to be more compact, more extensib
 
 The representation of transaction metadata has been both simplified and made more user-friendly, while remaining safe for more complex use-cases.  In fact, many people in the community have grown to expect transaction metadata to be JSON objects. However, they aren't. Or more specifically, they aren't necessarily. There are actually plenty of transaction metadata on-chain that aren't representable as valid JSON. Prior to version 6, Ogmios would give a so-called detailed JSON schema representation of those metadata, by encoding the binary encoding as a JSON object. This has created a lot of confusion for rookie users not yet familiar with Cardano entrails who would be expecting a plain JSON object. Plus, the format was unpractical to parse for client down the line as it used object keys as type discriminant, leaving decoders no choice to try various encoding alternatively.
 
-Starting from version 6, Ogmios returns transaction metadata as JSON object _when possible_ and fallback to CBOR otherwise. In fact, when metadata aren't representable as JSON object, this is probably because they are some elaborated binary encoding and users consuming them are most seemingly capable of decoding that themselves in the way they intended. Ogmios can be configured to always return the CBOR output using the `--include-metadata-cbor` flag on start.
+Starting from version 6, **by default** (see note below) Ogmios returns transaction metadata as JSON object _when possible_ and fallback to CBOR otherwise. In fact, when metadata aren't representable as JSON object, this is probably because they are some elaborated binary encoding and users consuming them are most seemingly capable of decoding that themselves in the way they intended. Ogmios can be configured to always return the CBOR output using the `--include-metadata-cbor` flag on start.
 
 To give a concrete example:
 
@@ -809,3 +809,7 @@ To give a concrete example:
 </table>
 
 When it isn't possible to represent the metadata as a plain JSON object, the `json` field is simply omitted and the metadata is only provided as CBOR.
+
+> [!INFO]
+>
+> The old behavior can be requested on-demand by enabling the `--metadata-detailed-schema` flag. When enabled, the `json` metadata will always be present and use the old declarative representation. This can be used in combination with the new `--include-metadata-cbor` flag as well.

clients/TypeScript/packages/schema/src/index.ts

@@ -94,10 +94,12 @@ export type Nonce = Neutral | DigestBlake2B256;
 export type Neutral = "neutral";
 export type Int64 = number;
 export type CostModel = Int64[];
-export type Metadatum = Integer | String | ArrayMetadatum | ObjectMetadatum;
+export type Metadatum = MetadatumNoSchema | MetadatumDetailedSchema;
+export type MetadatumNoSchema = Integer | String | ArrayMetadatum | ObjectMetadatum;
 export type Integer = bigint;
 export type String = string;
-export type ArrayMetadatum = Metadatum[];
+export type ArrayMetadatum = MetadatumNoSchema[];
+export type MetadatumDetailedSchema = Int | String1 | Bytes | List | Map;
 /**
  * An Ed25519 verification key.
  */
@@ -951,7 +953,26 @@ export interface MetadataLabels {
   };
 }
 export interface ObjectMetadatum {
-  [k: string]: Metadatum;
+  [k: string]: MetadatumNoSchema;
+}
+export interface Int {
+  int: bigint;
+}
+export interface String1 {
+  string: string;
+}
+export interface Bytes {
+  bytes: string;
+}
+export interface List {
+  list: MetadatumDetailedSchema[];
+}
+export interface Map {
+  map: MetadatumMap[];
+}
+export interface MetadatumMap {
+  k: MetadatumDetailedSchema;
+  v: MetadatumDetailedSchema;
 }
 /**
  * A signatory (EdDSA) for the transaction. The fields 'chainCode' and 'addressAttributes' are only present on bootstrap signatures (when spending from a Byron/Bootstrap address).

docs/static/cardano.json

@@ -1434,13 +1434,86 @@
   , "Metadatum":
     { "title": "Metadatum"
     , "oneOf":
-      [ { "title": "Integer", "type": "integer" }
-      , { "title": "String", "type": "string" }
-      , { "title": "Array<Metadatum>", "type": "array", "items": { "$ref": "cardano.json#/definitions/Metadatum" } }
-      , { "title": "Object<Metadatum>", "type": "object", "additionalProperties": { "$ref": "cardano.json#/definitions/Metadatum" } }
+      [ { "title": "Metadatum<NoSchema>"
+        , "oneOf":
+          [ { "title": "Integer", "type": "integer" }
+          , { "title": "String", "type": "string" }
+          , { "title": "Array<Metadatum>", "type": "array", "items": { "$ref": "cardano.json#/definitions/Metadatum/oneOf/0" } }
+          , { "title": "Object<Metadatum>", "type": "object", "additionalProperties": { "$ref": "cardano.json#/definitions/Metadatum/oneOf/0" } }
+          ]
+        }
+      , { "title": "Metadatum<DetailedSchema>"
+        , "oneOf":
+          [ { "type": "object"
+            , "title": "int"
+            , "additionalProperties": false
+            , "required": ["int"]
+            , "properties":
+              { "int":
+                  { "type": "integer"
+                  }
+              }
+            }
+          , { "type": "object"
+            , "title": "string"
+            , "additionalProperties": false
+            , "required": ["string"]
+            , "properties":
+              { "string":
+                  { "type": "string"
+                  }
+              }
+            }
+          , { "type": "object"
+            , "title": "bytes"
+            , "additionalProperties": false
+            , "required": ["bytes"]
+            , "properties":
+              { "bytes":
+                { "type": "string"
+                , "contentEncoding": "base16"
+                , "pattern": "^[0-9a-f]*$"
+                }
+              }
+            }
+          , { "type": "object"
+            , "title": "list"
+            , "additionalProperties": false
+            , "required": ["list"]
+            , "properties":
+              { "list":
+                { "type": "array"
+                , "items": { "$ref": "cardano.json#/definitions/Metadatum/oneOf/1" }
+                }
+              }
+            }
+          , { "type": "object"
+            , "title": "map"
+            , "additionalProperties": false
+            , "required": ["map"]
+            , "properties":
+              { "map":
+                { "type": "array"
+                , "items": { "$ref": "cardano.json#/definitions/MetadatumMap" }
+                }
+              }
+            }
+          ]
+        }
       ]
     }
 
+  , "MetadatumMap":
+    { "title": "MetadatumMap"
+    , "type": "object"
+    , "additionalProperties": false
+    , "required": ["k", "v"]
+    , "properties":
+      { "k": { "$ref": "cardano.json#/definitions/Metadatum/oneOf/1" }
+      , "v": { "$ref": "cardano.json#/definitions/Metadatum/oneOf/1" }
+      }
+    }
+
   , "Network":
     { "title": "Network"
     , "type": "string"

server/src/Ogmios/App/Configuration.hs

@@ -13,6 +13,7 @@ module Ogmios.App.Configuration
     -- * Configuration
     Configuration (..)
     , IncludeCbor (..)
+    , MetadataFormat (..)
     , includeAllCbor
     , omitOptionalCbor
     , Severity (..)
@@ -56,6 +57,7 @@ import Ogmios.Control.MonadLog
     )
 import Ogmios.Data.Json.Prelude
     ( IncludeCbor (..)
+    , MetadataFormat (..)
     , includeAllCbor
     , omitOptionalCbor
     )
@@ -84,6 +86,7 @@ data Configuration = Configuration
     , connectionTimeout :: !Int
     , maxInFlight :: !Int
     , includeCbor :: !IncludeCbor
+    , metadataFormat :: !MetadataFormat
     } deriving (Generic, Eq, Show)
 
 data NetworkParameters = NetworkParameters

server/src/Ogmios/App/Inspect/InspectTransaction.hs

@@ -19,7 +19,8 @@ import Ogmios.Data.Json
     , jsonToByteString
     )
 import Ogmios.Data.Json.Prelude
-    ( encodeMaybe
+    ( MetadataFormat (..)
+    , encodeMaybe
     )
 import System.Exit
     ( ExitCode (..)
@@ -46,4 +47,4 @@ inspectTransaction input =
                 errs
             exitWith (ExitFailure 1)
         Right (MultiEraDecoderSuccess transaction) ->
-            B8.putStrLn $ jsonToByteString $ encodeTx @StandardCrypto omitOptionalCbor transaction
+            B8.putStrLn $ jsonToByteString $ encodeTx @StandardCrypto (MetadataDetailedSchema, omitOptionalCbor) transaction

server/src/Ogmios/App/Server/WebSocket.hs

@@ -192,7 +192,7 @@ newWebSocketApp
     -> m WebSocketApp
 newWebSocketApp tr unliftIO = do
     NetworkParameters{slotsPerEpoch,networkMagic} <- asks (view typed)
-    Configuration{nodeSocket,maxInFlight,nodeConfig,includeCbor} <- asks (view typed)
+    Configuration{nodeSocket,maxInFlight,nodeConfig,includeCbor,metadataFormat} <- asks (view typed)
     sensors <- asks (view typed)
     let getGenesisConfig = GetGenesisConfig
             { getByronGenesis = readByronGenesis nodeConfig
@@ -203,7 +203,7 @@ newWebSocketApp tr unliftIO = do
 
     let codecs =
             ( mkChainSyncCodecs
-                (encodeBlock includeCbor)
+                (encodeBlock (metadataFormat, includeCbor))
                 encodePoint
                 encodeTip
 
@@ -214,7 +214,7 @@ newWebSocketApp tr unliftIO = do
 
             , mkTxMonitorCodecs
                 encodeTxId
-                (encodeTx includeCbor)
+                (encodeTx (metadataFormat, includeCbor))
 
             , mkTxSubmissionCodecs
                 encodeTxId

server/src/Ogmios/Data/Json.hs

@@ -153,12 +153,12 @@ encodeAcquireExpired = \case
 
 encodeBlock
     :: forall crypto. (Era (ByronEra crypto))
-    =>IncludeCbor
+    => (MetadataFormat, IncludeCbor)
     -> CardanoBlock crypto
     -> Json
 encodeBlock opts = \case
     BlockByron blk ->
-        Byron.encodeABlockOrBoundary @crypto opts (byronBlockRaw blk)
+        Byron.encodeABlockOrBoundary @crypto (snd opts) (byronBlockRaw blk)
     BlockShelley blk ->
         Shelley.encodeBlock opts blk
     BlockAllegra blk ->
@@ -245,7 +245,7 @@ encodeTx
     :: forall crypto.
         ( Crypto crypto
         )
-    => IncludeCbor
+    => (MetadataFormat, IncludeCbor)
     -> GenTx (CardanoBlock crypto)
     -> Json
 encodeTx opts = \case

server/src/Ogmios/Data/Json/Allegra.hs

@@ -48,7 +48,7 @@ type AuxiliaryScripts crypto =
 --
 encodeAuxiliaryData
     :: forall crypto era. (Era era, era ~ AllegraEra crypto)
-    => IncludeCbor
+    => (MetadataFormat, IncludeCbor)
     -> Al.AllegraTxAuxData era
     -> (Json, AuxiliaryScripts crypto)
 encodeAuxiliaryData opts (Al.AllegraTxAuxData blob scripts) =
@@ -61,7 +61,7 @@ encodeAuxiliaryData opts (Al.AllegraTxAuxData blob scripts) =
 
 encodeBlock
     :: Crypto crypto
-    => IncludeCbor
+    => (MetadataFormat, IncludeCbor)
     -> ShelleyBlock (TPraos crypto) (AllegraEra crypto)
     -> Json
 encodeBlock opts (ShelleyBlock (Ledger.Block blkHeader txs) headerHash) =
@@ -128,10 +128,10 @@ encodeTx
         ( Crypto crypto
         , era ~ AllegraEra crypto
         )
-    => IncludeCbor
+    => (MetadataFormat, IncludeCbor)
     -> Sh.ShelleyTx era
     -> Json
-encodeTx opts x =
+encodeTx (fmt, opts) x =
     encodeObject
         ( Shelley.encodeTxId (Ledger.txid @(AllegraEra crypto) (Sh.body x))
        <>
@@ -151,7 +151,7 @@ encodeTx opts x =
   where
     auxiliary = do
         hash <- Shelley.encodeAuxiliaryDataHash <$> Al.atbAuxDataHash (Sh.body x)
-        (labels, scripts) <- encodeAuxiliaryData opts <$> Sh.auxiliaryData x
+        (labels, scripts) <- encodeAuxiliaryData (fmt, opts) <$> Sh.auxiliaryData x
         pure
             ( encodeObject ("hash" .= hash <> "labels" .= labels)
             , scripts

server/src/Ogmios/Data/Json/Alonzo.hs

@@ -71,7 +71,7 @@ encodeAuxiliaryData
         ( Ledger.Script era ~ Al.AlonzoScript era
         , Ledger.Api.EraScript era
         )
-    => IncludeCbor
+    => (MetadataFormat, IncludeCbor)
     -> Al.AlonzoTxAuxData era
     -> (Json, AuxiliaryScripts era)
 encodeAuxiliaryData opts (Al.AlonzoTxAuxData blob timelocks plutus) =
@@ -99,7 +99,7 @@ encodeBinaryData =
 
 encodeBlock
     :: Crypto crypto
-    => IncludeCbor
+    => (MetadataFormat, IncludeCbor)
     -> ShelleyBlock (TPraos crypto) (AlonzoEra crypto)
     -> Json
 encodeBlock opts (ShelleyBlock (Ledger.Block blkHeader txs) headerHash) =
@@ -436,10 +436,10 @@ encodeTx
         ( Crypto crypto
         , era ~ AlonzoEra crypto
         )
-    => IncludeCbor
+    => (MetadataFormat, IncludeCbor)
     -> Al.AlonzoTx era
     -> Json
-encodeTx opts x =
+encodeTx (fmt, opts) x =
     encodeObject
         ( Shelley.encodeTxId (Ledger.txid @(AlonzoEra crypto) (Al.body x))
        <>
@@ -459,7 +459,7 @@ encodeTx opts x =
   where
     auxiliary = do
         hash <- Shelley.encodeAuxiliaryDataHash <$> Al.atbAuxDataHash (Al.body x)
-        (labels, scripts) <- encodeAuxiliaryData opts <$> Al.auxiliaryData x
+        (labels, scripts) <- encodeAuxiliaryData (fmt, opts) <$> Al.auxiliaryData x
         pure
             ( encodeObject ("hash" .= hash <> "labels" .= labels)
             , scripts

server/src/Ogmios/Data/Json/Babbage.hs

@@ -58,7 +58,7 @@ import qualified Ogmios.Data.Json.Shelley as Shelley
 encodeBlock
     :: ( Crypto crypto
        )
-    => IncludeCbor
+    => (MetadataFormat, IncludeCbor)
     -> ShelleyBlock (Praos crypto) (BabbageEra crypto)
     -> Json
 encodeBlock opts (ShelleyBlock (Ledger.Block blkHeader txs) headerHash) =
@@ -232,10 +232,10 @@ encodeTx
         ( Crypto crypto
         , era ~ BabbageEra crypto
         )
-    => IncludeCbor
+    => (MetadataFormat, IncludeCbor)
     -> Ba.AlonzoTx era
     -> Json
-encodeTx opts x =
+encodeTx (fmt, opts) x =
     encodeObject
         ( Shelley.encodeTxId (Ledger.txid @(BabbageEra crypto) (Ba.body x))
        <>
@@ -255,7 +255,7 @@ encodeTx opts x =
   where
     auxiliary = do
         hash <- Shelley.encodeAuxiliaryDataHash <$> Ba.btbAuxDataHash (Ba.body x)
-        (labels, scripts) <- Alonzo.encodeAuxiliaryData opts <$> Ba.auxiliaryData x
+        (labels, scripts) <- Alonzo.encodeAuxiliaryData (fmt, opts) <$> Ba.auxiliaryData x
         pure
             ( encodeObject ("hash" .= hash <> "labels" .= labels)
             , scripts

server/src/Ogmios/Data/Json/Conway.hs

@@ -80,7 +80,7 @@ encodeAnchor x = encodeObject
 encodeBlock
     :: ( Crypto crypto
        )
-    => IncludeCbor
+    => (MetadataFormat, IncludeCbor)
     -> ShelleyBlock (Praos crypto) (ConwayEra crypto)
     -> Json
 encodeBlock opts (ShelleyBlock (Ledger.Block blkHeader txs) headerHash) =
@@ -516,10 +516,10 @@ encodeTx
         ( Crypto crypto
         , era ~ ConwayEra crypto
         )
-   => IncludeCbor
+   => (MetadataFormat, IncludeCbor)
     -> Ba.AlonzoTx era
     -> Json
-encodeTx opts x =
+encodeTx (fmt, opts) x =
     encodeObject
         ( Shelley.encodeTxId (Ledger.txid @(ConwayEra crypto) (Cn.body x))
        <>
@@ -539,7 +539,7 @@ encodeTx opts x =
   where
     auxiliary = do
         hash <- Shelley.encodeAuxiliaryDataHash <$> Cn.ctbAdHash (Cn.body x)
-        (labels, scripts) <- Alonzo.encodeAuxiliaryData opts <$> Ba.auxiliaryData x
+        (labels, scripts) <- Alonzo.encodeAuxiliaryData (fmt, opts) <$> Ba.auxiliaryData x
         pure
             ( encodeObject ("hash" .= hash <> "labels" .= labels)
             , scripts