Pact5 seems to support less characters

[trendzetter]

Issue description

When opening Mercatus DEX. The list of tokens (https://www.mercatus.works/liquidity/tokens) is not displaying with error
Encountered an unexpected character \"Ö\

Steps to reproduce

Apparently pact5 doesn't accept all characters that pact4 does. Such as Ö.

Expected Behavior

Pact5 behaves as pact4 did before regarding character encoding.

Debug Information

example (note MesutÖzilDönerKebabMerkel42Inu):

curl 'https://api.chainweb.com/chainweb/0.0/mainnet01/chain/2/pact/api/v1/local' \
  -H 'accept: */*' \
  -H 'accept-language: nl-NL,nl;q=0.6' \
  -H 'content-type: application/json' \
  -H 'origin: https://www.mercatus.works' \
  -H 'priority: u=1, i' \
  -H 'referer: https://www.mercatus.works/' \
  -H 'sec-ch-ua: "Not A(Brand";v="8", "Chromium";v="132", "Brave";v="132"' \
  -H 'sec-ch-ua-mobile: ?0' \
  -H 'sec-ch-ua-platform: "Windows"' \
  -H 'sec-fetch-dest: empty' \
  -H 'sec-fetch-mode: cors' \
  -H 'sec-fetch-site: cross-site' \
  -H 'sec-gpc: 1' \
  -H 'user-agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/132.0.0.0 Safari/537.36' \
  --data-raw $'{"hash":"oGCNba63MK7DUx6tupLwf4hpmaoS6SQQ3g8T1NeajjU","sigs":[],"cmd":"{\\"networkId\\":null,\\"payload\\":{\\"exec\\":{\\"data\\":{},\\"code\\":\\"\\\\n              (namespace \'free)\\\\n  \\\\n              (module kaddex-read G\\\\n  \\\\n                (defcap G ()\\\\n                  true)\\\\n  \\\\n                (defun pair-info (pairList:list)\\\\n                  (let* (\\\\n                    (token0 (at 0 pairList))\\\\n                    (token1 (at 1 pairList))\\\\n                    (p (kaddex.exchange.get-pair token0 token1))\\\\n                    (reserveA (kaddex.exchange.reserve-for p token0))\\\\n                    (reserveB (kaddex.exchange.reserve-for p token1))\\\\n                    (totalBal (kaddex.tokens.total-supply (kaddex.exchange.get-pair-key token0 token1)))\\\\n                  )\\\\n                  [(kaddex.exchange.get-pair-key token0 token1)\\\\n                   reserveA\\\\n                   reserveB\\\\n                   totalBal\\\\n                 ]\\\\n                ))\\\\n              )\\\\n              (map (kaddex-read.pair-info) [[arkade.token coin] [coin free.babena] [coin free.crankk01] [coin free.cyberfly_token] [coin free.finux] [coin free.kishu-ken] [coin free.maga] [coin free.teckel] [coin free.wiza] [coin hypercent.prod-hype-coin] [coin kaddex.kdx] [coin kdlaunch.kdswap-token] [coin kdlaunch.token] [coin n_1a5237df794e7277f13d8576dedff685ee7992e1.JustaChillGuy] [coin n_2669414de420c0d40bbc3caa615e989eaba83d6f.highlander] [coin n_2d62eaa0fe32f40cee5dc199c50eac5db0f0c144.Fartcoin] [coin n_2d62eaa0fe32f40cee5dc199c50eac5db0f0c144.Trump] [coin n_518dfea5f0d2abe95cbcd8956eb97f3238e274a9.AZUKI] [coin n_53af468ba542ca7566ea01df9b1941bb014abcb5.UFO] [coin n_53af468ba542ca7566ea01df9b1941bb014abcb5.UNSTABLECOIN] [coin n_548ba28af7671a42fc8dbad7a1c9cd81f149f799.MikeHerronsToe] [coin n_582fed11af00dc626812cd7890bb88e72067f28c.bro] [coin n_625e9938ae84bdb7d190f14fc283c7a6dfc15d58.ktoshi] [coin n_69f9792008a10ab25971520a7d79ab19219973e4.KadenangGinto] [coin n_69f9792008a10ab25971520a7d79ab19219973e4.PangsthePangolin] [coin n_71c27e6720665fb572433c8e52eb89833b47b49b.Peppapig] [coin n_87b3adda1e0d14a18572865d32fe4adbc82d662c.MILEI] [coin n_9392a5730bb4de2ec689948f9d7e90f2010f980e.Kadena] [coin n_95d7fe012aa7e05c187b3fc8c605ff3b1a2c521d.MesutÖzilDönerKebabMerkel42Inu] [coin n_95d7fe012aa7e05c187b3fc8c605ff3b1a2c521d.ShrekYodaTrumpMarsX12Inu] [coin n_95d7fe012aa7e05c187b3fc8c605ff3b1a2c521d.SonGokuBezosPikachu12Inu] [coin n_9e2cba688e6e99ffb3027ec02eaeaf662ccf02f2.ReturnofTrump] [coin n_a3a44245bf822580d918d9522b07503d1b879e0e.BichonFriseKDA] [coin n_a3a44245bf822580d918d9522b07503d1b879e0e.Canis] [coin n_ac853eb6032cf7f9fc61632d02c405d83f5f583e.KDAbra] [coin n_b742b4e9c600892af545afb408326e82a6c0c6ed.zUSD] [coin n_b77f201bb942a9c036164a5ea50116fbb6929326.Chihuahua] [coin n_b77f201bb942a9c036164a5ea50116fbb6929326.DeepfakeAI] [coin n_b77f201bb942a9c036164a5ea50116fbb6929326.Doge] [coin n_b77f201bb942a9c036164a5ea50116fbb6929326.ShibaInuAi] [coin n_b7c1a37e5af79697662378c4ee7434d16e87d6ff.Astro] [coin n_b7c1a37e5af79697662378c4ee7434d16e87d6ff.Space] [coin n_c5a4b8c52f0866d66bc55864998a37cc089db47c.KEKW] [coin n_c89f6bb915bf2eddf7683fdea9e40691c840f2b6.cwc] [coin n_d8b93af515926aef1eeb5251d3c17bf8cf20234e.MKGA] [coin n_d8d407d0445ed92ba102c2ce678591d69e464006.TRILLIONCARBON] [coin n_e0db65ce44a2cb408642be0a425cff2063c0631c.MATTA] [coin n_e309f0fa7cf3a13f93a8da5325cdad32790d2070.heron] [coin n_e923e3cfd7a34ddf356fc1a6eb6085c9b80a3e52.AmirWifHat] [coin runonflux.flux] [coin n_2e999f869878d1580300b15f08d666c0d5b3db59.kodi] [coin n_53af468ba542ca7566ea01df9b1941bb014abcb5.BINARY] [coin n_5b358aab10724225a64b1c1b29959ae093473fce.TRUMPINO] [coin n_5b691573817a6635e6aa2f724b988f8ba73de3df.KadenaPlanet] ])\\\\n               \\"}},\\"signers\\":[],\\"meta\\":{\\"creationTime\\":1739174519,\\"ttl\\":600,\\"gasLimit\\":150000,\\"chainId\\":\\"2\\",\\"gasPrice\\":1e-7,\\"sender\\":\\"\\"},\\"nonce\\":\\"\\\\\\"2025-02-10T08:02:08.714Z\\\\\\"\\"}"}'

[rsoeldner]

Thank you for your report, we are investigating this issue and come back to you

[CryptoPascal31]

More clearly:
It appears that Pact-4 was supporting UTF-8 identifiers, while Pact-5 does not. This can be easily reproduced in REPL:

Example:

(module KaΔena G (defcap G () true))

Notice the "greek Delta"

With Pact 4, it works:

"Loaded module KaΔena, hash KptURNrvkrHDamBGlRoBckVB-cvK0kNCLDghpAUqQ8M"

while Pact-5 refuses it:

(interactive):1:10: Lexical Error: Encountered unexpected character "Δ", Last seen "a"

As such, this issue is IMHO critical since some modules and/or functions deployed in the Pact-4 era are not accessible anymore.