Merge pull request #70 from yomaytk/fix-elfconv_sh

Fix some bugs for Quick Start.

.gitignore

@@ -118,4 +118,5 @@ examples/tests
 SHA*
 elfconv-v*
 
-!examples/browser/exe.html
+!examples/browser/exe.html
+!bin/exe.html

Dockerfile

@@ -49,8 +49,9 @@ git pull && ./emsdk install latest && ./emsdk activate latest && . ./emsdk_env.s
 # takes long times to build wasi-sdk in arm64 because wasi-sdk doesn't release arm64 packages.
 RUN \
   if [ "$( uname -m )" = "x86_64" ]; then \
-    cd /root && export WASI_VERSION=21 && export WASI_VERSION_FULL=${WASI_VERSION}.0 && ( echo "export WASI_VERSION=21"; echo "export WASI_VERSION_FULL=${WASI_VERSION}.0"; echo "export WASI_SDK_PATH=/root/wasi-sdk-${WASI_VERSION_FULL}" ) >> /root/.bash_profile && \
-    wget https://github.com/WebAssembly/wasi-sdk/releases/download/wasi-sdk-${WASI_VERSION}/wasi-sdk-${WASI_VERSION_FULL}-linux.tar.gz && tar xvf wasi-sdk-${WASI_VERSION_FULL}-linux.tar.gz && rm wasi-sdk-${WASI_VERSION_FULL}-linux.tar.gz; \
+    cd /root && export WASI_OS=linux && export WASI_ARCH=x86_64 && export WASI_VERSION=24 && export WASI_VERSION_FULL=${WASI_VERSION}.0 && \
+    ( echo "export WASI_OS=linux"; echo "export WASI_ARCH=x86_64"; echo "export WASI_VERSION=24"; echo "export WASI_VERSION_FULL=${WASI_VERSION}.0"; echo "export WASI_SDK_PATH=/root/wasi-sdk-${WASI_VERSION_FULL}-${WASI_ARCH}-${WASI_OS}" ) >> /root/.bash_profile && \
+    wget https://github.com/WebAssembly/wasi-sdk/releases/download/wasi-sdk-${WASI_VERSION}/wasi-sdk-${WASI_VERSION_FULL}-${WASI_ARCH}-${WASI_OS}.tar.gz && tar xvf wasi-sdk-${WASI_VERSION_FULL}-${WASI_ARCH}-${WASI_OS}.tar.gz; \
   elif [ "$( uname -m )" = "aarch64" ]; then \
     cd /root && echo "export WASI_SDK_PATH=/root/wasi-sdk/build/install/opt/wasi-sdk" >> /root/.bash_profile && git clone --recursive https://github.com/WebAssembly/wasi-sdk.git; \
     cd wasi-sdk && NINJA_FLAGS=-v make package; \

README.md

@@ -4,16 +4,14 @@ elfconv converts an original ELF binary to the LLVM bitcode using [remill](https
 and elfconv uses [emscripten](https://github.com/emscripten-core/emscripten) (for browser) or [wasi-sdk](https://github.com/WebAssembly/wasi-sdk) (for WASI runtimes) to generate the WASM binary from the LLVM bitcode file.
 
 ## Status
-> [!WARNING]
-> "**elfconv is a work in progress**" and the test is insufficient, so you may fail to compile your ELF binary or execute the generated WASM binary. Current limitations are as follows.
+"**elfconv is a work in progress**" and the test is insufficient, so you may fail to compile your ELF binary or execute the generated WASM binary. Current limitations are as follows.
 - Only support of aarch64 ELF binary as an input binary
     - Furthermore, a part of aarch64 instructions are not supported. If your ELF binary's instruction is not supported, elfconv outputs the message (\[WARNING\] Unsupported instruction at 0x...)
 - No support for stripped binaries
 - No support for shared objects
 - a lot of Linux system calls are unimplemented (ref: [`runtime/syscalls/`](https://github.com/yomaytk/elfconv/blob/main/runtime/syscalls))
 ## Quick Start
 You can try elfconv using the docker container (amd64 and arm64) by executing the commands as follows and can execute the WASM application on the both browser and host environment (WASI runtimes).
-> [!WARNING]
 > The container generated by the Dockerfile includes the installation of [wasi-sdk](https://github.com/WebAssembly/wasi-sdk). However, wasi-sdk doesn't release the package for arm64, so we build wasi-sdk in the arm64 container image, and it takes a long time. If you try elfconv, it might be good to use elfconv [release packages](https://github.com/yomaytk/elfconv/releases), or comment out the lines of the installation of wasi-sdk (The relevant part is as follows in the Dockerfile) and try to execute only in the browser.
 > ```bash
 > elif [ "$( uname -m )" = "aarch64" ]; then \
@@ -25,13 +23,13 @@ You can try elfconv using the docker container (amd64 and arm64) by executing th
 ```bash
 $ git clone --recursive https://github.com/yomaytk/elfconv
 $ cd elfconv
-elfconv/$ docker build . -t elfconv-image
-elfconv/$ docker run -it --rm -p 8080:8080 --name elfconv-container elfconv-image
-### ENTRYPOINT: elfconv/scripts/container-entry-point.sh
+elfconv/$ docker build . -t <image-name>
+elfconv/$ docker run -it --rm -p 8080:8080 --name <container-name> <image-name>
 ### running build and test ...
 # You can test elfconv using `bin/elfconv.sh`
 ~/elfconv# cd bin
-~/elfconv/bin# TARGET=browser ./elfconv.sh /path/to/ELF # e.g. ../examples/eratosthenes_sieve/a.out
+~/elfconv/bin# TARGET=Browser ./elfconv.sh /path/to/ELF # e.g. ../examples/hello/a.out
+# exe.js and exe.wasm should be generated.
 ~/elfconv/bin# emrun --no_browser --port 8080 exe.wasm.html
 Web server root directory: /root/elfconv/bin
 Now listening at http://0.0.0.0:8080/
@@ -41,13 +39,12 @@ Now, the WASM application server has started, so that you can access it (e.g. ht
 ```bash
 $ git clone --recursive https://github.com/yomaytk/elfconv
 $ cd elfconv
-$ docker build . -t elfconv-image
-$ docker run -it --name elfconv-container elfconv-image
-### ENTRYPOINT: elfconv/scripts/container-entry-point.sh
+$ docker build . -t <image-name>
+$ docker run -it --name <container-name> <image-name>
 ### running build and test ...
 # You can test elfconv using `bin/elfconv.sh`
 ~/elfconv# cd bin
-~/elfconv/bin# TARGET=wasi ./elfconv.sh /path/to/ELF # e.g. ../examples/eratosthenes_sieve/a.out
+~/elfconv/bin# TARGET=Wasi ./elfconv.sh /path/to/ELF # e.g. ../examples/hello/a.out
 ~/elfconv/bin# wasmtime exe.wasm # wasmtime is preinstalled
 ```
 ## Source code build
@@ -84,12 +81,16 @@ After finishing the build, you can find the directory `elfconv/build/`, and you
 
 You can compile the ELF binary to the WASM binary using [`scripts/dev.sh`](https://github.com/yomaytk/elfconv/blob/main/scripts/dev.sh) as follows. `dev.sh` execute the translation (ELF -> LLVM bitcode by *'lifter'*) and compiles the [`runtime/`](https://github.com/yomaytk/elfconv/tree/main/runtime) (statically linked with generated LLVM bitcode) and generate the WASM binary. when you execute the script, you should explicitly specify the path of the elfconv directory (`/root/elfconv` on the container) with `NEW_ROOT` or rewrite the `ROOT_DIR` in `dev.sh`. 
 ```bash
+### Native
+~/elfconv/build# NEW_ROOT=/path/to/elfconv TARGET=Native ../scripts/dev.sh path/to/ELF # generate the Native binary (Host achitecture) under the elfconv/build/lifter
+~/elfconv/build# ./exe.${HOST_CPU}
+------------------------
 ### Browser
-~/elfconv/build# NEW_ROOT=/path/to/elfconv TARGET=browser ../scripts/dev.sh path/to/ELF # generate the WASM binary under the elfconv/build/lifter
+~/elfconv/build# NEW_ROOT=/path/to/elfconv TARGET=Browser ../scripts/dev.sh path/to/ELF # generate the WASM binary under the elfconv/build/lifter
 ~/elfconv/build# emrun --no_browser --port 8080 ./lifter/exe.wasm.html # execute the generated WASM binary with emscripten
 ------------------------
 ### Host (WASI Runtimes)
-~/elfconv/build# NEW_ROOT=/path/to/elfconv TARGET=wasi ../scripts/dev.sh path/to/ELF
+~/elfconv/build# NEW_ROOT=/path/to/elfconv TARGET=Wasi ../scripts/dev.sh path/to/ELF
 ~/elfconv/build# wasmedge ./lifter/exe.wasm
 ```
 ## Acknowledgement

backend/remill/lib/BC/TraceLifter.cpp

@@ -969,7 +969,7 @@ void TraceLifter::Impl::Optimize() {
   int opt_cnt = 1;
   for (auto lifted_func : no_indirect_lifted_funcs) {
     std::cout << "\r["
-              << "\033[1;32m"
+              << "\033[32m"
               << "INFO"
               << "\033[0m"
               << "]"
@@ -1001,7 +1001,7 @@ void TraceLifter::Impl::Optimize() {
   int opt_cnt2 = 1;
   for (auto lifted_func : no_indirect_lifted_funcs) {
     std::cout << "\r["
-              << "\033[1;32m"
+              << "\033[32m"
               << "INFO"
               << "\033[0m"
               << "]"

bin/elfconv.sh

@@ -8,23 +8,14 @@ setting() {
   UTILS_DIR=${ROOT_DIR}/utils
   BUILD_DIR=${ROOT_DIR}/build
   BUILD_LIFTER_DIR=${BUILD_DIR}/lifter
-  EMCC=emcc
-  EMAR=emar
   OPTFLAGS="-O3"
+  EMCC=emcc
   EMCCFLAGS="${OPTFLAGS} -I${ROOT_DIR}/backend/remill/include -I${ROOT_DIR}"
+  WASISDKCC=${WASI_SDK_PATH}/bin/clang++
+  WASISDKFLAGS="${OPTFLAGS} --sysroot=${WASI_SDK_PATH}/share/wasi-sysroot -D_WASI_EMULATED_PROCESS_CLOCKS -I${ROOT_DIR}/backend/remill/include -I${ROOT_DIR} -fno-exceptions"
+  WASISDK_LINKFLAGS="-lwasi-emulated-process-clocks"
   ELFCONV_MACROS="-DELFC_BROWSER_ENV=1"
-  ELFCONV_DEBUG_MACROS=
   ELFPATH=$( realpath "$1" )
-  WASMCC=$EMCC
-  WASMCCFLAGS=$EMCCFLAGS
-  WASMAR=$EMAR
-  WASISDKCXX=${WASI_SDK_PATH}/bin/clang++
-  WASISDKAR=${WASI_SDK_PATH}/bin/ar
-  WASISDKFLAGS="${OPTFLAGS} --sysroot=${WASI_SDK_PATH}/share/wasi-sysroot -I${ROOT_DIR}/backend/remill/include -I${ROOT_DIR} -fno-exceptions"
-
-  if [ "$TARGET" = "wasm-host" ]; then
-    ELFCONV_MACROS="-DELFC_WASI_ENV=1"
-  fi
 
 }
 
@@ -38,60 +29,49 @@ main() {
     exit 1
   fi
 
-  # build runtime
-  SYSCALLCPP="syscalls/SyscallBrowser.cpp"
-  echo -e "[\033[32mINFO\033[0m] Building elfconv-Runtime ..."
-  if [ "$TARGET" = "wasm-host" ]; then
-    WASMCC=$WASISDKCXX
-    WASMCCFLAGS=$WASISDKFLAGS
-    WASMAR=$WASISDKAR
-    SYSCALLCPP="syscalls/SyscallWasi.cpp"
+  # setting for WASI
+  wasi32_target_arch=''
+  if [ "$TARGET" = "Wasi" ]; then
+    wasi32_target_arch='wasi32'
   fi
-  cd "${RUNTIME_DIR}" || { echo "cd Failure"; exit 1; }
-    # shellcheck disable=SC2086
-    $WASMCC $WASMCCFLAGS $ELFCONV_MACROS $ELFCONV_DEBUG_MACROS -o Entry.o -c Entry.cpp && \
-    $WASMCC $WASMCCFLAGS $ELFCONV_MACROS $ELFCONV_DEBUG_MACROS -o Runtime.o -c Runtime.cpp && \
-    $WASMCC $WASMCCFLAGS $ELFCONV_MACROS $ELFCONV_DEBUG_MACROS -o Memory.o -c Memory.cpp && \
-    $WASMCC $WASMCCFLAGS $ELFCONV_MACROS $ELFCONV_DEBUG_MACROS -o Syscall.o -c $SYSCALLCPP && \
-    $WASMCC $WASMCCFLAGS $ELFCONV_MACROS $ELFCONV_DEBUG_MACROS -o VmIntrinsics.o -c VmIntrinsics.cpp && \
-    $WASMCC $WASMCCFLAGS $ELFCONV_MACROS $ELFCONV_DEBUG_MACROS -o Util.o -c "${UTILS_DIR}"/Util.cpp && \
-    $WASMCC $WASMCCFLAGS $ELFCONV_MACROS $ELFCONV_DEBUG_MACROS -o elfconv.o -c "${UTILS_DIR}"/elfconv.cpp && \
-    $WASMAR rcs libelfconv.a Entry.o Runtime.o Memory.o Syscall.o VmIntrinsics.o Util.o elfconv.o
-    mv libelfconv.a "${BIN_DIR}/"
-		rm *.o
-  echo -e "[\033[32mINFO\033[0m] Generate libelfconv.a."
 
-  # ELF -> LLVM bc
+  # ELF -> LLVM bitcode
   cp -p "${BUILD_LIFTER_DIR}/elflift" "${BIN_DIR}/"
-  echo -e "[\033[32mINFO\033[0m] Converting ELF to LLVM bitcode ..."
+  echo -e "[\033[32mINFO\033[0m] ELF -> LLVM bitcode..."
     cd "${BIN_DIR}" || { echo "cd Failure"; exit 1; }
     ./elflift \
     --arch aarch64 \
     --bc_out lift.bc \
     --target_elf "$ELFPATH" \
-    --dbg_fun_cfg "$2"
-  echo -e "[\033[32mINFO\033[0m] Generate lift.bc."
+    --dbg_fun_cfg "$2" \
+    --target_arch "$wasi32_target_arch"
+  echo -e "[\033[32mINFO\033[0m] LLVM bitcode (lift.bc) was generated."
 
   # LLVM bc -> target file
   case "$TARGET" in
-    browser)
-      echo -e "[\033[32mINFO\033[0m] Converting LLVM bitcode to WASM binary (for browser) ..."
+    Browser)
+      # We use https://github.com/mame/xterm-pty for the console on the browser.
+      ELFCONV_MACROS="-DELFC_BROWSER_ENV=1"
+      echo -e "[\033[32mINFO\033[0m] Compiling to Wasm and Js (for Browser)... "
       cd "${BIN_DIR}" || { echo "cd Failure"; exit 1; }
-        $WASMCC -c lift.bc -o lift.o && \
-        $WASMCC -o exe.wasm.html -L"./" -sWASM -sALLOW_MEMORY_GROWTH lift.o -lelfconv
-      echo -e "[\033[32mINFO\033[0m] Generate WASM binary."
-      return 0
+        $EMCC $EMCCFLAGS $ELFCONV_MACROS -sALLOW_MEMORY_GROWTH -sASYNCIFY -sEXPORT_ES6 -sENVIRONMENT=web --js-library ${ROOT_DIR}/xterm-pty/emscripten-pty.js \
+            -o exe.js lift.bc ${RUNTIME_DIR}/Entry.cpp ${RUNTIME_DIR}/Memory.cpp ${RUNTIME_DIR}/Runtime.cpp ${RUNTIME_DIR}/VmIntrinsics.cpp ${RUNTIME_DIR}/syscalls/SyscallBrowser.cpp \
+            ${UTILS_DIR}/elfconv.cpp ${UTILS_DIR}/Util.cpp
+      echo -e "[\033[32mINFO\033[0m] exe.wasm and exe.js were generated."
     ;;
-    wasi)
-      echo -e "[\033[32mINFO\033[0m] Converting LLVM bitcode to WASM binary (for server) ..."
+    Wasi)
+      echo -e "[\033[32mINFO\033[0m] Compiling to Wasm (for WASI)... "
+      ELFCONV_MACROS="-DELFC_WASI_ENV=1"
       cd "${BIN_DIR}" || { echo "cd Failure"; exit 1; }
-        $WASMCC -c lift.bc -o lift.o && \
-        $WASMCC -o exe.wasm -L"./" lift.o -lelfconv
-      echo -e "[\033[32mINFO\033[0m] Generate WASM binary."
-      return 0
+      $WASISDKCC $WASISDKFLAGS $WASISDK_LINKFLAGS $ELFCONV_MACROS -o exe.wasm lift.bc ${RUNTIME_DIR}/Entry.cpp ${RUNTIME_DIR}/Memory.cpp ${RUNTIME_DIR}/Runtime.cpp ${RUNTIME_DIR}/VmIntrinsics.cpp ${RUNTIME_DIR}/syscalls/SyscallWasi.cpp \
+          ${UTILS_DIR}/elfconv.cpp ${UTILS_DIR}/Util.cpp
+      echo -e "[\033[32mINFO\033[0m] exe.wasm was generated."
     ;;
   esac
 
+  rm ${BIN_DIR}/lift.bc
+  return 0
+
 }
 
 main "$@"

bin/exe.html

@@ -0,0 +1,29 @@
+<!-- requires ./exe.js ./exe.wasm -->
+
+<!DOCTYPE html>
+<html>
+  <head>
+    <link rel="stylesheet" href="https://cdn.jsdelivr.net/npm/[email protected]/css/xterm.css" />
+  </head>
+  <body>
+    <div id="terminal"></div>
+    <script type="module">
+      import 'https://cdn.jsdelivr.net/npm/[email protected]/lib/xterm.min.js';
+      import 'https://cdn.jsdelivr.net/npm/[email protected]/index.js';
+      import initEmscripten from './exe.js';
+
+      var xterm = new Terminal();
+      xterm.open(document.getElementById('terminal'));
+
+      // Create master/slave objects
+      const { master, slave } = openpty();
+
+      // Connect the master object to xterm.js
+      xterm.loadAddon(master);
+
+      await initEmscripten({
+        pty: slave
+      });
+    </script>
+  </body>
+</html>

lifter/Lift.cpp

@@ -134,7 +134,7 @@ int main(int argc, char *argv[]) {
   host_arch->PrepareModule(module.get());
 
   // Set wasm32-unknown-wasi and wasm32 data layout if necessary.
-  if (FLAGS_target_arch == "wasm32") {
+  if (FLAGS_target_arch == "wasi32") {
     auto wasm32_dl =
         llvm::DataLayout("e-m:e-p:32:32-p10:8:8-p20:8:8-i64:64-n32:64-S128-ni:1:10:20");
     module->setDataLayout(wasm32_dl.getStringRepresentation());
@@ -147,6 +147,5 @@ int main(int argc, char *argv[]) {
 
   remill::StoreModuleToFile(module.get(), FLAGS_bc_out);
 
-  printf("[\033[32mINFO\033[0m] Lift Done.\n");
   return 0;
 }