Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

An Implementation in Python with a Compiler #653

Closed
wants to merge 72 commits into from
Closed
Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
72 commits
Select commit Hold shift + click to select a range
da0c3b4
Copy files for rpl in to main dir
jcguu95 Mar 11, 2024
cd73f72
Add debug support to step1 (now main.py).
jcguu95 Mar 11, 2024
f7636c5
Import env, and worked on COMPILE.
jcguu95 Mar 11, 2024
4b537f7
Use a real debug logger.
jcguu95 Mar 11, 2024
402065f
Finish: def! (prim-opt), +-*/ (function).
jcguu95 Mar 11, 2024
3de4bbf
Clean the code. No logics changed.
jcguu95 Mar 11, 2024
6da0cf5
Finish: let* (prim-opt).
jcguu95 Mar 11, 2024
ad88939
Finish: do, if, fn* (prim-opts).
jcguu95 Mar 11, 2024
abc1538
Clean code. No logics changed.
jcguu95 Mar 11, 2024
cef411e
Add automatic tests and FIXME comments.
jcguu95 Mar 11, 2024
e512847
Remove comments.
jcguu95 Mar 11, 2024
8d62557
Seperate COMPILE and EXEC.
jcguu95 Mar 11, 2024
3db8a6b
Prevent compiler from polluting global bindings.
jcguu95 Mar 11, 2024
479d9d9
Finish: let* in the new setting.
jcguu95 Mar 11, 2024
57cfc0b
Scatter debugger in the compiled codes as well.
jcguu95 Mar 11, 2024
ed23b00
Renamed counter as i, and finish "do" primopt.
jcguu95 Mar 11, 2024
32ccb66
Fix bugs for arithmetics and result printing.
jcguu95 Mar 11, 2024
bc11d01
Refactor
jcguu95 Mar 11, 2024
4df900d
Make the compile switch clearer.
jcguu95 Mar 11, 2024
70f7b4d
Fix bug in last commit.
jcguu95 Mar 11, 2024
aa05747
Support nil.
jcguu95 Mar 11, 2024
bce3712
Shorter code. No logic changed.
jcguu95 Mar 11, 2024
980fe72
fn* is almost done, but there are some weird bugs.
jcguu95 Mar 12, 2024
a4668dc
Found subtle bug in `def!`; all tests passed.
jcguu95 Mar 12, 2024
61ae0bb
I'm concerned.. is this step cheating?
jcguu95 Mar 12, 2024
6b6d20e
Import core, and support scalars.
jcguu95 Mar 12, 2024
1858659
Two things:
jcguu95 Mar 12, 2024
58a6879
Tested atom, atom?, swap!, reset!, deref.
jcguu95 Mar 12, 2024
c384cf1
Tested reader @ as deref.
jcguu95 Mar 12, 2024
f627b37
Support primopt `quote`.
jcguu95 Mar 12, 2024
58f3a6b
Add major TODOs and clean up some comments.
jcguu95 Mar 12, 2024
a46552f
Try symlinking readme to front.
jcguu95 Mar 12, 2024
404376f
Rewrite TODOs and description in readme.
jcguu95 Mar 12, 2024
f404182
Implement fn* the right way without cheating.
jcguu95 Mar 12, 2024
07278d4
Try implementing defmacro! as a macro.
jcguu95 Mar 12, 2024
66c2dfd
Try implementing defmacro!
jcguu95 Mar 13, 2024
9aec4f8
Temporary Commit
jcguu95 Mar 14, 2024
39f897f
DONE defmacro! All tests passed :)
jcguu95 Mar 14, 2024
0b484ec
Move python\/compile to python-compile.
jcguu95 Mar 14, 2024
0e38138
Update README location.
jcguu95 Mar 14, 2024
500bcfc
Move codes around. No logic change.
jcguu95 Mar 14, 2024
39acea6
Support dict and vect compilation.
jcguu95 Mar 14, 2024
a2b062c
Add symlinks for testing easier.
jcguu95 Mar 14, 2024
864d13a
Define load-file. Pass all test 6.
jcguu95 Mar 14, 2024
4444bbb
Support vectors properly.
jcguu95 Mar 14, 2024
55f3a2a
Pass all hard tests from 3~7.
jcguu95 Mar 15, 2024
0c89eb1
Add tests for 7~9
jcguu95 Mar 15, 2024
f99c682
Passed all tests in step8 as well.
jcguu95 Mar 15, 2024
2745b97
Passed tests 2~8.
jcguu95 Mar 15, 2024
882c32d
Passed step 2,3,4,5,6,7,8,A
jcguu95 Mar 15, 2024
4f6e497
Refactor. All tests passed (23456789A)!
jcguu95 Mar 15, 2024
7222e7c
Clean up readme.
jcguu95 Mar 15, 2024
f6de0e3
Add a todo.
jcguu95 Mar 15, 2024
2153aa8
Finished file compiler.
jcguu95 Mar 15, 2024
b86e213
Raise recursion limit.
jcguu95 Mar 20, 2024
c07b337
Refactor. Logic Unchanged. Tests Passed.
jcguu95 Mar 20, 2024
b801b2d
Add TODOs.
jcguu95 Mar 20, 2024
87272da
Use compileall in compile_file. All tests passed.
jcguu95 Mar 20, 2024
02fe154
Remove logger, and compare performance.
jcguu95 Mar 20, 2024
ad5928d
Update readme.
jcguu95 Mar 20, 2024
ce198b9
Matching with upstream.
jcguu95 Dec 20, 2024
daf4005
Try resolve conflicts.
jcguu95 Dec 20, 2024
d005fe1
Merge branch 'kanaka:master' into python-compile
jcguu95 Dec 20, 2024
7f1bda1
Add impls info back.
jcguu95 Dec 20, 2024
de8289e
Remove original README. We no longer need it.
jcguu95 Dec 20, 2024
c129b43
Passed step0 test.
jcguu95 Dec 20, 2024
40977fe
Remove readme symlink.
jcguu95 Dec 20, 2024
ec8c941
Put python-compile readme to main page.
jcguu95 Dec 21, 2024
21189bd
Update readme.
jcguu95 Dec 21, 2024
f87119b
Update readme.
jcguu95 Dec 21, 2024
e2282a4
Add a TODO.
jcguu95 Dec 21, 2024
5125f0e
Clarify a todo.
jcguu95 Dec 21, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Remove logger, and compare performance.
  • Loading branch information
jcguu95 committed Mar 20, 2024
commit 02fe154ee1bc7a40fcdeb7ff1e4ef8450b699369
56 changes: 28 additions & 28 deletions impls/python-compile/compiler.py
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@
_consts = {} # TODO Can I make this non-global? It feels a bit leaky. I may have to make COMPILE a closure for this.

def compile_symbol (ast, env, prefix):
logger.debug(f"Compiling AST:\n{ast}\n")
#logger.debug(f"Compiling AST:\n{ast}\n")
assert(types._symbol_Q(ast))
compiled_strings = \
[f"""
Expand All @@ -14,7 +14,7 @@ def _{prefix} ():
# consts = []
def {prefix} (env):
result = env.get("{ast}")
logger.debug(f"result: {{result}}")
#logger.debug(f"result: {{result}}")
return result
return {prefix}
"""]
Expand All @@ -37,7 +37,7 @@ def {prefix} (env):
# The design of this compiler makes sure that _{prefix}_0 will
# have been defined before the following code being executed.
def compile_def (ast, env, prefix):
logger.debug(f"Compiling AST:\n{ast}\n")
#logger.debug(f"Compiling AST:\n{ast}\n")
assert(types._symbol_Q(ast[1]))
compiled_strings = COMPILE(ast[2], env, prefix=f"{prefix}_0")
compiled_strings += \
Expand All @@ -47,14 +47,14 @@ def _{prefix} ():
consts = [_{prefix}_0()]
def {prefix} (env):
result = env.set("{ast[1]}", consts[0](env))
logger.debug(f"result: {{result}}")
#logger.debug(f"result: {{result}}")
return result
return {prefix}
"""]
return compiled_strings

def compile_let (ast, env, prefix):
logger.debug(f"Compiling AST:\n{ast}\n")
#logger.debug(f"Compiling AST:\n{ast}\n")
compiled_strings = []
for i in range(1, len(ast[1]), 2):
compiled_strings += COMPILE(ast[1][i], env, prefix=f"{prefix}_{(i-1)//2}")
Expand Down Expand Up @@ -87,15 +87,15 @@ def {prefix} (env):
compiled_string += \
f"""
result = consts[{(i+2)//2}](let_env)
logger.debug(f"result: {{result}}")
#logger.debug(f"result: {{result}}")
return result
return {prefix}
"""
compiled_strings += [compiled_string]
return compiled_strings

def compile_do (ast, env, prefix):
logger.debug(f"Compiling AST:\n{ast}\n")
#logger.debug(f"Compiling AST:\n{ast}\n")
compiled_strings = []
for i in range(1, len(ast)):
compiled_strings += COMPILE(ast[i], env, prefix=f"{prefix}_{i-1}")
Expand All @@ -120,15 +120,15 @@ def {prefix} (env):"""
compiled_string += \
f"""
result = consts[{i}](env)
logger.debug(f"result: {{result}}")
#logger.debug(f"result: {{result}}")
return result
return {prefix}
"""
compiled_strings += [compiled_string]
return compiled_strings

def compile_if (ast, env, prefix):
logger.debug(f"Compiling AST:\n{ast}\n")
#logger.debug(f"Compiling AST:\n{ast}\n")
cond_compiled_strings = COMPILE(ast[1], env, prefix=f"{prefix}_0")
if_compiled_strings = COMPILE(ast[2], env, prefix=f"{prefix}_1")
else_compiled_strings = COMPILE(ast[3], env, prefix=f"{prefix}_2")
Expand All @@ -147,15 +147,15 @@ def {prefix} (env):
result = consts[1](env)
else:
result = consts[2](env)
logger.debug(f"result: {{result}}")
#logger.debug(f"result: {{result}}")
return result
return {prefix}
"""
compiled_strings = cond_compiled_strings + if_compiled_strings + else_compiled_strings + [compiled_string]
return compiled_strings

def compile_funcall (ast, env, prefix):
logger.debug(f"Compiling AST:\n{ast}\n")
#logger.debug(f"Compiling AST:\n{ast}\n")
compiled_string = \
f"""
# compile_funcall
Expand All @@ -176,7 +176,7 @@ def {prefix} (env):
compiled_string += " )"
compiled_string += \
f"""
logger.debug(f"result: {{result}}")
#logger.debug(f"result: {{result}}")
return result
return {prefix}
"""
Expand All @@ -186,7 +186,7 @@ def {prefix} (env):
return compiled_strings

def compile_literal (ast, env, prefix):
logger.debug(f"Compiling AST:\n{ast}\n")
#logger.debug(f"Compiling AST:\n{ast}\n")
# Using the global varaible _consts feels buggy, but I think
# and hope it isn't. There must be a way to pass this ast
# into the closure that will be generated by the function
Expand All @@ -202,18 +202,18 @@ def _{prefix} ():
]
def {prefix} (env):
result = consts[0]
logger.debug(f"result: {{result}}")
#logger.debug(f"result: {{result}}")
return result
# Once _{prefix} is called, the ast is embedded into the closure {prefix}, so it's time to quickly remove the link to that ast object.
popped = _consts.pop("{prefix}")
logger.debug(f"popped : {{popped}}")
logger.debug(f"_consts: {{_consts}}")
#logger.debug(f"popped : {{popped}}")
#logger.debug(f"_consts: {{_consts}}")
return {prefix}
"""]
return compiled_strings

def compile_fn (ast, env, prefix):
logger.debug(f"Compiling AST:\n{ast}\n")
#logger.debug(f"Compiling AST:\n{ast}\n")
params, body = ast[1], ast[2]
compiled_string = \
f"""
Expand All @@ -225,23 +225,23 @@ def _{prefix} ():
def {prefix} (env):
def {prefix}_fn (*args):
params, body = {params}, "{str(body)}"
logger.debug(f"params: {{params}}")
logger.debug(f" body : {{body}}")
logger.debug(f" args : {{args}}")
#logger.debug(f"params: {{params}}")
#logger.debug(f" body : {{body}}")
#logger.debug(f" args : {{args}}")
local_env = Env(env, params, types.List(args))
result = consts[0](local_env)
logger.debug(f"result: {{result}}")
#logger.debug(f"result: {{result}}")
return result
result = {prefix}_fn
logger.debug(f"result: {{result}}")
#logger.debug(f"result: {{result}}")
return result
return {prefix}
"""
compiled_strings = COMPILE(body, env, prefix=f"{prefix}_0") + [compiled_string]
return compiled_strings

def compile_hashmap (ast, env, prefix):
logger.debug(f"Compiling AST:\n{ast}\n")
#logger.debug(f"Compiling AST:\n{ast}\n")
assert(types._hash_map_Q(ast))
compiled_string = \
f"""
Expand All @@ -263,7 +263,7 @@ def _{prefix} ():
]
def {prefix} (env):
result = types._hash_map({hashmap_literal_string})
logger.debug(f"result: {{result}}")
#logger.debug(f"result: {{result}}")
return result
return {prefix}
"""
Expand Down Expand Up @@ -298,7 +298,7 @@ def {prefix} (env):
if err:
catch_env = Env(env, ["{ast[2][1]}"], [err])
result = consts[1](catch_env)
logger.debug(f"result: {{result}}")
#logger.debug(f"result: {{result}}")
return result
return {prefix}
"""
Expand Down Expand Up @@ -339,15 +339,15 @@ def macroexpand (ast, env):
count, unexpanded_ast = 0, ast
while is_macro_call(ast, env):
count += 1
logger.debug(f"Macro Expansion\n> AST:\n{ast}\n")
#logger.debug(f"Macro Expansion\n> AST:\n{ast}\n")
macro_fn = env.get(ast[0])
ast = macro_fn(*ast[1:])
logger.debug(f"Macro Expansion Finished ({count} fold(s)).\n> New AST:\n{ast}\n> Old AST:\n{unexpanded_ast}\n")
#logger.debug(f"Macro Expansion Finished ({count} fold(s)).\n> New AST:\n{ast}\n> Old AST:\n{unexpanded_ast}\n")
return ast

def COMPILE (ast, env, prefix="blk"):
# env is for macroexpansion
logger.debug(f"Compiling AST:\n{ast}\n")
#logger.debug(f"Compiling AST:\n{ast}\n")
ast = macroexpand(ast, env)
if types._symbol_Q(ast):
compiled_strings = compile_symbol(ast, env, prefix)
Expand Down
30 changes: 26 additions & 4 deletions impls/python-compile/readme.md
Original file line number Diff line number Diff line change
Expand Up @@ -12,18 +12,40 @@ in `./impls/python/`.
Quickly test them by running:

``` shell
[./mal]$ \
for ((i=2; i<=10; i++)); do
[ $i -eq 10 ] && make "test^python-compile^stepA" || make "test^python-compile^step${i}"
[ $? -ne 0 ] && { echo "Error occurred. Breaking loop."; break; }
done
```

## ~16 Times Faster Than The Interpreted Version

Per the official performance test suite, `python-compile` is near
16 times faster than the interpreted version.

``` shell
[./mal]$ make "perf^python"
----------------------------------------------
Performance test for python:
Running: env STEP=stepA_mal MAL_IMPL=js python_MODE=python ../python/run ../tests/perf1.mal
Elapsed time: 1 msecs
Running: env STEP=stepA_mal MAL_IMPL=js python_MODE=python ../python/run ../tests/perf2.mal
Elapsed time: 4 msecs
Running: env STEP=stepA_mal MAL_IMPL=js python_MODE=python ../python/run ../tests/perf3.mal
iters over 10 seconds: 9311

[./mal]$ make "perf^python-compile"
----------------------------------------------
Performance test for python-compile:
Running: env STEP=stepA_mal MAL_IMPL=js ../python-compile/run ../tests/perf1.mal
Running: env STEP=stepA_mal MAL_IMPL=js ../python-compile/run ../tests/perf2.mal
Running: env STEP=stepA_mal MAL_IMPL=js ../python-compile/run ../tests/perf3.mal
iters over 10 seconds: 148519
```

## TODO

+ **Compile Files and Compare Performance:** Analyze and compare
the performance metrics between the compiled MAL code and the
Python interpreter-based implementation.

+ Document how the design of the compiler.

+ **Clean Up Code and Submit Pull Request:** Review the codebase
Expand Down
Loading