Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Hitting errors attempting to run dpugen #25

Open
jafingerhut opened this issue Dec 14, 2023 · 5 comments
Open

Hitting errors attempting to run dpugen #25

jafingerhut opened this issue Dec 14, 2023 · 5 comments

Comments

@jafingerhut
Copy link

jafingerhut commented Dec 14, 2023 •
edited
Loading

All of the experiments I tried here were on an Ubuntu 20.04 Desktop Linux system, freshly installed. If that is part of my problem, please recommend what OS and version you recommend.

  • Try 1: attempt to use pip3 to install the dpugen package
sudo apt install python3.8-venv
cd $HOME
python3 -m venv venv-dpugen
source $HOME/venv-dpugen/bin/activate

pip3 install dpugen
# The following command recommended in the dpugen README page fails
# with the error shown:

# $ python3 -m dpugen.sai -h
# /home/andy/venv-dpugen/bin/python3: Error while finding module specification for 'dpugen.sai' (ModuleNotFoundError: No module named 'dpugen')
  • Try 2: attempt to clone the source repo and run it from there
udo apt install python3.8-venv
cd $HOME
python3 -m venv venv-dpugen
source $HOME/venv-dpugen/bin/activate

cd $HOME
git clone https://github.com/mgheorghe/dpugen
cd dpugen
pip3 install -r requirements.txt
export PYTHONPATH="$HOME/dpugen:$HOME/venv-dpugen/lib/python3.8/site-packages"

# Now I get this error when trying to run sai.py

$ ./dpugen/sai.py -h
Traceback (most recent call last):
  File "./dpugen/sai.py", line 10, in <module>
    import dpugen.saigen.enis
  File "/home/andy/dpugen/dpugen/saigen/enis.py", line 7, in <module>
    from dpugen.confbase import (
ImportError: cannot import name 'ipa' from 'dpugen.confbase' (/home/andy/dpugen/dpugen/confbase.py)

It appears that ipa is defined here: https://github.com/mgheorghe/dpugen/blob/main/work/test_sai_sample.py#L17

but it is not defined anywhere in the confbase.py source file. Even when I tried to change my local copy as shown in the diff below:

$ git diff . | cat
diff --git a/dpugen/confbase.py b/dpugen/confbase.py
index 0397ebf..045f28b 100644
--- a/dpugen/confbase.py
+++ b/dpugen/confbase.py
@@ -14,6 +14,7 @@ from munch import DefaultMunch
 
 from dpugen.dflt_params import dflt_params
 
+ipa = ipaddress.ip_address  # optimization so the . does not get executed multiple times
 socket_inet_ntoa = socket.inet_ntoa
 struct_pack = struct.pack

I get a different error when trying to run sai.py shown below:

$ echo $PYTHONPATH
/home/andy/dpugen:/home/andy/venv-dpugen/lib/python3.8/site-packages

$ ./dpugen/sai.py -h
Traceback (most recent call last):
  File "./dpugen/sai.py", line 18, in <module>
    from .confbase import ConfBase
ImportError: attempted relative import with no known parent package

Any suggestions for something to do differently?
@mgheorghe
Copy link
Owner

mgheorghe commented Dec 14, 2023 via email

@jafingerhut
Copy link
Author

I do not see any commits on main since my comment was made. Are you making commits on a different branch that I should try?

After your previous comment, on a freshly installed Ubuntu 20.04 VM I tried this sequence of commands:

sudo apt install python3-pip
pip3 install dpugen

and then the command you recommended gave this error message:

$ python3 -m dpugen.dash -o small.json
/usr/bin/python3: No module named dpugen.dash

@jafingerhut
Copy link
Author

OK, I switched back to the VM where I ran dpugen from source code, and the command:

python3 -m dpugen.dash -o small.json

is doing something for several minutes.

It is running about 8 threads to begin with, each of which gradually increases its memory usage over time, until the system runs out of RAM, one of the threads is probably killed off, and the rest continue to increase their memory usage until another is killed, etc.

How much RAM do you run this command on? I have tried up to 24 GBytes of RAM on the VM, and it still consumes everything available before threads start getting killed off, and there is only one left which continues to increase.

It finally finished after about 10 minutes, writing one file small.dpu_7.json that was 3.6 GBytes in size.

Is this expected behavior?

@mgheorghe
Copy link
Owner

mgheorghe commented Dec 15, 2023 via email

@jafingerhut
Copy link
Author

Here is a proposed PR that allows you to enable multi-threaded generation as a command line option, but without that option, the default behavior is to generate configs for each DPU sequentially.

Without that change, the latest code for a command like this one:

python3 -m dpugen.dash -o small.json

requires 128 GigaBytes of RAM to finish (about 16 GB of RAM for each of 8 threads, one per DPU).

With that change, the same command requires about 16GB of RAM, because the memory allocated for each of the 8 DPUs is freed up between the generation of configs for each DPU.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@jafingerhut @mgheorghe