Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

circular import error #704

Open
1 task done
jmlemetayer opened this issue Dec 6, 2024 · 8 comments
Open
1 task done

circular import error #704

jmlemetayer opened this issue Dec 6, 2024 · 8 comments
Labels
bug Something isn't working

Comments

@jmlemetayer
Copy link

Is there an existing issue for this?

  • I have searched the existing issues

Current Behavior

I am cross-compiling memray for an arm target using Yocto. The build went fine, but when I start memray I got an error:

ImportError: cannot import name 'FileFormat' from partially initialized module 'memray._memray' (most likely due to a circular import) (/usr/lib/python3.10/site-packages/memray/_memray.cpython-310-arm-linux-gnuea
bi.so)

The full stack is here:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 688, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 883, in exec_module
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "/usr/lib/python3.10/site-packages/memray/__init__.py", line 2, in <module>
    from ._memray import AllocationRecord
  File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 688, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 1184, in exec_module
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "src/memray/_memray.pyx", line 75, in init memray._memray
  File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 688, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 883, in exec_module
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "/usr/lib/python3.10/site-packages/memray/_metadata.py", line 4, in <module>
    from ._memray import FileFormat
ImportError: cannot import name 'FileFormat' from partially initialized module 'memray._memray' (most likely due to a circular import) (/usr/lib/python3.10/site-packages/memray/_memray.cpython-310-arm-linux-gnuea
bi.so)

Expected Behavior

No ImportError.

Steps To Reproduce

  1. My arm target is running Python 3.10.15
  2. Then I just have to memray -h to raise the error

Memray Version

Tried 1.14.0 and 1.15.0

Python Version

3.10

Operating System

Linux

Anything else?

I updated the file _metadata.py:

 from dataclasses import dataclass
 from datetime import datetime

-from ._memray import FileFormat


 @dataclass 
 class Metadata:
     start_time: datetime
     end_time: datetime
     total_allocations: int
     total_frames: int
     peak_memory: int
     command_line: str
     pid: int
     main_thread_id: int
     python_allocator: str
     has_native_traces: bool
     trace_python_allocators: bool
-    file_format: FileFormat
+    file_format: "FileFormat"

This fix the issue on my device.

I don't know why but it is working on my PC without the fix (Python 3.12.7).

@jmlemetayer jmlemetayer added the bug Something isn't working label Dec 6, 2024
@pablogsal
Copy link
Member

Unfortunately we cannot reproduce this in our side. Can you please give us detailed steps to get the error you are getting?

We build memray every day in our computers and in GitHub actions in many different platforms and this never appears so clearly there is something peculiar with your setup that's causing this. Without being able to do it on our side we won't be able to understand what's going on.

@jmlemetayer
Copy link
Author

Yeah this is more an informational post than a real bug.

But the circular import can be seen easily in the code:

  1. _memray.pyx line 75: from ._metadata import Metadata
  2. _metadata.py line 4: from ._memray import FileFormat

I think it only works because the imports are “type hints”, but for some reason the python on my device doesn't recognize them as such.

@jmlemetayer
Copy link
Author

Closing this issue for now. Feel free to reopen if necessary.

@jmlemetayer jmlemetayer closed this as not planned Won't fix, can't repro, duplicate, stale Dec 16, 2024
@godlygeek
Copy link
Contributor

I'm gonna reopen this, actually. I'm not really sure why this is happening only for you, but it's probably a good idea to break this circular import anyway...

@jmlemetayer Can you confirm that the issue doesn't occur if you make this patch, instead?

diff --git a/src/memray/_metadata.py b/src/memray/_metadata.py
index 5d5913b4..f61af98f 100644
--- a/src/memray/_metadata.py
+++ b/src/memray/_metadata.py
@@ -1,7 +1,9 @@
+import typing
 from dataclasses import dataclass
 from datetime import datetime

-from ._memray import FileFormat
+if typing.TYPE_CHECKING:
+    from ._memray import FileFormat


 @dataclass

@godlygeek godlygeek reopened this Dec 16, 2024
@jmlemetayer
Copy link
Author

Unfortunately this is not working:

$  memray -h
Traceback (most recent call last):
  File "/usr/bin/memray", line 5, in <module>
    from memray.__main__ import main
  File "/usr/lib/python3.10/site-packages/memray/__init__.py", line 2, in <module>
    from ._memray import AllocationRecord
  File "src/memray/_memray.pyx", line 75, in init memray._memray
  File "/usr/lib/python3.10/site-packages/memray/_metadata.py", line 10, in <module>
    class Metadata:
  File "/usr/lib/python3.10/site-packages/memray/_metadata.py", line 22, in Metadata
    file_format: FileFormat
NameError: name 'FileFormat' is not defined

As I said, I am using Yocto to build the firmware for my device. The recipe can be seen here:
https://git.openembedded.org/openembedded-core/tree/meta/recipes-devtools/python/python3_3.10.15.bb?h=kirkstone

I think this is due to a configuration problem. It seems that in my case, pgo and lto are disabled. I don't know if this is relevant or not...

@godlygeek
Copy link
Contributor

Unfortunately this is not working:

Er, oops, right. Can you instead try:

diff --git a/src/memray/_metadata.py b/src/memray/_metadata.py
index 5d5913b4..96d6416b 100644
--- a/src/memray/_metadata.py
+++ b/src/memray/_metadata.py
@@ -1,7 +1,11 @@
+from __future__ import annotations
+
+import typing
 from dataclasses import dataclass
 from datetime import datetime

-from ._memray import FileFormat
+if typing.TYPE_CHECKING:
+    from ._memray import FileFormat


 @dataclass

Same as before, but with the addition of from __future__ import annotations at the top (we're already using that in other modules)

@godlygeek
Copy link
Contributor

I think this is due to a configuration problem. It seems that in my case, pgo and lto are disabled. I don't know if this is relevant or not...

That should not make a difference, no...

@jmlemetayer
Copy link
Author

Yes, the last fix is working 👍

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

3 participants