Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[REG 2.65 -> 2.66] Python crashes on startup #30792

Closed
orgads opened this issue Feb 9, 2025 · 12 comments
Closed

[REG 2.65 -> 2.66] Python crashes on startup #30792

orgads opened this issue Feb 9, 2025 · 12 comments
Assignees
Labels
Auto-Resolve Auto resolve by bot Connected Kubernetes customer-reported Issues that are reported by GitHub users external to the Azure organization. question The issue doesn't require a change to the product in order to be resolved. Most issues start as that Service Attention This issue is responsible by Azure service team.
Milestone

Comments

@orgads
Copy link
Contributor

orgads commented Feb 9, 2025

Describe the bug

Just running python.exe makes it crash :(

Full backtrace:

Child-SP          RetAddr               Call Site
00000000`005eab00 00007ffe`10c417bd     python312!PyObject_LookupAttr+0x20
00000000`005eab30 00007ffd`4b1bd3e7     _ctypes+0x17bd
00000000`005eaba0 00007ffd`4b244bf6     python312!PyNumber_Negative+0x1bb
00000000`005eabf0 00007ffd`4b1be149     python312!PyObject_MakeTpCall+0xa2
00000000`005eac50 00007ffd`4b2095c1     python312!PyObject_FastCallDictTstate+0x115
00000000`005eaca0 00007ffd`4b209436     python312!PyObject_VectorcallDict+0x35
00000000`005eace0 00007ffd`4b1c1fbe     python312!PyObject_FastCall+0x27e
00000000`005eadb0 00007ffd`4b1de97f     python312!PySet_Add+0x60e
00000000`005eadf0 00007ffd`4b1de8e5     python312!PyObject_Vectorcall+0xcf
00000000`005eae70 00007ffd`4b1dfd9b     python312!PyObject_Vectorcall+0x35
00000000`005eaeb0 00007ffd`4b209ea8     python312!PyEval_EvalFrameDefault+0x88b
00000000`005eb0e0 00007ffd`4b20a385     python312!PyType_CalculateMetaclass+0x114
00000000`005eb120 00007ffd`4b2d650c     python312!PyEval_EvalCode+0xad
00000000`005eb1a0 00007ffd`4b2d63b0     python312!PyEval_GetBuiltin+0x248
00000000`005eb230 00007ffd`4b1c1fbe     python312!PyEval_GetBuiltin+0xec
00000000`005eb2e0 00007ffd`4b2442e5     python312!PySet_Add+0x60e
00000000`005eb320 00007ffd`4b24422f     python312!PyObject_Call+0x125
00000000`005eb360 00007ffd`4b1e36b6     python312!PyObject_Call+0x6f
00000000`005eb3b0 00007ffd`4b1ddf5c     python312!PyEval_EvalFrameDefault+0x41a6
00000000`005eb5e0 00007ffd`4b1de97f     python312!PyFunction_Vectorcall+0x17c
00000000`005eb640 00007ffd`4b241bb3     python312!PyObject_Vectorcall+0xcf
00000000`005eb6c0 00007ffd`4b241ae0     python312!PyObject_CallMethodObjArgs+0x143
00000000`005eb750 00007ffd`4b21d789     python312!PyObject_CallMethodObjArgs+0x70
00000000`005eb7a0 00007ffd`4b1b46ca     python312!PySys_GetObject+0x165
00000000`005eb820 00007ffd`4b234e9c     python312!PyImport_ImportModuleLevelObject+0x326
00000000`005eb8b0 00007ffd`4b1e50f8     python312!PyNumber_Invert+0x644
00000000`005eb960 00007ffd`4b209ea8     python312!PyEval_EvalFrameDefault+0x5be8
00000000`005ebb90 00007ffd`4b20a385     python312!PyType_CalculateMetaclass+0x114
00000000`005ebbd0 00007ffd`4b2d650c     python312!PyEval_EvalCode+0xad
00000000`005ebc50 00007ffd`4b2d63b0     python312!PyEval_GetBuiltin+0x248
00000000`005ebce0 00007ffd`4b1c1fbe     python312!PyEval_GetBuiltin+0xec
00000000`005ebd90 00007ffd`4b2442e5     python312!PySet_Add+0x60e
00000000`005ebdd0 00007ffd`4b24422f     python312!PyObject_Call+0x125
00000000`005ebe10 00007ffd`4b1e36b6     python312!PyObject_Call+0x6f
00000000`005ebe60 00007ffd`4b1ddf5c     python312!PyEval_EvalFrameDefault+0x41a6
00000000`005ec090 00007ffd`4b1de97f     python312!PyFunction_Vectorcall+0x17c
00000000`005ec0f0 00007ffd`4b241bb3     python312!PyObject_Vectorcall+0xcf
00000000`005ec170 00007ffd`4b241ae0     python312!PyObject_CallMethodObjArgs+0x143
00000000`005ec200 00007ffd`4b21d789     python312!PyObject_CallMethodObjArgs+0x70
00000000`005ec250 00007ffd`4b1b46ca     python312!PySys_GetObject+0x165
00000000`005ec2d0 00007ffd`4b234e9c     python312!PyImport_ImportModuleLevelObject+0x326
00000000`005ec360 00007ffd`4b1e50f8     python312!PyNumber_Invert+0x644
00000000`005ec410 00007ffd`4b209ea8     python312!PyEval_EvalFrameDefault+0x5be8
00000000`005ec640 00007ffd`4b20a385     python312!PyType_CalculateMetaclass+0x114
00000000`005ec680 00007ffd`4b2d650c     python312!PyEval_EvalCode+0xad
00000000`005ec700 00007ffd`4b2d63b0     python312!PyEval_GetBuiltin+0x248
00000000`005ec790 00007ffd`4b1c1fbe     python312!PyEval_GetBuiltin+0xec
00000000`005ec840 00007ffd`4b2442e5     python312!PySet_Add+0x60e
00000000`005ec880 00007ffd`4b24422f     python312!PyObject_Call+0x125
00000000`005ec8c0 00007ffd`4b1e36b6     python312!PyObject_Call+0x6f
00000000`005ec910 00007ffd`4b1ddf5c     python312!PyEval_EvalFrameDefault+0x41a6
00000000`005ecb40 00007ffd`4b1de97f     python312!PyFunction_Vectorcall+0x17c
00000000`005ecba0 00007ffd`4b241bb3     python312!PyObject_Vectorcall+0xcf
00000000`005ecc20 00007ffd`4b241ae0     python312!PyObject_CallMethodObjArgs+0x143
00000000`005eccb0 00007ffd`4b21d789     python312!PyObject_CallMethodObjArgs+0x70
00000000`005ecd00 00007ffd`4b1b46ca     python312!PySys_GetObject+0x165
00000000`005ecd80 00007ffd`4b234e9c     python312!PyImport_ImportModuleLevelObject+0x326
00000000`005ece10 00007ffd`4b1e50f8     python312!PyNumber_Invert+0x644
00000000`005ecec0 00007ffd`4b209ea8     python312!PyEval_EvalFrameDefault+0x5be8
00000000`005ed0f0 00007ffd`4b20a385     python312!PyType_CalculateMetaclass+0x114
00000000`005ed130 00007ffd`4b2d650c     python312!PyEval_EvalCode+0xad
00000000`005ed1b0 00007ffd`4b2d63b0     python312!PyEval_GetBuiltin+0x248
00000000`005ed240 00007ffd`4b1c1fbe     python312!PyEval_GetBuiltin+0xec
00000000`005ed2f0 00007ffd`4b2442e5     python312!PySet_Add+0x60e
00000000`005ed330 00007ffd`4b24422f     python312!PyObject_Call+0x125
00000000`005ed370 00007ffd`4b1e36b6     python312!PyObject_Call+0x6f
00000000`005ed3c0 00007ffd`4b1ddf5c     python312!PyEval_EvalFrameDefault+0x41a6
00000000`005ed5f0 00007ffd`4b1de97f     python312!PyFunction_Vectorcall+0x17c
00000000`005ed650 00007ffd`4b241bb3     python312!PyObject_Vectorcall+0xcf
00000000`005ed6d0 00007ffd`4b241ae0     python312!PyObject_CallMethodObjArgs+0x143
00000000`005ed760 00007ffd`4b21d789     python312!PyObject_CallMethodObjArgs+0x70
00000000`005ed7b0 00007ffd`4b1b46ca     python312!PySys_GetObject+0x165
00000000`005ed830 00007ffd`4b1c1928     python312!PyImport_ImportModuleLevelObject+0x326
00000000`005ed8c0 00007ffd`4b1c1fbe     python312!PyObject_GC_New+0x248
00000000`005ed990 00007ffd`4b2442e5     python312!PySet_Add+0x60e
00000000`005ed9d0 00007ffd`4b24422f     python312!PyObject_Call+0x125
00000000`005eda10 00007ffd`4b1e36b6     python312!PyObject_Call+0x6f
00000000`005eda60 00007ffd`4b1ddf5c     python312!PyEval_EvalFrameDefault+0x41a6
00000000`005edc90 00007ffd`4b1de97f     python312!PyFunction_Vectorcall+0x17c
00000000`005edcf0 00007ffd`4b241bb3     python312!PyObject_Vectorcall+0xcf
00000000`005edd70 00007ffd`4b241ae0     python312!PyObject_CallMethodObjArgs+0x143
00000000`005ede00 00007ffd`4b1b46aa     python312!PyObject_CallMethodObjArgs+0x70
00000000`005ede50 00007ffd`4b234e9c     python312!PyImport_ImportModuleLevelObject+0x306
00000000`005edee0 00007ffd`4b1e50f8     python312!PyNumber_Invert+0x644
00000000`005edf90 00007ffd`4b209ea8     python312!PyEval_EvalFrameDefault+0x5be8
00000000`005ee1c0 00007ffd`4b20a385     python312!PyType_CalculateMetaclass+0x114
00000000`005ee200 00007ffd`4b2d650c     python312!PyEval_EvalCode+0xad
00000000`005ee280 00007ffd`4b2d63b0     python312!PyEval_GetBuiltin+0x248
00000000`005ee310 00007ffd`4b1c1fbe     python312!PyEval_GetBuiltin+0xec
00000000`005ee3c0 00007ffd`4b2442e5     python312!PySet_Add+0x60e
00000000`005ee400 00007ffd`4b24422f     python312!PyObject_Call+0x125
00000000`005ee440 00007ffd`4b1e36b6     python312!PyObject_Call+0x6f
00000000`005ee490 00007ffd`4b1ddf5c     python312!PyEval_EvalFrameDefault+0x41a6
00000000`005ee6c0 00007ffd`4b1de97f     python312!PyFunction_Vectorcall+0x17c
00000000`005ee720 00007ffd`4b241bb3     python312!PyObject_Vectorcall+0xcf
00000000`005ee7a0 00007ffd`4b241ae0     python312!PyObject_CallMethodObjArgs+0x143
00000000`005ee830 00007ffd`4b21d789     python312!PyObject_CallMethodObjArgs+0x70
00000000`005ee880 00007ffd`4b1b46ca     python312!PySys_GetObject+0x165
00000000`005ee900 00007ffd`4b1c1928     python312!PyImport_ImportModuleLevelObject+0x326
00000000`005ee990 00007ffd`4b1c1fbe     python312!PyObject_GC_New+0x248
00000000`005eea60 00007ffd`4b2442e5     python312!PySet_Add+0x60e
00000000`005eeaa0 00007ffd`4b24422f     python312!PyObject_Call+0x125
00000000`005eeae0 00007ffd`4b1e36b6     python312!PyObject_Call+0x6f
00000000`005eeb30 00007ffd`4b1ddf5c     python312!PyEval_EvalFrameDefault+0x41a6
00000000`005eed60 00007ffd`4b1de97f     python312!PyFunction_Vectorcall+0x17c
00000000`005eedc0 00007ffd`4b241bb3     python312!PyObject_Vectorcall+0xcf
00000000`005eee40 00007ffd`4b241ae0     python312!PyObject_CallMethodObjArgs+0x143
00000000`005eeed0 00007ffd`4b21d789     python312!PyObject_CallMethodObjArgs+0x70
00000000`005eef20 00007ffd`4b1b46ca     python312!PySys_GetObject+0x165
00000000`005eefa0 00007ffd`4b234e9c     python312!PyImport_ImportModuleLevelObject+0x326
00000000`005ef030 00007ffd`4b1e50f8     python312!PyNumber_Invert+0x644
00000000`005ef0e0 00007ffd`4b209ea8     python312!PyEval_EvalFrameDefault+0x5be8
00000000`005ef310 00007ffd`4b20a385     python312!PyType_CalculateMetaclass+0x114
00000000`005ef350 00007ffd`4b2d650c     python312!PyEval_EvalCode+0xad
00000000`005ef3d0 00007ffd`4b2d63b0     python312!PyEval_GetBuiltin+0x248
00000000`005ef460 00007ffd`4b1c1fbe     python312!PyEval_GetBuiltin+0xec
00000000`005ef510 00007ffd`4b2442e5     python312!PySet_Add+0x60e
00000000`005ef550 00007ffd`4b24422f     python312!PyObject_Call+0x125
00000000`005ef590 00007ffd`4b1e36b6     python312!PyObject_Call+0x6f
00000000`005ef5e0 00007ffd`4b1ddf5c     python312!PyEval_EvalFrameDefault+0x41a6
00000000`005ef810 00007ffd`4b1de97f     python312!PyFunction_Vectorcall+0x17c
00000000`005ef870 00007ffd`4b241bb3     python312!PyObject_Vectorcall+0xcf
00000000`005ef8f0 00007ffd`4b241ae0     python312!PyObject_CallMethodObjArgs+0x143
00000000`005ef980 00007ffd`4b21d789     python312!PyObject_CallMethodObjArgs+0x70
00000000`005ef9d0 00007ffd`4b1b46ca     python312!PySys_GetObject+0x165
00000000`005efa50 00007ffd`4b1c1928     python312!PyImport_ImportModuleLevelObject+0x326
00000000`005efae0 00007ffd`4b1c1fbe     python312!PyObject_GC_New+0x248
00000000`005efbb0 00007ffd`4b1de97f     python312!PySet_Add+0x60e
00000000`005efbf0 00007ffd`4b1b5475     python312!PyObject_Vectorcall+0xcf
00000000`005efc70 00007ffd`4b1b5259     python312!PyObject_CallFunction+0xa9
00000000`005efd10 00007ffd`4b25a23b     python312!PyImport_Import+0x141
00000000`005efd90 00007ffd`4b35fec2     python312!PyImport_ImportModule+0x1f
00000000`005efdc0 00007ffd`4b1fe3e8     python312!PyThread_tss_is_created+0x18b8e
00000000`005efdf0 00007ffd`4b1fe339     python312!Py_RunMain+0xc4
00000000`005efe70 00007ffd`4b1fc0ed     python312!Py_RunMain+0x15
00000000`005efea0 00007ff7`24b91230     python312!Py_Main+0x25
00000000`005efef0 00007ffe`1bed7374     python+0x1230
00000000`005eff30 00007ffe`1c1bcc91     KERNEL32!BaseThreadInitThunk+0x14
00000000`005eff60 00000000`00000000     ntdll!RtlUserThreadStart+0x21

Related command

az extension add --upgrade --name connectedk8s --debug

Errors

An error occurred. Pip failed with status code 3221225477. Use --debug for more information.

Issue script & Debug output

cli.azure.cli.core.extension.operations: Command '['F:\Temp\az-prev\python.exe', '-m', 'pip', 'install', '--target', 'C:\Users\orgads\.azure\cliextensions\connectedk8s', 'F:\Temp\tmpw2jpebcj\connectedk8s-1.10.5-py2.py3-none-any.whl', '-vv', '--disable-pip-version-check', '--no-cache-dir']' returned non-zero exit status 3221225477.
cli.azure.cli.core.extension.operations: Pip failed so deleting anything we might have installed at C:\Users\orgads.azure\cliextensions\connectedk8s
cli.azure.cli.core.extension.operations: An error occurred whilst updating.
cli.azure.cli.core.extension.operations: An error occurred. Pip failed with status code 3221225477. Use --debug for more information.

Expected behavior

python should work

Environment Summary

azure-cli 2.66.0 *

core 2.66.0 *
telemetry 1.1.0

Extensions:
account 0.2.5
connectedk8s 1.9.2
ssh 2.0.2

Dependencies:
msal 1.31.0
azure-mgmt-resource 23.1.1

Python location 'F:\Temp\az-prev\python.exe'
Extensions directory 'C:\Users\orgads.azure\cliextensions'

Python (Windows) 3.12.7 (tags/v3.12.7:0b05ead, Oct 1 2024, 03:06:41) [MSC v.1941 64 bit (AMD64)]

Legal docs and information: aka.ms/AzureCliLegal

Additional context

No response

@orgads orgads added the bug This issue requires a change to an existing behavior in the product in order to be resolved. label Feb 9, 2025
Copy link

Hi @orgads,

2.66.0 is not the latest Azure CLI(2.68.0).

If you haven't already attempted to do so, please upgrade to the latest Azure CLI version by following https://learn.microsoft.com/en-us/cli/azure/update-azure-cli.

@azure-client-tools-bot-prd azure-client-tools-bot-prd bot added the Auto-Resolve Auto resolve by bot label Feb 9, 2025
@yonzhan
Copy link
Collaborator

yonzhan commented Feb 9, 2025

Thank you for opening this issue, we will look into it.

@orgads
Copy link
Contributor Author

orgads commented Feb 9, 2025

It reproduces also with 2.68. The difference is Python version. It was upgraded from 3.11 to 3.12 in 2.66.

@microsoft-github-policy-service microsoft-github-policy-service bot added the customer-reported Issues that are reported by GitHub users external to the Azure organization. label Feb 9, 2025
@yonzhan yonzhan added Azure CLI Team The command of the issue is owned by Azure CLI team question The issue doesn't require a change to the product in order to be resolved. Most issues start as that and removed bug This issue requires a change to an existing behavior in the product in order to be resolved. labels Feb 9, 2025
@yonzhan yonzhan added this to the Backlog milestone Feb 9, 2025
@yonzhan yonzhan added Service Attention This issue is responsible by Azure service team. Connected Kubernetes and removed Azure CLI Team The command of the issue is owned by Azure CLI team labels Feb 9, 2025
Copy link
Contributor

Thanks for the feedback! We are routing this to the appropriate team for follow-up. cc @akashkeshari.

@bebound
Copy link
Contributor

bebound commented Feb 10, 2025

Duplicate of #30517

I assume this extension requires a dependency that doesn't provide a wheel, and compiling from source code fails.

The extension owner should address this.

@orgads
Copy link
Contributor Author

orgads commented Feb 10, 2025

That's not the case. Even if I only run python.exe without any arguments, it crashes.

@bebound
Copy link
Contributor

bebound commented Feb 10, 2025

Even if I only run python.exe without any arguments, it crashes.

Then it's not related to CLI. I see you're using your own Python env, please check your env.

Similar issue: #30594

@orgads
Copy link
Contributor Author

orgads commented Feb 10, 2025

What do you mean? I ran python executable from CLI2 directory.

@bebound
Copy link
Contributor

bebound commented Feb 10, 2025

According to the log, you're calling cli.azure.cli.core.extension.operations: Command '['F:\Temp\az-prev\python.exe', '-m', 'pip', 'install', '--target',
The environment also indicates that az is using Python location 'F:\Temp\az-prev\python.exe'.

@orgads
Copy link
Contributor Author

orgads commented Feb 10, 2025

az-prev is azure cli 2.66 extracted from zip.

@bebound
Copy link
Contributor

bebound commented Feb 10, 2025

I see. Try this: #29630 (comment)

orgads added a commit to orgads/azure-cli that referenced this issue Feb 10, 2025
The scripts are installed in <base>/Scripts, but PYTHONPATH was set to
<scripts>/src, although this directory doesn't even exist. I guess the
deployment used to be in src, but this is no longer the case.

Setting PYTHONPATH is done to avoid conflicts with other installations
of Python on the same machine. When it is set incorrectly, Python tend
to crash on startup (typically when loading incompatible ctypes).

Fixes Azure#28004, Azure#29630, Azure#30517, Azure#30536, Azure#30594, Azure#30700, Azure#30792.
orgads added a commit to orgads/azure-cli that referenced this issue Feb 10, 2025
The scripts are installed in <base>/Scripts, but PYTHONPATH was set to
<scripts>/src, although this directory doesn't even exist. I guess the
deployment used to be in src, but this is no longer the case.

Setting PYTHONPATH is done to avoid conflicts with other installations
of Python on the same machine. When it is set incorrectly Python tends
to crash on startup (typically when loading incompatible ctypes).

Fixes Azure#28004, Azure#29630, Azure#30517, Azure#30536, Azure#30594, Azure#30700, Azure#30792.
@orgads
Copy link
Contributor Author

orgads commented Feb 10, 2025

@orgads orgads closed this as completed Feb 10, 2025
orgads added a commit to orgads/azure-cli that referenced this issue Feb 12, 2025
The scripts are installed in <base>/Scripts, but PYTHONPATH was set to
<scripts>/src, although this directory doesn't even exist. I guess the
deployment used to be in src, but this is no longer the case.

Setting PYTHONPATH is done to avoid conflicts with other installations
of Python on the same machine. When it is set incorrectly Python tends
to crash on startup (typically when loading incompatible ctypes).

Fixes Azure#28004, Azure#29630, Azure#30517, Azure#30536, Azure#30594, Azure#30700, Azure#30792.
orgads added a commit to orgads/azure-cli that referenced this issue Feb 12, 2025
Setting PYTHONPATH is done to avoid conflicts with other installations
of Python on the same machine. When it is set incorrectly Python tends
to crash on startup (typically when loading incompatible ctypes).

There were old scripts in <base>/Scripts, which set PYTHONPATH so that
Python prefers the bundled libs over the system ones. The problem with
these scripts is that they set the wrong path.

But these scripts no longer seem to be used. They were replaced by new
ones in wbin. The problem with those is that PYTHONPATH was not set at
all in them.

Update the scripts to set PYTHONPATH correctly, both for bash and cmd.

Fixes Azure#28004, Azure#29630, Azure#30517, Azure#30536, Azure#30594, Azure#30700, Azure#30792.
orgads added a commit to orgads/azure-cli that referenced this issue Feb 17, 2025
Setting PYTHONPATH is done to avoid conflicts with other installations
of Python on the same machine. When it is set incorrectly Python tends
to crash on startup (typically when loading incompatible ctypes).

There were old scripts in <base>/Scripts, which set PYTHONPATH so that
Python prefers the bundled libs over the system ones. The problem with
these scripts is that they set the wrong path.

But these scripts no longer seem to be used. They were replaced by new
ones in wbin. The problem with those is that PYTHONPATH was not set at
all in them.

Update the scripts to set PYTHONPATH correctly, both for bash and cmd.

Fixes Azure#25067, Azure#28004, Azure#29630, Azure#30517, Azure#30536, Azure#30594, Azure#30700, Azure#30792.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Auto-Resolve Auto resolve by bot Connected Kubernetes customer-reported Issues that are reported by GitHub users external to the Azure organization. question The issue doesn't require a change to the product in order to be resolved. Most issues start as that Service Attention This issue is responsible by Azure service team.
Projects
None yet
Development

No branches or pull requests

3 participants