Cannot allocate write+execute memory on M1 Mac

[jeromepin]

Describe the bug
When running an az command, it triggers a MemoryError on a M1 Mac :

MemoryError: Cannot allocate write+execute memory for ffi.callback(). You might be running on a system that prevents this. For more information, see https://cffi.readthedocs.io/en/latest/using.html#callbacks

Full stack trace

ERROR: The command failed with an unexpected error. Here is the traceback: ERROR: Cannot allocate write+execute memory for ffi.callback(). You might be running on a system that prevents this. For more information, see https://cffi.readthedocs.io/en/latest/using.html#callbacks Traceback (most recent call last): File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/knack/cli.py", line 233, in invoke cmd_result = self.invocation.execute(args) File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/azure/cli/core/commands/__init__.py", line 664, in execute raise ex File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/azure/cli/core/commands/__init__.py", line 727, in _run_jobs_serially results.append(self._run_job(expanded_arg, cmd_copy)) File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/azure/cli/core/commands/__init__.py", line 720, in _run_job six.reraise(*sys.exc_info()) File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/six.py", line 693, in reraise raise value File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/azure/cli/core/commands/__init__.py", line 698, in _run_job result = cmd_copy(params) File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/azure/cli/core/commands/__init__.py", line 331, in __call__ return self.handler(*args, **kwargs) File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/azure/cli/core/__init__.py", line 816, in default_command_handler return op(**command_args) File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/azure/cli/command_modules/profile/custom.py", line 154, in login subscriptions = profile.find_subscriptions_on_login( File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/azure/cli/core/_profile.py", line 210, in find_subscriptions_on_login subscriptions = subscription_finder.find_from_service_principal_id( File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/azure/cli/core/_profile.py", line 899, in find_from_service_principal_id token_entry = sp_auth.acquire_token(context, resource, client_id) File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/azure/cli/core/_profile.py", line 1211, in acquire_token return authentication_context.acquire_token_with_client_credentials(resource, client_id, self.secret) File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/adal/authentication_context.py", line 179, in acquire_token_with_client_credentials return self._acquire_token(token_func) File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/adal/authentication_context.py", line 128, in _acquire_token return token_func(self) File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/adal/authentication_context.py", line 177, in token_func return token_request.get_token_with_client_credentials(client_secret) File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/adal/token_request.py", line 312, in get_token_with_client_credentials token = self._oauth_get_token(oauth_parameters) File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/adal/token_request.py", line 112, in _oauth_get_token return client.get_token(oauth_parameters) File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/adal/oauth2_client.py", line 263, in get_token resp = requests.post(token_url.geturl(), File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/requests/api.py", line 116, in post return request('post', url, data=data, json=json, **kwargs) File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/requests/api.py", line 60, in request return session.request(method=method, url=url, **kwargs) File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/requests/sessions.py", line 533, in request resp = self.send(prep, **send_kwargs) File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/requests/sessions.py", line 646, in send r = adapter.send(request, **kwargs) File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/requests/adapters.py", line 439, in send resp = conn.urlopen( File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/urllib3/connectionpool.py", line 597, in urlopen httplib_response = self._make_request(conn, method, url, File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/urllib3/connectionpool.py", line 343, in _make_request self._validate_conn(conn) File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/urllib3/connectionpool.py", line 839, in _validate_conn conn.connect() File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/urllib3/connection.py", line 330, in connect self.ssl_context = create_urllib3_context( File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/urllib3/util/ssl_.py", line 283, in create_urllib3_context context.verify_mode = cert_reqs File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/urllib3/contrib/pyopenssl.py", line 413, in verify_mode self._ctx.set_verify( File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/OpenSSL/SSL.py", line 1028, in set_verify self._verify_helper = _VerifyHelper(callback) File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/OpenSSL/SSL.py", line 331, in __init__ self.callback = _ffi.callback( MemoryError: Cannot allocate write+execute memory for ffi.callback(). You might be running on a system that prevents this. For more information, see https://cffi.readthedocs.io/en/latest/using.html#callbacks To open an issue, please run: 'az feedback'

az worked for a few hours. Now I cannot run any command. az login works but still raises this error. az feedback too, as any other subcommand.

To Reproduce
Run any az command, even az --version.

Expected behavior
Being able to run any command

Environment summary

• Installation method: pip
• Python: 3.9 downloaded from the official Python website
• Shell: bash
• OS: 11.1 Darwin 20.2.0 Darwin Kernel Version 20.2.0 root:xnu-7195.60.75~1/RELEASE_ARM64_T8101 arm64
• az CLI version: 2.19.1 (in fact, even az --version doesn't work). I had to compile cryptography myself.
• OpenSSL: OpenSSL 1.1.1i 8 Dec 2020
• System resources :

Load Avg: 2.40, 2.07, 1.84  CPU usage: 9.58% user, 6.15% sys, 84.26% idle    SharedLibs: 220M resident, 48M data, 15M linkedit.
MemRegions: 172322 total, 2387M resident, 119M private, 1369M shared. PhysMem: 7463M used (1771M wired), 75M unused.
VM: 183T vsize, 3207M framework vsize, 33870926(32) swapins, 40398818(0) swapouts

Unused memory is kind of low, but it happened with 250M unused too.

Maybe it's an issue with OpenSSL itself, but I cannot understand why it worked until then.

It works for a little while after a reboot but ends up failing again soon after.

[yonzhan]

packaging

[yungezz]

HI @fengzhou-msft could you pls help to look at it? is it related to cryptopgraphy upgradign? thanks

[jeromepin]

It appears to be coming from PyOpenSSL itself.

[yiyongheng]

hey man , i also have this problem when i using the command mitmproxy. can you give me any idea to fix this ? thank you very much !

[jeromepin]

@yiyongheng I don't have a solution. I'm not using azure-cli on my M1 mac (it was just a test) thus I did not spent more time. But this is an issue with pyopenssl. The fix would be to either stop using pyopenssl or to "fix" it, but I don't have enough knowledge to do one or another. Ideally fixing pyopenssl would be best because it would benefit every software using it.

Maybe you could run azure-cli using rosetta ?

[jiasli]

@jeromepin, thanks for reporting this issue and the investigation. Azure CLI has not been tested on M1 Mac.

Once pyca/pyopenssl#873 is fixed, we will bump pyopenssl as soon as possible.