Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

python:3.11 image ImportError: libffi.so.6: cannot open shared object file #133

Open
vhnguyenae opened this issue Dec 14, 2023 · 5 comments

Comments

@vhnguyenae
Copy link

vhnguyenae commented Dec 14, 2023

Hi, our Lambda Python 3.11 function failed after December 5th with the error:

Runtime.ImportModuleError: Unable to import module 'datadog_lambda.handler': libffi.so.6: cannot open shared object file: No such file or directory
Traceback (most recent call last):

I'm aware of similar issues opened and closed in #127, #128, and #129. I tried again this morning with the latest image public.ecr.aws/lambda/python:3.11, and the issue still persists.

However, the error is raised only if the CDK stack (including the lambda) is built and deployed by CodeCatalyst. The deployment step is run with the CodeCatalyst Amazon EC2 for Linux (Arm64) image. If I build and deploy the CDK stack from my M2 MacBook, then the error doesn't occur.

So, I think the updated version of the Python lambda base image published on the 5th has something that is not compatible with the CodeCatalyst image when doing library packaging for the Docker container of the Lambda.

Since the trace is empty when the error happens, I have no clue how to debug. But it occurred only after December 5th, so I assume it's due to a change from that version. My current workaround is using the old image tag (public.ecr.aws/lambda/python:3.11.2023.11.18.02). If you need more information to investigate the issue, I'm happy to provide more details.
Thanks in advance for your help.

@jtuliani
Copy link

Thanks for flagging @vhnguyenae. Are you 100% sure that the latest Python 3.11 image doesn't resolve this? The error makes sense for the issue in #127 / #128 / #129 but should work OK with the latest image.

Please do provide as much detail as you can to reproduce the issue.

@vhnguyenae
Copy link
Author

Hi Jonathan, thanks for looking into the issue. I can't reproduce the error any more.
After switching between public.ecr.aws/lambda/python:3.11.2023.11.18.02 and back to public.ecr.aws/lambda/python:3.11, the issue disappears. So maybe the docker image caching of the lambda is cleaned up and rebuilt again with the new deployment.
You can close the issue, and sorry for the noise 😁

@rafrafek
Copy link

rafrafek commented Dec 20, 2023

Edit:

In my case - for Python 3.12 - updating cffi Python module to 1.16.0 resolved the issue!


Same issue now with CDK and Python 3.12 when running inside GitHub Actions. It fails during cdk synth operation when creating a dependency layer for a Python 3.12 lambda function.

It was working with no issues on Python 3.9.

See the GitHub action definition:

Action definition
  cdk-synth:
    runs-on: ubuntu-22.04
    permissions:
      id-token: write
      contents: read
    steps:
      - uses: actions/checkout@v3
      - name: Install poetry
        run: pipx install poetry==1.4.2
      - uses: actions/setup-python@v4
        with:
          python-version: '3.12'
          cache: 'poetry'
      - name: Install project dependencies
        run: poetry install --all-extras
      - name: Set up CDK
        run: npm install --global aws-cdk@2.115.0
      - name: Synth
        run: cdk synth $CDK_STACK

See logs:

Logs
Run cdk synth $CDK_STACK
  cdk synth $CDK_STACK
  shell: /usr/bin/bash -e {0}
  env:
    pythonLocation: /opt/hostedtoolcache/Python/3.12.1/x64
    PKG_CONFIG_PATH: /opt/hostedtoolcache/Python/3.12.1/x64/lib/pkgconfig
    Python_ROOT_DIR: /opt/hostedtoolcache/Python/3.12.1/x64
    Python2_ROOT_DIR: /opt/hostedtoolcache/Python/3.12.1/x64
    Python3_ROOT_DIR: /opt/hostedtoolcache/Python/3.12.1/x64
    LD_LIBRARY_PATH: /opt/hostedtoolcache/Python/3.12.1/x64/lib
/home/runner/.cache/pypoetry/virtualenvs/project-name-haHW2nBX-py3.12/lib/python3.12/site-packages/aws_cdk/aws_cloudwatch/__init__.py:1: SyntaxWarning: invalid escape sequence '\|'
  '''
/home/runner/.cache/pypoetry/virtualenvs/project-name-haHW2nBX-py3.12/lib/python3.12/site-packages/aws_cdk/aws_secretsmanager/__init__.py:1: SyntaxWarning: invalid escape sequence '\`'
  '''
Bundling asset StackName/Something/DepLayer/Code/Stage...
Unable to find image 'public.ecr.aws/sam/build-python3.12:latest' locally
latest: Pulling from sam/build-python3.12
7343d3ef5f77: Pulling fs layer
3ec3c9c2c638: Pulling fs layer
92841bde068d: Pulling fs layer
2a9f1ae020e1: Pulling fs layer
7b9f736bb562: Pulling fs layer
588108576ccb: Pulling fs layer
50a8f286ab7d: Pulling fs layer
3661b35580a6: Pulling fs layer
1b96cb9d3eb1: Pulling fs layer
1769ebdf741f: Pulling fs layer
af3abb3fd3e0: Pulling fs layer
1a02a45f4a7e: Pulling fs layer
7b9f736bb562: Waiting
588108576ccb: Waiting
50a8f286ab7d: Waiting
3661b35580a6: Waiting
1b96cb9d3eb1: Waiting
1769ebdf741f: Waiting
af3abb3fd3e0: Waiting
1a02a45f4a7e: Waiting
2a9f1ae020e1: Waiting
92841bde068d: Verifying Checksum
92841bde068d: Download complete
3ec3c9c2c638: Verifying Checksum
3ec3c9c2c638: Download complete
2a9f1ae020e1: Verifying Checksum
2a9f1ae020e1: Download complete
588108576ccb: Verifying Checksum
588108576ccb: Download complete
7343d3ef5f77: Verifying Checksum
7343d3ef5f77: Download complete
7343d3ef5f77: Pull complete
3ec3c9c2c638: Pull complete
92841bde068d: Pull complete
2a9f1ae020e1: Pull complete
50a8f286ab7d: Download complete
7b9f736bb562: Verifying Checksum
7b9f736bb562: Download complete
1b96cb9d3eb1: Verifying Checksum
1b96cb9d3eb1: Download complete
af3abb3fd3e0: Verifying Checksum
af3abb3fd3e0: Download complete
1a02a45f4a7e: Verifying Checksum
1a02a45f4a7e: Download complete
1769ebdf741f: Verifying Checksum
1769ebdf741f: Download complete
7b9f736bb562: Pull complete
3661b35580a6: Verifying Checksum
3661b35580a6: Download complete
588108576ccb: Pull complete
50a8f286ab7d: Pull complete
3661b35580a6: Pull complete
1b96cb9d3eb1: Pull complete
1769ebdf741f: Pull complete
af3abb3fd3e0: Pull complete
1a02a45f4a7e: Pull complete
Digest: sha256:be770c6b1354bbacf59162fe65f0e81dc2c1a3eff79b8d0ee4d689414cf5a474
Status: Downloaded newer image for public.ecr.aws/sam/build-python3.12:latest
Installing dependencies source=functions/something/requirements.txt md5=77a5bc97fb910b22e371279fbcc166cd ...
Collecting annotated-types==0.6.0 (from -r requirements.txt (line 1))
  Downloading annotated_types-0.6.0-py3-none-any.whl (12 kB)
Collecting aws-lambda-powertools[aws-sdk,tracer,validation]==2.28.1 (from -r requirements.txt (line 4))
  Downloading aws_lambda_powertools-2.28.1-py3-none-any.whl (707 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 707.1/707.1 kB 18.4 MB/s eta 0:00:00
Collecting aws-xray-sdk==2.12.1 (from -r requirements.txt (line 7))
  Downloading aws_xray_sdk-2.12.1-py2.py3-none-any.whl (101 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 101.7/101.7 kB 28.6 MB/s eta 0:00:00
Collecting boto3==1.34.4 (from -r requirements.txt (line 10))
  Downloading boto3-1.34.4-py3-none-any.whl (139 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 139.3/139.3 kB 39.3 MB/s eta 0:00:00
Collecting botocore==1.34.4 (from -r requirements.txt (line 13))
  Downloading botocore-1.34.4-py3-none-any.whl (11.8 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 11.8/11.8 MB 95.4 MB/s eta 0:00:00
Collecting certifi==2023.11.17 (from -r requirements.txt (line 16))
  Downloading certifi-2023.11.17-py3-none-any.whl (162 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 162.5/162.5 kB 33.7 MB/s eta 0:00:00
Collecting cffi==1.15.1 (from -r requirements.txt (line 19))
  Downloading cffi-1.15.1.tar.gz (508 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 508.5/508.5 kB 71.4 MB/s eta 0:00:00
  Installing build dependencies: started
  Installing build dependencies: finished with status 'done'
  Getting requirements to build wheel: started
  Getting requirements to build wheel: finished with status 'done'
  Preparing metadata (pyproject.toml): started
  Preparing metadata (pyproject.toml): finished with status 'done'
Collecting charset-normalizer==2.1.0 (from -r requirements.txt (line 84))
  Downloading charset_normalizer-2.1.0-py3-none-any.whl (39 kB)
Collecting cryptography==41.0.4 (from -r requirements.txt (line 87))
  Downloading cryptography-41.0.4-cp37-abi3-manylinux_2_28_x86_64.whl (4.4 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 4.4/4.4 MB 97.7 MB/s eta 0:00:00
Collecting dnspython==2.2.1 (from -r requirements.txt (line 111))
  Downloading dnspython-2.2.1-py3-none-any.whl (269 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 269.1/269.1 kB 59.1 MB/s eta 0:00:00
Collecting email-validator==2.1.0.post1 (from -r requirements.txt (line 114))
  Downloading email_validator-2.1.0.post1-py3-none-any.whl (32 kB)
Collecting fastjsonschema==2.19.0 (from -r requirements.txt (line 117))
  Downloading fastjsonschema-2.19.0-py3-none-any.whl (23 kB)
Collecting idna==3.3 (from -r requirements.txt (line 120))
  Downloading idna-3.3-py3-none-any.whl (61 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 61.2/61.2 kB 18.7 MB/s eta 0:00:00
Collecting jmespath==1.0.1 (from -r requirements.txt (line 123))
  Downloading jmespath-1.0.1-py3-none-any.whl (20 kB)
Collecting pycparser==2.21 (from -r requirements.txt (line 126))
  Downloading pycparser-2.21-py2.py3-none-any.whl (118 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 118.7/118.7 kB 34.4 MB/s eta 0:00:00
Collecting pydantic-core==2.14.5 (from -r requirements.txt (line 129))
  Downloading pydantic_core-2.14.5-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (2.1 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 2.1/2.1 MB 99.6 MB/s eta 0:00:00
Collecting pydantic[email]==2.5.2 (from -r requirements.txt (line 235))
  Downloading pydantic-2.5.2-py3-none-any.whl (381 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 381.9/381.9 kB 66.3 MB/s eta 0:00:00
Collecting pyjwt[crypto]==2.8.0 (from -r requirements.txt (line 238))
  Downloading PyJWT-2.8.0-py3-none-any.whl (22 kB)
Collecting python-dateutil==2.8.2 (from -r requirements.txt (line 241))
  Downloading python_dateutil-2.8.2-py2.py3-none-any.whl (247 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 247.7/247.7 kB 55.2 MB/s eta 0:00:00
Collecting requests==2.31.0 (from -r requirements.txt (line 244))
  Downloading requests-2.31.0-py3-none-any.whl (62 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 62.6/62.6 kB 16.4 MB/s eta 0:00:00
Collecting s3transfer==0.9.0 (from -r requirements.txt (line 247))
  Downloading s3transfer-0.9.0-py3-none-any.whl (82 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 82.0/82.0 kB 24.4 MB/s eta 0:00:00
Collecting sentry-sdk==1.39.1 (from -r requirements.txt (line 250))
  Downloading sentry_sdk-1.39.1-py2.py3-none-any.whl (254 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 254.1/254.1 kB 48.8 MB/s eta 0:00:00
Collecting six==1.16.0 (from -r requirements.txt (line 253))
  Downloading six-1.16.0-py2.py3-none-any.whl (11 kB)
Collecting typing-extensions==4.8.0 (from -r requirements.txt (line 256))
  Downloading typing_extensions-4.8.0-py3-none-any.whl (31 kB)
Collecting urllib3==1.26.11 (from -r requirements.txt (line 259))
  Downloading urllib3-1.26.11-py2.py3-none-any.whl (139 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 139.9/139.9 kB 36.5 MB/s eta 0:00:00
Collecting wrapt==1.16.0 (from -r requirements.txt (line 262))
  Downloading wrapt-1.16.0-cp312-cp312-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl (87 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 87.3/87.3 kB 24.0 MB/s eta 0:00:00
Building wheels for collected packages: cffi
  Building wheel for cffi (pyproject.toml): started
  Building wheel for cffi (pyproject.toml): finished with status 'error'
  error: subprocess-exited-with-error
  
  × Building wheel for cffi (pyproject.toml) did not run successfully.
  │ exit code: 1
  ╰─> [56 lines of output]
      Package libffi was not found in the pkg-config search path.
      Perhaps you should add the directory containing `libffi.pc'
      to the PKG_CONFIG_PATH environment variable
      Package 'libffi', required by 'virtual:world', not found
      Package libffi was not found in the pkg-config search path.
      Perhaps you should add the directory containing `libffi.pc'
      to the PKG_CONFIG_PATH environment variable
      Package 'libffi', required by 'virtual:world', not found
      Package libffi was not found in the pkg-config search path.
      Perhaps you should add the directory containing `libffi.pc'
      to the PKG_CONFIG_PATH environment variable
      Package 'libffi', required by 'virtual:world', not found
      Package libffi was not found in the pkg-config search path.
      Perhaps you should add the directory containing `libffi.pc'
      to the PKG_CONFIG_PATH environment variable
      Package 'libffi', required by 'virtual:world', not found
      Package libffi was not found in the pkg-config search path.
      Perhaps you should add the directory containing `libffi.pc'
      to the PKG_CONFIG_PATH environment variable
      Package 'libffi', required by 'virtual:world', not found
      running bdist_wheel
      running build
      running build_py
      creating build
      creating build/lib.linux-x86_64-cpython-312
      creating build/lib.linux-x86_64-cpython-312/cffi
      copying cffi/error.py -> build/lib.linux-x86_64-cpython-312/cffi
      copying cffi/cparser.py -> build/lib.linux-x86_64-cpython-312/cffi
      copying cffi/backend_ctypes.py -> build/lib.linux-x86_64-cpython-312/cffi
      copying cffi/ffiplatform.py -> build/lib.linux-x86_64-cpython-312/cffi
      copying cffi/model.py -> build/lib.linux-x86_64-cpython-312/cffi
      copying cffi/recompiler.py -> build/lib.linux-x86_64-cpython-312/cffi
      copying cffi/vengine_cpy.py -> build/lib.linux-x86_64-cpython-312/cffi
      copying cffi/verifier.py -> build/lib.linux-x86_64-cpython-312/cffi
      copying cffi/vengine_gen.py -> build/lib.linux-x86_64-cpython-312/cffi
      copying cffi/pkgconfig.py -> build/lib.linux-x86_64-cpython-312/cffi
      copying cffi/__init__.py -> build/lib.linux-x86_64-cpython-312/cffi
      copying cffi/lock.py -> build/lib.linux-x86_64-cpython-312/cffi
      copying cffi/setuptools_ext.py -> build/lib.linux-x86_64-cpython-312/cffi
      copying cffi/api.py -> build/lib.linux-x86_64-cpython-312/cffi
      copying cffi/commontypes.py -> build/lib.linux-x86_64-cpython-312/cffi
      copying cffi/cffi_opcode.py -> build/lib.linux-x86_64-cpython-312/cffi
      copying cffi/_cffi_include.h -> build/lib.linux-x86_64-cpython-312/cffi
      copying cffi/parse_c_type.h -> build/lib.linux-x86_64-cpython-312/cffi
      copying cffi/_embedding.h -> build/lib.linux-x86_64-cpython-312/cffi
      copying cffi/_cffi_errors.h -> build/lib.linux-x86_64-cpython-312/cffi
      running build_ext
      building '_cffi_backend' extension
      creating build/temp.linux-x86_64-cpython-312
      creating build/temp.linux-x86_64-cpython-312/c
      gcc -fno-strict-overflow -Wsign-compare -DNDEBUG -g -O3 -Wall -fPIC -DFFI_BUILDING=1 -DUSE__THREAD -DHAVE_SYNC_SYNCHRONIZE -I/usr/include/ffi -I/usr/include/libffi -I/var/lang/include/python3.12 -c c/_cffi_backend.c -o build/temp.linux-x86_64-cpython-312/c/_cffi_backend.o
      c/_cffi_backend.c:15:10: fatal error: ffi.h: No such file or directory
         15 | #include <ffi.h>
            |          ^~~~~~~
      compilation terminated.
      error: command '/usr/bin/gcc' failed with exit code 1
      [end of output]
  
  note: This error originates from a subprocess, and is likely not a problem with pip.
  ERROR: Failed building wheel for cffi
Failed to build cffi
ERROR: Could not build wheels for cffi, which is required to install pyproject.toml-based projects

Fails with the following error:

c/_cffi_backend.c:15:10: fatal error: ffi.h: No such file or directory
Failed to build cffi

@ShariqT
Copy link

ShariqT commented Dec 30, 2023

I just pulled down the latest image -- post the fix that was mentioned in other issues -- and still got this error.

@rafrafek
Copy link

rafrafek commented Jan 2, 2024

In my case - for Python 3.12 - updating cffi Python module to 1.16.0 resolved the issue!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

4 participants