Cryptography module 44.0.0 causing error "ImportError //lib/x86_64-linux-gnu/libc.so.6 /version `GLIBC_2.33' not found"

[gavin-aguiar]

Background:

Azure python functions apps running python versions 3.11 and below using cryptography module 44.0.0 fail with error

Exception:
Full Exception : Exception while executing function /Functions.api ---> Microsoft.Azure.WebJobs.Script.Workers.Rpc.RpcException /Result /Failure Exception /ImportError //lib/x86_64-linux-gnu/libc.so.6 /version `GLIBC_2.33' not found (required by /home/site/wwwroot/.python_packages/lib/site-packages/cryptography/hazmat/bindings/_rust.abi3.so). Cannot find module. Please check the requirements.txt file for the missing module. For more info, please refer the troubleshooting guide /https://aka.ms/functions-modulenotfound.........................continued

Root Cause:

Any Python function app, using azure-identity or any cryptography dependent module, built on the newest version of debian based system brings in a requirement of using GLIB_C version >= 2.33. Azure functions for python 3.11 and below are built on Bullseye with GLIB_C 2.31 is installed, and this causes a library-not-found issue.

Mitigation:

Multiple approaches to mitigate:

• Use Remote Build when deploying
  Retry performing the deployment using the Remote Build: https://learn.microsoft.com/en-us/azure/azure-functions/functions-deployment-technologies?tabs=windows#remote-build

• Use older action image for azure pipelines and GitHub Action
  Azure Pipelines: please change the vmImageName tag from “ubuntu-latest” to “ubuntu-20.04” for your yaml file- please refer to https://learn.microsoft.com/en-us/azure/devops/pipelines/agents/hosted?view=azure-devops&tabs=yaml#software
  GitHub Actions: please change the runs-on tag from “ubuntu-latest” to “ubuntu-20.04” for your yaml file- please refer to https://docs.github.com/en/actions/using-github-hosted-runners/using-github-hosted-runners/about-github-hosted-runners#supported-runners-and-hardware-resources

• For local build , please use Ubuntu 20.04 or any Debian 11 based image when doing a zip-deploy.
  [to be used only if 1 and 2 cannot be done] If you use azure-identity or any library leveraging azure-identity module

• Please pin the cryptography library to cryptography==43.0.3. After making this change, redeploy your app. For more details, refer to azure-sdk-for-python/issues/38725

Recommendation / Long-term Solution:

The official recommendation is to migrate to Python 3.12, which is currently in preview. The Python 3.12 image is built on Bookworm, and it brings in a later version of GLIB_C. Thus, the latest versions of cryptography are supported when using Python 3.12.

The ETA for 3.12 GA is the end of March 2025.

[laurens1984]

For information, option 2 using Azure Pipelines isn't going to be available from April 1st 2025. See https://devblogs.microsoft.com/devops/upcoming-updates-for-azure-pipelines-agents-images/#ubuntu for details.

[gitFire001]

Is there an ETA for this? We are already in production with the second mitigation you proposed: using an older action image for Azure Pipelines.

But with Ubuntu-20 expiring on April 1st, we need a resolution soon. Any updates?

[hallvictoria]

Hi @gitFire001, our recommendation is to migrate to Python 3.12. The latest versions of cryptography are compatible when using Python 3.12, and our GA target is end of this month.

[gitFire001]

@hallvictoria,
Could you please confirm if the fix is currently unavailable in Python 3.12 and whether it will be available exclusively in Python 3.12 by March 31st, or if it will also be available in other versions?

[hallvictoria]

The fix is already available in Python 3.12. Support for python 3.12 is still in preview though, with the GA target for end of this month.

For all other python versions, apps will need to use one of the other mitigation options listed in this issue.

[geekzter]

update: the sample YAML now uses a container image that is guaranteed to use Debian 11 (bullseye)

@laurens1984 @gitFire001 If you want to stick to 3.11 on Debian 11 (bullseye) in Azure Pipelines you can use the mcr.microsoft.com/devcontainers/python:3.11-bullseye container image with a container job

e.g.

  - job: devContainer
    container: mcr.microsoft.com/devcontainers/python:3.11-bullseye
    displayName: Use mcr.microsoft.com/devcontainers/python:3.11-bullseye container image
    pool:
      name: 'Azure Pipelines'
      vmImage: 'ubuntu-latest'
    steps:
    - task: FuncToolsInstaller@0
      displayName: 'Install Azure Functions Core Tools'
      inputs:
        version: 'latest'
    - bash: |
        lsb_release -a
        echo Azure Function Core Tools $(func --version)
        python --version
      displayName: 'Use Azure Functions Core Tools'

This provides the ability to use a container image of choice in Azure Pipelines. You may need to break up your jobs in YAML.

[miguelaguirrebwps]

Solved installing in requirements.txt:

cryptography==43.0.3

[briandelmsft]

Given that 3.12 looks like the solution here, are there any plans to actually be able to change language version in a consumption plan function?

According to this it's not supported to change the language version in consumption plan and redeploying is far from ideal.

[hallvictoria]

Hi @briandelmsft, there are currently no plans to support changing the language version without redeployment.

One of the reasons for this is because changing LinuxFx version only changes the python version -- it doesn't perform any rebuilds on the function app. That means that the app's built dependencies in .python_packages are the same for both the old language version and the new one. This can result in errors like No module named '_cffi_backend', where there's a mismatch between the dependencies which were built on an older version and the current version being used.

[briandelmsft]

Hi @briandelmsft, there are currently no plans to support changing the language version without redeployment.

One of the reasons for this is because changing LinuxFx version only changes the python version -- it doesn't perform any rebuilds on the function app. That means that the app's built dependencies in .python_packages are the same for both the old language version and the new one. This can result in errors like No module named '_cffi_backend', where there's a mismatch between the dependencies which were built on an older version and the current version being used.

@hallvictoria but if I was doing using WEBSITE_RUN_FROM_PACKAGE, is there any reason that changing the package to a zip built for 3.12 AND changing the runtime to 3.12 wouldn't work for a consumption plan function? It seems to work fine in all my testing.

[hallvictoria]

I haven't personally tested or reviewed that specific scenario, so I can't say with /complete/ certainty that it would work as anticipated. However, AFAIK as long as the app's built dependencies match the current python version and the zip is correctly formatted, then there shouldn't be any issues. For consumption specifically, I believe you'll need to manually sync triggers and restart the app to ensure the changes are picked up correctly -- assuming the URL hasn't changed, only the contents.

Is this scenario not working in your case?

[markbanksaamc]

I wrestled with this for a few days, including Microsoft telling me to entertain upgrading off Consumption Plan to higher paid plans :/

For me, the root cause was discovered after a lot of trial and error:

• I had built Python function using V3.12 using Azure Build pipeline. So the artifact is 3.12.
• The Azure Function App in Azure was created using Bicep IaC to target Python V3.10 and never got upgraded to V3.12.
• Then when releasing using Azure DevOps release pipeline, the Azure Functions Deploy task merrily pushed V3.12 artifact down onto the V3.10 function app, breaking cryptography and causing the function to not appear, without any errors logged.

So the solution is:

IaC Bicep scripts

Ensure your Microsoft.Web/sites properties|siteConfig has the correct versions:
linuxFxVersion = PYTHON|3.12
pythonVersion = 3.12

Build Pipeline

Use Python Version|Version Spec = 3.12

Release Pipeline

Azure Functions Deploy task
Runtime stack = PYTHON|3.12

To recap:

• Ensure Python 3.12 is consistant across IaC scripts, build and release pipelines
• Redeploy your Azure Function site to upgrade it to 3.12.
• Redeploy your Azure Function code (in 3.12) on top of that.
• Don't specify cryptography at all in your requirements file (let azure-identity handle that)

Everything should then be happy days!