Skip to content

These images ship vulnerable pip #1142

Closed

Description

@lachesis

They are vulnerable to the eMudhra compromise.

https://discuss.python.org/t/information-for-python-users-regarding-emudhra-emsign-ca-certificate-security-incident/109576/19

docker run --rm python:3.15-slim-trixie python -m pip --version
...
Digest: sha256:f5b92f2a669a4c347044f29367f31983280b648250ca94dea66f98c53bc87e58
Status: Downloaded newer image for python:3.15-slim-trixie
pip 26.2.1 from /usr/local/lib/python3.15/site-packages/pip (python 3.15)

Activity

  1. tianon commented on Oct 21, 2026

    @tianon
    Member
  2. yoobles commented on Oct 21, 2026

    @yoobles

    @tianon is it possible to get 3.11 and 3.12 docker images with updated pip as well?

  3. lachesis commented on Oct 21, 2026

    @lachesis
    Author

    What about Python 3.11 and 3.12? They remain supported for security fixes through the end of 2027.

    https://devguide.python.org/versions/

  4. tianon commented on Oct 21, 2026

    @tianon
    Member

    https://discuss.python.org/t/information-for-python-users-regarding-emudhra-emsign-ca-certificate-security-incident/109576/25:

    Here are expedited security releases of Python 3.15 and 3.14, and a special security release of 3.13, that upgrade pip and certifi:

    Are 3.11 & 3.12 planned ? I only saw back ports for 3.13-3.15 linked in Update bundled pip to 26.2.2 · Issue #159894 · python/cpython · GitHub

    No:

    We don’t normally update pip for security releases like Python 3.13 because of backward compatibility concerns, but since 3.13 only just entered security-only mode, doing a special release with installers seemed prudent.

    And pip itself can be updated via the usual means.

  5. yosifkit commented on Oct 21, 2026

    @yosifkit
    Member

    So, no, there will likely not be a pip bump in 3.11 or 3.12 since we follow what upstream publishes. We did bump it in the past and users suffered through many of those backward incompatible changes, so we don't do that anymore.

    If users need an updated pip in 3.11 or 3.12, they should update it in their Dockerfile.

    FROM python:3.11
    # pip install --upgrade pip==26.2.2
    # https://discuss.python.org/t/information-for-python-users-regarding-emudhra-emsign-ca-certificate-security-incident/109576
    RUN pip install "pip @ https://files.pythonhosted.org/packages/3f/06/3e4420302363fbb71be3c7f35ce042f54bfca330e3e66b3a2e3c4cca579f/pip-26.2.2-py3-none-any.whl#sha256=fd7f4dfa8190e014013b9f08f3e973f4e4c26ffd179a204895533a7254813a70"
    $ docker run -it --rm python:3.12 bash
    root@3abaf4510826:/# pip --version
    pip 25.0.1 from /usr/local/lib/python3.12/site-packages/pip (python 3.12)
    root@3abaf4510826:/# #pip install --upgrade pip==26.2.2
    root@3abaf4510826:/# pip install "pip @ https://files.pythonhosted.org/packages/3f/06/3e4420302363fbb71be3c7f35ce042f54bfca330e3e66b3a2e3c4cca579f/pip-26.2.2-py3-none-any.whl#sha256=fd7f4dfa8190e014013b9f08f3e973f4e4c26ffd179a204895533a7254813a70"
    Collecting pip@ https://files.pythonhosted.org/packages/3f/06/3e4420302363fbb71be3c7f35ce042f54bfca330e3e66b3a2e3c4cca579f/pip-26.2.2-py3-none-any.whl#sha256=fd7f4dfa8190e014013b9f08f3e973f4e4c26ffd179a204895533a7254813a70
      Downloading pip-26.2.2-py3-none-any.whl (1.8 MB)
         ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.8/1.8 MB 26.0 MB/s eta 0:00:00
    Installing collected packages: pip
      Attempting uninstall: pip
        Found existing installation: pip 25.0.1
        Uninstalling pip-25.0.1:
          Successfully uninstalled pip-25.0.1
    Successfully installed pip-26.2.2
    WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager, possibly rendering your system unusable. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv. Use the --root-user-action option if you know what you are doing and want to suppress this warning.
    root@3abaf4510826:/# pip --version
    pip 26.2.2 from /usr/local/lib/python3.12/site-packages/pip (python 3.12)
    root@3abaf4510826:/#
  6. yoobles commented on Oct 21, 2026

    @yoobles

    @yosifkit Wouldn't that strategy of updating via bare pip be subject to mitm?

    You would need to pin hashes like https://discuss.python.org/t/information-for-python-users-regarding-emudhra-emsign-ca-certificate-security-incident/109576

  7. yosifkit commented on Oct 21, 2026

    @yosifkit
    Member

    Fair point. I'll update my comment.

  8. lachesis commented on Oct 22, 2026

    @lachesis
    Author

    All users need an updated pip. Otherwise, any pipeline that uses these images becomes vulnerable to MITM compromise. A simple pip install requests on a compromised wifi network could lead to users installing a crypto miner, credential stealer, or dependency worm.

    I'm not sure whether upstream Python or Docker official images should own the solution, but IMO there's no reasonable world in which the downstream user should need to take an affirmative (and relatively tricky and fraught) action to upgrade pip.

    By the way, your given example is still insufficiently secure, because anyone who creates a venv using ensurepip will find themselves using the bundled (still insecure) version of pip.

  9. yoobles commented on Oct 22, 2026

    @yoobles

    FWIW I think this is easier from the Python side if they release a bugfix version.

  10. lachesis commented on Oct 22, 2026

    @lachesis
    Author

    There has been more discussion between the Python and Pip maintainers on the main thread. It seems the current consensus is that Python 3.11 and 3.12 will not be updated with a new pip for the same reason (not wanting backwards incompatible changes), and the pip maintainers do not have time to create new versions of pip 24 and 25.

    There has been some discussion of Docker acting as a downstream maintainer in this case, and being responsible for backporting security fixes to pip.

    I'm not really sure whether this should be a CPython fix, a pip maintainer fix, or a Docker image library fix. I know Debian took the liberty of patching pip to always use the system cert store, which would work great on the trixie versions of these images, though I know nothing about the alpine or Windows images.

    To make things even more confusing, it seems like uv has chosen to patch these installs of python. If you run:

    uv python install --reinstall cpython-3.12.15-linux-x86_64-gnu
    

    then look at the ensurepip directory, you can see that it now bundles 26.2.2 instead of the vulnerable versions of pip. The same is true for 3.11.17 as installed via uv python install.

    I've been a professional Python programmer for more than a decade, but I've never contributed to the project, so take this with a grain of salt. I'm not trying to minimize anyone's time, effort, or risk aversions. At the same time, it is hard for me to accept that I could install a version of Python that is listed as in-support for security fixes, and yet get pwned as soon as I do a pip install (anything) by some skid who vibe-coded a proxy that MITMs pypi.

  11. tianon commented on Oct 22, 2026

    @tianon
    Member

    That's a really good summary @lachesis - thanks for providing it ❤️

    The idea of patching to use the system certstore is compelling, IMO -- maybe not in all cases, as we want to reflect upstream faithfully, but for 3.11 and 3.12 that seems like it could be pretty sane, assuming the patches to do so aren't too invasive. 🤔

  12. tianon commented on Oct 22, 2026

    @tianon
    Member
  13. lachesis commented on Oct 22, 2026

    @lachesis
    Author

    Just for fun, I vibe-coded the evil proxy for about $0.30 in spend on one of the Chinese models.

    It listens on 443 and reads the SNI to see if the user is asking to talk to pypi. If not, it just forwards the connection with no interception (so browsers are still happy). If so, it instead intercepts the connection, strips any hashes from the simple index, intercepts any wheel downloads, repackages them with an "evil" file (a sentinel that does nothing in my case...), corrects their RECORD and file size, and serves them up. Older pip happily accepts this:

    $ virtualenv venv
    created virtual environment CPython3.14.8.final.0-64-x86_64 in 102ms
      creator CPython3Posix(dest=/tmp/venv, clear=False, no_vcs_ignore=False, venv_redirect=None, global=False)
      seeder FromAppData(download=False, pip=bundle, via=copy, app_data_dir=/home/user/.cache/virtualenv)
        added seed packages: pip==26.2.1
      activators BashActivator,CShellActivator,FishActivator,NushellActivator,PowerShellActivator,PythonActivator,XonshActivator
    $ . ./venv/bin/activate
    (venv) $ pip --version
    pip 26.2.1 from /tmp/venv/lib/python3.14/site-packages/pip (python 3.14)
    (venv) $ pip install --no-cache six
    Collecting six
      Downloading six-1.17.0-py2.py3-none-any.whl.metadata (1.7 kB)
    Downloading six-1.17.0-py2.py3-none-any.whl (11 kB)
    Installing collected packages: six
    Successfully installed six-1.17.0
    
    [notice] A new release of pip is available: 26.2.1 -> 26.2.2
    [notice] To update, run: pip install --upgrade pip
    (venv) $ find /tmp/venv -name sni_proxy_sentinel.txt
    /tmp/venv/lib/python3.14/site-packages/sni_proxy_sentinel.txt
    (venv) $ cat /tmp/venv/lib/python3.14/site-packages/sni_proxy_sentinel.txt
    SNI proxy lab sentinel. This file is inert.
    

    And of course, six works as normal. The only giveaway would be if you tried to browse to pypi.org in your browser, which would give you a cert error because it rejects the evil cert. That could be worked around by fingerprinting the pip TLS client-hello, which surely differs from modern Chrome/Safari/Firefox.

    The proxy could also be made to hijack upgrades for pip itself such that it didn't actually bring certifi forward, or otherwise included the backdoor in that upgrade.

    So, anyone who could convince you to send your traffic destined for pypi.org and files.pythonhosted.org to a server that they control - be that through an active network gateway MITM, a DNS hijack, or an ARP spoofing attack - would be able to pwn your machine the first time that you installed anything from pip.

    Note, pip 26.2.2 does not fall for this:

    (venv) $ pip install --no-cache --force-reinstall six
    Could not fetch URL https://pypi.org/simple/pip/: There was a problem confirming the ssl certificate: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1082) - skipping
    WARNING: Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1082)'))': /simple/six/
    
  14. tianon commented on Oct 22, 2026

    @tianon
    Member

    You might enjoy mitmproxy/mitmproxy#8500 😂

  15. tianon commented on Oct 24, 2026

    @tianon
    Member

    The idea of patching to use the system certstore is compelling, IMO -- maybe not in all cases, as we want to reflect upstream faithfully, but for 3.11 and 3.12 that seems like it could be pretty sane, assuming the patches to do so aren't too invasive. 🤔

    https://salsa.debian.org/python-team/packages/python-pip/-/blob/6f4f079131dcc2902600cee89aa54ef099d32345/debian/patches/certifi-debian-ca-certificates.patch does look pretty small -- probably have to adjust paths for applying it, but that's doable 🤔

    #1146

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions