LLDB install precludes installing Python in image

I had a Dockerfile that pulled the Swift image and installed Python on top of it using apt. The Dockerfile no longer works with the 5.0 image.

# apt-get install python-pip
[...]
Preparing to unpack .../python2.7-minimal_2.7.15~rc1-1ubuntu0.1_amd64.deb ...
new installation of python2.7-minimal; /usr/lib/python2.7/site-packages is a directory
which is expected a symlink to /usr/local/lib/python2.7/dist-packages.
please find the package shipping files in /usr/lib/python2.7/site-packages and
file a bug report to ship these in /usr/lib/python2.7/dist-packages instead
aborting installation of python2.7-minimal
dpkg: error processing archive /var/cache/apt/archives/python2.7-minimal_2.7.15~rc1-1ubuntu0.1_amd64.deb (--unpack):
 new python2.7-minimal package pre-installation script subprocess returned error exit status 1

Checking the site-packages directory, it appears to exist to support LLDB:

# ls -l /usr/lib/python2.7/site-packages
total 88
drwxr-xr-x 5 root root  4096 Apr 19 02:31 lldb
-rw-r--r-- 1 root root  8472 Apr 19 01:36 readline.so
-rw-r--r-- 1 root root 30130 Apr 19 02:31 six.py
-rw-r--r-- 1 root root 38747 Apr 19 01:58 six.pyc

I was able to fix this (ostensibly) by running mv /usr/lib/python2.7/site-packages /usr/lib/python2.7/dist-packages; ln -s dist-packages /usr/lib/python2.7/site-packages before running my apt commands. I was wondering if this is the right thing to do, and if this could be fixed at the source.

2 Likes

Thanks for posting. This is an issue I'm aware of and I opened [SR-10344] Linux packaging installs Python modules in wrong directory · Issue #4596 · apple/llvm-project · GitHub to track it recently.

Your workaround is the right thing to do (as much as any workaround is the "right" thing to do :slight_smile:) and it's what we're doing in the Kitura project - see Workaround for SR-10591 (#166) · Kitura/Package-Builder@f6adb98 · GitHub

I've been trying to figure out how to fix this properly - as you say, it's a problem with the shipped structure of the Linux toolchains. I think they should be putting Python modules into dist-packages not site-packages.

I found where this happens for readline - it's here: swift-lldb/CMakeLists.txt at 6b99dd2f4e7126ea0fbb8b7ae8526a15201d3484 · apple/swift-lldb · GitHub

But I think there are other places too, and I haven't tracked those down :frowning:

Thanks for your reply. I don't know if I can help much to figure out the root cause, but for now, knowing that my solution is seemingly the most advisable thing is good enough.

Are your docker files identical for Swift 4.2 and Swift 5 except for the Swift version? If not can you post any differences.

They are not the same anymore (for the sake of keeping my project going). The command in my original post worked with the Swift 4.2 image, but no longer works with the Swift 5 image:

apt-get install python-pip

Now, I have mv /usr/lib/python2.7/site-packages /usr/lib/python2.7/dist-packages; ln -s dist-packages /usr/lib/python2.7/site-packages for the Swift 5 image beforehand and the same command works.

Can you post the list of dependencies you are installing with Swift 4 and Swift 5?

The full list of dependencies is curl, python-pip and python-virtualenv. I didn't need to install curl in the Swift 4.2 container because it was already there.

I can't help but feel that your questions are a means to an end that is not satisfied by this answer. Just installing python-pip is enough to reproduce the problem I'm talking about. What are you trying to see?

1 Like

@fclout The reason I am asking is because there is no difference between Swift 4.2 and Swift 5 in regard to the contents of /usr/lib/python2.7/site-packages. This means there must be something different with your installation process that is causing this conflict.
python2.7-minimal would normally be installed during the installation of clang which is a dependency of Swift. clang as with all dependancies should be installed prior to installing Swift. If you are no longer installing clang with Swift 5 then this could be the cause of your problem.
The following is a list of dependencies that are installed when you install clang.

The following additional packages will be installed:
  binfmt-support binutils binutils-common binutils-x86-64-linux-gnu clang-6.0 gcc-7-base lib32gcc1 lib32stdc++6 libasan4 libatomic1 libbinutils libc-dev-bin libc6-dev libc6-i386
  libcilkrts5 libclang-common-6.0-dev libclang1-6.0 libffi-dev libgc1c2 libgcc-7-dev libgomp1 libitm1 libjsoncpp1 libllvm6.0 liblsan0 libmpx2 libobjc-7-dev libobjc4 libomp-dev
  libomp5 libpython-stdlib libpython2.7-minimal libpython2.7-stdlib libquadmath0 libstdc++-7-dev libtinfo-dev libtsan0 libubsan0 linux-libc-dev llvm-6.0 llvm-6.0-dev
  llvm-6.0-runtime manpages-dev python python-minimal python2.7 python2.7-minimal

As you can see this includes python python-minimal python2.7 python2.7-minimal

I am seeing similar errors when trying to install clang after Swift 5 is installed. This can be fixed with the correct combination of Depends , Provides and Replaces tags in the Swift .deb install package.

That said, it does appear that the Swift python files should be in dist-packages not site-packages according to the Debian / Python docs - https://wiki.debian.org/Python

  • dist-packages instead of site-packages. Third party Python software installed from Debian packages goes into dist-packages, not site-packages. This is to reduce conflict between the system Python, and any from-source Python build you might install manually.

The problem is reproducible simply by typing

$ docker run -it --entrypoint /bin/bash swift:5.0.1-xenial
# apt install python-pip

That's it. The following, however, works (interestingly, the 4.2.4 docker image requires you to run apt update, while the 5.0.1 doesn't):

$ docker run -it --entrypoint /bin/bash swift:4.2.4
# apt update && apt install python-pip

So either something broke in Swift 5 or the problem comes from the docker image.

edit:
After looking at the Dockerfiles, it does look as if the Swift 5 image doesn't install clang, while the Swift 4 image does.

1 Like

Swift 5 ships its own clang. Swift 4 required the system package.

Thanks Fryie! This is also what I see.

Regarding apt update, it’s probably just because the 5.0.1 image is still pretty fresh.