LLDB renaming for Linux packaging?

linux
(Ron Olson) #1

Hi all-

In the interest of expanding the reach of Swift, I’ve been working on creating an self-contained RPM package for Fedora/Red Hat/CentOS, etc., flavors of Linux. I have a working package that I have submitted for review in a pre-release state, given that it only build successfully using current 4.1-development snapshots. The intention is to have this available once the next version of Swift is officially released, then I’m planning on doing the same thing for Debian/Ubuntu, etc.

There’s a conflict, however, with LLDB, in that there’s the Swift one, and there’s the LLVM-project one. The LLVM toolchain is already available as an installable package and because the lldb executable has the same name in both, there’s an unresolvable conflict between the llvm package and the Swift one.

The only ‘easy’ option I can come up with is to prepend swift-* to the Swift-specific lldb executables. I fully understand that those in the know are shaking their heads and going ‘heh, not that easy pal…’ and while I’m willing to tackle the challenge, I was wondering if this was a worthwhile effort; there would have to be an indicator passed to tell Swift and LLDB that it’s being compiled for Linux packaging and use a different name. This gives rise to the possible precedent of having lldb be named in a different way on Linux than on the Mac, but offers the opportunity to not have conflicts with existing llvm/lldb installations.

So I guess the question is whether this is something that the Swift-LLDB dev team would be okay with, in which case I’d be happy to submit pull requests; if there are reasons why this is not a good idea, then I guess the existing tarball-based installation will suffice.

2 Likes
(Lukas Stabe 🙃) #2

I agree the current situation is somewhat unwieldy. The Arch Linux AUR package https://aur.archlinux.org/packages/swift-lldb/ has the same problem, and conflicts with lldb proper. If I’m correct in assuming that swift-lldb contains a superset of upstream lldb functionality, replacing normal lldb with swift-lldb works, but it is still a weird situation.

I wonder wether it might be possible to extract the swift-specific functionality into another tool or library that would depend on the system-provided lldb.

(Ron Olson) #3

I can’t speak to Arch (or Ubuntu in this case), but on Fedora/RH-based distros the installation will actually fail because of the existing lldb binary; overwriting is not permitted by default and just to pile on, I’d be concerned that the llvm-based packaging and the theoretical Swift-based ones would be on different release cadences, possibly leading to bugs/unexpected behavior; worse-case-scenario. :(

I floated the swift-* prepending on the very naive thought that it might be easier to search for all the places that swift expects to find ‘lldb’ and replace with ‘swift-lldb’, which I understand has the cascading effect of having lldb know to look for its swift-* dependencies.

I haven’t dived too deeply into the llvm toolchain, but it was my understanding that its ‘module’-based (not sure if that’s even the right term here), whereby Clang is just a module that translates C/C++ files into the special objects llvm wants, and lldb includes a similar architecture that understands how to interface interactively between Clang/C++ and the llvm, and presumably Swift works in a similar way. This, theoretically, makes the possibility of an separate tool or library more theoretically possible. The real issue, AFAIK, is whether the Swift-lldb makes actual code changes, which the project page sort-of suggests. I could be totally wrong about all of this; knowledge welcome!

I’m happy to help make the situation less-unwieldy if that’s what has to be done to make Swift more available on Linux. :)

1 Like
(David Zarzycki) #4

I think there are two Red Hat ways to solve this:

  1. Have one big SRPM/spec-file generate multiple binary RPMs. Therefore build LLVM, clang, lldb, lld, and swift as a part of one source package. This approach is close to how Apple thinks about and builds their toolchain for their “OS distro”.
  2. Use the “alternatives” infrastructure to arbitrate between llvm.lldb and swift.lldb. For example, try this: alternatives --display ld. (Which amusingly lists ld.gold and ld.bfd, but not ld.lld. Whoops. Somebody should fix that.)

Personally, I think #2 is easier and more realistic for most package maintainers to work with and reason about. It also avoids the fundamental problem with #1 from the perspective of distro maintainers: the Swift and LLVM release schedules aren’t coordinated.

(Lukas Stabe 🙃) #5

I don’t think #2 really solves that problem. It’s the approach the Arch AUR package uses, but if upstream LLDB releases a new version that isn’t released with Swift yet, other packages being updated to depend on the newer LLDB version will make the users system non-upgradable, because swift-lldb would not be available in that new version yet.

(Ron Olson) #6

I’m not at all familiar with the ‘alternatives’ infrastructure; what would I be looking for to read up?

1 Like
(David Zarzycki) #7

From the perspective of package maintainers:
https://fedoraproject.org/wiki/Packaging:Alternatives

(David Zarzycki) #8

If I were to go down the alternatives path, I’d only use it to arbitrate the lldb tool between the Swift version and the upstream version. I would NOT create swift-lldb-libs or swift-lldb-devel (to use RPM naming conventions) and I would not have swift-lldb install more than what is barely necessary. In other words, the only dependency on swift-lldb should be swift itself. All other dependancies on lldb libraries/headers should fall on the upstream package.

Also and IMHO, I’d setup the alternatives priorities such that “more feature complete” variants of lldb take precedence over newer version. That way end users that do NOT install Swift get the latest lldb, but end users that do install Swift get an lldb that works out of the box. The least likely scenario is some that installs Swift but cares more about the latest lldb than Swift support. They can manually change their alternatives config to get what they want.

Finally, if the alternatives approach turns out to be too much of a pain, one could consider installing Swift into a different prefix (say /usr/swift instead of /usr) and then dropping a file into /etc/profile.d that adds the path to the PATH environmental variable. This in fact might be the easiest solution.

(Ron Olson) #9

Oh, those alternatives (I was thinking it had something to do with Swift, not Linux itself); sorry about the confusion. :) Reading through the guidelines, it seems like there’s a case to be made; I opened a ticket with the packaging committee a few days ago and have updated it with the suggestion.

(Ryan Lovelett) #10

@Ron_Olson is there a link we can follow along with? Or possibly a link to the spec file so that we could build with?

(Ron Olson) #11

Sure! The initial “Request For Review” page is at here.

A shortcut to the spec file, as well as an rpm you can install is available on my Fedora People page. I update both to later git releases when I can.

1 Like
(Ryan Lovelett) #12

What are your thoughts on https://copr.fedorainfracloud.org?

At one point I had thought that was what Fedora was using as a sandbox for building packages. Not to mention it gives a fairly simple way to make RPMs for multiple versions of Fedora.

Unfortunately, I cannot find a lot of good docs on how to use it.

EDIT: Also thank you for the links

(Ron Olson) #13

I hadn’t really looked at copr; it’s somehow related to the ‘new-n-improved’ packaging process that was planned for 27, but apparently they scrapped that idea and went back to the drawing board. That said, I’ll come up to speed on it and see if it provides a viable solution.

You can see my copr Swift page here. I’ll be updating it as I further understand it.

In the meantime, I’m doing some deep spelunking into the code to see if I can figure out where in the many places Swift is looking for “lldb”, and seeing if I can get it to look for “swift-lldb”; so far I’ve only succeeded in breaking everything. :)

(Ryan Lovelett) #14

I was trying to run that spec in a Docker image and it kept reporting that there was no clang. Long story short I think you might add BuildRequires: which to the spec file. That way clang can be discovered.

(Ron Olson) #15

Thanks for the info! I’ve added it and testing it now; it hadn’t occurred to me that ‘which’ was an optional program (though I discovered by pushing builds to Koji that rsync is).

(Jim Ingham) #16

I am not at all familiar with package managers on Linux, so I can’t contribute anything there.

But note that the way swift currently works, ONLY the lldb that was built against a particular version of the swift compiler sources will be able to debug swift binaries built by that compiler. So the model where you install swift & lldb together is the only one that really makes sense.

If you just want to debug C++, then you can use any lldb you want, and the llvm.org one is functionally the same as the github one - though as you say the release cycles are different.

But if you want to debug swift you really have to gang the swift & it’s associated lldb together to get a package that makes any sense.

In answer to another question, making swift a pure add-on to lldb is something we’re pretty far from being able to do at present. Hoping for that to happen as a solution to this problem would lead to sadness…

(Lukas Stabe 🙃) #17

Do you think it would be feasible to teach swift/swift-lldb to optionally search for it’s binaries and libraries using another name (lldb-swift would probably be good enough), so packagers can create the package in a way that also allows lldb proper to be installed at the same time?

(Jim Ingham) #18

As far as I can remember, the only thing that needs to know the name of the lldb binary is the swift driver when it starts up the Swift REPL - since the REPL is actually just a mode of lldb. Other than that lldb doesn’t know its name in any other places I can find other than in the usage string printed out for “lldb --help”.

But lldb actually ships with a bunch of binaries (lldb-server, lldb-argdumper, etc.) Do you only need to change the lldb binary’s name, or do you need to change the name of all duplicated binaries and/or libraries?

If the former, then this should be pretty straight-forward. Otherwise it gets a little more gnarly. There are a couple of helper apps lldb needs to find (lldb-rpc-server & lldb-argdumper). If you had to change the name of modules you may need to play tricks to get the python module to be called lldb. Changing the name of the module would break the testsuite & scripts that use the module. The test suite does launch lldb in a few places so we’d have to fix that.

But I think all this is just work, it shouldn’t be hard. BTW, I’m not weighing in yet on whether I would love having the name changed yet. I have to think about that a little more. But it would be doable.

(Ron Olson) #19

I believe they would all need to be changed because the standard llvm-lldb uses those names too. Basically it should be possible to have both Swift’s and the LLVM lldb installed simultaneously; replacing an existing install of lldb, regardless of which package it came from, is not allowed, given the likelihood of “user surprise” when their software starts breaking.

I’ve been trying to do the work of building the lldb executables with different names and have had zero success so far; it either errors out or, strangely, ignores my name changes and builds things as lldb-* anyway. Given that it takes, on my hardware, a minimum of an hour just to get to it failing out, even with my custom build-preset, it’s pretty slow going.

The plan, assuming I have any success, is to be able to tell the CMake scripts that Swift is being built for packaging, and to use the other name for the lldb executables. Then figure out whether it’d be proper to submit the pull requests or just apply the changes as a series of in-place patches during building for packages. This is assuming I get it working, of course. :)

(David Zarzycki) #20

If changing the names is too much of a pain in the short term, one could just install Swift into /usr/swift instead of /usr and then have the RPM drop a script into /etc/profile.d/ (or whatever it is called) to add /usr/swift/bin to $PATH. This would also avoid the Red Hat alternatives approach. The only problem with this approach is that distro maintainers probably aren’t keen on projects requiring PATH updates (not every user has their login scripts source /etc/profile.d).