Hello SPM


(Eloy Duran) #1

Hi SPM team,

First of all, major congrats to you and the entire Swift team for your work so far and going public yesterday. I’m very glad to see this all happening in a true OSS way :clap:

I’m reaching out on behalf of the CocoaPods team. CocoaPods and the Swift Package Manager will occupy a similar space, and we are now in the process of realigning our roadmap given what we know about the Swift Package Manager.

The Swift Package Manager has had the opportunity to plan its roadmap with knowledge of currently available solutions. We'd welcome a discussion of potential issues where our projects overlap, and together decide what's best for the community.

We already have some questions, but we’ll look into those ourselves first and otherwise send those out as separate emails. We understand you’re probably also very busy in these first few days after the announcement so for now I’ll leave you with best wishes and that we’re looking forward to working together towards best solutions for the community.

Kind regards,
Eloy Durán


(Daniel Dunbar) #2

Hi SPM team,

First of all, major congrats to you and the entire Swift team for your work so far and going public yesterday. I’m very glad to see this all happening in a true OSS way :clap:

Thanks!

I’m reaching out on behalf of the CocoaPods team. CocoaPods and the Swift Package Manager will occupy a similar space, and we are now in the process of realigning our roadmap given what we know about the Swift Package Manager.

Does CocoaPods maintain a roadmap for what features are planned / scheduled?

The Swift Package Manager has had the opportunity to plan its roadmap with knowledge of currently available solutions. We'd welcome a discussion of potential issues where our projects overlap, and together decide what's best for the community.

Absolutely, we would welcome the same. Was there a specific discussion you were hoping to start on this thread?

We already have some questions, but we’ll look into those ourselves first and otherwise send those out as separate emails. We understand you’re probably also very busy in these first few days after the announcement so for now I’ll leave you with best wishes and that we’re looking forward to working together towards best solutions for the community.

Thanks for reaching out, I would love to find ways for all of us to collaborate. I think we all want the same thing -- beautiful, easy, scalable development and a rich ecosystem.

Here are some of my random initial thoughts/questions/ideas from a purely technical perspective:

1. Is CocoaPods technically able to use Swift-based libraries? If there were clear factorings of support functionality that we could build as, say, SPM packages, would CocoaPods be able to use them?

2. The chicken and egg problem is something which makes evaluating design choices very difficult. Informally, the rule that Max and I have often talked about is that "80%+ of projects" should be able to use the convention based system with only the "leading package specification" (let package = Package(...)) and no other custom configuration. However, we can't currently objectively evaluate whether a particular design choice satisfies that criteria without a large body of projects that already work (or almost work) with SPM. I would like to investigate whether we can make use of CocoaPods existing corpus of projects (and infrastructure for testing them) for these kinds of purposes. This would help us quantify things like "this is how much work X percent of projects have to do to adopt SPM" and "X percent of projects need feature Y to work".

3. One of my personal beliefs is that part of the reason software packaging & integration has historically been so messy is because no one has been able to be a "force for change" and cause projects to adapt. For C/C++/Obj-C projects, there are a lot of "unnecessary" variations in the project structure which add complexity for very little overall gain. For example, some things are simpler if you can assume that any package X installs its headers neatly under an "X" subdirectory -- but since historically this was never done, no has ever been able to encourage projects that don't do this to change.

One of my hopes is that, eventually, Swift and SPM support can function as this force for change. If we can come up with a stricter set of requirements that improves our ability to make a very reliable, smooth development experience, then it may be that the value of supporting Swift and SPM is enough to encourage upstream projects to undertake the pain of changing their code base to meet those requirements.

We saw this with Clang once OS adoption (e.g., FreeBSD, Darwin, Linux kernel) began -- initially there was a lot of push back on Clang to support every single GCC-ism that existing projects relied on. However, once it became clear that supporting Clang added value to the project, people started updating older projects to accommodate Clang.

The reason I bring this up in relation to CocoaPods is that I suspect CocoaPods is currently in the camp of "we have a large body of existing clients and cannot aggressively try and simplify our model if it means they need to change", whereas for SPM we are much more likely to be in the camp of "we would like to require all projects to behave this way, even if it means many existing projects will need to be adapted, because we think it will allow us to build a better experience in the long run".

What is your perspective on this problem? How big of an issue has this been for CocoaPods to evolve?

- Daniel

···

On Dec 6, 2015, at 4:18 AM, Eloy Durán via swift-build-dev <swift-build-dev@swift.org> wrote:

Kind regards,
Eloy Durán
_______________________________________________
swift-build-dev mailing list
swift-build-dev@swift.org
https://lists.swift.org/mailman/listinfo/swift-build-dev


(Eloy Duran) #3

Does CocoaPods maintain a roadmap for what features are planned / scheduled?

The scheduled ones at the moment are in the 1.0.0 milestone https://github.com/CocoaPods/CocoaPods/milestones/1.0.0

Absolutely, we would welcome the same. Was there a specific discussion you were hoping to start on this thread?

For now just the exact one that we’re now having, you have some great questions already.

Thanks for reaching out, I would love to find ways for all of us to collaborate. I think we all want the same thing -- beautiful, easy, scalable development and a rich ecosystem.

Definitely.

Here are some of my random initial thoughts/questions/ideas from a purely technical perspective:

1. Is CocoaPods technically able to use Swift-based libraries? If there were clear factorings of support functionality that we could build as, say, SPM packages, would CocoaPods be able to use them?

Yes, it does support them right now. Here’s an example lib of one of our core team members (Kyle Fuller): https://github.com/kylef/stencil and its specification https://github.com/kylef/Stencil/blob/master/Stencil.podspec.json.

One of our other core team members (Boris Bügling) has already started on a Swift tool to generate CocoaPods specifications from SPM specifications: https://github.com/neonichu/Chocolat. Some metadata that we deem important, at least in our centralised service, is missing from SPM specifications: https://github.com/neonichu/Chocolat/issues/2. One of those is a license, which you already mention in your docs as something you’d be interested in looking into, the other is a description of the library, which you might also need if/when you’d ever build an index, adding these sooner rather than later would make it much easier for Boris’ tool to work perfect.

I’ll let Boris do the speaking about what we’d really need here and to answer further questions on this front.

2. The chicken and egg problem is something which makes evaluating design choices very difficult. Informally, the rule that Max and I have often talked about is that "80%+ of projects" should be able to use the convention based system with only the "leading package specification" (let package = Package(...)) and no other custom configuration. However, we can't currently objectively evaluate whether a particular design choice satisfies that criteria without a large body of projects that already work (or almost work) with SPM. I would like to investigate whether we can make use of CocoaPods existing corpus of projects (and infrastructure for testing them) for these kinds of purposes. This would help us quantify things like "this is how much work X percent of projects have to do to adopt SPM" and "X percent of projects need feature Y to work”.

We have quite some data on the thousands of libraries that we currently support and even stats on integration which might be useful to you. I’ll leave it to yet another of our team members (Orta Therox) to give you access to that data and show you around.

3. One of my personal beliefs is that part of the reason software packaging & integration has historically been so messy is because no one has been able to be a "force for change" and cause projects to adapt. For C/C++/Obj-C projects, there are a lot of "unnecessary" variations in the project structure which add complexity for very little overall gain. For example, some things are simpler if you can assume that any package X installs its headers neatly under an "X" subdirectory -- but since historically this was never done, no has ever been able to encourage projects that don't do this to change.

One of my hopes is that, eventually, Swift and SPM support can function as this force for change. If we can come up with a stricter set of requirements that improves our ability to make a very reliable, smooth development experience, then it may be that the value of supporting Swift and SPM is enough to encourage upstream projects to undertake the pain of changing their code base to meet those requirements.

I think that it will be quite easy for you to go forward with the strict structure approach and you should totally do so. Swift is still new, the people are excited to work on their projects and are already used to having to update them as Swift matured, and last but not least, you are Apple :slight_smile: Many of the people that did not want to use CocoaPods would not do so because it’s not how Apple did things and some of those were definitely technically valid as we couldn’t integrate as nicely with Xcode as Apple ever could.

Having said that, I think this really only applies to Swift libraries. Existing Objective-C libraries might be doable if the author is using their lib in Swift or otherwise interested in overhauling their project’s layout, but C/C++ projects will prove hard unless Swift users are going to want to go the extra mile to overhaul project layouts for those, but these can get very hard to get right. Right now you’re making the choice to tell people to install those differently (e.g. homebrew), but that makes integration a bit harder for users, especially for a platform like iOS. And that’s not even considering the issue of how to link those dependencies into an app.

The reason I bring this up in relation to CocoaPods is that I suspect CocoaPods is currently in the camp of "we have a large body of existing clients and cannot aggressively try and simplify our model if it means they need to change", whereas for SPM we are much more likely to be in the camp of "we would like to require all projects to behave this way, even if it means many existing projects will need to be adapted, because we think it will allow us to build a better experience in the long run".

What is your perspective on this problem? How big of an issue has this been for CocoaPods to evolve?

You’re absolutely right about these problems, they have been very real. When we started on CocoaPods, we were entering into a long established ecosystem and we too had a chicken and egg problem. There was no way that people would adopt their existing projects to CocoaPods, so it had to basically work with all of them. I absolutely love convention over configuration, but this definitely made that very hard to do so. The hardest part has always been projects that use Xcode projects:

* Because these documents are too opaque to properly reflect upon and e.g. ensure a project doesn’t vendor a third party lib that could lead to clashes or what other libs and the versions thereof the project actually depends on. This was especially hard with libraries where the one maintaining the CocoaPods specification wasn’t the lib’s author and wasn’t always inclined to do their absolute best to ensure the specifications validity for everybody. However, over time people did start going CocoaPods first with their libraries, which made this problem largely go away and was a big win in transparency of what dependencies you were using imo.

* Integrating the libs into the user’s project in a way that made the most sense from the way people were doing development with Xcode, which are including the source so that they are always fully navigable and debuggable, meant we had to generate Xcode projects ourselves. We have never been happy with this, because the undocumented/unsupported nature of it tends to lead to a lot of friction. I’m hoping that the way SPM is going to integrate into Xcode might also open up a possibility for CocoaPods to integrate artefacts without having to deal with Xcode details or possibly by using an Xcode specific simple, documented, and supported specification on how to integrate artefacts.

I hope I addressed all your questions to the full extent, please do let me know otherwise.

– Eloy


(Paul Young) #4

Glad to hear about the potential for collaboration here!

Daniel – I wanted to support your comments by saying that when Carthage shipped, it required projects to have build schemes defined (and shared). This highlighted projects which _didn’t_ have one or both of those things, and it’s my understanding that it prompted updates along those lines.

My perception of this is that:

a) people hadn’t created (shared) schemes because they didn’t know about or understand them, rather than purposefully avoiding them

b) people were happy to make the changes because the benefit outweighed the effort

c) this was often referred to as “adding Carthage support” but was really just adding standard Xcode behavior

d) projects were more accessible outside of use with Carthage as a result

All this to say – I think having a preferred way of structuring projects would be welcomed, despite some instability in the beginning.

···

On Sun, Dec 6, 2015 at 7:32 PM, Daniel Dunbar via swift-build-dev <swift-build-dev@swift.org> wrote:

On Dec 6, 2015, at 4:18 AM, Eloy Durán via swift-build-dev <swift-build-dev@swift.org> wrote:

Hi SPM team,

First of all, major congrats to you and the entire Swift team for your work so far and going public yesterday. I’m very glad to see this all happening in a true OSS way :clap:

Thanks!

I’m reaching out on behalf of the CocoaPods team. CocoaPods and the Swift Package Manager will occupy a similar space, and we are now in the process of realigning our roadmap given what we know about the Swift Package Manager.

Does CocoaPods maintain a roadmap for what features are planned / scheduled?

The Swift Package Manager has had the opportunity to plan its roadmap with knowledge of currently available solutions. We'd welcome a discussion of potential issues where our projects overlap, and together decide what's best for the community.

Absolutely, we would welcome the same. Was there a specific discussion you were hoping to start on this thread?

We already have some questions, but we’ll look into those ourselves first and otherwise send those out as separate emails. We understand you’re probably also very busy in these first few days after the announcement so for now I’ll leave you with best wishes and that we’re looking forward to working together towards best solutions for the community.

Thanks for reaching out, I would love to find ways for all of us to collaborate. I think we all want the same thing -- beautiful, easy, scalable development and a rich ecosystem.
Here are some of my random initial thoughts/questions/ideas from a purely technical perspective:
1. Is CocoaPods technically able to use Swift-based libraries? If there were clear factorings of support functionality that we could build as, say, SPM packages, would CocoaPods be able to use them?
2. The chicken and egg problem is something which makes evaluating design choices very difficult. Informally, the rule that Max and I have often talked about is that "80%+ of projects" should be able to use the convention based system with only the "leading package specification" (let package = Package(...)) and no other custom configuration. However, we can't currently objectively evaluate whether a particular design choice satisfies that criteria without a large body of projects that already work (or almost work) with SPM. I would like to investigate whether we can make use of CocoaPods existing corpus of projects (and infrastructure for testing them) for these kinds of purposes. This would help us quantify things like "this is how much work X percent of projects have to do to adopt SPM" and "X percent of projects need feature Y to work".
3. One of my personal beliefs is that part of the reason software packaging & integration has historically been so messy is because no one has been able to be a "force for change" and cause projects to adapt. For C/C++/Obj-C projects, there are a lot of "unnecessary" variations in the project structure which add complexity for very little overall gain. For example, some things are simpler if you can assume that any package X installs its headers neatly under an "X" subdirectory -- but since historically this was never done, no has ever been able to encourage projects that don't do this to change.
One of my hopes is that, eventually, Swift and SPM support can function as this force for change. If we can come up with a stricter set of requirements that improves our ability to make a very reliable, smooth development experience, then it may be that the value of supporting Swift and SPM is enough to encourage upstream projects to undertake the pain of changing their code base to meet those requirements.
We saw this with Clang once OS adoption (e.g., FreeBSD, Darwin, Linux kernel) began -- initially there was a lot of push back on Clang to support every single GCC-ism that existing projects relied on. However, once it became clear that supporting Clang added value to the project, people started updating older projects to accommodate Clang.
The reason I bring this up in relation to CocoaPods is that I suspect CocoaPods is currently in the camp of "we have a large body of existing clients and cannot aggressively try and simplify our model if it means they need to change", whereas for SPM we are much more likely to be in the camp of "we would like to require all projects to behave this way, even if it means many existing projects will need to be adapted, because we think it will allow us to build a better experience in the long run".
What is your perspective on this problem? How big of an issue has this been for CocoaPods to evolve?
- Daniel

Kind regards,
Eloy Durán
_______________________________________________
swift-build-dev mailing list
swift-build-dev@swift.org
https://lists.swift.org/mailman/listinfo/swift-build-dev


(Daniel Dunbar) #5

Does CocoaPods maintain a roadmap for what features are planned / scheduled?

The scheduled ones at the moment are in the 1.0.0 milestone https://github.com/CocoaPods/CocoaPods/milestones/1.0.0

Cool, thanks.

Absolutely, we would welcome the same. Was there a specific discussion you were hoping to start on this thread?

For now just the exact one that we’re now having, you have some great questions already.

Thanks for reaching out, I would love to find ways for all of us to collaborate. I think we all want the same thing -- beautiful, easy, scalable development and a rich ecosystem.

Definitely.

Here are some of my random initial thoughts/questions/ideas from a purely technical perspective:

1. Is CocoaPods technically able to use Swift-based libraries? If there were clear factorings of support functionality that we could build as, say, SPM packages, would CocoaPods be able to use them?

Yes, it does support them right now. Here’s an example lib of one of our core team members (Kyle Fuller): https://github.com/kylef/stencil and its specification https://github.com/kylef/Stencil/blob/master/Stencil.podspec.json.

Sorry, I wasn't clear here.

What I meant was is the *implementation* of CocoaPods actually able to use Swift libraries. I'm wondering if it would ever be viable to factor out shared libraries that both projects used, but I can imagine that using (and installing) those libraries from the Ruby based CocoaPods might be non-trivial?

One of our other core team members (Boris Bügling) has already started on a Swift tool to generate CocoaPods specifications from SPM specifications: https://github.com/neonichu/Chocolat. Some metadata that we deem important, at least in our centralised service, is missing from SPM specifications: https://github.com/neonichu/Chocolat/issues/2. One of those is a license, which you already mention in your docs as something you’d be interested in looking into, the other is a description of the library, which you might also need if/when you’d ever build an index, adding these sooner rather than later would make it much easier for Boris’ tool to work perfect.

I think both of those are things it makes sense to add soon (and encourage people to author).

I’ll let Boris do the speaking about what we’d really need here and to answer further questions on this front.

2. The chicken and egg problem is something which makes evaluating design choices very difficult. Informally, the rule that Max and I have often talked about is that "80%+ of projects" should be able to use the convention based system with only the "leading package specification" (let package = Package(...)) and no other custom configuration. However, we can't currently objectively evaluate whether a particular design choice satisfies that criteria without a large body of projects that already work (or almost work) with SPM. I would like to investigate whether we can make use of CocoaPods existing corpus of projects (and infrastructure for testing them) for these kinds of purposes. This would help us quantify things like "this is how much work X percent of projects have to do to adopt SPM" and "X percent of projects need feature Y to work”.

We have quite some data on the thousands of libraries that we currently support and even stats on integration which might be useful to you. I’ll leave it to yet another of our team members (Orta Therox) to give you access to that data and show you around.

Ok, great! Do you have existing infrastructure for analysis of those libraries? Would it be easy to write a tool to, e.g., count #s and types of libraries? Or create a summary of source layout styles?

3. One of my personal beliefs is that part of the reason software packaging & integration has historically been so messy is because no one has been able to be a "force for change" and cause projects to adapt. For C/C++/Obj-C projects, there are a lot of "unnecessary" variations in the project structure which add complexity for very little overall gain. For example, some things are simpler if you can assume that any package X installs its headers neatly under an "X" subdirectory -- but since historically this was never done, no has ever been able to encourage projects that don't do this to change.

One of my hopes is that, eventually, Swift and SPM support can function as this force for change. If we can come up with a stricter set of requirements that improves our ability to make a very reliable, smooth development experience, then it may be that the value of supporting Swift and SPM is enough to encourage upstream projects to undertake the pain of changing their code base to meet those requirements.

I think that it will be quite easy for you to go forward with the strict structure approach and you should totally do so. Swift is still new, the people are excited to work on their projects and are already used to having to update them as Swift matured, and last but not least, you are Apple :slight_smile: Many of the people that did not want to use CocoaPods would not do so because it’s not how Apple did things and some of those were definitely technically valid as we couldn’t integrate as nicely with Xcode as Apple ever could.

Having said that, I think this really only applies to Swift libraries. Existing Objective-C libraries might be doable if the author is using their lib in Swift or otherwise interested in overhauling their project’s layout, but C/C++ projects will prove hard unless Swift users are going to want to go the extra mile to overhaul project layouts for those, but these can get very hard to get right. Right now you’re making the choice to tell people to install those differently (e.g. homebrew), but that makes integration a bit harder for users, especially for a platform like iOS. And that’s not even considering the issue of how to link those dependencies into an app.

Right. We do want to support C-family projects natively (not just system dependencies) in the long term. As you know much of the Swift project itself is in C++; we are pretty highly motivated to support this just to have a better development / build model for ourselves.

I hope to eventually be able to apply leverage on C-family projects, too. There is still no good way of Integrating C-family projects into one another; anyone who has tried to write a small tool that uses LLVM will know exactly what I am talking about. In some ways, this situation is actually even worse than for Swift projects because of the diversity and complexity of C-family project build systems. I believe that if we could support some strong conventions for C/C++ projects, to the point that it was suddenly easy to build dependencies together, then we would have a very compelling system that project owners would actually be excited to support.

One evolutionary approach I have considered is that we could, for C-family projects, provide built in tools to define adaptor packages which integrate non-conforming projects into the system. This is analogous to how we support module-map packages today, even though in an ideal world projects would ship module definitions with their project. This would allow the ecosystem to build up, and people would begin to see the value of it. At that point, the owners of the adaptor packages could upstream their work into the project itself. And then we would all live in a better world! :slight_smile:

The reason I bring this up in relation to CocoaPods is that I suspect CocoaPods is currently in the camp of "we have a large body of existing clients and cannot aggressively try and simplify our model if it means they need to change", whereas for SPM we are much more likely to be in the camp of "we would like to require all projects to behave this way, even if it means many existing projects will need to be adapted, because we think it will allow us to build a better experience in the long run".

What is your perspective on this problem? How big of an issue has this been for CocoaPods to evolve?

You’re absolutely right about these problems, they have been very real. When we started on CocoaPods, we were entering into a long established ecosystem and we too had a chicken and egg problem. There was no way that people would adopt their existing projects to CocoaPods, so it had to basically work with all of them. I absolutely love convention over configuration, but this definitely made that very hard to do so. The hardest part has always been projects that use Xcode projects:

* Because these documents are too opaque to properly reflect upon and e.g. ensure a project doesn’t vendor a third party lib that could lead to clashes or what other libs and the versions thereof the project actually depends on. This was especially hard with libraries where the one maintaining the CocoaPods specification wasn’t the lib’s author and wasn’t always inclined to do their absolute best to ensure the specifications validity for everybody. However, over time people did start going CocoaPods first with their libraries, which made this problem largely go away and was a big win in transparency of what dependencies you were using imo.

Yup. One of our very explicit goals is to minimize the the amount of stuff that is "opaque" to swiftpm, so that we at least have an open road to very aggressively auditing what a package does and assisting users in writing the definition correctly. There is a tension between this goal and quickly supporting all of the things people want to do (especially w.r.t. C-family projects), but I think it is important that we get this right.

* Integrating the libs into the user’s project in a way that made the most sense from the way people were doing development with Xcode, which are including the source so that they are always fully navigable and debuggable, meant we had to generate Xcode projects ourselves. We have never been happy with this, because the undocumented/unsupported nature of it tends to lead to a lot of friction. I’m hoping that the way SPM is going to integrate into Xcode might also open up a possibility for CocoaPods to integrate artefacts without having to deal with Xcode details or possibly by using an Xcode specific simple, documented, and supported specification on how to integrate artifacts.

I understand the tension of needing to generate projects and the downsides. I have a lot of experience working with LLVM's auto generated (via CMake) Xcode project, and it is just very difficult to build a seamless experience when adapting two very different project models this way.

We are trying something new here, which is to build a package manager that exposes an API, and then rely on clients to build good integration by using that API well. TBH this is a bit of an experiment, but I believe based on our experience with libclang/SourceKit that it is absolutely the right way to go and will let us do something great. Of course, this is all currently TBD. We don't yet vend a formal library much less any real APIs, so this plan will become more fully formed in the coming months...

I hope I addressed all your questions to the full extent, please do let me know otherwise.

One of the specific things I was curious about was how CocoaPods has historically handled changes to the podspec definition format? How does this work if a podspec and its dependencies start requiring incompatible versions of CocoaPods itself? Have you just managed to avoid this, or are there legacy features you have ended up supporting but wish you didn't need to? I see things like issue #840 which I assume reflects some of how this worked out in practice.

- Daniel

···

On Dec 7, 2015, at 5:22 AM, Eloy Durán via swift-build-dev <swift-build-dev@swift.org> wrote:

– Eloy

_______________________________________________
swift-build-dev mailing list
swift-build-dev@swift.org
https://lists.swift.org/mailman/listinfo/swift-build-dev


(Max Howell) #6

One of our other core team members (Boris Bügling) has already started on a Swift tool to generate CocoaPods specifications from SPM specifications: https://github.com/neonichu/Chocolat. Some metadata that we deem important, at least in our centralised service, is missing from SPM specifications: https://github.com/neonichu/Chocolat/issues/2. One of those is a license, which you already mention in your docs as something you’d be interested in looking into, the other is a description of the library, which you might also need if/when you’d ever build an index, adding these sooner rather than later would make it much easier for Boris’ tool to work perfect.

I intend to have swift-build output a file list as an optional instantiation. That and a target list. Presumably things like that would help here so I’ll try and knock it out some evening.

WRT to licensing, I think we intend to move pretty quickly on adding this aspect to Package.swift.


(orta) #7

We have quite some data on the thousands of libraries that we currently support and even stats on integration which might be useful to you. I’ll leave it to yet another of our team members (Orta Therox) to give you access to that data and show you around.

I’ve exported all of the metadata with all our current knowledge about libraries in the Cocoa ecosystem.

https://docs.google.com/spreadsheets/d/1HnfzWRtNQW1cwY3CJZiwAbTb5VSNJC90rIYUzZHcQQQ/edit#gid=1479796730&vpid=A1

I don’t necessarily expect any of this to be essential in terms of decisions making, but I think that this can be useful to understand things like: licensing, language shifts, file sizes etc. We use these values for generating our Quality Estimates, which do some of the things you want to do in the future.

https://guides.cocoapods.org/making/quality-indexes.html

We use this for providing ways in which libraries can improve after they’ve been shipped, by basing the search ranking off this number.

- ./orta


(orta) #8

I have a local machine with a download of every pod, from about 3 months ago. I can probably script this sort of thing, happy to take this one to direct mail if you have an idea of what you want.

Given that paths are, generally speaking, added in the Podspec, this can also be generated from the CocoaPods/Specs repo itself too.

···

On Dec 7, 2015, at 12:30 PM, Daniel Dunbar via swift-build-dev <swift-build-dev@swift.org> wrote:

Ok, great! Do you have existing infrastructure for analysis of those libraries? Would it be easy to write a tool to, e.g., count #s and types of libraries? Or create a summary of source layout styles?


(Eloy Duran) #9

Sorry, I wasn't clear here.

What I meant was is the *implementation* of CocoaPods actually able to use Swift libraries. I'm wondering if it would ever be viable to factor out shared libraries that both projects used, but I can imagine that using (and installing) those libraries from the Ruby based CocoaPods might be non-trivial?

Ah, I see. It would involve some work, but I’m sure it’s something we can overcome, when the benefit of sharing said components outweighs the complexity of writing some CRuby/Swift bindings. Right now I’m not sure which components that would be –maybe the Xcode integration part in the future?–, but we’ll keep this possibility in mind as we go forward.

I think both of those are things it makes sense to add soon (and encourage people to author).

Ace. I see you already commented on that Chocolat ticket, will answer there.

One evolutionary approach I have considered is that we could, for C-family projects, provide built in tools to define adaptor packages which integrate non-conforming projects into the system. This is analogous to how we support module-map packages today, even though in an ideal world projects would ship module definitions with their project. This would allow the ecosystem to build up, and people would begin to see the value of it. At that point, the owners of the adaptor packages could upstream their work into the project itself. And then we would all live in a better world! :slight_smile:

Oooh, this sounds good. I’d love to hear more about this when.if you have more details. It sounds like this might be an interesting place where we can provide an adapter that would make libraries covered by CP only to be usable by SPM while they are not natively supported yet, or ever.

One of the specific things I was curious about was how CocoaPods has historically handled changes to the podspec definition format?

- A lot of build options would initially be set through a simple `xcconfig` dictionary and a `compiler_flags` list.

- A big reason for me to choose Ruby for CP was that, because of Ruby’s nature:
  * lib authors could amend the way a library was integrated with a Ruby post-install hook in their lib’s podspec that had complete access to the library integration code
  * users could basically monkey-patch the complete system from their Podfile (the root spec, shall we say)
  This was an often recommended (we’d give them code suggestions of how/what to patch) and ask them (both lib authors and end-users) to report back to us with the solutions they came up with.

In both cases we’d use the data that we received, about what people actually needed, to decide what the podspec definition should look like. Because of this there haven’t been many podspec additions that we felt we needed to revert.

Obviously this complete freedom comes at a price and now that we pretty much have everything in the format that we need we’ve started locking that down. Using the `xcconfig` dictionary is discouraged and the post-install hooks have been replaced from the podspec format for a shell command: https://guides.cocoapods.org/syntax/podspec.html#prepare_command

(post-install hooks are still a supported and valuable thing in the Podfile format, though. The end-user should be able to do whatever they need.)

How does this work if a podspec and its dependencies start requiring incompatible versions of CocoaPods itself?

A ‘spec repo’ may contain a file that indicates what the minimal CP version that is required to work with the specs in that spec repo. The CLI tool simply stops working and informs the user to upgrade if their version is older than the minimal required version for that spec repo.

For example, the minimal required version for the public central spec repo is 0.32.1 (over a year old): https://github.com/CocoaPods/Specs/blob/master/CocoaPods-version.yml.

But as we use a simple DB for our index (FS, probably in a SCM repo) users that really can’t upgrade their CP version yet, or don’t want to yet because of other time constraints, can easily use an older tree / branch. (This is something that people, specifically larger companies, have definitely done in the past.) Because of this flexibility, we were free to make breaking changes when we really deemed them necessary.

Have you just managed to avoid this, or are there legacy features you have ended up supporting but wish you didn't need to? I see things like issue #840 which I assume reflects some of how this worked out in practice.

* The Podfile format is indeed something that in hindsight is very clear should have been the way as described in #840 from the get-go, but it’s also going to be pretty much the only real big breaking change we’ll be making, so I don’t see it as a big problem. People have still been able to build their complex multi-platform projects, this API change would be a one time major clean-up of everything we now know people require.

* Static libraries with resources were a major pain, because the way the lib code often referenced them (through the NSBundle API) meant that all the resources would have to go into the main app bundle, which could easily lead to duplicate filenames. We ended up adding functionality to create ‘resource bundles’ and over time lib authors started using this pattern more and more, but the complexity we needed to deal with for this is something I really wish we didn’t needed to deal with. https://guides.cocoapods.org/syntax/podspec.html#resource_bundles. We had hoped that dynamic frameworks becoming available on iOS would finally solve all such woes, but alas those tend to load way too slow (https://github.com/artsy/eigen/issues/586). Have you given any thought yet to how you’d like to deal with resources and embedding those in a bundle?

* Subspecs are something that I really don’t like and wish we didn’t need to support. https://guides.cocoapods.org/syntax/podspec.html#subspec. It was pretty much only added to support the monolithic libraries that were being build, especially from the pre-iOS era. I don’t like having source that I don’t use in my app, which includes dependencies of such code. So the only reason we saw to deal with that was to allow a CP user to specify what subsets of the monolithic library they wanted to use.

  Subspecs make some implementation more complex than we’d like and I personally prefer the approach of micro libs and a spec for each, *but* I should add that many in the community actually really like using subspecs.

* Another thing that I remember that I was very glad we got rid of was determining platform specific settings at spec evaluation time, which made specifications very hard to use in environments where users were targeting multiple platforms, let alone once more platforms were being introduced. i.e. you used to be able to do something like:

    common_files = ‘Common/..'
    if config.osx?
      spec.source_files = common_files + ‘OSX/…'
    else
      spec.source_files = common_files + ‘iOS/…’
    end

  Nowadays you specify them all and their platform scope:

    spec.source_files = ‘Common/…’
    spec.osx.source_files = ‘OSX/...’
    spec.ios.source_files = ‘iOS/...'

– Eloy


(Daniel Dunbar) #10

Circling back on this...

Ok, great! Do you have existing infrastructure for analysis of those libraries? Would it be easy to write a tool to, e.g., count #s and types of libraries? Or create a summary of source layout styles?

I have a local machine with a download of every pod, from about 3 months ago. I can probably script this sort of thing, happy to take this one to direct mail if you have an idea of what you want.

Given that paths are, generally speaking, added in the Podspec, this can also be generated from the CocoaPods/Specs repo itself too.

The high level question we would want to answer would be "how much work is it for people to conform to convention X". I could envision trying to automate that by taking the corpus of pods, applying some amount of automated transformations based on the known or proposed conventions and the Podspec, and then seeing if the project builds. If it wasn't too much work to build such a system then it could be invaluable in vetting convention proposals, but I'm not currently familiar enough with the ecosystem to know if this is viable or not.

If you think it is, that is totally something I would consider taking off list if/when I had some time to hack on it...

- Daniel

···

On Dec 7, 2015, at 10:23 AM, orta therox via swift-build-dev <swift-build-dev@swift.org> wrote:

On Dec 7, 2015, at 12:30 PM, Daniel Dunbar via swift-build-dev <swift-build-dev@swift.org <mailto:swift-build-dev@swift.org>> wrote:

_______________________________________________
swift-build-dev mailing list
swift-build-dev@swift.org
https://lists.swift.org/mailman/listinfo/swift-build-dev


(orta) #11

Interesting, so this is feasible, once a pod is downloaded a then Podspec can be ran against it to get access to all of the files. I think an app that could read the Podspec json and perform some transformation is feasible for most swift pods.

We’ve already got tools for converting a CocoaPods project to work with the Package.swift <https://github.com/neonichu/schoutedenapus> which you can take some inspiration from.

···

--

[A.] Orta Therox

w/ Artsy <http://artsy.net/>CocoaPods <http://cocoapods.org/> / CocoaDocs <http://cocoadocs.org/> / GIFs.app <https://itunes.apple.com/us/app/gifs/id961850017?l=en&mt=12>
@orta <http://twitter.com/orta> / orta.github.com <http://orta.github.com/>
Artsy is totally hiring iOS Devs <https://artsy.net/job/mobile-engineer> ATM

On 12 Dec 2015, at 12:10, Daniel Dunbar <daniel_dunbar@apple.com> wrote:

Circling back on this...

On Dec 7, 2015, at 10:23 AM, orta therox via swift-build-dev <swift-build-dev@swift.org <mailto:swift-build-dev@swift.org>> wrote:

On Dec 7, 2015, at 12:30 PM, Daniel Dunbar via swift-build-dev <swift-build-dev@swift.org <mailto:swift-build-dev@swift.org>> wrote:

Ok, great! Do you have existing infrastructure for analysis of those libraries? Would it be easy to write a tool to, e.g., count #s and types of libraries? Or create a summary of source layout styles?

I have a local machine with a download of every pod, from about 3 months ago. I can probably script this sort of thing, happy to take this one to direct mail if you have an idea of what you want.

Given that paths are, generally speaking, added in the Podspec, this can also be generated from the CocoaPods/Specs repo itself too.

The high level question we would want to answer would be "how much work is it for people to conform to convention X". I could envision trying to automate that by taking the corpus of pods, applying some amount of automated transformations based on the known or proposed conventions and the Podspec, and then seeing if the project builds. If it wasn't too much work to build such a system then it could be invaluable in vetting convention proposals, but I'm not currently familiar enough with the ecosystem to know if this is viable or not.

If you think it is, that is totally something I would consider taking off list if/when I had some time to hack on it...

- Daniel

_______________________________________________
swift-build-dev mailing list
swift-build-dev@swift.org <mailto:swift-build-dev@swift.org>
https://lists.swift.org/mailman/listinfo/swift-build-dev


(orta) #12

Thinking about this a bit further, performing an automated transition from CocoaPods -> SPM would be feasible given all the metadata is kept in a file, but you’d probably end up breaking carthage support for those libraries. It would also need to update the Xcode projects that may be connected to the source files, which I think would turn into a bit more trouble. Likely could be scripted with the xcodeproj gem though.

···

--

[A.] Orta Therox

w/ Artsy <http://artsy.net/>CocoaPods <http://cocoapods.org/> / CocoaDocs <http://cocoadocs.org/> / GIFs.app <https://itunes.apple.com/us/app/gifs/id961850017?l=en&mt=12>
@orta <http://twitter.com/orta> / orta.github.com <http://orta.github.com/>
Artsy is totally hiring iOS Devs <https://artsy.net/job/mobile-engineer> ATM

On 18 Dec 2015, at 09:16, orta therox <orta.therox@gmail.com> wrote:

Interesting, so this is feasible, once a pod is downloaded a then Podspec can be ran against it to get access to all of the files. I think an app that could read the Podspec json and perform some transformation is feasible for most swift pods.

We’ve already got tools for converting a CocoaPods project to work with the Package.swift <https://github.com/neonichu/schoutedenapus> which you can take some inspiration from.

--

[A.] Orta Therox

w/ Artsy <http://artsy.net/>CocoaPods <http://cocoapods.org/> / CocoaDocs <http://cocoadocs.org/> / GIFs.app <https://itunes.apple.com/us/app/gifs/id961850017?l=en&mt=12>
@orta <http://twitter.com/orta> / orta.github.com <http://orta.github.com/>
Artsy is totally hiring iOS Devs <https://artsy.net/job/mobile-engineer> ATM

On 12 Dec 2015, at 12:10, Daniel Dunbar <daniel_dunbar@apple.com <mailto:daniel_dunbar@apple.com>> wrote:

Circling back on this...

On Dec 7, 2015, at 10:23 AM, orta therox via swift-build-dev <swift-build-dev@swift.org <mailto:swift-build-dev@swift.org>> wrote:

On Dec 7, 2015, at 12:30 PM, Daniel Dunbar via swift-build-dev <swift-build-dev@swift.org <mailto:swift-build-dev@swift.org>> wrote:

Ok, great! Do you have existing infrastructure for analysis of those libraries? Would it be easy to write a tool to, e.g., count #s and types of libraries? Or create a summary of source layout styles?

I have a local machine with a download of every pod, from about 3 months ago. I can probably script this sort of thing, happy to take this one to direct mail if you have an idea of what you want.

Given that paths are, generally speaking, added in the Podspec, this can also be generated from the CocoaPods/Specs repo itself too.

The high level question we would want to answer would be "how much work is it for people to conform to convention X". I could envision trying to automate that by taking the corpus of pods, applying some amount of automated transformations based on the known or proposed conventions and the Podspec, and then seeing if the project builds. If it wasn't too much work to build such a system then it could be invaluable in vetting convention proposals, but I'm not currently familiar enough with the ecosystem to know if this is viable or not.

If you think it is, that is totally something I would consider taking off list if/when I had some time to hack on it...

- Daniel

_______________________________________________
swift-build-dev mailing list
swift-build-dev@swift.org <mailto:swift-build-dev@swift.org>
https://lists.swift.org/mailman/listinfo/swift-build-dev