Thank you for confirming that using -g works. Adding all this instrumentation has decreased the frequency of the crash. I'm running some experimental reproducers. I will let you know when I find something promising.
Currently, reporting transmits over to Ubuntu's setup, but it would be nice if we could tap in the kernel infrastructure it has for local reporting.
Do you mean /proc/sys/kernel/core_pattern? That unfortunately doesn't seem to be namespaced in containers and it only available to be changed for privileged containers:
Therefore I don't think core_pattern can really be used for a Swift on Server story as it modifies the actual kernel state (shared across everything) and not the container. So this would only be usable in environments where we have real virtualisation.
Well, bummer. I guess this would require upstream work to change (probably with a side-by-side namespaced version or similar kernel changes.)
I’m at a loss at the moment to suggest different ways to go about this. I still strongly want symbolicated backtraces for all crashes I need to investigate, and in many cases I cannot install a separate package at that point (because, say, at the time my code builds swiftpm hasn’t been built yet, for one.) Maybe it could be a library with a constructor whose code we ship as part of, say, apple/swift, that gets linked on demand by the appropriate projects?
Yup, I think that's probably what it'll have to be if we think we can't rely on SwiftPM packages. I reckon we could bootstrap it (as Ian and others are) as SwiftPM package just because it's easier and faster to iterate and then when we're ready integrate it into apple/swift.
Even after a potential move into apple/swift, the SwiftPM package could be kept for further improvements or extended functionality. Once we have a built-in mechanism we could also have a kill switch for it in case there is a 'more advanced' version delivered in an alternative way.
1 Like
I reckon we could bootstrap it (as Ian and others are) as SwiftPM package just because it's easier and faster to iterate and then when we're ready integrate it into
apple/swift.
exactly
Over the last two weeks turning on Thread Sanitizer has gradually exposed several race conditions (in the vicinity of this crash) on Linux which never showed up on OSX. Most of these were benign but last night I got one which definitely could bite. For working this particular issue, Thread Sanitizer has been much more useful than stack traces. I haven't seen a crash in two weeks, but this is most likely a side effect of the additional instrumentation which is now going into the build. However, with this morning's fix I may have fixed the (or at least a) cause of the crashes.