Let's say I'm designing a language feature/transformation that involves some degree of full/partial language coverage.
.swiftmodulemodule interface files need to work (round-trip printing and parsing) for all Swift modules. @harlanhaskins @jrose
- A new SIL transform needs to correctly handle all potential SIL programs without crashing or miscompiling (or breaking OSSA). @Michael_Gottesman @Andrew_Trick
- Differentiating function bodies (as part of differentiable programming in Swift) needs to work for all supported Swift function bodies (represented in syntax trees and the AST, and eventually SIL and LLVM IR). @rxwei
How do I test language coverage?
I can do my best, find bugs, and check in unit tests preventing regression. Or I can have some foresight and preemptively think of untested buggy edge cases, and check in unit tests. Or I can add integration tests based on popular Swift community packages. Or I can...
There are many technology options to solve this testing use case. Which is best? Example criteria:
- Flexibility and expressivity (what can I do with this tech)
- Practicality (effort of implementing and maintaining it)
- Maintainability (modularity, separation of concerns, is it written as a SwiftPM package)
Right now, we have the following technologies:
- Swift Source Compatibility Suite: highly pragmatic and maintainable. Not expressive, limited to regression testing (only works with existing code, doesn't preemptively discover any bugs for new code).
- Swift Evolve: pragmatic and maintainable. Very limited expressivity so far.
- Naive fuzzing. See @practicalswift and their fuzzing tests - it mostly catches garbage Swift code that crashes the parser or name lookup.
I like property-based testing, at least for use case (3) above. @marcrasi and @shabalin implemented variants of generators of random Swift code, for different purposes: autodiff correctness (Marc) and autodiff performance for different language features (Denys).
- Marc's work: Randomly-generated Tests for Swift AD
- Denys' work: generate a simple Python AST and generate Swift code from it with specific language features enabled. (not yet open-sourced, ask us if you're interested)
Are there good property-based testing libraries suitable for generating simple ASTs, lower-able to Swift? SwiftCheck is hefty, I think it could be distilled to a simpler core with less abstraction and fewer cute syntax. @codafi
Any other solution ideas that hit a criteria-satisfaction sweet spot?