Assertion failed: (isa<X>(Val) && "cast<Ty>() argument of incompatible type!") from clang when trying to compile with an unusual target

Development Compiler
1

When i'm compiling swift code to LLVM IR using my slightly patched version of swift 4.2 with the unusual target triple avr-atmel-linux-gnueabihf, I'm getting a trap within the constructor for swift::irgen::IRGenModule.

Specifically, the last line, IsSwiftErrorInRegister = clang::CodeGen::swiftcall::isSwiftErrorLoweredInRegister( ClangCodeGen->CGM());

Clang source code shows this method is simply...

// Is swifterror lowered to a register by the target ABI.
bool swiftcall::isSwiftErrorLoweredInRegister(CodeGenModule &CGM) {
  return getSwiftABIInfo(CGM).isSwiftErrorInRegister();
}

getSwiftABIInfo traps with...

Assertion failed: (isa<X>(Val) && "cast<Ty>() argument of incompatible type!"), function cast, file /Users/carlpeto/avr/swift/llvm/include/llvm/Support/Casting.h, line 248.

The code in question looks simple...

static const SwiftABIInfo &getSwiftABIInfo(CodeGenModule &CGM) {
  return cast<SwiftABIInfo>(CGM.getTargetCodeGenInfo().getABIInfo());
}

So my best guess is that CGM.getTargetCodeGenInfo().getABIInfo() returns ABI info that's not Swift ABI info, whatever that means.

Looking through clang source code I can see something that looks like an issue.

Tracing all of the above back to source, the ABI information that comes out depends on the target triple as you would expect.

The thing that looks suspicious is in CodeGenModule::getTargetCodeGenInfo() where the TargetCodeGenInfo is being created with ABI information in it, most major targets (e.g. ARM, x86_64, SystemZ) are creating TargetCodeGenInfo using a specialised ABIInfo, for example for ARM it's ARMABIInfo. In all these cases where I know Swift works, the ABI Info class derives from a special subclass called SwiftABIInfo. For AVR, Mips and some other architectures, it does not. AVR uses DefaultABIInfo for example.

This then would cause the crash when compiling because swift tries to load clang context for emitting IR and correctly asserts because DefaultABIInfo cannot be cast to SwiftABIInfo.

Firstly, can anyone recommend a simple workaround (e.g. disabling parts of clang that aren't doing anything useful here). I am not interested in clang emission i don't think.

Secondly, if there's no way to limit this, can I just change AVRTargetCodeGenInfo to use ABI info for swift, e.g.

namespace {
class AVRTargetCodeGenInfo : public TargetCodeGenInfo {
public:
 AVRTargetCodeGenInfo(CodeGenTypes &CGT)
   : TargetCodeGenInfo(new DefaultABIInfo(CGT)) { }

becomes...

namespace {
class AVRTargetCodeGenInfo : public TargetCodeGenInfo {
public:
 AVRTargetCodeGenInfo(CodeGenTypes &CGT)
   : TargetCodeGenInfo(new SwiftABIInfo(CGT)) { }

Would this work? Do I need to add extra features to make it all work?

(I know the question looks like clang rather than swift but I think this infrastructure isn't really core clang and looks like it's just been added to clang to support swift.)

2

The above guess was right. When compiling using swift, if the target triple includes an architecture that has not yet been updated for swift ABI support in the clang libraries, the above assert is thrown.

To fix it, clang needs to be patched. To do it properly would be a fairly serious piece of work. If someone is reading this after seeing this error themselves, it's probably because they did what I did and patched lib/Basic/LangOptions.cpp (or similar) to support a new architecture without adding ABI support to clang.

Here's a sample patch I used to add basic ABI support...

diff --git a/lib/CodeGen/TargetInfo.cpp b/lib/CodeGen/TargetInfo.cpp
index 2078ea1fea..a08fe505fd 100644
--- a/lib/CodeGen/TargetInfo.cpp
+++ b/lib/CodeGen/TargetInfo.cpp
@@ -7095,10 +7095,79 @@ MIPSTargetCodeGenInfo::initDwarfEHRegSizeTable(CodeGen::CodeGenFunction &CGF,
 //===----------------------------------------------------------------------===//
 
 namespace {
+  /* Basic hack, created an ABI class just as copied code from DefaultABIInfo
+  but derived from SwiftABIInfo and with swift suitable implementation copied
+  from the SystemZ implementation of SwiftABIInfo methods.
+  Not suitable for production use, for experiments only. May contain bugs. */
+class AVRABIInfo : public SwiftABIInfo {
+
+public:
+  AVRABIInfo(CodeGenTypes &CGT) : SwiftABIInfo(CGT) {}
+
+  ABIArgInfo classifyReturnType(QualType RetTy) const;
+  ABIArgInfo classifyArgumentType(QualType RetTy) const;
+
+  bool shouldPassIndirectlyForSwift(ArrayRef<llvm::Type*> scalars,
+                                    bool asReturnValue) const override {
+    return occupiesMoreThan(CGT, scalars, /*total*/ 4);
+  }
+  bool isSwiftErrorInRegister() const override {
+    return false;
+  }
+
+  /* copied from DefaultABIInfo */
+  void computeInfo(CGFunctionInfo &FI) const override {
+    if (!getCXXABI().classifyReturnType(FI))
+      FI.getReturnInfo() = classifyReturnType(FI.getReturnType());
+    for (auto &I : FI.arguments())
+      I.info = classifyArgumentType(I.type);
+  }
+
+  Address EmitVAArg(CodeGenFunction &CGF, Address VAListAddr,
+                    QualType Ty) const override {
+    return EmitVAArgInstr(CGF, VAListAddr, Ty, classifyArgumentType(Ty));
+  }
+};
+
+ABIArgInfo AVRABIInfo::classifyArgumentType(QualType Ty) const {
+  Ty = useFirstFieldIfTransparentUnion(Ty);
+
+  if (isAggregateTypeForABI(Ty)) {
+    // Records with non-trivial destructors/copy-constructors should not be
+    // passed by value.
+    if (CGCXXABI::RecordArgABI RAA = getRecordArgABI(Ty, getCXXABI()))
+      return getNaturalAlignIndirect(Ty, RAA == CGCXXABI::RAA_DirectInMemory);
+
+    return getNaturalAlignIndirect(Ty);
+  }
+
+  // Treat an enum type as its underlying type.
+  if (const EnumType *EnumTy = Ty->getAs<EnumType>())
+    Ty = EnumTy->getDecl()->getIntegerType();
+
+  return (Ty->isPromotableIntegerType() ? ABIArgInfo::getExtend(Ty)
+                                        : ABIArgInfo::getDirect());
+}
+
+ABIArgInfo AVRABIInfo::classifyReturnType(QualType RetTy) const {
+  if (RetTy->isVoidType())
+    return ABIArgInfo::getIgnore();
+
+  if (isAggregateTypeForABI(RetTy))
+    return getNaturalAlignIndirect(RetTy);
+
+  // Treat an enum type as its underlying type.
+  if (const EnumType *EnumTy = RetTy->getAs<EnumType>())
+    RetTy = EnumTy->getDecl()->getIntegerType();
+
+  return (RetTy->isPromotableIntegerType() ? ABIArgInfo::getExtend(RetTy)
+                                           : ABIArgInfo::getDirect());
+}
+
 class AVRTargetCodeGenInfo : public TargetCodeGenInfo {
 public:
   AVRTargetCodeGenInfo(CodeGenTypes &CGT)
-    : TargetCodeGenInfo(new DefaultABIInfo(CGT)) { }
+    : TargetCodeGenInfo(new AVRABIInfo(CGT)) { }
 
   void setTargetAttributes(const Decl *D, llvm::GlobalValue *GV,
                            CodeGen::CodeGenModule &CGM) const override {

This is enough for now. We can consider this closed I think.