Hi all,
I'm working on a text editor with syntax highlighting and I'm using a C framework for my parser (to find out what characters to highlight).
The parser receives a String (as char), a start Int and a length Int and gives back an AST made out of structs called token. The return type is UnsafeMutablePointer<token>?.
struct token {
unsigned short type; //!< Type for the token
short can_open; //!< Can token open a matched pair?
short can_close; //!< Can token close a matched pair?
short unmatched; //!< Has token been matched yet?
size_t start; //!< Starting offset in the source string
size_t len; //!< Length of the token in the source string
struct token * next; //!< Pointer to next token in the chain
struct token * prev; //!< Pointer to previous token in the chain
struct token * child; //!< Pointer to child chain
struct token * tail; //!< Pointer to last token in the chain
struct token * mate; //!< Pointer to other token in matched pair
};
Then I go over this tree using a recursive function:
let t = mmd_engine_parse_substring(e, location, length)
// Match the String with the Syntax
self.handleTokenTree(t)
func handleTokenTree(_ t: UnsafeMutablePointer<token>?) {
// A binary search tree of token nodes, each of which can point to a child and/or a "next" sibling token. That builds up the structure of the document!.pointee. You walk through sibling nodes, recursively handling children:
while t != nil {
// Recursive function to walk the token tree and apply formatting based on the token types
self.applyHighlightingForTokenTree(t)
if t!.pointee.child != nil {
// Recurse into children tokens
handleTokenTree(t!.pointee.child)
}
// Next sibling token
t = t!.pointee.next
}
// Cleanup the memory
token_free(t)
}
This code worked great in Swift 4.2 and also works perfectly fine in Objective-C.
struct token * t = mmd_engine_parse_substring(e, extendedRange.location, extendedRange.length);
// Match the String with the Syntax
[self handleTokenTree:t];
-(void)handleTokenTree:(token *)t
{
// A binary search tree of token nodes, each of which can point to a child and/or a "next" sibling token. That builds up the structure of the document!.pointee. You walk through sibling nodes, recursively handling children:
while (t != nil) {
// Recursive function to walk the token tree and apply formatting based on the token types
[self applySyntaxHighlightingForTokenTree:t];
if (t->child != nil) {
// Recurse into children tokens
[self handleTokenTree:t->child];
}
// Next sibling token
t = t->next;
}
// Cleanup the memory
token_free(t);
}
But in Swift 5, the token loses it's values.
For example, the parser parses the text and returns the token with a proper child and tail. But when it reaches Swift (I placed a breakpoint in the line between let t and handleTokenTree), the token doesn't have the child or the tail and has nil/NULL for everything except for type and mate (which was supposed to be nil/NULL).
I have made sure the issue is just with Swift 5 by trying different devices, different OS versions, Swift 4.2, Objective-C and debugging, using both breakpoints and print statements. The issue occurs only in Swift 5.
Another thing I've noticed, though I'm not sure if it has any effect is, I've kept an eye on the pointer address and Swift 5 adds a lot of leading zeros to the pointer. For example, if the pointer address is 0x109a4000, Swift 5 makes it into 0x000000109a4000.
What can I do to fix the issue except for using Objective-C?
I'm currently using Objective-C as a workaround, but would love to get this resolved and use Swift 5 since the rest of my code is in Swift 5 and I want to calculate offsets using Swift's UTF8/16Views.
This issue has been driving me crazy since I moved to Swift 5 with the release last Monday.
Any help would be much appreciated 
