NSMutableData memory leak

Ian_Partridge2 · May 16, 2016, 4:34pm

import Foundation
while true {
var myData: NSMutableData? = NSMutableData(capacity: 0)
myData = nil
}

Running this infinite loop with swift-corelibs-foundation shows a steady
memory leak, with the process's RSS increasing over time. No leak is seen
with Foundation on Darwin.

Instrumenting with Valgrind's massif profiler shows this stacktrace is
responsible for the leak:

67.36% (114,349B) (heap allocation functions) malloc/new/new,
--alloc-fns, etc.
->65.01% (110,352B) 0x59F7A89: _CFDataInit
  ->65.01% (110,352B) 0x5B8A8DF:
_TTSf4n_n_n_g_n___TFC10Foundation6NSDatacfT5bytesGSqGSpT___6lengthSi4copySb11deallocatorGSqFTGSpT__Si_T___S0_
    ->65.01% (110,352B) 0x5B873ED:
_TFC10Foundation13NSMutableDataCfT8capacitySi_GSqS0__
      ->65.01% (110,352B) 0x40105D: main

I've stepped through the code with a debugger and observed that the
requested capacity is thrown away
<swift-corelibs-foundation/NSData.swift at df239bbbdf5bcdd9ea31c394c6af4dd7c328f99d · apple/swift-corelibs-foundation · GitHub;
[1]
to begin with. The leak occurs regardless of the capacity requested.

The deinitializer for NSData does call through to _CFDeinit(), which does
then call the finalize()
<swift-corelibs-foundation/CFRuntime.c at ea6179dd35be2c7d9a8f953579f626a5f1be6511 · apple/swift-corelibs-foundation · GitHub;
[2]
function and hence through to __CFDataDeallocate()
<swift-corelibs-foundation/CFData.c at ea3014bd7883e428727272118cbf37dc56522be6 · apple/swift-corelibs-foundation · GitHub;
[3].
However, once in __CFDataDeallocate(), the code to free the buffer is
skipped, because __kCFDontDeallocate is set.

If I hack _CFDataInit() so that __kCFDontDeallocate isn't set (by
commenting out this line
<swift-corelibs-foundation/CFData.c at ea3014bd7883e428727272118cbf37dc56522be6 · apple/swift-corelibs-foundation · GitHub;
[4])
then I get crashes elsewhere - so this obviously isn't the right approach.

I can see that some work has been done in this area
<https://github.com/apple/swift-corelibs-foundation/commit/ea3014bd7883e428727272118cbf37dc56522be6>
[5]
previously by Philippe so I'm wondering if anyone can advise on what might
be going on here?

The init?(length:) initializer avoids CFData entirely and calls malloc()
and free() directly. I'm not sure why that approach was taken and whether
it's relevant to my issue.

Any help would be gratefully received!

Thanks,

[1]

github.com

apple/swift-corelibs-foundation/blob/df239bbbdf5bcdd9ea31c394c6af4dd7c328f99d/Foundation/NSData.swift#L904


      
                  replaceBytes(in: NSMakeRange(0, data.length), withBytes: data.bytes)
              }
              
              public func replaceBytes(in range: NSRange, withBytes replacementBytes: UnsafePointer<Void>, length replacementLength: Int) {
                  CFDataReplaceBytes(_cfMutableObject, CFRangeMake(range.location, range.length), UnsafePointer<UInt8>(bytes), replacementLength)
              }
          }
          
          
extension NSMutableData {
              
              public convenience init?(capacity: Int) {
                  self.init(bytes: nil, length: 0)
              }
              
              public convenience init?(length: Int) {
                  let memory = malloc(length)
                  self.init(bytes: memory, length: length, copy: false) { buffer, amount in
                      free(buffer)
                  }
              }
          }

[2]

github.com

apple/swift-corelibs-foundation/blob/ea6179dd35be2c7d9a8f953579f626a5f1be6511/CoreFoundation/Base.subproj/CFRuntime.c#L1773


      
          struct _CFSwiftBridge __CFSwiftBridge = { { NULL } };
          
          

          
// Call out to the CF-level finalizer, because the object is going to go away.
          CF_SWIFT_EXPORT void _CFDeinit(CFTypeRef cf) {
              uint32_t cfinfo = *(uint32_t *)&(((CFRuntimeBase *)cf)->_cfinfo);
              CFTypeID typeID = (cfinfo >> 8) & 0x03FF; // mask up to 0x0FFF
              CFRuntimeClass *cfClass = __CFRuntimeClassTable[typeID];
              void (*func)(CFTypeRef) = __CFRuntimeClassTable[typeID]->finalize;
              if (NULL != func) {
                  func(cf);
              }
          }
          #endif
          
          
bool _CFIsSwift(CFTypeID type, CFSwiftRef obj) {
              if (type == _kCFRuntimeNotATypeID) {
                  return false;
              }
              if (obj->isa == (uintptr_t)__CFConstantStringClassReferencePtr) return false;
              return obj->isa != __CFRuntimeObjCClassTable[type];

[3]

github.com

apple/swift-corelibs-foundation/blob/ea3014bd7883e428727272118cbf37dc56522be6/CoreFoundation/Collections.subproj/CFData.c#L294


      
          	    } else {
          		bytes = malloc(size);
          	    }
          	}
              }
              return bytes;
          }
          
          
static void __CFDataDeallocate(CFTypeRef cf) {
              CFMutableDataRef data = (CFMutableDataRef)cf;
              if (!__CFDataBytesInline(data) && !__CFDataGetInfoBit(data, __kCFDontDeallocate)) {
                  CFAllocatorRef deallocator = data->_bytesDeallocator;
                  if (deallocator != NULL) {
                      _CFAllocatorDeallocateGC(deallocator, data->_bytes);
                      CFRelease(deallocator);
                      data->_bytes = NULL;
                  } else {
                      if (__CFDataUseAllocator(data)) {
                          _CFAllocatorDeallocateGC(__CFGetAllocator(data), data->_bytes);
                      } else if (!__CFDataAllocatesCollectable(data) && data->_bytes) {
                          free(data->_bytes);

[4]

github.com

apple/swift-corelibs-foundation/blob/ea3014bd7883e428727272118cbf37dc56522be6/CoreFoundation/Collections.subproj/CFData.c#L337


      
              dispatch_once(&initOnce, ^{ __kCFDataTypeID = _CFRuntimeRegisterClass(&__CFDataClass); });
              return __kCFDataTypeID;
          }
          
          
CF_SWIFT_EXPORT void _CFDataInit(CFMutableDataRef memory, CFOptionFlags flags, CFIndex capacity, const uint8_t *bytes, CFIndex length, Boolean noCopy) {
              Boolean isMutable = ((flags & __kCFMutable) != 0);
              Boolean isGrowable = ((flags & __kCFGrowable) != 0);
              
              __CFDataSetNumBytesUsed(memory, 0);
              __CFDataSetLength(memory, 0);
              __CFDataSetInfoBits(memory, __kCFDontDeallocate);
              
              if (isMutable && isGrowable) {
                  __CFDataSetCapacity(memory, __CFDataRoundUpCapacity(1));
                  __CFDataSetNumBytes(memory, __CFDataNumBytesForCapacity(__CFDataRoundUpCapacity(1)));
                  __CFSetMutableVariety(memory, kCFMutable);
              } else {
                  /* Don't round up capacity */
                  __CFDataSetCapacity(memory, capacity);
                  __CFDataSetNumBytes(memory, __CFDataNumBytesForCapacity(capacity));
                  __CFSetMutableVariety(memory, kCFFixedMutable);

[5]
https://github.com/apple/swift-corelibs-foundation/commit/ea3014bd7883e428727272118cbf37dc56522be6

···

--
Ian Partridge

Philippe_Hausler · May 16, 2016, 4:43pm

I think there is probably some likely issue with the fact that NSMutableData is a subclass and the layout initialization is not properly setup during the init for that object.

__kCFDontDeallocate is used to mark the allocated memory as managed by the deallocator blocks. So I bet the problem is that the code-path for NSMutableData(capacity: 0) is hitting the don’t deallocate code path incorrectly (where as the other versions are hitting the appropriate flagging; hence the crashes)

Does this also occur in the Darwin builds of this? (Using SwiftFoundtion instead of Foundation)

···

On May 16, 2016, at 9:34 AM, Ian Partridge via swift-corelibs-dev <swift-corelibs-dev@swift.org> wrote:

import Foundation

while true {
  var myData: NSMutableData? = NSMutableData(capacity: 0)
  myData = nil
}

Running this infinite loop with swift-corelibs-foundation shows a steady memory leak, with the process's RSS increasing over time. No leak is seen with Foundation on Darwin.

Instrumenting with Valgrind's massif profiler shows this stacktrace is responsible for the leak:

67.36% (114,349B) (heap allocation functions) malloc/new/new, --alloc-fns, etc.
->65.01% (110,352B) 0x59F7A89: _CFDataInit
  ->65.01% (110,352B) 0x5B8A8DF: _TTSf4n_n_n_g_n___TFC10Foundation6NSDatacfT5bytesGSqGSpT___6lengthSi4copySb11deallocatorGSqFTGSpT__Si_T___S0_
    ->65.01% (110,352B) 0x5B873ED: _TFC10Foundation13NSMutableDataCfT8capacitySi_GSqS0__
      ->65.01% (110,352B) 0x40105D: main
I've stepped through the code with a debugger and observed that the requested capacity is thrown away <https://github.com/apple/swift-corelibs-foundation/blob/df239bbbdf5bcdd9ea31c394c6af4dd7c328f99d/Foundation/NSData.swift#L904> [1] to begin with. The leak occurs regardless of the capacity requested.

The deinitializer for NSData does call through to _CFDeinit(), which does then call the finalize() <https://github.com/apple/swift-corelibs-foundation/blob/ea6179dd35be2c7d9a8f953579f626a5f1be6511/CoreFoundation/Base.subproj/CFRuntime.c#L1773> [2] function and hence through to __CFDataDeallocate() <https://github.com/apple/swift-corelibs-foundation/blob/ea3014bd7883e428727272118cbf37dc56522be6/CoreFoundation/Collections.subproj/CFData.c#L294> [3]. However, once in __CFDataDeallocate(), the code to free the buffer is skipped, because __kCFDontDeallocate is set.

If I hack _CFDataInit() so that __kCFDontDeallocate isn't set (by commenting out this line <https://github.com/apple/swift-corelibs-foundation/blob/ea3014bd7883e428727272118cbf37dc56522be6/CoreFoundation/Collections.subproj/CFData.c#L337> [4]) then I get crashes elsewhere - so this obviously isn't the right approach.

I can see that some work has been done in this area <https://github.com/apple/swift-corelibs-foundation/commit/ea3014bd7883e428727272118cbf37dc56522be6> [5] previously by Philippe so I'm wondering if anyone can advise on what might be going on here?

The init?(length:) initializer avoids CFData entirely and calls malloc() and free() directly. I'm not sure why that approach was taken and whether it's relevant to my issue.

Any help would be gratefully received!

Thanks,

[1] https://github.com/apple/swift-corelibs-foundation/blob/df239bbbdf5bcdd9ea31c394c6af4dd7c328f99d/Foundation/NSData.swift#L904
[2] https://github.com/apple/swift-corelibs-foundation/blob/ea6179dd35be2c7d9a8f953579f626a5f1be6511/CoreFoundation/Base.subproj/CFRuntime.c#L1773
[3] https://github.com/apple/swift-corelibs-foundation/blob/ea3014bd7883e428727272118cbf37dc56522be6/CoreFoundation/Collections.subproj/CFData.c#L294
[4] https://github.com/apple/swift-corelibs-foundation/blob/ea3014bd7883e428727272118cbf37dc56522be6/CoreFoundation/Collections.subproj/CFData.c#L337
[5] Additional sanitization passes on the copy/deallocation capture state… · apple/swift-corelibs-foundation@ea3014b · GitHub

--
Ian Partridge

_______________________________________________
swift-corelibs-dev mailing list
swift-corelibs-dev@swift.org
https://lists.swift.org/mailman/listinfo/swift-corelibs-dev

Ian_Partridge2 · May 16, 2016, 4:45pm

Yes, I have been investigating this using a new TestNSData test under XCode.

···

On 16 May 2016 at 17:43, Philippe Hausler <phausler@apple.com> wrote:

I think there is probably some likely issue with the fact that NSMutableData is a subclass and the layout initialization is not properly setup during the init for that object.

__kCFDontDeallocate is used to mark the allocated memory as managed by the deallocator blocks. So I bet the problem is that the code-path for NSMutableData(capacity: 0) is hitting the don’t deallocate code path incorrectly (where as the other versions are hitting the appropriate flagging; hence the crashes)

Does this also occur in the Darwin builds of this? (Using SwiftFoundtion instead of Foundation)

--
Ian Partridge

Ian_Partridge2 · May 18, 2016, 3:41pm

We're continuing to investigate this, and have raised SR-1552 to track
it - [SR-1552] NSMutableData(capacity:) memory leak on Linux · Issue #4349 · apple/swift-corelibs-foundation · GitHub

···

--
Ian Partridge