Swift 4 TensorFlow is 10x slower when running in xcode versus jupyter notebook

Related Projects
1

Hi Everyone,

Please excuse me and redirect me to the appropriate forum to ask this question if necessary.
My question concerns the swift-4-tensorflow framework.

I recently started reading David Foster's "Generative Deep Learning" book. I want to follow the examples, but I prefer swift over Python, and additionaly I prefer running my code in xcode instead of jupyter notebook.

I successfully converted his first notebook example python code to Swift using the "TensorFlow" swift library and I can run it in xcode. However, when training the network the training speed is dramatically slower (more than 20x) than when running the same example in notebook/python.

Any idea why? I'm pretty sure in both cases the training is only using the CPU.

EDIT: I do get the following warning when running in xcode right before the training starts:

020-03-07 14:03:32.859407: I tensorflow/core/platform/cpu_feature_guard.cc:137] Your CPU supports instructions that this TensorFlow binary was not compiled to use: SSE4.1 SSE4.2 AVX AVX2 FMA

Thank you and all the best,
-d

2

It's hard to debug performance without more information. Could you share your Python and Swift code?

It sounds like you're comparing the performance of Swift for TensorFlow vs Keras in Python? S4TF and Keras have different programming models, so it might not be an apples-to-apples comparison.

3

If you've used Python TensorFlow before, this warning may look familiar! It occurs because TensorFlow must be built explicitly with SSE/AVX/FMA support:

bazel build -c opt --copt=-mavx --copt=-msse4.1 --copt=-msse4.2 --copt=-mfma

Swift for TensorFlow toolchains are currently built without these CPU extensions for maximum compatibility with different CPUs. We could consider building with some of these extensions by default though, especially on macOS which is more homogeneous.

Here's a more complete explanation of this warning in the context of Python TensorFlow.

1 Like
4

Also: you'll have more visibility asking questions on the Swift for TensorFlow mailing list!
The Swift for TensorFlow team will definitely see your questions there.

5

That is true!

Here is David's code: https://github.com/davidADSP/GDL_code/blob/master/02_01_deep_learning_deep_neural_network.ipynb

And here is mine (I hope it's not too long)

import Cocoa
import TensorFlow
import Python

let kNumberOfClasses = 10

class ViewController: NSViewController {
    
    var model:PythonObject!
    var trainingData_x:PythonObject!
    var trainingData_y:PythonObject!

    override func viewDidLoad() {
        super.viewDidLoad()
        setupTrainingData()
        setupModel()
        compileModel()
        trainModel()
    }
    
    func setupTrainingData() {
        let kerasUtils = Python.import("keras.utils")
        let cifar = Python.import("keras.datasets").cifar10

        let cifarData = cifar.load_data()
        let trainingData = cifarData[0]
        let testData = cifarData[1]
        
        trainingData_x = trainingData[0]
        var testData_x = testData[0]
        
        trainingData_x = trainingData_x.astype("float32") / 255.0
        testData_x = testData_x.astype("float32") / 255.0
        
        trainingData_y = trainingData[1]
        var testData_y = testData[1]

        trainingData_y = kerasUtils.to_categorical(trainingData_y, kNumberOfClasses)
        testData_y = kerasUtils.to_categorical(testData_y, kNumberOfClasses)
        
        let value = trainingData_x[54,12,13,1]
        print(value)
    }
    
    func setupModel() {
        let Sequential = Python.import("keras.models").Sequential
        let Flatten = Python.import("keras.layers").Flatten
        let Dense = Python.import("keras.layers").Dense

        model = Sequential([
            Dense(200, activation: "relu", input_shape: [32,32,3]),
            Flatten(),
            Dense(150, activation: "relu"),
            Dense(10, activation: "softmax")
        ])
        
        print(model.summary())
    }
    
    func compileModel() {
        let Adam = Python.import("keras.optimizers").Adam
        
        let opt = Adam(lr: 0.0005)
        model.compile(loss:"categorical_crossentropy", optimizer: opt, metrics: ["accuracy"])
    }
    
    func trainModel() {
        model.fit(trainingData_x, trainingData_y, batch_size: 32, epochs: 10, shuffle: true)
    }
}

I will take a look at the mailing list.
Thank you!

6

In particular, the last macOS that supported hardware without SSE4.1 and 4.2 was 10.13, so those seem pretty safe for S4TF. We should also investigate better support for building fat Swift libraries with x86_64h support, which would let you use F16C, AVX2 and FMA while still having a fallback in the x86_64 slice.

2 Likes
7

Ooo, I didn't expect that you're using Python interop in Swift to call Keras, instead of using Swift for TensorFlow deep learning APIs.

I haven't tried running or profiling your code, but I imagine calling Python (C APIs) via Swift has overhead compared with directly using CPython: calling Python via Swift involves at least running additional Swift code.

Minimizing Python API calls (e.g. by moving Python code from Swift to Python, and importing the resulting Python code) should reduce some overheads.

But really, I think you should try learning Swift if you have time and interest! Here's a Swift ML model training walkthrough that shows you how to define and train a model in Swift. Models written in Swift are more idiomatic and should be more concise and performant.

1 Like
8

The library here (TensorFlow) is actually a C++ library rather than a Swift library, which, specifically for macOS I suppose would be possible to build in a fat configuration (although may become computationally expensive at ~4hrs per slice).

1 Like
9

Ow ok, I see.

My fear was, reading everywhere that Keras is essentialy a high level API of tensorflow, that I would be having to go to low-level in code, while Keras has all the benefits to offer for someone just starting out with deep learning.

But maybe I'm wrong and the same thing can be done with (S4-)TensorfFlow?

Stil it would be nice to see some official Keras integration to swfit in the future imho, as most resources I have come across use Keras.

Thanks for the info. I will try to convert all the Python code to Swift then :slight_smile: (i use swift on a daily basis since I am primarily an iOS developer, so maybe the effort makes sense :p)

10

Swift for TensorFlow has high-level deep learning APIs too, like Sequential from Keras!

Here's the Python model definition from "Your first deep neural network":

from keras.layers import Input, Flatten, Dense
from keras.models import Model

NUM_CLASSES = 10
input_layer = Input((32,32,3))
x = Flatten()(input_layer)
x = Dense(200, activation = 'relu')(x)
x = Dense(150, activation = 'relu')(x)
output_layer = Dense(NUM_CLASSES, activation = 'softmax')(x)
model = Model(input_layer, output_layer)
model.summary()

And here's what it looks like in Swift (no input shape inference):

import TensorFlow

let inputSize = 32 * 32 * 3
let classCount = 10

var model = Sequential {
  Flatten<Float>()
  Dense<Float>(inputSize: inputSize, outputSize: 200, activation: relu)
  Dense<Float>(inputSize: 200, outputSize: 150, activation: relu)
  Dense<Float>(inputSize: 150, outputSize: classCount, activation: softmax)
}

Check out GitHub - tensorflow/swift-models: Models and examples built with Swift for TensorFlow for models and examples written in Swift - I think it's a nice starting point for learning Swift for TensorFlow.

This example shows how to train custom models ported from PyTorch and Keras on CIFAR-10. Model training loops are just regular Swift loops that differentiate models wrt a loss function and apply an optimizer.

1 Like
11

Ow great, that looks nice!

Are there also examples available focused on S4TF in a macOS xcode project environment?
Most of them are assuming using swift in the command line or with Notebooks.

The reason I am asking is because I want to build a user-friendly macOS app with a UI so my client does not have to deal with commands etc.

Thank you!

12

GitHub - tensorflow/swift-models: Models and examples built with Swift for TensorFlow is a SwiftPM package, so you can open it in Xcode.

I usually run:

git clone https://github.com/tensorflow/swift-models.git
cd swift-models
swift package generate-xcodeproj
open TensorFlowModels.xcodeproj # opens Xcode

There are other ways to use SwiftPM within Xcode too: https://developer.apple.com/documentation/swift_packages

1 Like
13

This is what I've been searching for.

Thank you!

1 Like
14

Hello, Dan Zhang. Is this how initialization will look like with function builder?

15

Yes, Sequential uses a function builder called LayerBuilder.