buildbot configuration + etc


(Travis Tilley) #1

Is there any documentation somewhere on what the exact commands are that
the buildbot CI performs and how the packages on swift.org are created?

I've noticed that:

1) LLDB doesn't compile on either OSX or ubuntu 15 via the normal script
for various reasons (codesign on OSX, no idea on ubuntu) in the default
configuration. This seems highly counter-intuitive.

2) The buildbot presets don't have sane defaults and I can't find any
documentation on what "sane" might mean for those builds.

3) I still can't build the full suite using the following command (which,
again, may or may not be sane):

nice ./swift/utils/build-script \

  --preset=buildbot_osx_package \
  installable_package=/tmp/swift.tar.gz \

install_destdir=/Library/Developer/Toolchains/swift-2.2-SNAPSHOT-2015-12-14/
\

install_toolchain_dir=/Library/Developer/Toolchains/swift-2.2-SNAPSHOT-2015-12-14.xctoolchain
\

install_symroot=/Library/Developer/Toolchains/swift-2.2-SNAPSHOT-2015-12-14/
\
  symbols_package=/tmp/swift-symbols.tar.gz \
  darwin_toolchain_bundle_identifier=org.swift.20151214 \
  darwin_toolchain_display_name=20151214 \
  darwin_toolchain_xctoolchain_name=swift darwin_toolchain_version=swift

​All that I actually -want- to do is start xcode using a freshly built
swift toolchain. That is it. I don't even need any of the SDKs other than
OSX built. But the build directory install structure doesn't have anything
like what `​launch-with-toolchain` appears to expect...

Not sure where i'm going wrong. Please feel free to tell me to RTFM if
there is a FM to read.

-Travis Tilley


(Dmitri Gribenko) #2

+Mishal Awadah, Mishal Shah

···

On Mon, Dec 14, 2015 at 8:04 PM, Travis Tilley via swift-dev < swift-dev@swift.org> wrote:

Is there any documentation somewhere on what the exact commands are that
the buildbot CI performs and how the packages on swift.org are created?

I've noticed that:

1) LLDB doesn't compile on either OSX or ubuntu 15 via the normal script
for various reasons (codesign on OSX, no idea on ubuntu) in the default
configuration. This seems highly counter-intuitive.

2) The buildbot presets don't have sane defaults and I can't find any
documentation on what "sane" might mean for those builds.

3) I still can't build the full suite using the following command (which,
again, may or may not be sane):

nice ./swift/utils/build-script \

  --preset=buildbot_osx_package \
  installable_package=/tmp/swift.tar.gz \

install_destdir=/Library/Developer/Toolchains/swift-2.2-SNAPSHOT-2015-12-14/
\

install_toolchain_dir=/Library/Developer/Toolchains/swift-2.2-SNAPSHOT-2015-12-14.xctoolchain
\

install_symroot=/Library/Developer/Toolchains/swift-2.2-SNAPSHOT-2015-12-14/
\
  symbols_package=/tmp/swift-symbols.tar.gz \
  darwin_toolchain_bundle_identifier=org.swift.20151214 \
  darwin_toolchain_display_name=20151214 \
  darwin_toolchain_xctoolchain_name=swift darwin_toolchain_version=swift

​All that I actually -want- to do is start xcode using a freshly built
swift toolchain. That is it. I don't even need any of the SDKs other than
OSX built. But the build directory install structure doesn't have anything
like what `​launch-with-toolchain` appears to expect...

Not sure where i'm going wrong. Please feel free to tell me to RTFM if
there is a FM to read.

-Travis Tilley

_______________________________________________
swift-dev mailing list
swift-dev@swift.org
https://lists.swift.org/mailman/listinfo/swift-dev

--
main(i,j){for(i=2;;i++){for(j=2;j<i;j++){if(!(i%j)){j=0;break;}}if
(j){printf("%d\n",i);}}} /*Dmitri Gribenko <gribozavr@gmail.com>*/


(Mish Awadah) #3

I’m sorry for the experience Travis, but this is certainly one of our pain points. There’s a lot at play and we haven’t finished setting up our public CI either. Here’s what I do locally to build a toolchain on OS X:

function build_osx_package() {
                                                                                                                                                                                       
    YEAR=$(date +"%Y")
    MONTH=$(date +"%m")
    DAY=$(date +"%d")
    TOOLCHAIN_VERSION="swift-2.2-SNAPSHOT-${YEAR}-${MONTH}-${DAY}-a"
    ARCHIVE_DIR="${TOOLCHAIN_VERSION}-${BUILD_NUMBER}"
    ARCHIVE="${TOOLCHAIN_VERSION}-osx.tar.gz"
    SYM_ARCHIVE="${TOOLCHAIN_VERSION}-osx-symbols.tar.gz"
    BUNDLE_IDENTIFIER="org.swift.${YEAR}${MONTH}${DAY}"
    DISPLAY_NAME="Swift 2.2 Development Snapshot"
    TOOLCHAIN_NAME="${TOOLCHAIN_VERSION}"
                                                                                                                                                                                       
    SWIFT_SOURCE_ROOT="${SRC_DIR}"
    SWIFT_BUILD_ROOT="${SRC_DIR}/build"
    SWIFT_INSTALLABLE_PACKAGE="${SRC_DIR}/${ARCHIVE}"
    SWIFT_INSTALL_DIR="${SRC_DIR}/swift-nightly-install"
    SWIFT_INSTALL_SYMROOT="${SRC_DIR}/swift-nightly-symroot"
    SWIFT_TOOLCHAIN_DIR="/Applications/Xcode.app/Contents/Developer/Toolchains/${TOOLCHAIN_NAME}.xctoolchain"
    SYMBOLS_PACKAGE="${SRC_DIR}/${SYM_ARCHIVE}"
                                                                                                                                                                                       
    ./swift/utils/build-script --preset="buildbot_osx_package" install_destdir="${SWIFT_INSTALL_DIR}" installable_package="${SWIFT_INSTALLABLE_PACKAGE}" install_toolchain_dir="${SWIF\
T_TOOLCHAIN_DIR}" install_symroot="${SWIFT_INSTALL_SYMROOT}" symbols_package="${SYMBOLS_PACKAGE}" darwin_toolchain_bundle_identifier="${BUNDLE_IDENTIFIER}" darwin_toolchain_display_n\
ame="${DISPLAY_NAME}" darwin_toolchain_xctoolchain_name="${TOOLCHAIN_NAME}" darwin_toolchain_version="${TOOLCHAIN_VERSION}"
}

It should work. If not, please file a bug with your output and configuration and let us know about it.

- mish

···

On Dec 14, 2015, at 8:32 PM, Dmitri Gribenko <gribozavr@gmail.com> wrote:

+Mishal Awadah, Mishal Shah

On Mon, Dec 14, 2015 at 8:04 PM, Travis Tilley via swift-dev <swift-dev@swift.org <mailto:swift-dev@swift.org>> wrote:
Is there any documentation somewhere on what the exact commands are that the buildbot CI performs and how the packages on swift.org <http://swift.org/> are created?

I've noticed that:

1) LLDB doesn't compile on either OSX or ubuntu 15 via the normal script for various reasons (codesign on OSX, no idea on ubuntu) in the default configuration. This seems highly counter-intuitive.

2) The buildbot presets don't have sane defaults and I can't find any documentation on what "sane" might mean for those builds.

3) I still can't build the full suite using the following command (which, again, may or may not be sane):

nice ./swift/utils/build-script \
  --preset=buildbot_osx_package \
  installable_package=/tmp/swift.tar.gz \
  install_destdir=/Library/Developer/Toolchains/swift-2.2-SNAPSHOT-2015-12-14/ \
  install_toolchain_dir=/Library/Developer/Toolchains/swift-2.2-SNAPSHOT-2015-12-14.xctoolchain \
  install_symroot=/Library/Developer/Toolchains/swift-2.2-SNAPSHOT-2015-12-14/ \
  symbols_package=/tmp/swift-symbols.tar.gz \
  darwin_toolchain_bundle_identifier=org.swift.20151214 \
  darwin_toolchain_display_name=20151214 \
  darwin_toolchain_xctoolchain_name=swift darwin_toolchain_version=swift

​All that I actually -want- to do is start xcode using a freshly built swift toolchain. That is it. I don't even need any of the SDKs other than OSX built. But the build directory install structure doesn't have anything like what `​launch-with-toolchain` appears to expect...

Not sure where i'm going wrong. Please feel free to tell me to RTFM if there is a FM to read.

-Travis Tilley

_______________________________________________
swift-dev mailing list
swift-dev@swift.org <mailto:swift-dev@swift.org>
https://lists.swift.org/mailman/listinfo/swift-dev

--
main(i,j){for(i=2;;i++){for(j=2;j<i;j++){if(!(i%j)){j=0;break;}}if
(j){printf("%d\n",i);}}} /*Dmitri Gribenko <gribozavr@gmail.com <mailto:gribozavr@gmail.com>>*/


(Travis Tilley) #4

Thank you very kindly for the response. This is extremely helpful. I'll let
you know my progress tomorrow (it's almost midnight in my time zone). :wink:

​-Travis Tilley​

···

On Mon, Dec 14, 2015 at 11:42 PM, Mish Awadah <mawadah@apple.com> wrote:

I’m sorry for the experience Travis, but this is certainly one of our pain
points. There’s a lot at play and we haven’t finished setting up our public
CI either. Here’s what I do locally to build a toolchain on OS X:

*function* build_osx_package() {

    YEAR=$(*date* +"%Y")

    MONTH=$(*date* +"%m")

    DAY=$(*date* +"%d")

    TOOLCHAIN_VERSION="swift-2.2-SNAPSHOT-${YEAR}-${MONTH}-${DAY}-a"

    ARCHIVE_DIR="${TOOLCHAIN_VERSION}-${BUILD_NUMBER}"

    ARCHIVE="${TOOLCHAIN_VERSION}-osx.tar.gz"

    SYM_ARCHIVE="${TOOLCHAIN_VERSION}-osx-symbols.tar.gz"

    BUNDLE_IDENTIFIER="org.swift.${YEAR}${MONTH}${DAY}"

    DISPLAY_NAME="Swift 2.2 Development Snapshot"

    TOOLCHAIN_NAME="${TOOLCHAIN_VERSION}"

    SWIFT_SOURCE_ROOT="${SRC_DIR}"

    SWIFT_BUILD_ROOT="${SRC_DIR}/build"

    SWIFT_INSTALLABLE_PACKAGE="${SRC_DIR}/${ARCHIVE}"

    SWIFT_INSTALL_DIR="${SRC_DIR}/swift-nightly-install"

    SWIFT_INSTALL_SYMROOT="${SRC_DIR}/swift-nightly-symroot"

    SWIFT_TOOLCHAIN_DIR=
"/Applications/Xcode.app/Contents/Developer/Toolchains/${TOOLCHAIN_NAME}.xctoolchain"

    SYMBOLS_PACKAGE="${SRC_DIR}/${SYM_ARCHIVE}"

    ./swift/utils/build-script --preset="buildbot_osx_package"
install_destdir="${SWIFT_INSTALL_DIR}" installable_package=
"${SWIFT_INSTALLABLE_PACKAGE}" install_toolchain_dir="${SWIF\
T_TOOLCHAIN_DIR}" install_symroot="${SWIFT_INSTALL_SYMROOT}"
symbols_package="${SYMBOLS_PACKAGE}" darwin_toolchain_bundle_identifier=
"${BUNDLE_IDENTIFIER}" darwin_toolchain_display_n\
ame="${DISPLAY_NAME}" darwin_toolchain_xctoolchain_name=
"${TOOLCHAIN_NAME}" darwin_toolchain_version="${TOOLCHAIN_VERSION}"

}

It should work. If not, please file a bug with your output and
configuration and let us know about it.

- mish

On Dec 14, 2015, at 8:32 PM, Dmitri Gribenko <gribozavr@gmail.com> wrote:

+Mishal Awadah, Mishal Shah

On Mon, Dec 14, 2015 at 8:04 PM, Travis Tilley via swift-dev < > swift-dev@swift.org> wrote:

Is there any documentation somewhere on what the exact commands are that
the buildbot CI performs and how the packages on swift.org are created?

I've noticed that:

1) LLDB doesn't compile on either OSX or ubuntu 15 via the normal script
for various reasons (codesign on OSX, no idea on ubuntu) in the default
configuration. This seems highly counter-intuitive.

2) The buildbot presets don't have sane defaults and I can't find any
documentation on what "sane" might mean for those builds.

3) I still can't build the full suite using the following command (which,
again, may or may not be sane):

nice ./swift/utils/build-script \

  --preset=buildbot_osx_package \
  installable_package=/tmp/swift.tar.gz \
  install_destdir=/Library/Developer/Toolchains/swift-2.2-SNAPSHOT-2015-12-14/
\
  install_toolchain_dir=/Library/Developer/Toolchains/swift-2.2-SNAPSHOT-2015-12-14.xctoolchain
\
  install_symroot=/Library/Developer/Toolchains/swift-2.2-SNAPSHOT-2015-12-14/
\
  symbols_package=/tmp/swift-symbols.tar.gz \
  darwin_toolchain_bundle_identifier=org.swift.20151214 \
  darwin_toolchain_display_name=20151214 \
  darwin_toolchain_xctoolchain_name=swift darwin_toolchain_version=swift

​All that I actually -want- to do is start xcode using a freshly built
swift toolchain. That is it. I don't even need any of the SDKs other than
OSX built. But the build directory install structure doesn't have anything
like what `​launch-with-toolchain` appears to expect...

Not sure where i'm going wrong. Please feel free to tell me to RTFM if
there is a FM to read.

-Travis Tilley

_______________________________________________
swift-dev mailing list
swift-dev@swift.org
https://lists.swift.org/mailman/listinfo/swift-dev

--
main(i,j){for(i=2;;i++){for(j=2;j<i;j++){if(!(i%j)){j=0;break;}}if
(j){printf("%d\n",i);}}} /*Dmitri Gribenko <gribozavr@gmail.com>*/


(Travis Tilley) #5

Wait... are you sure that's the appropriate configuration? Shouldn't
the xctoolchain destination be in /Library/Developer/Toolchains/ rather
than the xcode directory proper? Or is the actual end destination handled
entirely by `toolchain-installer'?

Only somewhat related: SRC_DIR isn't defined, so I have my version looking
like this so far (and will add toolchain-installer afterwards
after verifying the install and figuring out how things work there).

function build_osx_package() { YEAR=$(date +"%Y") MONTH=$(date +"%m")
DAY=$(date +"%d") TOOLCHAIN_VERSION="swift-2.2-SNAPSHOT-${YEAR}-${MONTH}-
${DAY}-a" ARCHIVE_DIR="${TOOLCHAIN_VERSION}-${BUILD_NUMBER}" ARCHIVE="
${TOOLCHAIN_VERSION}-osx.tar.gz" SYM_ARCHIVE="${TOOLCHAIN_VERSION}
-osx-symbols.tar.gz" BUNDLE_IDENTIFIER="org.swift.${YEAR}${MONTH}${DAY}"
DISPLAY_NAME="Swift 2.2 Development Snapshot" TOOLCHAIN_NAME="
${TOOLCHAIN_VERSION}" SRC_DIR="$(pwd)" SWIFT_SOURCE_ROOT="${SRC_DIR}"
SWIFT_BUILD_ROOT="${SRC_DIR}/build" SWIFT_INSTALLABLE_PACKAGE="${SRC_DIR}/
${ARCHIVE}" SWIFT_INSTALL_DIR="${SRC_DIR}/swift-nightly-install"
SWIFT_INSTALL_SYMROOT="${SRC_DIR}/swift-nightly-symroot"
SWIFT_TOOLCHAIN_DIR="/Applications/Xcode.app/Contents/Developer/Toolchains/
${TOOLCHAIN_NAME}.xctoolchain" SYMBOLS_PACKAGE="${SRC_DIR}/${SYM_ARCHIVE}"
./swift/utils/build-script --preset="buildbot_osx_package" \
install_destdir="${SWIFT_INSTALL_DIR}" \ installable_package="
${SWIFT_INSTALLABLE_PACKAGE}" \ install_toolchain_dir="
${SWIFT_TOOLCHAIN_DIR}" \ install_symroot="${SWIFT_INSTALL_SYMROOT}" \
symbols_package="${SYMBOLS_PACKAGE}" \ darwin_toolchain_bundle_identifier="
${BUNDLE_IDENTIFIER}" \ darwin_toolchain_display_name="${DISPLAY_NAME}" \
darwin_toolchain_xctoolchain_name="${TOOLCHAIN_NAME}" \
darwin_toolchain_version="${TOOLCHAIN_VERSION}" } build_osx_package

-Travis Tilley

···

On Mon, Dec 14, 2015 at 11:42 PM, Mish Awadah <mawadah@apple.com> wrote:

I’m sorry for the experience Travis, but this is certainly one of our pain
points. There’s a lot at play and we haven’t finished setting up our public
CI either. Here’s what I do locally to build a toolchain on OS X:

*function* build_osx_package() {

    YEAR=$(*date* +"%Y")

    MONTH=$(*date* +"%m")

    DAY=$(*date* +"%d")

    TOOLCHAIN_VERSION="swift-2.2-SNAPSHOT-${YEAR}-${MONTH}-${DAY}-a"

    ARCHIVE_DIR="${TOOLCHAIN_VERSION}-${BUILD_NUMBER}"

    ARCHIVE="${TOOLCHAIN_VERSION}-osx.tar.gz"

    SYM_ARCHIVE="${TOOLCHAIN_VERSION}-osx-symbols.tar.gz"

    BUNDLE_IDENTIFIER="org.swift.${YEAR}${MONTH}${DAY}"

    DISPLAY_NAME="Swift 2.2 Development Snapshot"

    TOOLCHAIN_NAME="${TOOLCHAIN_VERSION}"

    SWIFT_SOURCE_ROOT="${SRC_DIR}"

    SWIFT_BUILD_ROOT="${SRC_DIR}/build"

    SWIFT_INSTALLABLE_PACKAGE="${SRC_DIR}/${ARCHIVE}"

    SWIFT_INSTALL_DIR="${SRC_DIR}/swift-nightly-install"

    SWIFT_INSTALL_SYMROOT="${SRC_DIR}/swift-nightly-symroot"

    SWIFT_TOOLCHAIN_DIR=
"/Applications/Xcode.app/Contents/Developer/Toolchains/${TOOLCHAIN_NAME}.xctoolchain"

    SYMBOLS_PACKAGE="${SRC_DIR}/${SYM_ARCHIVE}"

    ./swift/utils/build-script --preset="buildbot_osx_package"
install_destdir="${SWIFT_INSTALL_DIR}" installable_package=
"${SWIFT_INSTALLABLE_PACKAGE}" install_toolchain_dir="${SWIF\
T_TOOLCHAIN_DIR}" install_symroot="${SWIFT_INSTALL_SYMROOT}"
symbols_package="${SYMBOLS_PACKAGE}" darwin_toolchain_bundle_identifier=
"${BUNDLE_IDENTIFIER}" darwin_toolchain_display_n\
ame="${DISPLAY_NAME}" darwin_toolchain_xctoolchain_name=
"${TOOLCHAIN_NAME}" darwin_toolchain_version="${TOOLCHAIN_VERSION}"

}

It should work. If not, please file a bug with your output and
configuration and let us know about it.

- mish

On Dec 14, 2015, at 8:32 PM, Dmitri Gribenko <gribozavr@gmail.com> wrote:

+Mishal Awadah, Mishal Shah

On Mon, Dec 14, 2015 at 8:04 PM, Travis Tilley via swift-dev < > swift-dev@swift.org> wrote:

Is there any documentation somewhere on what the exact commands are that
the buildbot CI performs and how the packages on swift.org are created?

I've noticed that:

1) LLDB doesn't compile on either OSX or ubuntu 15 via the normal script
for various reasons (codesign on OSX, no idea on ubuntu) in the default
configuration. This seems highly counter-intuitive.

2) The buildbot presets don't have sane defaults and I can't find any
documentation on what "sane" might mean for those builds.

3) I still can't build the full suite using the following command (which,
again, may or may not be sane):

nice ./swift/utils/build-script \

  --preset=buildbot_osx_package \
  installable_package=/tmp/swift.tar.gz \
  install_destdir=/Library/Developer/Toolchains/swift-2.2-SNAPSHOT-2015-12-14/
\
  install_toolchain_dir=/Library/Developer/Toolchains/swift-2.2-SNAPSHOT-2015-12-14.xctoolchain
\
  install_symroot=/Library/Developer/Toolchains/swift-2.2-SNAPSHOT-2015-12-14/
\
  symbols_package=/tmp/swift-symbols.tar.gz \
  darwin_toolchain_bundle_identifier=org.swift.20151214 \
  darwin_toolchain_display_name=20151214 \
  darwin_toolchain_xctoolchain_name=swift darwin_toolchain_version=swift

​All that I actually -want- to do is start xcode using a freshly built
swift toolchain. That is it. I don't even need any of the SDKs other than
OSX built. But the build directory install structure doesn't have anything
like what `​launch-with-toolchain` appears to expect...

Not sure where i'm going wrong. Please feel free to tell me to RTFM if
there is a FM to read.

-Travis Tilley

_______________________________________________
swift-dev mailing list
swift-dev@swift.org
https://lists.swift.org/mailman/listinfo/swift-dev

--
main(i,j){for(i=2;;i++){for(j=2;j<i;j++){if(!(i%j)){j=0;break;}}if
(j){printf("%d\n",i);}}} /*Dmitri Gribenko <gribozavr@gmail.com>*/


(Mishal Shah) #6

Hi Travis,

(Hopefully this message will be sent, last message was blocked due to 40KB limit)

How to create OS X Toolchain?

Xcode toolchain is created using buildbot_osx_package preset, which supports two types of packages.

- Installer(.pkg)
This is similer to what is provide on swift.org <http://swift.org/> website, however you will need installer and application cert.

- Tarball(tar.gz)
This is not code signed and it will contain complete path from "install_toolchain_dir" argument, to install it you can use "sudo darwin install package.tar.gz"

Arguments for buildbot_osx_package preset:

install_destdir - "the path to use as the filesystem root for the installation"
installable_package - "the path to the archive of the installation directory"
install_toolchain_dir - "installation prefix"
install_symroot - "the path to install debug symbols into"
symbols_package - "the path to the archive of the symbols directory"
darwin_toolchain_bundle_identifier - "CFBundleIdentifier for xctoolchain info plist"
darwin_toolchain_display_name - "Display Name for xctoolcain info plist"
darwin_toolchain_xctoolchain_name - "Directory name for xctoolchain"
darwin_toolchain_version - "Version for xctoolchain info plist and installer pkg"
darwin_toolchain_app_cert - "Application Cert name to codesign xctoolchain"
darwin_toolchain_installer_cert - "Installer Cert name to create installer pkg"
darwin_toolchain_installer_package - "The path to installer pkg"

Examples:

Installer:
SRC_DIR="$(pwd)"

swift/utils/build-script \
--preset="buildbot_osx_package" \
install_destdir="${SRC_DIR}/swift_install" \
installable_package="${SRC_DIR}/swift-2.2-dev-osx.tar.gz" \
install_toolchain_dir="/Library/Developer/Toolchains/swift-2.2-dev.xctoolchain" \
install_symroot="${SRC_DIR}/swift_symroot" \
symbols_package="${SRC_DIR}/swift-2.2-dev-osx-symbols.tar.gz" \
darwin_toolchain_bundle_identifier="com.test.xyz <http://com.test.xyz/>" \
darwin_toolchain_display_name="Swift 2.2 Dev" \
darwin_toolchain_xctoolchain_name="swift-2.2-dev" \
darwin_toolchain_version="2.2-dev" \
darwin_toolchain_app_cert="Mac Developer: Name (xxxxxxxx)" \
darwin_toolchain_installer_cert="Developer ID Installer: Name (xxxxxxxx)" \
darwin_toolchain_installer_package="${SRC_DIR}/swift-2.2-dev-osx.pkg"

tar.gz only:
SRC_DIR="$(pwd)"

swift/utils/build-script \
--preset="buildbot_osx_package" \
install_destdir="${SRC_DIR}/swift_install" \
installable_package="${SRC_DIR}/swift-2.2-dev-osx.tar.gz" \
install_toolchain_dir="/Library/Developer/Toolchains/swift-2.2-dev.xctoolchain" \
install_symroot="${SRC_DIR}/swift_symroot" \
symbols_package="${SRC_DIR}/swift-2.2-dev-osx-symbols.tar.gz" \
darwin_toolchain_bundle_identifier="com.test.xyz <http://com.test.xyz/>" \
darwin_toolchain_display_name="Swift 2.2 Dev" \
darwin_toolchain_xctoolchain_name="swift-2.2-dev" \
darwin_toolchain_version="2.2-dev"

How to install tar.gz?
sudo darwinup install package.tar.gz

Hope this helps.

Thanks,
Mishal Shah


(Jens Persson) #7

I'm also trying to figure out more about the build process. I don't even
understand in what context the buld_osx_package function should be (should
it be just in it's own file or in some context where eg SRC_DIR and
BUILD_NUMBER are defined)?

What I want to do most of the times is just to try out the current master,
so I do:
swift/utils/update-checkout -a

And then I want to build something like the dev snapshots, but I don't want
to wait for too long, so I want to build only for os x (skip ios, tvos,
watchos) and no tests, and as I want to see any improvements in the
optimizer I want --no-assertions, --no-swift-stdlib-assertions.

I've not managed to make enough sense out of looking at the code and
comments of the build tools to feel confident in writing my own
build-preset or anything like that.

In order to get something that worked, I looked at what build_osx_package
preset expanded into, and then modified those flags according to the above
and passed that to build-script.

I suppose a lot of people are able to figure out a more sane/proper way to
do this, but I can't without more documentation / instructions ...

···

On Tue, Dec 15, 2015 at 6:45 AM, Travis Tilley via swift-dev < swift-dev@swift.org> wrote:

Wait... are you sure that's the appropriate configuration? Shouldn't
the xctoolchain destination be in /Library/Developer/Toolchains/ rather
than the xcode directory proper? Or is the actual end destination handled
entirely by `toolchain-installer'?

Only somewhat related: SRC_DIR isn't defined, so I have my version looking
like this so far (and will add toolchain-installer afterwards
after verifying the install and figuring out how things work there).

function build_osx_package() { YEAR=$(date +"%Y") MONTH=$(date +"%m")
DAY=$(date +"%d") TOOLCHAIN_VERSION="swift-2.2-SNAPSHOT-${YEAR}-${MONTH}-
${DAY}-a" ARCHIVE_DIR="${TOOLCHAIN_VERSION}-${BUILD_NUMBER}" ARCHIVE="
${TOOLCHAIN_VERSION}-osx.tar.gz" SYM_ARCHIVE="${TOOLCHAIN_VERSION}
-osx-symbols.tar.gz" BUNDLE_IDENTIFIER="org.swift.${YEAR}${MONTH}${DAY}"
DISPLAY_NAME="Swift 2.2 Development Snapshot" TOOLCHAIN_NAME="
${TOOLCHAIN_VERSION}" SRC_DIR="$(pwd)" SWIFT_SOURCE_ROOT="${SRC_DIR}"
SWIFT_BUILD_ROOT="${SRC_DIR}/build" SWIFT_INSTALLABLE_PACKAGE="${SRC_DIR}/
${ARCHIVE}" SWIFT_INSTALL_DIR="${SRC_DIR}/swift-nightly-install"
SWIFT_INSTALL_SYMROOT="${SRC_DIR}/swift-nightly-symroot"
SWIFT_TOOLCHAIN_DIR=
"/Applications/Xcode.app/Contents/Developer/Toolchains/${TOOLCHAIN_NAME}
.xctoolchain" SYMBOLS_PACKAGE="${SRC_DIR}/${SYM_ARCHIVE}"
./swift/utils/build-script --preset="buildbot_osx_package" \
install_destdir="${SWIFT_INSTALL_DIR}" \ installable_package="
${SWIFT_INSTALLABLE_PACKAGE}" \ install_toolchain_dir="
${SWIFT_TOOLCHAIN_DIR}" \ install_symroot="${SWIFT_INSTALL_SYMROOT}" \
symbols_package="${SYMBOLS_PACKAGE}" \ darwin_toolchain_bundle_identifier=
"${BUNDLE_IDENTIFIER}" \ darwin_toolchain_display_name="${DISPLAY_NAME}"
\ darwin_toolchain_xctoolchain_name="${TOOLCHAIN_NAME}" \
darwin_toolchain_version="${TOOLCHAIN_VERSION}" } build_osx_package

-Travis Tilley

On Mon, Dec 14, 2015 at 11:42 PM, Mish Awadah <mawadah@apple.com> wrote:

I’m sorry for the experience Travis, but this is certainly one of our
pain points. There’s a lot at play and we haven’t finished setting up our
public CI either. Here’s what I do locally to build a toolchain on OS X:

*function* build_osx_package() {

    YEAR=$(*date* +"%Y")

    MONTH=$(*date* +"%m")

    DAY=$(*date* +"%d")

    TOOLCHAIN_VERSION="swift-2.2-SNAPSHOT-${YEAR}-${MONTH}-${DAY}-a"

    ARCHIVE_DIR="${TOOLCHAIN_VERSION}-${BUILD_NUMBER}"

    ARCHIVE="${TOOLCHAIN_VERSION}-osx.tar.gz"

    SYM_ARCHIVE="${TOOLCHAIN_VERSION}-osx-symbols.tar.gz"

    BUNDLE_IDENTIFIER="org.swift.${YEAR}${MONTH}${DAY}"

    DISPLAY_NAME="Swift 2.2 Development Snapshot"

    TOOLCHAIN_NAME="${TOOLCHAIN_VERSION}"

    SWIFT_SOURCE_ROOT="${SRC_DIR}"

    SWIFT_BUILD_ROOT="${SRC_DIR}/build"

    SWIFT_INSTALLABLE_PACKAGE="${SRC_DIR}/${ARCHIVE}"

    SWIFT_INSTALL_DIR="${SRC_DIR}/swift-nightly-install"

    SWIFT_INSTALL_SYMROOT="${SRC_DIR}/swift-nightly-symroot"

    SWIFT_TOOLCHAIN_DIR=
"/Applications/Xcode.app/Contents/Developer/Toolchains/${TOOLCHAIN_NAME}.xctoolchain"

    SYMBOLS_PACKAGE="${SRC_DIR}/${SYM_ARCHIVE}"

    ./swift/utils/build-script --preset="buildbot_osx_package"
install_destdir="${SWIFT_INSTALL_DIR}" installable_package=
"${SWIFT_INSTALLABLE_PACKAGE}" install_toolchain_dir="${SWIF\
T_TOOLCHAIN_DIR}" install_symroot="${SWIFT_INSTALL_SYMROOT}"
symbols_package="${SYMBOLS_PACKAGE}" darwin_toolchain_bundle_identifier=
"${BUNDLE_IDENTIFIER}" darwin_toolchain_display_n\
ame="${DISPLAY_NAME}" darwin_toolchain_xctoolchain_name=
"${TOOLCHAIN_NAME}" darwin_toolchain_version="${TOOLCHAIN_VERSION}"

}

It should work. If not, please file a bug with your output and
configuration and let us know about it.

- mish

On Dec 14, 2015, at 8:32 PM, Dmitri Gribenko <gribozavr@gmail.com> wrote:

+Mishal Awadah, Mishal Shah

On Mon, Dec 14, 2015 at 8:04 PM, Travis Tilley via swift-dev < >> swift-dev@swift.org> wrote:

Is there any documentation somewhere on what the exact commands are that
the buildbot CI performs and how the packages on swift.org are created?

I've noticed that:

1) LLDB doesn't compile on either OSX or ubuntu 15 via the normal script
for various reasons (codesign on OSX, no idea on ubuntu) in the default
configuration. This seems highly counter-intuitive.

2) The buildbot presets don't have sane defaults and I can't find any
documentation on what "sane" might mean for those builds.

3) I still can't build the full suite using the following command
(which, again, may or may not be sane):

nice ./swift/utils/build-script \

  --preset=buildbot_osx_package \
  installable_package=/tmp/swift.tar.gz \
  install_destdir=/Library/Developer/Toolchains/swift-2.2-SNAPSHOT-2015-12-14/
\
  install_toolchain_dir=/Library/Developer/Toolchains/swift-2.2-SNAPSHOT-2015-12-14.xctoolchain
\
  install_symroot=/Library/Developer/Toolchains/swift-2.2-SNAPSHOT-2015-12-14/
\
  symbols_package=/tmp/swift-symbols.tar.gz \
  darwin_toolchain_bundle_identifier=org.swift.20151214 \
  darwin_toolchain_display_name=20151214 \
  darwin_toolchain_xctoolchain_name=swift
darwin_toolchain_version=swift

​All that I actually -want- to do is start xcode using a freshly built
swift toolchain. That is it. I don't even need any of the SDKs other than
OSX built. But the build directory install structure doesn't have anything
like what `​launch-with-toolchain` appears to expect...

Not sure where i'm going wrong. Please feel free to tell me to RTFM if
there is a FM to read.

-Travis Tilley

_______________________________________________
swift-dev mailing list
swift-dev@swift.org
https://lists.swift.org/mailman/listinfo/swift-dev

--
main(i,j){for(i=2;;i++){for(j=2;j<i;j++){if(!(i%j)){j=0;break;}}if
(j){printf("%d\n",i);}}} /*Dmitri Gribenko <gribozavr@gmail.com>*/

_______________________________________________
swift-dev mailing list
swift-dev@swift.org
https://lists.swift.org/mailman/listinfo/swift-dev

--
bitCycle AB | Smedjegatan 12 | 742 32 Östhammar | Sweden
http://www.bitcycle.com/
Phone: +46-73-753 24 62
E-mail: jens@bitcycle.com