Database API/Framework

Hi all-

I've been working with Swift more on Linux than macOS and am very excited to see the formation of this group. Looking at the server api workgroup's focus, I notice it's primarily low-level interaction with the operating system. When might it be a good time to bring up the possibility of creating a database-specific framework for those folks who want to work directly with Postgres, MySQL, even DB2 and Oracle; I'm thinking a JDBC-inspired framework that drivers could be written against.

Thanks,

Ron

Hi Ron:

As you say, that's outside of the scope of the work group, however I do
think its a valuable area that the various existing web framework groups
might be interested in collaborating on. Having a common, recognized
framework that the database providers or third parties would be interested
in writing database drivers for would be a huge step forward.

I know that Vapor has its Fluent ORM framework, and that its an area that
the Kitura team are looking at as well, so your post might kick-start some
wider discussion.

Chris

···

From: Ron Olson via swift-server-dev <swift-server-dev@swift.org>
To: swift-server-dev@swift.org
Date: 25/10/2016 18:54
Subject: [swift-server-dev] Database API/Framework
Sent by: swift-server-dev-bounces@swift.org

Hi all-

I've been working with Swift more on Linux than macOS and am very
excited to see the formation of this group. Looking at the server api
workgroup's focus, I notice it's primarily low-level interaction with
the operating system. When might it be a good time to bring up the
possibility of creating a database-specific framework for those folks
who want to work directly with Postgres, MySQL, even DB2 and Oracle; I'm
thinking a JDBC-inspired framework that drivers could be written
against.

Thanks,

Ron
_______________________________________________
swift-server-dev mailing list
swift-server-dev@swift.org
https://lists.swift.org/mailman/listinfo/swift-server-dev

Hi Chris,

it is a little unclear what the scope of the group is. Presumably you have some very specific things in mind?

Of course we’ve seen https://swift.org/server-apis/#focus-areas\. But what does it actually mean? :-) Looking at it:
- base networking: TCP/UDP, DNS. Covered already by libdispatch and the OS’ses?
- security: OpenSSL? or some wrapper around both, CC and OpenSSL?
- provide low level HTTP parsing: http_parser? Is anyone using something
  else?
All that seems kinda covered/standard already? Unless maybe you include non-Unix stuff like Windows.

Is this primarily for things like providing a standardised ’swiftier' OpenSSL wrapper (similar to how Objective-GCD wraps libdispatch)?
Or would that just be module maps which are shipped by default with Swift and hence can be relied on by all frameworks?

But maybe I’m just a little too impatient :->

hh

···

On 25 Oct 2016, at 20:41, Chris Bailey via swift-server-dev <swift-server-dev@swift.org> wrote:

Hi Ron:

As you say, that's outside of the scope of the work group, however I do think its a valuable area that the various existing web framework groups might be interested in collaborating on. Having a common, recognized framework that the database providers or third parties would be interested in writing database drivers for would be a huge step forward.

I know that Vapor has its Fluent ORM framework, and that its an area that the Kitura team are looking at as well, so your post might kick-start some wider discussion.

Chris

From: Ron Olson via swift-server-dev <swift-server-dev@swift.org>
To: swift-server-dev@swift.org
Date: 25/10/2016 18:54
Subject: [swift-server-dev] Database API/Framework
Sent by: swift-server-dev-bounces@swift.org

Hi all-

I've been working with Swift more on Linux than macOS and am very
excited to see the formation of this group. Looking at the server api
workgroup's focus, I notice it's primarily low-level interaction with
the operating system. When might it be a good time to bring up the
possibility of creating a database-specific framework for those folks
who want to work directly with Postgres, MySQL, even DB2 and Oracle; I'm
thinking a JDBC-inspired framework that drivers could be written
against.

Thanks,

Ron
_______________________________________________
swift-server-dev mailing list
swift-server-dev@swift.org
https://lists.swift.org/mailman/listinfo/swift-server-dev

_______________________________________________
swift-server-dev mailing list
swift-server-dev@swift.org
https://lists.swift.org/mailman/listinfo/swift-server-dev

Hey everyone,

So I’m currently working on a pure Swift Postgres driver. It has no
dependency on libpq at all. I’m parsing the Postgres Wire Protocol (V3)
myself and everything seems great. Here are some problems/issues I’m facing
that db driver implementers like myself (don’t shoot me this is my first
shot at a db driver ;) ) would like to see resolved by this swift server
group or we could use some guidance on:

1. A standard socket api, there are a few right now. Currently I’m using
IBM’s BlueSocket but really we just need one socket api that works cross
platform. From the blog post it seems like that falls under this groups
umbrella/objectives

2. A lil more guidance around timestamps + time zones. These are super
important for decent compliant swift db drivers. Can we rely on Foundation
NSDate and timezones to work well cross platform without issues here?

3. A standard lightweight crypto library to handle authentication. This is
something all database driver authors will have to deal with (MD5 hashing
etc). During the summer I had been switching between a few libraries but
would be great to have this unified, cross platform support for this. Again
I think this falls under the umbrella of this organization.

4. Coming from a diverse backend/server background, I’ve seen different
lang+environments handle sql interfaces differently. Most have left it up
to third party frameworks/ecosystems (ruby/node/python) but some like
Golang provide at least a default SQL interface that driver authors can
adapt to. I’m starting to lean towards the latter of the two but I’m by no
means and expert on anything :). This probably needs more community
engagement/proposal.

In short, for me at least, these 4 points would be great to be addressed
(the 4th one perhaps is debatable and not as urgent), for database driver
authors would greatly help a lot. Pretty much every server side swift
framework right now has their own ORM-ish libs/frameworks. That’s awesome!
My hope is we can start writing some of the drivers in pure Swift and help
all those frameworks/libs get better too.

Just my 2 cents. Perhaps this deserves a thread of its own. Perhaps it
doesn’t. I apologize in advance if this is distracting from the current
thread :)

···

--
Haris Amin
Sent with Airmail

On October 25, 2016 at 3:21:07 PM, Paulo Faria via swift-server-dev ( swift-server-dev@swift.org) wrote:

Hello, Helge.

Yes, most of the things we’ll cover are already provided by some
frameworks, but having an official library joining efforts will be great
for the community. We’ll provide full-fledged Swift libraries for the
topics mentioned. How they’re going to be developed it’s still to be
decided.

Cheers,
Paulo

On Oct 25, 2016, at 4:56 PM, Helge Heß via swift-server-dev < swift-server-dev@swift.org> wrote:

Hi Chris,

it is a little unclear what the scope of the group is. Presumably you

have some very specific things in mind?

Of course we’ve seen https://swift.org/server-apis/#focus-areas\. But what

does it actually mean? :-) Looking at it:

- base networking: TCP/UDP, DNS. Covered already by libdispatch and the

OS’ses?

- security: OpenSSL? or some wrapper around both, CC and OpenSSL?
- provide low level HTTP parsing: http_parser? Is anyone using something
else?
All that seems kinda covered/standard already? Unless maybe you include

non-Unix stuff like Windows.

Is this primarily for things like providing a standardised ’swiftier'

OpenSSL wrapper (similar to how Objective-GCD wraps libdispatch)?

Or would that just be module maps which are shipped by default with Swift

and hence can be relied on by all frameworks?

But maybe I’m just a little too impatient :->

hh

On 25 Oct 2016, at 20:41, Chris Bailey via swift-server-dev < swift-server-dev@swift.org> wrote:

Hi Ron:

As you say, that's outside of the scope of the work group, however I do

think its a valuable area that the various existing web framework groups
might be interested in collaborating on. Having a common, recognized
framework that the database providers or third parties would be interested
in writing database drivers for would be a huge step forward.

I know that Vapor has its Fluent ORM framework, and that its an area

that the Kitura team are looking at as well, so your post might kick-start
some wider discussion.

Chris

From: Ron Olson via swift-server-dev <swift-server-dev@swift.org>
To: swift-server-dev@swift.org
Date: 25/10/2016 18:54
Subject: [swift-server-dev] Database API/Framework
Sent by: swift-server-dev-bounces@swift.org

Hi all-

I've been working with Swift more on Linux than macOS and am very
excited to see the formation of this group. Looking at the server api
workgroup's focus, I notice it's primarily low-level interaction with
the operating system. When might it be a good time to bring up the
possibility of creating a database-specific framework for those folks
who want to work directly with Postgres, MySQL, even DB2 and Oracle; I'm
thinking a JDBC-inspired framework that drivers could be written
against.

Thanks,

Ron
_______________________________________________
swift-server-dev mailing list
swift-server-dev@swift.org
https://lists.swift.org/mailman/listinfo/swift-server-dev

_______________________________________________
swift-server-dev mailing list
swift-server-dev@swift.org
https://lists.swift.org/mailman/listinfo/swift-server-dev

_______________________________________________
swift-server-dev mailing list
swift-server-dev@swift.org
https://lists.swift.org/mailman/listinfo/swift-server-dev

_______________________________________________
swift-server-dev mailing list
swift-server-dev@swift.org
https://lists.swift.org/mailman/listinfo/swift-server-dev

Hello, Helge.

Yes, most of the things we’ll cover are already provided by some frameworks, but having an official library joining efforts will be great for the community. We’ll provide full-fledged Swift libraries for the topics mentioned. How they’re going to be developed it’s still to be decided.

Cheers,
Paulo

···

On Oct 25, 2016, at 4:56 PM, Helge Heß via swift-server-dev <swift-server-dev@swift.org> wrote:

Hi Chris,

it is a little unclear what the scope of the group is. Presumably you have some very specific things in mind?

Of course we’ve seen https://swift.org/server-apis/#focus-areas\. But what does it actually mean? :-) Looking at it:
- base networking: TCP/UDP, DNS. Covered already by libdispatch and the OS’ses?
- security: OpenSSL? or some wrapper around both, CC and OpenSSL?
- provide low level HTTP parsing: http_parser? Is anyone using something
else?
All that seems kinda covered/standard already? Unless maybe you include non-Unix stuff like Windows.

Is this primarily for things like providing a standardised ’swiftier' OpenSSL wrapper (similar to how Objective-GCD wraps libdispatch)?
Or would that just be module maps which are shipped by default with Swift and hence can be relied on by all frameworks?

But maybe I’m just a little too impatient :->

hh

On 25 Oct 2016, at 20:41, Chris Bailey via swift-server-dev <swift-server-dev@swift.org> wrote:

Hi Ron:

As you say, that's outside of the scope of the work group, however I do think its a valuable area that the various existing web framework groups might be interested in collaborating on. Having a common, recognized framework that the database providers or third parties would be interested in writing database drivers for would be a huge step forward.

I know that Vapor has its Fluent ORM framework, and that its an area that the Kitura team are looking at as well, so your post might kick-start some wider discussion.

Chris

From: Ron Olson via swift-server-dev <swift-server-dev@swift.org>
To: swift-server-dev@swift.org
Date: 25/10/2016 18:54
Subject: [swift-server-dev] Database API/Framework
Sent by: swift-server-dev-bounces@swift.org

Hi all-

I've been working with Swift more on Linux than macOS and am very
excited to see the formation of this group. Looking at the server api
workgroup's focus, I notice it's primarily low-level interaction with
the operating system. When might it be a good time to bring up the
possibility of creating a database-specific framework for those folks
who want to work directly with Postgres, MySQL, even DB2 and Oracle; I'm
thinking a JDBC-inspired framework that drivers could be written
against.

Thanks,

Ron
_______________________________________________
swift-server-dev mailing list
swift-server-dev@swift.org
https://lists.swift.org/mailman/listinfo/swift-server-dev

_______________________________________________
swift-server-dev mailing list
swift-server-dev@swift.org
https://lists.swift.org/mailman/listinfo/swift-server-dev

_______________________________________________
swift-server-dev mailing list
swift-server-dev@swift.org
https://lists.swift.org/mailman/listinfo/swift-server-dev

Hi,

So I’m currently working on a pure Swift Postgres driver.

nice, I’m working on one as well (currently can run only basic queries, nothing fancy).

1. A standard socket api, there are a few right now. Currently I’m using IBM’s BlueSocket but really we just need one socket api that works cross platform.

I would be interested what people are really missing here (at that basic level). GCD channels are really quite nice for that kind of stuff (scheduling IO) and are already part of Swift 3. (I personally wonder how they scale compared to something like libuv, but for now I assume they do ;-)

For making the Unix calls a little nicer Swift extensions on the Unix types are IMO pretty awesome, something along those lines:

  https://github.com/NozeIO/Noze.io/blob/master/Sources/net/SocketAddress.swift

I would avoid creating wrappers which are then just wrapped again by the higher level frameworks …

The only real thing I’m missing is SSL support. But that already depends quite a lot on how the higher level frameworks deal with streams in general.

2. A lil more guidance around timestamps + time zones. These are super important for decent compliant swift db drivers. Can we rely on Foundation NSDate and timezones to work well cross platform without issues here?

If it is part of Foundation you can assume it is supposed to work, right? :-)

3. A standard lightweight crypto library to handle authentication. This is something all database driver authors will have to deal with (MD5 hashing etc). During the summer I had been switching between a few libraries but would be great to have this unified, cross platform support for this. Again I think this falls under the umbrella of this organization.

+1 (And that should probably be part of Foundation as it is required just the same on the client side).

4. Coming from a diverse backend/server background, I’ve seen different lang+environments handle sql interfaces differently. Most have left it up to third party frameworks/ecosystems (ruby/node/python) but some like Golang provide at least a default SQL interface that driver authors can adapt to. I’m starting to lean towards the latter of the two but I’m by no means and expert on anything :). This probably needs more community engagement/proposal.

IMHO that sounds more like something the higher level frameworks (ORMs) should be concerned about (as they will define what they want to map from and how, and probably in pretty different ways). But maybe not. Personally I’d like to avoid something like JDBC as it provides little value at extra overhead.

As an example: if I have received a DispatchData from the socket containing a PG row values, I don’t want to first convert that to a JDBC or EOAdaptor record-Dictionary<Key,Any>> which is then mapped to a Customer.purchaseStatus enum. I would rather like to pass up that DispatchData to the highest level and only have it converted at the very last point and only if necessary - sometimes an ORM, but other times it may be just directly streamed into a different format, say JSON (in such cases you can often accomplish zero-copy).

My hope is we can start writing some of the drivers in pure Swift and help all those frameworks/libs get better too.

Not sure a generic ‘driver' interface will help a lot here. I think you help most by providing a SwiftPQ library which can then be integrated by the ORMs/frameworks in a way which fits best.

Anywayz, interesting to see what people come up with :-)

hh

···

On 25 Oct 2016, at 21:34, Haris Amin <aminharis7@gmail.com> wrote:

Hi Helge:

As the initial target areas are low-level function, yes, each of the
frameworks has an implementation of most of them today. Those
implementations tend to be limited in a number of ways:
C libraries are often used directly (via a modulemap)
This means that there's no general "Swift API" that developers can use -
you have to interface with the C APIs directly and deal with
Unsafe/OpaquePointers etc. Ideally we want Swift developers to only have
to write Swift code
Where a Swift API wraps a C library its generally for a specific use case
As the framework teams have a very specific use case - using it in their
framework - the APIs are very directed to what they need to do, rather
than being general purpose for a wider set of use cases
There's multiple, similar, packages
Most frameworks have their own server socket implementation (mostly by
calling Glibc and Darwin), HTTP parser implementation, etc. Not only does
this mean we can consolidate the efforts on a single implementation, but
it means that anyone wanting to add additional platform support (like
Windows) only has to do so once.

Chris

···

From: Helge Heß via swift-server-dev <swift-server-dev@swift.org>
To: swift-server-dev@swift.org
Date: 25/10/2016 19:57
Subject: Re: [swift-server-dev] Database API/Framework
Sent by: swift-server-dev-bounces@swift.org

Hi Chris,

it is a little unclear what the scope of the group is. Presumably you have
some very specific things in mind?

Of course we’ve seen https://swift.org/server-apis/#focus-areas\. But what
does it actually mean? :-) Looking at it:
- base networking: TCP/UDP, DNS. Covered already by libdispatch and the
OS’ses?
- security: OpenSSL? or some wrapper around both, CC and OpenSSL?
- provide low level HTTP parsing: http_parser? Is anyone using something
  else?
All that seems kinda covered/standard already? Unless maybe you include
non-Unix stuff like Windows.

Is this primarily for things like providing a standardised ’swiftier'
OpenSSL wrapper (similar to how Objective-GCD wraps libdispatch)?
Or would that just be module maps which are shipped by default with Swift
and hence can be relied on by all frameworks?

But maybe I’m just a little too impatient :->

hh

On 25 Oct 2016, at 20:41, Chris Bailey via swift-server-dev <swift-server-dev@swift.org> wrote:

Hi Ron:

As you say, that's outside of the scope of the work group, however I do

think its a valuable area that the various existing web framework groups
might be interested in collaborating on. Having a common, recognized
framework that the database providers or third parties would be interested
in writing database drivers for would be a huge step forward.

I know that Vapor has its Fluent ORM framework, and that its an area

that the Kitura team are looking at as well, so your post might kick-start
some wider discussion.

Chris

From: Ron Olson via swift-server-dev <swift-server-dev@swift.org>

To: swift-server-dev@swift.org
Date: 25/10/2016 18:54
Subject: [swift-server-dev] Database API/Framework
Sent by: swift-server-dev-bounces@swift.org

Hi all-

I've been working with Swift more on Linux than macOS and am very
excited to see the formation of this group. Looking at the server api
workgroup's focus, I notice it's primarily low-level interaction with
the operating system. When might it be a good time to bring up the
possibility of creating a database-specific framework for those folks
who want to work directly with Postgres, MySQL, even DB2 and Oracle; I'm

thinking a JDBC-inspired framework that drivers could be written
against.

Thanks,

Ron
_______________________________________________
swift-server-dev mailing list
swift-server-dev@swift.org
https://lists.swift.org/mailman/listinfo/swift-server-dev

_______________________________________________
swift-server-dev mailing list
swift-server-dev@swift.org
https://lists.swift.org/mailman/listinfo/swift-server-dev

_______________________________________________
swift-server-dev mailing list
swift-server-dev@swift.org
https://lists.swift.org/mailman/listinfo/swift-server-dev

I mentioned JDBC because I happened to be looking at some Java code at the time, but I really meant 'generic-interface' to provide the lowest-level-but-no-lower-than-necessary access to the database, regardless of whether it was sqlite or postgres. I think Go does a pretty good job of providing that thin layer of abstraction and it's up the driver writer to implement. After that what somebody wants to do with the results is their business.

Personally I don't like ORM layers; most of the time I'm writing complex queries that don't fit neatly into an ORM paradigm and I end up fighting with the framework.

Ron

···

On 25 Oct 2016, at 15:44, Helge Heß via swift-server-dev wrote:

Hi,

On 25 Oct 2016, at 21:34, Haris Amin <aminharis7@gmail.com> wrote:

So I’m currently working on a pure Swift Postgres driver.

nice, I’m working on one as well (currently can run only basic queries, nothing fancy).

1. A standard socket api, there are a few right now. Currently I’m using IBM’s BlueSocket but really we just need one socket api that works cross platform.

I would be interested what people are really missing here (at that basic level). GCD channels are really quite nice for that kind of stuff (scheduling IO) and are already part of Swift 3. (I personally wonder how they scale compared to something like libuv, but for now I assume they do ;-)

For making the Unix calls a little nicer Swift extensions on the Unix types are IMO pretty awesome, something along those lines:

  https://github.com/NozeIO/Noze.io/blob/master/Sources/net/SocketAddress.swift

I would avoid creating wrappers which are then just wrapped again by the higher level frameworks …

The only real thing I’m missing is SSL support. But that already depends quite a lot on how the higher level frameworks deal with streams in general.

2. A lil more guidance around timestamps + time zones. These are super important for decent compliant swift db drivers. Can we rely on Foundation NSDate and timezones to work well cross platform without issues here?

If it is part of Foundation you can assume it is supposed to work, right? :-)

3. A standard lightweight crypto library to handle authentication. This is something all database driver authors will have to deal with (MD5 hashing etc). During the summer I had been switching between a few libraries but would be great to have this unified, cross platform support for this. Again I think this falls under the umbrella of this organization.

+1 (And that should probably be part of Foundation as it is required just the same on the client side).

4. Coming from a diverse backend/server background, I’ve seen different lang+environments handle sql interfaces differently. Most have left it up to third party frameworks/ecosystems (ruby/node/python) but some like Golang provide at least a default SQL interface that driver authors can adapt to. I’m starting to lean towards the latter of the two but I’m by no means and expert on anything :). This probably needs more community engagement/proposal.

IMHO that sounds more like something the higher level frameworks (ORMs) should be concerned about (as they will define what they want to map from and how, and probably in pretty different ways). But maybe not. Personally I’d like to avoid something like JDBC as it provides little value at extra overhead.

As an example: if I have received a DispatchData from the socket containing a PG row values, I don’t want to first convert that to a JDBC or EOAdaptor record-Dictionary<Key,Any>> which is then mapped to a Customer.purchaseStatus enum. I would rather like to pass up that DispatchData to the highest level and only have it converted at the very last point and only if necessary - sometimes an ORM, but other times it may be just directly streamed into a different format, say JSON (in such cases you can often accomplish zero-copy).

My hope is we can start writing some of the drivers in pure Swift and help all those frameworks/libs get better too.

Not sure a generic ‘driver' interface will help a lot here. I think you help most by providing a SwiftPQ library which can then be integrated by the ORMs/frameworks in a way which fits best.

Anywayz, interesting to see what people come up with :-)

hh

_______________________________________________
swift-server-dev mailing list
swift-server-dev@swift.org
https://lists.swift.org/mailman/listinfo/swift-server-dev

Helge,

Expand on Chris's answer, modern languages used on server side come with abstractions native to their language. Taking the example of of HTTP protocol:
* Go comes with http package - net/http - Go Packages
* Netty (defacto standard Java net lib) comes with: io.netty.handler.codec.http (Netty API Reference (4.1.101.Final))
* etc ...
Our task should be to design an API that works well in Swift. Node's http_parser is "just" an example of an API and a possible way of accelerating the implementation of Swift's HTTP parser.

Thanks
Gregor

···

On 26 Oct 2016, at 12:39, Chris Bailey via swift-server-dev <swift-server-dev@swift.org> wrote:

Hi Helge:

As the initial target areas are low-level function, yes, each of the frameworks has an implementation of most of them today. Those implementations tend to be limited in a number of ways:
  • C libraries are often used directly (via a modulemap)
This means that there's no general "Swift API" that developers can use - you have to interface with the C APIs directly and deal with Unsafe/OpaquePointers etc. Ideally we want Swift developers to only have to write Swift code
  • Where a Swift API wraps a C library its generally for a specific use case
As the framework teams have a very specific use case - using it in their framework - the APIs are very directed to what they need to do, rather than being general purpose for a wider set of use cases
  • There's multiple, similar, packages
Most frameworks have their own server socket implementation (mostly by calling Glibc and Darwin), HTTP parser implementation, etc. Not only does this mean we can consolidate the efforts on a single implementation, but it means that anyone wanting to add additional platform support (like Windows) only has to do so once.

Chris

From: Helge Heß via swift-server-dev <swift-server-dev@swift.org>
To: swift-server-dev@swift.org
Date: 25/10/2016 19:57
Subject: Re: [swift-server-dev] Database API/Framework
Sent by: swift-server-dev-bounces@swift.org

Hi Chris,

it is a little unclear what the scope of the group is. Presumably you have some very specific things in mind?

Of course we’ve seen https://swift.org/server-apis/#focus-areas\. But what does it actually mean? :-) Looking at it:
- base networking: TCP/UDP, DNS. Covered already by libdispatch and the OS’ses?
- security: OpenSSL? or some wrapper around both, CC and OpenSSL?
- provide low level HTTP parsing: http_parser? Is anyone using something
else?
All that seems kinda covered/standard already? Unless maybe you include non-Unix stuff like Windows.

Is this primarily for things like providing a standardised ’swiftier' OpenSSL wrapper (similar to how Objective-GCD wraps libdispatch)?
Or would that just be module maps which are shipped by default with Swift and hence can be relied on by all frameworks?

But maybe I’m just a little too impatient :->

hh

> On 25 Oct 2016, at 20:41, Chris Bailey via swift-server-dev <swift-server-dev@swift.org> wrote:
>
> Hi Ron:
>
> As you say, that's outside of the scope of the work group, however I do think its a valuable area that the various existing web framework groups might be interested in collaborating on. Having a common, recognized framework that the database providers or third parties would be interested in writing database drivers for would be a huge step forward.
>
> I know that Vapor has its Fluent ORM framework, and that its an area that the Kitura team are looking at as well, so your post might kick-start some wider discussion.
>
> Chris
>
>
>
>
> From: Ron Olson via swift-server-dev <swift-server-dev@swift.org>
> To: swift-server-dev@swift.org
> Date: 25/10/2016 18:54
> Subject: [swift-server-dev] Database API/Framework
> Sent by: swift-server-dev-bounces@swift.org
>
>
>
> Hi all-
>
> I've been working with Swift more on Linux than macOS and am very
> excited to see the formation of this group. Looking at the server api
> workgroup's focus, I notice it's primarily low-level interaction with
> the operating system. When might it be a good time to bring up the
> possibility of creating a database-specific framework for those folks
> who want to work directly with Postgres, MySQL, even DB2 and Oracle; I'm
> thinking a JDBC-inspired framework that drivers could be written
> against.
>
> Thanks,
>
> Ron
> _______________________________________________
> swift-server-dev mailing list
> swift-server-dev@swift.org
> https://lists.swift.org/mailman/listinfo/swift-server-dev
>
>
> _______________________________________________
> swift-server-dev mailing list
> swift-server-dev@swift.org
> https://lists.swift.org/mailman/listinfo/swift-server-dev

_______________________________________________
swift-server-dev mailing list
swift-server-dev@swift.org
https://lists.swift.org/mailman/listinfo/swift-server-dev

_______________________________________________
swift-server-dev mailing list
swift-server-dev@swift.org
https://lists.swift.org/mailman/listinfo/swift-server-dev

Hello everyone!

I just wanted to say that I look forward to contributing to this group! I have been working on server side Swift for a while and I’ve created the project, Edge, on Github. GitHub - skylab-inc/Lightning: A Swift Multiplatform Single-threaded Non-blocking Web and Networking Framework

It uses the libdispatch (Dispatch) api to create a nodejs like asynchronous server APIs. I would love to be able to contribute some of my work to the Server APIs project.

Thank you,

Tyler

s/Expand/Expanding/

···

On 26 Oct 2016, at 14:35, Gregor Milos via swift-server-dev <swift-server-dev@swift.org> wrote:

Expand

Hi Gregor,

Expand on Chris's answer, modern languages used on server side come with abstractions native to their language.

now that is a pretty bold claim. C, C++, Java, JavaScript, Ruby and Objective-C, don’t. Python does, but few use that. But yes, Go seems to come with one … :-)

* Netty (defacto standard Java net lib)

Hu?! Since when is Netty the `defacto standard Java net lib`? Netty is a prime example of just being one of many options coming as a 3rd party package (being a good one for sure).

OK OK, nitpicking aside, so the goal this:

Our task should be to design an API that works well in Swift.

Understood. So the plan is indeed to replace the kernel of the HL frameworks with a common Swift `http` server module. A peer to https://golang.org/pkg/net/http/\. Fair enough.

Node's http_parser is "just" an example of an API and a possible way of accelerating the implementation of Swift's HTTP parser.

The http_parser is not just 'an example of an API', it is an actual implementation of a HTTP parser which can be used from Swift right away. And not a bad one :-) I guess my original question was why people invent their own instead of just using the implementations directly accessible already. Same for OpenSSL, Expat, etc. I really like the Swift-C mapping and how convenient it is to use.

Unless of course we are not talking about providing core support for HL frameworks (which provide the nice API of their choice to backend developers), but for directly providing nice APIs to backend developers as part of the Swift project. (so that they don’t have to grab say Kitura just to build a small micro service).

But I think I got you, you want to wrap that kind of thing (or rewrite in pure Swift) in a nicer, Swiftier API. Thanks for clarifying that.

hh

···

On 26 Oct 2016, at 15:35, Gregor Milos <gmilos@apple.com> wrote:

On 26 Oct 2016, at 12:39, Chris Bailey via swift-server-dev <swift-server-dev@swift.org> wrote:

Hi Helge:

As the initial target areas are low-level function, yes, each of the frameworks has an implementation of most of them today. Those implementations tend to be limited in a number of ways:
  • C libraries are often used directly (via a modulemap)
This means that there's no general "Swift API" that developers can use - you have to interface with the C APIs directly and deal with Unsafe/OpaquePointers etc. Ideally we want Swift developers to only have to write Swift code
  • Where a Swift API wraps a C library its generally for a specific use case
As the framework teams have a very specific use case - using it in their framework - the APIs are very directed to what they need to do, rather than being general purpose for a wider set of use cases
  • There's multiple, similar, packages
Most frameworks have their own server socket implementation (mostly by calling Glibc and Darwin), HTTP parser implementation, etc. Not only does this mean we can consolidate the efforts on a single implementation, but it means that anyone wanting to add additional platform support (like Windows) only has to do so once.

Chris

From: Helge Heß via swift-server-dev <swift-server-dev@swift.org>
To: swift-server-dev@swift.org
Date: 25/10/2016 19:57
Subject: Re: [swift-server-dev] Database API/Framework
Sent by: swift-server-dev-bounces@swift.org

Hi Chris,

it is a little unclear what the scope of the group is. Presumably you have some very specific things in mind?

Of course we’ve seen https://swift.org/server-apis/#focus-areas\. But what does it actually mean? :-) Looking at it:
- base networking: TCP/UDP, DNS. Covered already by libdispatch and the OS’ses?
- security: OpenSSL? or some wrapper around both, CC and OpenSSL?
- provide low level HTTP parsing: http_parser? Is anyone using something
else?
All that seems kinda covered/standard already? Unless maybe you include non-Unix stuff like Windows.

Is this primarily for things like providing a standardised ’swiftier' OpenSSL wrapper (similar to how Objective-GCD wraps libdispatch)?
Or would that just be module maps which are shipped by default with Swift and hence can be relied on by all frameworks?

But maybe I’m just a little too impatient :->

hh

On 25 Oct 2016, at 20:41, Chris Bailey via swift-server-dev <swift-server-dev@swift.org> wrote:

Hi Ron:

As you say, that's outside of the scope of the work group, however I do think its a valuable area that the various existing web framework groups might be interested in collaborating on. Having a common, recognized framework that the database providers or third parties would be interested in writing database drivers for would be a huge step forward.

I know that Vapor has its Fluent ORM framework, and that its an area that the Kitura team are looking at as well, so your post might kick-start some wider discussion.

Chris

From: Ron Olson via swift-server-dev <swift-server-dev@swift.org>
To: swift-server-dev@swift.org
Date: 25/10/2016 18:54
Subject: [swift-server-dev] Database API/Framework
Sent by: swift-server-dev-bounces@swift.org

Hi all-

I've been working with Swift more on Linux than macOS and am very
excited to see the formation of this group. Looking at the server api
workgroup's focus, I notice it's primarily low-level interaction with
the operating system. When might it be a good time to bring up the
possibility of creating a database-specific framework for those folks
who want to work directly with Postgres, MySQL, even DB2 and Oracle; I'm
thinking a JDBC-inspired framework that drivers could be written
against.

Thanks,

Ron
_______________________________________________
swift-server-dev mailing list
swift-server-dev@swift.org
https://lists.swift.org/mailman/listinfo/swift-server-dev

_______________________________________________
swift-server-dev mailing list
swift-server-dev@swift.org
https://lists.swift.org/mailman/listinfo/swift-server-dev

_______________________________________________
swift-server-dev mailing list
swift-server-dev@swift.org
https://lists.swift.org/mailman/listinfo/swift-server-dev

_______________________________________________
swift-server-dev mailing list
swift-server-dev@swift.org
https://lists.swift.org/mailman/listinfo/swift-server-dev

Additional advantage of a standardised APIs+implementation is standardised license. MIT (for NGINX http_parser) is not a big deal, but the more dependencies you're pulling into your app, the harder it's to track it all.

Some more comments inline:

Hi Gregor,

Expand on Chris's answer, modern languages used on server side come with abstractions native to their language.

now that is a pretty bold claim. C, C++, Java, JavaScript, Ruby and Objective-C, don’t. Python does, but few use that. But yes, Go seems to come with one … :-)

* Netty (defacto standard Java net lib)

Hu?! Since when is Netty the `defacto standard Java net lib`? Netty is a prime example of just being one of many options coming as a 3rd party package (being a good one for sure).

It's all about the definition of "modern" ;). More seriously though, the core of my point is that good APIs are invariably designed for the language in question, rather than foreign imports. The distribution model (compiler/stdlib/common libraries/package manager) is orthogonal. At this point I don't think it's clear how we are going to deliver the implementation of APIs designed by this work-group either. https://swift.org/server-apis/ talks about the process somewhat, but where the code belongs will largely depend on what it evolves to look like.

OK OK, nitpicking aside, so the goal this:

Our task should be to design an API that works well in Swift.

Understood. So the plan is indeed to replace the kernel of the HL frameworks with a common Swift `http` server module. A peer to https://golang.org/pkg/net/http/\. Fair enough.

Node's http_parser is "just" an example of an API and a possible way of accelerating the implementation of Swift's HTTP parser.

The http_parser is not just 'an example of an API', it is an actual implementation of a HTTP parser which can be used from Swift right away. And not a bad one :-) I guess my original question was why people invent their own instead of just using the implementations directly accessible already. Same for OpenSSL, Expat, etc. I really like the Swift-C mapping and how convenient it is to use.

Yes, it can be used from Swift and yes, it's a good parser. But the fact it's not a Swift HTTP parser is immediately obvious. We always wrap it in layer of Swift that deals with all the UnsafeMutablePointers, state management and something that exposes API conforming to https://swift.org/documentation/api-design-guidelines/\.

Hope this all makes sense.
Gregor

···

On 26 Oct 2016, at 15:39, Helge Heß via swift-server-dev <swift-server-dev@swift.org> wrote:
On 26 Oct 2016, at 15:35, Gregor Milos <gmilos@apple.com> wrote:

Unless of course we are not talking about providing core support for HL frameworks (which provide the nice API of their choice to backend developers), but for directly providing nice APIs to backend developers as part of the Swift project. (so that they don’t have to grab say Kitura just to build a small micro service).

But I think I got you, you want to wrap that kind of thing (or rewrite in pure Swift) in a nicer, Swiftier API. Thanks for clarifying that.

hh

On 26 Oct 2016, at 12:39, Chris Bailey via swift-server-dev <swift-server-dev@swift.org> wrote:

Hi Helge:

As the initial target areas are low-level function, yes, each of the frameworks has an implementation of most of them today. Those implementations tend to be limited in a number of ways:
  • C libraries are often used directly (via a modulemap)
This means that there's no general "Swift API" that developers can use - you have to interface with the C APIs directly and deal with Unsafe/OpaquePointers etc. Ideally we want Swift developers to only have to write Swift code
  • Where a Swift API wraps a C library its generally for a specific use case
As the framework teams have a very specific use case - using it in their framework - the APIs are very directed to what they need to do, rather than being general purpose for a wider set of use cases
  • There's multiple, similar, packages
Most frameworks have their own server socket implementation (mostly by calling Glibc and Darwin), HTTP parser implementation, etc. Not only does this mean we can consolidate the efforts on a single implementation, but it means that anyone wanting to add additional platform support (like Windows) only has to do so once.

Chris

From: Helge Heß via swift-server-dev <swift-server-dev@swift.org>
To: swift-server-dev@swift.org
Date: 25/10/2016 19:57
Subject: Re: [swift-server-dev] Database API/Framework
Sent by: swift-server-dev-bounces@swift.org

Hi Chris,

it is a little unclear what the scope of the group is. Presumably you have some very specific things in mind?

Of course we’ve seen https://swift.org/server-apis/#focus-areas\. But what does it actually mean? :-) Looking at it:
- base networking: TCP/UDP, DNS. Covered already by libdispatch and the OS’ses?
- security: OpenSSL? or some wrapper around both, CC and OpenSSL?
- provide low level HTTP parsing: http_parser? Is anyone using something
else?
All that seems kinda covered/standard already? Unless maybe you include non-Unix stuff like Windows.

Is this primarily for things like providing a standardised ’swiftier' OpenSSL wrapper (similar to how Objective-GCD wraps libdispatch)?
Or would that just be module maps which are shipped by default with Swift and hence can be relied on by all frameworks?

But maybe I’m just a little too impatient :->

hh

On 25 Oct 2016, at 20:41, Chris Bailey via swift-server-dev <swift-server-dev@swift.org> wrote:

Hi Ron:

As you say, that's outside of the scope of the work group, however I do think its a valuable area that the various existing web framework groups might be interested in collaborating on. Having a common, recognized framework that the database providers or third parties would be interested in writing database drivers for would be a huge step forward.

I know that Vapor has its Fluent ORM framework, and that its an area that the Kitura team are looking at as well, so your post might kick-start some wider discussion.

Chris

From: Ron Olson via swift-server-dev <swift-server-dev@swift.org>
To: swift-server-dev@swift.org
Date: 25/10/2016 18:54
Subject: [swift-server-dev] Database API/Framework
Sent by: swift-server-dev-bounces@swift.org

Hi all-

I've been working with Swift more on Linux than macOS and am very
excited to see the formation of this group. Looking at the server api
workgroup's focus, I notice it's primarily low-level interaction with
the operating system. When might it be a good time to bring up the
possibility of creating a database-specific framework for those folks
who want to work directly with Postgres, MySQL, even DB2 and Oracle; I'm
thinking a JDBC-inspired framework that drivers could be written
against.

Thanks,

Ron
_______________________________________________
swift-server-dev mailing list
swift-server-dev@swift.org
https://lists.swift.org/mailman/listinfo/swift-server-dev

_______________________________________________
swift-server-dev mailing list
swift-server-dev@swift.org
https://lists.swift.org/mailman/listinfo/swift-server-dev

_______________________________________________
swift-server-dev mailing list
swift-server-dev@swift.org
https://lists.swift.org/mailman/listinfo/swift-server-dev

_______________________________________________
swift-server-dev mailing list
swift-server-dev@swift.org
https://lists.swift.org/mailman/listinfo/swift-server-dev

_______________________________________________
swift-server-dev mailing list
swift-server-dev@swift.org
https://lists.swift.org/mailman/listinfo/swift-server-dev

Hi Tyler:

Awesome stuff. If your interested in signing up as a stakeholder - which
implies your willing to spend some time providing input and trying to
adopt the APIs that are created into your framework - you can do that by
adding your name to the group here:
        GitHub - swift-server/work-group: Historical workgroup organisation

Chris

···

From: Tyler Cloutier via swift-server-dev <swift-server-dev@swift.org>
To: swift-server-dev@swift.org
Date: 26/10/2016 04:23
Subject: [swift-server-dev] Lib dispatch and Edge
Sent by: swift-server-dev-bounces@swift.org

Hello everyone!

I just wanted to say that I look forward to contributing to this group! I
have been working on server side Swift for a while and I’ve created the
project, Edge, on Github. GitHub - skylab-inc/Lightning: A Swift Multiplatform Single-threaded Non-blocking Web and Networking Framework

It uses the libdispatch (Dispatch) api to create a nodejs like
asynchronous server APIs. I would love to be able to contribute some of my
work to the Server APIs project.

Thank you,

Tyler
_______________________________________________
swift-server-dev mailing list
swift-server-dev@swift.org
https://lists.swift.org/mailman/listinfo/swift-server-dev

Hi,

Is there any progress/activity on this front? On the Swift Server Workgroup page I see Postgres client, and there are several DB related projects around, but nothing heading towards unified DB API.

I am author of Python OLAP/multi-dimensional semantic layer toolkit Cubes and was pondering with the idea of rewriting it in Swift. The project is mostly about composing relational algebra entities through metadata and translating it into a concrete physical database implementation.

I think that we need the following set of separate APIs and Libraries:

Low Level/Adapter Database

Type: API

Similar to: Python PEP 249 -- Python Database API Specification v2.0

Functionality:

  • connection
  • execution (both synchronous and asynchronous)
  • metadata (schema) reflection
  • physical/concrete data type abstraction

Relational Algebra/Structured Query

Type: Library

Similar: SQL Alchemy Core - Expression API

Note: this is not an ORM layer and should not ever be. It is very low level without any assumptions about developer's intent, such as "magical joins". ORM, if someone wishes, might be implemented on top of it.

Functionality:

  • Expressions
  • Statements – both Query and DDL
  • Tree Rewrite Rules (for example use-case see below)
  • Data Types
  • basic DB schema metadata objects such as Table, Column

Relational Algebra to Dialect Compiler

Type: API

Functionality:

  • translation of expressions and statements to their native (usually textual) representations
  • dialect specific expression/statement rewrites, such as transparent support of Date data type in SQLite that gets translated into a String

See also:

Is there any interest going towards this direction? I am willing to contribute some of my sabbatical time (while it lasts) into the effort, if there are people willing to form a Database API group.

What do you think?

Stefan

3 Likes

hey @Stiivi this sounds interesting! as you noticed, there are few db drivers in progress but nothing like what you propose. to collect feedback i suggest you make a pitch in the server/pitches category, see sswg incubation process