Posix Module


(Helge Heß) #1

Hi,

I guess this kinda belongs here:

I wonder whether we can have a standard Posix module with all the standard Posix stuff in it to avoid the

  #if os(Linux)
    import Glibc
  #else
    import Darwin
  #endif

for things which are standardised in Posix (and hence the same, even on Windoze). I currently have an `xsys` module to alias the definitions, but this is kinda crap :slight_smile:

  https://github.com/NozeIO/Noze.io/tree/master/Sources/xsys

Or is there a better way to do this already?

Thanks,
  Helge


(Daniel Dunbar) #2

Given the Glibc overlay lives in Swift, this probably is more appropriate for swift-dev rather than the server APIs project specifically.

- Daniel

···

On Oct 28, 2016, at 8:51 AM, Helge Heß via swift-server-dev <swift-server-dev@swift.org> wrote:

Hi,

I guess this kinda belongs here:

I wonder whether we can have a standard Posix module with all the standard Posix stuff in it to avoid the

#if os(Linux)
   import Glibc
#else
   import Darwin
#endif

for things which are standardised in Posix (and hence the same, even on Windoze). I currently have an `xsys` module to alias the definitions, but this is kinda crap :slight_smile:

https://github.com/NozeIO/Noze.io/tree/master/Sources/xsys

Or is there a better way to do this already?

Thanks,
Helge

_______________________________________________
swift-server-dev mailing list
swift-server-dev@swift.org
https://lists.swift.org/mailman/listinfo/swift-server-dev


(Michael Gottesman) #3

Actually swift-evolution. There was a thread about it already:

https://lists.swift.org/pipermail/swift-evolution/Week-of-Mon-20160307/012085.html

It was only a draft but people IIRC people viewed it favorably (don't quote me on this though, my memory might be wrong). Perhaps no one got behind implementing it? (again my memory is gone).

Michael

···

On Oct 28, 2016, at 3:56 PM, Daniel Dunbar via swift-server-dev <swift-server-dev@swift.org> wrote:

Given the Glibc overlay lives in Swift, this probably is more appropriate for swift-dev rather than the server APIs project specifically.

- Daniel

On Oct 28, 2016, at 8:51 AM, Helge Heß via swift-server-dev <swift-server-dev@swift.org> wrote:

Hi,

I guess this kinda belongs here:

I wonder whether we can have a standard Posix module with all the standard Posix stuff in it to avoid the

#if os(Linux)
  import Glibc
#else
  import Darwin
#endif

for things which are standardised in Posix (and hence the same, even on Windoze). I currently have an `xsys` module to alias the definitions, but this is kinda crap :slight_smile:

https://github.com/NozeIO/Noze.io/tree/master/Sources/xsys

Or is there a better way to do this already?

Thanks,
Helge

_______________________________________________
swift-server-dev mailing list
swift-server-dev@swift.org
https://lists.swift.org/mailman/listinfo/swift-server-dev

_______________________________________________
swift-server-dev mailing list
swift-server-dev@swift.org
https://lists.swift.org/mailman/listinfo/swift-server-dev


(Johannes Weiss) #4

Hi,

Well, correct handling of POSIX functions in Swift is unfortunately a bit harder than just re-exporting Glibc/Darwin depending on the platform.

Errno is an issue. In _many_ libraries I have seen code like this:

let rv = some_syscall(params)
if tv != 0 {
    throw POSIXError(code: errno)
}

this looks correct but technically it isn't. There's no guarantee in Swift that between calling 'some_syscall' and capturing 'errno', errno isn't overridden. ARC might insert 'release' calls at any point. Unfortunately, release calls might cause 'deinit's to be run and they might use syscalls (which could override errno).

So yes, just re-exporting Glibc/Darwin is easy but using these functions correctly is hard.

Another assumption I have seen in Swift libraries is assuming that a successful system call resets errno to 0, that's not true either.

But as others have said, this might be a swift-{users,evolution} question?

Cheers,
  Johannes

···

On 28 Oct 2016, at 4:51 pm, Helge Heß via swift-server-dev <swift-server-dev@swift.org> wrote:

Hi,

I guess this kinda belongs here:

I wonder whether we can have a standard Posix module with all the standard Posix stuff in it to avoid the

#if os(Linux)
   import Glibc
#else
   import Darwin
#endif

for things which are standardised in Posix (and hence the same, even on Windoze). I currently have an `xsys` module to alias the definitions, but this is kinda crap :slight_smile:

https://github.com/NozeIO/Noze.io/tree/master/Sources/xsys

Or is there a better way to do this already?

Thanks,
Helge

_______________________________________________
swift-server-dev mailing list
swift-server-dev@swift.org
https://lists.swift.org/mailman/listinfo/swift-server-dev


(Tyler Cloutier) #5

I think this goes beyond just that proposal, though. It’s not just a problem of conditionally importing both frameworks as there is sometimes a slight mismatch in the API. This results in me having to do unfortunate things like this:

https://github.com/SwiftOnEdge/Edge/blob/8bdbb4555dd49ecfad51498234f4d91cdf07d4aa/Sources/POSIXExtensions/Socket.swift#L18

A standard POSIX Swift API would reduce some serious pain points in developing for the server. All of this translational work from C APIs to Swift APIs is happening in every project that does low level I/O. Ideally it would be as thin a wrapper as is possible while still making the APIs feel Swift-y.

Can you imagine not having to mess about with errno anymore? I dream of the day.

Tyler

···

On Oct 29, 2016, at 8:32 PM, Michael Gottesman via swift-server-dev <swift-server-dev@swift.org> wrote:

Actually swift-evolution. There was a thread about it already:

https://lists.swift.org/pipermail/swift-evolution/Week-of-Mon-20160307/012085.html

It was only a draft but people IIRC people viewed it favorably (don't quote me on this though, my memory might be wrong). Perhaps no one got behind implementing it? (again my memory is gone).

Michael

On Oct 28, 2016, at 3:56 PM, Daniel Dunbar via swift-server-dev <swift-server-dev@swift.org> wrote:

Given the Glibc overlay lives in Swift, this probably is more appropriate for swift-dev rather than the server APIs project specifically.

- Daniel

On Oct 28, 2016, at 8:51 AM, Helge Heß via swift-server-dev <swift-server-dev@swift.org> wrote:

Hi,

I guess this kinda belongs here:

I wonder whether we can have a standard Posix module with all the standard Posix stuff in it to avoid the

#if os(Linux)
import Glibc
#else
import Darwin
#endif

for things which are standardised in Posix (and hence the same, even on Windoze). I currently have an `xsys` module to alias the definitions, but this is kinda crap :slight_smile:

https://github.com/NozeIO/Noze.io/tree/master/Sources/xsys

Or is there a better way to do this already?

Thanks,
Helge

_______________________________________________
swift-server-dev mailing list
swift-server-dev@swift.org
https://lists.swift.org/mailman/listinfo/swift-server-dev

_______________________________________________
swift-server-dev mailing list
swift-server-dev@swift.org
https://lists.swift.org/mailman/listinfo/swift-server-dev

_______________________________________________
swift-server-dev mailing list
swift-server-dev@swift.org
https://lists.swift.org/mailman/listinfo/swift-server-dev


(Johannes Weiss) #6

ARC might insert 'release' calls at any point.

Are you sure this is true? `errno` is stored at the TLS. The only way "ARC might insert 'release' calls at any point.” is if that point is in another thread. And if that `deinit` is called in another thread it won’t affect the `errno` of the current thread. What am I missing here?

I did speak to Joe Groff who's working on the Swift compiler team before about that and he confirmed that there are issues regarding that. I know `errno` is thread-safe (because it's stored in thread-local storage) but the problem can manifest even single-threaded code.

Let's assume this code:

--- SNIP ---
public class SomeClass {
    public let someValue: Int = 42
    deinit {
        /* some failing syscall */
    }
}

let x = SomeClass()
let rv = write(x.someValue, nil, 0)
let errnoSave = errno
if rv != 0 {
    raise POSIXError(code: errnoSave)
}
--- SNAP ---

The instance `x` isn't needed anymore as soon as we have called write. So the compiler might put this code in there:

--- SNIP ---
let x = SomeClass()
let rv = write(x.someValue, nil, 0)
/* ARC generated */ x.release()
let errnoSave = errno /* wrong errno value :frowning: */
if rv != 0 {
    raise POSIXError(code: errnoSave)
}
--- SNAP ---

and this would be causing a problem as x.release() would trigger the deinit.

Now I'm not saying this will happen every single time but it might happen depending on your code, your Swift compiler version and your optimisation levels.

Cheers,
  Johannes

···

On 30 Oct 2016, at 6:45 pm, Paulo Faria <paulo@zewo.io> wrote:

On Oct 30, 2016, at 4:11 PM, Johannes Weiß via swift-server-dev <swift-server-dev@swift.org> wrote:

Hi,

Well, correct handling of POSIX functions in Swift is unfortunately a bit harder than just re-exporting Glibc/Darwin depending on the platform.

Errno is an issue. In _many_ libraries I have seen code like this:

let rv = some_syscall(params)
if tv != 0 {
  throw POSIXError(code: errno)
}

this looks correct but technically it isn't. There's no guarantee in Swift that between calling 'some_syscall' and capturing 'errno', errno isn't overridden. ARC might insert 'release' calls at any point. Unfortunately, release calls might cause 'deinit's to be run and they might use syscalls (which could override errno).

So yes, just re-exporting Glibc/Darwin is easy but using these functions correctly is hard.

Another assumption I have seen in Swift libraries is assuming that a successful system call resets errno to 0, that's not true either.

But as others have said, this might be a swift-{users,evolution} question?

Cheers,
Johannes

On 28 Oct 2016, at 4:51 pm, Helge Heß via swift-server-dev <swift-server-dev@swift.org> wrote:

Hi,

I guess this kinda belongs here:

I wonder whether we can have a standard Posix module with all the standard Posix stuff in it to avoid the

#if os(Linux)
import Glibc
#else
import Darwin
#endif

for things which are standardised in Posix (and hence the same, even on Windoze). I currently have an `xsys` module to alias the definitions, but this is kinda crap :slight_smile:

https://github.com/NozeIO/Noze.io/tree/master/Sources/xsys

Or is there a better way to do this already?

Thanks,
Helge

_______________________________________________
swift-server-dev mailing list
swift-server-dev@swift.org
https://lists.swift.org/mailman/listinfo/swift-server-dev

_______________________________________________
swift-server-dev mailing list
swift-server-dev@swift.org
https://lists.swift.org/mailman/listinfo/swift-server-dev


(Helge Heß) #7

Hi Johannes,

So yes, just re-exporting Glibc/Darwin is easy but using these functions correctly is hard.

yes, all that is understood but it has absolutely no relevance to the namespace under which those "hard-to-use-functions" are exposed?

All I :slight_smile: ask for is that those Posix or C99 funds are exposed under a common name whether on Linux, Darwin, Windoze or BeOS.

Another assumption I have seen in Swift libraries is assuming that a successful system call resets errno to 0, that's not true either.

File a bug report against the respective libs. Of course this is not true and never has been true in C either.

But as others have said, this might be a swift-{users,evolution} question?

It probably is, so are sockets, encryption and database access :slight_smile: I think you need to start somewhere, and Swift Server is the one place which has to deal with this specific issue right now.

hh

···

On 30 Oct 2016, at 19:11, Johannes Weiß <johannesweiss@apple.com> wrote:


(Helge Heß) #8

I think this goes beyond just that proposal, though. It’s not just a problem of conditionally importing both frameworks as there is sometimes a slight mismatch in the API. This results in me having to do unfortunate things like this:

https://github.com/SwiftOnEdge/Edge/blob/8bdbb4555dd49ecfad51498234f4d91cdf07d4aa/Sources/POSIXExtensions/Socket.swift#L18

I wouldn’t go that far, I mean this is just a struct. There is nothing wrong with not using the convenience initialiser and doing a `var address = addr()` and then initialising the values. No portability issue here (BTW this worked fine before, SE-0060 b0rked it, maybe we are getting the old struct-convenience-initializer behaviour back later, I’m kinda counting on it :-).

On a higher level you wouldn’t want to access the ai_* values directly anyways (but rather something which gives you a string value, etc). But I wouldn’t even put those wrappers I have in my `xsys` module into that Swift `Posix` module. This one should be just the Posix API exposed, same like in C. The latter may be more like an extra `SwiftyPosix` module.

Is anyone else interested in having a module which enhances the Posix/CLib structs to be swiftyer? I think I currently have some code for socket addresses, the mentioned addr and for time value stuff (time_t, timespec, timeval).

A standard POSIX Swift API would reduce some serious pain points in developing for the server. All of this translational work from C APIs to Swift APIs is happening in every project that does low level I/O. Ideally it would be as thin a wrapper as is possible while still making the APIs feel Swift-y.

I wouldn’t even make it a wrapper. Just expose the Posix/C-stdlib subsystems under a common name. We could still do `#if linux` for cases where it is really necessary. (it shouldn’t be for the regular Posix socket demo doing a connect(), send(), recv() and close() …).

The platform which is a bit special wrt Posix is Windows. That may indeed require some renaming (all the socket stuff being prefixed with WSA etc). Though a higher level socket lib may want to do Windows differently anyways, not sure.

Can you imagine not having to mess about with errno anymore? I dream of the day.

Well, yes. But that should really live at a higher level. I was just talking about the

  #if os(Linux)
    import Glibc
    public let connect = Glibc.connect
  #else
    import Darwin
    public let connect = Darwin.connect
  #endif

… which is just ridiculous :slight_smile:

My understanding is that there likely wouldn’t be a big resistance towards having a common Posix/CLib module. There ‘just’ needs to be someone implementing a pull request for the Swift codebase (aka do the work).
That’s not me, easier for me to just do the funky mappings :slight_smile:

hh

P.S.: I think the discussion kinda belongs here because server-side is the main use case for X-platform stuff until the Android port begins to fly ;->

···

On 30 Oct 2016, at 07:29, Tyler Cloutier via swift-server-dev <swift-server-dev@swift.org> wrote:


(Johannes Weiss) #9

Hi Helge,

[...]

So yes, just re-exporting Glibc/Darwin is easy but using these functions correctly is hard.

yes, all that is understood but it has absolutely no relevance to the namespace under which those "hard-to-use-functions" are exposed?

All I :slight_smile: ask for is that those Posix or C99 funds are exposed under a common name whether on Linux, Darwin, Windoze or BeOS.

Understood but that topic is already covered on swift-evolution. I'm more interested in making those functions usable correctly in Swift. Right now, "hard-to-use function" is an understatement IMHO, I have yet to see a guaranteed correct use of `errno` in any Swift open-source library but that should be discussed on swift-{evolution, users}.

Another assumption I have seen in Swift libraries is assuming that a successful system call resets errno to 0, that's not true either.

File a bug report against the respective libs. Of course this is not true and never has been true in C either.

Doing that. And, yes, that errno isn't guaranteed to be set to 0 on success has always been true in C. But capturing the correct value of errno is very straightforward in C and it's definitely not in Swift.

But as others have said, this might be a swift-{users,evolution} question?

It probably is, so are sockets, encryption and database access :slight_smile: I think you need to start somewhere, and Swift Server is the one place which has to deal with this specific issue right now.

Not sure if I fully agree. I see two areas here:

1) making the basic OS functionality available to Swift. IMHO that covers being able to call the OS's syscalls and being able to deal with the errors in a robust way. That for me belongs to swift-evolution but this work group could for example come up with proposals.

2) building libraries on top of these basic primitives. There's a bit more opinion involved here on how to do things. For example obvious questions about the programming model depending if based on Dispatch (DisptchIO or DispatchSource), one of the CSP libraries (eg. lib{dill, mill, venice}), blocking IO ;), or something else. But we should try to not bake too much opinion in these basic building blocks because that might make them incompatible with layers further up (like Zewo, Vapor, Kitura).

  Also we have to consider here that right now DispatchIO/DispatchSource is probably a reasonable place to start as Dispatch comes with Swift and a Swift API. But when (in maybe 1.5+ years) Swift gains a memory and concurrency model, Dispatch might not be the best way of doing things anymore.
So I think it's too early to answer these questions in the Swift standard library right now. But in this work group we can start to discuss and implement(!) basic building blocks that help building server programs right now (or in the near future). These efforts would then make it a) easier to develop server software right now and b) reduce duplication in the existing server frameworks.

  Summing up, I think this workgroup could cover the layer above syscalls focussing on the areas that common server software needs (eg. networking, security, HTTP parsing, ...). Hopefully the layer can be as free as possible of decisions that massively influence the stack on top. It'd be great to see Vapor, Zewo and Kitura adopting these lower level primitives. And that should be possible without changing their overall architecture.

I hope that makes some sense. And for the record: The scope for the discussions/libraries that come out of this doesn't need to be massive. Personally, I'd be very happy if we'd produce some very few basic libraries that are used as a common ground to bootstrap network applications.

All the best,
  Johannes


(Helge Heß) #10

Hi Johannes,

All I :slight_smile: ask for is that those Posix or C99 funds are exposed under a common name whether on Linux, Darwin, Windoze or BeOS.

Understood but that topic is already covered on swift-evolution.

OK, great.

I'm more interested in making those functions usable correctly in Swift. Right now, "hard-to-use function" is an understatement IMHO

IMHO you are exaggerating quite a bit, but OK :slight_smile:

It probably is, so are sockets, encryption and database access :slight_smile: I think you need to start somewhere, and Swift Server is the one place which has to deal with this specific issue right now.

Not sure if I fully agree. I see two areas here:

1) making the basic OS functionality available to Swift. IMHO that covers being able to call the OS's syscalls and being able to deal with the errors in a robust way. That for me belongs to swift-evolution but this work group could for example come up with proposals.

OK.

2) building libraries on top of these basic primitives. There's a bit more opinion involved here on how to do things. For example obvious questions about the programming model depending if based on Dispatch (DisptchIO or DispatchSource), one of the CSP libraries (eg. lib{dill, mill, venice}), blocking IO ;), or something else. But we should try to not bake too much opinion in these basic building blocks because that might make them incompatible with layers further up (like Zewo, Vapor, Kitura).

I think I completely agree with this. As mentioned I also think that this means it would be so low level, it essentially is little more than 'making the basic OS functionality available to Swift’. Similar to what has been done to CoreGraphics or GCD APIs.

Presumably something everyone is fighting with is TLS, primarily due to the ‘nice’ API of the relevant library :slight_smile: Having a nice Swift TLS API which doesn’t tie into specific transports would be really great (actually having one for C would be great too :-).

Also we have to consider here that right now DispatchIO/DispatchSource is probably a reasonable place to start as Dispatch comes with Swift and a Swift API. But when (in maybe 1.5+ years) Swift gains a memory and concurrency model, Dispatch might not be the best way of doing things anymore.

I was wondering about this as well. It may not make sense to invest too much into modelling something bigger when such language changes are on the horizon?
Don’t know, interested to see what people come up with :slight_smile:

hh

···

On 31 Oct 2016, at 10:36, Johannes Weiß <johannesweiss@apple.com> wrote:


(Johannes Weiss) #11

Hi Helge,

FWIW, I posted to swift-evolution:
https://lists.swift.org/pipermail/swift-evolution/Week-of-Mon-20161031/028627.html

Cheers,
  johannes

···

On 1 Nov 2016, at 10:41, Helge Heß via swift-server-dev <swift-server-dev@swift.org> wrote:

Hi Johannes,

On 31 Oct 2016, at 10:36, Johannes Weiß <johannesweiss@apple.com> wrote:

All I :slight_smile: ask for is that those Posix or C99 funds are exposed under a common name whether on Linux, Darwin, Windoze or BeOS.

Understood but that topic is already covered on swift-evolution.

OK, great.

I'm more interested in making those functions usable correctly in Swift. Right now, "hard-to-use function" is an understatement IMHO

IMHO you are exaggerating quite a bit, but OK :slight_smile:

It probably is, so are sockets, encryption and database access :slight_smile: I think you need to start somewhere, and Swift Server is the one place which has to deal with this specific issue right now.

Not sure if I fully agree. I see two areas here:

1) making the basic OS functionality available to Swift. IMHO that covers being able to call the OS's syscalls and being able to deal with the errors in a robust way. That for me belongs to swift-evolution but this work group could for example come up with proposals.

OK.

2) building libraries on top of these basic primitives. There's a bit more opinion involved here on how to do things. For example obvious questions about the programming model depending if based on Dispatch (DisptchIO or DispatchSource), one of the CSP libraries (eg. lib{dill, mill, venice}), blocking IO ;), or something else. But we should try to not bake too much opinion in these basic building blocks because that might make them incompatible with layers further up (like Zewo, Vapor, Kitura).

I think I completely agree with this. As mentioned I also think that this means it would be so low level, it essentially is little more than 'making the basic OS functionality available to Swift’. Similar to what has been done to CoreGraphics or GCD APIs.

Presumably something everyone is fighting with is TLS, primarily due to the ‘nice’ API of the relevant library :slight_smile: Having a nice Swift TLS API which doesn’t tie into specific transports would be really great (actually having one for C would be great too :-).

Also we have to consider here that right now DispatchIO/DispatchSource is probably a reasonable place to start as Dispatch comes with Swift and a Swift API. But when (in maybe 1.5+ years) Swift gains a memory and concurrency model, Dispatch might not be the best way of doing things anymore.

I was wondering about this as well. It may not make sense to invest too much into modelling something bigger when such language changes are on the horizon?
Don’t know, interested to see what people come up with :slight_smile:

hh

_______________________________________________
swift-server-dev mailing list
swift-server-dev@swift.org
https://lists.swift.org/mailman/listinfo/swift-server-dev