Custom String Interpolation and Compile-time Interpretation applied to Logging


(Vadim Shpakovski) #15

Sorry, but the “Improving OS Log” part in the topic inspires us to think about a ‘fixed’ os_log. As this function is a part of the Standard Library, it seems like you will modify it anyway :innocent: Thanks.


(Ben Cohen) #16

Yeah, sorry, that is misleading and that's why I posted to try and redirect. Maybe we can retitle the thread. The goal here is to talk about the design of the language feature.

os_log is not part of the standard library.

edit: to flesh this out a bit – some component parts of what is needed for a logging implementation should probably make it into the standard library, but how Apple's specific technology for logging on devices etc would be at a level above this. For example, there are also discussions going on in the server work group about logging and we would ideally share some stuff there. But the goal of this thread is to talk about what language features we need in order to start building up that stack.


(Ravi Kandhadai Madhavan) #17
  1. The naming doesn’t seem quite Swifty. It seems like osLog should be OSLog, and some of the other API be unabbreviated. e.g. msg = message.

Thanks for pointing that out and we will evolve the naming conventions and syntax to be more Swifty. Just to clarify: osLog is a free function as described here. It is not a type. However, it could be made a method or, in the future, could be abstracted by a more general logging API.

  1. Will the os_log API remain visible at the same time as the new API? It would be great if we could separate the two to minimize confusion.

That is a good suggestion. While os_log should be a part of the ABI, it certainly could be made unavailable or, at least, discouraged when the new APIs become available.

  1. Will these improvements be applied to the other os_ APIs?

Yes.


(Nobody1707) #18

I didn't realize that the "constexpr" support would be so powerful so soon. This makes it seem like it'd be possible to implement a fmt style printing library in Swift 5 (or whichever version is getting compile time evaluation), which is exciting. String interpolation gets a little awkward when you want to print formatted numbers. Right now the nicest way is to use String(format:) or vprintf().


(Joe Groff) #19

Even without the constant evaluation part, it is possible now to extend String.Interpolation with APIs for different kinds of formatting in interpolation segments. @Erica_Sadun had some good examples in a recent blog post: https://ericasadun.com/2018/12/12/the-beauty-of-swift-5-string-interpolation/ Along the same lines of what she shows for Optional, you could add appendInterpolation(radix:precision:width:) for printf-like control of numeric formatting too.


(Ravi Kandhadai Madhavan) #20

I'm confused as to why this proposal is coming up.

This proposal is posted here for the following reasons:

(a) We think the ideas and the implementation presented here could be useful in many other contexts. E.g. one of the limitations that prevented some libraries from adopting string interpolation was that they wanted the ability to split the interpolation into static and dynamic components, and compile/optimize them differently. In fact, the design of the new custom string interpolation protocol was partly motivated by such use cases. What this proposal shows is that custom string interpolation can be combined with compile-time interpretation feature to obtain all benefits of a static format string and a varargs pair.

(b) Though os_log is not a general-purpose logging system, it is a highly sophisticated logging system, with asynchronous message construction, privacy qualifiers and custom formatting. Any logging system that shares some of these features would likely face similar challenges. Ideally, we would like to enable developers implementing other logging system also benefit from some of the abstractions and optimizations we are designing for this work. This is one of the motivations for using a general-purpose compile-time interpreter, and also for aspiring to make the implementation of types like PackedLogMsg visible and available to everyone.
(See also @Ben_Cohen's comments in this thread.)

(c) Finally, we really want to share with the community the new powerful, compiler features that we plan to bring in and provide some context and motivation for that.

I'm talking about the context of "I've shipped an app to customers and they're trying to request help via a support feature in the app" ← I can't programmatically ask for logs to attach to that support request.

I will try redirect this concern to the right team.


(Ravi Kandhadai Madhavan) #21

Yes.


(Ravi Kandhadai Madhavan) #22

The names of the APIs would likely evolve. But, here osLog is meant to be a free function that logs to Apple's unified logging system. It not a general-purpose logging API for Swift, which is an orthogonal line of work.


(Ravi Kandhadai Madhavan) #23

That is an interesting question and it is relevant to this proposal. Ideally, we would like to support this. But this is not something the proposal enables out of the box. There are some limitations coming from the logging system itself that complicates wrapping logging calls (as you had mentioned). Furthermore, the optimization proposed here causes a few problems to doing this. (There are certain hacky workarounds like using @_transparent etc. But, it would not be a very desirable user model and will have its own limitation.) Therefore, at this point I do not have a concrete answer to this. But, this is something quite in sync with the overall extensibility aspect that we are trying to address.


(Paul Solt) #24

My thoughts exactly, thanks @davedelong

At first when I looked into os_log() I thought it would be useful, but there's no way I'm going to do sysdiagnoise to get logs back from users.

There should be some kind of logging solution built into the SDK/platform/library/etc that is actually useful for a developer to integrate and easily get logs to view.

All of the features of os_log() are cool, and this new string interpolation definitely makes it way more approachable (Swifty).

BUT it still doesn't address my biggest reason not to use it at all. It's not easy to get logs back from users.


(Russ Bishop) #25

I think this is an excellent proposal and I'm also looking forward to the compile-time interpreter landing.

Regardless of the merits of os_log, this demonstrates how framework authors can build powerful abstractions that make really nice to use APIs that give us safety at compile time while melting away at runtime.

Consider this a big +1 from me for exploring this area further.


(Gavin Eadie) #26

I'm a lurker here but, in past lives, have implemented some, and used other, ways of generating human readable text by the aggregation of static and dynamic values via a 'templating' syntax. I think this is the core of this proposal so I wanted to add a thought, or two, about the needs I've had of such implementations.

.. The final product that is made visible (the text emitted into the logging system in the osLog case) may be used in other ways. It could be user-visible alert text, written to storage, a Tweet, etc, so I would encourage decoupling logging from this proposal. If osLog has special needs that limit the flexibility of the feature, I'd worry.

.. Syntax for conditionals: emitting alternative texts depending on a boolean value? One I used a lot is emitting the right word for zero, singular or plural occurrences of a value "there {cowCount, "are no cows", "is only one cow", "are too many cows"}." ..

.. With delayed binding schemes like this, it is very easy to throw the kitchen sink into the mix. I'd keep the core implementation sparse and offer a generous extension mechanism. Custom types are catered for, could localization, or my conditional idea above, be custom extensions, for example?

I'm going to look at the various interpolation/interpretation proposals and play with the code more. Please pardon me if my above remarks are off base here, or already taken into consideration.


(Karl) #27

Just wanted to add: this idea of hoovering up all information is the "old way" of thinking. Users also have privacy rights, which these days are backed by laws like the EU's GDPR. If your logs contain private information, you need to consider who has access to these logs and how long they are stored for.

If you're not considering this in your logging right now, you definitely should.


(Matthew Johnson) #28

I think it's perfectly acceptable to provide an in-app feature that allows users to submit logs along with a bug report as long as there is disclosure on the nature of the information contained in the logs. It is the user themselves who submits the log after all. I think this is a use case that can be addressed while still respecting the privacy of the user.


(Gavin Eadie) #29

Talking to myself .. I've now read a lot of the background to this proposal, looked at code and messed around in a Playground, and been somewhat mortified to appreciate the immense scope of the work. My remarks above are naive in comparison and are best passed over. I live and learn.


(Karl) #30

So, looking at the prototype:

I feel that the interpolations could read more nicely.

  • The private: argument label gets in the way. I would prefer to put the value first, and the privacy modifier afterwards.

    // Prototype.
    osLog("Login time: \(private: currentTime, .time) ms")
    // Shuffled.
    osLog("Login time: \(time: currentTime, .private) ms")
    

    (Making privacy an enum might also allow us to specify more fine-grained permissions controls one day, such as hashing or truncating the private values when exported).

  • Rather than make the formatting options an enum, have you considered adding more addInterpolation overloads?

    // Prototype.
    osLog("Network delay: \(delay, .decimal(2)) ms")
    osLog("Header: \(flags, .hex)")
    // With overloading.
    osLog("Network delay: \(delay, decimalPlaces: 2) ms")
    osLog("Header: \(hex: flags)")
    
  • I'm not sure about the name osLog. It seems like this would be better expressed in Swift using namespacing - e.g. OS.log(...). Similarly, PackedLogMsg -> OS.LogMessage

Otherwise, this is a very substantial improvement to the os_log API :+1:. Thanks for sharing your progress at this stage.


(Ravi Kandhadai Madhavan) #31

Thanks for raising some interesting points. I would like to use this context to explain some of the extensibility aspects of the proposal (possibly answering some of your questions). Particularly, what this proposal offers currently and some future directions to explore (as a community).

I want to bring out the different levels of extensibility here (in the increasing order of their generality):

Level 1: Customizing log messages passed to osLog: This is kind of what the proposal enables as such, as it is narrowly focused on just osLog. You can customize the string interpolation methods of PackedLogMsg as you like e.g. to append tags/prefixes/suffices etc. to messages. The updations can use static, interpretable state. For example, you can choose between singular/plural based on the count of something (as long as the count is compile-time constant). If you look at the format string construction, it is already quite complicated. Having said that, all of this is limited by two things: (a) the interpretable language fragment (which doesn't include impure functions with side-effects, globals, classes, exceptions etc), and (b) the stdlib APIs supported by the interpreter. Note that the interpreter would only support a tiny fraction of string and array operations to accomplish the goals of this proposal. So doing advanced processing on strings will need extensions to the interpreter. (Often complex stdlib APIs are not interpretable and require special modeling inside the interpreter.)

Level 2: Customizing the backend and making it work with other loggers: The next level of extension would be to be able to use a custom logger instead of the unified-logging system, which in the simplest case, can just be writing to a log file. This is kind of plausible with this proposal (But is not the focus, at least until the basic functionality is accomplished). This can be achieved, if you make the APIs that does the write to file accept a PackedLogMsg (which could well be customized to not create a byte buffer and format string but instead, say, just convert everything to string). E.g. you can create a function

func writeToLogFile(msg: PackedLogMsg) {
    msg.getAsString().writeToFile(...)
}

extension PackedLogMsg {
   func getAsString() -> String {
      // construct a string using the tracked format string and "argument encoders" tracked by PackedLogMsg.
   }
}

What you get out of this implementation is that PackedLogMsg is constructed at compile-time and only getAsString() and writing to file has to happen at runtime. The code for getAsString is a bit complex but it can be constructed with the public members of PackedLogMsg (viz. the format string property and encode method). Ideally, the extension may want to write to file asynchronously and also construct the full message asynchronously (like osLog) for performance. This is doable with this proposal, but again has the limitation of being within interpretable/optimizable language fragment. This is how we think the proposed design can benefit/inter-operate with server-side logging (in the future). (See Server-side logging discussions: Server Logging API)

From here, I am going to get a bit futuristic and present some ideas. (They are completely outside the scope of this proposal) and I am just sharing ideas for potential extensions/generalizations of this idea, and the interpreter more broadly.

c) Level 3: A general InterpretableStringInterpolation struct: There could be many other cases where one may want to optimize a custom string interpolation by interpreting constant parts of it. The compiler can expose a general custom string interpolation struct on which it will run the interpreter and try to fold its uses, independent of what it is used for. It could be for logging or for some other purpose. An open question/challenge here is how to design the properties and interfaces of the struct to make it general enough. (This is a non-trivial challenge especially when some parts of the interpolation are constant and some are not as in the osLog case.) PackedLogMsg can then be seen one instance of this.

d) Level 4: A general-purpose "constexpr" for Swift: Such a language feature would enable Swift users invoke the interpreter and ask it to statically fold a value. This is a major feature and has several open questions/challenges. (A remark: the original compile-time interpretation pitch proposed a static-assert construct, which can also invoke the interpreter on arbitrary code but does not fold the result.)

The experience we gain and the challenges we address with this work may help inform a larger effort to evolve such user-visible language features as a part of Swift. This is completely outside the scope of this work.


(Jean-Daniel) #32

AFAIK, signpost is not yet available in Swift because it relies on storing the static C string into a custom Section in the generated object file. To solve this, Apple should either change the way signpost works (by using the standard C string section), or we should update Swift to support string static C string into a custom section.


(Ravi Kandhadai Madhavan) #33

@Jean-Daniel os_signpost is available in Swift and has an almost similar interface to os_log - taking a format string and varargs: https://developer.apple.com/documentation/os/3019241-os_signpost. (An article on using it: https://www.swiftbysundell.com/daily-wwdc/getting-started-with-signposts)


(Jean-Daniel) #34

My bad. I mixed it up with os_activity(), which is the one not available and requiring that static string be generated in section __TEXT,__oslogstring,cstring_literals


"printing" Foundation.Data