Custom String Interpolation and Compile-time Interpretation applied to Logging

Development Compiler
21

Yes.

22

The names of the APIs would likely evolve. But, here osLog is meant to be a free function that logs to Apple's unified logging system. It not a general-purpose logging API for Swift, which is an orthogonal line of work.

23

That is an interesting question and it is relevant to this proposal. Ideally, we would like to support this. But this is not something the proposal enables out of the box. There are some limitations coming from the logging system itself that complicates wrapping logging calls (as you had mentioned). Furthermore, the optimization proposed here causes a few problems to doing this. (There are certain hacky workarounds like using @_transparent etc. But, it would not be a very desirable user model and will have its own limitation.) Therefore, at this point I do not have a concrete answer to this. But, this is something quite in sync with the overall extensibility aspect that we are trying to address.

1 Like
24

My thoughts exactly, thanks @davedelong

At first when I looked into os_log() I thought it would be useful, but there's no way I'm going to do sysdiagnoise to get logs back from users.

There should be some kind of logging solution built into the SDK/platform/library/etc that is actually useful for a developer to integrate and easily get logs to view.

All of the features of os_log() are cool, and this new string interpolation definitely makes it way more approachable (Swifty).

BUT it still doesn't address my biggest reason not to use it at all. It's not easy to get logs back from users.

1 Like
25

I think this is an excellent proposal and I'm also looking forward to the compile-time interpreter landing.

Regardless of the merits of os_log, this demonstrates how framework authors can build powerful abstractions that make really nice to use APIs that give us safety at compile time while melting away at runtime.

Consider this a big +1 from me for exploring this area further.

1 Like
26

I'm a lurker here but, in past lives, have implemented some, and used other, ways of generating human readable text by the aggregation of static and dynamic values via a 'templating' syntax. I think this is the core of this proposal so I wanted to add a thought, or two, about the needs I've had of such implementations.

.. The final product that is made visible (the text emitted into the logging system in the osLog case) may be used in other ways. It could be user-visible alert text, written to storage, a Tweet, etc, so I would encourage decoupling logging from this proposal. If osLog has special needs that limit the flexibility of the feature, I'd worry.

.. Syntax for conditionals: emitting alternative texts depending on a boolean value? One I used a lot is emitting the right word for zero, singular or plural occurrences of a value "there {cowCount, "are no cows", "is only one cow", "are too many cows"}." ..

.. With delayed binding schemes like this, it is very easy to throw the kitchen sink into the mix. I'd keep the core implementation sparse and offer a generous extension mechanism. Custom types are catered for, could localization, or my conditional idea above, be custom extensions, for example?

I'm going to look at the various interpolation/interpretation proposals and play with the code more. Please pardon me if my above remarks are off base here, or already taken into consideration.

1 Like
27

Just wanted to add: this idea of hoovering up all information is the "old way" of thinking. Users also have privacy rights, which these days are backed by laws like the EU's GDPR. If your logs contain private information, you need to consider who has access to these logs and how long they are stored for.

If you're not considering this in your logging right now, you definitely should.

4 Likes
28

I think it's perfectly acceptable to provide an in-app feature that allows users to submit logs along with a bug report as long as there is disclosure on the nature of the information contained in the logs. It is the user themselves who submits the log after all. I think this is a use case that can be addressed while still respecting the privacy of the user.

29

Talking to myself .. I've now read a lot of the background to this proposal, looked at code and messed around in a Playground, and been somewhat mortified to appreciate the immense scope of the work. My remarks above are naive in comparison and are best passed over. I live and learn.

1 Like
30

So, looking at the prototype:

I feel that the interpolations could read more nicely.

  • The private: argument label gets in the way. I would prefer to put the value first, and the privacy modifier afterwards.

    // Prototype.
osLog("Login time: \(private: currentTime, .time) ms")
// Shuffled.
osLog("Login time: \(time: currentTime, .private) ms")

    (Making privacy an enum might also allow us to specify more fine-grained permissions controls one day, such as hashing or truncating the private values when exported).

  • Rather than make the formatting options an enum, have you considered adding more addInterpolation overloads?

    // Prototype.
osLog("Network delay: \(delay, .decimal(2)) ms")
osLog("Header: \(flags, .hex)")
// With overloading.
osLog("Network delay: \(delay, decimalPlaces: 2) ms")
osLog("Header: \(hex: flags)")

  • I'm not sure about the name osLog. It seems like this would be better expressed in Swift using namespacing - e.g. OS.log(...). Similarly, PackedLogMsg -> OS.LogMessage

Otherwise, this is a very substantial improvement to the os_log API :+1:. Thanks for sharing your progress at this stage.

2 Likes
31

Thanks for raising some interesting points. I would like to use this context to explain some of the extensibility aspects of the proposal (possibly answering some of your questions). Particularly, what this proposal offers currently and some future directions to explore (as a community).

I want to bring out the different levels of extensibility here (in the increasing order of their generality):

Level 1: Customizing log messages passed to osLog: This is kind of what the proposal enables as such, as it is narrowly focused on just osLog. You can customize the string interpolation methods of PackedLogMsg as you like e.g. to append tags/prefixes/suffices etc. to messages. The updations can use static, interpretable state. For example, you can choose between singular/plural based on the count of something (as long as the count is compile-time constant). If you look at the format string construction, it is already quite complicated. Having said that, all of this is limited by two things: (a) the interpretable language fragment (which doesn't include impure functions with side-effects, globals, classes, exceptions etc), and (b) the stdlib APIs supported by the interpreter. Note that the interpreter would only support a tiny fraction of string and array operations to accomplish the goals of this proposal. So doing advanced processing on strings will need extensions to the interpreter. (Often complex stdlib APIs are not interpretable and require special modeling inside the interpreter.)

Level 2: Customizing the backend and making it work with other loggers: The next level of extension would be to be able to use a custom logger instead of the unified-logging system, which in the simplest case, can just be writing to a log file. This is kind of plausible with this proposal (But is not the focus, at least until the basic functionality is accomplished). This can be achieved, if you make the APIs that does the write to file accept a PackedLogMsg (which could well be customized to not create a byte buffer and format string but instead, say, just convert everything to string). E.g. you can create a function

func writeToLogFile(msg: PackedLogMsg) {
    msg.getAsString().writeToFile(...)
}

extension PackedLogMsg {
   func getAsString() -> String {
      // construct a string using the tracked format string and "argument encoders" tracked by PackedLogMsg.
   }
}

What you get out of this implementation is that PackedLogMsg is constructed at compile-time and only getAsString() and writing to file has to happen at runtime. The code for getAsString is a bit complex but it can be constructed with the public members of PackedLogMsg (viz. the format string property and encode method). Ideally, the extension may want to write to file asynchronously and also construct the full message asynchronously (like osLog) for performance. This is doable with this proposal, but again has the limitation of being within interpretable/optimizable language fragment. This is how we think the proposed design can benefit/inter-operate with server-side logging (in the future). (See Server-side logging discussions: [Discussion] Server Logging API - #45 by johannesweiss)

From here, I am going to get a bit futuristic and present some ideas. (They are completely outside the scope of this proposal) and I am just sharing ideas for potential extensions/generalizations of this idea, and the interpreter more broadly.

c) Level 3: A general InterpretableStringInterpolation struct: There could be many other cases where one may want to optimize a custom string interpolation by interpreting constant parts of it. The compiler can expose a general custom string interpolation struct on which it will run the interpreter and try to fold its uses, independent of what it is used for. It could be for logging or for some other purpose. An open question/challenge here is how to design the properties and interfaces of the struct to make it general enough. (This is a non-trivial challenge especially when some parts of the interpolation are constant and some are not as in the osLog case.) PackedLogMsg can then be seen one instance of this.

d) Level 4: A general-purpose "constexpr" for Swift: Such a language feature would enable Swift users invoke the interpreter and ask it to statically fold a value. This is a major feature and has several open questions/challenges. (A remark: the original compile-time interpretation pitch proposed a static-assert construct, which can also invoke the interpreter on arbitrary code but does not fold the result.)

The experience we gain and the challenges we address with this work may help inform a larger effort to evolve such user-visible language features as a part of Swift. This is completely outside the scope of this work.

2 Likes
32

AFAIK, signpost is not yet available in Swift because it relies on storing the static C string into a custom Section in the generated object file. To solve this, Apple should either change the way signpost works (by using the standard C string section), or we should update Swift to support string static C string into a custom section.

33

@Jean-Daniel os_signpost is available in Swift and has an almost similar interface to os_log - taking a format string and varargs: Apple Developer Documentation. (An article on using it: Getting started with signposts | Swift by Sundell)

1 Like
34

My bad. I mixed it up with os_activity(), which is the one not available and requiring that static string be generated in section __TEXT,__oslogstring,cstring_literals

35

This was a really good idea — any chance we see a formal pitch for it? I'd hate to see it fade away.

3 Likes
37

Update from @ravikandhadai on this topic:

3 Likes