Support for Device Frames

Development Swift-DocC
1

Hi All!

I’m excited to share a proposal for a new experimental feature that allows authors to wrap images and videos inside device frames.

When developing UI libraries or apps, we sometimes need to show previews of how things look when rendered on a device, like a phone or a tablet. This new experimental feature is set to allow you to wrap your images and videos in a frame, making it easier to create complex assets and maintain them overtime.

DeviceFrames are supposed to work in tandem with captions for both @Image and @Video.

@Image

Add the following parameters to the @Image directive:

  • deviceFrame: Freeform string input. See Adding your own frames*
@Image(
    source: "overview-hero.png", 
    alt: "A screenshot of an app containing an illustration of a sleeping sloth.",
    deviceFrame: phone
)

Explanation: Wraps the image inside a device frame, with the caption right under it. Images that do not fit the device frame’s aspect ratios are top aligned and stretched to fit the width.

Examples

@Image(
    source: "some-image",
    deviceFrame: phone
)
  • deviceFrameCrop: One of topHalf, bottomHalf. Defaults to none, resulting in no cropping.
@Image(
    source: "notification.png", 
    alt: "A screenshot of an app notification.",
    deviceFrame: phone,
    deviceFrameCrop: topHalf,
)

Explanation: Renders only the top/bottom half of a frame, re-calculating the aspect ratio and size of the frame. This allows authors to focus user’s attention to either part of the device.

@Video

Add the following parameter to the @Video directive:

  • deviceFrame: Freeform string input. See Adding your own frames*

Examples

@Video(
    source: "page-controls-scrub.mp4",
    poster: "page-controls-scrub.png",
    deviceFrame: tablet,
)
  • deviceFrameCrop: One of topHalf, bottomHalf. Defaults to none, resulting in no cropping.
@Video(
    source: "video.mp4",
    deviceFrame: tv,
    deviceFrameCrop: bottomHalf,
)

Explanation: Renders only the top/bottom half of a frame, re-calculating the aspect ratio and size of the frame. This allows authors to focus user’s attention to either part of the device.

Adding your own frames

Device frames can be added using the theme-settings.json file. Add all your frames under the theme.device-frames path, then use the same keys you defined for your deviceFrame when you author your docs.

All the parameters are required.

  1. The Numeric values represent pixels, measured when the image is rendered at 100%.
    1. screenTop and screenLeft are the amount of pixels from edge of image to edge device screen.
    2. screenWidth and screenHeight are the pixel dimensions of the device screen.
    3. frameWidth and frameHeight are the pixel dimensions of the entire frame. This will generally match your image size and represents how big the frame will be rendered on your screen, when at 100% size.
  2. The URLs are absolute paths to the device frame image.
    1. If the path starts with / its relative to the root of your website. The images can be hosted along with your docc-archive or on a remote server. This is the same concept as providing your own custom icons.
{
  "theme": {
    "device-frames": {
      "custom-device": {
        "screenTop": 27,
        "screenLeft": 27,
        "screenWidth": 655,
        "screenHeight": 490,
        "frameWidth": 708,
        "frameHeight": 543,
        "lightUrl": "/img/device-frames/custom-device.png",
        "darkUrl": "/img/device-frames/custom-device~dark.png",
      }
    }
  }
}

Usage:

@Video(
    source: "video.mp4",
    deviceFrame: "custom-device",
    deviceFrameCrop: topHalf,
)
10 Likes
2

Generally seems reasonable to me. Can/will the compiler emit a diagnostic if you reference a device frame that doesn't exist in the JSON file?

Are there any other uses for this other than device chrome that would motivate making the parameter name more general? I could maybe see folks wanting to add standardized web browser chrome or a terminal window frame, but I think maybe those cases are better solved at the time a screenshot is produced.

In the JSON, is frameWidth and frameHeight necessary? It's not needed for other image-related DocC features. What happens if those values differ from the actual image dimensions?

1 Like
3

Hello,

Can/will the compiler emit a diagnostic if you reference a device frame that doesn't exist in the JSON file?

As far as I know, it cant really do that. The JSON file is just passed over to the renderer.

Are there any other uses for this other than device chrome that would motivate making the parameter name more general?

As you said, you could have a browser window, a terminal, a mobile device, TV, laptop, anything really. The biggest advantage, apart from not having to bake-in the actual frames, saving time, is later updating said frames, if say a new device comes out and you want to use it.

In the JSON, is frameWidth and frameHeight necessary?

We need those in order to determine how big the frame should be. Because it's no longer just an image, but rather a complex set of overlayed elements, with aspect ratios and responsive functionality, we need those dimensions. Also, if you use an SVG, which I recommend, those can be without a predefined size.

4

For the DocC directive changes I feel that the duplication of device frame information across @Image and @Video could lead to inconsistencies if we add more device configuration in the future or if we add more content that can have a device frame.

Have you considered introducing a @DeviceFrame directive that wraps the media that it frames? In that solution the device information arguments isn't duplicated across image and video, future arguments can be added in one place, and support for future content can be added in one place as well.

Having a separate directive for device frames would also make it easier to disallow device frames in places certain places (for example in the introduction section on a tutorial page where the image serves as a background)

I've translated some of the examples from the original post to use a wrapping @DeviceFrame directive instead.

I think it would make sense to shorten the "deviceFrameCrop" argument to "crop" but I'm not sure if the "deviceFrame" argument should become an unlabeled argument or a labeled argument. Below I've labeled it "style".

@DeviceFrame(style: phone) {
  @Image(source: "some-image")
}

@DeviceFrame(style: phone, crop: topHalf) {
  @Image(
    source: "notification.png", 
    alt: "A screenshot of an app notification."
  )
}

@DeviceFrame(style: phone, crop: topHalf) {
  @Image(
    source: "notification.png", 
    alt: "A screenshot of an app notification."
  )
}

@DeviceFrame(style: tablet) {
  @Video(
    source: "page-controls-scrub.mp4",
    poster: "page-controls-scrub.png"
  )
}
4 Likes
5

I did actually think about this, but form a short convo with Ethan we agreed the parameter approach might be better. I am unsure about the DocC specifics honestly.

From a RenderJSON change perspective, it should be mostly fine.

6

Could you or @ethankusters share some of the conclusions about the proposed authoring syntax from that conversion?

7

I much prefer this solution, it’s more composable and could be more easily expanded upon

1 Like
8

I see a pretty clear trade-off here.

Adding a new @DeviceFrame directive allows the @Image and @Video directives to remain as simple and approachable as possible but makes the operation of adding a device frame to an image or video more opaque and difficult to discover.

In general I'm not a huge fan of wrapping directives as a way to set configuration of a child element – for DeviceFrame it makes a little more sense since the device frame literally wraps the image in the eventual UI – but I still don't like it as a precedent. Adding this as a parameter makes it immediately clear to the user what's possible here while a @DeviceFrame directive just invites questions: "What can I put inside a device frame directive? If I put multiple images in a device frame does the frame wrap both or apply individually to each? Can I have an empty device frame? Can I put prose in a device frame?" All of these questions now have to be answered via diagnostics and trial/error. Editor support and documentation can help here but only so much.

Adding a parameter to @Image makes it immediately clear what is and what is not supported here while keeping things very discoverable to the user in documentation and any completion strategies their editor might have. The drawback is that if we keep adding more and more parameters to @Image and @Video it will make things difficult for newcomers to Swift-DocC – we don't want to end up in a world where @Image has one required parameter and 10 optional ones.

Personally, I think device frames are a common enough need in documentation that they deserve the first-class treatment of being a direct parameter on @Image. And I don't forsee needing to add many more parameters to @Image – Device Frames are really an exception to the rule that I'd expect image manipulation to be done in a separate tool. For example, I don't expect Swift-DocC to ever add support for image filters or cropping – most image operations should be done in a separate tool and baked into the image Swift-DocC displays.

Jumping straight to splitting off a separate directive to add this piece of configuration to images and videos seems like it errs on over-optimization when we don't have examples of other parameters @Image and @Video are likely to add.

I think this is right – it's less discoverable to the user but is more composable and easily expanded upon. So in order to justify the reduced discoverability and usability we should have an idea of how that composability and expansion might be used in the future.

That being said, I'm not fully convinced either way here and I think this is a compelling argument:

Really what I want is a better syntax for adding configuration to base directives – something equivalent to SwiftUI's view modifiers: border(_:width:) | Apple Developer Documentation. Something that can be as discoverable as a parameter without inviting the confusion of a fully open parent directive. But I think a change like that is well out of scope of this proposal. :blush: With the syntax currently available to us – I currently see the additional parameter approach as the better trade-off.

3 Likes
9

I thought that this file was included in the DocC catalog, so in theory it would be possible to consume the data. If that's not always the case and we can't provide diagnostics and autocomplete for this feature then I am not in support of the way device frames are declared.

Should we consider a more general name than deviceFrame then, e.g. frame or imageFrame? (but if we expect those cases to be far less common, then let's not sacrifice discoverability)

10

I had the same initial reaction as @Lancelotbronner, but I find this rational compelling. Wrapper directives are often too open-ended, and allow for too many possibilities that have to be guarded against.

:exploding_head: I love this idea. For the future, of course :slight_smile:

11

I agree and I think that is the reason this feature is being pitched as "experimental":

In order to have to proper (non-experimental) support for device frames Swift-DocC will need to have some story for out-of-the-box support of frames so that the average user doesn't need to declare them at all. I believe this is being left to a future pitch. (In general we also need a better story for the UX of theming – diagnostics, etc – and I think improvements to declaring custom frames will come with that).

1 Like
12

Okay, this seems like a reasonable stepping stone to that then since I'd imagine DocC will end up emitting that json file in the future.

1 Like
13

I think this is the core of our difference of opinion. In my mental model the device frame isn't a configuration of an image, it's a distinct element that contains an image.

In my opinion adding a @DeviceFrame directive doesn't set any present for configuration of inner elements because the frame information isn't configuration that belongs on the inner elements.

There is no reason why the @Image documentation couldn't link to @DeviceFrame directive and describe how to wrap an image in a frame. I would find it a bit odd for the @Image documentation to list which of its parameters that are also supported on @Video directives (and vice-versa) but it would be a natural place for the @DeviceFrame documentation to list what elements it can frame.

If the various device frame configurations are added as parameters to the directives that support frames that makes it hard to answer the question: "What content can I put in a frame?"

If the various device frame configurations are added as parameters to the directives that support frames it would invite a similar question that's much harder to answer: "what other directives can I add a device frame to?"

Not only is it hard to find the answer to this question but it's harder to write the documentation that provides this answer.

  • One way would be to write an article that list all the device frame configurations and all the directives that support them, but at that point why isn't it a dedicated directive?

  • Another way would be to cross link between all the directives that support device frames to list the other directives that also support device frames, but at that point why not have a single page with all this information?

In my mental model—where the device frame is its own distinct element that contains other content—the frame wraps the content it is given. Documentation and diagnostics could easily limit this to a single media element — like we do with @ContentAndMedia today — but maybe there's a future use case for presenting multiple images in a carousel within a single device frame.

I can't think of a good current use case for this but maybe there will be some in the future. For example, maybe someone would want to create a custom device frame that looks like a sticky note and use it to put little annotations in their documentation. A Note callout would probably be the better element for that but maybe this developer really likes the appearance of sticky notes.

If the various device frame configurations are added as parameters to the content that supports it then this could lead to inconsistent syntax in the future. A paragraph of text can't have directive arguments so we would need a different syntax if we ever supported text within a device frame.

A different question that's hard to answer if the various device frame configurations are added as parameters to the directives that support frames is: "where can and can't I use a device frame"?

It's not unlikely that editor support could lead the developer down the wrong path by offering completion for the device frame arguments in a place where they're not supported (e.g. when the image is used as a background) needing a diagnostic to suggest removing the completed arguments.

If the editor had code completion for allowed directives (see second-to-last paragraph of this post) then it wouldn't complete the device frame directive in places that expect a non-framed image.

I agree that device frames deserve the first-class treatment but in my opinion that means having a dedicated directive for it instead of inlining the device frame arguments on the framed directives.

In my opinion the strongest argument for inlining the device frame configuration as image and video parameters is completion discoverability. This is a compelling argument but I personally doesn't feel that it measures up to all the limitations it puts on future enhancements and the current problems it doesn't address.

I think that it would be easier to improve the completion discoverability of the dedicated directive - for example by adding support for completing the allowed child directives in a given scope — than what it would be to scale the inlined device frame arguments and handle the special cases where images used as backgrounds shouldn't support frames.

It's much harder to change any authoring syntax after it has been added and I don't feel that this proposal makes a strong enough case that its proposed syntax will scale or that no future use cases exist which this syntax would need to scale to support.

2 Likes
14

I think this is the strongest argument and the question we need to answer here. Is a DeviceFrame a distinct element or is it a piece of configuration that Image and Video both happen to support?

To me it seems clear that the DeviceFrame as @dhristov pitched it is a piece of configuration because it doesn't support arbitrary content. An Image can add a device frame and a Video can add a device frame. This stands in direct contrast to something like the @Row/@Column directives which are distinct elements that hold arbitrary content.

Writing:

@DeviceFrame {
   @Image()
}

invites an obvious connection to

@Row {
   @Column {
       @Image
   }
}

which we actually don't want the user to connect. These are very different pieces of syntax that behave in a very different way.

What is the argument for DeviceFrame being it's own distinct element?

I agree with the limitations of the pitched syntax as you've described them – for example, if we want to add further configuration to image device frames in the future, we could end up with a confusing number of parameters on the @Image directive. But I think it's worse to break the current precedent we have of what a container directive is in a DocC article. Today, in DocC articles (this isn't true in Tutorials and is part of what makes Tutorial syntax difficult to learn) with they key exception of @Metadata and @Options (discussed here) a container directive can contain any kind of markup. We discussed the importance of this in the pitch for @Options:

Introducing @DeviceFrame as a separate directive now introduces confusion for all other container directive use in DocC articles. The user needs to ask – "Is this a special directive that sets configuration for a specific set of child directives or this a container directive that holds markup?"

I still think we should explore alternative syntax for adding configuration to directives. Maybe something like this:

@Image(source: "sloth") {
   My image caption here.
}
.deviceFrame(phone)

which would solve a lot of the documentation and scalability issues we've discussed.

But I think the correct intermediate solution here is to add this as a parameter instead of breaking the mental model of how container directives work in DocC articles.

2 Likes
15

What about flipping the whole thing? I haven’t given it much though so maybe this is ridiculous, but how about:

@Image(source: “images/example.png”) {
    caption for the image here
    @DeviceFrame(*params*)
}

It would make it clear that it is something which pertains to images and videos without “polluting” their parameters and be extensible in the future since it’s its own directive.

I also feel it can’t really be mistaken with the content because I can’t immediately think of anything, definitely correct me on that.

I think it also communicates that it is additional configuration which isn’t essential to understanding and using @Image, which I think is a bonus.

What do you people think of this?

16

Thank you for bringing this up! It's definitely something we should consider and there's a lot of precedent for this in Tutorials. @Chapter for example takes an optional Image directive as a child but the @Image isn't rendered inline, it's instead configuration for an image associated with the Chapter itself.

Again though, I'd refer to my earlier point:

I don't think that this feature (especially in its experimental phase) justifies introducing a new way of writing directives to DocC articles. I'd really like to ensure that the directive use in DocC articles is consistently approachable and understandable without much prior knowledge – this isn't true for Tutorials but Tutorials justify that by offering a significantly different and more powerful user-experience.

Unless we can justify a need for the expandability and composability that an additional directive provides – I don't think we should be introducing @DeviceFrame at this time (along with a new concept for configuration-setting for directives in DocC articles) when an additional parameter will suffice.

17

Hi @dhristov,

To move things along a little here – what do you think of simplifying things here by removing the second deviceFrameCrop parameter? We would move forward with adding the single deviceFrame parameter to both @Image and @Video as an experimental feature. For folks who do need cropped device frames, they have a workaround of just adding an additional custom frame that is cropped to their desired dimensions.

We can see how folks make use of the feature while it's still in its experimental phase and if it becomes obvious that we need more configuration than the single-parameter-approach allows – we can re-open the discussion about a dedicated @DeviceFrame directive.

In the case, that we do decide we need @DeviceFrame or even .deviceFrame() we would offer a FixIt and migration period to move folks from the deviceFrame parameter – since there's just a single parameter to support I think a migration period would be very achievable. (I personally think the chances of us finding a need for more configuration beyond the single parameter is rather low – just want to make it clear that we have a path forward in that scenario.)

Thoughts?

CC: @ronnqvist I'm curious if this addresses your concerns as well.

@Image(
    source: "notification.png",
    alt: "A screenshot of an app notification.",
    deviceFrame: phone
)

@Video(
    source: "page-controls-scrub.mp4",
    poster: "page-controls-scrub.png",
    alt: "A scrub control moving across the screen.",
    deviceFrame: laptop
)
1 Like
18

Fine by me. The crop param was just to make life a bit easier, but honestly I dont see many people using it anyway.

As for the syntax, I think the parameter approach is simpler than an extra directive.

1 Like
19

We have different mental models about what a device frame is. Neither of us has the data to predict the mental model of the broader developer community so I don't think either mental model should be used as a strong indication about what the broader community feel is a more natural syntax.

What connection don't we want the developer to make here? Most directives with child directives have restrictions regarding what sub directives they can contain and not all directives support free-form prose.

As far as I know @Row, @TabNavigator, and @Steps don't support free-from prose but they each support some sub directives. I wouldn't mind calling these directives "containers".

This addresses some concerns but raises other concerns.

Since this pitch has already talked about a "device frame crop" configuration there is at least one example of a "future" enhancement that we may want to add but there isn't a solid plan for how to add it. In this future there would already be one inlined device frame parameter so the natural non-breaking change would be to add more deviceFrame prefixed parameters to both @Image and @Video. Adding only one device frame parameter doesn't really make any difference if it implies that future device frame configurations are also added as inline parameters to both @Image and @Video. There also isn't a plan for how to deal with hypothetical new types of content that can be framed and framing non-directives wouldn't work with this syntax. This thread hasn't brought up any concrete examples of future device framed content but I also don't feel that it's made a strong case that there will be no future cases.

One unfortunate effect of adding a deviceFrame parameter directly to @Image and @Video is that images and videos will "support" frames by default and cases where the frame doesn't make sense—for example if the image is used as a background—have to opt out by raising a diagnostic. Having device frames be opt-out instead of opt-in means that all future places that support images or video need to remember to opt-out.

It's very hard to change authoring syntax and once we commit to a syntax we're likely going to have to support it for a very long time. With the Swift-DocC public Swift API we have the ability to deprecate API and slowly phase it out but we currently don't have support for directive deprecation warnings. This is why I'm so very hesitant towards adding authoring syntax that may change. Even if the risk of it needing to change in the future is low, to cost of making that change is really high.

I tried to look for other pitches that add a directive as "experimental" but I couldn't find any. I see that your PR hides the deviceFrame argument from the generated documentation but I don't think that's enough. I'm concerned that a developer would find this parameter and start using it whiteout realizing that it's experimental and then be upset—or worse, be unaware—if we make breaking syntax changes to it in the future. For other experimental DocC features we add a temporary command line argument to DocC to enable the feature. I wonder if we could do the same here (for example: --experimental-enable-device-frames). That would make it very clear that this feature isn't stable and that it could change in the future.

As long as we allow ourselves to revisit the syntax before making this non-experimental and we make it very explicit that this is an experimental feature (for example by requiring a command line flag), I could be convinced that a single deviceFrame parameters is an okay solution when that's the only device frame configuration we need/have.

20

This is a great idea – it's generally been our policy to gate all experimental features behind a feature flag and I think we should update follow that precedent here.

I've updated the PR to include that change:

I agree there are unresolved concerns here. I think the best path forward is to land the feature as @dhristov pitched it and see if in-practice there's a need for more involved syntax. If so, we can put forward a new pitch with that syntax and, if not, we can publish a pitch to formalize this feature, include some baked-in device frame support in Swift-DocC and remove the feature flag.

Thank you for all of your great feedback here! I really appreciate it.

1 Like