Adding additional information to the ABI JSON

Evolution Pitches
1

Hi folks,

I hope you are all well. What do folks think about augmenting the event JSON ABI to include additional information test information, such as bug. timeLimit and tags (which is included in the experimental ABI JSON).

Motivation

The event JSON stream offers great data about the Swift Testing tests. Tools can be written to extract information from the JSON. However, the event JSON stream is currently missing some traits information about the test itself. This information is can be important for tools wanting to use the missing information.

Proposed solution

We propose adding tags, bug and timeLimit to the test payload where the Modified Backus-Naur Form (BNF) delta would be:

diff --git a/Documentation/ABI/JSON.md b/Documentation/ABI/JSON.md
index e4ff24a..1a82996 100644
--- a/Documentation/ABI/JSON.md
+++ b/Documentation/ABI/JSON.md
@@ -157,10 +157,26 @@ additional `"testCases"` field describing the individual test cases.
   ["displayName": <string>,] ; the user-supplied custom display name
   "sourceLocation": <source-location>, ; where the test is defined
   "id": <test-id>,
-  "isParameterized": <bool> ; is this a parameterized test function or not?
+  "isParameterized": <bool>, ; is this a parameterized test function or not?
+  ["tags": <array:tag>,] ; the tags associated with this test function
+  ["bugs": <array:bug>,] ; the bugs associated with this test function
+  ["timeLimit": <number>] ; the time limit associated with this test function
+
 }
 
 <test-id> ::= <string> ; an opaque string representing the test case
+
+<tags> ::= <string> ; a string representation of a tag
+
+<bug> ::= {
+  ["url": <string>,] ; the bug url
+  ["id": <string>,] ; the bug id
+  "title": <string> ; the human readable bug title
+} ;
+

An example output of the test kind is

{
    "kind":"test",
    "payload":{
        "tags": [".purple",".blue",".red",".green",".orange",".traitRelated",".yellow"],
        "bugs":[
            {
                "url": "myInternalScheme://defect/9876"
            },
            {
                "url": "http://example.com/defect/1234",
                "title": "another defect"
            }
        ],
        "timeLimit": 5,
        <...SNIP...>
    },
    <...SNIP...>
}

I have a draft PR which add this functionality, which guards the new fields behind an ABI version.

Looking forward to your feedback.

Sam

EDIT: I made some changes to the proposed Modified BNF changes

1 Like
2

The timeLimit is currently in seconds. Should the property be renamed to include the units in the name to make it less ambiguous? e.g.: timeLimitInSeconds instead of timeLimit?

3

For anyone interested, the proposal PR has been published. Here is a human-readable version.

4

I’m a big fan of exposing more of the traits that provide rich context for tests for tools to use.

I just wish Xcode would expose more of this information…

RE: timeLimitInSeconds I’m in favor, just to avoid ambiguity for posterity

5

Side note about timeLimit — just to clarify, the API only supports expressing time limits in terms of minutes, but the ABI JSON will resurface that in terms of seconds?