External pass pipeline YAML format


(Vasileios Kalintiris) #1

Hi all,

Please, let me know if I should post this to another list.

I'm trying to figure out what is the expected YAML format of the
-external-pass-pipeline-filename option.

Dumping the pass pipeline under -O and feeding it back to the compiler with
this option doesn't work (from swift-3.1-branch).

I thought to ask here because I'm not entirely sure that the relevant YAML
parsing code from SILPassPipelinePlan::getPassPipelineFromFile() accepts
valid YAML input.

Thanks,
Vasileios


(Daniel Dunbar) #2

Have you seen `utils/pass-pipeline/scripts/pipelines_build_script.py`, it looks like it might give some hints?

···

--
$ sgit grep external-pass-pipeline-filename
include/swift/Option/FrontendOptions.td:def external_pass_pipeline_filename : Separate<["-"], "external-pass-pipeline-filename">,
utils/pass-pipeline/scripts/pipelines_build_script.py: '-external-pass-pipeline-filename\;-Xfrontend\;%s' % data_file]
--

- Daniel

On Jun 28, 2017, at 5:27 AM, Vasileios Kalintiris via swift-dev <swift-dev@swift.org> wrote:

Hi all,

Please, let me know if I should post this to another list.

I'm trying to figure out what is the expected YAML format of the -external-pass-pipeline-filename option.

Dumping the pass pipeline under -O and feeding it back to the compiler with this option doesn't work (from swift-3.1-branch).

I thought to ask here because I'm not entirely sure that the relevant YAML parsing code from SILPassPipelinePlan::getPassPipelineFromFile() accepts valid YAML input.

Thanks,
Vasileios
_______________________________________________
swift-dev mailing list
swift-dev@swift.org
https://lists.swift.org/mailman/listinfo/swift-dev


(Vasileios Kalintiris) #3

Have you seen `utils/pass-pipeline/scripts/pipelines_build_script.py`, it

looks like it might give some hints?

I've tried generating a pass pipeline with
utils/pass-pipeline/scripts/pipelines_generator.py
and use it with the -external-pass-pipeline-filename. However, AFAICT the
generated pipeline is not in the correct format that the compiler expects:

[
    [
        "HighLevel",
        "run_n_times",
        2,
        "SimplifyCFG",
        ...
        "GlobalARCOpts"
    ],
    [
        "EarlyLoopOpt",
        "run_n_times",
        1,
        "LowerAggregateInstrs",
        ...
        "SwiftArrayOpts"
    ],
    ...
]

Each generated pass pipeline contains the "run_n_times"|"run_to_fixed_point"
field, followed by the number of iterations, which is not what the compiler
expects.

I had no luck even when I tried to re-format the file containing the
pipelines to something that I believe the compiler would expect based on
the source code of SILPassPipelinePlan::getPassPipelineFromFile():

[
    [
        "HihLevel",
        "SimplifyCFG",
        ...
        "GlobalARCOpts"
    ],
    ...
]

or even:

[
    [
       "HighLevel",
       [ "SimplifyCFG" ],
       ...
       [ "GlobalARCOpts" ]
    ],
    ...
]

I suspect that we don't use the pass pipeline python scripts in our
buildbots anymore and the relevant bits, ie. the code in
SILPassPipelinePlan::getPassPipelineFromFile and/or the python scripts,
have not been kept up-to-date.

···

On Wed, Jun 28, 2017 at 6:18 PM, Daniel Dunbar <daniel_dunbar@apple.com> wrote:

Have you seen `utils/pass-pipeline/scripts/pipelines_build_script.py`, it
looks like it might give some hints?
--
$ sgit grep external-pass-pipeline-filename
include/swift/Option/FrontendOptions.td:def external_pass_pipeline_filename
: Separate<["-"], "external-pass-pipeline-filename">,
utils/pass-pipeline/scripts/pipelines_build_script.py:
'-external-pass-pipeline-filename\;-Xfrontend\;%s' % data_file]
--

- Daniel

> On Jun 28, 2017, at 5:27 AM, Vasileios Kalintiris via swift-dev < > swift-dev@swift.org> wrote:
>
> Hi all,
>
> Please, let me know if I should post this to another list.
>
> I'm trying to figure out what is the expected YAML format of the
-external-pass-pipeline-filename option.
>
> Dumping the pass pipeline under -O and feeding it back to the compiler
with this option doesn't work (from swift-3.1-branch).
>
> I thought to ask here because I'm not entirely sure that the relevant
YAML parsing code from SILPassPipelinePlan::getPassPipelineFromFile()
accepts valid YAML input.
>
> Thanks,
> Vasileios
> _______________________________________________
> swift-dev mailing list
> swift-dev@swift.org
> https://lists.swift.org/mailman/listinfo/swift-dev


(Michael Gottesman) #4

> Have you seen `utils/pass-pipeline/scripts/pipelines_build_script.py`, it looks like it might give some hints?

I've tried generating a pass pipeline with utils/pass-pipeline/scripts/pipelines_generator.py and use it with the -external-pass-pipeline-filename. However, AFAICT the generated pipeline is not in the correct format that the compiler expects:

[
    [
        "HighLevel",
        "run_n_times",
        2,
        "SimplifyCFG",
        ...
        "GlobalARCOpts"
    ],
    [
        "EarlyLoopOpt",
        "run_n_times",
        1,
        "LowerAggregateInstrs",
        ...
        "SwiftArrayOpts"
    ],
    ...
]

Each generated pass pipeline contains the "run_n_times"|"run_to_fixed_point" field, followed by the number of iterations, which is not what the compiler expects.

I had no luck even when I tried to re-format the file containing the pipelines to something that I believe the compiler would expect based on the source code of SILPassPipelinePlan::getPassPipelineFromFile():

[
    [
        "HihLevel",
        "SimplifyCFG",
        ...
        "GlobalARCOpts"
    ],
    ...
]

or even:

[
    [
       "HighLevel",
       [ "SimplifyCFG" ],
       ...
       [ "GlobalARCOpts" ]
    ],
    ...
]

I suspect that we don't use the pass pipeline python scripts in our buildbots anymore and the relevant bits, ie. the code in SILPassPipelinePlan::getPassPipelineFromFile and/or the python scripts, have not been kept up-to-date.

Yes. I think that is true. Here is what I would suggest:

1. It would be really trivial to change this to use yamltraits.
2. If you make this change, make sure that a test is added (maybe to sil-opt) that makes sure that we can roundtrip from the dumper.

MIchael

···

On Jun 28, 2017, at 10:15 AM, Vasileios Kalintiris via swift-dev <swift-dev@swift.org> wrote:

On Wed, Jun 28, 2017 at 6:18 PM, Daniel Dunbar <daniel_dunbar@apple.com <mailto:daniel_dunbar@apple.com>> wrote:
Have you seen `utils/pass-pipeline/scripts/pipelines_build_script.py`, it looks like it might give some hints?
--
$ sgit grep external-pass-pipeline-filename
include/swift/Option/FrontendOptions.td:def external_pass_pipeline_filename : Separate<["-"], "external-pass-pipeline-filename">,
utils/pass-pipeline/scripts/pipelines_build_script.py: '-external-pass-pipeline-filename\;-Xfrontend\;%s' % data_file]
--

- Daniel

> On Jun 28, 2017, at 5:27 AM, Vasileios Kalintiris via swift-dev <swift-dev@swift.org <mailto:swift-dev@swift.org>> wrote:
>
> Hi all,
>
> Please, let me know if I should post this to another list.
>
> I'm trying to figure out what is the expected YAML format of the -external-pass-pipeline-filename option.
>
> Dumping the pass pipeline under -O and feeding it back to the compiler with this option doesn't work (from swift-3.1-branch).
>
> I thought to ask here because I'm not entirely sure that the relevant YAML parsing code from SILPassPipelinePlan::getPassPipelineFromFile() accepts valid YAML input.
>
> Thanks,
> Vasileios
> _______________________________________________
> swift-dev mailing list
> swift-dev@swift.org <mailto:swift-dev@swift.org>
> https://lists.swift.org/mailman/listinfo/swift-dev

_______________________________________________
swift-dev mailing list
swift-dev@swift.org
https://lists.swift.org/mailman/listinfo/swift-dev


(Vasileios Kalintiris) #5

I suspect that we don't use the pass pipeline python scripts in our

buildbots anymore and the relevant bits, ie. the code in
SILPassPipelinePlan::getPassPipelineFromFile and/or the python scripts,
have not been kept up-to-date.

It seems that the following format works for the time being:

[
  [ "Pipeline1", [ "PassName1" ],
    [ "Pipeline2", [ "PassName2" ],
      [ "Pipeline3", [ "PassName3" ],
        [ ...
        ]
      ]
    ]
  ]
]

However, it works only when LLVM's assertions are disabled. I'll try to
come-up with a patch that will accept a saner format with multiple passes
per pipeline entry. Against which branch should I perform a pull request?

- Vasileios

···

On Wed, Jun 28, 2017 at 7:15 PM, Vasileios Kalintiris <vkal@uber.com> wrote:

> Have you seen `utils/pass-pipeline/scripts/pipelines_build_script.py`,
it looks like it might give some hints?

I've tried generating a pass pipeline with utils/pass-pipeline/scripts/pipelines_generator.py
and use it with the -external-pass-pipeline-filename. However, AFAICT the
generated pipeline is not in the correct format that the compiler expects:

[
    [
        "HighLevel",
        "run_n_times",
        2,
        "SimplifyCFG",
        ...
        "GlobalARCOpts"
    ],
    [
        "EarlyLoopOpt",
        "run_n_times",
        1,
        "LowerAggregateInstrs",
        ...
        "SwiftArrayOpts"
    ],
    ...
]

Each generated pass pipeline contains the "run_n_times"|"run_to_fixed_point"
field, followed by the number of iterations, which is not what the compiler
expects.

I had no luck even when I tried to re-format the file containing the
pipelines to something that I believe the compiler would expect based on
the source code of SILPassPipelinePlan::getPassPipelineFromFile():

[
    [
        "HihLevel",
        "SimplifyCFG",
        ...
        "GlobalARCOpts"
    ],
    ...
]

or even:

[
    [
       "HighLevel",
       [ "SimplifyCFG" ],
       ...
       [ "GlobalARCOpts" ]
    ],
    ...
]

I suspect that we don't use the pass pipeline python scripts in our
buildbots anymore and the relevant bits, ie. the code in
SILPassPipelinePlan::getPassPipelineFromFile and/or the python scripts,
have not been kept up-to-date.

On Wed, Jun 28, 2017 at 6:18 PM, Daniel Dunbar <daniel_dunbar@apple.com> > wrote:

Have you seen `utils/pass-pipeline/scripts/pipelines_build_script.py`,
it looks like it might give some hints?
--
$ sgit grep external-pass-pipeline-filename
include/swift/Option/FrontendOptions.td:def
external_pass_pipeline_filename : Separate<["-"],
"external-pass-pipeline-filename">,
utils/pass-pipeline/scripts/pipelines_build_script.py:
'-external-pass-pipeline-filename\;-Xfrontend\;%s' % data_file]
--

- Daniel

> On Jun 28, 2017, at 5:27 AM, Vasileios Kalintiris via swift-dev < >> swift-dev@swift.org> wrote:
>
> Hi all,
>
> Please, let me know if I should post this to another list.
>
> I'm trying to figure out what is the expected YAML format of the
-external-pass-pipeline-filename option.
>
> Dumping the pass pipeline under -O and feeding it back to the compiler
with this option doesn't work (from swift-3.1-branch).
>
> I thought to ask here because I'm not entirely sure that the relevant
YAML parsing code from SILPassPipelinePlan::getPassPipelineFromFile()
accepts valid YAML input.
>
> Thanks,
> Vasileios
> _______________________________________________
> swift-dev mailing list
> swift-dev@swift.org
> https://lists.swift.org/mailman/listinfo/swift-dev


(Vasileios Kalintiris) #6

I plan on writing a patch for this. Based on which branch should I create
the pull request? Is the swift-3.1-branch frozen?

···

On Thu, Jun 29, 2017 at 7:27 PM, Michael Gottesman <mgottesman@apple.com> wrote:

On Jun 28, 2017, at 10:15 AM, Vasileios Kalintiris via swift-dev < > swift-dev@swift.org> wrote:

> Have you seen `utils/pass-pipeline/scripts/pipelines_build_script.py`,
it looks like it might give some hints?

I've tried generating a pass pipeline with utils/pass-pipeline/scripts/pipelines_generator.py
and use it with the -external-pass-pipeline-filename. However, AFAICT the
generated pipeline is not in the correct format that the compiler expects:

[
    [
        "HighLevel",
        "run_n_times",
        2,
        "SimplifyCFG",
        ...
        "GlobalARCOpts"
    ],
    [
        "EarlyLoopOpt",
        "run_n_times",
        1,
        "LowerAggregateInstrs",
        ...
        "SwiftArrayOpts"
    ],
    ...
]

Each generated pass pipeline contains the "run_n_times"|"run_to_fixed_point"
field, followed by the number of iterations, which is not what the compiler
expects.

I had no luck even when I tried to re-format the file containing the
pipelines to something that I believe the compiler would expect based on
the source code of SILPassPipelinePlan::getPassPipelineFromFile():

[
    [
        "HihLevel",
        "SimplifyCFG",
        ...
        "GlobalARCOpts"
    ],
    ...
]

or even:

[
    [
       "HighLevel",
       [ "SimplifyCFG" ],
       ...
       [ "GlobalARCOpts" ]
    ],
    ...
]

I suspect that we don't use the pass pipeline python scripts in our
buildbots anymore and the relevant bits, ie. the code in
SILPassPipelinePlan::getPassPipelineFromFile and/or the python scripts,
have not been kept up-to-date.

Yes. I think that is true. Here is what I would suggest:

1. It would be really trivial to change this to use yamltraits.
2. If you make this change, make sure that a test is added (maybe to
sil-opt) that makes sure that we can roundtrip from the dumper.

MIchael

On Wed, Jun 28, 2017 at 6:18 PM, Daniel Dunbar <daniel_dunbar@apple.com> > wrote:

Have you seen `utils/pass-pipeline/scripts/pipelines_build_script.py`,
it looks like it might give some hints?
--
$ sgit grep external-pass-pipeline-filename
include/swift/Option/FrontendOptions.td:def
external_pass_pipeline_filename : Separate<["-"],
"external-pass-pipeline-filename">,
utils/pass-pipeline/scripts/pipelines_build_script.py:
'-external-pass-pipeline-filename\;-Xfrontend\;%s' % data_file]
--

- Daniel

> On Jun 28, 2017, at 5:27 AM, Vasileios Kalintiris via swift-dev < >> swift-dev@swift.org> wrote:
>
> Hi all,
>
> Please, let me know if I should post this to another list.
>
> I'm trying to figure out what is the expected YAML format of the
-external-pass-pipeline-filename option.
>
> Dumping the pass pipeline under -O and feeding it back to the compiler
with this option doesn't work (from swift-3.1-branch).
>
> I thought to ask here because I'm not entirely sure that the relevant
YAML parsing code from SILPassPipelinePlan::getPassPipelineFromFile()
accepts valid YAML input.
>
> Thanks,
> Vasileios
> _______________________________________________
> swift-dev mailing list
> swift-dev@swift.org
> https://lists.swift.org/mailman/listinfo/swift-dev

_______________________________________________
swift-dev mailing list
swift-dev@swift.org
https://lists.swift.org/mailman/listinfo/swift-dev


(Michael Gottesman) #7

I plan on writing a patch for this. Based on which branch should I create the pull request? Is the swift-3.1-branch frozen?

master.

···

On Jun 29, 2017, at 12:50 PM, Vasileios Kalintiris <vkal@uber.com> wrote:

On Thu, Jun 29, 2017 at 7:27 PM, Michael Gottesman <mgottesman@apple.com <mailto:mgottesman@apple.com>> wrote:

On Jun 28, 2017, at 10:15 AM, Vasileios Kalintiris via swift-dev <swift-dev@swift.org <mailto:swift-dev@swift.org>> wrote:

> Have you seen `utils/pass-pipeline/scripts/pipelines_build_script.py`, it looks like it might give some hints?

I've tried generating a pass pipeline with utils/pass-pipeline/scripts/pipelines_generator.py and use it with the -external-pass-pipeline-filename. However, AFAICT the generated pipeline is not in the correct format that the compiler expects:

[
    [
        "HighLevel",
        "run_n_times",
        2,
        "SimplifyCFG",
        ...
        "GlobalARCOpts"
    ],
    [
        "EarlyLoopOpt",
        "run_n_times",
        1,
        "LowerAggregateInstrs",
        ...
        "SwiftArrayOpts"
    ],
    ...
]

Each generated pass pipeline contains the "run_n_times"|"run_to_fixed_point" field, followed by the number of iterations, which is not what the compiler expects.

I had no luck even when I tried to re-format the file containing the pipelines to something that I believe the compiler would expect based on the source code of SILPassPipelinePlan::getPassPipelineFromFile():

[
    [
        "HihLevel",
        "SimplifyCFG",
        ...
        "GlobalARCOpts"
    ],
    ...
]

or even:

[
    [
       "HighLevel",
       [ "SimplifyCFG" ],
       ...
       [ "GlobalARCOpts" ]
    ],
    ...
]

I suspect that we don't use the pass pipeline python scripts in our buildbots anymore and the relevant bits, ie. the code in SILPassPipelinePlan::getPassPipelineFromFile and/or the python scripts, have not been kept up-to-date.

Yes. I think that is true. Here is what I would suggest:

1. It would be really trivial to change this to use yamltraits.
2. If you make this change, make sure that a test is added (maybe to sil-opt) that makes sure that we can roundtrip from the dumper.

MIchael

On Wed, Jun 28, 2017 at 6:18 PM, Daniel Dunbar <daniel_dunbar@apple.com <mailto:daniel_dunbar@apple.com>> wrote:
Have you seen `utils/pass-pipeline/scripts/pipelines_build_script.py`, it looks like it might give some hints?
--
$ sgit grep external-pass-pipeline-filename
include/swift/Option/FrontendOptions.td:def external_pass_pipeline_filename : Separate<["-"], "external-pass-pipeline-filename">,
utils/pass-pipeline/scripts/pipelines_build_script.py: '-external-pass-pipeline-filename\;-Xfrontend\;%s' % data_file]
--

- Daniel

> On Jun 28, 2017, at 5:27 AM, Vasileios Kalintiris via swift-dev <swift-dev@swift.org <mailto:swift-dev@swift.org>> wrote:
>
> Hi all,
>
> Please, let me know if I should post this to another list.
>
> I'm trying to figure out what is the expected YAML format of the -external-pass-pipeline-filename option.
>
> Dumping the pass pipeline under -O and feeding it back to the compiler with this option doesn't work (from swift-3.1-branch).
>
> I thought to ask here because I'm not entirely sure that the relevant YAML parsing code from SILPassPipelinePlan::getPassPipelineFromFile() accepts valid YAML input.
>
> Thanks,
> Vasileios
> _______________________________________________
> swift-dev mailing list
> swift-dev@swift.org <mailto:swift-dev@swift.org>
> https://lists.swift.org/mailman/listinfo/swift-dev

_______________________________________________
swift-dev mailing list
swift-dev@swift.org <mailto:swift-dev@swift.org>
https://lists.swift.org/mailman/listinfo/swift-dev