The ability to call the Swift compiler with response files was added in PR 15853 in GitHub. This allows you to invoke the compiler at the command line with the following syntax:
myresponsefile.resp is a text file containing command line arguments. This allows you to pass large numbers of command line arguments that would otherwise exceed the system limits.
Since the compiler creates separate jobs for the frontend, these jobs will fail if the number of command line arguments exceeds the system limits. In this situation, the command line argument vector would need to be written to a temporary response file, and then replaced with
Two possible approaches are:
- Wrap the command line argument vector within
TaskQueue. (Either in
- Wrap the command line argument vector just before adding a task to the
In Option 1, we would need to implement the argument vector wrapping in both the Default and Unix implementations of
TaskQueue. Additionally, we would need to implement it in the
DummyTaskQueue if we want the
-driver-skip-execution flags to print the response files instead of the entire list of command line arguments.
In Option 2, we can modify the code in a single location and cover all three code paths.
@jrose - Does Option 2 sound like a reasonable solution, or do you have other suggestions for how to approach this?
Not every job supports response files, so Option 1 is probably non-viable. I’ll copy in my concerns from the previous PR:
- If we have response files, should we get rid of
- Do these need to be preserved under
-save-temps, so that we can reproduce the build?
- What does it look like in a frontend crash log (with PrettyStackTrace) when using a response file? (I suspect this will just work but someone needs to check.)
- Will this interfere with
swift::driver::createCompilerInvocation, which does all its work in-place? I think not in this design, but we should make sure we have a test.
- There should probably be a flag to force the use of response files in the frontend, like
-driver-use-filelists…but judging from #15743, that should at least have both “enable” and “disable” variants, and should possibly be a customizable limit.
But I think this is a good plan, and the motivation of “we actually have many search paths, -D flags, and -Xcc flags” is a valid one.
@David_Ungar2, @Graydon_Hoare, what are your thoughts?
I am attempting to implement Option 2, and I’m running into a few issues.
Ensure that the replaced command line argument vector persists as long as the
Comp. The newly generated string containing the response file name needs to be passed through
Comp.getArgs().MakeArgString, and the temporary file itself needs to be registered with
Comp.addTemporaryFile(). This issue seems straightforward to address.
The new argument vector that gets passed into
TQ->addTask needs to persist as long as the
Cmd. The natural way to achieve this would be to modify the
Arguments property attached
Cmd. The problem is that
const in this context, so we cannot modify its argument list. In order to avoid this, we would have to move the implementation of response files several layers above
addPendingJobToTaskQueue. I’m not sure this is something we want to do. I also tried creating a
cost char *arg, mirroring other calls to
TQ->AddTask. This didn’t work, and the TaskQueue was reading junk memory when it attempted to execute the command.
Do you have any ideas about how we might resolve #2? Since I can’t modify the argument list directly on
Cmd, passing a new argument vector into
TQ->AddTask would seem to be the right approach. I am just not sure how to make sure it doesn’t get cleaned up before it is used.