Wrapping the Command Line Argument Vector in a Response File if it Exceeds System Limits

The ability to call the Swift compiler with response files was added in PR 15853 in GitHub. This allows you to invoke the compiler at the command line with the following syntax:

swiftc @myresponsefile.resp

where myresponsefile.resp is a text file containing command line arguments. This allows you to pass large numbers of command line arguments that would otherwise exceed the system limits.

Since the compiler creates separate jobs for the frontend, these jobs will fail if the number of command line arguments exceeds the system limits. In this situation, the command line argument vector would need to be written to a temporary response file, and then replaced with @temporary-reponse-file.resp.

Two possible approaches are:

  1. Wrap the command line argument vector within TaskQueue. (Either in addTask or execute).
  2. Wrap the command line argument vector just before adding a task to the TaskQueue in Compilation.cpp::addPendingJobToTaskQueue.

In Option 1, we would need to implement the argument vector wrapping in both the Default and Unix implementations of TaskQueue. Additionally, we would need to implement it in the DummyTaskQueue if we want the -driver-print-jobs and -driver-skip-execution flags to print the response files instead of the entire list of command line arguments.

In Option 2, we can modify the code in a single location and cover all three code paths.

@jrose - Does Option 2 sound like a reasonable solution, or do you have other suggestions for how to approach this?

Not every job supports response files, so Option 1 is probably non-viable. I’ll copy in my concerns from the previous PR:

  • If we have response files, should we get rid of -filelist and -primary-filelist?
  • Do these need to be preserved under -save-temps, so that we can reproduce the build?
  • What does it look like in a frontend crash log (with PrettyStackTrace) when using a response file? (I suspect this will just work but someone needs to check.)
  • Will this interfere with swift::driver::createCompilerInvocation, which does all its work in-place? I think not in this design, but we should make sure we have a test.
  • There should probably be a flag to force the use of response files in the frontend, like -driver-use-filelists…but judging from #15743, that should at least have both “enable” and “disable” variants, and should possibly be a customizable limit.

But I think this is a good plan, and the motivation of “we actually have many search paths, -D flags, and -Xcc flags” is a valid one.

@David_Ungar2, @Graydon_Hoare, what are your thoughts?

I am attempting to implement Option 2, and I’m running into a few issues.

  1. Ensure that the replaced command line argument vector persists as long as the Compilation object, Comp. The newly generated string containing the response file name needs to be passed through Comp.getArgs().MakeArgString, and the temporary file itself needs to be registered with Comp.addTemporaryFile(). This issue seems straightforward to address.

  2. The new argument vector that gets passed into TQ->addTask needs to persist as long as the Job object, Cmd. The natural way to achieve this would be to modify the Arguments property attached Cmd. The problem is that Cmd is const in this context, so we cannot modify its argument list. In order to avoid this, we would have to move the implementation of response files several layers above addPendingJobToTaskQueue. I’m not sure this is something we want to do. I also tried creating a cost char *arg[], mirroring other calls to TQ->AddTask. This didn’t work, and the TaskQueue was reading junk memory when it attempted to execute the command.

Do you have any ideas about how we might resolve #2? Since I can’t modify the argument list directly on Cmd, passing a new argument vector into TQ->AddTask would seem to be the right approach. I am just not sure how to make sure it doesn’t get cleaned up before it is used.