[SR-11898] Per module version of `target.swift-extra-clang-flags` setting

[swift-ci]

Previous ID	SR-11898
Radar	rdar://problem/57711772
Original Reporter	dylansturg (JIRA User)
Type	New Feature

Additional Detail from JIRA

Votes	0
Component/s	LLDB for Swift
Labels	New Feature
Assignee	@adrian-prantl
Priority	Medium

md5: 929fb7170a494f3fcbbc8113125d3b16

Issue Description:

Background
LLDB supports a global setting `target.swift-extra-clang-flags` to feeds additional flags from the compiler invocation into each SwiftASTContext created during the debugging session. These flags appear to be applied to every Swift module that's loaded during the debugging session.

In order to stop serializing debugging options (e.g. absolute search paths), @Adrian_Prantl suggested passing `-no-serialize-debugging-options` to the Swift compiler and passing the necessary Clang flags to LLDB using the `target.swift-extra-clang-flags` setting. https://forums.swift.org/t/improving-path-remappings-for-extraclangargs-swiftastcontext/26886/3

Feature Request
It would be nice to have a way to specify `target.swift-extra-clang-flags` on a per module basis. Each module might have separate Clang flags, and the project's Clang flags don't need to be used at all for system libraries (e.g. UIKit, Foundation, etc.).

Can LLDB support a setting similar to `target.swift-extra-clang-flags` that allows specifying Clang flags per-module? Perhaps a setting that's a per module map of a module name/identifier to a list of Clang flags like what is used for the existing `target.swift-extra-clang-flags` setting.

[beccadax]

@swift-ci create

[adrian-prantl]

more context:
-serialize-debugging-options is an AST serialization option and not part of the debug info. That means it is not affected by -fdebug-prefix-map. This makes it unsuitable for distributable or reproducible builds.

You would still need to ensure that all the per-module clang flags don't conflict. Is the goal here to allow a library author to ship prefix-map'able clang flags with each shared library?

[swift-ci]

Comment by Dylan Sturgeon (JIRA)

I agree it's still possible to have per-module conflicts, but much less likely (at least for the Bazel based builds that I'm working with). I'm definitely open to alternatives that remove the chance of conflicts; this proposal seemed like a decent compromise. You're correct that the goal is to allow libraries to specify/ship their own set of prefix-map'able Clang flags. When debugging an app, we will collect and apply all of the Clang flags from each library.

[adrian-prantl]

When you said "per module" are referring to what LLDB calls a module (= a dylib) or to a Swift module?

I would like to figure out what the imagined workflow is and then design a feature to support this. LLDB doesn't have a mechanism to tie settings to individual lldb::Modules let alone Swift modules. But if we work backwards from what the intended workflow is, I'm sure we can design something useful.

It would help to have a working example for this:
Let's say

1. dynamic library Foo depends on Clang library OpenSSL, it exports clang flags "-I${OPEN_SSL_INCLUDE_DIR}"
2. dynamic library Bar depends on Clang library z, it exports clang flags "-I${ZLIB_INCLUDE_DIR}"
3. these flags are combined into target.swift-extra-clang-flags
4. some mechanism writes out a remapping dictionary that contains "${OPEN_SSL_INCLUDE_DIR},/usr/local/include" and "${ZLIB_INCLUDE_DIR},~/zlib/include"
5. LLDB applies the prefix map to target.swift-extra-clang-flags

Does that accurately reflect you use-case? If not, please let me know, it's important for me to understand the requirements.

The way I'm seeing this, you need a custom script to perform step 4 anyway, so it could conceivably also collect the clang flags from all the dynamic libraries. For example the build system could write them into Foo.dsym/Resources/clang-flags.txt or some other previously-agreed location where the script from step 4 would find them.

If this is a viable workflow, there are things we can do in the tools to make it more convenient, such as printing warnings when paths mentioned in the extra clang flags don't exist; if this isn't I'm very interested in learning why not.

[swift-ci]

Comment by Dylan Sturgeon (JIRA)

I was originally referring to a Swift module, but a different granularity might make more sense. We don't generally create dylibs from libraries, instead everything is statically linked.

Your example is something we could do, but there are some concerns that the conglomerate of Clang flags might lead to inconsistencies between the Swift compiler invocation(s) to build originally and the state recreated by LLDB while debugging. One example that came up was potentially for finding the wrong header due to order/presence of different include dirs used by LLDB versus the Swift compiler when building originally. Collecting the flags and creating a dictionary of the flags is certainly feasible from our perspective.

Here's an example of what happens in our build system:

Suppose you have an Objective-C library A that depends on another Objective-C library B. They each have an include dir and define: "A_INCLUDE_DIR", "-DA" and "B_INCLUDE_DIR", "-DB" respectively.

Now suppose there's also a Swift library S that depends on A, which has an include dir "S_INCLUDE_DIR".

We can collect all of the include dirs and defines while building A, B, and S. I think that's steps 1 and 2 from your example? And we can take those outputs and create a remapping dictionary or massage them into some other format as needed. For now, I'm merging them all together into a single target.swift-extra-clang-flags value. For the current setup using that single target.swift-extra-clang-flags, I think (correct me if I'm wrong) LLDB assumes all flags apply to every module that it touches. By applying all include dirs and defines, there's a possibility that the compiler invocations from LLDB discover different headers or headers in a different context (e.g. due to different defines) than was originally discovered by the Swift compiler at build time.

I think it would be more reliable (i.e. less likely to run into conflicts/inconsistencies) if I could instruct LLDB which include dirs and defines are specifically for A, which are for B, and which are for S.

[adrian-prantl]

> One example that came up was potentially for finding the wrong header due to order/presence of different include dirs used by LLDB versus the Swift compiler when building originally.

I would expect this problem to get worse if you have per-module Swift flags. You may not be aware that even for something like printing the value of a variable LLDB has to perform most operations in the global expression context in order to perform dynamic type resolution (think of an interface defined via a protocol type where the actual dynamic type came from a different library). If you have per-module flags, how do you guarantee a deterministic order between the flags?

Looking at your example, I wonder if I perhaps misunderstood your intention initially: it sounds like you are more interested in isolating the flags needed for different modules to avoid cross-pollution rather than a distribution mechanism?

If that is the case, at the moment neither LLDB nor the Swift compiler are architecturally set up for this. The Swift compiler uses a single Clang compiler invocation for all Clang module imports. We could conceivably have LLDB create multiple Clang invocations for top-level module imports and then ASTImport them into a shared context and vend that to Swift directly, similar to how DWARFImporter works.

That said, we might be better of just productizing DWARFImporter itself to be good enough to cover your use-case: While compiling the binary .pcm artifacts for Clang modules, clang also emit DWARF debug information for (most) declarations in the Clang module. The DWARFImporter is a new LLDB component I am working on that can find a Clang declaration in DWARF, create a Clang AST from it and then vend that to Swift's ClangImporter, side-stepping the clang module import. It is enabled by default on swift/master, however there is way of forcing it as an import mechanism at the moment; it will only kick in when a Clang module import failed as a fallback.

[swift-ci]

Comment by Dylan Sturgeon (JIRA)

You're correct that the concern is about cross-pollution, not a distribution mechanism.

It sounds like rather than try to invest more in improving usage of Clang flags with LLDB, it would be better to invest in the existing DWARFImporter so that we can use DWARF debug information. Removing the Clang module import step from LLDB would be helpful, because I think it will entirely remove the concerns with Clang flag cross-pollution. If the DWARFImporter is an option, then I don't think this per-module Clang flags feature is necessary.