Following the work done by @jdevlieghere in D143690, this changes how Clang module build
events are emitted.
Instead of one Progress event per module being built, a single Progress event is used to
encompass all modules, and each module build is sent as an `Increment` update.
Differential Revision: https://reviews.llvm.org/D147248
Update the Clang diagnostic consumer (in ClangModulesDeclVendor) to report
progress on Clang module builds, as both progress events and expression logs.
Module build remarks are enabled by with clang's `-Rmodule-build` flag.
With this change, command line users of lldb will see progress events showing
which modules are being built, and - by how long they stay on screen - how much
time it takes to build them. IDEs that show progress events can show these
updates if desired.
This does not show module-import remarks, although that may be added as a
future change.
Differential Revision: https://reviews.llvm.org/D140056
D127284 introduced a new language option which is not benign from modules
perspective. Before this patch lldb would set up the compiler invocation and
later enable incremental processing. Post-D127284 this does not work because
the option causes a module hash mismatch for implicit modules.
In addition, D127284 enables parsing statements on the global scope if
incremental processing is on and thus `syntax_error_for_lldb_to_find` was
rightfully not recognized as a declaration and is considered a statement
which produces a slightly different diagnostic.
Thanks to Michael Buch for the help in understanding this issue. This patch
should appease the lldb bots.
More discussion available at: https://reviews.llvm.org/D127284
When a process gets restarted TypeSystem objects associated with it
may get deleted, and any CompilerType objects holding on to a
reference to that type system are a use-after-free in waiting. Because
of the SBAPI, we don't have tight control over where CompilerTypes go
and when they are used. This is particularly a problem in the Swift
plugin, where the scratch TypeSystem can be restarted while the
process is still running. The Swift plugin has a lock to prevent
abuse, but where there's a lock there can be bugs.
This patch changes CompilerType to store a std::weak_ptr<TypeSystem>.
Most of the std::weak_ptr<TypeSystem>* uglyness is hidden by
introducing a wrapper class CompilerType::WrappedTypeSystem that has a
dyn_cast_or_null() method. The only sites that need to know about the
weak pointer implementation detail are the ones that deal with
creating TypeSystems.
rdar://101505232
Differential Revision: https://reviews.llvm.org/D136650
In C++20 modules imports must be together and at the start of the module.
Rather than growing more ad-hoc flags to test state, this keeps track of the
phase of of a valid module TU (first decl, global module frag, module,
private module frag). If the phasing is broken (with some diagnostic) the
pattern does not conform to a valid C++20 module, and we set the state
accordingly.
We can thus issue diagnostics when imports appear in the wrong places and
decouple the C++20 modules state from other module variants (modules-ts and
clang modules). Additionally, we attempt to diagnose wrong imports before
trying to find the module where possible (the latter will generally emit an
unhelpful diagnostic about the module not being available).
Although this generally simplifies the handling of C++20 module import
diagnostics, the motivation was that, in particular, it allows detecting
invalid imports like:
import module A;
int some_decl();
import module B;
where being in a module purview is insufficient to identify them.
Differential Revision: https://reviews.llvm.org/D118893
Recently we observed high memory pressure caused by clang during some parallel builds.
We discovered that we have several projects that have a large number of #define directives
in their TUs (on the order of millions), which caused huge memory consumption in clang due
to a lot of allocations for MacroInfo. We would like to reduce the memory overhead of
clang for a single #define to reduce the memory overhead for these files, to allow us to
reduce the memory pressure on the system during highly parallel builds. This change achieves
that by removing the SmallVector in MacroInfo and instead storing the tokens in an array
allocated using the bump pointer allocator, after all tokens are lexed.
The added unit test with 1000000 #define directives illustrates the problem. Prior to this
change, on arm64 macOS, clang's PP bump pointer allocator allocated 272007616 bytes, and
used roughly 272 bytes per #define. After this change, clang's PP bump pointer allocator
allocates 120002016 bytes, and uses only roughly 120 bytes per #define.
For an example test file that we have internally with 7.8 million #define directives, this
change produces the following improvement on arm64 macOS: Persistent allocation footprint for
this test case file as it's being compiled to LLVM IR went down 22% from 5.28 GB to 4.07 GB
and the total allocations went down 14% from 8.26 GB to 7.05 GB. Furthermore, this change
reduced the total number of allocations made by the system for this clang invocation from
1454853 to 133663, an order of magnitude improvement.
The recommit fixes the LLDB build failure.
Differential Revision: https://reviews.llvm.org/D117348
Most of our code was including Log.h even though that is not where the
"lldb" log channel is defined (Log.h defines the generic logging
infrastructure). This worked because Log.h included Logging.h, even
though it should.
After the recent refactor, it became impossible the two files include
each other in this direction (the opposite inclusion is needed), so this
patch removes the workaround that was put in place and cleans up all
files to include the right thing. It also renames the file to LLDBLog to
better reflect its purpose.
LLDB is currently always activating C++ when parsing expressions as LLDB itself
is using C++ features when creating the final AST that will be codegen'd
(specifically, references to variables, namespaces and using declarations are
used).
This is causing problems for users that have variables in non-C++ programs (e.g.
plain C or Objective-C) that have names which are keywords in C++. Expressions
referencing those variables fail to parse as LLDB's Clang parser thinks those
identifiers are C++ keywords and not identifiers that may belong to a
declaration.
We can't just disable C++ in the expression parser for those situations as
replacing the functionality of the injected C++ code isn't trivial. So this
patch is just disabling most keywords that are exclusive to C++ in LLDB's Clang
parser when we are in a non-C++ expression. There are a few keywords we can't
disable for now:
* `using` as that's currently used in some situations to inject variables into the expression function.
* `__null` as that's used by LLDB to define `NULL`/`Nil`/`nil`.
Getting rid of these last two keywords is possible but is a large enough change
that this will be handled in follow up patches.
Note that this only changes the keyword status of those tokens but this patch
does not remove any C++ functionality from the expression parser. The type
system still follows C++ rules and so does the rest of the expression parser.
There is another small change that gives the hardcoded macro definitions in LLDB
a higher precedence than the macros imported from the Objective-C modules. The
reason for this is that the Objective-C modules in LLDB are actually parsed in
Objective-C++ mode and they end up providing the C++ definitions of certain
system macros (like `NULL` being defined as `nullptr`). So we have to move the
LLDB definition forward and surround the definition from the module with an
`#ifdef` to make sure that we use the correct LLDB definition that doesn't
reference C++ keywords. Or to give an example, this is how the expression source
code changes:
Before:
```
#define NULL (nullptr) // injected module definition
#ifndef NULL
#define NULL (__null) // hardcoded LLDB definition
#endif
```
After:
```
#ifndef NULL
#define NULL (__null) // hardcoded LLDB definition
#endif
#ifndef NULL
#define NULL (nullptr) // injected module definition
#endif
```
Fixes rdar://10356912
Reviewed By: shafik
Differential Revision: https://reviews.llvm.org/D82770
Extract all the provider related logic from Reproducer.h and move it
into its own header ReproducerProvider.h. These classes are seeing most
of the development these days and this reorganization reduces
incremental compilation from ~520 to ~110 files when making changes to
the new header.
Summary:
When the ClangModulesDeclVendor currently fails it just prints very basic and often incomplete diagnostics without any source locations:
```
(lldb) p @import Foundation
error: while importing modules:
'foo/bar.h' file not found
could not build module 'Darwin'
[...]
```
or even just
```
(lldb) p @import Foundation
error: while importing modules:
could not build module 'Darwin'
[...]
```
These diagnostics help neither the user nor us with figuring out what is the reason for the failure.
This patch wires up a full TextDiagnosticPrinter in the ClangModulesDeclVendor and makes
sure we always return the error stream to the user when we fail to compile our modules.
Fixes rdar://63216849
Reviewers: aprantl, jdoerfert
Reviewed By: aprantl
Subscribers: JDevlieghere
Differential Revision: https://reviews.llvm.org/D79947
Summary:
This change represents the move of ClangASTImporter, ClangASTMetadata,
ClangExternalASTSourceCallbacks, ClangUtil, CxxModuleHandler, and
TypeSystemClang from lldbSource to lldbPluginExpressionParserClang.h
This explicitly removes knowledge of clang internals from lldbSymbol,
moving towards a more generic core implementation of lldb.
Reviewers: JDevlieghere, davide, aprantl, teemperor, clayborg, labath, jingham, shafik
Subscribers: emaste, mgorny, arphaman, jfb, usaxena95, lldb-commits
Tags: #lldb
Differential Revision: https://reviews.llvm.org/D73661
This adds a conversion function from clang::Decl to CompilerDecl. It checks
that the TypeSystemClang in the CompilerDecl actually fits to the clang::Decl
AST during creation, thus preventing the creation of CompilerDecl instances with
inconsistent state.
This is how it should've been and brings it more in line with
std::string_view. There should be no functional change here.
This is mostly mechanical from a custom clang-tidy check, with a lot of
manual fixups. It uncovers a lot of minor inefficiencies.
This doesn't actually modify StringRef yet, I'll do that in a follow-up.
Summary:
A *.cpp file header in LLDB (and in LLDB) should like this:
```
//===-- TestUtilities.cpp -------------------------------------------------===//
```
However in LLDB most of our source files have arbitrary changes to this format and
these changes are spreading through LLDB as folks usually just use the existing
source files as templates for their new files (most notably the unnecessary
editor language indicator `-*- C++ -*-` is spreading and in every review
someone is pointing out that this is wrong, resulting in people pointing out that this
is done in the same way in other files).
This patch removes most of these inconsistencies including the editor language indicators,
all the different missing/additional '-' characters, files that center the file name, missing
trailing `===//` (mostly caused by clang-format breaking the line).
Reviewers: aprantl, espindola, jfb, shafik, JDevlieghere
Reviewed By: JDevlieghere
Subscribers: dexonsmith, wuzish, emaste, sdardis, nemanjai, kbarton, MaskRay, atanasyan, arphaman, jfb, abidh, jsji, JDevlieghere, usaxena95, lldb-commits
Tags: #lldb
Differential Revision: https://reviews.llvm.org/D73258
Summary:
This commit renames ClangASTContext to TypeSystemClang to better reflect what this class is actually supposed to do
(implement the TypeSystem interface for Clang). It also gets rid of the very confusing situation that we have both a
`clang::ASTContext` and a `ClangASTContext` in clang (which sometimes causes Clang people to think I'm fiddling
with Clang's ASTContext when I'm actually just doing LLDB work).
I also have plans to potentially have multiple clang::ASTContext instances associated with one ClangASTContext so
the ASTContext naming will then become even more confusing to people.
Reviewers: #lldb, aprantl, shafik, clayborg, labath, JDevlieghere, davide, espindola, jdoerfert, xiaobai
Reviewed By: clayborg, labath, xiaobai
Subscribers: wuzish, emaste, nemanjai, mgorny, kbarton, MaskRay, arphaman, jfb, usaxena95, jingham, xiaobai, abidh, JDevlieghere, lldb-commits
Tags: #lldb
Differential Revision: https://reviews.llvm.org/D72684
Summary:
I often struggle to understand what exactly LLDB is doing by looking at our expression evaluation logging as our messages look like this:
```
CompleteTagDecl[2] on (ASTContext*)0x7ff31f01d240 Completing (TagDecl*)0x7ff31f01d568 named DeclName1
```
From the log messages it's unclear what this ASTContext is. Is it the scratch context, the expression context, some decl vendor context or a context from a module?
The pointer value isn't helpful for anyone unless I'm in a debugger where I could inspect the memory at the address. But even with a debugger it's not easy to
figure out what this ASTContext is without having deeper understanding about all the different ASTContext instances in LLDB (e.g., valid SourceLocation
from the file system usually means that this is the Objective-C decl vendor, a file name from multiple expressions is probably the scratch context, etc.).
This patch adds a name field to ClangASTContext instances that we can use to store a name which can be used for logging and debugging. With this
our log messages now look like this:
```
CompleteTagDecl[2] on scratch ASTContext. Completing (TagDecl*)0x7ff31f01d568 named Foo
```
We can now also just print a ClangASTContext from the debugger and see a useful name in the `m_display_name` field, e.g.
```
m_display_name = "AST for /Users/user/test/main.o";
```
Reviewers: shafik, labath, JDevlieghere, mib
Reviewed By: shafik
Subscribers: clayborg, lldb-commits
Tags: #lldb
Differential Revision: https://reviews.llvm.org/D72391
Summary:
This code is handling debug info paths starting with /proc/self/cwd,
which is one of the mechanisms people use to obtain "relocatable" debug
info (the idea being that one starts the debugger with an appropriate
cwd and things "just work").
Instead of resolving the symlinks inside DWARFUnit, we can do the same
thing more elegantly by hooking into the existing Module path remapping
code. Since llvm::DWARFUnit does not support any similar functionality,
doing things this way is also a step towards unifying llvm and lldb
dwarf parsers.
Reviewers: JDevlieghere, aprantl, clayborg, jdoerfert
Subscribers: lldb-commits
Tags: #lldb
Differential Revision: https://reviews.llvm.org/D71770
Instead of returning NamedDecls and then calling GetASTContext
to find back the ClangASTContext we used can just implement the
FindDecl variant that returns CompilerDecls (and implement the
other function by throwing away the ClangASTContext part of the
compiler decl).
Summary:
As discussed on the mailing list [1] we have to make a decision for how to proceed with the modern-type-lookup.
This patch removes modern-type-lookup from LLDB. This just removes all the code behind the modern-type-lookup
setting but it does *not* remove any code from Clang (i.e., the ExternalASTMerger and the clang-import-test stay around
for now).
The motivation for this is that I don't think that the current approach of implementing modern-type-lookup
will work out. Especially creating a completely new lookup system behind some setting that is never turned on by anyone
and then one day make one big switch to the new system seems wrong. It doesn't fit into the way LLVM is developed and has
so far made the transition work much more complicated than it has to be.
A lot of the benefits that were supposed to come with the modern-type-lookup are related to having a better organization
in the way types move across LLDB and having less dependencies on unrelated LLDB code. By just looking at the current code (mostly
the ClangASTImporter) I think we can reach the same goals by just incrementally cleaning up, documenting, refactoring
and actually testing the existing code we have.
[1] http://lists.llvm.org/pipermail/lldb-dev/2019-December/015831.html
Reviewers: shafik, martong
Subscribers: rnkovacs, christof, arphaman, JDevlieghere, usaxena95, lldb-commits, friss
Tags: #lldb
Differential Revision: https://reviews.llvm.org/D71562
We have no use case in LLDB where we actually do want to change the ASTContext after
it the ClangASTContext has been constructed. All callers of setASTContext are just setting
the ASTContext directly after construction, so we might as well make this a Constructor
instead of supporting this tricky use case.
llvm-svn: 373330
Summary:
This introduces a layer between DeclVendor and the currently implemented
DeclVendors (ClangModulesDeclVendor and AppleObjCDeclVendor). This
allows the removal of DeclVendor::GetImporterSource which is extremely
clang-specific, freeing up the interface to be more general.
A good follow up to this would be to remove the remaining instances of
clang in DeclVendor, either by moving things to ClangDeclVendor or by
using wrappers (e.g. CompilerDecl instead of clang::NamedDecl).
Differential Revision: https://reviews.llvm.org/D66451
llvm-svn: 369424
Summary:
Currently, clang's FileManager uses NULL as an indicator that a particular file
did not exist, but would not propagate errors like permission issues. Instead,
teach FileManager to use llvm::ErrorOr internally and return rich errors for
failures.
Reviewers: arphaman, bruno, martong, shafik
Subscribers: nemanjai, kbarton, MaskRay, jkorous, dexonsmith, kadircet, jsji, cfe-commits, lldb-commits
Tags: #clang, #lldb
Differential Revision: https://reviews.llvm.org/D65534
llvm-svn: 367618
Summary:
We assume in LLDB that every type comes from an ASTContext with an associated ClangASTContext.
However the types inside the ClangModuleDeclVendor don't have a ClangASTContext so we end up
crashing whenever we create a CompilerType for one of these types.
Simplest way to trigger this bug is to just look up NSObject from a module:
(lldb) expr @import Foundation
(lldb) type lookup NSObject
Assertion failed: (m_type_system != nullptr), function CompilerType, file /Users/teemperor/llvm1/llvm-project/lldb/source/Symbol/CompilerType.cpp, line 39.
This patch just creates a ClangASTContext for the ASTContext used by ClangModuleDeclVendor.
Reviewers: davide, shafik
Reviewed By: davide
Subscribers: lldb-commits
Tags: #lldb
Differential Revision: https://reviews.llvm.org/D64989
llvm-svn: 366653
Summary:
Currently the ClangModulesDeclVendor is spamming the expression log with the compiler flags it is using, which creates a log that looks like this:
```
clang
-fmodules
-fimplicit-module-maps
```
This patch removes all these newlines and just prints the compiler flags in one line as you see in the command line:
```
clang -fmodules -fimplicit-module-maps [...]
```
Reviewers: shafik, davide
Reviewed By: davide
Subscribers: davide, abidh, lldb-commits
Tags: #lldb
Differential Revision: https://reviews.llvm.org/D64858
llvm-svn: 366347
Before a Debugger gets a Target, target settings are routed to a global set
of settings. Even without this, some part of the LLDB which exist independently
of the Debugger object (the Module cache, the Symbol vendors, ...) access
directly the global default store for those settings.
Of course, if you modify one of those global settings while they are being read,
bad things happen. We see this quite a bit with FileSpecList settings. In
particular, we see many cases where one debug session changes
target.exec-search-paths while another session starts up and it crashes when
one of those accesses invalid FileSpecs.
This patch addresses the specific FileSpecList issue by adding locking to
OptionValueFileSpecList and never returning by reference.
Reviewers: clayborg
Subscribers: lldb-commits
Differential Revision: https://reviews.llvm.org/D60468
llvm-svn: 359028
Remove CompilerInstance::VirtualFileSystem and
CompilerInstance::setVirtualFileSystem, instead relying on the VFS in
the FileManager. CompilerInstance and its clients already went to some
trouble to make these match. Now they are guaranteed to match.
As part of this, I added a VFS parameter (defaults to nullptr) to
CompilerInstance::createFileManager, to avoid repeating construction
logic in clients that just wanted to customize the VFS.
https://reviews.llvm.org/D59377
llvm-svn: 357037
My apologies for the large patch. With the exception of ConstString.h
itself it was entirely produced by sed.
ConstString has exactly one const char * data member, so passing a
ConstString by reference is not any more efficient than copying it by
value. In both cases a single pointer is passed. But passing it by
value makes it harder to accidentally return the address of a local
object.
(This fixes rdar://problem/48640859 for the Apple folks)
Differential Revision: https://reviews.llvm.org/D59030
llvm-svn: 355553
