This reverts commit d6713ad80d.
This changed was reverted because of greendragon failures such
as
Unresolved Tests (2):
lldb-api :: debuginfod/Normal/TestDebuginfod.py
lldb-api :: debuginfod/SplitDWARF/TestDebuginfodDWP.py
I believe I've got the tests properly configured to only run on Linux
x86(_64), as I don't have a Linux AArch64/Arm device to diagnose what's
going wrong with the tests (I suspect there's some issue with generating
`.note.gnu.build-id` sections...)
The actual code fixes have now been reviewed 3 times:
https://github.com/llvm/llvm-project/pull/79181 (moved shell tests to
API tests), https://github.com/llvm/llvm-project/pull/85693 (Changed
some of the testing infra), and
https://github.com/llvm/llvm-project/pull/86812 (didn't get the tests
configured quite right). The Debuginfod integration for symbol
acquisition in LLDB now works with the `executable` and `debuginfo`
Debuginfod network requests working properly for normal, `objcopy
--only-keep-debug` stripped, split-dwarf, and `objcopy
--only-keep-debug` stripped *plus* split-dwarf symbols/binaries.
The reasons for the multiple attempts have been tests on platforms I
don't have access to (Linux AArch64/Arm + MacOS x86_64). I believe I've
got the tests properly disabled for everything except for Linux x86(_64)
now. I've built & tested on MacOS AArch64 and Linux x86_64.
---------
Co-authored-by: Kevin Frei <freik@meta.com>
This is a followup to
https://github.com/llvm/llvm-project/pull/86359
"[lldb] [ObjectFileMachO] LLVM_COV is not mapped into firmware memory
(#86359)"
where I treat LLVM_COV segments in a Mach-O binary as non-loadable.
There is another codepath in
`DynamicLoaderStatic::LoadAllImagesAtFileAddresses` which is called to
set the load addresses for a Module to the file addresses. It has no
logic to detect a segment that is not loaded in virtual memory
(ObjectFileMachO::SectionIsLoadable), so it would set the load address
for this LLVM_COV segment to the file address and shadow actual code,
breaking lldb behavior.
This method currently sets the load address for any section that doesn't
have a load address set already. This presumes that a Module was added
to the Target, some mechanism set the correct load address for SOME
segments, and then this method is going to set the other segments to a
no-slide value, assuming they were forgotten.
ObjectFile base class doesn't, today, vend a SectionIsLoadable method,
but we do have ObjectFile::SetLoadAddress and at a higher level,
Module::SetLoadAddress, when we're setting the same slide to all
segments.
That's the behavior we want in this method. If any section has a load
address, we don't touch this Module. Otherwise we set all sections to
have a load address that is the same as the file address.
I also audited the other parts of lldb that are calling
SectionList::SectionLoadAddress and looked if they should be more
correctly using Module::SetLoadAddress for the entire binary. But in
most cases, we have the potential for different slides for different
sections so this section-by-section approach must be taken.
rdar://125800290
The previous diff (and it's subsequent fix) were reverted as the tests
didn't work properly on the AArch64 & ARM LLDB buildbots. I made a
couple more minor changes to tests (from @clayborg's feedback) and
disabled them for non Linux-x86(_64) builds, as I don't have the ability
do anything about an ARM64 Linux failure. If I had to guess, I'd say the
toolchain on the buildbots isn't respecting the `-Wl,--build-id` flag.
Maybe, one day, when I have a Linux AArch64 system I'll dig in to it.
From the reverted PR:
I've migrated the tests in my
https://github.com/llvm/llvm-project/pull/79181 from shell to API (at
@JDevlieghere's suggestion) and addressed a couple issues that were
exposed during testing.
The tests first test the "normal" situation (no DebugInfoD involvement,
just normal debug files sitting around), then the "no debug info"
situation (to make sure the test is seeing failure properly), then it
tests to validate that when DebugInfoD returns the symbols, things work
properly. This is duplicated for DWP/split-dwarf scenarios.
---------
Co-authored-by: Kevin Frei <freik@meta.com>
Size was clearly not correct here. This call has been here since
the initial reformat of all of lldb so it has likely always been
incorrect.
(although registers don't typically have an endian, they are
just values, in the remote protocol register data is in target
endian)
This might have been a problem for Neon registers on big endian
AArch64, but only if the debug server describes them as integers.
lldb-server does not, they've always been vectors which doesn't
take this code path.
Not adding a test because the way I've mocked up a big endian
target in the past is using s390x as the architecture. This
apparently has some form of vector extension that may be 128 bit
but lldb doesn't support it.
This patch introduces a new `IterationMarker` enum (happy to take
alternative name suggestions), which callbacks, like the one in
`SymbolFileDWARFDebugMap::ForEachSymbolFile`, can return in order to
indicate whether the caller should continue iterating or bail.
For now this patch just changes the `ForEachSymbolFile` callback to use
this new enum. In the future we could change the various
`DWARFIndex::GetXXX` callbacks to do the same.
This makes the callbacks easier to read and hopefully reduces the chance
of bugs like https://github.com/llvm/llvm-project/pull/87177.
While adding register fields I realised that the AUXV values for Linux
and FreeBSD disagree here.
So I've added a FreeBSD specific HWCAP value that I can use from FreeBSD
specific code.
The alternative is translating GetAuxValue calls depending on platform,
which requires that we know what we are at all times.
Another way would be to convert the entries' values when we construct
the AuxVector but the platform specific call that reads the data just
returns a raw array. So adding another layer here is more disruption.
These are probably actually unreachable - perhaps an lldb developer
would be interested in rephrasing this change to move the new cases into
some unreachable/unsupported bucket, rather than my half-hearted guess
at what the desired behavior would be (completely untested, because
they're probably untestable/unreachable - maybe debugging from modules?)
We check if the next character after `N.` is `*` before we check its
length. Using `split` on the string is cleaner and less error prone than
using indices with `find` and `substr`.
Note: this does not make `N.` mean anything, it just prevents assertion
failures. `N.` is treated the same as an unrecognized breakpoint name:
```
(lldb) breakpoint enable 1
1 breakpoints enabled.
(lldb) breakpoint enable 1.*
1 breakpoints enabled.
(lldb) breakpoint enable 1.
0 breakpoints enabled.
(lldb) breakpoint enable xyz
0 breakpoints enabled.
```
Found via LLDB fuzzers.
We have the ability to load .dwp files with a .debug_info.dwo section
that exceeds 4GB. There were 4 locations that were using 32 bit offsets
and lengths to extract variable locations, and if a DIE was over the 4GB
barrier, we would truncate the block offset for the variable locations
and the variable expression would be garbage. This fixes the issues. It
isn't possible to add a test for this as we don't want to create a 4GB
.dwp file on test machines.
An inverted condition causes `SymbolFileDWARFDebugMap::FindTypes` to
bail out after inspecting the first .o file in each module.
The same kind of bug is found in
`SymbolFileDWARFDebugMap::ParseDeclsForContext`.
Correct both early exit conditions and add a regression test for lookup
of up a type defined in a secondary compilation unit.
Fixes#87176
Avoid deadlocks in the Alarm class by releasing the lock before invoking
callbacks. This deadlock manifested itself in the ProgressManager:
1. On the main thread, the ProgressManager acquires its lock in
ProgressManager::Decrement and calls Alarm::Create.
2. On the main thread, the Alarm acquires its lock in Alarm::Create.
3. On the alarm thread, the Alarm acquires its lock after waiting on the
condition variable and calls ProgressManager::Expire.
4. On the alarm thread, the ProgressManager acquires its lock in
ProgressManager::Expire.
Note how the two threads are acquiring the locks in different orders.
Deadlocks can be avoided by always acquiring locks in the same order,
but since the two mutexes here are private implementation details,
belong to different classes, that's not straightforward. Luckily, we
don't need to have the Alarm mutex locked when invoking the callbacks.
That exactly how this patch solves the issue.
This is fixing a report from ubsan which I don't think is super high
value, but our testsuite hits it on
TestDataFormatterObjCNSContainer.py so I'd like to work around it. We
are getting
```
runtime error: left shift of negative value -8827055269646171913
3159 int64_t data_payload_signed =
3160 ((int64_t)((int64_t)unobfuscated
-> 3161 << m_objc_debug_taggedpointer_ext_payload_lshift) >>
3162 m_objc_debug_taggedpointer_ext_payload_rshift);
```
At this point `unobfuscated` is 0x85800000000000f7 and
`m_objc_debug_taggedpointer_ext_payload_lshift` is 9, so
`(int64_t)0x85800000000000f7<<9` shifts off the "sign" bit and then some
zeroes etc, and that's how we get this error.
We're only trying to extract some bits in the middle of the doubleword,
so the fact that we're "losing" the sign is not a bug. Change the inner
cast to (uint64_t).
In
commit 2f63718f85
Author: Jason Molenda <jmolenda@apple.com>
Date: Tue Mar 26 09:07:15 2024 -0700
[lldb] Don't clear a Module's UnwindTable when adding a SymbolFile
(#86603)
I stopped clearing a Module's UnwindTable when we add a SymbolFile to
avoid the memory management problems with adding a symbol file
asynchronously while the UnwindTable is being accessed on another
thread. This broke the target-symbols-add-unwind.test shell test on
Linux which removes the DWARF debub_frame section from a binary, loads
it, then loads the unstripped binary with the DWARF debug_frame section
and checks that the UnwindPlans for a function include debug_frame.
I originally decided that I was willing to sacrifice the possiblity of
additional unwind sources from a symbol file because we rely on assembly
emulation so heavily, they're rarely critical. But there are targets
where we we don't have emluation and rely on things like DWARF
debug_frame a lot more, so this probably wasn't a good choice.
This patch adds a new UnwindTable::Update method which looks for any new
sources of unwind information and adds it to the UnwindTable, and calls
that after a new SymbolFile has been added to a Module.
This implements coalescing of progress events using a timeout, as
discussed in the RFC on Discourse [1]. This PR consists of two commits
which, depending on the feedback, I may split up into two PRs. For now,
I think it's easier to review this as a whole.
1. The first commit introduces a new generic `Alarm` class. The class
lets you to schedule a function (callback) to be executed after a given
timeout expires. You can cancel and reset a callback before its
corresponding timeout expires. It achieves this with the help of a
worker thread that sleeps until the next timeout expires. The only
guarantee it provides is that your function is called no sooner than the
requested timeout. Because the callback is called directly from the
worker thread, a long running callback could potentially block the
worker thread. I intentionally kept the implementation as simple as
possible while addressing the needs for the `ProgressManager` use case.
If we want to rely on this somewhere else, we can reassess whether we
need to address those limitations.
2. The second commit uses the Alarm class to coalesce progress events.
To recap the Discourse discussion, when multiple progress events with
the same title execute in close succession, they get broadcast as one to
`eBroadcastBitProgressCategory`. The `ProgressManager` keeps track of
the in-flight progress events and when the refcount hits zero, the Alarm
class is used to schedule broadcasting the event. If a new progress
event comes in before the alarm fires, the alarm is reset (and the
process repeats when the new progress event ends). If no new event comes
in before the timeout expires, the progress event is broadcast.
[1]
https://discourse.llvm.org/t/rfc-improve-lldb-progress-reporting/75717/
Fixing a crash in lldb when `symbols.auto-download` setting is enabled.
When doing a backtrace, this feature has lldb search for a SymbolFile
for stack frames when we are backtracing, and add them either
synchoronously or asynchronously, depending on the specific setting
used.
Module::SetSymbolFileFileSpec clears the Module's UnwindTable, once we
find a new SymbolFile. We may be adding a source of unwind information
that we did not have when lldb was working only with the executable
binary.
What happens in practice is that we're using a reference to the Module's
UnwindTable, and then the other thread getting the SymbolFile clears it
and now the first thread is referring to freed memory and we can crash.
When built with address sanitizer, it crashes much more reliably.
Given that unwind information used for exception handling -- eh_frame,
compact unwind -- is present in executable binaries, the only thing
we're likely to *add* would be DWARF's `debug_frame` if that was also
available. The actual value of re-creating the UnwindTable when we have
added a SymbolFile is not large.
I also tried fixing this by changing the Module to have a shared_ptr to
the UnwindTable, so we could have two different UnwindTable's in use
simultaneously for a brief period. This would be fine TODAY, but it
introduces a very subtle bug that someone will have a heck of a time
figuring out in the future.
In the end, I believe the safest approach is to sacrifice the possible
marginal gain of reconstructing the UnwindTable once a SymbolFile has
been added, to sidestep this whole problem area.
Also, in `Module::GetUnwindTable()`, call `DownloadSymbolFileAsync`
before we create the UnwindTable for the first time, in case the symbol
file is fetched synchronously, we will have it for that possible
marginal gain.
The idea behind the address-expression is that it handles all the common
expressions that produce addresses. It handles actual valid expressions
that return a scalar, and it handles useful cases that the various
source languages don't support. At present, the fallback handles:
<symbol_name>{+-}<offset>
which isn't valid C but is very handy.
This patch adds handling of:
$<reg_name>
and
$<reg_name>{+-}<offset>
That's kind of pointless in C because the C expression parser handles
that expression already. But some languages don't have a straightforward
way to represent register values like this (swift) so having this
fallback is quite a quality of life improvement.
I added a test which tests that I didn't mess up either of these
fallbacks, though it doesn't test the actually handling of registers
that I added, since the expression parser for C succeeds in that case
and returns before this code gets run.
I will add a test on the swift fork for that checks that this works the
same way for a swift frame after this check.
It is possible to gather code coverage in a firmware environment, where
the __LLVM_COV segment will not be mapped in memory but does exist in
the binary, see
https://llvm.org/devmtg/2020-09/slides/PhippsAlan_EmbeddedCodeCoverage_LLVM_Conf_Talk_final.pdf
The __LLVM_COV segment in the binary happens to be at the same address
as the __DATA segment, so if lldb treats this segment as loaded, it
shadows the __DATA segment and address->symbol resolution can fail.
For these non-userland code cases, we need to mark __LLVM_COV as not a
loadable segment.
rdar://124475661
This implements coalescing of progress events using a timeout, as
discussed in the RFC on Discourse [1]. This PR consists of two commits
which, depending on the feedback, I may split up into two PRs. For now,
I think it's easier to review this as a whole.
1. The first commit introduces a new generic `Alarm` class. The class
lets you to schedule a function (callback) to be executed after a given
timeout expires. You can cancel and reset a callback before its
corresponding timeout expires. It achieves this with the help of a
worker thread that sleeps until the next timeout expires. The only
guarantee it provides is that your function is called no sooner than the
requested timeout. Because the callback is called directly from the
worker thread, a long running callback could potentially block the
worker thread. I intentionally kept the implementation as simple as
possible while addressing the needs for the `ProgressManager` use case.
If we want to rely on this somewhere else, we can reassess whether we
need to address those limitations.
2. The second commit uses the Alarm class to coalesce progress events.
To recap the Discourse discussion, when multiple progress events with
the same title execute in close succession, they get broadcast as one to
`eBroadcastBitProgressCategory`. The `ProgressManager` keeps track of
the in-flight progress events and when the refcount hits zero, the Alarm
class is used to schedule broadcasting the event. If a new progress
event comes in before the alarm fires, the alarm is reset (and the
process repeats when the new progress event ends). If no new event comes
in before the timeout expires, the progress event is broadcast.
[1]
https://discourse.llvm.org/t/rfc-improve-lldb-progress-reporting/75717/
commit e66b670f3b
Author: Nathan Lanza <nathanlanza@gmail.com>
Date: Thu Mar 21 19:53:48 2024 -0400
triggers:
lldb/source/Plugins/TypeSystem/Clang/TypeSystemClang.cpp:478:16:
error: enumeration value 'CIR' not handled in switch
[-Werror,-Wswitch]
This patch teaches lldb to handle clang::Language::CIR the same way as
clang::Language::LLVM_IR.
Finally getting back to Debuginfod tests:
I've migrated the tests in my [earlier
PR](https://github.com/llvm/llvm-project/pull/79181) from shell to API
(at @JDevlieghere's suggestion) and addressed a couple issues that came
about during testing.
The tests first test the "normal" situation (no DebugInfoD involvement,
just normal debug files sitting around), then the "no debug info"
situation (to make sure the test is seeing failure properly), then it
tests to validate that when Debuginfod returns the symbols, things work
properly. This is duplicated for DWP/split-dwarf scenarios.
---------
Co-authored-by: Kevin Frei <freik@meta.com>
This is another step towards supporting DWARF5 checksums and inline
source code in LLDB. This is a reland of #85468 but without the
functional change of storing the support file from the line table (yet).
AddressableBits is in the Utility module of LLDB. It currently directly
refers to Process, which is from the Target LLDB module. This is a
layering violation which concretely means that it is impossible to link
anything that uses Utility without it also using Target as well. This is
generally not an issue for LLDB (since everything is built together) but
it may make it difficult to write unit tests for AddressableBits later
on.
This option doesn't exist. It is currently displayed by `help target
var` due to a bug introduced by 41ae8e7445 in 2018.
Some code for `target var` and `frame var` is shared, and some hard-code
constants are used in order to filter out options that belong only to
`frame var`. However, the aforementioned commit failed to update these
constants properly. This patch addresses the issue by having a _single_
place where the filtering of options needs to be done.
This caused following warnings in an LLDB build:
```
[237/1072] Building CXX object tools/l...lusLanguage.dir/LibCxxSliceArray.cpp.o
/Volumes/Data/llvm-project/lldb/source/Plugins/Language/CPlusPlus/LibCxxSliceArray.cpp:38:53: warning: format specifies type 'unsigned long long' but the argument has type 'size_t' (aka 'unsigned long') [-Wformat]
38 | stream.Printf("stride=%" PRIu64 " size=%" PRIu64, stride, size);
| ~~~~~~~~~ ^~~~~~
/Volumes/Data/llvm-project/lldb/source/Plugins/Language/CPlusPlus/LibCxxSliceArray.cpp:38:61: warning: format specifies type 'unsigned long long' but the argument has type 'size_t' (aka 'unsigned long') [-Wformat]
38 | stream.Printf("stride=%" PRIu64 " size=%" PRIu64, stride, size);
| ~~~~~~~~~ ^~~~
2 warnings generated.
```
This patch simply changes the format specifiers to use the `%zu` for
`size_t`s.
The code that prints ValueObjects is duplicated across two different
cases of the dwim-print command, and a subsequent commit will add a
third case. As such, this commit factors out the common code into a
lambda. A free function was considered, but there is too much
function-local context required in that.
We also reword some of the comments so that they stop counting cases,
making it easier to add other cases later.
`EvaluateExpression` does not always create a new persistent result. If the expression
is a bare persistent variable, then a new persistent result is not created. This means
the caller can't assume a new persistent result is created for each evaluation.
However, `dwim-print` was doing exactly that: assuming a new persistent result for each
evaluation. This resulted in a bug:
```
(lldb) p int $j = 23
(lldb) p $j
(lldb) p $j
```
The first `p $j` would not create a persistent result, and so `dwim-print` would
inadvertently delete `$j`. The second `p $j` would fail.
The fix is to try `expr` as a persistent variable, after trying `expr` as a frame
variable. For persistent variables, this avoids calling `EvaluateExpression`.
Resolves https://github.com/llvm/llvm-project/issues/84806
rdar://124688427
llvm-project/lldb/source/Host/common/Alarm.cpp:37:5:
error: 'lock_guard' may not intend to support class template argument deduction [-Werror,-Wctad-maybe-unsupported]
std::lock_guard alarm_guard(m_alarm_mutex);
^
Currently, we always show the argument passed to dsymForUUID in the
corresponding progress update. Most of the time this is a UUID, but it
can also be an absolute path. The former is pretty uninformative and the
latter needlessly noisy.
This changes the progress update to print the UUID and the module name,
if both are available. Otherwise, we print the UUID or the module name
depending on which one is available.
We now also unconditionally pass the module file spec and architecture
to DownloadObjectAndSymbolFile, while previously this was conditional on
the file existing on-disk. This should be harmless:
- We already check that the file exists in DownloadObjectAndSymbolFile.
- It doesn't make sense to check the filesystem for the architecutre.
rdar://124643548
The commit introduces a new, generic, Alarm class. The class lets you to
schedule functions (callbacks) that will execute after a predefined
timeout. Once scheduled, you can cancel and reset a callback, given the
timeout hasn't expired yet.
The alarm class worker thread that sleeps until the next timeout
expires. When the thread wakes up, it checks for all the callbacks that
have expired and calls them in order. Because the callback is called
from the worker thread, the only guarantee is that a callback is called
no sooner than the timeout. A long running callback could potentially
block the worker threads and delay other callbacks from getting called.
I intentionally kept the implementation as simple as possible while
addressing the needs for the use case of coalescing progress events as
discussed in [1]. If we want to rely on this somewhere else, we can
reassess whether we need to address this class' limitations.
[1] https://discourse.llvm.org/t/rfc-improve-lldb-progress-reporting/75717/
Some languages may create artificial functions that have no real user
code, even though there is line table information for them. One such
case is with coroutine code that receives the CoroSplitter
transformation in LLVM IR. That code transformation creates many
different Functions, cloning one Instruction into many Instructions in
many different Functions and copying the associated debug locations.
It would be difficult to make that pass delete debug locations of cloned
instructions in a language agnostic way (is it even possible?), but LLDB
can ignore certain locations by querying its Language APIs and having it
decide based on, for example, mangling information.
Darwin AArch64 application processors are run with Top Byte Ignore mode
enabled so metadata may be stored in the top byte, it needs to be
ignored when reading/writing memory. David Spickett handled this already
in the base class Process::ReadMemory but ProcessMachCore overrides that
method (to avoid the memory cache) and did not pick up the same change.
I add a test case that creates a pointer with metadata in the top byte
and dereferences it with a live process and with a corefile.
rdar://123784501
by handling *all* errors in IRExecDiagnosticHandler. The function that
call this handles all unhandled errors with an `exit(1)`.
rdar://124459751
I don't really have a testcase for this, since the crash report I got
for this involved the Swift language plugin.
MSVC fails when there is ambiguity (multiple options) around implicit
type conversion operators.
Make ConstString's conversion operator to string_view explicit to avoid
ambiguity with one to StringRef and remove an unused local variable that
MSVC also fails on.
