Recently my coworker @jeffreytan81 pointed out that Minidumps don't show
breakpoints when collected. This was prior blocked because Minidumps
could only contain 1 exception, now that we support N signals/sections
we can save all the threads stopped on breakpoints.
In my prior two save core API's, I experimented on how to save stacks
with the new API. I incorrectly left these in, as the existing
`m_thread_by_range_end` was the correct choice.
I have removed the no-op collection, and moved to use the proper one.
It's worth noting this was not caught by testing because we do not
verify where the items are contained in the minidump. This would require
a test being aware of how minidumps are structured, or adding a textual
tool that we can then scan the output of.
Summary:
This improves the performance of ObjectFileMacho::ParseSymtab by
removing eager and expensive work in favor of doing it later in a
less-expensive fashion.
Experiment:
My goal was to understand LLDB's startup time.
First, I produced a Debug build of LLDB (no dSYM) and a
Release+NoAsserts build of LLDB. The Release build debugged the Debug
build as it debugged a small C++ program. I found that
ObjectFileMachO::ParseSymtab accounted for somewhere between 1.2 and 1.3
seconds consistently. After applying this change, I consistently
measured a reduction of approximately 100ms, putting the time closer to
1.1s and 1.2s on average.
Background:
ObjectFileMachO::ParseSymtab will incrementally create symbols by
parsing nlist entries from the symtab section of a MachO binary. As it
does this, it eagerly tries to determine the size of symbols (e.g. how
long a function is) using LC_FUNCTION_STARTS data (or eh_frame if
LC_FUNCTION_STARTS is unavailable). Concretely, this is done by
performing a binary search on the function starts array and calculating
the distance to the next function or the end of the section (whichever
is smaller).
However, this work is unnecessary for 2 reasons:
1. If you have debug symbol entries (i.e. STABs), the size of a function
is usually stored right after the function's entry. Performing this work
right before parsing the next entry is unnecessary work.
2. Calculating symbol sizes for symbols of size 0 is already performed
in `Symtab::InitAddressIndexes` after all the symbols are added to the
Symtab. It also does this more efficiently by walking over a list of
symbols sorted by address, so the work to calculate the size per symbol
is constant instead of O(log n).
In #95312 Minidump file creation was moved from being created at the
end, to the file being emitted in chunks. This causes some undesirable
behavior where the file can still be present after an error has
occurred. To resolve this we will now delete the file upon an error.
Recently in #107731 this change was revereted due to excess memory size
in `TestSkinnyCore`. This was due to a bug where a range's end was being
passed as size. Creating massive memory ranges.
Additionally, and requiring additional review, I added more unit tests
and more verbose logic to the merging of save core memory regions.
@jasonmolenda as an FYI.
Reapplies #106293, testing identified issue in the merging code. I used
this opportunity to strip CoreFileMemoryRanges to it's own file and then
add unit tests on it's behavior.
A follow up to #106473 Minidump wasn't collecting fs or gs_base. This
patch extends the x86_64 register context and gated reading it behind an
lldb specific flag. Additionally these registers are explicitly checked
in the tests.
This patch addresses a bug where `cs`/`fs` and other segmentation flags
were being identified as having a type of `32b` and `64b` for `rflags`.
In that case the register value was returning the fail value `0xF...`
and this was corrupting some minidumps. Here we just read it as a 64b
value and truncate it.
In addition to that fix, I added comparing the registers from the live
process to the loaded core for the generic minidump test. Prior only
being ARM register tests. This explains why this was not detected
before.
This patch removes all of the Set.* methods from Status.
This cleanup is part of a series of patches that make it harder use the
anti-pattern of keeping a long-lives Status object around and updating
it while dropping any errors it contains on the floor.
This patch is largely NFC, the more interesting next steps this enables
is to:
1. remove Status.Clear()
2. assert that Status::operator=() never overwrites an error
3. remove Status::operator=()
Note that step (2) will bring 90% of the benefits for users, and step
(3) will dramatically clean up the error handling code in various
places. In the end my goal is to convert all APIs that are of the form
` ResultTy DoFoo(Status& error)
`
to
` llvm::Expected<ResultTy> DoFoo()
`
How to read this patch?
The interesting changes are in Status.h and Status.cpp, all other
changes are mostly
` perl -pi -e 's/\.SetErrorString/ = Status::FromErrorString/g' $(git
grep -l SetErrorString lldb/source)
`
plus the occasional manual cleanup.
This patch adds the option to specify specific memory ranges to be
included in a given core file. The current implementation lets user
specified ranges either be in addition to a certain save style, or
independent of them via the newly added custom enum.
To achieve being inclusive of save style, I've moved from a std::vector
of ranges to a RangeDataVector, and to join overlapping ranges to
prevent duplication of memory ranges in the core file.
As a non function bonus, when SBSavecore was initially created, the
header was included in the lldb-private interfaces, and I've fixed that
and moved it the forward declare as an oversight. CC @bulbazord in case
we need to include that into swift.
This PR is in response to a bug my coworker @mbucko discovered where on
MacOS Minidumps were being created where the 64b memory regions were
readable, but were not being listed in
`SBProcess.GetMemoryRegionList()`. This went unnoticed in #95312 due to
all the linux testing including /proc/pid maps. On MacOS generated dumps
(or any dump without access to /proc/pid) we would fail to properly map
Memory Regions due to there being two independent methods for 32b and
64b mapping.
In this PR I addressed this minor bug and merged the methods, but in
order to add test coverage required additions to `obj2yaml` and
`yaml2obj` which make up the bulk of this patch.
Lastly, there are some non-required changes such as the addition of the
`Memory64ListHeader` type, to make writing/reading the header section of
the Memory64List easier.
This PR is in reference to porting LLDB on AIX.
Link to discussions on llvm discourse and github:
1. https://discourse.llvm.org/t/port-lldb-to-ibm-aix/80640
2. #101657
The complete changes for porting are present in this draft PR:
https://github.com/llvm/llvm-project/pull/102601
The changes on this PR are intended to avoid namespace collision for
certain typedefs between lldb and other platforms:
1. tid_t --> lldb::tid_t
2. offset_t --> lldb::offset_t
Reapply #100443 and #101770. These were originally reverted due to a
test failure and an MSAN failure. I changed the test attribute to
restrict to x86 (following the other existing tests). I could not
reproduce the test or the MSAN failure and no repo steps were provided.
This patch improves the ability of a ObjectFileELF instance to read the
.dynamic section. It adds the ability to read the .dynamic section from
the PT_DYNAMIC program header which is useful for ELF files that have no
section headers and for ELF files that are read from memory. It cleans
up the usage of the .dynamic entries so that
ObjectFileELF::ParseDynamicSymbols() is the only code that parses
.dynamic entries, teaches that function the read and store the string
values for each .dynamic entry. We now dump the .dynamic entries in the
output of "image dump objfile". It also cleans up the code that gets the
dynamic string table so that it can grab it from the DT_STRTAB and
DT_STRSZ .dynamic entries for when we have a ELF file with no section
headers or we are reading it from memory.
This fixes the following:
```
[6603/7618] Building CXX object
tools\lldb\source\Plugins\ObjectFile\PECOFF\CMakeFiles\lldbPluginObjectFilePECOFF.dir\WindowsMiniDump.cpp.obj
C:\src\git\llvm-project\lldb\source\Plugins\ObjectFile\PECOFF\WindowsMiniDump.cpp(29,25):
warning: object backing the pointer will be destroyed at the end of the
full-expression [-Wdangling-gsl]
29 | const auto &outfile = core_options.GetOutputFile().value();
| ^~~~~~~~~~~~~~~~~~~~~~~~~~~~
1 warning generated.
```
This patch improves the ability of a ObjectFileELF instance to read the .dynamic section. It adds the ability to read the .dynamic section from the PT_DYNAMIC program header which is useful for ELF files that have no section headers and for ELF files that are read from memory. It cleans up the usage of the .dynamic entries so that ObjectFileELF::ParseDynamicSymbols() is the only code that parses .dynamic entries, teaches that function the read and store the string values for each .dynamic entry. We now dump the .dynamic entries in the output of "image dump objfile". It also cleans up the code that gets the dynamic string table so that it can grab it from the DT_STRTAB and DT_STRSZ .dynamic entries for when we have a ELF file with no section headers or we are reading it from memory.
In #100443, Mach-o and Minidump now only call process API's that take a
`SaveCoreOption` as the container for the style and information if a
thread should be included in the core or not. This introduced a bug
where in subsequent method calls we were not honoring the defaults of
both implementations.
~~To solve this I have made a copy of each SaveCoreOptions that is
mutable by the respective plugin. Originally I wanted to leave the
SaveCoreOptions as non const so these default value mutations could be
shown back to the user. Changing that behavior is outside of the scope
of this bugfix, but is context for why we are making a copy.~~
Removed const on the savecoreoptions so defaults can be inspected by the
user
CC: @Michael137
In #98403 I enabled the SBSaveCoreOptions object, which allows users via
the scripting API to define what they want saved into their core file.
As the first option I've added a threadlist, so users can scan and
identify which threads and corresponding stacks they want to save.
In order to support this, I had to add a new method to `Process.h` on
how we identify which threads are to be saved, and I had to change the
book keeping in minidump to ensure we don't double save the stacks.
Important to @jasonmolenda I also changed the MachO coredump to accept
these new APIs.
Prevent passing a nullptr to std::string::insert in
ObjectFileMachO::GetDependentModules. Calling GetCString on an empty
ConstString will return a nullptr, which is undefined behavior. Instead,
use the GetString helper which will return an empty string in that case.
rdar://132388027
.sbss section was recognized as eSectionTypeOther, but has to be
eSectionTypeZeroFill.
Fixed tests for RISCV target:
TestClassLoadingViaMemberTypedef.TestCase
TestClassTemplateNonTypeParameterPack.TestCaseClassTemplateNonTypeParameterPack
TestThreadSelectionBug.TestThreadSelectionBug
lldbsuite.test.lldbtest.TestOffsetof
lldbsuite.test.lldbtest.TestOffsetofCpp
TestTargetDumpTypeSystem.TestCase
TestCPP11EnumTypes.CPP11EnumTypesTestCase
TestCppIsTypeComplete.TestCase
TestExprCrash.ExprCrashTestCase
TestDebugIndexCache.DebugIndexCacheTestcase
This PR adds `SBSaveCoreOptions`, which is a container class for options
when LLDB is taking coredumps. For this first iteration this container
just keeps parity with the extant API of `file, style, plugin`. In the
future this options object can be extended to allow users to take a
subset of their core dumps.
PT_LOAD and PT_TLS segments are top level sections in the ObjectFileELF
section list. The two segments can often have the same program header
p_vaddr and p_paddr values and this can cause section load list issues
in LLDB if we load the PT_TLS segments. What happens is the
SectionLoadList::m_addr_to_sect, when a library is loaded, will first
map one of the sections named "PT_LOAD[0]" with the load address that
matches the p_vaddr entry from the program header. Then the "PT_TLS[0]"
would come along and try to load this section at the same address. This
would cause the "PT_LOAD[0]" section to be unloaded as the
SectionLoadList::m_addr_to_sect would replace the value for the matching
p_vaddr with the last section to be seen. The sizes of the PT_TLS and
PT_LOAD that have the same p_vaddr value don't need to have the same
byte size, so this could cause lookups to fail for an addresses in the
"PT_LOAD[0]" section or any of its children if the offset is greater
than the offset size of the PT_TLS segment. It could also cause us to
incorrectly attribute addresses from the "PT_LOAD[0]" to the "PT_TLS[0]"
segment when doing lookups for offset that are less than the size of the
PT_TLS segment.
This fix stops us from loading PT_TLS segments in the section load lists
and will prevent the bugs that resulted from this. No addresses the the
DWARF refer to TLS data with a "file address" in any way. They all have
TLS DWARF location expressions to locate these variables. We also don't
have any support for having actual thread specific sections and having
those sections resolve to something different for each thread, so there
currently is no point in loading thread specific sections. Both the
ObjectFileMachO and ObjectFileCOFF both ignore thread specific sections
at the moment, so this brings the ObjectFileELF to parity with those
plug-ins.
I added a test into an existing test to verify that things work as
expected.
Prior to this fix with a real binary, the output of "target dump
section-load-list" would look like this for the old LLDB:
```
// (lldb) target dump section-load-list
// addr = 0x0000000000000000, section = 0x55d46ab8c510: 0xfffffffffffffffd container [0x0000000000000000-0x0000000000000628) r-- 0x00000000 0x00000628 0x00000000 a.out.PT_LOAD[0]
// addr = 0x0000000000001000, section = 0x55d46ab8b0c0: 0xfffffffffffffffc container [0x0000000000001000-0x0000000000001185) r-x 0x00001000 0x00000185 0x00000000 a.out.PT_LOAD[1]
// addr = 0x0000000000002000, section = 0x55d46ac040f0: 0xfffffffffffffffb container [0x0000000000002000-0x00000000000020cc) r-- 0x00002000 0x000000cc 0x00000000 a.out.PT_LOAD[2]
// addr = 0x0000000000003db0, section = 0x55d46ab7cef0: 0xfffffffffffffff6 container [0x0000000000003db0-0x0000000000003db4) r-- 0x00002db0 0x00000000 0x00000000 a.out.PT_TLS[0]
```
And this for the fixed LLDB:
```
// (lldb) target dump section-load-list
// addr = 0x0000000000000000, section = 0x105f0a9a8: 0xfffffffffffffffd container [0x0000000000000000-0x0000000000000628) r-- 0x00000000 0x00000628 0x00000000 a.out.PT_LOAD[0]
// addr = 0x0000000000001000, section = 0x105f0adb8: 0xfffffffffffffffc container [0x0000000000001000-0x0000000000001185) r-x 0x00001000 0x00000185 0x00000000 a.out.PT_LOAD[1]
// addr = 0x0000000000002000, section = 0x105f0af48: 0xfffffffffffffffb container [0x0000000000002000-0x00000000000020cc) r-- 0x00002000 0x000000cc 0x00000000 a.out.PT_LOAD[2]
// addr = 0x0000000000003db0, section = 0x105f0b078: 0xfffffffffffffffa container [0x0000000000003db0-0x0000000000004028) rw- 0x00002db0 0x00000274 0x00000000 a.out.PT_LOAD[3]
```
We can see that previously the "PT_LOAD[3]" segment would be removed
from the section load list, and after the fix it remains and there is on
PT_TLS in the loaded sections.
ObjectFileMachO::SetLoadAddress sets the address of each segment in a
binary in a Target, but it ignores segments that are not loaded in the
virtual address space. It was marking segments that were purely BSS --
having no content in the file, but in zero-initialized memory when
running in the virtual address space -- as not-loadable, unless they
were named "DATA". This works pretty well for typical userland binaries,
but in less Darwin environments, there may be BSS segments with other
names, that ARE loadable.
I looked at the origin of SectionIsLoadable's check for this, and it was
a cleanup by Greg in 2018 where we had three different implementations
of the idea in ObjectFileMachO and one of them skipped zero-file-size
segments (BSS), which made it into the centralized SectionIsLoadable
method.
Also add some logging to the DynamicLoader log channel when loading a
binary - it's the first place I look when debugging segment address
setting bugs, and it wasn't emitting anything.
rdar://129870649
In #95312 I incorrectly set `m_expected_directories` to uint, this broke
the windows build and is the incorrect type.
`size_t` is more accurate because this value only ever represents the
expected upper bound of the directory vector.
Currently, LLDB does not support taking a minidump over the 4.2gb limit imposed by uint32. In fact, currently it writes the RVA's and the headers to the end of the file, which can become corrupted due to the header offset only supporting a 32b offset.
This change reorganizes how the file structure is laid out. LLDB will precalculate the number of directories required and preallocate space at the top of the file to fill in later. Additionally, thread stacks require a 32b offset, and we provision empty descriptors and keep track of them to clean up once we write the 32b memory list.
For
[MemoryList64](https://learn.microsoft.com/en-us/windows/win32/api/minidumpapiset/ns-minidumpapiset-minidump_memory64_list),
the RVA to the start of the section itself will remain in a 32b addressable space. We achieve this by predetermining the space the memory regions will take, and only writing up to 4.2 gb of data with some buffer to allow all the MemoryDescriptor64s to also still be 32b addressable.
I did not add any explicit tests to this PR because allocating 4.2gb+ to test is very expensive. However, we have 32b automation tests and I validated with in several ways, including with 5gb+ array/object and would be willing to add this as a test case.
Add comments and a test for delay-init libraries on macOS. I originally
added the support in 954d00e87c a month
ago, but without these additional clarifications.
rdar://126885033
Currently in Core dumps, the entire pthread is copied, including the
unused space beyond the stack pointer. This causes large amounts of core
dump inflation when the number of threads is high, but the stack usage
is low. Such as when an application is using a thread pool.
This change will optimize for these situations in addition to generally
improving the core dump performance for all of lldb.
It was pointed out that ordering is crucial here, so note that.
I also looked into using a vector instead, as described in
https://llvm.org/docs/ProgrammersManual.html#dss-sortedvectorset.
Which this is in theory perfect for, but we have at least 2 places
that update the map and both would need to sort/unique each time.
Plus this code is pretty bug prone.
If there is future refactoring it's one thing to consider.
Instead of updating the member of the ObjectFileELF instance. This means
that if one object file asks another to parse the symbol table, that
first object's can update its address class map with the same changes
that the other object did.
(I'm not returning a reference to the other object's m_address_class_map
member because there may be other things in there not related to the
symbol table being parsed)
This will fix the code added in
https://github.com/llvm/llvm-project/pull/90622 which broke the test
`Expr/TestStringLiteralExpr.test` on 32 bit Arm Linux.
This happened because we had the program file, then asked for a better
object file, which returned the same program file again. This creates a
second ObjectFileELF for the same file, so when we tell the second
instance to parse the symbol table it actually calls into the first
instance, leaving the address class map of the second instance empty.
Which caused us to put an Arm breakpoint instuction at a Thumb return
address and broke the ability to call mmap.
Detects AArch64 trampolines in order to be able to step in a function
through a trampoline on AArch64.
---------
Co-authored-by: Vincent Belliard <v-bulle@github.com>
Section unification cannot just use names, because it's valid for ELF
binaries to have multiple sections with the same name. We should check
other section properties too.
Fixes#88001.
rdar://124467787
Summary:
AddMemoryList() was returning the last error status returned by
ReadMemory(). So if an invalid memory region was read last, the function
would return an error.
Test Plan:
./bin/llvm-lit -sv
~/src/llvm-project/lldb/test/API/functionalities/process_save_core_minidump/TestProcessSaveCoreMinidump.py
Reviewers:
kevinfrei,clayborg
Subscribers:
Tasks:
Tags:
There are users reporting saving minidump from lldb-dap does not work.
Turns out our stack trace request always evaluate a function call which
caused JIT object file like "__lldb_caller_function" to be created which
can fail minidump builder to get its module size.
This patch fixes "getModuleFileSize" for ObjectFileJIT so that module
list can be saved. I decided to create a lldb-dap test so that this
end-to-end functionality can be validated from our tests (instead of
only command line lldb).
The patch also improves several other small things in the workflow:
1. It logs any minidump stream failure so that it is easier to find out
what stream saving fails. In future, we should show process and error to
end users.
2. It handles error from "getModuleFileSize" llvm::Expected<T> otherwise
it will complain the error is not handled.
---------
Co-authored-by: jeffreytan81 <jeffreytan@fb.com>
It is possible to gather code coverage in a firmware environment, where
the __LLVM_COV segment will not be mapped in memory but does exist in
the binary, see
https://llvm.org/devmtg/2020-09/slides/PhippsAlan_EmbeddedCodeCoverage_LLVM_Conf_Talk_final.pdf
The __LLVM_COV segment in the binary happens to be at the same address
as the __DATA segment, so if lldb treats this segment as loaded, it
shadows the __DATA segment and address->symbol resolution can fail.
For these non-userland code cases, we need to mark __LLVM_COV as not a
loadable segment.
rdar://124475661
[lldb] Add SBProcess methods for get/set/use address masks (#83095)
I'm reviving a patch from phabracator, https://reviews.llvm.org/D155905
which was approved but I wasn't thrilled with all the API I was adding
to SBProcess for all of the address mask types / memory regions. In this
update, I added enums to control type address mask type (code, data,
any) and address space specifiers (low, high, all) with defaulted
arguments for the most common case. I originally landed this via
https://github.com/llvm/llvm-project/pull/83095 but it failed on CIs
outside of arm64 Darwin so I had to debug it on more environments
and update the patch.
This patch is also fixing a bug in the "addressable bits to address
mask" calculation I added in AddressableBits::SetProcessMasks. If lldb
were told that 64 bits are valid for addressing, this method would
overflow the calculation and set an invalid mask. Added tests to check
this specific bug while I was adding these APIs.
This patch changes the value of "no mask set" from 0 to
LLDB_INVALID_ADDRESS_MASK, which is UINT64_MAX. A mask of all 1's
means "no bits are used for addressing" which is an impossible mask,
whereas a mask of 0 means "all bits are used for addressing" which
is possible.
I added a base class implementation of ABI::FixCodeAddress and
ABI::FixDataAddress that will apply the Process mask values if they
are set to a value other than LLDB_INVALID_ADDRESS_MASK.
I updated all the callers/users of the Mask methods which were
handling a value of 0 to mean invalid mask to use
LLDB_INVALID_ADDRESS_MASK.
I added code to the all AArch64 ABI Fix* methods to apply the
Highmem masks if they have been set. These will not be set on a
Linux environment, but in TestAddressMasks.py I test the highmem
masks feature for any AArch64 target, so all AArch64 ABI plugins
must handle it.
rdar://123530562
