I backed this out due to a problem on one of the bots that myself and
others have problems reproducing locally. I'd like to try to land it
again, at least to gain more information.
Summary:
This improves the performance of ObjectFileMacho::ParseSymtab by
removing eager and expensive work in favor of doing it later in a
less-expensive fashion.
Experiment:
My goal was to understand LLDB's startup time.
First, I produced a Debug build of LLDB (no dSYM) and a
Release+NoAsserts build of LLDB. The Release build debugged the Debug
build as it debugged a small C++ program. I found that
ObjectFileMachO::ParseSymtab accounted for somewhere between 1.2 and 1.3
seconds consistently. After applying this change, I consistently
measured a reduction of approximately 100ms, putting the time closer to
1.1s and 1.2s on average.
Background:
ObjectFileMachO::ParseSymtab will incrementally create symbols by
parsing nlist entries from the symtab section of a MachO binary. As it
does this, it eagerly tries to determine the size of symbols (e.g. how
long a function is) using LC_FUNCTION_STARTS data (or eh_frame if
LC_FUNCTION_STARTS is unavailable). Concretely, this is done by
performing a binary search on the function starts array and calculating
the distance to the next function or the end of the section (whichever
is smaller).
However, this work is unnecessary for 2 reasons:
1. If you have debug symbol entries (i.e. STABs), the size of a function
is usually stored right after the function's entry. Performing this work
right before parsing the next entry is unnecessary work.
2. Calculating symbol sizes for symbols of size 0 is already performed
in `Symtab::InitAddressIndexes` after all the symbols are added to the
Symtab. It also does this more efficiently by walking over a list of
symbols sorted by address, so the work to calculate the size per symbol
is constant instead of O(log n).
Resubmissions of https://github.com/llvm/llvm-project/pull/112596 with
buildbot fixes.
Allow LLDB to parse the dynamic symbol table from an ELF file or memory
image in an ELF file that has no section headers. This patch uses the
ability to parse the PT_DYNAMIC segment and find the DT_SYMTAB,
DT_SYMENT, DT_HASH or DT_GNU_HASH to find and parse the dynamic symbol
table if the section headers are not present. It also adds a helper
function to read data from a .dynamic key/value pair entry correctly
from the file or from memory.
Allow LLDB to parse the dynamic symbol table from an ELF file or memory
image in an ELF file that has no section headers. This patch uses the
ability to parse the PT_DYNAMIC segment and find the DT_SYMTAB,
DT_SYMENT, DT_HASH or DT_GNU_HASH to find and parse the dynamic symbol
table if the section headers are not present. It also adds a helper
function to read data from a .dynamic key/value pair entry correctly
from the file or from memory.
This patch adds desired feature flags in JIT compiler to enable
hard-float instructions if target supports them and allows to use floats
and doubles in lldb expressions.
Fited tests:
lldb-shell :: Expr/TestAnonNamespaceParamFunc.cpp
lldb-shell :: Expr/TestIRMemoryMap.test
lldb-shell :: Expr/TestStringLiteralExpr.test
lldb-shell :: SymbolFile/DWARF/debug-types-expressions.test
Similar as #99336
Depens on: https://github.com/llvm/llvm-project/pull/114741
Reviewed By: SixWeining
Pull Request: https://github.com/llvm/llvm-project/pull/114742
Recently my coworker @jeffreytan81 pointed out that Minidumps don't show
breakpoints when collected. This was prior blocked because Minidumps
could only contain 1 exception, now that we support N signals/sections
we can save all the threads stopped on breakpoints.
In my prior two save core API's, I experimented on how to save stacks
with the new API. I incorrectly left these in, as the existing
`m_thread_by_range_end` was the correct choice.
I have removed the no-op collection, and moved to use the proper one.
It's worth noting this was not caught by testing because we do not
verify where the items are contained in the minidump. This would require
a test being aware of how minidumps are structured, or adding a textual
tool that we can then scan the output of.
Summary:
This improves the performance of ObjectFileMacho::ParseSymtab by
removing eager and expensive work in favor of doing it later in a
less-expensive fashion.
Experiment:
My goal was to understand LLDB's startup time.
First, I produced a Debug build of LLDB (no dSYM) and a
Release+NoAsserts build of LLDB. The Release build debugged the Debug
build as it debugged a small C++ program. I found that
ObjectFileMachO::ParseSymtab accounted for somewhere between 1.2 and 1.3
seconds consistently. After applying this change, I consistently
measured a reduction of approximately 100ms, putting the time closer to
1.1s and 1.2s on average.
Background:
ObjectFileMachO::ParseSymtab will incrementally create symbols by
parsing nlist entries from the symtab section of a MachO binary. As it
does this, it eagerly tries to determine the size of symbols (e.g. how
long a function is) using LC_FUNCTION_STARTS data (or eh_frame if
LC_FUNCTION_STARTS is unavailable). Concretely, this is done by
performing a binary search on the function starts array and calculating
the distance to the next function or the end of the section (whichever
is smaller).
However, this work is unnecessary for 2 reasons:
1. If you have debug symbol entries (i.e. STABs), the size of a function
is usually stored right after the function's entry. Performing this work
right before parsing the next entry is unnecessary work.
2. Calculating symbol sizes for symbols of size 0 is already performed
in `Symtab::InitAddressIndexes` after all the symbols are added to the
Symtab. It also does this more efficiently by walking over a list of
symbols sorted by address, so the work to calculate the size per symbol
is constant instead of O(log n).
In #95312 Minidump file creation was moved from being created at the
end, to the file being emitted in chunks. This causes some undesirable
behavior where the file can still be present after an error has
occurred. To resolve this we will now delete the file upon an error.
Recently in #107731 this change was revereted due to excess memory size
in `TestSkinnyCore`. This was due to a bug where a range's end was being
passed as size. Creating massive memory ranges.
Additionally, and requiring additional review, I added more unit tests
and more verbose logic to the merging of save core memory regions.
@jasonmolenda as an FYI.
Reapplies #106293, testing identified issue in the merging code. I used
this opportunity to strip CoreFileMemoryRanges to it's own file and then
add unit tests on it's behavior.
A follow up to #106473 Minidump wasn't collecting fs or gs_base. This
patch extends the x86_64 register context and gated reading it behind an
lldb specific flag. Additionally these registers are explicitly checked
in the tests.
This patch addresses a bug where `cs`/`fs` and other segmentation flags
were being identified as having a type of `32b` and `64b` for `rflags`.
In that case the register value was returning the fail value `0xF...`
and this was corrupting some minidumps. Here we just read it as a 64b
value and truncate it.
In addition to that fix, I added comparing the registers from the live
process to the loaded core for the generic minidump test. Prior only
being ARM register tests. This explains why this was not detected
before.
This patch removes all of the Set.* methods from Status.
This cleanup is part of a series of patches that make it harder use the
anti-pattern of keeping a long-lives Status object around and updating
it while dropping any errors it contains on the floor.
This patch is largely NFC, the more interesting next steps this enables
is to:
1. remove Status.Clear()
2. assert that Status::operator=() never overwrites an error
3. remove Status::operator=()
Note that step (2) will bring 90% of the benefits for users, and step
(3) will dramatically clean up the error handling code in various
places. In the end my goal is to convert all APIs that are of the form
` ResultTy DoFoo(Status& error)
`
to
` llvm::Expected<ResultTy> DoFoo()
`
How to read this patch?
The interesting changes are in Status.h and Status.cpp, all other
changes are mostly
` perl -pi -e 's/\.SetErrorString/ = Status::FromErrorString/g' $(git
grep -l SetErrorString lldb/source)
`
plus the occasional manual cleanup.
This patch adds the option to specify specific memory ranges to be
included in a given core file. The current implementation lets user
specified ranges either be in addition to a certain save style, or
independent of them via the newly added custom enum.
To achieve being inclusive of save style, I've moved from a std::vector
of ranges to a RangeDataVector, and to join overlapping ranges to
prevent duplication of memory ranges in the core file.
As a non function bonus, when SBSavecore was initially created, the
header was included in the lldb-private interfaces, and I've fixed that
and moved it the forward declare as an oversight. CC @bulbazord in case
we need to include that into swift.
This PR is in response to a bug my coworker @mbucko discovered where on
MacOS Minidumps were being created where the 64b memory regions were
readable, but were not being listed in
`SBProcess.GetMemoryRegionList()`. This went unnoticed in #95312 due to
all the linux testing including /proc/pid maps. On MacOS generated dumps
(or any dump without access to /proc/pid) we would fail to properly map
Memory Regions due to there being two independent methods for 32b and
64b mapping.
In this PR I addressed this minor bug and merged the methods, but in
order to add test coverage required additions to `obj2yaml` and
`yaml2obj` which make up the bulk of this patch.
Lastly, there are some non-required changes such as the addition of the
`Memory64ListHeader` type, to make writing/reading the header section of
the Memory64List easier.
This PR is in reference to porting LLDB on AIX.
Link to discussions on llvm discourse and github:
1. https://discourse.llvm.org/t/port-lldb-to-ibm-aix/80640
2. #101657
The complete changes for porting are present in this draft PR:
https://github.com/llvm/llvm-project/pull/102601
The changes on this PR are intended to avoid namespace collision for
certain typedefs between lldb and other platforms:
1. tid_t --> lldb::tid_t
2. offset_t --> lldb::offset_t
Reapply #100443 and #101770. These were originally reverted due to a
test failure and an MSAN failure. I changed the test attribute to
restrict to x86 (following the other existing tests). I could not
reproduce the test or the MSAN failure and no repo steps were provided.
This patch improves the ability of a ObjectFileELF instance to read the
.dynamic section. It adds the ability to read the .dynamic section from
the PT_DYNAMIC program header which is useful for ELF files that have no
section headers and for ELF files that are read from memory. It cleans
up the usage of the .dynamic entries so that
ObjectFileELF::ParseDynamicSymbols() is the only code that parses
.dynamic entries, teaches that function the read and store the string
values for each .dynamic entry. We now dump the .dynamic entries in the
output of "image dump objfile". It also cleans up the code that gets the
dynamic string table so that it can grab it from the DT_STRTAB and
DT_STRSZ .dynamic entries for when we have a ELF file with no section
headers or we are reading it from memory.
This fixes the following:
```
[6603/7618] Building CXX object
tools\lldb\source\Plugins\ObjectFile\PECOFF\CMakeFiles\lldbPluginObjectFilePECOFF.dir\WindowsMiniDump.cpp.obj
C:\src\git\llvm-project\lldb\source\Plugins\ObjectFile\PECOFF\WindowsMiniDump.cpp(29,25):
warning: object backing the pointer will be destroyed at the end of the
full-expression [-Wdangling-gsl]
29 | const auto &outfile = core_options.GetOutputFile().value();
| ^~~~~~~~~~~~~~~~~~~~~~~~~~~~
1 warning generated.
```
This patch improves the ability of a ObjectFileELF instance to read the .dynamic section. It adds the ability to read the .dynamic section from the PT_DYNAMIC program header which is useful for ELF files that have no section headers and for ELF files that are read from memory. It cleans up the usage of the .dynamic entries so that ObjectFileELF::ParseDynamicSymbols() is the only code that parses .dynamic entries, teaches that function the read and store the string values for each .dynamic entry. We now dump the .dynamic entries in the output of "image dump objfile". It also cleans up the code that gets the dynamic string table so that it can grab it from the DT_STRTAB and DT_STRSZ .dynamic entries for when we have a ELF file with no section headers or we are reading it from memory.
In #100443, Mach-o and Minidump now only call process API's that take a
`SaveCoreOption` as the container for the style and information if a
thread should be included in the core or not. This introduced a bug
where in subsequent method calls we were not honoring the defaults of
both implementations.
~~To solve this I have made a copy of each SaveCoreOptions that is
mutable by the respective plugin. Originally I wanted to leave the
SaveCoreOptions as non const so these default value mutations could be
shown back to the user. Changing that behavior is outside of the scope
of this bugfix, but is context for why we are making a copy.~~
Removed const on the savecoreoptions so defaults can be inspected by the
user
CC: @Michael137
In #98403 I enabled the SBSaveCoreOptions object, which allows users via
the scripting API to define what they want saved into their core file.
As the first option I've added a threadlist, so users can scan and
identify which threads and corresponding stacks they want to save.
In order to support this, I had to add a new method to `Process.h` on
how we identify which threads are to be saved, and I had to change the
book keeping in minidump to ensure we don't double save the stacks.
Important to @jasonmolenda I also changed the MachO coredump to accept
these new APIs.
Prevent passing a nullptr to std::string::insert in
ObjectFileMachO::GetDependentModules. Calling GetCString on an empty
ConstString will return a nullptr, which is undefined behavior. Instead,
use the GetString helper which will return an empty string in that case.
rdar://132388027
.sbss section was recognized as eSectionTypeOther, but has to be
eSectionTypeZeroFill.
Fixed tests for RISCV target:
TestClassLoadingViaMemberTypedef.TestCase
TestClassTemplateNonTypeParameterPack.TestCaseClassTemplateNonTypeParameterPack
TestThreadSelectionBug.TestThreadSelectionBug
lldbsuite.test.lldbtest.TestOffsetof
lldbsuite.test.lldbtest.TestOffsetofCpp
TestTargetDumpTypeSystem.TestCase
TestCPP11EnumTypes.CPP11EnumTypesTestCase
TestCppIsTypeComplete.TestCase
TestExprCrash.ExprCrashTestCase
TestDebugIndexCache.DebugIndexCacheTestcase
This PR adds `SBSaveCoreOptions`, which is a container class for options
when LLDB is taking coredumps. For this first iteration this container
just keeps parity with the extant API of `file, style, plugin`. In the
future this options object can be extended to allow users to take a
subset of their core dumps.
PT_LOAD and PT_TLS segments are top level sections in the ObjectFileELF
section list. The two segments can often have the same program header
p_vaddr and p_paddr values and this can cause section load list issues
in LLDB if we load the PT_TLS segments. What happens is the
SectionLoadList::m_addr_to_sect, when a library is loaded, will first
map one of the sections named "PT_LOAD[0]" with the load address that
matches the p_vaddr entry from the program header. Then the "PT_TLS[0]"
would come along and try to load this section at the same address. This
would cause the "PT_LOAD[0]" section to be unloaded as the
SectionLoadList::m_addr_to_sect would replace the value for the matching
p_vaddr with the last section to be seen. The sizes of the PT_TLS and
PT_LOAD that have the same p_vaddr value don't need to have the same
byte size, so this could cause lookups to fail for an addresses in the
"PT_LOAD[0]" section or any of its children if the offset is greater
than the offset size of the PT_TLS segment. It could also cause us to
incorrectly attribute addresses from the "PT_LOAD[0]" to the "PT_TLS[0]"
segment when doing lookups for offset that are less than the size of the
PT_TLS segment.
This fix stops us from loading PT_TLS segments in the section load lists
and will prevent the bugs that resulted from this. No addresses the the
DWARF refer to TLS data with a "file address" in any way. They all have
TLS DWARF location expressions to locate these variables. We also don't
have any support for having actual thread specific sections and having
those sections resolve to something different for each thread, so there
currently is no point in loading thread specific sections. Both the
ObjectFileMachO and ObjectFileCOFF both ignore thread specific sections
at the moment, so this brings the ObjectFileELF to parity with those
plug-ins.
I added a test into an existing test to verify that things work as
expected.
Prior to this fix with a real binary, the output of "target dump
section-load-list" would look like this for the old LLDB:
```
// (lldb) target dump section-load-list
// addr = 0x0000000000000000, section = 0x55d46ab8c510: 0xfffffffffffffffd container [0x0000000000000000-0x0000000000000628) r-- 0x00000000 0x00000628 0x00000000 a.out.PT_LOAD[0]
// addr = 0x0000000000001000, section = 0x55d46ab8b0c0: 0xfffffffffffffffc container [0x0000000000001000-0x0000000000001185) r-x 0x00001000 0x00000185 0x00000000 a.out.PT_LOAD[1]
// addr = 0x0000000000002000, section = 0x55d46ac040f0: 0xfffffffffffffffb container [0x0000000000002000-0x00000000000020cc) r-- 0x00002000 0x000000cc 0x00000000 a.out.PT_LOAD[2]
// addr = 0x0000000000003db0, section = 0x55d46ab7cef0: 0xfffffffffffffff6 container [0x0000000000003db0-0x0000000000003db4) r-- 0x00002db0 0x00000000 0x00000000 a.out.PT_TLS[0]
```
And this for the fixed LLDB:
```
// (lldb) target dump section-load-list
// addr = 0x0000000000000000, section = 0x105f0a9a8: 0xfffffffffffffffd container [0x0000000000000000-0x0000000000000628) r-- 0x00000000 0x00000628 0x00000000 a.out.PT_LOAD[0]
// addr = 0x0000000000001000, section = 0x105f0adb8: 0xfffffffffffffffc container [0x0000000000001000-0x0000000000001185) r-x 0x00001000 0x00000185 0x00000000 a.out.PT_LOAD[1]
// addr = 0x0000000000002000, section = 0x105f0af48: 0xfffffffffffffffb container [0x0000000000002000-0x00000000000020cc) r-- 0x00002000 0x000000cc 0x00000000 a.out.PT_LOAD[2]
// addr = 0x0000000000003db0, section = 0x105f0b078: 0xfffffffffffffffa container [0x0000000000003db0-0x0000000000004028) rw- 0x00002db0 0x00000274 0x00000000 a.out.PT_LOAD[3]
```
We can see that previously the "PT_LOAD[3]" segment would be removed
from the section load list, and after the fix it remains and there is on
PT_TLS in the loaded sections.
ObjectFileMachO::SetLoadAddress sets the address of each segment in a
binary in a Target, but it ignores segments that are not loaded in the
virtual address space. It was marking segments that were purely BSS --
having no content in the file, but in zero-initialized memory when
running in the virtual address space -- as not-loadable, unless they
were named "DATA". This works pretty well for typical userland binaries,
but in less Darwin environments, there may be BSS segments with other
names, that ARE loadable.
I looked at the origin of SectionIsLoadable's check for this, and it was
a cleanup by Greg in 2018 where we had three different implementations
of the idea in ObjectFileMachO and one of them skipped zero-file-size
segments (BSS), which made it into the centralized SectionIsLoadable
method.
Also add some logging to the DynamicLoader log channel when loading a
binary - it's the first place I look when debugging segment address
setting bugs, and it wasn't emitting anything.
rdar://129870649
In #95312 I incorrectly set `m_expected_directories` to uint, this broke
the windows build and is the incorrect type.
`size_t` is more accurate because this value only ever represents the
expected upper bound of the directory vector.
Currently, LLDB does not support taking a minidump over the 4.2gb limit imposed by uint32. In fact, currently it writes the RVA's and the headers to the end of the file, which can become corrupted due to the header offset only supporting a 32b offset.
This change reorganizes how the file structure is laid out. LLDB will precalculate the number of directories required and preallocate space at the top of the file to fill in later. Additionally, thread stacks require a 32b offset, and we provision empty descriptors and keep track of them to clean up once we write the 32b memory list.
For
[MemoryList64](https://learn.microsoft.com/en-us/windows/win32/api/minidumpapiset/ns-minidumpapiset-minidump_memory64_list),
the RVA to the start of the section itself will remain in a 32b addressable space. We achieve this by predetermining the space the memory regions will take, and only writing up to 4.2 gb of data with some buffer to allow all the MemoryDescriptor64s to also still be 32b addressable.
I did not add any explicit tests to this PR because allocating 4.2gb+ to test is very expensive. However, we have 32b automation tests and I validated with in several ways, including with 5gb+ array/object and would be willing to add this as a test case.
Add comments and a test for delay-init libraries on macOS. I originally
added the support in 954d00e87c a month
ago, but without these additional clarifications.
rdar://126885033
Currently in Core dumps, the entire pthread is copied, including the
unused space beyond the stack pointer. This causes large amounts of core
dump inflation when the number of threads is high, but the stack usage
is low. Such as when an application is using a thread pool.
This change will optimize for these situations in addition to generally
improving the core dump performance for all of lldb.
It was pointed out that ordering is crucial here, so note that.
I also looked into using a vector instead, as described in
https://llvm.org/docs/ProgrammersManual.html#dss-sortedvectorset.
Which this is in theory perfect for, but we have at least 2 places
that update the map and both would need to sort/unique each time.
Plus this code is pretty bug prone.
If there is future refactoring it's one thing to consider.
Instead of updating the member of the ObjectFileELF instance. This means
that if one object file asks another to parse the symbol table, that
first object's can update its address class map with the same changes
that the other object did.
(I'm not returning a reference to the other object's m_address_class_map
member because there may be other things in there not related to the
symbol table being parsed)
This will fix the code added in
https://github.com/llvm/llvm-project/pull/90622 which broke the test
`Expr/TestStringLiteralExpr.test` on 32 bit Arm Linux.
This happened because we had the program file, then asked for a better
object file, which returned the same program file again. This creates a
second ObjectFileELF for the same file, so when we tell the second
instance to parse the symbol table it actually calls into the first
instance, leaving the address class map of the second instance empty.
Which caused us to put an Arm breakpoint instuction at a Thumb return
address and broke the ability to call mmap.
Detects AArch64 trampolines in order to be able to step in a function
through a trampoline on AArch64.
---------
Co-authored-by: Vincent Belliard <v-bulle@github.com>
