Commit Graph

1809 Commits

Author SHA1 Message Date
Joseph Huber
6183826212 [libc][NFC] Clean up the memory buffer handling for RPC
We do a lot of arithmetic on void pointers here, so include a helper and
make some more consistent names. Changes no functionality.

Reviewed By: JonChesterfield

Differential Revision: https://reviews.llvm.org/D150576
2023-05-15 11:30:23 -05:00
Joseph Huber
45b899b92f [libc] Cache ownership of the shared buffer in the port
This patch adds another variable to cache cases where we know that we
own the buffer. This allows us to skip the atomic load on the inbox
because we already know its state. This is legal immediately after
opening a port, or when sending immediately after a recieve. This
caching nets a significant (~17%) speedup for the basic open, send,
recieve combination.

Reviewed By: JonChesterfield

Differential Revision: https://reviews.llvm.org/D150516
2023-05-15 06:16:23 -05:00
Joseph Huber
9417d9fc38 [libc] Make the bump pointer explicitly return null on buffer oveerrun
We use a simple bump ptr in the `libc` tests. If we run out of data we
can currently return other static memory and have weird failure cases.
We should fail more explicitly here by returning a null pointer instead.

Reviewed By: sivachandra

Differential Revision: https://reviews.llvm.org/D150529
2023-05-15 06:15:52 -05:00
Guillaume Chatelet
c5dede880d [libc] Add optimized memset for RISCV
This patch adds two versions of `memset` optimized for architectures where unaligned accesses are either illegal or extremely slow.
It is currently enabled for RISCV 64 and RISCV 32 but it could be used for ARM 32 architectures as well.

Here is the before / after output of libc.benchmarks.memory_functions.opt_host --benchmark_filter=BM_Memset on a quad core Linux starfive RISCV 64 board running at 1.5GHz.

Before
```
Run on (4 X 1500 MHz CPU s)
CPU Caches:
  L1 Instruction 32 KiB (x4)
  L1 Data 32 KiB (x4)
  L2 Unified 2048 KiB (x1)
------------------------------------------------------------------------
Benchmark              Time             CPU   Iterations UserCounters...
------------------------------------------------------------------------
BM_Memset/0/0        506 ns          252 ns      2883584 bytes_per_cycle=0.238312/s bytes_per_second=340.908M/s items_per_second=3.96043M/s __llvm_libc::memset,memset Google A
BM_Memset/1/0        296 ns          189 ns      2900992 bytes_per_cycle=0.234589/s bytes_per_second=335.583M/s items_per_second=5.29382M/s __llvm_libc::memset,memset Google B
BM_Memset/2/0       2110 ns         1049 ns       678912 bytes_per_cycle=0.24687/s bytes_per_second=353.151M/s items_per_second=953.527k/s __llvm_libc::memset,memset Google D
BM_Memset/3/0        397 ns          254 ns      3055616 bytes_per_cycle=0.238479/s bytes_per_second=341.147M/s items_per_second=3.93224M/s __llvm_libc::memset,memset Google L
BM_Memset/4/0       1119 ns          621 ns      1079296 bytes_per_cycle=0.244925/s bytes_per_second=350.368M/s items_per_second=1.61047M/s __llvm_libc::memset,memset Google M
BM_Memset/5/0        605 ns          349 ns      1644544 bytes_per_cycle=0.241364/s bytes_per_second=345.274M/s items_per_second=2.8614M/s __llvm_libc::memset,memset Google Q
BM_Memset/6/0        472 ns          271 ns      2310144 bytes_per_cycle=0.238615/s bytes_per_second=341.341M/s items_per_second=3.68799M/s __llvm_libc::memset,memset Google S
BM_Memset/7/0        262 ns          143 ns      3956736 bytes_per_cycle=0.225812/s bytes_per_second=323.026M/s items_per_second=7.0087M/s __llvm_libc::memset,memset Google U
BM_Memset/8/0        454 ns          261 ns      2940928 bytes_per_cycle=0.238883/s bytes_per_second=341.725M/s items_per_second=3.82716M/s __llvm_libc::memset,memset Google W
BM_Memset/9/0       8768 ns         5998 ns       115712 bytes_per_cycle=0.249196/s bytes_per_second=356.478M/s items_per_second=166.724k/s __llvm_libc::memset,uniform 384 to 4096
```

After
```
BM_Memset/0/0        117 ns         69.5 ns      9761792 bytes_per_cycle=0.935152/s bytes_per_second=1.30639G/s items_per_second=14.3834M/s __llvm_libc::memset,memset Google A
BM_Memset/1/0       97.8 ns         58.5 ns     13002752 bytes_per_cycle=0.892814/s bytes_per_second=1.24725G/s items_per_second=17.0848M/s __llvm_libc::memset,memset Google B
BM_Memset/2/0        326 ns          163 ns      5156864 bytes_per_cycle=1.54408/s bytes_per_second=2.15706G/s items_per_second=6.1192M/s __llvm_libc::memset,memset Google D
BM_Memset/3/0        132 ns         65.4 ns     11455488 bytes_per_cycle=0.876411/s bytes_per_second=1.22433G/s items_per_second=15.2803M/s __llvm_libc::memset,memset Google L
BM_Memset/4/0        222 ns          120 ns      6405120 bytes_per_cycle=1.44398/s bytes_per_second=2.01722G/s items_per_second=8.30758M/s __llvm_libc::memset,memset Google M
BM_Memset/5/0        119 ns         79.2 ns      8930304 bytes_per_cycle=1.13327/s bytes_per_second=1.58317G/s items_per_second=12.6189M/s __llvm_libc::memset,memset Google Q
BM_Memset/6/0        123 ns         64.0 ns     11609088 bytes_per_cycle=1.008/s bytes_per_second=1.40817G/s items_per_second=15.6365M/s __llvm_libc::memset,memset Google S
BM_Memset/7/0       85.9 ns         52.1 ns     12423168 bytes_per_cycle=0.641164/s bytes_per_second=917.192M/s items_per_second=19.1937M/s __llvm_libc::memset,memset Google U
BM_Memset/8/0        114 ns         67.1 ns     10347520 bytes_per_cycle=0.911968/s bytes_per_second=1.274G/s items_per_second=14.9015M/s __llvm_libc::memset,memset Google W
BM_Memset/9/0       1326 ns          785 ns       907264 bytes_per_cycle=1.89716/s bytes_per_second=2.6503G/s items_per_second=1.27348M/s __llvm_libc::memset,uniform 384 to 4096
```

Again not as good as current glibc but it's a first step in the right direction.
```
BM_Memset/0/0        108 ns         53.6 ns     12894208 bytes_per_cycle=1.02858/s bytes_per_second=1.4369G/s items_per_second=18.668M/s glibc::memset,memset Google A
BM_Memset/1/0       84.6 ns         47.6 ns     14284800 bytes_per_cycle=1.00197/s bytes_per_second=1.39974G/s items_per_second=21.0256M/s glibc::memset,memset Google B
BM_Memset/2/0        160 ns         85.8 ns      8927232 bytes_per_cycle=3.30805/s bytes_per_second=4.62129G/s items_per_second=11.6596M/s glibc::memset,memset Google D
BM_Memset/3/0       78.9 ns         53.6 ns     13326336 bytes_per_cycle=1.14058/s bytes_per_second=1.59338G/s items_per_second=18.674M/s glibc::memset,memset Google L
BM_Memset/4/0       99.2 ns         60.8 ns     11460608 bytes_per_cycle=2.54751/s bytes_per_second=3.55884G/s items_per_second=16.4587M/s glibc::memset,memset Google M
BM_Memset/5/0       93.0 ns         56.1 ns     12219392 bytes_per_cycle=1.73379/s bytes_per_second=2.42207G/s items_per_second=17.8157M/s glibc::memset,memset Google Q
BM_Memset/6/0       89.4 ns         47.2 ns     14692352 bytes_per_cycle=1.34846/s bytes_per_second=1.88377G/s items_per_second=21.1795M/s glibc::memset,memset Google S
BM_Memset/7/0       84.0 ns         50.0 ns     14468096 bytes_per_cycle=0.911198/s bytes_per_second=1.27293G/s items_per_second=19.994M/s glibc::memset,memset Google U
BM_Memset/8/0       93.4 ns         52.8 ns     13063168 bytes_per_cycle=1.06642/s bytes_per_second=1.48977G/s items_per_second=18.9524M/s glibc::memset,memset Google W
BM_Memset/9/0        438 ns          241 ns      2853888 bytes_per_cycle=6.1185/s bytes_per_second=8.54744G/s items_per_second=4.15064M/s glibc::memset,uniform 384 to 4096
```

Reviewed By: sivachandra

Differential Revision: https://reviews.llvm.org/D150433
2023-05-15 07:30:54 +00:00
Mark de Wever
65429b9af6 Reland "[CMake] Bumps minimum version to 3.20.0."
The owner of the last two failing buildbots updated CMake.

This reverts commit e8e8707b4a.
2023-05-13 11:42:25 +02:00
Tue Ly
36b702901a [libc][math] Implement fast division / modulus for UInt / (uint32_t * 2^e).
This is to improve a performance bottleneck of printf for long double.

Reviewed By: michaelrj

Differential Revision: https://reviews.llvm.org/D150475
2023-05-12 21:01:15 -04:00
Joseph Huber
182e5acb11 [libc] Check the RPC server once again after the kernel exits
We support asynchronous sends, that means that the kernel can issue a
send, then exit the kernel as we do with the `EXIT` syscall. Because of
the condition it's therefore possible for the kernel to exit and break
from the loop before we check the server again. This can potentially
cause us to ignore an `EXIT` call from the GPU.

Reviewed By: JonChesterfield, lntue

Differential Revision: https://reviews.llvm.org/D150456
2023-05-12 12:49:19 -05:00
Joseph Huber
f6e5f90fd4 [libc] Fix undeclared 'free' function in stream test
Summary:
We need this function from the test.cpp but need to declare it manually.
2023-05-11 12:49:40 -05:00
Joseph Huber
d21e507cfc [libc] Implement a generic streaming interface in the RPC
Currently we provide the `send_n` and `recv_n` functions. These were
somewhat divergent and not tested on the GPU. This patch changes the
support to be more common. We do this my making the CPU provide an array
equal the to at least the lane size while the GPU can rely on the
private memory address of its stack variables. This allows us to send
data back and forth generically.

Reviewed By: JonChesterfield

Differential Revision: https://reviews.llvm.org/D150379
2023-05-11 11:55:41 -05:00
Joseph Huber
30093d6be2 [libc][obvious] Fix undefined variable after name change
I forgot that we still used these variables in the loaders.

Differential Revision: https://reviews.llvm.org/D150362
2023-05-11 09:00:08 -05:00
Joseph Huber
4a2e50e4f4 [libc][NFC] Clean up some code in the RPC implementation.
Small cleanup of the server code and fixes a constant name not following
the naming convention.

Differential Revision: https://reviews.llvm.org/D150361
2023-05-11 08:22:55 -05:00
Guillaume Chatelet
eea5d9cc41 [libc][benchmark] Do not force static linking
Being able to link statically depends on other CMake options and choice of libc.
2023-05-11 09:09:28 +00:00
Guillaume Chatelet
b51e6bfcb6 [libc] Allows cross compilation of membenchmarks
This patch makes sure:
 - we pass the correct compiler options when building Google benchmarks,
 - we only import the C++ version of the memory functions.

The change in libc/cmake/modules/LLVMLibCTestRules.cmake is here to make sure CMake can generate the right command line in the presence of the CMAKE_CROSSCOMPILING_EMULATOR option.

Relevant documentation:
https://cmake.org/cmake/help/latest/variable/CMAKE_CROSSCOMPILING_EMULATOR.html
https://cmake.org/cmake/help/latest/command/add_custom_command.html#command:add_custom_command
"
If COMMAND specifies an executable target name (created by the `add_executable()` command), it will automatically be replaced by the location of the executable created at build time if either of the following is true:
 - The target is not being cross-compiled (i.e. the CMAKE_CROSSCOMPILING variable is not set to true).
 - New in version 3.6: The target is being cross-compiled and an emulator is provided (i.e. its CROSSCOMPILING_EMULATOR target property is set). In this case, the contents of CROSSCOMPILING_EMULATOR will be prepended to the command before the location of the target executable.
"

Reviewed By: gchatelet

Differential Revision: https://reviews.llvm.org/D150200
2023-05-11 08:57:50 +00:00
Jon Chesterfield
bbeae142bf [libc][rpc] Allocate a single block of shared memory instead of three
Allows moving the pointer swap between server and client into reset.
Single allocation simplifies whatever allocates the client/server, currently
the libc loaders.

Reviewed By: jhuber6

Differential Revision: https://reviews.llvm.org/D150337
2023-05-11 03:04:56 +01:00
Joseph Huber
c8c19e1c31 [libc] Fix RPC interface when sending and recieving aribtrary packets
The interface exported by the RPC library allows users to simply send
and recieve fixed sized packets without worrying about the data motion
underneath. However, this was broken in the current implementation. We
can think of the send and recieve implementations in terms of waiting
for ownership of the buffer, using the buffer, and posting ownership to
the other side. Our implementation of `recv` was incorrect in the
following scenarios.

recv -> send // we still own the buffer and should give away ownership
recv -> close // The other side is not waiting for data, this will
                 result in multiple openings of the same port

This patch attempts to fix this with an admittedly hacky fix where we
track if the previous implementation was a recv and post conditionally.

Reviewed By: JonChesterfield

Differential Revision: https://reviews.llvm.org/D150327
2023-05-10 18:51:38 -05:00
Jon Chesterfield
f497611f43 [libc][rpc] Allocate locks array within process
Replaces the globals currently used. Worth changing to a bitmap
before allowing runtime number of ports >> 64. One bit per port is likely
to be cheap enough that sizing for the worst case is always fine, otherwise
in the future we can change to dynamically allocating it.

Reviewed By: jhuber6

Differential Revision: https://reviews.llvm.org/D150309
2023-05-11 00:41:51 +01:00
Joseph Huber
bcc2021efd [libc] Prevent changing ownership of the port once opened
The Port type has stipuations that the same exact mask used to open it
needs to close it. This can currently be violated by calling its move
constructor to put it somewhere else. We still need the move constructor
to handle the open and closing functions. So, we simply make these
constructors private and only allow a few classes to have move
priviledges on it.

Reviewed By: JonChesterfield, lntue

Differential Revision: https://reviews.llvm.org/D150118
2023-05-10 18:34:29 -05:00
Guillaume Chatelet
f4a3549250 [libc] Add optimized memcpy for RISCV
This patch adds two versions of memcpy optimized for architectures where unaligned accesses are either illegal or extremely slow.
It is currently enabled for RISCV 64 and RISCV 32 but it could be used for ARM 32 architectures as well.

Here is the before / after output of `libc.benchmarks.memory_functions.opt_host --benchmark_filter=BM_Memcpy` on a quad core Linux starfive RISCV 64 board running at 1.5GHz.

Before:
```
Run on (4 X 1500 MHz CPU s)
CPU Caches:
  L1 Instruction 32 KiB (x4)
  L1 Data 32 KiB (x4)
  L2 Unified 2048 KiB (x1)
------------------------------------------------------------------------
Benchmark              Time             CPU   Iterations UserCounters...
------------------------------------------------------------------------
BM_Memcpy/0/0        474 ns          474 ns      1483776 bytes_per_cycle=0.243492/s bytes_per_second=348.318M/s items_per_second=2.11097M/s __llvm_libc::memcpy,memcpy Google A
BM_Memcpy/1/0        210 ns          209 ns      3649536 bytes_per_cycle=0.233819/s bytes_per_second=334.481M/s items_per_second=4.77519M/s __llvm_libc::memcpy,memcpy Google B
BM_Memcpy/2/0       1814 ns         1814 ns       396288 bytes_per_cycle=0.247899/s bytes_per_second=354.622M/s items_per_second=551.402k/s __llvm_libc::memcpy,memcpy Google D
BM_Memcpy/3/0       89.3 ns         89.2 ns      7459840 bytes_per_cycle=0.217415/s bytes_per_second=311.014M/s items_per_second=11.2071M/s __llvm_libc::memcpy,memcpy Google L
BM_Memcpy/4/0        134 ns          134 ns      3815424 bytes_per_cycle=0.226584/s bytes_per_second=324.131M/s items_per_second=7.44567M/s __llvm_libc::memcpy,memcpy Google M
BM_Memcpy/5/0       52.8 ns         52.6 ns     11001856 bytes_per_cycle=0.194893/s bytes_per_second=278.797M/s items_per_second=19.0284M/s __llvm_libc::memcpy,memcpy Google Q
BM_Memcpy/6/0        180 ns          180 ns      4101120 bytes_per_cycle=0.231884/s bytes_per_second=331.713M/s items_per_second=5.55957M/s __llvm_libc::memcpy,memcpy Google S
BM_Memcpy/7/0        195 ns          195 ns      3906560 bytes_per_cycle=0.232951/s bytes_per_second=333.239M/s items_per_second=5.1217M/s __llvm_libc::memcpy,memcpy Google U
BM_Memcpy/8/0        152 ns          152 ns      4789248 bytes_per_cycle=0.227507/s bytes_per_second=325.452M/s items_per_second=6.58187M/s __llvm_libc::memcpy,memcpy Google W
BM_Memcpy/9/0       6036 ns         6033 ns       118784 bytes_per_cycle=0.249158/s bytes_per_second=356.423M/s items_per_second=165.75k/s __llvm_libc::memcpy,uniform 384 to 4096
```

After:
```
BM_Memcpy/0/0        126 ns          126 ns      5770240 bytes_per_cycle=1.04707/s bytes_per_second=1.46273G/s items_per_second=7.9385M/s __llvm_libc::memcpy,memcpy Google A
BM_Memcpy/1/0       75.1 ns         75.0 ns     10204160 bytes_per_cycle=0.691143/s bytes_per_second=988.687M/s items_per_second=13.3289M/s __llvm_libc::memcpy,memcpy Google B
BM_Memcpy/2/0        333 ns          333 ns      2174976 bytes_per_cycle=1.39297/s bytes_per_second=1.94596G/s items_per_second=3.00002M/s __llvm_libc::memcpy,memcpy Google D
BM_Memcpy/3/0       49.6 ns         49.5 ns     16092160 bytes_per_cycle=0.710161/s bytes_per_second=1015.89M/s items_per_second=20.1844M/s __llvm_libc::memcpy,memcpy Google L
BM_Memcpy/4/0       57.7 ns         57.7 ns     11213824 bytes_per_cycle=0.561557/s bytes_per_second=803.314M/s items_per_second=17.3228M/s __llvm_libc::memcpy,memcpy Google M
BM_Memcpy/5/0       48.0 ns         47.9 ns     16437248 bytes_per_cycle=0.346708/s bytes_per_second=495.97M/s items_per_second=20.8571M/s __llvm_libc::memcpy,memcpy Google Q
BM_Memcpy/6/0       67.5 ns         67.5 ns     10616832 bytes_per_cycle=0.614173/s bytes_per_second=878.582M/s items_per_second=14.8142M/s __llvm_libc::memcpy,memcpy Google S
BM_Memcpy/7/0       84.7 ns         84.6 ns     10480640 bytes_per_cycle=0.819077/s bytes_per_second=1.14424G/s items_per_second=11.8174M/s __llvm_libc::memcpy,memcpy Google U
BM_Memcpy/8/0       61.7 ns         61.6 ns     11191296 bytes_per_cycle=0.550078/s bytes_per_second=786.893M/s items_per_second=16.2279M/s __llvm_libc::memcpy,memcpy Google W
BM_Memcpy/9/0        981 ns          981 ns       703488 bytes_per_cycle=1.52333/s bytes_per_second=2.12807G/s items_per_second=1019.81k/s __llvm_libc::memcpy,uniform 384 to 4096
```

It is not as good as glibc for now so there's room for improvement. I suspect a path pumping 16 bytes at once given the doubled numbers for large copies.
```
BM_Memcpy/0/1        146 ns         82.5 ns      8576000 bytes_per_cycle=1.35236/s bytes_per_second=1.88922G/s items_per_second=12.1169M/s glibc memcpy,memcpy Google A
BM_Memcpy/1/1        112 ns         63.7 ns     10634240 bytes_per_cycle=0.628018/s bytes_per_second=898.387M/s items_per_second=15.702M/s glibc memcpy,memcpy Google B
BM_Memcpy/2/1        315 ns          180 ns      4079616 bytes_per_cycle=2.65229/s bytes_per_second=3.7052G/s items_per_second=5.54764M/s glibc memcpy,memcpy Google D
BM_Memcpy/3/1       85.3 ns         43.1 ns     15854592 bytes_per_cycle=0.774164/s bytes_per_second=1107.45M/s items_per_second=23.2249M/s glibc memcpy,memcpy Google L
BM_Memcpy/4/1        105 ns         54.3 ns     13427712 bytes_per_cycle=0.7793/s bytes_per_second=1114.8M/s items_per_second=18.4109M/s glibc memcpy,memcpy Google M
BM_Memcpy/5/1       77.1 ns         43.2 ns     16476160 bytes_per_cycle=0.279808/s bytes_per_second=400.269M/s items_per_second=23.1428M/s glibc memcpy,memcpy Google Q
BM_Memcpy/6/1        112 ns         62.7 ns     11236352 bytes_per_cycle=0.676078/s bytes_per_second=967.137M/s items_per_second=15.9387M/s glibc memcpy,memcpy Google S
BM_Memcpy/7/1        131 ns         65.5 ns     11751424 bytes_per_cycle=0.965616/s bytes_per_second=1.34895G/s items_per_second=15.2762M/s glibc memcpy,memcpy Google U
BM_Memcpy/8/1        104 ns         55.0 ns     12314624 bytes_per_cycle=0.583336/s bytes_per_second=834.468M/s items_per_second=18.1937M/s glibc memcpy,memcpy Google W
BM_Memcpy/9/1        932 ns          466 ns      1480704 bytes_per_cycle=3.17342/s bytes_per_second=4.43321G/s items_per_second=2.14679M/s glibc memcpy,uniform 384 to 4096
```

Reviewed By: sivachandra

Differential Revision: https://reviews.llvm.org/D150202
2023-05-10 08:42:07 +00:00
Siva Chandra Reddy
bb2ebbd190 [libc][NFC] Simplify string-table generation internals.
Reviewed By: michaelrj

Differential Revision: https://reviews.llvm.org/D150088
2023-05-08 18:36:08 +00:00
Jon Chesterfield
616f830127 [libc][rpc][nfc] Encapsulate access to outbox pointer
Reviewed By: jhuber6

Differential Revision: https://reviews.llvm.org/D150065
2023-05-08 17:37:54 +01:00
Joseph Huber
91b6d319cb [libc] Make the opcode parameter a compile time constant
Currently the opcode is only valid if it is the same between all of the
ports. This is possible to violate if the opcode is places into a memory
location and then read in a non-uniform manner by the warp / wavefront.
Moving this to a compile time constant makes it impossible to break this
invariant.

Reviewed By: JonChesterfield

Differential Revision: https://reviews.llvm.org/D150115
2023-05-08 11:16:29 -05:00
Roland McGrath
46b76a54e1 [libc] Use Linux errno and signal strings for Fuchsia
The exact set of supported values is determined by the <errno.h>
and <signal.h> headers, which don't (yet) come from llvm-libc on
Fuchsia. The mappings of SIG* and E* codes to psignal/strsignal
and perror/strerror text used in Fuchsia libc today is the same
as for Linux.

Reviewed By: abrachet

Differential Revision: https://reviews.llvm.org/D150026
2023-05-07 17:22:21 -07:00
Kazu Hirata
282798fefd [libc] Fix typos in documentation 2023-05-06 11:26:32 -07:00
Mark de Wever
e8e8707b4a Revert "Reland "[CMake] Bumps minimum version to 3.20.0.""
Unfortunatly not all buildbots are updated.

This reverts commit ffb807ab53.
2023-05-06 17:03:56 +02:00
Mark de Wever
ffb807ab53 Reland "[CMake] Bumps minimum version to 3.20.0."
All build bots should be updated now.

This reverts commit 44d38022ab.
2023-05-06 11:43:02 +02:00
Joseph Huber
023bfc196b [libc][docs] Fix incorrect CMake argument in GPU documentation
Summary;
This was changed a long time ago to drop the `LLVM_` prefix.

Differential Revision: https://reviews.llvm.org/D150012
2023-05-05 20:16:47 -05:00
Joseph Huber
71d2b65e36 [libc] Make the RPC interfaces move only
This patch uses the changed interface in D149972 to make these classes
move-only. The `Port` class could be further refined to be
construct-only in a future patch, but for now this makes it more
difficult to misuse the interface.

Depends on D149972

Reviewed By: JonChesterfield

Differential Revision: https://reviews.llvm.org/D149974
2023-05-05 13:45:23 -05:00
Joseph Huber
b8e878da5c [libc] Rework 'cpp:optional' to support move construction
This patch replaces the existing `cpp::optional` type with a newer
version that has more features. This class is heavily inspired by the
old `llvm::Optional` class. Currently the limitations of this class is
that we only handle types with trivial constructors or operators.

Reviewed By: sivachandra

Differential Revision: https://reviews.llvm.org/D149972
2023-05-05 13:45:22 -05:00
Joseph Huber
aea866c12c [libc] Support concurrent RPC port access on the GPU
Previously we used a single port to implement the RPC. This was
sufficient for single threaded tests but can potentially cause deadlocks
when using multiple threads. The reason for this is that GPUs make no
forward progress guarantees. Therefore one group of threads waiting on
another group of threads can spin forever because there is no guarantee
that the other threads will continue executing. The typical workaround
for this is to allocate enough memory that a sufficiently large number
of work groups can make progress. As long as this number is somewhat
close to the amount of total concurrency we can obtain reliable
execution around a shared resource.

This patch enables using multiple ports by widening the arrays to a
predetermined size and indexes into them. Empty ports are currently
obtained via a trivial linker scan. This should be imporoved in the
future for performance reasons. Portions of D148191 were applied to
achieve parallel support.

Depends on D149581

Reviewed By: JonChesterfield

Differential Revision: https://reviews.llvm.org/D149598
2023-05-05 10:12:19 -05:00
Joseph Huber
2e93dd0613 [libc] Fix hanging test on NVPTX due to lack of warp sync
Previously this wasn't implemented because it's effectively a no-op.
However, this should be safe to emit on sm_60 architectures. It's
important because it carries semantic importance for whether or not
something can be moved. So we should always emit this instrinsic.

Differential Revision: https://reviews.llvm.org/D149923
2023-05-04 21:38:24 -05:00
Joseph Huber
901266dad3 [libc] Change GPU startup and loader to use multiple kernels
The GPU has a different execution model to standard `_start`
implementations. On the GPU, all threads are active at the start of a
kernel. In order to correctly intitialize and call the constructors we
want single threaded semantics. Previously, this was done using a
makeshift global barrier with atomics. However, it should be easier to
simply put the portions of the code that must be single threaded in
separate kernels and then call those with only one thread. Generally,
mixing global state between kernel launches makes optimizations more
difficult, similarly to calling a function outside of the TU, but for
testing it is better to be correct.

Depends on D149527 D148943

Reviewed By: JonChesterfield

Differential Revision: https://reviews.llvm.org/D149581
2023-05-04 19:31:41 -05:00
Joseph Huber
507edb52f9 [libc] Enable multiple threads to use RPC on the GPU
The execution model of the GPU expects that groups of threads will
execute in lock-step in SIMD fashion. It's both important for
performance and correctness that we treat this as the smallest possible
granularity for an RPC operation. Thus, we map multiple threads to a
single larger buffer and ship that across the wire.

This patch makes the necessary changes to support executing the RPC on
the GPU with multiple threads. This requires some workarounds to mimic
the model when handling the protocol from the CPU. I'm not completely
happy with some of the workarounds required, but I think it should work.

Uses some of the implementation details from D148191.

Reviewed By: JonChesterfield

Differential Revision: https://reviews.llvm.org/D148943
2023-05-04 19:31:41 -05:00
Mikhail R. Gadelha
4c9c1a4e4f [libc] Enable linux directory entries syscalls in riscv64
This patch updates the struct dirent to be on par with glibc (by adding
a missing d_type member) and update the readdir call to use SYS_getdents64
instead of SYS_getdents.

Reviewed By: sivachandra

Differential Revision: https://reviews.llvm.org/D147738
2023-05-04 19:07:16 -03:00
Jon Chesterfield
75b7b9f292 [libc][rpc] Update locking to work on volta
Carefully work around not knowing the thread mask that nvptx intrinsic
functions require.

If the warp is converged when calling try_lock, a single rpc call will handle
all lanes within it. Otherwise more than one rpc call with thread masks that
compose to the unknown one will occur.

Reviewed By: jhuber6

Differential Revision: https://reviews.llvm.org/D149897
2023-05-04 22:58:04 +01:00
Jon Chesterfield
8aaaa1c3b9 Revert "[libc][rpc] Update locking to work on volta"
This reverts commit b132373864.
2023-05-04 22:44:33 +01:00
Jon Chesterfield
b132373864 [libc][rpc] Update locking to work on volta
Carefully work around not knowing the thread mask that nvptx intrinsic
functions require.

If the warp is converged when calling try_lock, a single rpc call will handle
all lanes within it. Otherwise more than one rpc call with thread masks that
compose to the unknown one will occur.

Reviewed By: jhuber6

Differential Revision: https://reviews.llvm.org/D149897
2023-05-04 22:30:54 +01:00
Jon Chesterfield
5cba7717fc Revert "[libc][rpc] Land helpers from D148943"
This reverts commit 09ceb4729f.
2023-05-04 21:45:18 +01:00
Jon Chesterfield
09ceb4729f [libc][rpc] Land helpers from D148943 2023-05-04 20:53:55 +01:00
Joseph Huber
9d0258e363 [libc] Remove support for atomic test due to failing on sm_60
This test fails on sm_60 because of the atomics codegen. We test atomics
indirectly with the `rpc` so we still have coverage.

Differential Revision: https://reviews.llvm.org/D149887
2023-05-04 14:14:13 -05:00
Siva Chandra Reddy
b77265711b Revert "[libc] Improve the add_libc_test rule."
This reverts commit fb6faf4798 as the
aarch64 builders are failing.
2023-05-04 19:09:19 +00:00
Siva Chandra Reddy
fb6faf4798 [libc] Improve the add_libc_test rule.
A target for the test named ${fq_target_name} has been added. It depends
on ${fq_target_name}.__unit__ and ${fq_target_name}.__hermetic__ as
relevant.

Reviewed By: jhuber6

Differential Revision: https://reviews.llvm.org/D149730
2023-05-04 18:53:13 +00:00
Joseph Huber
cbcf55d32b [libc] Maintain proper alignment for the hermetic tests malloc
We use a bump pointer to implement malloc for the hermetic tests.
Currently, we bump the pointer up by any amount. This means that calling
`malloc(1)` will misalign the buffer so any following `malloc(8)`
accesses will not be aligned. This causes problems in architectures
which require alignment.

Reviewed By: sivachandra

Differential Revision: https://reviews.llvm.org/D149863
2023-05-04 13:22:20 -05:00
Jon Chesterfield
a4ba7da367 [libc][rpc] Pass lane_mask into lock functions
Will be necessary for correct locking on volta.

API-only change to help with rebasing the rest of the stack on this.

Reviewed By: jhuber6

Differential Revision: https://reviews.llvm.org/D149871
2023-05-04 12:35:28 -05:00
Joseph Huber
62a2a07cbe [libc][Docs] Add warning about running GPU tests with parallelism
Now that a large amount of GPU tests can be run in parallel, we should
document spurious failures. It is a well-known issue that launching many
GPU applications in parallel can lead to various problems, the worst of
which being an indefinite hang.

Differential Revision: https://reviews.llvm.org/D149857
2023-05-04 10:18:06 -05:00
Joseph Huber
f4002c1415 [libc] Enable running libc unit tests on NVPTX
The previous patches added the necessary support for global constructors
used to register tests. This patch enables the NVPTX target to build
and run the unit tests on the GPU. Currently this only tests the ctype
tests, but adding more should be straightforward from here on.

This ran all the ctest unit tests when run on an sm_70.

Depends on D149517 D149527

Reviewed By: sivachandra

Differential Revision: https://reviews.llvm.org/D149532
2023-05-04 08:36:29 -05:00
Jon Chesterfield
f19f7497ca [libc][rpc] Factor try_lock, unlock into functions on Process
Reduces line noise and localises changes needed for volta

Reviewed By: jhuber6

Differential Revision: https://reviews.llvm.org/D149845
2023-05-04 13:58:25 +01:00
Joseph Huber
2e1c0ec629 [libc] Support global constructors and destructors on NVPTX
This patch adds the necessary hacks to support global constructors and
destructors. This is an incredibly hacky process caused by the primary
fact that Nvidia does not provide any binary tools and very little
linker support. We first had to emit references to these functions and
their priority in D149451. Then we dig them out of the module once it's
loaded to manually create the list that the linker should have made for
us. This patch also contains a few Nvidia specific hacks, but it passes
the test, albeit with a stack size warning from `ptxas` for the
callback. But this should be fine given the resource usage of a common
test.

This also adds a dependency on LLVM to the NVPTX loader, which hopefully doesn't
cause problems with our CUDA buildbot.

Depends on D149451

Reviewed By: tra

Differential Revision: https://reviews.llvm.org/D149527
2023-05-04 07:13:00 -05:00
Jon Chesterfield
3cd230e0d9 [libc][rpc] Fold can send/recv into buffer_unavailable
Left out of D149788 to simplify the diff

Reviewed By: jhuber6

Differential Revision: https://reviews.llvm.org/D149807
2023-05-04 13:09:35 +01:00
Joseph Huber
632fa3798c [libc] Enable running libc unit tests on AMDGPU
The previous patches added the necessary support for global constructors
used to register tests. This patch enables the AMDGPU target to build
and run the unit tests on the GPU. Currently this only tests the `ctype`
tests, but adding more should be straightforward from here on.

Reviewed By: sivachandra

Differential Revision: https://reviews.llvm.org/D149517
2023-05-04 06:32:52 -05:00
Jon Chesterfield
cb7e3da054 [libc][rpc] Treat pointers as arrays consistently
Noticed in passing. Either way compiles.

Reviewed By: jhuber6

Differential Revision: https://reviews.llvm.org/D149808
2023-05-04 10:52:45 +01:00