Fabian Ritter
a4fd3dba6e
[AMDGPU] Use wider loop lowering type for LowerMemIntrinsics ( #112332 )
...
When llvm.memcpy or llvm.memmove intrinsics are lowered as a loop in
LowerMemIntrinsics.cpp, the loop consists of a single load/store pair
per iteration. We can improve performance in some cases by emitting
multiple load/store pairs per iteration. This patch achieves that by
increasing the width of the loop lowering type in the GCN target and
letting legalization split the resulting too-wide access pairs into
multiple legal access pairs.
This change only affects lowered memcpys and memmoves with large (>=
1024 bytes) constant lengths. Smaller constant lengths are handled by
ISel directly; non-constant lengths would be slowed down by this change
if the dynamic length was smaller or slightly larger than what an
unrolled iteration copies.
The chosen default unroll factor is the result of microbenchmarks on
gfx1030. This change leads to speedups of 15-38% for global memory and
1.9-5.8x for scratch in these microbenchmarks.
Part of SWDEV-455845.
2024-10-28 09:04:19 +01:00
..
2024-06-22 00:40:59 +02:00
2024-10-21 09:39:52 -07:00
2024-01-16 21:54:58 -08:00
2024-10-16 09:43:16 +02:00
2024-09-25 07:04:09 +01:00
2024-07-25 19:21:28 +02:00
2023-07-31 20:15:45 -04:00
2022-11-28 11:51:36 -05:00
2024-06-25 14:35:19 +05:30
2024-10-07 07:35:31 +02:00
2024-10-07 07:35:31 +02:00
2024-10-21 09:39:52 -07:00
2024-10-21 09:39:52 -07:00
2024-07-15 11:51:44 +04:00
2023-08-22 14:25:31 +01:00
2024-02-27 10:44:53 +05:30
2024-02-27 10:44:53 +05:30
2024-02-27 10:44:53 +05:30
2023-06-05 16:59:07 +00:00
2024-07-15 11:51:44 +04:00
2024-04-08 08:44:13 +01:00
2024-04-16 02:36:12 -04:00
2024-02-22 09:47:36 +01:00
2024-08-07 12:13:39 +02:00
2024-10-08 09:41:24 -04:00
2024-02-20 10:54:44 +00:00
2024-02-20 10:54:44 +00:00
2024-03-07 19:09:18 -08:00
2024-09-11 15:04:55 +02:00
2024-09-11 15:04:55 +02:00
2024-08-01 09:42:04 +04:00
2024-07-15 11:51:44 +04:00
2024-01-16 21:54:58 -08:00
2024-09-23 13:20:21 +04:00
2024-05-06 14:07:11 +05:30
2024-09-23 13:20:21 +04:00
2024-10-21 13:21:16 +05:30
2024-02-29 13:57:59 +01:00
2024-05-06 14:07:11 +05:30
2024-02-29 15:38:54 +01:00
2024-05-06 14:07:11 +05:30
2024-02-29 15:38:54 +01:00
2024-05-06 14:07:11 +05:30
2024-07-30 18:18:16 +04:00
2024-03-06 09:51:48 -05:00
2023-02-21 10:42:39 +01:00
2024-07-15 11:51:44 +04:00
2024-03-09 09:07:22 +00:00
2023-12-06 11:23:23 +05:30
2024-03-09 09:07:22 +00:00
2024-03-09 09:07:22 +00:00
2024-10-21 09:39:52 -07:00
2024-10-08 09:49:20 -04:00
2024-01-16 21:54:58 -08:00
2024-09-16 19:25:34 +02:00
2024-08-22 23:27:33 +04:00
2024-08-22 23:27:33 +04:00
2024-08-22 23:27:33 +04:00
2024-08-01 09:42:04 +04:00
2024-09-06 11:00:30 -04:00
2024-10-21 09:39:52 -07:00
2024-07-30 18:18:16 +04:00
2024-01-16 21:54:58 -08:00
2024-10-08 09:41:24 -04:00
2024-10-08 09:41:24 -04:00
2024-10-09 09:27:28 +04:00
2024-08-22 23:27:33 +04:00
2024-01-16 21:54:58 -08:00
2024-10-08 09:49:20 -04:00
2024-09-11 15:04:55 +02:00
2024-09-11 15:04:55 +02:00
2024-07-15 11:51:44 +04:00
2024-06-18 13:20:49 +01:00
2024-09-24 11:08:41 +04:00
2024-09-24 11:08:41 +04:00
2024-09-24 11:08:41 +04:00
2024-09-24 11:08:41 +04:00
2024-09-24 11:08:41 +04:00
2022-11-28 11:51:36 -05:00
2023-11-01 19:48:30 +00:00
2022-11-28 11:51:36 -05:00
2024-10-21 13:21:16 +05:30
2024-07-15 11:51:44 +04:00
2024-10-01 10:29:35 +02:00
2020-09-30 10:35:25 -04:00
2024-08-01 09:42:04 +04:00
2024-10-21 09:39:52 -07:00
2024-10-21 09:39:52 -07:00
2024-10-21 09:39:52 -07:00
2024-10-21 09:39:52 -07:00
2024-01-16 21:54:58 -08:00
2024-10-16 10:27:44 -04:00
2024-10-16 12:26:01 -04:00
2024-10-16 12:26:01 -04:00
2024-09-24 11:08:41 +04:00
2024-07-28 11:26:29 +04:00
2024-10-01 09:25:36 -04:00
2024-01-16 21:54:58 -08:00
2022-06-27 12:15:29 +01:00
2024-07-30 18:31:18 +04:00
2024-09-24 11:08:41 +04:00
2024-03-02 20:10:02 +01:00
2024-10-16 12:26:01 -04:00
2024-08-21 17:47:36 +03:00
2024-07-30 18:18:16 +04:00
2024-10-16 12:26:01 -04:00
2024-03-02 20:10:02 +01:00
2024-10-16 12:26:01 -04:00
2024-10-16 12:26:01 -04:00
2024-09-24 11:08:41 +04:00
2022-06-27 12:15:29 +01:00
2024-01-16 21:54:58 -08:00
2024-01-16 21:54:58 -08:00
2024-01-16 21:54:58 -08:00
2024-07-30 18:18:16 +04:00
2024-01-16 21:54:58 -08:00
2024-09-24 11:08:41 +04:00
2024-05-30 14:02:15 +01:00
2024-10-10 13:37:44 +02:00
2024-07-30 18:18:16 +04:00
2024-10-01 09:25:36 -04:00
2024-07-28 11:26:29 +04:00
2024-01-16 21:54:58 -08:00
2024-01-16 21:54:58 -08:00
2024-10-01 09:25:36 -04:00
2024-01-16 21:54:58 -08:00
2024-10-01 09:25:36 -04:00
2024-01-16 21:54:58 -08:00
2024-02-29 15:38:54 +01:00
2024-07-28 11:26:29 +04:00
2024-07-30 18:18:16 +04:00
2024-01-16 21:54:58 -08:00
2024-03-02 20:10:02 +01:00
2024-10-10 14:01:49 +04:00
2024-10-01 09:25:36 -04:00
2024-10-16 12:26:01 -04:00
2024-01-16 21:54:58 -08:00
2024-01-16 21:54:58 -08:00
2024-07-28 11:26:29 +04:00
2024-01-16 21:54:58 -08:00
2024-10-16 12:26:01 -04:00
2024-01-16 21:54:58 -08:00
2024-09-25 15:02:23 +02:00
2024-07-28 11:26:29 +04:00
2024-03-02 20:10:02 +01:00
2024-10-10 14:01:49 +04:00
2024-07-15 11:51:44 +04:00
2024-07-15 11:51:44 +04:00
2023-07-31 12:15:39 +05:30
2024-03-06 09:51:48 -05:00
2024-06-15 09:55:04 +02:00
2024-03-06 09:51:48 -05:00
2024-03-06 09:51:48 -05:00
2024-03-26 11:01:08 +01:00
2024-06-14 20:56:43 +02:00
2024-06-14 20:56:43 +02:00
2024-06-14 20:56:43 +02:00
2024-03-06 09:51:48 -05:00
2023-11-07 09:34:16 +01:00
2024-02-27 10:44:53 +05:30
2024-07-15 11:51:44 +04:00
2024-05-06 14:07:11 +05:30
2024-06-18 13:20:49 +01:00
2022-11-28 11:51:36 -05:00
2024-03-06 09:51:48 -05:00
2024-03-06 09:51:48 -05:00
2024-02-27 10:44:53 +05:30
2024-04-08 08:44:13 +01:00
2024-03-06 09:49:58 -06:00
2023-12-11 11:06:50 +00:00
2024-07-15 11:51:44 +04:00
2024-03-06 09:51:48 -05:00
2024-05-10 13:02:05 +02:00
2024-07-15 11:51:44 +04:00
2024-01-16 21:54:58 -08:00
2024-07-15 11:51:44 +04:00
2024-08-19 18:22:29 +01:00
2024-05-06 14:07:11 +05:30
2024-05-06 14:07:11 +05:30
2024-03-29 15:59:50 -04:00
2024-07-28 11:26:29 +04:00
2024-07-08 14:01:32 +01:00
2024-04-08 08:44:13 +01:00
2024-04-08 08:44:13 +01:00
2024-06-28 23:03:39 +02:00
2024-10-08 09:49:20 -04:00
2024-09-25 13:02:29 +02:00
2024-09-25 13:02:29 +02:00
2024-09-25 13:02:29 +02:00
2024-06-28 23:03:39 +02:00
2024-02-27 10:44:53 +05:30
2024-02-27 10:44:53 +05:30
2024-02-27 10:44:53 +05:30
2024-02-27 10:44:53 +05:30
2024-02-27 10:44:53 +05:30
2024-06-15 18:19:52 +05:30
2024-10-10 13:37:44 +02:00
2024-10-11 08:50:49 +02:00
2024-10-10 13:37:44 +02:00
2024-10-10 13:37:44 +02:00
2024-10-11 08:50:49 +02:00
2024-09-25 13:02:29 +02:00
2024-03-09 09:07:22 +00:00
2024-05-30 12:40:42 +03:00
2024-03-29 15:59:50 -04:00
2024-05-30 12:40:42 +03:00
2024-03-29 15:59:50 -04:00
2024-10-10 13:37:44 +02:00
2024-05-22 10:55:19 -04:00
2024-03-23 13:12:44 +01:00
2024-09-25 07:04:09 +01:00
2024-06-14 19:14:19 +02:00
2024-10-21 09:39:52 -07:00
2024-10-11 11:40:27 +02:00
2024-03-06 09:51:48 -05:00
2024-09-19 09:00:21 +02:00
2024-10-21 09:39:52 -07:00
2024-10-21 09:39:52 -07:00
2024-07-15 11:51:44 +04:00
2024-10-21 09:39:52 -07:00
2024-10-21 09:39:52 -07:00
2024-07-15 11:51:44 +04:00
2024-10-24 16:19:18 -07:00
2024-01-18 10:47:45 +00:00
2024-01-18 10:47:45 +00:00
2024-01-18 10:47:45 +00:00
2024-10-21 09:39:52 -07:00
2023-06-05 16:59:07 +00:00
2024-04-01 09:48:37 +08:00
2024-07-30 18:18:16 +04:00
2024-07-15 11:51:44 +04:00
2024-07-30 18:18:16 +04:00
2024-10-21 09:39:52 -07:00
2024-07-15 11:51:44 +04:00
2023-12-15 13:45:03 +01:00
2024-02-27 10:44:53 +05:30
2023-08-09 14:20:51 +02:00
2023-12-15 13:45:03 +01:00
2023-12-15 13:45:03 +01:00
2023-12-15 13:45:03 +01:00
2024-05-27 14:09:17 +02:00
2023-08-09 14:20:51 +02:00
2023-08-09 14:20:51 +02:00
2023-08-09 14:20:51 +02:00
2023-08-09 14:20:51 +02:00
2024-01-12 11:20:01 -06:00
2023-08-09 14:20:51 +02:00
2023-08-09 14:20:51 +02:00
2024-01-12 11:20:01 -06:00
2023-08-09 14:20:51 +02:00
2023-08-09 14:20:51 +02:00
2023-12-15 13:45:03 +01:00
2023-12-15 13:45:03 +01:00
2024-07-30 18:18:16 +04:00
2024-06-15 18:19:52 +05:30
2024-07-15 11:51:44 +04:00
2024-09-11 17:16:06 +01:00
2023-12-15 13:45:03 +01:00
2024-02-27 10:44:53 +05:30
2023-08-09 14:20:51 +02:00
2023-12-15 13:45:03 +01:00
2024-03-25 09:01:46 +00:00
2024-01-10 10:50:13 +00:00
2024-05-27 14:09:17 +02:00
2023-12-15 13:45:03 +01:00
2023-12-15 13:45:03 +01:00
2023-12-15 13:45:03 +01:00
2023-08-09 14:20:51 +02:00
2023-10-30 16:44:22 +00:00
2023-08-09 14:20:51 +02:00
2023-08-09 14:20:51 +02:00
2024-03-25 09:01:46 +00:00
2023-08-09 14:20:51 +02:00
2023-08-09 14:20:51 +02:00
2023-08-09 14:20:51 +02:00
2023-08-09 14:20:51 +02:00
2023-08-09 14:20:51 +02:00
2023-08-09 14:20:51 +02:00
2023-12-15 13:45:03 +01:00
2023-12-15 13:45:03 +01:00
2024-07-15 11:51:44 +04:00
2024-07-15 11:51:44 +04:00
2024-10-21 09:39:52 -07:00
2024-10-21 09:39:52 -07:00
2024-10-21 09:39:52 -07:00
2024-07-15 11:51:44 +04:00
2024-07-15 11:51:44 +04:00
2024-07-28 11:26:29 +04:00
2024-01-16 21:54:58 -08:00
2024-10-28 09:04:19 +01:00
2024-07-26 08:43:30 +02:00
2024-07-30 18:18:16 +04:00
2024-10-16 12:26:01 -04:00
2022-11-28 11:51:36 -05:00
2024-07-01 15:26:09 +02:00
2024-10-10 14:01:49 +04:00
2024-10-10 14:01:49 +04:00
2024-10-21 09:39:52 -07:00
2024-01-16 21:54:58 -08:00
2024-07-15 11:51:44 +04:00
2024-07-25 19:21:28 +02:00
2024-10-08 09:41:24 -04:00
2024-10-08 09:41:24 -04:00
2024-07-15 11:51:44 +04:00
2024-01-26 09:56:06 -06:00
2020-07-23 20:56:19 +02:00
2024-04-24 08:52:25 +02:00
2024-10-21 09:39:52 -07:00
2024-10-21 09:39:52 -07:00
2024-10-21 09:39:52 -07:00
2024-09-04 18:54:07 +05:30
2024-07-15 11:51:44 +04:00
2023-08-22 14:25:31 +01:00
2024-08-22 17:14:53 +01:00
2024-10-16 09:43:16 +02:00
2024-05-06 14:07:11 +05:30
2024-05-06 14:07:11 +05:30
2024-02-27 10:44:53 +05:30
2024-02-27 10:44:53 +05:30
2023-07-31 12:15:39 +05:30
2024-06-15 18:19:52 +05:30
2023-07-31 12:15:39 +05:30
2023-07-31 12:15:39 +05:30
2023-07-31 12:15:39 +05:30
2024-08-07 10:34:09 -07:00
2024-01-16 21:54:58 -08:00
2024-08-01 09:42:04 +04:00
2024-10-02 12:08:21 +04:00
2024-10-02 12:08:17 +04:00
2024-09-25 13:02:29 +02:00
2024-06-14 19:14:19 +02:00
2023-07-04 12:22:38 +01:00
2024-03-29 12:41:47 -04:00
2024-10-15 08:23:05 +01:00
2024-07-23 13:59:15 +05:30
2024-10-21 09:39:52 -07:00
2024-07-25 19:21:28 +02:00
2024-04-16 02:36:12 -04:00
2024-07-23 13:59:15 +05:30
2024-10-15 08:23:05 +01:00
2024-06-14 19:14:19 +02:00
2024-07-15 11:51:44 +04:00
2024-07-15 11:51:44 +04:00
2024-06-22 00:40:59 +02:00
2024-01-16 21:54:58 -08:00
2022-05-18 11:04:33 +01:00
2024-09-04 17:37:34 +08:00
2024-06-03 10:42:10 +05:30
2023-07-04 12:22:38 +01:00
2024-09-11 15:04:55 +02:00
2024-07-23 13:59:15 +05:30
2024-04-16 02:36:12 -04:00
2024-09-11 15:04:55 +02:00
2024-06-03 10:42:10 +05:30
2024-09-24 11:08:41 +04:00
2024-07-23 13:59:15 +05:30
2024-10-21 09:39:52 -07:00
2024-10-21 09:39:52 -07:00
2024-10-21 09:39:52 -07:00
2024-10-21 09:39:52 -07:00
2024-10-21 09:39:52 -07:00
2024-10-21 09:39:52 -07:00
2024-10-21 09:39:52 -07:00
2024-10-21 09:39:52 -07:00
2024-10-21 09:39:52 -07:00
2024-10-21 09:39:52 -07:00
2024-06-03 10:42:10 +05:30
2023-07-04 12:22:38 +01:00