# Clang/P2996

Clang/P2996 is a fork of the [llvm/llvm-project](https://github.com/llvm/llvm-project) implementing experimental support for ISO C++ (WG21) proposal [**P2996**](https://wg21.link/p2996) (_Reflection for C++26_) in the `clang` compiler front-end. This project was initiated by a passionate group of C++ enthusiasts in [Bloomberg's Engineering department](https://www.TechAtBloomberg.com).

The Clang/P2996 fork is highly experimental; sharp edges abound and occasional crashes should be expected. Memory usage has not been optimized and is, in many cases, wasteful. **DO NOT use this project to build any artifacts destined for production.**

That said, this project represents the most complete implementation of P2996 to date. It is possible to use Clang/P2996 to build nontrivial reflection-heavy programs. We encourage all interested parties to try it out, and we welcome any pull requests or feedback.

The intent of this project is to continue tracking changes to P2996 as it makes its way through WG21's stages of development. That said, this is an aspiration and not a promise. The project is provided "as is." Subsequent development will be on a "best effort" basis, and we offer no guarantees as to its continued maintenance.


## Menu

- [Rationale](#rationale)
- [Acknowledgments](#acknowledgments)
- [Quick start](#quick-start)
- [Implementation status](#implementation-status)
  - [Incomplete features](#incomplete-features)
  - [P2996 test cases](#p2996-test-cases)
- [License](#license)
- [Code of conduct](#code-of-conduct)
- [Implementation design notes](#implementation-design-notes)
  - [Representing reflections](#representing-reflections)
  - [Reflect expressions](#reflect-expressions)
  - [Splices](#splices)
  - [Metafunctions](#metafunctions)
  - [Tracking source locations](#tracking-source-locations)
  - [Name mangling](#name-mangling)
- [Update history](#update-history)


## Rationale
The project's primary goal is to explore the implementation feasibility of P2996 in a major open source compiler. Through this, we hope to also provide an environment in which C++ enthusiasts can experiment with additional reflection features not already proposed by P2996. Finally, we aim to discover and raise awareness of possible concerns with bringing P2996 to `clang` (i.e., proposed features that may not fit easily within the existing architecture).


## Acknowledgments
This project owes an enormous debt to the historical [paper/p2320](https://github.com/lock3/meta/tree/paper/p2320) LLVM fork previously developed by Andrew Sutton and Wyatt Childers from their time at Lock3 Software. Their work not only provided our initial roadmap for implementing reflection in `clang`, but also taught us much about `clang`'s internals -- none of the initial Clang/P2996 developers had any prior development experience with `clang` (or indeed, any modern C++ compiler).

The initial implementation of Clang/P2996 was led by Dan Katz, with significant contributions from Georgi Koyrushki, Mark Sciabica, Sergei Murzin, and Kay Hicketts. All are engineers at Bloomberg, and they are grateful to their organization for supporting this work.


## Quick start
The upstream LLVM project provides an excellent [Getting Started](https://llvm.org/docs/GettingStarted.html) guide with instructions for building `clang` and `libc++`. After building both of these targets, support for reflection can be enabled in Clang/P2996 by compiling with both the `-std=c++26` and `-freflection` flags.

The following additional experimental features can be enabled on top of `-freflection`:
- Parameter reflection as proposed by [P3096](https://wg21.link/p3096) (`-fparameter-reflection`).
- Partial support for expansion statements as proposed by [P1306](https://wg21.link/p1306) (`-fexpansion-statements`). Note that expansions over constexpr ranges are not supported.
- Consteval blocks as proposed by [P3289](https://wg21.link/p3289) (`-fconsteval-blocks`).
- Newly proposed reflection syntax from [P3381](https://wg21.link/p3381) (`-freflection-new-syntax`).

For convenience, a unified `-freflection-latest` flag enables all of these features. Note that this fork does not implement any features proposed by P3294 ("Code Injection with Token Sequences").


## Implementation status
At present, Clang/P2996 supports:

* Reflection and splicing of all entities supported by P2996 (i.e., types, functions, variables, class members, templates, namespaces, constant values)
* All metafunctions proposed by P2996
* [P3096](https://wg21.link/p3096) metafunction extensions for function parameters (enabled with `-fparameter-reflection`)
* [P1306](https://wg21.link/p1306) expansion statements over _expansion-init-lists_ and _destructurable expressions_
* [P3289](https://wg21.link/p3289) `consteval` blocks
* [P3491](https://wg21.link/p3491) define_static_{string,object,array} 

Any implemented language or library extensions that are _not_ candidates for inclusion in P2996 are enabled by separate feature flags (as documented above). Any behaviors divergent from what is described by P2996, unless implemented experimentally for possible adoption by P2996, should be considered bugs.

### Incomplete features
Nearly all of P2996 is supported. We make an effort to keep the [issue tracker](https://github.com/bloomberg/clang-p2996/issues) updated with all known bugs, noncomformant behaviors, and missing features.

It has been our goal to provide a working compiler capable of building executables, but it has not been a non-goal to support the full family of features and tools offered by `clang` (e.g., AST output, `clang-tidy`, etc). Several shortcuts were therefore taken when it comes to defining things like the formatting of a reflection in a text dump. We are open to pull requests implementing such things, but are otherwise content to wait until P2996 is further along in the WG21 process.

### P2996 test cases
A significant number of tests have been written for this project, covering both the reflection and splicing operators proposed for the core language and the various metafunctions proposed for the Standard Library (found [here](/clang/test/Reflection/) and [here](/libcxx/test/std/experimental/reflection/) respectively). Among these tests are many of the examples from the P2996 paper (e.g., "_Parsing Command-Line Options II_", "_Compile-Time Ticket Counter_", and "_Emulating Typeful Reflection_"). We expect for this body of tests to continue to grow, and hope that it may furthermore assist validation of future implementations of P2996.

At this time, our test cases primarily verify correct uses of P2996 facilities; that is, we have not yet written tests to validate expected diagnostics for ill-formed programs. As such, this is probably an area having a nontrivial number of bugs and crashes waiting to be discovered.


## License
Following the example of the upstream LLVM project, new code developed for Clang/P2996 is licensed under Apache 2.0 with LLVM extensions ([LICENSE.TXT](/LICENSE.TXT)).


## Code of conduct
The Clang/P2996 project has adopted a [Code of Conduct](https://github.com/bloomberg/.github/blob/main/CODE_OF_CONDUCT.md). If you have any concerns about the Code, or behavior which you have experienced in the project, please contact us at opensource@bloomberg.net.


# Implementation design notes
Below are some high level notes about our implementation of P2996, along with some thoughts regarding certain "sharp edges" of the proposal as it relates to `clang`.

## Representing reflections
P2996 proposes a _value-based_ reflection model, whereby reflections are encoded as opaque scalar values usable during translation. The evaluation of an expression can compute a reflection, so it becomes important for an `APValue` to be capable of representing a reflection. A new `APValue::Reflection` kind is introduced for this purpose.

Most reflections are internally represented as a `void *` tagged with an enum value identifying the kind of entity reflected (similar to e.g., `TemplateArgument`). The pointer identifies one of:

* A type (represented by a `QualType`)
* A declaration of an evaluatable entity (represented by a `ValueDecl *`)
* A template (represented by a `TemplateName`)
* A namespace (represented by a `Decl *`)
* A base class specifier (represented by a `CXXBaseSpecifier *`)
* A description of a hypothetical data member, for representing the result of a call to `std::meta::data_member_spec` (represented by a `TagDataMemberSpec *`).

Representing all of these kinds of reflections is straightforward. Less straightforward is the representation of _values_ and _objects_, the natural representations of which are _also_ `APValue`s. The puzzling situation emerges in which an `APValue` must sometimes be able to hold a reflection, but a reflection must sometimes also be able to hold an `APValue`.

Our initial implementation of P2996 used a separate `ReflectionValue` class to model a reflection, and made `ReflectionValue` one of the several different "kinds" of structs that might be held by an `APValue`. When a reflection of a value or object was required, we stored the reflected result in a separate dynamically allocated `APValue`.

We eventually settled on a more efficient, if more invasive, change: We merged the `ReflectionValue` class into the `APValue` class, and added a `ReflectionDepth` "counter" to the representation of every `APValue`. This reduces the task of "taking a reflection of a value" to little more than incrementing its "reflection depth". The details are slightly more involved, but this model requires no dynamic allocations to represent any reflections.


## Reflect expressions
We generally found parsing reflect expressions (e.g., `^int`) to be an easier problem than parsing splices. An expression of the form `^E`, where `E` is a named entity or a constant expression, is parsed as a `CXXReflectExpr` with the operand represented by a `APValue` data member.

### Reflection contexts
The operand of a reflect expression is unevaluated, but we found that parsing reflections follows slightly different rules from other unevaluated contexts. Examples here include allowing direct reference to class members without taking their address (e.g., `^S::fn<int>`) and resolving a namespace to a `NamespaceAliasDecl` without dealiasing.

To implement these special rules, we introduced a new `ExpressionEvaluationContext::ReflectionContext` enum value, together with a `Sema::isReflectionContext()` method for checking whether parsing is taking place within the context of a reflect expression.


## Splices
While the act of reflecting over an entity always produces an expression, the act of splicing a reflection might produce:
* A type (i.e., `QualType`)
* A reference to an entity or constant expression (e.g., `ConstantExpr *`)
* A namespace
* A template name

It thus becomes clear that a _splice_, without further context, can only be given a coherent definition at the grammatical level; its semantics are dependent on both the context and the evaluation of the spliced expression. 

If the operand of the splice can be evaluated at parse time, then this presents little difficulty. However, if the splice is value-dependent on a template parameter (e.g., the common use case of splicing a reflection received as a template argument), then it may not be possible to determine what the splice "is" until template substitution. P2996 addresses this in a familiar way, requiring that all such splices be preceded with e.g., a leading `typename` to disambiguate the class of entity; an expression is assumed in the absence of such a disambiguator.

To help with such cases, we decompose a splice `[:R:]` into two objects:
1. A `CXXSpliceSpecifierExpr` representing the expression whose evaluation will yield the `APValue` of the entity being spliced, and
2. An AST object holding a pointer to the `CXXSpliceSpecifierExpr` (e.g., `ReflectionSpliceType` for types, `CXXExprSpliceExpr` for expressions).


### Parsing splices of expressions
Splices of expressions usually appear in the context of a _cast-expression_ (e.g., `[:R:] + 13`), but they can also appear on the right-hand side of a _member access expression_. The existing machinery in `SemaMemberExpr.cpp` assumes that the member being accessed has not yet been resolved and performs name lookup to find a `Decl`. However, in the case of an expression like `a.[:R:]`, we already have the `Decl` from a `APValue` obtained through evaluation of `R`. Therefore, we introduce overloads of functions like `Sema::ActOnMemberAccessExpr` to support the case where the right-hand of the member access is a splice. A special `CXXDependentMemberSpliceExpr` is introduced to cover cases when the reflection is dependent on a template parameter, in order to "hold" the underlying `CXXSpliceSpecifierExpr` until template substitution.


### Nested-name-specifiers containing splices
Splices of types and namespaces can appear as the (possibly dependent) leading component of a nested-name-specifier. This is interesting in cases such as `typename [:R:]::member`, for which the `typename` keyword clarifies that `member` will resolve to a type, but tells us nothing about what sort of entity `[:R:]` might be. We address this by adding a new `SpecifierKind::IndeterminateSplice` kind to `NestedNameSpecifier`, which holds a `CXXSpliceSpecifierExpr` to be rebuilt thereafter into a namespace or type during template substitution.


### Splicing namespaces
The most recent revisions of P2996 remove the `namespace [: :]` syntax and only allow splices of namespaces in the following contexts:
* Using directives (e.g., `using namespace [:R:];`)
* Namespace alias definitions (e.g., `namespace A = [:R:];`)
* Within nested-name-specifiers (e.g., `[:R:]::Member`)

Use of splices in namespace definitions (e.g., `namespace [:R:] { ... }`) is neither supported by this compiler nor proposed by P2996.

Since both using directives and namespace aliases may appear in templated contexts, it becomes possible to have a namespace name that is dependent on a template parameter. Support for this has not yet been implemented in this compiler.


## Metafunctions
The library functions proposed by P2996 for producing and operating on reflections are coloquially referred to as _metafunctions_; as a general rule, these functions cannot be implemented without compiler intrinsics. Drawing inspiration from the earlier `paper/p2320` implementation from Lock3, we used a single `__metafunction` intrinsic to implement a library of 60+ metafunctions.

An expression of the form

    __metafunction ( ID, expr0, ..., exprN )
is parsed as a `CXXMetafunctionExpr`. The `ID` argument is expected to be a non-dependent enum constant identifying the metafunction, and is evaluated at parse time. The remaining arguments are parsed as expressions and held by the `CXXMetafunctionExpr` until constant evaluation time. The only validation performed at parse time is to check that the number of arguments is valid for the metafunction identified by `ID`.

Corresponding to each metafunction is a global instance of the `Metafunction` class, which provides:
* The `QualType` of the value resulting from its evaluation;
* How many arguments it accepts; and
* A `Metafunction::evaluate` member function used to evaluate the metafunction for some given inputs.

The `Metafunction::Lookup` static member function accepts a metafunction `ID` and returns a pointer to the corresponding `Metafunction` instance.

The core of the `VisitCXXMetafunctionExpr` function, implemented in `ExprConstant.cpp` to evaluate a metafunction, invokes a callback held by the `CXXMetafunctionExpr`. In turn, this calls the appropriate `Metafunction::evaluate` function with the argument expressions held by the `CXXMetafunctionExpr`. This approach works well for our experimental purposes, but interacts poorly with precompiled headers and C++20 modules due to the unserializable nature of the evaluation callback, as detailed below.


### The `Sema` problem
Many of the metafunctions proposed by P2996 are "observational", in the sense that they query properties from the AST while leaving it unchanged. These include `is_type`, `is_public`, `parent_of`, and many others.

However, several of the most powerful proposed metafunctions are _generative_: their evaluation can **modify** the state of the AST, producing side effects during constant evaluation that are not otherwise posssible in C++ today. It is exactly these side-effects that make some of the most interesting examples from the P2996 paper possible in the first place (e.g., "[Compile-time ticket counter](https://wg21.link/p2996r2#compile-time-ticket-counter)").

Although AST nodes can be directly constructed using only `ASTContext`, doing so elides the semantic analysis of whether the would-be entities "make any sense". This is a recipe for compiler crashes and assertion failures, since the resulting function calls, class definitions, and template instantiations will be constructed regardless of violations of invariants that later stages of compilation (e.g., CodeGen) expect semantic analysis to have already enforced. We concluded that metafunctions like `define_class`, `reflect_invoke`, and `substitute` cannot be coherently implemented in `clang` without access to the `Sema` object.

This presents a problem for `clang` though, as its physical design is carefully architected to specifically disallow this. All constant evaluation is implemented by `AST/ExprConstant.cpp`, and there is no means of accessing the `Sema` object from this file. Indeed, referencing any part of the `Sema` layer from the `AST` layer seems contrary to the deliberate design of the `clang` codebase.

The result is an apparent tension between the architecture of `clang` and P2996, or any other paper allowing the constant evaluation of a function to produce side effects in the AST. Between Lock3's implementation and Clang/P2996, two different approaches have been tried to address this tension.

#### Lock3/P2320: The `EvalContext` Model
The Lock3 implementation defines a `ReflectionCallback` struct presenting a polymorphic interface to a narrow set of features provided by `Sema` (e.g., "evaluate this type trait"). This interface is defined in `/AST/`, but implementated as `ReflectionCallbackImpl` in `/Sema/`. An `EvalContext` struct, essentially an `<ASTContext, ReflectionCallback *>` pair, provides the "context" needed to evaluate any compile-time expression.

`EvalContext` replaces `ASTContext` in the interface of the `Expr::Evaluate*` family of functions. Evaluations like:
```cpp
Expr::EvalValue Result;
E->EvaluateAsRValue(Result, S.Context, /*InConstantContext=*/true);
```

are then replaced with calls like:
```cpp
Expr::EvalValue Result;
Expr::EvalContext Ctx(S.Context, S.GetReflectionCallbackObj());
E->EvaluateAsRValue(Result, Ctx, /*InConstantContext=*/true);
```

This is a fairly invasive change, requiring modifications to many different files (especially `ExprConstant.cpp`). It also has the drawback of allowing evaluation to proceed even if the `ReflectionCallbackImpl` object isn't readily available from `Sema`, i.e.,
```cpp
Expr::EvalContext Ctx(S.Context, /*ReflectionCallback=*/nullptr);
```

In some cases, this can result in compiler crashes, especially if a metafunction is evaluated in a context that had no means of accessing the `Sema` object. These can almost certainly be fixed, but it will require more discipline with respect to which code is allowed to evaluate expressions.

#### Clang/P2996: The "Evaluation Callback" Model
Our implementation's approach is to bind a reference to the `Sema` object to the "evaluation" callback held by the `CXXMetafunctionExpr`. This callback takes only objects from the `AST` layer as arguments, but dispatches to the `Metafunction::evaluate` function from the `Sema` layer. The invocation from `ExprConstant.cpp` also passes an "Evaluator" callback, which allows the `Sema` metafunction implementation to evaluate an expression in the existing constant evaluation context without access to the `EvalInfo` object. This is important, for instance, to ensure that an lvalue referencing an object dynamically allocated earlier in the constant expression can be evaluated by the metafunction.

While this requires less invasive changes than the `EvalContext` model, a new problem is introduced: the "evaluation callback" owned by the `CXXMetafunctionExpr` cannot be serialized or deserialized between processes. Since `clang` stores the serialized AST as a part of its representation of C++20 modules and precompiled headers, our proposed implementation of `CXXMetafunctionExpr` breaks both of these features. Even if we were to try to serialize the callback state, the `Sema` object referred to by the callback likely no longer exists at deserialization time: after all, the process compiling a module is likely different from the process importing it.

Given that the `ASTReader` and `Sema` objects are owned by the same `CompilerInstance`, one may be able to construct the `ASTReader` with a reference to `Sema`, thereby allowing it to reconstruct metafunction evaluation callbacks that have references to the "new" `Sema` instance belonging to the deserializing process. We have not attempted this change, and aren't fully convinced that such a direction would be desirable.

#### Retrospective: The `Sema` Problem
We consider this problem not fully solved. On the one hand, the `EvalContext` model avoids the storage of a non-serializable callback in the AST, obviating the issues with C++20 modules. On the other hand, it restricts the contexts from which an expression can be safely evaluated (i.e., those with access to either `Sema` or to an `EvalContext` constructed with `Sema`). The Clang/P2996 model makes it "easier" to evaluate an expression by "hiding" the `Sema` reference in the evaluation callback of the `CXXMetafunctionExpr`, but this makes (de)serialization more tricky.

The takeaway seems to be: in the presence of the changes proposed by P2996, constant folding can no longer be performed without `Sema`. Either of the above approaches can probably be made to work in a pinch (assuming P2996's eventual adoption by WG21), but some third design approach that more fully re-imagines the relationship between `Sema` and the AST might be better yet.


### Retrospective: Metafunction Design
Almost all of `clang`'s machinery for evaluating constant expressions lives in `ExprConstant.cpp`, so our decision to implement P2996 metafunctions in a separate `/Sema/Metafunctions.cpp` deviates from existing practice. This model nevertheless has a few nice fatures. Implementing all such 60+ functions in an isolated file has helped delineate our "experimental" code from existing production code. More importantly, the model succeeds in maintaining the separation of `Sema` code from `AST`.

On the other hand, implementing the metafunctions outside of `ExprConstant.cpp` means that they lack access to much of the constant evaluation state (e.g., the callstack). For metafunctions in need of access to this state, we use the evaluation of synthesized expressions as a "communication channel" with the constant evaluator. In some cases (e.g., `value_of`, `is_accessible`), this required the invention of a novel `Expr` type just to pipe the required state from the constant evaluator back to the metafunction implementation (see `LValueValueOfExpr` and `StackLocationExpr`). While this does work, it leads to wasteful allocation of AST nodes by the compiler. This increased memory usage might become noticeable when compiling large translation units, or several translation units in parallel.


### The difficulty of `define_class`
The family of functions used to define a tag type (e.g., `ActOnTag`, `ActOnStartDefinition`, `ActOnStartMemberDeclarations`) provide a sort of "declarative language" that is used by the `Parser` as it encounters class member definitions. Our implementation of `define_class` carefully reuses this machinery, while constructing `Scope`s and `DeclContext`s to ensure the resulting definition is a redeclaration of the provided incomplete type (and not just a new and unrelated type). Special cases are needed for when the type being completed is a specialization of a class template. The implementation ends up feeling like a sloppy and fragile reimplementation of the `Parser`, rather than a clean set of calls to existing machinery in `Sema`.


## Tracking source locations
Clang's ability to report accurate error messages requires it to track the locations of various tokens (e.g., expression locations, `(` and `)` for function calls, `<` and `>` for template specializations). When generating AST nodes from metafunctions or when splicing reflections, answering the question of where certain tokens are becomes awkward since we're synthesizing nodes from token sequences that are "shaped differently" than otherwise expected. While not usually a blocker in practice, the disharmony between the desire of `clang` to individually track the location of many individual tokens, and the desire of metaprogramming frameworks to synthesize a variety of AST nodes that don't easily map to the same tokens expected by `clang`, is worth noting.


## Name mangling
As P2996 proposes mechanisms for _Static_ Reflection, all of the business around reflections and splices and metafunctions has mostly wrapped up by the time CodeGen begins. The most notable exception is the need to mangle the names of templates having a `std::meta::info` value as an argument. As noted previously, this is a rather important use-case: since a reflection can only be spliced if it's a constant expression, a function that wants to splice a reflection that was received as an argument must receive it as a template argument. Some of the more powerful metafunctions (e.g., `substitute`, `reflect_invoke`, `value_of`) reduce the frequency with which reflections must be passed as template arguments, but it remains necessary in some cases.

Given a specialization of `template <std::meta::info R> fn()`, the compiler must be able to mangle a representation of any entity that can be reflected by `R`. This includes the full class of values of literal types - an expansion from the values of _structural_ types that can already appear as non-type template arguments. This raises questions around whether all (or which) such values should be well-formed template arguments - or perhaps, if all such values should be allowed, but some should cause the associated specialization to have internal linkage. These are questions that we expect to see clarified as P2996 advances through WG21.