clice/openspec/changes/explore-improve-folding-range-support/design.md

## Context

`clice` currently implements folding ranges in `src/feature/folding_ranges.cpp`. The implementation is primarily an AST visitor with extra handling for conditional compilation and `#pragma region` data from `CompilationUnitRef::directives()`. It already covers many structural folds that clangd does not currently expose, such as namespaces, records, function parameter lists, lambda captures, call argument lists, access-specifier sections, and initializer lists.

The request path is currently split across:

- `src/feature/folding_ranges.cpp` for collection and rendering
- `src/server/master_server.cpp` for request plumbing and capability advertisement
- generated `kota` LSP protocol types for request/response shapes
- `tests/unit/feature/folding_range_tests.cpp` for unit coverage

That split reveals three immediate shortcomings:

- the current collector has no comment path at all
- folding-specific client capabilities such as `lineFoldingOnly`, `rangeLimit`, and `collapsedText` are not threaded through the request path
- directive-related tests are mostly placeholders and do not assert important behavior

The comparison target for this exploration change should be fixed and versioned. At tag `llvmorg-21.1.8`, clangd's folding behavior is centered on `clang-tools-extra/clangd/SemanticSelection.cpp`, with request plumbing in `ClangdServer.cpp` and `ClangdLSPServer.cpp`, protocol types in `Protocol.h` and `Protocol.cpp`, and regression coverage in `test/folding-range.test` plus `unittests/SemanticSelectionTests.cpp`. Those files have been downloaded into `openspec/changes/explore-improve-folding-range-support/reference/clangd/llvmorg-21.1.8/`, and the side-by-side analysis lives in `comparison.md`.

Compared with that clangd baseline, the current gap is clear:

- clangd already has behavior that `clice` still lacks:
  - multiline comment folding
  - contiguous `//` comment-group folding
  - `lineFoldingOnly` rendering behavior wired from client capabilities into folding generation
  - consistent use of standard public folding kinds
  - a more complete and assertion-backed folding-range test matrix
- `clice` already has behavior that clangd does not:
  - richer AST-structure folding
  - `#pragma region` and some conditional-compilation folding
  - `collapsedText`
- `clice` still has obvious opportunities that are not fully implemented yet:
  - fully closing the last `#if/#elif/#else` branch at `#endif`
  - folding inactive branches
  - folding multiline macro definitions
  - grouping contiguous `#include` / `import` blocks
  - capability-aware `kind` and `collapsedText` rendering

In addition, the downloaded clangd source confirms that clangd still does not implement PP conditional regions, include grouping, or access-specifier folding in `SemanticSelection.cpp`; those are explicitly left as FIXME items upstream. The real parity target is therefore narrower than "match everything clangd does": comments, line-only rendering, standard kinds, and test discipline are the confirmed baseline gaps. Everything around directive groups, inactive branches, and richer structural categories remains a clice-specific extension opportunity.

## Goals / Non-Goals

**Goals:**

- Download a focused clangd reference set from `llvmorg-21.1.8` into this change directory and use it as the explicit comparison baseline for this branch.
- Preserve `clice`'s current advantage in AST-structure folding instead of regressing to clangd's much narrower block-only baseline.
- Fill the high-value baseline gaps that clangd already covers, especially multiline comments and `lineFoldingOnly`.
- Turn preprocessor metadata into a differentiating `clice` capability covering conditional branches, macro definitions, and include/import grouping.
- Make folding-range output respect client capabilities with predictable fallback behavior.
- Lock behavior down with unit and integration tests across AST, comments, preprocessor handling, and protocol negotiation.

**Non-Goals:**

- Import clangd implementation code directly into `clice` production paths or make the build depend on the downloaded reference files.
- Achieve byte-for-byte or range-for-range parity with clangd in this change.
- Add fine-grained folding for every C++ syntax detail such as template parameter lists, requires-clauses, or attribute arguments before their value is proven.
- Introduce editor-specific behavior that only exists to satisfy one frontend.
- Add cross-file or index-backed folding behavior.

## Decisions

### 1. Download a focused clangd reference set into the change directory before implementation work

The branch should first download a small, reviewable set of clangd's folding-related sources from tag `llvmorg-21.1.8` into `openspec/changes/explore-improve-folding-range-support/reference/clangd/llvmorg-21.1.8/`. The downloaded set should include the implementation, request plumbing, protocol types, and relevant tests that explain folding behavior, rather than the whole LLVM tree.

Why:

- it creates a stable review artifact for this exploration branch
- later implementation work can point at local upstream code instead of external URLs
- it keeps the eventual runtime change honest about what is parity work and what is a clice-specific extension
- it avoids adding a repo-level vendor location for a one-branch study artifact

Alternative considered:

- Put the files under `third_party/`. Rejected because this is an exploration artifact, not a production dependency.

### 2. Split the folding-range pipeline into collection, normalization, and rendering

The current implementation mixes "how a range is discovered" with "how it is emitted as LSP". The new design separates this into three layers:

- collection: produce internal `RawFoldingRange` entries from AST, comment scanning, and preprocessor metadata
- normalization: sort, deduplicate, validate, and reconcile nested or overlapping ranges
- rendering: decide line/column boundaries, `kind`, and `collapsedText` based on client capabilities

Why:

- `lineFoldingOnly`, `collapsedText`, and standards-compatible kind downgrading are rendering concerns and should not pollute collection logic
- comments, macros, and include/import groups do not naturally belong inside the AST visitor
- future range limiting or prioritization should also live in normalization/rendering instead of collector code

Follow-up discussion narrows this design point: the existing `RawFoldingRange` model is finished for the current pipeline work and should not be redesigned here. The missing part is an explicit options object, passed as `Opts`/`FoldingRangeOptions`, that lets callers configure renderer behavior such as `line_folding_only`.

Alternative considered:

- Keep generating final LSP ranges directly inside the visitor. Rejected because capability negotiation and multi-source collection will keep making the function larger and harder to test.

### 3. Keep rich internal categories, but only promise standard-compatible public kinds

Internally, the implementation may still distinguish namespace, class, function body, macro definition, conditional branch, and similar categories so tests, prioritization, and `collapsedText` selection remain precise. However, public LSP output should default to standard kinds only:

- comment folds -> `comment`
- contiguous include/import groups -> `imports`
- all other structural and preprocessor folds -> `region`

If some client later proves it needs clice-specific kinds, that can be evaluated separately. This change does not make non-standard kind strings part of the compatibility contract.

Why:

- many current custom strings will not be understood by clients and do not produce stable UI semantics
- the real differentiator is what `clice` can fold, not the literal `kind` label
- once public kinds are standardized, `collapsedText` and range boundaries become the primary user-visible expression

Alternative considered:

- Continue exposing all custom kinds directly. Rejected because that leaves client compatibility up to luck rather than protocol design.

### 4. Use the downloaded clangd files as a behavior reference, not as a direct implementation template

clangd's folding logic is text- and token-oriented rather than AST-oriented. `clice` should study the upstream behavior to match the useful parts, but it should not force its own collector architecture to look like clangd's when `CompilationUnitRef::directives()` and the existing AST visitor provide better raw data.

Why:

- parity should be measured at the behavior boundary, not by mirroring file structure
- `clice` already has data sources that clangd does not, especially for directive metadata
- this keeps the change focused on correctness and value, not on source-level imitation

Alternative considered:

- Rewrite `clice` folding collection to resemble clangd's text parser closely. Rejected because that would discard existing strengths without a clear benefit.

### 5. Implement comment folding through lexical/source scanning, not AST

Multiline comments are handled independently in clangd's pseudo-parser path, and `clice` should do the same. The design adds a comment collector that scans the main-file source or token stream directly:

- fold multiline `/* ... */` block comments
- fold contiguous `//` comment groups
- do not fold single-line comments
- preserve source spans that let the renderer adjust closing boundaries for `lineFoldingOnly` mode

Why:

- comments are not AST structure, so trying to derive them from AST produces fragile behavior
- lexical scanning naturally handles adjacent-comment grouping and block-comment boundaries

Alternative considered:

- Only support block comments. Rejected because clangd already demonstrates that contiguous `//` comment groups are a useful folding case.

### 6. Rework preprocessor folding around complete branch blocks instead of the current half-open stack

Today `collect_condition_directives()` only closes the previous branch when it sees `#else`, but when it sees `#endif` it only pops the stack and does not emit a folding range for the final `#if/#elif/#else` branch. As a result, `#if` folding is incomplete.

The new design treats conditional compilation as an explicit branch-group model:

- maintain the ordered branch chain for each `#if` group
- allow every branch to close at the next `#elif`, `#else`, or `#endif`
- distinguish active and inactive branches
- allow inactive branches to produce region folds, optionally with distinct `collapsedText`

Why:

- this is the minimum sound model needed to fix the current logical gap
- `Condition::ConditionValue` already records true/false/skipped state and can drive inactive-branch folding directly

Alternative considered:

- Patch only the `#endif` closing case. Rejected because nested conditions, inactive branches, and range ordering would remain structurally weak.

### 7. Add dedicated directive-based collectors for macros and include/import groups

`clice` already collects:

- `directive.macros`
- `directive.includes`
- `directive.imports`

The new design therefore adds directive-based folding collectors for:

- multiline `#define` macro definitions, using continuation backslashes or stable definition ranges
- contiguous `#include` blocks, merged into a single `imports` folding range
- contiguous `import Foo;` / `import Foo:Bar;` module-import blocks, also emitted as `imports`

Why:

- the necessary data already exists in preprocessing metadata and does not require new AST modeling
- this is one of the easiest places for `clice` to provide value beyond clangd

Alternative considered:

- Leave include/import grouping for a later change. Rejected because the metadata already exists, the implementation cost is relatively low, and the editor-facing value is immediate.

### 8. Separate clangd parity capabilities from clice-only protocol improvements

This change should treat comment folding, `lineFoldingOnly`, and standard public kinds as clangd parity work. `collapsedText` gating and deterministic `rangeLimit` trimming remain clice-side protocol improvements. The downloaded clangd `Protocol.h` / `Protocol.cpp` reference does not expose `collapsedText`, so the design and tests should not imply that clangd already provides that capability.

Why:

- it keeps the comparison honest
- it allows reviewer discussion to separate "must match upstream baseline" from "valuable extra behavior"
- it keeps spec language compatible with LSP without overstating clangd

Alternative considered:

- Treat all capability work as a clangd parity gap. Rejected because clangd's known folding path does not establish that broader claim.

### 9. Folding-range output must be explicitly bound to client capabilities

The master server currently only advertises `foldingRangeProvider = true`, but it does not read or propagate folding-specific client capabilities. The new design requires the session to track at least:

- `lineFoldingOnly`
- whether `collapsedText` is supported
- optional `rangeLimit`

Capability state should be translated into a feature-layer options object before rendering. The initial option needed by the current discussion is:

```cpp
struct FoldingRangeOptions {
    bool line_folding_only = false;
};
```

The feature API should accept that options object separately from the source collector inputs, for example as `folding_ranges(unit, opts, encoding)`. Later protocol work can extend the same object for collapsed-text gating or range limiting without changing collectors.

Rendering rules:

- when `opts.line_folding_only = true`, only emit ranges that remain meaningful as line-based folds, adjusting end lines where necessary
- when the client does not support `collapsedText`, omit it
- when a `rangeLimit` is declared, trim results deterministically rather than arbitrarily

Alternative considered:

- Continue always returning exact columns and `collapsedText`. Rejected because that relies on client tolerance instead of following the protocol contract.
- Thread capability state into collectors directly. Rejected because it would reopen the raw model and collection contract even though line-only behavior is a renderer policy.

### 10. Organize tests by source category and protocol behavior

Tests will be split into two dimensions:

- source-category unit tests: AST structure, comments, conditional compilation, multiline macros, `#pragma region`, and include/import groups
- protocol-behavior tests: `lineFoldingOnly`, `collapsedText` support, public kind mapping, and range limiting

In particular, the current `tests/unit/feature/folding_range_tests.cpp` contains `Directive` and `PragmaRegion` cases that do not actually assert results. This change upgrades them into strong assertion-based tests.

Alternative considered:

- Rely mostly on manual editor validation. Rejected because folding details regress easily, especially for preprocessor handling and line-only rendering.

## Risks / Trade-offs

- [The downloaded clangd reference set could sprawl or become noisy in review] -> Mitigation: keep only the small folding-related file set needed for comparison under the change directory and record the exact URLs in `comparison.md`.
- [Client capabilities must flow from initialize state into request-time rendering] -> Mitigation: introduce a dedicated folding-options structure so session details do not leak broadly into the feature layer.
- [Inactive-branch and macro-definition ranges can be unstable around expansion locations] -> Mitigation: prefer spelling/main-file ranges and explicitly filter or special-case macro-expansion ranges when necessary.
- [Adding comments, macros, and include/import groups can increase the number of ranges quickly] -> Mitigation: implement stable sorting and `rangeLimit` trimming in the normalization layer.
- [Mapping public kinds back to standard values changes current metadata output] -> Mitigation: the folds themselves remain; the user-visible change is mostly in optional metadata, and tests plus change notes will make that explicit.
- [Multiple collectors may produce overlapping or duplicate ranges] -> Mitigation: normalize by source category and boundary rules so collectors do not amplify noise.

## Migration Plan

1. Download the focused clangd `llvmorg-21.1.8` folding reference files into `openspec/changes/explore-improve-folding-range-support/reference/clangd/llvmorg-21.1.8/`.
2. Record the confirmed clangd-vs-clice comparison in this change, including exact URLs, which behaviors are parity gaps, and which are clice-specific extensions.
3. Keep the existing `RawFoldingRange` data flow, add `FoldingRangeOptions` for `line_folding_only`, and add standard kind mapping.
4. Add the comment collector and assertion-backed tests for multiline comment folding.
5. Rewrite conditional-directive and `#pragma region` collection so `#if` branches close correctly through `#endif`.
6. Add multiline macro folding and grouped include/import collectors.
7. Wire folding client capabilities through initialize/request handling and add integration coverage.
8. Add `rangeLimit` trimming and regression cleanup after the new collectors are in place.

Rollback strategy:

- If the downloaded reference set becomes more distracting than useful, keep only the documented comparison notes and delete the change-local downloads before merging.
- If protocol negotiation proves unstable, keep the new collectors but temporarily disable outward behavior changes tied to `collapsedText` or `rangeLimit`.
- If a particular new fold category proves noisy, roll it back collector-by-collector instead of reverting the entire folding-range refactor.

## Open Questions

- Are `test/folding-range.test` and `unittests/SemanticSelectionTests.cpp` enough for ongoing comparison, or will later implementation work need more upstream folding-related tests?
- Should multiline macro folding cover only the macro body, or the full `#define NAME(...)` line plus body as one fold region?
- Should `rangeLimit` prioritize outer structure, top-of-file regions, or longer ranges when trimming results?
- For structural AST folds originating from macro expansion, should `clice` preserve current behavior or restrict itself to cases with stable spelling ranges only?