On Darwin, the --isysroot flag must also be specified. This
happens when either %flang or %flang_fc1 is expanded. As -fc1 must
be the first argument, %flang_fc1 must be used in tests, instead of
%flang -fc1.
Commas being optional in FORMAT statements, the tokenization of things
like 3I9HHOLLERITH is tricky. After tokenizing the initial '3', we don't
want to then take apparent identifier "I9HHOLLERITH" as the next token.
So the prescanner just consumes the letter ("I") as its own token in
this context.
A recent bug report complained that this can lead to incorrect results
when (in this case) the letter is a defined preprocessing macro. I
updated the prescanner to check that the letter is actually followed by
an instance of a problematic Hollerith literal.
And this broke two tests in the Fujitsu Fortran test suite that Linaro
runs, as it couldn't detect a following Hollerith literal that wasn't on
the same source line. We can't do look-ahead line continuation
processing in NextToken(), either.
So here's a second attempt at fixing the original problem: namely, the
letter that follows a decimal integer token is checked to see whether
it's the name of a defined macro.
In order to properly expose the Hollerith editing item in something like
FORMAT(3I9HHOLLERITH) as its own token, the tokenization routine in the
prescanner has special handling for digit strings followed by letters
("3I" above). This handler's effects are too broad, and prevent a macro
name from being recognized as such in a reported bug; make the test for
a hidden Hollerith more precise.
Fixes https://github.com/llvm/llvm-project/issues/121931.
A free form source line that begins with a macro should still be
classified as a source line, and have its continuation lines work, even
if the macro expands to an empty replacement.
Fixes https://github.com/llvm/llvm-project/issues/117297.
The character 'e' or 'd' (either case) shouldn't be tokenized as part of
a real literal during preprocessing if it is not followed by an
optionally-signed digit string. Doing so prevents it from being
recognized as a macro name, or as the start of one.
Fixes https://github.com/llvm/llvm-project/issues/115676.
When a fixed form source line begins with the name of a macro, don't
emit the usual warning message about a non-decimal character in the
label field. (The check for a macro was only being applied to free form
source lines, and the label field checking was unconditional).
Fixes https://github.com/llvm/llvm-project/issues/116914.
The preprocessor implements "defined(X)" and "defined X" in if/elif
directive expressions in such a way that they only work at the top
level, not when they appear in macro expansions. Fix that, which is a
little tricky due to the need to detect the "defined" keyword before
applying any macro expansion to its argument, and add a bunch of tests.
Fixes https://github.com/llvm/llvm-project/issues/114064.
When running fixed-form source through the compiler under -E, don't
aggressively remove space characters, since the parser won't be parsing
the result and some tools might need to see the spaces in the -E
preprocessed output.
Fixes https://github.com/llvm/llvm-project/issues/112279.
…INCLUDE lines
The compiler current treats an INCLUDE line as essentially a synonym for
a preprocessing #include directive. The causes macros that have been
defined at the point where the INCLUDE line is processed to be replaced
within the text of the included file.
This behavior is surprising to users who expect an INCLUDE line to be
expanded into its contents *after* preprocessing has been applied to the
original source file, with no further macro expansion.
Change INCLUDE line processing to use a fresh instance of Preprocessor
containing no macro definitions except _CUDA in CUDA Fortran
compilations and, if the original file was being preprocessed, the
standard definitions of __FILE__, __LINE__, and so forth.
Codes using traditional C preprocessors will sometimes put a keyword
macro name in a free form continuation line in order to get macro
replacement of part of an identifier, as in
call subr_&
&N&
&(1.)
where N is a keyword macro. f18 already handles this case, but not when
there is white space between the macro name and the following
continuation marker character '&'. Allow white space to appear.
Fixes https://github.com/llvm/llvm-project/issues/106931.
…tion
See new test. A #line (or #) directive after a line ending with & and
before its continuation shouldn't elicit an error about mismatched
parentheses.
Fixes https://github.com/llvm/llvm-project/issues/100073.
The prescanner checks lines that begin with a keyword macro name to see
whether they should be treated as a comment or compiler directive
instead of a source line. This fails when the potential keyword macro
name is extended with identifier characters via Fortran line
continuation. Disable line continuation during this check.
Support the predefined macro __TIMESTAMP__ as interpreted by GCC. It
expands to a character literal with the time of last modification of the
top-level source file in asctime(3) format, e.g. "Tue Jul 4 10:18:05
1776".
Don't treat !DIR$ or an active !$ACC, !$OMP, &c. as a comment when they
appear after a semicolon, but instead treat them as a compiler directive
sentinel.
…literals
To help port codes from compilers using pre-ANSI C preprocessors, which
didn't care much about context when replacing macros, support the
replacement of keyword macros in quoted character literals when (and
only when) the name of the keyword macro constitutes the entire
significant portion of a free form continuation line. See the new test
case for a motivating example.
Fixes https://github.com/llvm/llvm-project/issues/96781.
…ne continuation
Allow preprocessing directives to appear between a source line and its
continuation, including conditional compilation directives (#if, #ifdef,
&c.).
Fixes https://github.com/llvm/llvm-project/issues/95476.
…rective
Implement fixed-form line continuation when the continuation line is the
result of text produced by an #include or other preprocessing directive.
This accommodates the somewhat common practice of putting dummy or
actual arguments into a header file and #including it into several code
sites.
Fixes https://github.com/llvm/llvm-project/issues/78928.
…e produced by keyword macro replacement
When later initial keyword macro replacement will yield a line that is
not Fortran source, don't interpret "&" as a Fortran source line
continuation marker during tokenization of the line.
Fixes https://github.com/llvm/llvm-project/issues/82579.
Some applications like to use a CPP-style #include directive to pull in
a common list of arguments, dummy arguments, or COMMON block variables
after a free-form & line continuation marker. This works naturally with
compilers that run an actual cpp pass over the input before doing
anything specific to Fortran, but it's a case that I missed with this
integrated preprocessor.
…Warn()
Many warning messages were being emitted unconditionally. Ensure that
all warnings are conditional on a true result from a call to
common::LanguageFeatureControl::ShouldWarn() so that it is easy for a
driver to disable them all, or, in the future, to provide per-warning
control over them.
When prescanning a Fortran source file with preprocessing enabled in
free source form, interpret a line-ending backslash as a source line
continuation marker as a C preprocessor would. This usage isn't
completely portable, but it is supported by GNU Fortran and appears in
the source for FPM package manager.
I implemented legacy "token pasting" via line continuation for
call prefix&
&MACRO&
&suffix(1)
in a recent patch; this patch addresses the related cases
call prefix&
&MACRO&
&(1)
and
call &
&MACRO&
&suffix(1)
Fixes the latest https://github.com/llvm/llvm-project/issues/79590.
Ensure that #include FOO undergoes macro replacement. But, as is the
case with C/C++, continue to not perform macro replacement in a #include
directive with <angled brackets>.
Some Fortran codes use line continuation as a form of token pasting; see
https://github.com/llvm/llvm-project/issues/78797. This works in
compilers that run a C-like preprocessor and then apply line
continuation to its output; f18 implements line continuation during
tokenization and preprocessing, but can still handle this case.
In the rare case when an identifier is split across two or more
continuation lines, this patch allows its parts to be distinct
preprocessing tokens for the purpose of macro replacemnt. They (or their
replacement texts) can be effectively rejoined later as a single
identifier when the cooked character stream is tokenized in parsing.
Fixes https://github.com/llvm/llvm-project/issues/78797.
If DEFAULT_SYSROOT is not specfied when building flang, then the
-isysroot flag is needed to link binaries against system libraries
on Darwin. It's also needed when linking against a non-default
sysroot.
When a reference to a function-like macro begins during the rescanning
of the expansion of another macro but is not completed by the end of
that expansion, it is necessary to abort that rescanning of that
expansion and try again when more tokens can be acquired. (See the new
unclosed-FLM.F90 test case.) All other Fortran preprocessors to which I
have access can handle this situation.
During function-like macro expansion in a standard C/C++ preprocessor,
the macro being expanded is disabled from recursive macro expansion. The
implementation in this compiler's preprocessor, however, was too broad;
the macro expansion needs to be disabled for the "rescanning" phase
only, not for actual argument expansion.
(Also corrects an obsolete comment elsewhere that was noticed during
reduction of an original test case.)
At present, the prescanner emits an error if a source line or compiler
directive, after macro replacement or not, contains a token with a
non-Fortran character. In the particular case of the '!' character, the
code that checks for bad character will accept the '!' if it appears
after a ';', since the '!' might begin a compiler directive.
This current implementation fails when a compiler directive appears
after some other character that might (by means of further source
processing not visible to the prescanner) be replaced with a ';' or
newline.
Extend the bad character check for '!' to actually check for a compiler
directive sentinel instead.
…fter macro expansion
When compiler directives (!$omp) and/or their continuations (!$omp &)
are produced by macro expansion, handle those continuations. Also allow
a continuation marker (&) to appear in a macro actual argument.
D155499 fixed an issue with implicit continuations. The fixes included a
nested parenthesis check during definition of a macro which is then
carried over in the scanner state.
This leads to the following corner case to fail:
subroutine foo(a, d)
implicit none
integer :: a
integer :: d
! An implicit continuation won't be considered unless
! the definition of "bar" above is removed/commented
call sub(1,
2)
end subroutine foo
The definition of bar is indeed unbalanced but it is not even used in
the code, so it should not impact whether we apply implicit continuation
in the expansion of sub.
This change aims at addressing this issue by removing the balance check
and constraining a bit more when we consider implicit continuations:
only when we see a left parenthesis after a function-like macro, not a
object-like macro. In this case I think it is OK to (unconditionally)
implicitly continue to the next line in search of the corresponding
right parenthesis. This is, to my understanding, similar to what the C
preprocessor would do according to the description in [1].
[1] https://www.spinellis.gr/blog/20060626/
Differential Revision: https://reviews.llvm.org/D157414
The prescanner performs implicit line continuation when it looks
like the parenthesized arguments of a call to a function-like macro
may span multiple lines. In an attempt to work more like a
Fortran-oblivious C preprocessor, the prescanner will act as if
the following lines had been continuations so that the function-like
macro could be invoked.
This still seems like a good idea, but a recent bug report on
LLVM's GitHub issue tracker shows one way in which it could trigger
inadvertently and mess up a program. So this patch makes the
conditions for implicit line continuation much more strict.
First, the leading parenthesis has to have been preceded by an
identifier that's known to be a macro name. (It doesn't have to
be a function-like macro, since it's possible for a keyword-like
macro to expand to the name of a function-like macro.) Second,
no macro definition can ever have had unbalanced parentheses in
its replacement text.
Also cleans up some parenthesis recognition code to fix some
issues found in testing, so that a token with leading or trailing
spaces can still be recognized as a parenthesis or comma.
Fixes https://github.com/llvm/llvm-project/issues/63844.
Differential Revision: https://reviews.llvm.org/D155499
A fixed form continuation line should not be the first line of a statement;
emit a warning if it looks like one is so that the programmer can look
for a missing card.
Differential Revision: https://reviews.llvm.org/D142763
If a line is over 72 characters long, flang's preprocessor cuts it there
and continues on the next line.
For this purpose it uses the standard way of continuing line with & on each line.
However, it doesn't work with long compiler directives, like OpenMP or OpenACC ones.
The line that continues the directive also has to
contain the corresponding sentinel at the beginning.
This change implements the described functionality.
Also, some code was refactored in order to simplify and reuse existing code.
Reviewed By: klausler
Differential Revision: https://reviews.llvm.org/D126301
Using recently established message severity codes, upgrade
non-fatal messages to usage and portability warnings as
appropriate.
Differential Revision: https://reviews.llvm.org/D121246
When preprocessing "# ARG" in function-like macro expansion,
the preprocessor needs to pop the previously-pushed '#' token
from the end of the resulting token sequence after detecting the
argument name. The code to do this was just wrong in a couple of
ways.
Differential Revision: https://reviews.llvm.org/D117148
Rename the current -E option to "-E -Xflang -fno-reformat".
Add a new Parsing::EmitPreprocessedSource() routine to convert the
cooked character stream output of the prescanner back to something
more closely resembling output from a traditional preprocessor;
call this new routine when -E appears.
The new -E output is suitable for use as fixed form Fortran source to
compilation by (one hopes) any Fortran compiler. If the original
top-level source file had been free form source, the output will be
suitable for use as free form source as well; otherwise there may be
diagnostics about missing spaces if they were indeed absent in the
original fixed form source.
Unless the -P option appears, #line directives are interspersed
with the output (but be advised, f18 will ignore these if presented
with them in a later compilation).
An effort has been made to preserve original alphabetic character case
and source indentation.
Add -P and -fno-reformat to the new drivers.
Tweak test options to avoid confusion with prior -E output; use
-fno-reformat where needed, but prefer to keep -E, sometimes
in concert with -P, on most, updating expected results accordingly.
Differential Revision: https://reviews.llvm.org/D106727
Allow the lit test suite to run under Windows. This encompasses the following changes:
* Define `lit_tools_dir` for flang's test configuration
* Replace `(<command> || true)` idiom with `not <command>`
* Add `REQUIRES: shell` on tests that invoke a shell script
Reviewed By: awarzynski
Differential Revision: https://reviews.llvm.org/D89368
Update the preprocessor regression tests to use the new driver if the new driver is built (FLANG_BUILD_NEW_DRIVER=On), otherwise the tests will still run using f18.
Summary of changes:
- Introduce %flang to the regression tests, which points to the new driver if it is built or otherwise points to f18
- Update all tests in flang/test/Preprocessing/ to use %flang
Differential Revision: https://reviews.llvm.org/D94805
Hew more closely to the C17 standard; perform macro replacement
of arguments to function-like macros unless they're being stringified
or pasted. Test with a model "assert" macro idiom that exposed
the problem.
Differential Revision: https://reviews.llvm.org/D87650
