Skip to content

Commit 54a80ab

Browse files
mattmccutchen-ccimattmccutchen-cci
authored
Merge pull request #687 from correctcomputation/merge-from-microsoft-20210825
Merge from Microsoft 2021-08-25
2 parents 5d364ab + c69e157 commit 54a80ab

15 files changed

+1301
-269
lines changed

‎clang/docs/checkedc/LLVM-Upgrade-Notes.md

Lines changed: 38 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,38 @@
1+
# Notes on LLVM/Clang Upgrades
2+
3+
This file gives details about the commit hashes up to which the `baseline` and
4+
`master` branches of the `checkedc-clang` repository have been upgraded with
5+
respect to the upstream LLVM/Clang repository.
6+
7+
8+
# Upgrade to LLVM/Clang 11.x
9+
10+
The commit hash on the `main` branch of the upstream LLVM/Clang repository
11+
up to which the `baseline` and the `master` branches of the `checkedc-clang`
12+
repository have been upgraded:
13+
<branch-point-of-11-on-main> : 2e10b7a39b930ef8d9c4362509d8835b221fbc0a
14+
15+
Cherry-picked commits (on `master` of `checkedc-clang`) from the upstream
16+
LLVM/Clang repository:
17+
Details not available.
18+
19+
The branch `release_11.x` of the `checkedc-clang` repository is upgraded to
20+
<llvm-11.1.0-commit-hash> : 1fdec59bffc11ae37eb51a1b9869f0696bfd5312
21+
22+
23+
# Upgrade to LLVM/Clang 12.x
24+
25+
The commit hash on the `main` branch of the upstream LLVM/Clang repository
26+
up to which the `baseline` and the `master` branches of the `checkedc-clang`
27+
repository have been upgraded:
28+
<branch-point-of-12-on-main> : 8e464dd76befbc4a39a1d21968a3d5f543e29312
29+
30+
Cherry-picked commits (on `master` of `checkedc-clang`) from the upstream
31+
LLVM/Clang repository:
32+
33+
- Commit hash 0024efc69ea6cd0b630cd11cef5991b7edb73ffc on the `main` branch
34+
of the upstream LLVM/Clang repo (See LLVM/Clang bug
35+
[#48920](https://bugs.llvm.org/show_bug.cgi?id=48920) for details.)
36+
37+
The branch `release_12.x` of the `checkedc-clang` repository is upgraded to
38+
<llvm-12.0.1-commit-hash> : fed41342a82f5a3a9201819a82bf7a48313e296b

‎clang/docs/checkedc/Update-to-latest-LLVM-sources.md

Lines changed: 242 additions & 72 deletions
Original file line numberDiff line numberDiff line change
@@ -1,81 +1,251 @@
11
# Instructions for updating to the latest LLVM/Clang sources
22

3-
We are staying in sync with the LLVM/Clang mainline sources. The baseline branch is a pristine copy of
4-
LLVM/Clang sources. We periodically update the baseline branch and then push the changes to other branches.
5-
6-
The first step is to create updated baseline branches:
7-
1. Create new branches of your local baseline branches (we suggest creating new
8-
branches so that you can run automated testing. You'll need new branches
9-
that you can push to GitHub).
10-
2. Update those branches to the latest sources.
11-
3. Run testing on those branches to make sure things are stable.
12-
13-
The second step is to create updated master branches:
14-
1. Create branches of your updated baseline branches.
15-
2. Merge changes from the Checked C master branches into those branches.
16-
3. Fix merge conflicts and run testing. You will likely need to fix some issues
17-
and re-run testing until all issues are fixed.
18-
19-
The third step is to merge your changes back into your baseline and master branches.
20-
21-
## Create updated branches of your baseline branches
22-
23-
First create remote to the mirrored GitHub repo that contains the updated sources
24-
for LLVM/Clang. Go to your LLVM/Clang repo and do:
25-
26-
git remote add mirror https://github.com/llvm-mirror/llvm-project
27-
28-
Then branch your baseline branch and merge changes into it:
29-
3+
We are staying in sync with the upstream LLVM/Clang repository. The `baseline`
4+
branch of the `checkedc-clang` repository is a pristine copy of LLVM/Clang
5+
sources, and the `master` branch contains LLVM/Clang sources plus the support
6+
for Checked C language extensions. We periodically upgrade the `baseline` branch
7+
and the `master` branch to the sources of the most recent releases from the
8+
upstream LLVM/Clang repository.
9+
10+
The branching and merging policies of the `checkedc-clang` repository are
11+
aligned with the versioning and branching of upstream LLVM/Clang repository in
12+
order to closely follow the LLVM/Clang releases. LLVM/Clang releases are
13+
versioned as <major-version>.<minor-version>.<patch-version>.
14+
The LLVM/Clang repository has a branch called `main`, and several release
15+
branches (created off `main`) called `release/<major-version>.x`, each
16+
corresponding to the major version number of a release. New features go on the
17+
`main` branch. At an appropriate commit on the `main` branch, a release branch
18+
is created. Bug fixes go on the release branch and only some of these bug fixes
19+
get merged back into the `main` branch. Typically, LLVM/Clang makes two final
20+
releases for each major version.
21+
22+
Now we describe our approach for upgrading the `checkedc-clang` repository to
23+
LLVM/Clang sources corresponding to the two releases for each major version,
24+
for example, 12.0.0 and 12.0.1. The upgrade is performed in two phases as
25+
described below. This description is followed by detailed instructions for each
26+
phase.
27+
28+
**Phase 1**: We upgrade the `master` branch of the `checkedc-clang` repository
29+
up to the commit hash on the `main` branch of the LLVM/Clang repository at which
30+
the branch `release/12.x` is created - we shall refer to this commit hash as
31+
&lt;branch-point-of-12-on-main&gt;. We get this commit hash by executing the
32+
command `git merge-base main release/12.x` in a clone of the LLVM/Clang
33+
repository. Note that &lt;branch-point-of-12-on-main&gt;
34+
needs to be the full commit hash and not just the 7-digit abbreviation. This
35+
phase constitutes the major portion of the developer effort towards the merge.
36+
37+
**Phase 2**: We create a branch called `release_12.x` off the `master` branch in
38+
the `checkedc-clang` repository. Then there are two sub-phases:
39+
- When we wish to make a Checked C compiler release based on the 12.0.0
40+
release of LLVM/Clang:
41+
- We first upgrade the `release_12.x` branch (of the `checkedc-clang`
42+
repository) to the sources corresponding to the 12.0.0 release on the
43+
`release/12.x` branch of the LLVM/Clang repository.
44+
- Next, we merge Checked C features and bug fixes from the `master` branch
45+
to the `release_12.x` branch.
46+
- The release of the Checked C compiler happens from the `release_12.x`
47+
branch.
48+
- When we wish to make a Checked C compiler release based on the 12.0.1
49+
release of LLVM/Clang:
50+
- We upgrade the same `release_12.x` branch (of the `checkedc-clang`
51+
repository) to the sources corresponding to the 12.0.1 release on the
52+
`release/12.x` branch of the LLVM/Clang repository.
53+
- Next, we merge Checked C features and bug fixes from the `master` branch
54+
to the `release_12.x` branch.
55+
- The release of the Checked C compiler happens from the `release_12.x`
56+
branch.
57+
58+
59+
We do not expect substantial developer effort to be expended in Phase 2 as it is
60+
highly unlikely that the bug fixes on the release branches of the LLVM/Clang
61+
repository will break Checked C build and tests.
62+
63+
Note that Phase 1 and the sub-phases of Phase 2 may not be executed one after
64+
another in one continuous stretch. They may be spread over several weeks and may
65+
be interspersed with Checked C development on the `master` branch.
66+
67+
## Detailed Instructions - Phase 1
68+
69+
1. Create a new branch off the `baseline` branch in the `checkedc-clang`
70+
repository. Let us call this new branch `updated_baseline`.
71+
```
72+
git checkout -b updated_baseline baseline
73+
```
74+
2. Upgrade the branch `updated_baseline` to &lt;branch-point-of-12-on-main&gt;
75+
commit hash on the `main` branch of the LLVM/Clang repository. Record the commit
76+
hash &lt;branch-point-of-12-on-main&gt; in the document LLVM-Upgrade-Notes.md.
77+
```
78+
git remote add upstream https://github.com/llvm/llvm-project.git
79+
git pull upstream <branch-point-of-12-on-main>
80+
```
81+
3. Execute LLVM/Clang tests on the branch `updated_baseline` on your local
82+
machine to make sure that the upgrade is stable.
83+
84+
4. Merge the `updated_baseline` branch back to the `baseline` branch. We expect
85+
branch `updated_baseline` to be a descendant of branch `baseline`.
86+
```
3087
git checkout baseline
31-
git checkout -b updated-baseline
32-
git pull mirror master
33-
34-
## Run testing on your branched baseline branches.
35-
36-
You can run testing locally or push your branched baseline branches to GitHub
37-
and use automated testing. This will show you whether there are any unexpected
38-
failures in your baseline branch.
39-
40-
If you use automated testing, make sure to clean the build directory first.
41-
Enough changes will likely have accumulated that things may go wrong without doing
42-
that first.
43-
44-
## Branch your new baseline branches and merge master changes
45-
46-
You can now branch your baseline branches to create a new master branch:
47-
48-
git checkout -b updated-master
88+
git merge --ff-only updated_baseline
89+
```
90+
5. Create a new branch, say, `updated_baseline_master_12` from the `baseline`
91+
branch.
92+
```
93+
git checkout -b updated_baseline_master_12 baseline
94+
```
95+
6. Merge the `master` branch of the checkedc-clang repository into the
96+
`updated_baseline_master_12` branch. This merge may cause several merge
97+
conflicts and test case failures. After the merge command, git may suggest to
98+
you to set `merge.renamelimit` to a specific value. It is a good idea to set it
99+
as suggested and re-run the merge. The last section in this document gives
100+
guidelines on fixing the merge conflicts and test failures.
101+
```
102+
git checkout updated_baseline_master_12
103+
git merge master > merge_conflict_list.txt
104+
```
105+
7. The resolution of test failures may require you to cherry-pick commits from
106+
the upstream LLVM/Clang repository. Record the cherry-picked commits in the
107+
document LLVM-Upgrade-Notes.md. Let &lt;commit-to-cherry-pick&gt; be one
108+
such commit hash on branch &lt;cherry-pick-branch&gt; of the LLVM/Clang
109+
repository.
110+
```
111+
git fetch upstream <cherry-pick-branch>
112+
git cherry-pick -x <commit-to-cherry-pick>
113+
```
114+
8. While the merge conflicts and test failures are being fixed, it is possible
115+
that the `master` branch has had several commits. Merge the `master` branch into
116+
`updated_baseline_master_12` branch, and resolve the new conflicts and failures
117+
if necessary.
118+
```
119+
git merge master >> merge_conflict_list.txt
120+
```
121+
9. After all the merge conflicts and test failures on the local machine are
122+
fixed, push the `baseline` and `updated_baseline_master_12` branches to the
123+
remote repository and execute the ADO tests on branch
124+
`updated_baseline_master_12`. Commits to the `master` branch should be held off
125+
during the final execution of these tests, i.e. just prior to executing step 10
126+
below.
127+
```
128+
git push origin baseline
129+
git push origin updated_baseline_master_12
130+
```
131+
10. After the ADO tests are successful, merge the `updated_baseline_master_12`
132+
branch into `master` and push to the remote repository. If all the changes on
133+
the `master` branch have been merged into the `updated_baseline_master_12`
134+
branch in step 8, then it should be possible to do a fast-forward merge of
135+
branch `updated_baseline_master_12` into the `master` branch.
136+
```
137+
git checkout master
138+
git merge --ff-only updated_baseline_master_12
139+
git push origin master
140+
```
141+
142+
## Detailed Instructions - Phase 2
143+
We give the instructions for the first sub-phase of Phase 2. Similar steps
144+
have to be followed for the other sub-phase.
145+
146+
1. Create a new branch, called `release_12.x`, from the `master` branch in the
147+
`checkedc-clang` repository. If the branch already exists, then merge the
148+
`master` branch into it. Resolve merge conflicts and test failures if any.
149+
```
150+
git checkout -b release_12.x master
151+
OR
152+
git checkout release_12.x
49153
git merge master
50-
51-
You will very likely have merge conflicts and some test failures. The test
52-
failures usually stem from incorrect merges or Checked C-specific data not being
53-
initialized by new or changed constructor methods.
54-
55-
You may also need to pick up changes from LLVM/Clang for fixes to any unexpected
56-
baseline failures.
57-
58-
You can push your updated master branches up to GitHub for automated
59-
testing. If you haven't cleaned the build directory as described earlier,
60-
make sure you do that.
61-
62-
You'll want to run automated tests on Linux and Windows x86/x64, as well as
154+
```
155+
2. Upgrade the branch `release_12.x` to the LLVM/Clang sources corresponding to
156+
the 12.0.0 release. The commit hash associated with the 12.0.0 release is given
157+
by the LLVM/Clang tag `llvmorg-12.0.0`. Resolve merge conflicts and test
158+
failures if any. Record the fact that the `release_12.x` branch has been
159+
upgraded upto the commit hash given by the `llvmorg-12.0.0` tag, in the document
160+
LLVM-Upgrade-Notes.md.
161+
```
162+
git remote add upstream https://github.com/llvm/llvm-project.git
163+
git pull upstream llvmorg-12.0.0
164+
```
165+
3. Push the branch `release_12.x` to the remote repository, run ADO tests
166+
on it and fix test failures to make sure that the branch is stable.
167+
```
168+
git push origin release_12.x
169+
```
170+
## Practical Tips
171+
172+
1. If you use automated testing, make sure to clean the build directory first.
173+
Enough changes will likely have accumulated that things may go wrong without
174+
doing that first.
175+
176+
2. The test failures usually stem from incorrect merges or Checked C-specific data
177+
not being initialized by new or changed constructor methods.
178+
179+
3. You'll want to run automated tests on Linux and Windows x86/x64, as well as
63180
Linux LNT tests. You may find in some cases that tests need to be updated
64181
based on new warnings or slightly different compiler behavior.
65182

66-
## Merge your branched baseline and master branches
67-
68-
Once all testing is passing, you can merge your branches back into
69-
your baseline and master branches.
70-
71-
72-
git checkout baseline
73-
git merge updated-baseline
74-
git checkout master
75-
git merge updated-master
76-
77-
## Push the updated branches to GitHub
78-
79-
The changes will be extensive enough that you don't want to do a pull request
183+
4. The changes will be extensive enough that you don't want to do a pull request
80184
on GitHub. Just push the branches up to GitHub.
81185

186+
187+
## Handling Merge Conflicts and Test Failures
188+
189+
1. When `merge.renamelimit` is set to a large value, the output of the
190+
`git merge` command may have several lines like:
191+
- CONFLICT (rename/delete): ... in master renamed to ... in HEAD. Version
192+
HEAD ... left in tree.
193+
- CONFLICT (rename/delete): ... in master renamed to ... in HEAD. Version
194+
master ... left in tree.
195+
196+
If the version in HEAD is left in tree, it is very likely a valid change -
197+
the file may have been moved. If the version in `master` is left in tree,
198+
the files may be valid files that have been added as part of Checked C
199+
support:
200+
- `llvm/projects/checkedc-wrapper/lit.site.cfg.py`
201+
- `llvm/projects/checkedc-wrapper/lit.cfg.py`
202+
203+
Others may need more investigation. For example, in the LLVM/Clang 11 to 12
204+
merge, two files which were reported by git as "Version master ... left in
205+
tree" were old versions lying around that needed to be deleted:
206+
- `clang/include/clang/StaticAnalyzer/Core/PathSensitive/SMTAPI.h`
207+
- `clang/include/clang/AST/TypeNodes.def`
208+
209+
2. While resolving merge conflicts, it is a good idea to have three reference
210+
code snapshots to perform diffs of individual files to help distinguish between
211+
checkedc-specific changes and llvm-11-to-12 changes:
212+
- The `master` branch of the `checkedc-clang` repository at the commit hash
213+
which was merged into `updated_baseline_master_12`.
214+
- The pristine LLVM/Clang source at &lt;branch-point-of-11-on-main&gt;
215+
- The pristine LLVM/Clang source at &lt;branch-point-of-12-on-main&gt;
216+
217+
The description of the three snapshots above holds only for the initial
218+
merge from LLVM 11 to 12 in phase 1 step 6. For a merge of additional
219+
Checked C feature development in phase 1 step 8, here are the relevant
220+
snapshots and their commit hashes, which we get from the in-progress merge:
221+
- A snapshot of branch `updated_baseline_master_12` after phase 1 step 7, or
222+
an earlier iteration of step 8. The commit hash that corresponds to this
223+
snapshot is `HEAD` - use command `git rev-parse HEAD` to dump the commit
224+
hash.
225+
- A snapshot of the `master` branch that participated in the previous merge
226+
that produced the above snapshot. The commit hash that corresponds to this
227+
snapshot is the common ancestor - use command
228+
`git merge-base HEAD MERGE_HEAD` to dump the commit hash.
229+
- A snapshot of the current `master` branch. The commit hash that
230+
corresponds to this is `MERGE_HEAD` - use command `git rev-parse MERGE_HEAD`
231+
to dump the commit hash.
232+
233+
Here are some alternatives to the above approach of using source snapshots
234+
to manually perform diffs of the relevant files:
235+
- Run `git mergetool` - you may need to have a merge tool that is suitable
236+
for your environment (Linux/Windows) installed and configured
237+
(ex.: `kdiff3`).
238+
- Set `git config merge.conflictstyle diff3` before running `git merge` so
239+
that the conflict markup written to the file shows all three versions rather
240+
than just two.
241+
- Run
242+
`git show :1:clang/path/to/some/File.cpp > clang/path/to/some/File.1.cpp`
243+
(and similarly with 2 or 3 in place of 1) to write out the three versions
244+
for inspection with other tools.
245+
246+
3. When bitfields are modified (or need modification to accommodate changes
247+
related to Checked C) in files like `clang/include/clang/AST/Stmt.h` (see
248+
commit hash af12895f631), and `clang/include/clang/AST/DeclBase.h` (see the diff
249+
between commits bc556e5 and e08e3f6 - this change was required for a previous
250+
merge), more manual investigation may be needed to check the correctness of the
251+
changes.

‎clang/include/clang/AST/Stmt.h

Lines changed: 11 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -102,8 +102,10 @@ class alignas(void *) Stmt {
102102

103103
/// The statement class.
104104
unsigned sClass : 8;
105+
/// The checked scope specifier for the statement.
106+
unsigned CSS : 2;
105107
};
106-
enum { NumStmtBits = 8 };
108+
enum { NumStmtBits = 10 };
107109

108110
class NullStmtBitfields {
109111
friend class ASTStmtReader;
@@ -1207,6 +1209,14 @@ class alignas(void *) Stmt {
12071209

12081210
const char *getStmtClassName() const;
12091211

1212+
CheckedScopeSpecifier getCheckedScopeSpecifier() const {
1213+
return static_cast<CheckedScopeSpecifier>(StmtBits.CSS);
1214+
}
1215+
1216+
void setCheckedScopeSpecifier(CheckedScopeSpecifier CSS) {
1217+
StmtBits.CSS = CSS;
1218+
}
1219+
12101220
/// SourceLocation tokens are not useful in isolation - they are low level
12111221
/// value objects created/interpreted by SourceManager. We assume AST
12121222
/// clients will have a pointer to the respective SourceManager.

0 commit comments

Comments
 (0)