[clang Dependency Scanning] Out-of-Date Scanning File System Cache Entry Reporting C-APIs

[qiongsiwu]

This PR implements the C-APIs to report a scanning file system cache's out-of-date entries. The C-APIs contains a function to return a set of file system cache out-of-date entries, functions to facilitate looping through all the entries, and reporting the relevant information from the entries.

The APIs are based on llvm#144105.

rdar://152247357

[qiongsiwu]

Note to reviewers: I don't see a good way to cleanly test these APIs through c-index tests. Ideally, I'd like to create an underlying service whose file system cache I can manipulate directly to test these APIs directly (something similar to https://github.com/llvm/llvm-project/pull/144105/files#diff-25ec4b41a8aed83f312f2bb7b409f379f38dd9d1597d6e4e45e17681260dfcc2R205).

Do we think it is reasonable to create unit tests for these C-APIs? If so, I will try making some unit tests.

[qiongsiwu]

@swift-ci please test llvm.

[qiongsiwu]

@swift-ci please test llvm.

[qiongsiwu]

The two failing tests do not relate to this PR. rdar://154865941 is tracking them.

[qiongsiwu]

Gentle ping for review. Thanks!

[jansvoboda11]

Adding the unit test is a nice bonus, but I'd be in favor in keeping the old c-index-test test around. I think it's nice that c-index-test is a nice one-stop-shop for anything libclang.

[qiongsiwu]

Adding the unit test is a nice bonus, but I'd be in favor in keeping the old c-index-test test around. I think it's nice that c-index-test is a nice one-stop-shop for anything libclang.

This is a good point. The difficulty is that it is not easy to create out-date-entries exactly as we desired to make sure we have good coverage of all the APIs. I can recover the removed c-index-test that at least runs through the APIs obtaining the out-of-date entry set and disposing it. The set will contain no entry. Does that sound reasonable?

[qiongsiwu]

@swift-ci please test llvm.

[jansvoboda11]

I'm finding this test a bit odd. I'd expect it to either:

• use the real FS and only use the C API, which gives us a nice example of the API usage,
• use parts of the C++ API to inject files via the VFS, which removes usage of the real FS.

This currently mixes both approaches, and I'm wondering why?

[qiongsiwu]

I did not find a good way to manipulate the file system to set up the exact scenario where we have one entry for each out-of-date kind. The test sets up the scenario by manipulating the FS directly. Once that is done, it only uses the C-API to obtain diagnostics information obtained from the DependencyScanningService.

I am all ears for a different way of setting up the test.

[jansvoboda11]

I think what's confusing to me about this test is that the DependencyScanningWorkerFilesystem populates SharedCache with the contents of InMemoryFS, but Service then implicitly checks the cache against the on-disk FS. That also means llvm::sys::fs::createTemporaryFile() takes effect much later in the test function than I'd expect. This setup seems very fragile to me.

I'd suggest to drop the usage of InMemoryFS, set up the real FS, run a proper scan using the C API, then manipulate the real FS to make the cache out of date, and verify that through the new API.

[jansvoboda11]

I have something like this in mind:

TEST(DependencyScanningFSCacheOutOfDate, Basic) {
  CXDependencyScannerServiceOptions ServiceOptions =
      clang_experimental_DependencyScannerServiceOptions_create();
  CXDependencyScannerService Service =
      clang_experimental_DependencyScannerService_create_v1(ServiceOptions);
  CXDependencyScannerWorker Worker =
      clang_experimental_DependencyScannerWorker_create_v0(Service);

  llvm::SmallString<128> Dir;
  ASSERT_FALSE(llvm::sys::fs::createUniqueDirectory("tmp", Dir));

  llvm::SmallString<128> Include = Dir;
  llvm::sys::path::append(Include, "include");
  ASSERT_FALSE(llvm::sys::fs::create_directories(Include));

  llvm::SmallString<128> Include2 = Dir;
  llvm::sys::path::append(Include2, "include2");
  ASSERT_FALSE(llvm::sys::fs::create_directories(Include2));

  // Initially, we keep include/a.h empty and only write contents later.
  llvm::SmallString<128> HeaderA = Include;
  llvm::sys::path::append(HeaderA, "a.h");
  {
    std::error_code EC;
    llvm::raw_fd_ostream HeaderAFile(HeaderA, EC);
    ASSERT_FALSE(EC);
  }

  // Initially, we keep include/b.h missing and only create include2/b.h.
  llvm::SmallString<128> HeaderB2 = Include2;
  llvm::sys::path::append(HeaderB2, "b.h");
  {
    std::error_code EC;
    llvm::raw_fd_ostream HeaderB2File(HeaderB2, EC);
    ASSERT_FALSE(EC);
  }

  llvm::SmallString<128> TU = Dir;
  llvm::sys::path::append(TU, "tu.c");
  {
    std::error_code EC;
    llvm::raw_fd_ostream TUFile(TU, EC);
    ASSERT_FALSE(EC);
    TUFile << R"(
      #include "a.h"
      #include "b.h"
    ")";
  }

  const char *Argv[] = {"-c", TU.c_str(),      "-I", Include.c_str(),
                        "-I", Include2.c_str()};
  size_t Argc = std::size(Argv);
  CXDependencyScannerWorkerScanSettings ScanSettings =
      clang_experimental_DependencyScannerWorkerScanSettings_create(
          Argc, Argv, /*ModuleName=*/nullptr, /*WorkingDirectory=*/Dir.c_str(),
          /*MLOContext=*/nullptr, /*MLO=*/nullptr);

  CXDepGraph *Graph = new CXDepGraph;
  CXErrorCode ScanResult =
      clang_experimental_DependencyScannerWorker_getDepGraph(
          Worker, ScanSettings, Graph);
  ASSERT_EQ(ScanResult, CXError_Success);

  // Now, we change the size of include/a.h.
  {
    std::error_code EC;
    llvm::raw_fd_ostream HeaderAFile(HeaderA, EC);
    ASSERT_FALSE(EC);
    HeaderAFile << "// New content!\n";
  }

  // Now, we populate include/b.h.
  llvm::SmallString<128> HeaderB = Include;
  llvm::sys::path::append(HeaderB, "b.h");
  {
    std::error_code EC;
    llvm::raw_fd_ostream HeaderBFile(HeaderB, EC);
    ASSERT_FALSE(EC);
  }

  CXDepScanFSOutOfDateEntrySet Entries =
      clang_experimental_DependencyScannerService_getFSCacheOutOfDateEntrySet(
          Service);

  size_t NumEntries =
      clang_experimental_DepScanFSCacheOutOfDateEntrySet_getNumOfEntries(
          Entries);
  ASSERT_EQ(NumEntries, 2u);

  for (size_t Idx = 0; Idx < NumEntries; Idx++) {
    CXDepScanFSOutOfDateEntry Entry =
        clang_experimental_DepScanFSCacheOutOfDateEntrySet_getEntry(Entries,
                                                                    Idx);
    CXDepScanFSCacheOutOfDateKind Kind =
        clang_experimental_DepScanFSCacheOutOfDateEntry_getKind(Entry);
    CXString Path =
        clang_experimental_DepScanFSCacheOutOfDateEntry_getPath(Entry);
    ASSERT_TRUE(Kind == NegativelyCached || Kind == SizeChanged);
    switch (Kind) {
    case NegativelyCached:
      ASSERT_STREQ(clang_getCString(Path), HeaderB.c_str());
      break;
    case SizeChanged:
      ASSERT_STREQ(clang_getCString(Path), HeaderA.c_str());
      ASSERT_EQ(
          clang_experimental_DepScanFSCacheOutOfDateEntry_getCachedSize(Entry),
          16u);
      ASSERT_EQ(
          clang_experimental_DepScanFSCacheOutOfDateEntry_getActualSize(Entry),
          0u);
      break;
    }
  }

  clang_experimental_DepScanFSCacheOutOfDateEntrySet_disposeSet(Entries);
}