Merge latest FastBuild into missmah/fastbuild #1
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
- assign deterministic thread names to TCPConnectionPool threads
[Improvement][Linux] Reduce process overhead (particularly for short-lived ones) - start with a short sleep interval, increasing the sleep interval while receiving on input - when receiving input, reset the sleep interval - this allows short-lived processes to terminate upto 7ms faster - longer lived processes that emit output before exiting tend to also terminate more quickly [Improvement][Linux] Reduce per-process buffer size usage to scale better on multi-core systems - set smaller buffer for stdOut capture (512KiB instead of 1MiB) and don't increase default buffer size for stdErr Fix occasional failure of test in Process.cpp setting pipe buffer size on Linux: - typical Linux limit is 64MiB and we consumed ~2MiB per core. We now consume~512KiB per core from changes above - the assert was also removed as conceivably a system with many CPU cores (~256 or more) might exhaust the limit in the future
- saves ~4s on Windows, less on OSX/Linux Details: - replace sleep with Semaphore in TestDataTransfer - reduce delay sleeps in TestConnectionStuckDuringSend - override default Connect timeout in TestConnectionFailure
…n closing connections - also cuts TCPConnectionPool test time by ~8s on Windows
- read m_ShuttingDown via AtomicLoadRelaxed, consistent with other reads
- Semaphore could be accessed on connection thread after being destroyed on main thread - ConnectionInfo was freed after main thread synchronization which could cause occasional test failures due to memory leaks being reported
…sses - If the child process lifetime extends past FBuild or FBuildWorker, the socket would not be freed - Unfreed sockets are particularly problematic for the socket FBuildWorker listens on
…ng FNEW) - allocations will show up even if file/line number info is unavailable - this used to work, but was broken by a change to remove overhead in Release builds
- Tests are hanging, but only on Travis CI
- markup TEST_ASSERT macros for static analysis - missing cast
…oject references a Copy()
* Adjust GetRelativePath case sensitivity -As per the other code in PathUtils, change GetRelativePath to assume case insensitivity on windows and macosx. -Due to the requirements of some of the current tests, preserves case in the output relative file path. * Change check from "not linux" to "windows or mac" -Was checking for using case insensitive paths by checking that __LINUX__ was not defined. Instead, check that __WINDOWS__ or __OSX__ is defined. * Fix case insensitive comparison * Fix another compA/compB mixup * Whitespace to force rebuild
… robustly distributed)
- fix Clang 8.x.x config - add Clang 9.x.x config
- accelerates compilation of these files ~10x
… shift) - problem detected by PVS Studio analysis
- the timer can be safely retrieved on Windows whether the frequency has been retrieved or not
* Minor documentation fix * Fix documentation typos
- FASTBuild bffs auto detect VS 2019 or 2017 tool chain - Detects: -- Vendorized Toolchain (External/) -- Installation via environment variable -- Installation in default Visual Studio path -- Installation in default BuiltTools path (VS2017 only)
- bad html tags - tabs
…"custom" .CompilerFamily
- Remove deprecated AllowDBMigration flag - Fix whitspace in some for loops
[Fix][Windows] Terminated remote jobs are once again System Failues (Regression in v1.0) [Improvement][OSX][Linux] Remote Clang and GCC disk space erorrs are System Failures and are re-scheduled
Linux boxes suddently started complaining about expired certificates. The certificate is fine (no errors on other platforms). As a stab in the dark, try making the url explicitly use https.
Travis CI Linux nodes are incorrectly reporting that fastbuild.org's cert has expired. It has not, as evidenced by it working on Windows and OSX Travis nodes, and also for wget invoked on Linux outside of Travis. This seems to be a Travis configuraiton problem.
…ger cause parsing errors
* Travis CI: Update FASTBuild to v1.01 * Extending #if parsing with boolean && and || (#733) * Extending #if parsing with boolean && and || * Fix build on gcc/g++ * Updated documentation * Extending #if parsing with boolean && and || * Fix build on gcc/g++ * Updated documentation * Apply && and || operator precedence (&& before ||) * Tokenizer now recognises two character operators && and ||, #if handler modified accordingly * Remove tests from if_directive.bff, add them into TestBFFParsing.cpp * Actually evaluate the first defined result during the || pass * Kludge tests a bit until better understood * Formatting and cleanup * Added errors for #if operator parsing, applied them to tests. Co-authored-by: George Nakos <[email protected]> Co-authored-by: ffulin <[email protected]> * Fix bad merge * Delete stale file [skip ci] * Post intergation: #if supports || and && - simplify logic - add tests for some additional cases - simplify tests - code style - add documentation for new errors - update documentation to reflect precedence rules * [Improvement] Increase thread stack size to avoid unexpected problems with cache plugins * CachePlugin improvements [Note] DB Version changed [Improvement] Cache plugin can be initialized via InitEx which provides details of operating mode (read/write/verbose) [Improvement] Settings exposes .CachePluginDLLConfig to all passing of custom config to cache plugins [Improvement][Windows] Cache plugin DLL function names no longer need to be mangled * Fix tests for VS older that VS2019 - __pragma in macro behavior seems inconsistent - just do things manually for now * Replace master with better wording (#756) * Replace master with better wording * Force rebuild * Replace blacklist with better wording (#757) * Added “AllowResponseFile” setting. (#501) Co-authored-by: ffulin <[email protected]> * Post-integration cleanup: AllowResponseFiles - add documentation - fix unreachable code - update integraiton/markup files - increment database version * Fix typo in test bff [skip ci] * [Improvement] ObjectLists can be chained using .CompilerInputObjectLists * Fix memory leak in cache test * Code style cleanup: - remove use of auto - replace some manual loops with range-based for - add some missing const * LZ4: Remove .gitignore and Makefile * [Fix][Windows] Prevent deadlock in FBuildWorker on shutdown when it's sent a WM_QUIT by another application * [Improvement] VCXProject and XCodeProject include .editorconfig files by default [Fix] Fix documentation errors for VCXProject and XCodeProject * Update Visual Studio 2019 to 14.26.28801 (16.6.4) * Emit monitor events for EXEC_NODE (#768) * [WIP] Add new `docs/troubleshooting/cache.html` file to describe `cache debugging` strategies (#729) This document is a direct result of the discussions and debugging here: #727 NOTE: There remain some `TODO`(s) and unfinished sections to this file. I don't have all of the knowledge required to finish these sections properly. Hopefully @ffulin can fill in the blanks here. * Update `docs/documentation.html` to add a hyperlink to this new file. Signed-off-by: Layla <[email protected]> * Further improvements to Cache troubleshooting documentation * Documentation styling improvements and fixes * Add .SourceMapping_Experimental to Compiler node (#769) * Post integration: Cleanup SourceMapping_Experimental - minor code style fixes - enable test on Windows with Clang - disable distributed compilation temporartily when used and document that * [Fix] ForEach correctly handles comma separator in parallel loops * Update FASTBuild to use VS2019 16.7.0 (14.27.29110) - fix various newly surfaced compiler warnings * Fix *ExcludePattern inconsistencies with slash direction [Fix] CSAssembly() handles forward and backslashes correctly on all platforms in .CompilerInputExcludePattern [Fix] Exec() handles forward and backslashes correctly on all platforms in .ExecInputExcludePattern [Fix] ObjectList() handles forward and backslashes correctly on all platforms in .CompilerInputExcludePattern [Fix] Test() handles forward and backslashes correctly on all platforms in .TestInputExcludePattern * Generate integration files for recently added SourceMapping_Experimental [skip ci] * Response file improvements [Improvement] Compiler() response file use can be forced with .ForceResponseFile [Improvement] Library() response file use can be forced with .LibrarianForceResponseFile [Improvement] Executable()/DLL() response file use can be forced with .LinkerForceResponseFile [Improvement][Linux] Command line limits (128 KiB) are enforced on Linux to make problems more obvious [Fix][Linux] Various "CanUseResponseFile" options are respected on Linux * Possible fix for file permissions problems on CI Uniquify response file names over time as a possible work around for permissions isuses on some machines - files reliably fail to open on the CI for some reason when re-used in rapid succession - this may be related to security software of some kind * Increase timeout on Response file permissions workaround for Windows * Improve output on failing assertion * Enhance assertion message further * Fix Args test - prevent tests executing code that relies on WorkerThread having previously created directories - revert some changes made during troubleshooting this issue * [Fix] Fix a failure path writing response files that was not stopping compilation correctly * Update documentation for ForceResponseFile options [skip ci] * v1.02 Co-authored-by: ggnkua <[email protected]> Co-authored-by: George Nakos <[email protected]> Co-authored-by: Brian Burgin <[email protected]> Co-authored-by: Axel Wefers <[email protected]> Co-authored-by: Joe Kirchoff <[email protected]> Co-authored-by: Layla <[email protected]> Co-authored-by: Javier Blazquez <[email protected]>
* Travis CI: Update FASTBuild to v1.01 * Extending #if parsing with boolean && and || (#733) * Extending #if parsing with boolean && and || * Fix build on gcc/g++ * Updated documentation * Extending #if parsing with boolean && and || * Fix build on gcc/g++ * Updated documentation * Apply && and || operator precedence (&& before ||) * Tokenizer now recognises two character operators && and ||, #if handler modified accordingly * Remove tests from if_directive.bff, add them into TestBFFParsing.cpp * Actually evaluate the first defined result during the || pass * Kludge tests a bit until better understood * Formatting and cleanup * Added errors for #if operator parsing, applied them to tests. Co-authored-by: George Nakos <[email protected]> Co-authored-by: ffulin <[email protected]> * Fix bad merge * Delete stale file [skip ci] * Post intergation: #if supports || and && - simplify logic - add tests for some additional cases - simplify tests - code style - add documentation for new errors - update documentation to reflect precedence rules * [Improvement] Increase thread stack size to avoid unexpected problems with cache plugins * CachePlugin improvements [Note] DB Version changed [Improvement] Cache plugin can be initialized via InitEx which provides details of operating mode (read/write/verbose) [Improvement] Settings exposes .CachePluginDLLConfig to all passing of custom config to cache plugins [Improvement][Windows] Cache plugin DLL function names no longer need to be mangled * Fix tests for VS older that VS2019 - __pragma in macro behavior seems inconsistent - just do things manually for now * Replace master with better wording (#756) * Replace master with better wording * Force rebuild * Replace blacklist with better wording (#757) * Added “AllowResponseFile” setting. (#501) Co-authored-by: ffulin <[email protected]> * Post-integration cleanup: AllowResponseFiles - add documentation - fix unreachable code - update integraiton/markup files - increment database version * Fix typo in test bff [skip ci] * [Improvement] ObjectLists can be chained using .CompilerInputObjectLists * Fix memory leak in cache test * Code style cleanup: - remove use of auto - replace some manual loops with range-based for - add some missing const * LZ4: Remove .gitignore and Makefile * [Fix][Windows] Prevent deadlock in FBuildWorker on shutdown when it's sent a WM_QUIT by another application * [Improvement] VCXProject and XCodeProject include .editorconfig files by default [Fix] Fix documentation errors for VCXProject and XCodeProject * Update Visual Studio 2019 to 14.26.28801 (16.6.4) * Emit monitor events for EXEC_NODE (#768) * [WIP] Add new `docs/troubleshooting/cache.html` file to describe `cache debugging` strategies (#729) This document is a direct result of the discussions and debugging here: #727 NOTE: There remain some `TODO`(s) and unfinished sections to this file. I don't have all of the knowledge required to finish these sections properly. Hopefully @ffulin can fill in the blanks here. * Update `docs/documentation.html` to add a hyperlink to this new file. Signed-off-by: Layla <[email protected]> * Further improvements to Cache troubleshooting documentation * Documentation styling improvements and fixes * Add .SourceMapping_Experimental to Compiler node (#769) * Post integration: Cleanup SourceMapping_Experimental - minor code style fixes - enable test on Windows with Clang - disable distributed compilation temporartily when used and document that * [Fix] ForEach correctly handles comma separator in parallel loops * Update FASTBuild to use VS2019 16.7.0 (14.27.29110) - fix various newly surfaced compiler warnings * Fix *ExcludePattern inconsistencies with slash direction [Fix] CSAssembly() handles forward and backslashes correctly on all platforms in .CompilerInputExcludePattern [Fix] Exec() handles forward and backslashes correctly on all platforms in .ExecInputExcludePattern [Fix] ObjectList() handles forward and backslashes correctly on all platforms in .CompilerInputExcludePattern [Fix] Test() handles forward and backslashes correctly on all platforms in .TestInputExcludePattern * Generate integration files for recently added SourceMapping_Experimental [skip ci] * Response file improvements [Improvement] Compiler() response file use can be forced with .ForceResponseFile [Improvement] Library() response file use can be forced with .LibrarianForceResponseFile [Improvement] Executable()/DLL() response file use can be forced with .LinkerForceResponseFile [Improvement][Linux] Command line limits (128 KiB) are enforced on Linux to make problems more obvious [Fix][Linux] Various "CanUseResponseFile" options are respected on Linux * Possible fix for file permissions problems on CI Uniquify response file names over time as a possible work around for permissions isuses on some machines - files reliably fail to open on the CI for some reason when re-used in rapid succession - this may be related to security software of some kind * Increase timeout on Response file permissions workaround for Windows * Improve output on failing assertion * Enhance assertion message further * Fix Args test - prevent tests executing code that relies on WorkerThread having previously created directories - revert some changes made during troubleshooting this issue * [Fix] Fix a failure path writing response files that was not stopping compilation correctly * Update documentation for ForceResponseFile options [skip ci] * v1.02 * Add ListDependencies() function to generate dependency text files for UE4 (#760) * Post integration: ListDependencies - Emit output messages consistently with other nodes - Emit error codes in failure messages if output file writing fails - minor code style fixes * Apple Silicon configs (#774) * Post integratoin: Support for Apply silicon natgive binaries - use longer sleep in TestTimer::Validate only on OSX ARM - minor code style tweaks * Travis CI: Update tests to use v1.02 binaries * Small clean-up, remove LibraryNode::GetLibrarian and use m_Librarian instead of GetName (#710) Co-authored-by: Francisco Facioni <[email protected]> Co-authored-by: ffulin <[email protected]> * Improve include path extraction in ProjectGeneratorBase (#701) * Add `/imsvc` option supported by clang-cl. * Add `-idirafter` option supported by GCC and Clang. * Extract include paths according to the priority of options that introduce them. Eg. path added via `-I` will be searched before the one that is added via `-isystem` even if it appears later on the command line. * Fix extraction of the option argument from `-isystem-afterDIR` which didn't work correctly due to a missing `break`. Previously `-afterDIR` was extracted in that case as the include path. * Add test for the order of include paths in generated project files. * Add all recognized options for specifying include paths to the documentation. Co-authored-by: ffulin <[email protected]> * Post integration: Improved Intellisense/CodeSense inlude path extraction - fix various code style issues * Improve extraction of libraries from command line for GCC (#702) * Support `-l:filename` syntax which allows to link a specific file. * Support `-Bstatic` and `-Bdynamic` options which control whether `-lfoo` checks for existence of a corresponding dynamic library file (`libfoo.so`) or not. * Change interpretation of `-l` options to match behavior of GNU ld - `-Bdynamic` is the default for systems that support dynamic linking, so for `-l` options we check for dynamic libraries first. * Post integration: linker command line dependency extraction fixes/improvements - code style cleanup * Add ability to manually specifiy workers via an environment variable. (#689) E.g FASTBUILD_WORKERS=10.0.0.1;10.0.0.2;10.0.0.3 Co-authored-by: eclazi <blank> * Post integration: FASTBUILD_WORKERS environment variable - move logic from FBuild -> WorkerBrokerage - code style - add documentation - rename variabels and functions related to init of the brokerage for clarity * Rename AutoPtr -> UniquePtr * Remove some unnecessary includes * Fix BFFFuzzer compilation * Code cleanup - Add missing const - Remove some goto instances - Add missing ovveride keywords - Use constexpr in a few places - Add some missing default cases to switch statements * Code style fixes (Thanks to Joe Kirchoff) * Protocol: Fix compilation when TCP_DEBUG is enabled - an array of packet string names was out of date * MemDebug: Fix debug memory fill for remainder bytes - memory fill of remainder was wiritng pattern incorrectly - additionally simplify code slightly * Improve Static Analysis strictness for VS2019: - enable all static analysis checks and opt-out of those that cause errors - will may be able to keep more enabled in the future * Fix OSX and Linux compilation * [Improvement] MSVC linker failure LNK1201 is automatically retried to work around VS2019 bug * Tidy VS2019 static analysis flags * [Fix] Visual Studio solution header for VS2019 so version selector works correctly * Update brokerage file to use ip address instead of host name (#770) * Add domain name to hostname results * Update brokerage file to include ip address Brokerage filename now contains ip address in addition to the host name Connection treats host string as a + delimited array and will attempt resolving each entry to an ip address * fix ::getdomainname check * Change all brokerage timers to use seconds for consistency * whitespace * Continue attempting connection to additional worker IPs * Ensure hostname is updated in WorkerBrokerage if changed Get hostname from WorkerBrokerage and pass to SetStatus() instead of also retrieving in WorkerWindow Handle empty strings in SetStatus so display text is formatted correctly if host name hasn't been acquired yet in WorkerBrokerage * Use ipaddress for brokerage filename If ip address is not yet available, use host name instead In addition to host name, write domain name and FQDN to brokerage file if domain name is not blank Add Network:GetDomainName() Reset TCPConnectionPool::Connect() to previous functionality where only one hostname/ip address is accepted instead of the + delimited string Reset GetHostName to previous functionality where only host name is returned, instead of FQDN Reset PROTOCOL_VERSION to 21 * revert TCPConnectionPool.cpp to before PR * revert WorkerBrokerage::FindWorkers() to before PR * Fix linux\apple typo Co-authored-by: ffulin <[email protected]> * Post-integration cleanup: Worker brokerage by IP - various style fixes * [New] -wsl option can forward to Windows Subsystem for Linux in Visual Studio * Rename FBuild.exe project/library from FBuildApp to FBuild * BFF parses fixes for User Functions: [Fix] Escaped characters in string literal args to User Functions were not escaped correctly [Fix] $ Subsititutions in string literal args to User Functions were not handled in the correct scope * [Fix] #if directives were not correctly handled when used inside list declarations (and possibly other cases) - change how the EndOfFile token is managed when parsing if directives to avoid edge cases. EndOfFile now never appears incorrectly in the middle of a stream, which would cause inconsistent behavior depending on what was being parsed when we encountered an ifdef * Add missing newline character in ListDependenciesNode::EmitOutputMessage() log (#796) Co-authored-by: Maxime Houlier <[email protected]> * [Fix] Fix features documentation typo (Thanks to Aleksey Kladov) * Simplify Visual Studio project generation with some bff helper functions * Compile with GCC9 and Clang10 on Linux by default * Core: adapt to Aarch64 on Linux (ARM64) (#792) * Post integration: ARM Linux compatibility - minor whitespace fixes * Tidy up FASTBuild VS projects: - consolidate use of helper function(s) - add solution items for miscellaneous files * Linux: Set thread stack size always above PTHREAD_STACK_MIN. (#791) PTHREAD_STACK_MIN=131072 on Aarch64 on Raspberry Pi OS, which made thread creation with stackSize=65536 fail. * Post commit: Respect PTHREAD_STACK_MIN on OSX & Linux - minor code style fixes * [New] Support integration of FASTBuild, Windows Subsystem for Linux and Visual Studio [Note] DB version has changed: Rebuild will occur [New] Support integration of FASTBuild, Windows Subsystem for Linux and Visual Studio - add various project generation options to assist with compiling and debugging - add "-wsl" helper command line option to allow invocation of commands in WSL * [Improvement] Add explicit support for Clang on windows in clang-cl mode (Thanks to Lambert Clara) - add a new compiler famiily and auto-detect clang-cl use - this allows clang to be used in "cl" mode, while allowing us to handle some differences versus the real cl.exe * Fix dispatch re-entry check in Tracing::Callbacks::DispatchCallbacksDebugSpam() to allow further logs to be recorded. (#789) This issue occured with PROTOCOL_DEBUG_ENABLED defined, which enables the use of DEBUGSPAM() to output protocol debug log, and was preventing any further logs. Co-authored-by: Maxime Houlier <[email protected]> * Fix bad merge causing incorrect WARNING: output * [Fix] Missing errors for preprocessors that write to stdout on failure * [Improvement] Add .Libraries2 to Executable and DLL which can be accessed with %1[1] * Add lz4 1.9.3 source (not yet used) - will be used soon. Submitting to keep in sync with canonical Perforce repo * Add config to build FASTBuild with Clang 11 on Windows (unused) - submitting to keep github in sync with canonical Perforce depot * v1.03 Co-authored-by: ggnkua <[email protected]> Co-authored-by: George Nakos <[email protected]> Co-authored-by: Brian Burgin <[email protected]> Co-authored-by: Axel Wefers <[email protected]> Co-authored-by: Joe Kirchoff <[email protected]> Co-authored-by: Layla <[email protected]> Co-authored-by: Javier Blazquez <[email protected]> Co-authored-by: poppolopoppo <[email protected]> Co-authored-by: Francisco Facioni <[email protected]> Co-authored-by: Francisco Facioni <[email protected]> Co-authored-by: dummyunit <[email protected]> Co-authored-by: eclazi <[email protected]> Co-authored-by: Joe Kirchoff <[email protected]> Co-authored-by: mhoulier <[email protected]> Co-authored-by: Maxime Houlier <[email protected]> Co-authored-by: Nicolas <[email protected]>
* Travis CI: Update FASTBuild to v1.01 * Extending #if parsing with boolean && and || (#733) * Extending #if parsing with boolean && and || * Fix build on gcc/g++ * Updated documentation * Extending #if parsing with boolean && and || * Fix build on gcc/g++ * Updated documentation * Apply && and || operator precedence (&& before ||) * Tokenizer now recognises two character operators && and ||, #if handler modified accordingly * Remove tests from if_directive.bff, add them into TestBFFParsing.cpp * Actually evaluate the first defined result during the || pass * Kludge tests a bit until better understood * Formatting and cleanup * Added errors for #if operator parsing, applied them to tests. Co-authored-by: George Nakos <[email protected]> Co-authored-by: ffulin <[email protected]> * Fix bad merge * Delete stale file [skip ci] * Post intergation: #if supports || and && - simplify logic - add tests for some additional cases - simplify tests - code style - add documentation for new errors - update documentation to reflect precedence rules * [Improvement] Increase thread stack size to avoid unexpected problems with cache plugins * CachePlugin improvements [Note] DB Version changed [Improvement] Cache plugin can be initialized via InitEx which provides details of operating mode (read/write/verbose) [Improvement] Settings exposes .CachePluginDLLConfig to all passing of custom config to cache plugins [Improvement][Windows] Cache plugin DLL function names no longer need to be mangled * Fix tests for VS older that VS2019 - __pragma in macro behavior seems inconsistent - just do things manually for now * Replace master with better wording (#756) * Replace master with better wording * Force rebuild * Replace blacklist with better wording (#757) * Added “AllowResponseFile” setting. (#501) Co-authored-by: ffulin <[email protected]> * Post-integration cleanup: AllowResponseFiles - add documentation - fix unreachable code - update integraiton/markup files - increment database version * Fix typo in test bff [skip ci] * [Improvement] ObjectLists can be chained using .CompilerInputObjectLists * Fix memory leak in cache test * Code style cleanup: - remove use of auto - replace some manual loops with range-based for - add some missing const * LZ4: Remove .gitignore and Makefile * [Fix][Windows] Prevent deadlock in FBuildWorker on shutdown when it's sent a WM_QUIT by another application * [Improvement] VCXProject and XCodeProject include .editorconfig files by default [Fix] Fix documentation errors for VCXProject and XCodeProject * Update Visual Studio 2019 to 14.26.28801 (16.6.4) * Emit monitor events for EXEC_NODE (#768) * [WIP] Add new `docs/troubleshooting/cache.html` file to describe `cache debugging` strategies (#729) This document is a direct result of the discussions and debugging here: #727 NOTE: There remain some `TODO`(s) and unfinished sections to this file. I don't have all of the knowledge required to finish these sections properly. Hopefully @ffulin can fill in the blanks here. * Update `docs/documentation.html` to add a hyperlink to this new file. Signed-off-by: Layla <[email protected]> * Further improvements to Cache troubleshooting documentation * Documentation styling improvements and fixes * Add .SourceMapping_Experimental to Compiler node (#769) * Post integration: Cleanup SourceMapping_Experimental - minor code style fixes - enable test on Windows with Clang - disable distributed compilation temporartily when used and document that * [Fix] ForEach correctly handles comma separator in parallel loops * Update FASTBuild to use VS2019 16.7.0 (14.27.29110) - fix various newly surfaced compiler warnings * Fix *ExcludePattern inconsistencies with slash direction [Fix] CSAssembly() handles forward and backslashes correctly on all platforms in .CompilerInputExcludePattern [Fix] Exec() handles forward and backslashes correctly on all platforms in .ExecInputExcludePattern [Fix] ObjectList() handles forward and backslashes correctly on all platforms in .CompilerInputExcludePattern [Fix] Test() handles forward and backslashes correctly on all platforms in .TestInputExcludePattern * Generate integration files for recently added SourceMapping_Experimental [skip ci] * Response file improvements [Improvement] Compiler() response file use can be forced with .ForceResponseFile [Improvement] Library() response file use can be forced with .LibrarianForceResponseFile [Improvement] Executable()/DLL() response file use can be forced with .LinkerForceResponseFile [Improvement][Linux] Command line limits (128 KiB) are enforced on Linux to make problems more obvious [Fix][Linux] Various "CanUseResponseFile" options are respected on Linux * Possible fix for file permissions problems on CI Uniquify response file names over time as a possible work around for permissions isuses on some machines - files reliably fail to open on the CI for some reason when re-used in rapid succession - this may be related to security software of some kind * Increase timeout on Response file permissions workaround for Windows * Improve output on failing assertion * Enhance assertion message further * Fix Args test - prevent tests executing code that relies on WorkerThread having previously created directories - revert some changes made during troubleshooting this issue * [Fix] Fix a failure path writing response files that was not stopping compilation correctly * Update documentation for ForceResponseFile options [skip ci] * v1.02 * Add ListDependencies() function to generate dependency text files for UE4 (#760) * Post integration: ListDependencies - Emit output messages consistently with other nodes - Emit error codes in failure messages if output file writing fails - minor code style fixes * Apple Silicon configs (#774) * Post integratoin: Support for Apply silicon natgive binaries - use longer sleep in TestTimer::Validate only on OSX ARM - minor code style tweaks * Travis CI: Update tests to use v1.02 binaries * Small clean-up, remove LibraryNode::GetLibrarian and use m_Librarian instead of GetName (#710) Co-authored-by: Francisco Facioni <[email protected]> Co-authored-by: ffulin <[email protected]> * Improve include path extraction in ProjectGeneratorBase (#701) * Add `/imsvc` option supported by clang-cl. * Add `-idirafter` option supported by GCC and Clang. * Extract include paths according to the priority of options that introduce them. Eg. path added via `-I` will be searched before the one that is added via `-isystem` even if it appears later on the command line. * Fix extraction of the option argument from `-isystem-afterDIR` which didn't work correctly due to a missing `break`. Previously `-afterDIR` was extracted in that case as the include path. * Add test for the order of include paths in generated project files. * Add all recognized options for specifying include paths to the documentation. Co-authored-by: ffulin <[email protected]> * Post integration: Improved Intellisense/CodeSense inlude path extraction - fix various code style issues * Improve extraction of libraries from command line for GCC (#702) * Support `-l:filename` syntax which allows to link a specific file. * Support `-Bstatic` and `-Bdynamic` options which control whether `-lfoo` checks for existence of a corresponding dynamic library file (`libfoo.so`) or not. * Change interpretation of `-l` options to match behavior of GNU ld - `-Bdynamic` is the default for systems that support dynamic linking, so for `-l` options we check for dynamic libraries first. * Post integration: linker command line dependency extraction fixes/improvements - code style cleanup * Add ability to manually specifiy workers via an environment variable. (#689) E.g FASTBUILD_WORKERS=10.0.0.1;10.0.0.2;10.0.0.3 Co-authored-by: eclazi <blank> * Post integration: FASTBUILD_WORKERS environment variable - move logic from FBuild -> WorkerBrokerage - code style - add documentation - rename variabels and functions related to init of the brokerage for clarity * Rename AutoPtr -> UniquePtr * Remove some unnecessary includes * Fix BFFFuzzer compilation * Code cleanup - Add missing const - Remove some goto instances - Add missing ovveride keywords - Use constexpr in a few places - Add some missing default cases to switch statements * Code style fixes (Thanks to Joe Kirchoff) * Protocol: Fix compilation when TCP_DEBUG is enabled - an array of packet string names was out of date * MemDebug: Fix debug memory fill for remainder bytes - memory fill of remainder was wiritng pattern incorrectly - additionally simplify code slightly * Improve Static Analysis strictness for VS2019: - enable all static analysis checks and opt-out of those that cause errors - will may be able to keep more enabled in the future * Fix OSX and Linux compilation * [Improvement] MSVC linker failure LNK1201 is automatically retried to work around VS2019 bug * Tidy VS2019 static analysis flags * [Fix] Visual Studio solution header for VS2019 so version selector works correctly * Update brokerage file to use ip address instead of host name (#770) * Add domain name to hostname results * Update brokerage file to include ip address Brokerage filename now contains ip address in addition to the host name Connection treats host string as a + delimited array and will attempt resolving each entry to an ip address * fix ::getdomainname check * Change all brokerage timers to use seconds for consistency * whitespace * Continue attempting connection to additional worker IPs * Ensure hostname is updated in WorkerBrokerage if changed Get hostname from WorkerBrokerage and pass to SetStatus() instead of also retrieving in WorkerWindow Handle empty strings in SetStatus so display text is formatted correctly if host name hasn't been acquired yet in WorkerBrokerage * Use ipaddress for brokerage filename If ip address is not yet available, use host name instead In addition to host name, write domain name and FQDN to brokerage file if domain name is not blank Add Network:GetDomainName() Reset TCPConnectionPool::Connect() to previous functionality where only one hostname/ip address is accepted instead of the + delimited string Reset GetHostName to previous functionality where only host name is returned, instead of FQDN Reset PROTOCOL_VERSION to 21 * revert TCPConnectionPool.cpp to before PR * revert WorkerBrokerage::FindWorkers() to before PR * Fix linux\apple typo Co-authored-by: ffulin <[email protected]> * Post-integration cleanup: Worker brokerage by IP - various style fixes * [New] -wsl option can forward to Windows Subsystem for Linux in Visual Studio * Rename FBuild.exe project/library from FBuildApp to FBuild * BFF parses fixes for User Functions: [Fix] Escaped characters in string literal args to User Functions were not escaped correctly [Fix] $ Subsititutions in string literal args to User Functions were not handled in the correct scope * [Fix] #if directives were not correctly handled when used inside list declarations (and possibly other cases) - change how the EndOfFile token is managed when parsing if directives to avoid edge cases. EndOfFile now never appears incorrectly in the middle of a stream, which would cause inconsistent behavior depending on what was being parsed when we encountered an ifdef * Add missing newline character in ListDependenciesNode::EmitOutputMessage() log (#796) Co-authored-by: Maxime Houlier <[email protected]> * [Fix] Fix features documentation typo (Thanks to Aleksey Kladov) * Simplify Visual Studio project generation with some bff helper functions * Compile with GCC9 and Clang10 on Linux by default * Core: adapt to Aarch64 on Linux (ARM64) (#792) * Post integration: ARM Linux compatibility - minor whitespace fixes * Tidy up FASTBuild VS projects: - consolidate use of helper function(s) - add solution items for miscellaneous files * Linux: Set thread stack size always above PTHREAD_STACK_MIN. (#791) PTHREAD_STACK_MIN=131072 on Aarch64 on Raspberry Pi OS, which made thread creation with stackSize=65536 fail. * Post commit: Respect PTHREAD_STACK_MIN on OSX & Linux - minor code style fixes * [New] Support integration of FASTBuild, Windows Subsystem for Linux and Visual Studio [Note] DB version has changed: Rebuild will occur [New] Support integration of FASTBuild, Windows Subsystem for Linux and Visual Studio - add various project generation options to assist with compiling and debugging - add "-wsl" helper command line option to allow invocation of commands in WSL * [Improvement] Add explicit support for Clang on windows in clang-cl mode (Thanks to Lambert Clara) - add a new compiler famiily and auto-detect clang-cl use - this allows clang to be used in "cl" mode, while allowing us to handle some differences versus the real cl.exe * Fix dispatch re-entry check in Tracing::Callbacks::DispatchCallbacksDebugSpam() to allow further logs to be recorded. (#789) This issue occured with PROTOCOL_DEBUG_ENABLED defined, which enables the use of DEBUGSPAM() to output protocol debug log, and was preventing any further logs. Co-authored-by: Maxime Houlier <[email protected]> * Fix bad merge causing incorrect WARNING: output * [Fix] Missing errors for preprocessors that write to stdout on failure * [Improvement] Add .Libraries2 to Executable and DLL which can be accessed with %1[1] * Add lz4 1.9.3 source (not yet used) - will be used soon. Submitting to keep in sync with canonical Perforce repo * Add config to build FASTBuild with Clang 11 on Windows (unused) - submitting to keep github in sync with canonical Perforce depot * v1.03 * Switch dev branch tests to use FASTBuild v1.03 binaries * Switch binaries back to v1.02 OSX version requires a newer OS install which will need to be resolved * Use FASTBuild v1.03 binaries for CI - OSX binaries have been rebuilt to correctly support older OSX versions * Ensure compilation on OSX targets OSX 10.7 * Fix various compilation warnings * Fix various clang warnings: - superfluous ; - unused macros - 0 (or NULL) used for pointer instead of nullptr - mismatched include casing - use of deprecated types - and others * Fix missing line endings in various test files * Various clang-cl fixes: [Fix] Unity fixup is applied when using Clang-cl with static analysis so errors are correctly reported [Fix] Compilation with Clang doesn't fail when output is empty (can occur with Clang-cl with empty files) [Fix] SourceMapping_Experimental is applied correcly when using Clang-cl [Fix] Distributed compilation with Clang-cl is auto disabled when using SourceMapping_Experimental (as remapping is currently not distributed) [Improvement] When using Clang with static analysis, results are written to output file if no output is written by Clang itself [Fix] Warnings are printed for remotely built objects when using Clang-cl even if "warning as errors" is off * Fix Clang11 config for building FASTBuild: - switch to clang-cl for Clang11 to make management of Windows headers etc simpler - using Clang 11 when building with Clang on Windows (but keep Clang 7 use for Travis CI) * Simplify LZ4 build config: - disable all warnings instead of trying to disable them manually - there are too many warnings and they differ between Clang versions as well as clang.exe vs clang-cl.exe * Cleanup clang configs used to build FASTBuild on Windows: - use clang-cl for all versions * Update Visual Studio 2019 version to 16.8.3 (14.28.29333) * [Improvement] Distributed compilation and cache performance improved slightly due to LZ4 update (1.9.3) - Compression speed for preprocessed data improved ~5% - Decompression speed for preprocessed data improved ~5% - Compression speed for object files improved 5% - Decompression speed for object files improved upto ~10% * Fix MemoryLeak tracker when running on Windows with Clang * -fixuperrorpaths fixes [Fix] -fixuperrorpaths is correctly applied to Clang warnings using (x,y) style line numbering [Fix][OSX][Linux] -fixuperrorpaths with VBCC doesn't change slash direction on Linux/OSX * [Improvement] Dependency database memory usage reduced slightly - save 24 bytes per node by avoiding wasted bytes due to padding and oversized enum types * Remote Thread tracking: - Track which thread is used to build a job on remote hosts. This will be used for further changes to improve build visualization - Change ThreadIndex to uint16_t to avoid any increase in Job class size * ProfileManager: Fix FormatU64 so it handles printing zero correctly * [Improvement] Network Protocol bandwidth usage reduced slightly [Note] Distributed compilation protocol has changed: please update workers [Improvement] Network Protocol bandwidth usage reduced slightly - coupling small optimization with returning info about the thread used to compile remotely which will be used for future functionality * Fix typo in changelog: Compiation -> Compilation * -fastcancel is on by default [Improvement] FastCancel is enabled by default and can be disabled with -nofastcancel [Improvement] Remove deprecated -nooutputbuffering (option hasn't done anything for a long time) * Array: EmplaceBack returns reference to newly created element * [New] -profile option emits fbuild_profile.json in Chrome tracing format, which can be used to visualize the build * Update fbuild.bash-completion script for recent changes * [Improvement] Distributed compilation scheduling latency reduced by upto 100ms in certain scenarios Also: - improve accuracy of -profile display for distributed jobs, particularly when a remote jobs is returned and we cancel the local race * Minor DB serialization optimizations [Note] DB version changed - rebuild will occur [Improvement] Build database read/write optimized slightly * [Improvement][Windows] Reduce CPU overhead of FBuild slightly, particularly with many CPU cores - (Windows only) Limit the MainThread semaphore signal count to 1. Since it processes all work every time it awakens, additional signals are redundant. * WorkerThread: Improve profile macro setup - capture lifetime of worker threads more accurately - add macro to show per-thread tmp dir creation - make thread names display better for system with 10 or more cpu cores * Node: Fix return of void * FBuildWorker CPU usage, UI latency and shutdown speed improvements [Improvement] FBuildWorker idle CPU usage reduced, particularly for systems with many cpu cores [Improvement] FBuildWorker shuts down more quickly due to improved thread management [Improvement][Windows] Avoid per-CPU core stall updating UI, which could also delay exiting (up to 100ms per core) * [Improvement] Jobs are scheduled in a more efficient order when using distributed compilation (both locally and remotely) [Improvement] Jobs are scheduled in a more efficient order when using distributed compilation (both locally and remotely) - Jobs start being preprocessed in order of recursive cost, but can complete in any order. - We now sort the distributable jobs (the second/compilation pass) to ensure the most expensive jobs are started first (whether handed out to workers, or completed locally) - NOTE: The m_DistributableJobs_Available queue order is reversed (compared to before) so jobs are now removed from the end. This allows the same sorting helper to be used as for the main queue. * [New] Dependencies can be exported in DOT format using -dot and -dotfull - this allows visualiation of dependencies in tools that support the DOT file format * Ass some profiling macros * LightCache improvements/fixes: [Improvement] LightCache use is no longer inhibited by unused includes using macros: #include SOME_MACRO [Fix] Fix typo in 'Could not resolve macro' LightCache message * [Fix] Executable and DLL '%1[1]' is now correctly expanded even if '%1[0]' is not used * Fix Compiler() example for VS2019 in the documentation * BuildProfiler: Ensure backslashes in args are escaped in JSON output - args can contain backslashes if passed on the command line * Scheduling improvements - ObjectList/Library [Improvement] Compilation ordering improved by remove unnecesary dependencies from ObjectList and Library [Note] DB version changed * [Fix] Output messages no longer missing when using LightCache with -cachewrite and no distribution * Update .gitignore to exclude linux binaries * Update copyright to 2021 * v1.04 - Version bump - Documentation updates Co-authored-by: ggnkua <[email protected]> Co-authored-by: George Nakos <[email protected]> Co-authored-by: Brian Burgin <[email protected]> Co-authored-by: Axel Wefers <[email protected]> Co-authored-by: Joe Kirchoff <[email protected]> Co-authored-by: Layla <[email protected]> Co-authored-by: Javier Blazquez <[email protected]> Co-authored-by: poppolopoppo <[email protected]> Co-authored-by: Francisco Facioni <[email protected]> Co-authored-by: Francisco Facioni <[email protected]> Co-authored-by: dummyunit <[email protected]> Co-authored-by: eclazi <[email protected]> Co-authored-by: Joe Kirchoff <[email protected]> Co-authored-by: mhoulier <[email protected]> Co-authored-by: Maxime Houlier <[email protected]> Co-authored-by: Nicolas <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
No description provided.