Skip to content
This repository has been archived by the owner on Dec 25, 2023. It is now read-only.

Commit

Permalink
File format (.xet) change. DdsToXet can upgrade old Xet files to the …
Browse files Browse the repository at this point in the history
…new format. Assets in the DDS directory are exported at build time into media directory. Upgrade to DirectStorage v1.0.2. Many misc. improvements.
  • Loading branch information
allenhux-intel committed Jun 10, 2022
1 parent 858a5e6 commit 189a5cf
Show file tree
Hide file tree
Showing 56 changed files with 916 additions and 739 deletions.
434 changes: 285 additions & 149 deletions DdsToXet/DdsToXet.cpp

Large diffs are not rendered by default.

15 changes: 15 additions & 0 deletions DdsToXet/DdsToXet.vcxproj
Original file line number Diff line number Diff line change
Expand Up @@ -58,11 +58,16 @@
<SDLCheck>true</SDLCheck>
<PreprocessorDefinitions>_DEBUG;_CONSOLE;%(PreprocessorDefinitions)</PreprocessorDefinitions>
<ConformanceMode>true</ConformanceMode>
<LanguageStandard>stdcpp17</LanguageStandard>
</ClCompile>
<Link>
<SubSystem>Console</SubSystem>
<GenerateDebugInformation>true</GenerateDebugInformation>
</Link>
<PostBuildEvent>
<Command>cd /d $(TargetDir)
convert.bat $(SolutionDir)dds media</Command>
</PostBuildEvent>
</ItemDefinitionGroup>
<ItemDefinitionGroup Condition="'$(Configuration)|$(Platform)'=='Release|x64'">
<ClCompile>
Expand All @@ -72,20 +77,30 @@
<SDLCheck>true</SDLCheck>
<PreprocessorDefinitions>NDEBUG;_CONSOLE;%(PreprocessorDefinitions)</PreprocessorDefinitions>
<ConformanceMode>true</ConformanceMode>
<LanguageStandard>stdcpp17</LanguageStandard>
</ClCompile>
<Link>
<SubSystem>Console</SubSystem>
<EnableCOMDATFolding>true</EnableCOMDATFolding>
<OptimizeReferences>true</OptimizeReferences>
<GenerateDebugInformation>true</GenerateDebugInformation>
</Link>
<PostBuildEvent>
<Command>cd /d $(TargetDir)
convert.bat $(SolutionDir)dds media</Command>
</PostBuildEvent>
</ItemDefinitionGroup>
<ItemGroup>
<ClCompile Include="DdsToXet.cpp" />
</ItemGroup>
<ItemGroup>
<ClInclude Include="..\TileUpdateManager\XetFileHeader.h" />
</ItemGroup>
<ItemGroup>
<CopyFileToFolders Include="..\scripts\convert.bat">
<FileType>Document</FileType>
</CopyFileToFolders>
</ItemGroup>
<Import Project="$(VCTargetsPath)\Microsoft.Cpp.targets" />
<ImportGroup Label="ExtensionTargets" />
</Project>
8 changes: 8 additions & 0 deletions DdsToXet/DdsToXet.vcxproj.filters
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,9 @@
<UniqueIdentifier>{93995380-89BD-4b04-88EB-625FBE52EBFB}</UniqueIdentifier>
<Extensions>h;hh;hpp;hxx;h++;hm;inl;inc;ipp;xsd</Extensions>
</Filter>
<Filter Include="scripts">
<UniqueIdentifier>{e864060a-b20d-4932-b5fe-ebfa3a47eda9}</UniqueIdentifier>
</Filter>
</ItemGroup>
<ItemGroup>
<ClCompile Include="DdsToXet.cpp">
Expand All @@ -20,4 +23,9 @@
<Filter>Header Files</Filter>
</ClInclude>
</ItemGroup>
<ItemGroup>
<CopyFileToFolders Include="..\scripts\convert.bat">
<Filter>scripts</Filter>
</CopyFileToFolders>
</ItemGroup>
</Project>
16 changes: 16 additions & 0 deletions DdsToXet/DdsToXet_vs2022.vcxproj
Original file line number Diff line number Diff line change
Expand Up @@ -58,11 +58,16 @@
<SDLCheck>true</SDLCheck>
<PreprocessorDefinitions>_DEBUG;_CONSOLE;%(PreprocessorDefinitions)</PreprocessorDefinitions>
<ConformanceMode>true</ConformanceMode>
<LanguageStandard>stdcpp17</LanguageStandard>
</ClCompile>
<Link>
<SubSystem>Console</SubSystem>
<GenerateDebugInformation>true</GenerateDebugInformation>
</Link>
<PostBuildEvent>
<Command>cd /d $(TargetDir)
convert.bat $(SolutionDir)dds media</Command>
</PostBuildEvent>
</ItemDefinitionGroup>
<ItemDefinitionGroup Condition="'$(Configuration)|$(Platform)'=='Release|x64'">
<ClCompile>
Expand All @@ -72,19 +77,30 @@
<SDLCheck>true</SDLCheck>
<PreprocessorDefinitions>NDEBUG;_CONSOLE;%(PreprocessorDefinitions)</PreprocessorDefinitions>
<ConformanceMode>true</ConformanceMode>
<LanguageStandard>stdcpp17</LanguageStandard>
</ClCompile>
<Link>
<SubSystem>Console</SubSystem>
<EnableCOMDATFolding>true</EnableCOMDATFolding>
<OptimizeReferences>true</OptimizeReferences>
<GenerateDebugInformation>true</GenerateDebugInformation>
</Link>
<PostBuildEvent>
<Command>cd /d $(TargetDir)
convert.bat $(SolutionDir)dds media</Command>
</PostBuildEvent>
</ItemDefinitionGroup>
<ItemGroup>
<ClCompile Include="DdsToXet.cpp" />
</ItemGroup>
<ItemGroup>
<ClInclude Include="..\TileUpdateManager\XetFileHeader.h" />
<ClInclude Include="XeTv2.h" />
</ItemGroup>
<ItemGroup>
<CopyFileToFolders Include="..\scripts\convert.bat">
<FileType>Document</FileType>
</CopyFileToFolders>
</ItemGroup>
<Import Project="$(VCTargetsPath)\Microsoft.Cpp.targets" />
<ImportGroup Label="ExtensionTargets" />
Expand Down
11 changes: 11 additions & 0 deletions DdsToXet/DdsToXet_vs2022.vcxproj.filters
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,9 @@
<UniqueIdentifier>{93995380-89BD-4b04-88EB-625FBE52EBFB}</UniqueIdentifier>
<Extensions>h;hh;hpp;hxx;h++;hm;inl;inc;ipp;xsd</Extensions>
</Filter>
<Filter Include="scripts">
<UniqueIdentifier>{6d432c77-37cd-43cf-9691-b7787e8ae3f1}</UniqueIdentifier>
</Filter>
</ItemGroup>
<ItemGroup>
<ClCompile Include="DdsToXet.cpp">
Expand All @@ -19,5 +22,13 @@
<ClInclude Include="..\TileUpdateManager\XetFileHeader.h">
<Filter>Header Files</Filter>
</ClInclude>
<ClInclude Include="XeTv2.h">
<Filter>Header Files</Filter>
</ClInclude>
</ItemGroup>
<ItemGroup>
<CopyFileToFolders Include="..\scripts\convert.bat">
<Filter>scripts</Filter>
</CopyFileToFolders>
</ItemGroup>
</Project>
115 changes: 115 additions & 0 deletions DdsToXet/XeTv2.h
Original file line number Diff line number Diff line change
@@ -0,0 +1,115 @@
//*********************************************************
//
// Copyright 2020 Intel Corporation
//
// Permission is hereby granted, free of charge, to any
// person obtaining a copy of this software and associated
// documentation files(the "Software"), to deal in the Software
// without restriction, including without limitation the rights
// to use, copy, modify, merge, publish, distribute, sublicense,
// and/or sell copies of the Software, and to permit persons to
// whom the Software is furnished to do so, subject to the
// following conditions :
// The above copyright notice and this permission notice shall
// be included in all copies or substantial portions of the Software.
//
// THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
// EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
// MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
// NONINFRINGEMENT.IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT
// HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY,
// WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
// OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER
// DEALINGS IN THE SOFTWARE.
//
//*********************************************************

#pragma once

#include "DDS.h"

/*-----------------------------------------------------------------------------
File Layout:
- Header
- Array of Per-tile info: file offset, # bytes, index into metadata array
- Array of Per-metadata blob info: file offset, size of metadata blob
- Metadata blobs, each blob is aligned
- Texture Data, each tile is aligned
- packed mips, treat as not aligned
-----------------------------------------------------------------------------*/
struct XetFileHeaderV2
{
static UINT GetMagic() { return 0x20544558; }
static UINT GetAlignment() { return 4096; }
static UINT GetTileSize() { return 65536; }
static UINT GetVersion() { return 2; }

UINT m_magic{ GetMagic() };
UINT m_version{ GetVersion() };
DirectX::DDS_HEADER m_ddsHeader;
DirectX::DDS_HEADER_DXT10 m_extensionHeader;

UINT m_numMetadataBlobs;

struct MipInfo
{
UINT m_numStandardMips;
UINT m_numPackedMips;
UINT m_numTilesForPackedMips;
UINT m_numTilesForStandardMips; // the number of TileData[] entries after the header
};
MipInfo m_mipInfo;

// use subresource tile dimensions to generate linear tile index
struct StandardMipInfo
{
UINT m_widthTiles;
UINT m_heightTiles;
UINT m_depthTiles;

// convenience value, can be computed from sum of previous subresource dimensions
UINT m_subresourceTileIndex;
};

// if required, compute dimensions by shifting mip 0 dimensions by mip level
struct PackedMipInfo
{
UINT m_rowPitch;
UINT m_slicePitch;
UINT m_fileOffset;
};

struct SubresourceInfo
{
union
{
StandardMipInfo m_standardMipInfo;
PackedMipInfo m_packedMipInfo;
};
};

// indices < m_numStandardMips are standard mips, >= are packed mips
SubresourceInfo m_subresourceInfo[16];

// array TileData[m_numTilesForStandardMips] for each tile
struct TileData
{
UINT m_offset; // file offset to tile data
UINT m_numBytes; // # bytes for the tile
UINT m_metadataIndex; // index of metadata to use
};

// array of MetaData[m_numMetadataBlobs] (may be size 0)
struct MetaData
{
UINT m_offset; // file offset to metadata blob
UINT m_numBytes; // size of the metadata blob
};

// metadata here (aligned)
// tile data here (aligned)

// packed mip data from m_packedMipInfo.m_fileOffset until EOF
};
28 changes: 13 additions & 15 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,9 +6,11 @@ This repository contains an [MIT licensed](LICENSE) demo of _DirectX12 Sampler F

The demo requires ***Windows 10 20H1 (aka May 2020 Update, build 19041)*** or later and a GPU with Sampler Feedback Support, such as Intel Iris Xe Graphics as found in 11th Generation Intel&reg; Core&trade; processors and discrete GPUs (driver version **[30.0.100.9667](https://downloadcenter.intel.com/product/80939/Graphics) or later**).

This repository has been updated with DirectStorage 1.0.0 for Windows&reg; from https://www.nuget.org/packages/Microsoft.Direct3D.DirectStorage/
- Note the legacy streaming code using ReadFile() had sector alignment constraints for the file internals that are not required by DirectStorage for Windows.
- Documentation for accessing nuget packages may be added in the future, but there are many online resources.
This repository has been updated with DirectStorage 1.0.2 for Windows&reg; from https://www.nuget.org/packages/Microsoft.Direct3D.DirectStorage/

Notes:
- The file format has changed since large textures were provided as "releases." See the [log](#log) below.
- The legacy streaming code using ReadFile() had sector alignment constraints for the file internals that are not required by DirectStorage for Windows.

See also:

Expand All @@ -20,9 +22,9 @@ See also:
Textures derived from [Hubble Images](https://www.nasa.gov/mission_pages/hubble/multimedia/index.html), see the [Hubble Copyright](https://hubblesite.org/copyright)


Note the textures shown above, which total over 13GB, are not part of the repo. A few 16k x 16k textures are available as a [release](https://github.com/GameTechDev/SamplerFeedbackStreaming/releases/tag/1)
Note the textures shown above, which total over 13GB, are not part of the repo. A few 16k x 16k textures are available as [release 1](https://github.com/GameTechDev/SamplerFeedbackStreaming/releases/tag/1) and [release 2](https://github.com/GameTechDev/SamplerFeedbackStreaming/releases/tag/2)

Test textures are provided, as is a mechanism to convert from BCx format DDS files into the custom .XET format.
Test textures are provided. At build time, BCx textures (BC7 and BC1 tested) in the dds/ directory are converted into the custom .XET format and placed in the ($TargetDir)/media directory (e.g. x64/Release/media)

## Build Instructions

Expand Down Expand Up @@ -282,17 +284,13 @@ ID3D12CommandList* pCommandLists[] = { commandLists.m_beforeDrawCommands, m_comm
## Log
- 2021-06-21: initial commit
- 2021-07-23: use windows events to reduce cpu overhead
- 2021-08-10: use WaitOnAddress to further reduce cpu overhead. some 16k x 16k textures (BC7 format) posted as "release 1".
- 2021-08-28: proof-of-concept culling: textures for objects behind view are evicted
- 2021-09-20: fixed race condition that could result in hang on exit
- 2021-10-21: code refactor to improve sampler feedback streaming library API
- 2021-12-03: added BC1 asset collection as "release 2." All texture assets (.xet files) can reside in the same directory despite format differences, and can co-exist in the same GPU heap. Also minor source tweaks, including fix to not cull base "terrain" object.
- 2021-12-15: "-addAliasingBarriers" command line option to add an aliasing barrier to assist PIX analysis. Can also be enabled in config.json.
- 2022-01-19: eliminated aliasing barrier perf cost (still requires command line/config to enable). config file parser fixes. terrain texture not applied to planets for prettier out-of-box demo experience.
- 2022-03-14: DirectStorage integrated into mainline
- 2022-06-10: File format (.xet) change. DdsToXet can upgrade old Xet files to the new format. Assets in the DDS directory are exported at build time into media directory. Upgrade to DirectStorage v1.0.2. Many misc. improvements.
- 2022-05-05: Workaround for rare race condition. Many tweaks and improvements.
- 2022-03-14: DirectStorage 1.0.0 integrated into mainline
- 2021-12-15: "-addAliasingBarriers" command line option to add an aliasing barrier to assist PIX analysis. Can also be enabled in config.json.
- 2021-12-03: added BC1 asset collection as "release 2." All texture assets (.xet files) can reside in the same directory despite format differences, and can co-exist in the same GPU heap. Also minor source tweaks, including fix to not cull base "terrain" object.
- 2021-10-21: code refactor to improve sampler feedback streaming library API
- 2021-08-10: Added some 16k x 16k textures (BC7 format) posted as "release 1".
## License
Expand Down
3 changes: 2 additions & 1 deletion SamplerFeedbackStreaming_vs2022.sln
Original file line number Diff line number Diff line change
Expand Up @@ -13,8 +13,9 @@ Project("{8BC9CEB8-8B4A-11D0-8D11-00A0C91BC942}") = "DdsToXet", "DdsToXet\DdsToX
EndProject
Project("{8BC9CEB8-8B4A-11D0-8D11-00A0C91BC942}") = "Expanse", "src\Expanse_vs2022.vcxproj", "{E9BCC846-C1E9-402F-AC99-0E72195202AC}"
ProjectSection(ProjectDependencies) = postProject
{45087328-C272-4BB6-BB09-95D899D2276A} = {45087328-C272-4BB6-BB09-95D899D2276A}
{12A36A45-4A15-48E3-B886-257E81FD57C6} = {12A36A45-4A15-48E3-B886-257E81FD57C6}
{369039E2-4C18-40D9-A7FE-E3D87BA23149} = {369039E2-4C18-40D9-A7FE-E3D87BA23149}
{45087328-C272-4BB6-BB09-95D899D2276A} = {45087328-C272-4BB6-BB09-95D899D2276A}
EndProjectSection
EndProject
Global
Expand Down
Loading

0 comments on commit 189a5cf

Please sign in to comment.