Added support for specifying an extracted type tree data file

Documentation/command-analyze.md

@@ -16,6 +16,7 @@ UnityDataTool analyze <path> [options]
 | `-s, --skip-references` | Skip CRC and reference extraction (faster, smaller DB) | `false` |
 | `-v, --verbose` | Show more information during analysis | `false` |
 | `--no-recurse` | Do not recurse into sub-directories | `false` |
+| `-d, --typetree-data <file>` | Load an external TypeTree data file before processing (Unity 6.5+) | — |
 
 ## Examples
 
@@ -90,7 +91,13 @@ System.ArgumentException: Invalid object id.
 
 This error occurs when SerializedFiles are built without TypeTrees. The command will skip these files and continue.
 
-**Solution:** Enable **ForceAlwaysWriteTypeTrees** in your Unity build settings. See [Unity Content Format](../../Documentation/unity-content-format.md) for details.
+**Solutions:**
+- Enable **ForceAlwaysWriteTypeTrees** in your Unity build settings. See [Unity Content Format](../../Documentation/unity-content-format.md) for details.
+- If your bundles were built with external TypeTree data (Unity 6.5+), use the `--typetree-data` option to load the TypeTree data file before analysis:
+
+```bash
+UnityDataTool analyze /path/to/bundles --typetree-data /path/to/typetree.bin
+```
 
 ### SQL Constraint Errors
 

Documentation/command-dump.md

@@ -15,6 +15,7 @@ UnityDataTool dump <path> [options]
 | `-f, --output-format <format>` | Output format | `text` |
 | `-s, --skip-large-arrays` | Skip dumping large arrays | `false` |
 | `-i, --objectid <id>` | Only dump object with this ID | All objects |
+| `-d, --typetree-data <file>` | Load an external TypeTree data file before processing (Unity 6.5+) | — |
 
 ## Examples
 
@@ -87,6 +88,14 @@ UnityDataTool serialized-file metadata /path/to/file
 
 The `TypeTree Definitions` field will show `No` when TypeTrees are absent.
 
+**External TypeTree data (Unity 6.5+):**
+
+If your bundles were built with TypeTree data extracted to a separate file, use the `--typetree-data` option to load it:
+
+```bash
+UnityDataTool dump /path/to/file.bundle --typetree-data /path/to/typetree.bin
+```
+
 ---
 
 ## Output Format

Documentation/command-serialized-file.md

@@ -305,8 +305,8 @@ Each element of `scriptTypes` in the JSON output contains:
 
 Notes:
 
-* For SerializedFiles inside AssetBundles the Unity Version is frequently stripped ("0.0.0").  See [BuildAssetBundleOptions.AssetBundleStripUnityVersion](https://docs.unity3d.com/ScriptReference/BuildAssetBundleOptions.AssetBundleStripUnityVersion.html).
-* For AssetBundles the version string may take the form "<version>\n<assetbundle-format-version>".  The assetbundle-format-version rarely changes, and is currently 2.
+* For SerializedFiles inside AssetBundles the Unity Version can be stripped, in which case the Unity Version value is "0.0.0".  See [BuildAssetBundleOptions.AssetBundleStripUnityVersion](https://docs.unity3d.com/ScriptReference/BuildAssetBundleOptions.AssetBundleStripUnityVersion.html).
+* For AssetBundles where the typetrees have been stripped out the version string takes the form `<unity-version>\n<assetbundle-format-version>`.  The assetbundle-format-version rarely changes, and is currently 2. Examples "6000.0.65f1\n2" and "0.0.0\n2".
 * The Unity Editor will attempt to load SerializedFiles regardless of the Platform.  But the Runtime will only load files built with the correct platform value.
 
 ---

Documentation/unity-content-format.md

@@ -88,6 +88,14 @@ Player built content, unless the Player was built with TypeTrees enabled.
 >[!TIP]
 >The `binary2text` tool supports an optional argument `-typeinfo` to enable dumping out the TypeTrees in a SerializedFile header.  That is a useful way to learn more about TypeTrees and to see exactly how Unity data is represented in the binary format.
 
+#### Extracted Typetrees
+
+Starting with Unity 6.5 and Addressables 2.9 it is possible to extract the TypeTrees from all the SerializedFiles in an Addressable build into a shared file.  This can reduce the size of the build output, because the TypeTree information is no longer duplicated in each file.
+
+For details see [Addressable AssetBundle memory considerations](https://docs.unity3d.com/Packages/com.unity.addressables@2.9/manual/memory-assetbundles.html)
+
+UnityDataTools commands that open serialized files using the UnityFileSystem API require access to the TypeTrees.  For example `dump` and `analyze`.  Use the `--typetree-data` option to specify the `.typetreedata` file when examining a build that has extracted TypeTrees.
+
 ### Platform details for using UnityDataTool with Player Data
 
 The output structure and file formats for a Unity Player build are quite platform specific.

Documentation/unitydatatool.md

@@ -41,6 +41,22 @@ Use `--help` with any command for details: `UnityDataTool analyze --help`
 
 Use `--version` to print the tool version.
 
+## External TypeTree Data
+
+Starting with Unity 6.5, asset bundles can be built with TypeTree data extracted into a separate file. When bundles are built this way, the TypeTree data file must be loaded before the bundles can be processed.
+
+The `--typetree-data` (`-d`) option is available on the [`analyze`](command-analyze.md) and [`dump`](command-dump.md) commands:
+
+```bash
+# Analyze bundles that use an external TypeTree data file
+UnityDataTool analyze /path/to/bundles --typetree-data /path/to/typetree.bin
+
+# Dump a bundle with external TypeTree data
+UnityDataTool dump /path/to/file.bundle -d /path/to/typetree.bin
+```
+
+> **Note:** This option requires a version of UnityFileSystemApi from Unity 6.5 or newer. Using it with an older version of the library will produce an error message.
+
 
 ## Installation
 

TestCommon/Data/AssetBundleTypeTreeVariations/AssetBundle-NoTypeTree/AssetBundle-NoTypeTree.manifest

@@ -0,0 +1,9 @@
+ManifestFileVersion: 0
+UnityVersion: 6000.0.65f1
+CRC: 2966982943
+HashAppended: 0
+AssetBundleManifest:
+  AssetBundleInfos:
+    Info_0:
+      Name: small.bundle
+      Dependencies: {}

TestCommon/Data/AssetBundleTypeTreeVariations/AssetBundle-NoTypeTree/small.bundle.manifest

@@ -0,0 +1,23 @@
+ManifestFileVersion: 0
+UnityVersion: 6000.0.65f1
+CRC: 1950084276
+Hashes:
+  AssetFileHash:
+    serializedVersion: 2
+    Hash: 85d7a0628d93941ceb548a6dfa4daaa5
+  TypeTreeHash:
+    serializedVersion: 2
+    Hash: 8fe56aa230288d26e5aa9f8e595de7aa
+  IncrementalBuildHash:
+    serializedVersion: 2
+    Hash: 23708f44cf6b8ab8a9a6d6474bc8166a
+HashAppended: 0
+ClassTypes:
+- Class: 114
+  Script: {fileID: 11500000, guid: 1eb29b0714ed25b4b9acbb23c6791a75, type: 3}
+- Class: 115
+  Script: {instanceID: 0}
+SerializeReferenceClassIdentifiers: []
+Assets:
+- Assets/Assets/BasicScriptableObject.asset
+Dependencies: []

TestCommon/Data/AssetBundleTypeTreeVariations/AssetBundle-NoTypeTreeNoVersion/AssetBundle-NoTypeTreeNoVersion.manifest

@@ -0,0 +1,9 @@
+ManifestFileVersion: 0
+UnityVersion: 6000.0.65f1
+CRC: 3103323536
+HashAppended: 0
+AssetBundleManifest:
+  AssetBundleInfos:
+    Info_0:
+      Name: small.bundle
+      Dependencies: {}

TestCommon/Data/AssetBundleTypeTreeVariations/AssetBundle-NoTypeTreeNoVersion/small.bundle.manifest

@@ -0,0 +1,23 @@
+ManifestFileVersion: 0
+UnityVersion: 6000.0.65f1
+CRC: 728949141
+Hashes:
+  AssetFileHash:
+    serializedVersion: 2
+    Hash: 59d745ce3c1a250c02ed018d24740e27
+  TypeTreeHash:
+    serializedVersion: 2
+    Hash: 8fe56aa230288d26e5aa9f8e595de7aa
+  IncrementalBuildHash:
+    serializedVersion: 2
+    Hash: 923313b95e26f50f9389a0be92d4feda
+HashAppended: 0
+ClassTypes:
+- Class: 114
+  Script: {fileID: 11500000, guid: 1eb29b0714ed25b4b9acbb23c6791a75, type: 3}
+- Class: 115
+  Script: {instanceID: 0}
+SerializeReferenceClassIdentifiers: []
+Assets:
+- Assets/Assets/BasicScriptableObject.asset
+Dependencies: []

TestCommon/Data/AssetBundleTypeTreeVariations/README.md

@@ -1,50 +1,72 @@
-This folder contains variations of the typetree representations in the newest SerializedFile formats.
+# AssetBundle TypeTree Variations
 
-* v22 is used in recent versions of Unity
-* v23 is introduced in Unity 6.5
+This folder contains variations of the TypeTree representations in the newest SerializedFile formats.
 
-They are all builds of the same tiny Addressables project.
+- **v22** is used in recent versions of Unity
+- **v23** is introduced in Unity 6.5
 
-# Summary of Content
+## Folder Overview
 
-## packedassets_assets_all.bundle
+| Folder | Format | TypeTree Mode | Build Method | Built With |
+|--------|--------|---------------|--------------|------------|
+| `v22/` | v22 | Inline | Addressables | Unity 6000.0.65f1 |
+| `v23_extracted/` | v23 | Extracted to separate file | Addressables | Unity 6000.6.0a1 |
+| `v23_Inline/` | v23 | Inline | Addressables | Unity 6000.6.0a1 |
+| `AssetBundle-NoTypeTree/` | v22 | Disabled (`DisableWriteTypeTree`) | `BuildPipeline.BuildAssetBundles` | Unity 6000.0.65f1 |
+| `AssetBundle-NoTypeTreeNoVersion/` | v22 | Disabled (`DisableWriteTypeTree` + `AssetBundleStripUnityVersion`) | `BuildPipeline.BuildAssetBundles` | Unity 6000.0.65f1 |
 
-* Main AssetBundle
-* It builds a simple prefab that includes a MonoBehavior (an instance of the MyScripts.Data class)
-* The MonoBehaviour has a SerializedReference field that stores an instance of a scripting class.
+## Addressable Builds (v22, v23_extracted, v23_Inline)
 
-## MonoScript_monoscripts_dde848dc9848681e340a8b4fa9bd7578.bundle
+These three folders are builds of the same tiny Addressables project. They each contain the following files:
 
-* Autogenerated AssetBundle
-* Contains the MonoScript that tracks the MonoBehaviour type.
+### packedassets_assets_all.bundle
 
-## AssetBundle.typetreedata
+- Main AssetBundle.
+- Builds a simple prefab that includes a MonoBehaviour (an instance of the `MyScripts.Data` class).
+- The MonoBehaviour has a `SerializedReference` field that stores an instance of a scripting class.
 
-* An archive file with the typetree info from both bundles.
-* Only present in the v23 build when Addressables is built with Extract Typetrees enabled.
+### MonoScript_monoscripts_dde848dc9848681e340a8b4fa9bd7578.bundle
 
-## prefab_with_serializedreference.serializedfile
+- Auto-generated AssetBundle.
+- Contains the MonoScript that tracks the MonoBehaviour type.
 
-* Serialized file that is inside packedassets_assets_all.bundle
-* Actual name inside AssetBundle is CAB-394ff12e47c27ee4c30d41d2747acd4b
-* It contains 2 GameObjects, 2 Transforms, a MonoBehavior and an AssetBundle (visible using  `UnityDataTool sf objectlist prefab_with_serializedreference.serializedfile`)
-* The TypeTree array has type info for AssetBundle (type 142), Transform (type 4), GameObject (type 1), MonoBehaviour (type 114)
-* Note: the TypeTree for the MonoBehavior is specific to the MyScripts.Data class, e.g. it includes the SerializedReference field MyData, which is not part of other classes derived from MonoBehaviour.  The ScriptID hash is used to distinguish the precise type when there are multiple MonoBehaviours referenced in the same file.
-* The SerializedReference TypeTree array references C# type Assembly: Assembly-CSharp NameSpace: MyScripts Class: Data
+### AssetBundle.typetreedata
 
+- An archive file with the TypeTree info from both bundles.
+- **Only present in `v23_extracted/`** (when Addressables is built with *Extract Typetrees* enabled).
 
-## monoscriptbundle.serializedfile
+### prefab_with_serializedreference.serializedfile
 
-* Serialized file that is inside MonoScript_monoscripts_dde848dc9848681e340a8b4fa9bd7578.bundle
-* Extracted using UnityDataTool archive extract 
-* Actual name inside AssetBundle is CAB-d57a1d89ac0708bf030936c59479c685
+- Serialized file extracted from `packedassets_assets_all.bundle`.
+- Actual name inside AssetBundle: `CAB-394ff12e47c27ee4c30d41d2747acd4b`
+- Contains: 2 GameObjects, 2 Transforms, a MonoBehaviour, and an AssetBundle
+  (visible using `UnityDataTool sf objectlist prefab_with_serializedreference.serializedfile`).
+- The TypeTree array has type info for AssetBundle (142), Transform (4), GameObject (1), MonoBehaviour (114).
+- The TypeTree for the MonoBehaviour is specific to `MyScripts.Data` — it includes the `SerializedReference` field `MyData`, which is not part of other `MonoBehaviour`-derived classes. The `ScriptID` hash distinguishes the precise type when multiple MonoBehaviours are referenced in the same file.
+- The SerializedReference TypeTree array references C# type `Assembly: Assembly-CSharp, NameSpace: MyScripts, Class: Data`.
 
-# Binary2text tips
+### monoscriptbundle.serializedfile
 
-* The `-typeinfo` argument is a way to see the actual contents of the typetrees.
-* To use binary2text with the extracted versions of the serialized files you must pass in AssetBundle.typetreedata using the `typetreefile` argument.
+- Serialized file extracted from `MonoScript_monoscripts_dde848dc9848681e340a8b4fa9bd7578.bundle`.
+- Actual name inside AssetBundle: `CAB-d57a1d89ac0708bf030936c59479c685`
+
+## Built-in AssetBundle Builds (AssetBundle-NoTypeTree, AssetBundle-NoTypeTreeNoVersion)
+
+These are builds made by Unity 6000.0.65f1 of a single small ScriptableObject asset (from the BuildReportInspector package test project). The built-in AssetBundle support was used (`BuildPipeline.BuildAssetBundles`). The archive files have LZMA compression.
+
+Each folder contains `small.bundle` (the AssetBundle) and associated manifest files.
+
+| Folder | Build Flags |
+|--------|-------------|
+| `AssetBundle-NoTypeTree/` | `BuildAssetBundleOptions.DisableWriteTypeTree` |
+| `AssetBundle-NoTypeTreeNoVersion/` | `BuildAssetBundleOptions.DisableWriteTypeTree` + `AssetBundleStripUnityVersion` |
+
+## binary2text Tips
+
+- The `-typeinfo` argument shows the actual contents of the TypeTrees.
+- To use binary2text with the extracted versions of the serialized files, you must pass in `AssetBundle.typetreedata` using the `-typetreefile` argument.
 
 Example:
 ```
-binary2text.exe prefab_with_serializedreference.serializedfile -typetreefile AssetBundle.typetreedata -typeinfo
-```
+binary2text v23_extracted/prefab_with_serializedreference.serializedfile -typetreefile v23_extracted/AssetBundle.typetreedata -typeinfo
+```

UnityDataTool.Tests/ExtractedTypeTreeTests.cs

@@ -0,0 +1,121 @@
+using System;
+using System.IO;
+using System.Threading.Tasks;
+using Microsoft.Data.Sqlite;
+using NUnit.Framework;
+
+namespace UnityDataTools.UnityDataTool.Tests;
+
+#pragma warning disable NUnit2005, NUnit2006
+
+public class ExtractedTypeTreeTests
+{
+    private string m_TestOutputFolder;
+    private string m_DataFolder;
+    private string m_SerializedFile;
+    private string m_SerializedFilePath;
+    private string m_TypeTreeDataFile;
+
+    [OneTimeSetUp]
+    public void OneTimeSetup()
+    {
+        m_TestOutputFolder = Path.Combine(TestContext.CurrentContext.TestDirectory, "test_folder_typetree");
+        Directory.CreateDirectory(m_TestOutputFolder);
+
+        m_DataFolder = Path.Combine(TestContext.CurrentContext.TestDirectory, "Data", "ExtractedTypeTree");
+        m_SerializedFile = "sfwithextractedtypetrees1";
+        m_SerializedFilePath = Path.Combine(m_DataFolder, m_SerializedFile);
+        m_TypeTreeDataFile = Path.Combine(m_DataFolder, "sfwithextractedtypetrees1.typetreedata");
+    }
+
+    [SetUp]
+    public void Setup()
+    {
+        Directory.SetCurrentDirectory(m_TestOutputFolder);
+    }
+
+    [TearDown]
+    public void Teardown()
+    {
+        SqliteConnection.ClearAllPools();
+
+        foreach (var file in new DirectoryInfo(m_TestOutputFolder).EnumerateFiles())
+        {
+            file.Delete();
+        }
+
+        foreach (var dir in new DirectoryInfo(m_TestOutputFolder).EnumerateDirectories())
+        {
+            dir.Delete(true);
+        }
+    }
+
+    [Test]
+    public async Task Analyze_WithTypeTreeData_DatabaseCorrect(
+        [Values("-d", "--typetree-data")] string option)
+    {
+        var databasePath = SQLTestHelper.GetDatabasePath(m_TestOutputFolder);
+
+        Assert.AreEqual(0, await Program.Main(new string[] { "analyze", m_DataFolder, option, m_TypeTreeDataFile }));
+
+        using var db = SQLTestHelper.OpenDatabase(databasePath);
+
+        var objectCount = SQLTestHelper.QueryInt(db, "SELECT COUNT(*) FROM objects");
+        Assert.Greater(objectCount, 0, "Expected objects in database when TypeTree data file is provided");
+    }
+
+    [Test]
+    public async Task Analyze_WithoutTypeTreeData_ReportsFailure()
+    {
+        var databasePath = SQLTestHelper.GetDatabasePath(m_TestOutputFolder);
+
+        using var swOut = new StringWriter();
+        using var swErr = new StringWriter();
+        var currentOut = Console.Out;
+        var currentErr = Console.Error;
+        try
+        {
+            Console.SetOut(swOut);
+            Console.SetError(swErr);
+
+            await Program.Main(new string[] { "analyze", m_DataFolder });
+
+            var output = swOut.ToString() + swErr.ToString();
+
+            Assert.That(output, Does.Contain("Failed files: 1"),
+                "Expected failure when analyzing without TypeTree data file");
+        }
+        finally
+        {
+            Console.SetOut(currentOut);
+            Console.SetError(currentErr);
+        }
+    }
+
+    [Test]
+    public async Task Dump_WithTypeTreeData_Succeeds(
+        [Values("-d", "--typetree-data")] string option)
+    {
+        Assert.AreEqual(0, await Program.Main(new string[] { "dump", m_SerializedFilePath, "-o", m_TestOutputFolder, option, m_TypeTreeDataFile }));
+        var outputFile = Path.Combine(m_TestOutputFolder, m_SerializedFile + ".txt");
+        var txt = File.ReadAllText(outputFile);
+
+        // Confirm that the file contains an expected line (based on the content of this file)
+        Assert.IsTrue(txt.Contains("m_GameObject (PPtr<GameObject>)"));
+
+        File.Delete(outputFile);
+    }
+
+    [Test]
+    public async Task Dump_WithoutTypeTreeData_Fails()
+    {
+        Assert.AreNotEqual(0, await Program.Main(new string[] { "dump", m_SerializedFilePath }));
+    }
+
+    [Test]
+    public async Task TypeTreeData_FileNotFound_ReturnsError()
+    {
+        var result = await Program.Main(new string[] { "analyze", m_DataFolder, "--typetree-data", "nonexistent_file.bin" });
+        Assert.AreNotEqual(0, result, "Expected non-zero return code when TypeTree data file does not exist");
+    }
+}