Major performance optimizations for HFS+ DMG extraction - #76
Open
statico wants to merge 6 commits into
Open
Conversation
- dmg.go: Add LZMA compression support, fix partition offset handling - hfsplus.go: Rewrite Files() for efficiency, add infinite loop protection - types.go: Fix multi-extent file reading, fix String() recursion 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Added LRU cache field to Partition struct - Implemented initCache() for lazy cache initialization - Added findChunkIndex() for binary search chunk lookup - Rewrote Partition.ReadAt to use caching and binary search - Fixed DMG.ReadAt cache bug (cache.Add was in wrong branch) - Added benchmark tests for performance measurement
- Added sync.Pool for bytes.Buffer output buffers - Added sized input buffer pools (64K, 128K, 256K, 512K, 1M) - Pre-size output buffers to avoid growth allocations - Memory for full extraction: 64MB → 19MB (3.3x reduction)
- Removed log.Debugf in B-tree traversal (caused allocations) - Added bytes.Reader pool for zlib decompression - Allocations dropped from 72K to 51K
- readBTRecordAt: direct uint16 reads for keyLength and recordType - readBTreeNodeAtOffset: direct byte buffer read for record offsets - Allocations: 51K → 47K
- Use ReadAt for keyLength and recordType instead of SectionReader with seeks - Cleaner code path, same performance
Owner
|
Sorry I'm taking so long to respond 😩 this is a great PR; but with AI stuff it's hard to quickly review because it so much code, but I'm certain it's better code than I could write myself, but being the owner I owe it to my users to thoroughly review it. Just letting you know I've seen this, I think it looks great, but in need some time to real read it. |
Owner
|
please resolve merge conflicts |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Important
This is stacked on #75
Authored with Claude Code, this PR dramatically improves the performance of HFS+ filesystem extraction from DMG files. An internal operation has gone from taking 2+ minutes to 1.83s with these changes.
Changes
Partition.ReadAtto avoid re-decompressing the same chunksDMG.ReadAtwhere cache entries were never stored (cache.Add was in wrong branch)sync.Pool) for decompression buffers to reduce GC pressureencoding/binary.Readwith direct byte parsing in performance-critical pathsPerformance Improvements
Root Cause
The primary issue was
Partition.ReadAt(used by HFS+ code) had no caching - every read of a B-tree node would re-decompress the same DMG chunks repeatedly. A typical HFS+ mount would decompress the same blocks thousands of times, causing massive memory churn and 46GB of allocations for a 46MB DMG.Testing
All existing tests pass. Added new benchmarks using Fork.dmg to validate performance.