Skip to content

Refactor and update parsing#9

Draft
moofMonkey wants to merge 19 commits intoButterflyStats:masterfrom
moofMonkey:update-parsing
Draft

Refactor and update parsing#9
moofMonkey wants to merge 19 commits intoButterflyStats:masterfrom
moofMonkey:update-parsing

Conversation

@moofMonkey
Copy link
Copy Markdown

@moofMonkey moofMonkey commented Oct 14, 2023

Fixes #7
Fixes #8

  • Fix docker build
  • Fix stringtables parsing
  • Fix deprecation warnings
  • Add support for stringtables using varint_bitcounts
  • Fix fp_PushN
  • Clean up and optimize fieldpath code
  • Implement resource manifest parsing to avoid embedding resources trie in butterfly binary/sources
  • Rework flattened serializers
    • Handle all arrays as if they're dynamic
    • Move property#type to fs_info
    • Avoid storing property paths as strings and introduce new property getter API to reduce RAM usage
    • Construct entities layout inside flattened_serializer instead of using std::vector/std::unordered_map to reduce RAM and CPU usage
  • Fix m_fGameTime logic - m_fGameTime was removed in favor of tick-based time calculation
  • Implement proper entities baselines support
  • Implement stringtables callbacks
  • Implement entities props callbacks nor specify changed paths in on_entity callback?
  • Eliminate BUTTERFLY_THREADSAFE
    • Eliminate global object pools
    • Replace static std::vector with llvm::SmallVector in entity::parse
  • Discuss and reimplement JS/Python APIs using event-driven approach
  • Discuss whether we want to store resource paths as strings or provide API for resource path resolution
  • Discuss clang-format
    Would it be fine if I add Google/LLVM-styled clang-format?
    If so, is SpacesInParentheses: true worth preserving in butterfly code?
  • Fix Windows builds
  • Run unit tests against matches in different game patches?
  • Add support for GitHub CI?
  • Store statically sized arrays without additional allocations?
    This might introduce problems if array size is parsed incorrectly, but could theoretically reduce RAM/CPU usage.

Known problems:

  • Possible crash on exit in case of global butterfly::parser due to global object pools heap use-after-free in entity::~entity
    This affects 03-props, 06-events and 07-combatlog
  • entity_classes class_id should either assert or return UINT32_MAX on lookup failure

Next PRs TODO:

  1. Proper VPK API
  2. KV parsing API

@moofMonkey moofMonkey changed the title Update parsing Refactor and update parsing Oct 14, 2023

private:
uint8_t size_ = 0;
std::array<uint16_t, 7> ar{};
Copy link
Copy Markdown
Author

@moofMonkey moofMonkey Oct 14, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Even though current DEFAULT_MAX_PATH_DEPTH is 10, no parsers currently handle this.
Having only 7 path elements gives us perfect 16 byte fieldpath size/alignment.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

format changes FS out-of-bounds

1 participant