Docstrings not processed correctly

[lczech]

Hey Sergey,

a small issue that I noticed: Doxygen-style docstrings referencing function parameters via @p or \p annotations (see here) are not processed correctly. The parameter name after that annotation just disappears in the Binder-generated documentation:

/**
 * @brief Test to use the \p value or the @p value annotation.
 */
inline int my_test( int value );

becomes

M("genesis::utils::math").def("my_test", (int (*)(int)) &genesis::utils::math::my_test, "Test to use the  or the  annotation.\n\nC++: genesis::utils::math::my_test(int) --> int", pybind11::arg("value"));

I am not sure if that also affects other doxygen annotations - those are the ones I noticed for now.

Cheers
Lucas

[lyskov]

Hm… Binder only do very little of post-processing of doc strings that LLVM provides (see https://github.com/RosettaCommons/binder/blob/master/source/util.cpp#L333) so I am guessing for some reason LLVM strip some of the info...

[lczech]

Thanks, I see. Asking ChatGPT about this revealed:

The reason you’re losing @​p, \tparam, and other Doxygen tags is that the Clang “comment AST” you’re walking down is already a parsed representation (a FullComment) in which text‐only nodes (e.g. TextComment, ParagraphComment) carry only the literal words, and all of the Doxygen commands have been turned into specialized AST nodes that your get_text simply ignores.

You have two easy fixes:

• Grab the raw comment text (including all tags) straight from Clang’s RawComment API.
• Extend your get_text to handle the various command nodes (ParamCommandComment, InlineCommandComment, BlockCommandComment, etc.).

Using the RawComment API

auto &SM = ast_context.getSourceManager();
if (auto *RC = ast_context.getRawCommentForDeclNoCache(decl)) {
  // This contains the entire /** … */ or ///… text, tags intact
  StringRef raw = RC->getRawText(SM);
  text = raw.str();
  // … then do your escaping, trimming, etc.
}

Extending the walker to include command nodes

#include "clang/AST/Comment.h"
using namespace clang;
using namespace clang::comments;

string get_text(Comment const *C, SourceManager const &SM, SourceLocation &prev) {
  if (auto *TC = dyn_cast<TextComment>(C)) {
    return TC->getText().str();
  }
  else if (auto *PC = dyn_cast<ParamCommandComment>(C)) {
    // e.g. @p foo → "@p foo"
    return "@" + PC->getCommandName().str() + " " + 
           PC->getParamName();
  }
  else if (auto *IC = dyn_cast<InlineCommandComment>(C)) {
    // e.g. \c int → "\c int"
    SmallString<32> buf;
    IC->print(buf, SM, /*traits=*/nullptr);
    return buf.str();
  }
  else if (auto *BC = dyn_cast<BlockCommandComment>(C)) {
    // e.g. \brief → "@brief"
    return "@" + BC->getCommandName().str() + "\n";
  }
  else {
    // your existing ParagraphComment/recursive logic…
  }
}

Hope it's okay to suggest AI slop here - I have not worked with LLVM, so this is the best I can do quickly.

So apparently, the issue affects all kinds of doxygen-style comments, and hence might be worth fixing. Let me know what you think!