[APIView] Deterministic line IDs?

[tjprescott]

Now that we have a tree-based token system, we should be able to identify lines by traversing the tree, instead of requiring parser authors to provide a "deterministic, unique value".

This issue would investigate our ability to generate unique line IDs deterministically using the tree structure.

[smw-ms]

Language	UsesTreeStyleParser	UsesObjectPathAsLineID?	Mental Model	Source of Truth	Example CodeFile
C#	✅ true	Yes	<Namespace>.<Type>.<Member>(<FullyQualifiedParameterTypes>) [Azure.Messaging.EventHubs.EventProcessorClient.EventProcessorClient(Azure.Storage.Blobs.BlobContainerClient, System.String, System.String, Azure.Messaging.EventHubs.EventProcessorClientOptions)]	Complier symbol	https://microsoft-my.sharepoint.com/:u:/p/almend/IQDl4mqKn8Y3TbUnN_H6UqjRASujwxiW6GdIqUjQ6iLFPPo?e=MlPZkC
Python	✅ true	Yes	package.module.Class.member [azure.template.template_main]	API Tree	https://microsoft-my.sharepoint.com/:u:/p/almend/IQDbbyM4yjFjRq80z9l1JH-MAXhURlrrBj8xcdkKsBCihuY
Java	✅ true	No	Some LineIds are like (JAVADOC_LINE_667) others (package-com.azure.resourcemanager.dynatrace.fluent.models others (maven-lineid-name-Microsoft Azure SDK for Dynatrace Management)	---------	https://microsoft-my.sharepoint.com/:u:/p/summerwarren/IQA48uU1WZJ7Rp3Rgmjn09K1AZLIysubZuqUOYAhLa2za60?e=jd6de8
Javascript	✅ true	Yes	<package>/<subpath> ! <symbol> [# <member>] : <kind> [@azure/planetarycomputer/models!TerrainAlgorithm:type]	API Tree	https://microsoft-my.sharepoint.com/:u:/p/almend/IQD1yOeVAv84R5nyWg9LP3vDAaKZ38ZTPyoX55f-U6PvhnY?e=Yl7NKB
Go	✅ true	Mostly Yes but fragile — Go types/fields follow a package/type/member path, but method LineIds embed the receiver’s rendered syntax (including receiver variable name/formatting) rather than a canonical package.Type.Method(signature) path, so method IDs can shift if the printer/parser conventions change. [armnetapp-(a *Account) UnmarshalJSON]	<ModuleImportPath>.<Type>.<Member> [armnetapp/fake.VolumeQuotaRulesServer-Get]	AST symbols (go/ast + go/types)	https://microsoft-my.sharepoint.com/:u:/p/summerwarren/IQCH3y0dNQbXS5G5RbjyjgDxAfQommBuuG2kN2cQ4IS3ftw?e=q9Kg5A
Swift	✅ true	Yes	<Module>.<Type>.<Member>(<ParameterLabel>:<Type>) [AzureCommunicationCalling.VideoOptions.init(localVideoStreams:[LocalVideoStream])]	API Tree	https://microsoft-my.sharepoint.com/:u:/p/summerwarren/IQByTQz-IrGNTY6yS7h-218_AW0VjeiXKPzMbIZSoNjNwCw?e=cd3Zad
Rust	✅ true	Yes	<Crate>::<ModulePath>::<Type>::<Member>(…) [root_mod_azure_storage_blob_external_type_alias_typespec::error::Result]	API Tree	https://microsoft-my.sharepoint.com/:u:/p/summerwarren/IQB2pcYPJ1AvQ4xZIajsNRP8AXT492bqcPVwEcJ6EKPZ4uY?e=rM4mgH
TypeSpec	✅ true	Yes	<Namespace>.<API-node>.<child-node> [Search.facets.@get] [Search.ProductType.PowerBIVisuals.@doc]	API Tree	https://microsoft-my.sharepoint.com/:u:/p/summerwarren/IQAt-nUDAWdnQJhiSRkakb_DAWpjQ4XcrOMULN-RuAtZ_x4?e=IJUi6N
C	❌ false	N/A	N/A	N/A	N/A
C++	❌ false	N/A	N/A	N/A	N/A
Json	❌ false	N/A	N/A	N/A	N/A
Protocol	❌ false	N/A	N/A	N/A	N/A
Swagger	❌ false	N/A	N/A	N/A	N/A
XML	❌ false	N/A	N/A	N/A	N/A

[smw-ms]

Only these fields matter to rebuild LineIds:

• Children → defines the tree (parent → child path)
• Child order (array index) → deterministic fallback
• Tokens[].RenderClasses → tells you what each token is
• Tokens[].Value → the actual names/strings
• RelatedToLine (when present) → anchor to the real node when needed

LineId = parentPath + info derived from Tokens + (RelatedToLine if present)

Example 1 (Java)

  "LineId": "JAVADOC_LINE_3038",
  "Tokens": [ { "IsDocumentation": true } ],
  "Children": [],
  "RelatedToLine": "com.azure.resourcemanager.dynatrace.models.MetricRules.public-MetricRules-withFilteringTags(List<FilteringTag>-filteringTags)"
} 

Process
Detect doc
From Tokens[].IsDocumentation == true

Use RelatedToLine as the anchor
Rebuild/normalize the LineId for:
com.azure.resourcemanager.dynatrace.models.MetricRules.public-MetricRules-withFilteringTags(List-filteringTags)

Append doc path + ordinal
Use tree position (or a running doc index):
/doc/line#3038

Result

/package/com.azure.resourcemanager.dynatrace.models
  /class/MetricRules
  /method/withFilteringTags(List<FilteringTag>)
  /doc/line#3038

In Short
If IsDocumentation and RelatedToLine exist → attach this node under the rebuilt path of RelatedToLine, plus /doc/line#N.

Example 2 (Java)

{
  "LineId": "maven-lineid-description-This package contains Microsoft Azure SDK for Dynatrace Management SDK. For documentation on how to use this package, please see https://aka.ms/azsdk/java/mgmt.  Package tag package-2023-04-27.",
  "Tokens": [
    {
      "Value": "description",
      "RenderClasses": ["keyword", "mavenKey"]
    },
    {
      "Value": ":",
      "RenderClasses": ["punctuation"]
    },
    {
      "Value": "This package contains Microsoft Azure SDK for Dynatrace Management SDK. For documentation on how to use this package, please see https://aka.ms/azsdk/java/mgmt.  Package tag package-2023-04-27.",
      "RenderClasses": ["mavenValue"]
    }
  ],
  "Children": []
}

Result

/maven/description[@value="This package contains Microsoft Azure SDK for Dynatrace Management SDK. For documentation on how to use this package, please see https://aka.ms/azsdk/java/mgmt.  Package tag package-2023-04-27."]

Example 3 (Java)

{
  "LineId": "package-com.azure.resourcemanager.dynatrace.fluent.models",
  "Tokens": [
    { "Value": "package", "RenderClasses": ["keyword"] },
    {
      "Value": "com.azure.resourcemanager.dynatrace.fluent.models",
      "RenderClasses": ["java-package","package","packageName"]
    },
    { "Value": "{", "RenderClasses": ["punctuation"] }
  ],
  "Children": [
    {
      "Tokens": [
        {
          "Value": "@Fluent",
          "RenderClasses": ["typeName","annotationName"]
        }
      ],
      "Children": [],
      "RelatedToLine": "com.azure.resourcemanager.dynatrace.fluent.models.AppServiceInfoInner"
    }
  ]
}

Result

/package/com.azure.resourcemanager.dynatrace.fluent.models

[tjprescott]

What I'd like to see is, given a token file (or any language) that we calculate the following:

1. Total Line IDs (every line in the token file with an elementId)
2. Unique Reachable Line IDs (every line where you can calculate a unique line ID solely from the tokenfile tree structure. No language knowledge or heuristics are allowed)
3. Nonunique Reachable Line IDs (every line where you can calculate a non-unique line ID solely from the tokenfile tree structure)
4. Unreachable Line IDs (basically 1 - 2 - 3 (leftovers))

Step 1 is seeing the percent breakdowns for 2-4 on arbitrary tokenfiles. We'd like to get a sense of the average scores for the tier 1 languages.

[smw-ms]

Language	Total Line IDs	Unique Reachable Line IDs (%)	Non‑unique Reachable Line IDs (%)	Unreachable Line IDs (%)	Notes
C#
Python
Java
JavaScript
Go
Swift
Rust
TypeSpec