-
Notifications
You must be signed in to change notification settings - Fork 566
[Performance] EWMA SQL Query Plan Section #5147
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: main
Are you sure you want to change the base?
Conversation
- Limiting the total number of entries.
src/Microsoft.Health.Fhir.Core/Features/Search/Expressions/SearchParameterExpression.cs
Dismissed
Show dismissed
Hide dismissed
src/Microsoft.Health.Fhir.Core/Features/Search/Expressions/StringExpression.cs
Dismissed
Show dismissed
Hide dismissed
Undo changes on Search Results.
Adding new logic to handle exceptions, not blocking query execution.
| stopwatch.Stop(); | ||
|
|
||
| if (!string.IsNullOrEmpty(hash)) | ||
| { | ||
| _queryPlanSelector.ReportExecutionTime(hash, reuseQueryCachingPlan, stopwatch.Elapsed.TotalMilliseconds); |
Check notice
Code scanning / CodeQL
Generic catch clause Note
Show autofix suggestion
Hide autofix suggestion
Copilot Autofix
AI about 2 months ago
The best way to fix this issue is to replace the generic catch (Exception exception) clause with one or more specific catch clauses that match the expected exception types thrown by the operations in the try block. Examine the methods used here:
sqlSearchOptions.Expression.GetHashedUniqueExpressionIdentifier()_queryPlanSelector.GetRecommendedQueryPlanCacheSetting(hash)
Assuming these methods are unlikely to throw anything but programming-related or argument exceptions, the most precise approach is to catch exceptions such as ArgumentException, InvalidOperationException, and potentially NullReferenceException or any custom exception the application expects from those functions. In case the expected exceptions are not documented, it's safest (in absence of other project code) to catch ArgumentException and InvalidOperationException.
Only lines 276-280 should be updated. No imports or additional dependencies are required for these exception types.
-
Copy modified line R276 -
Copy modified lines R281-R285
| @@ -273,11 +273,16 @@ | ||
| _logger.LogInformation("Got Query Plan Caching Setting {ReuseQueryCachingPlan} for hash {Hash}.", reuseQueryCachingPlan, hash); | ||
| } | ||
| } | ||
| catch (Exception exception) | ||
| catch (ArgumentException exception) | ||
| { | ||
| // Swallow any exceptions and just run the query with the default setting. | ||
| _logger.LogWarning(exception, "Failed to get Query Plan Caching Setting, running with default."); | ||
| } | ||
| catch (InvalidOperationException exception) | ||
| { | ||
| // Swallow any exceptions and just run the query with the default setting. | ||
| _logger.LogWarning(exception, "Failed to get Query Plan Caching Setting, running with default."); | ||
| } | ||
|
|
||
| Stopwatch stopwatch = Stopwatch.StartNew(); | ||
| SearchResult searchResult = await SearchImpl(sqlSearchOptions, reuseQueryCachingPlan, cancellationToken); |
…elector. Adding more validations validating the generation of hashes.
src/Microsoft.Health.Fhir.Core/Configs/OperationsConfiguration.cs
Outdated
Show resolved
Hide resolved
src/Microsoft.Health.Fhir.Core/Features/Search/Hackathon/QueryPlanSelector.cs
Outdated
Show resolved
Hide resolved
src/Microsoft.Health.Fhir.SqlServer.UnitTests/Features/Search/Expressions/ExpressionTests.cs
Outdated
Show resolved
Hide resolved
src/Microsoft.Health.Fhir.SqlServer/Features/Search/SqlServerSearchService.cs
Outdated
Show resolved
Hide resolved
src/Microsoft.Health.Fhir.Core/Features/Search/Hackathon/QueryPlanSelector.cs
Outdated
Show resolved
Hide resolved
|
When the times comes the new files need to be moved to a folder with a proper name. |
| .GetField("_ewmaScores", System.Reflection.BindingFlags.NonPublic | System.Reflection.BindingFlags.Instance) | ||
| .GetValue(ewma) as System.Collections.Concurrent.ConcurrentDictionary<string, double?>; | ||
|
|
||
| Assert.True(scores.ContainsKey(metric)); |
Check notice
Code scanning / CodeQL
Inefficient use of ContainsKey Note
indexer
Show autofix suggestion
Hide autofix suggestion
Copilot Autofix
AI about 2 months ago
To fix the issue, replace the two separate operations: first checking presence of the key with scores.ContainsKey(metric) (line 34) and then accessing its value with scores[metric] (line 35), with a single call to scores.TryGetValue(metric, out var value). This allows both the test for key presence and for the value to be handled efficiently and more idiomatically. Specifically, within the test, use Assert.True(scores.TryGetValue(metric, out var value)); instead of Assert.True(scores.ContainsKey(metric)); and then check Assert.Null(value); instead of reading scores[metric] again. No new methods or imports are required, only a code change within the relevant lines of EwmaTests.cs.
-
Copy modified lines R34-R35
| @@ -31,8 +31,8 @@ | ||
| .GetField("_ewmaScores", System.Reflection.BindingFlags.NonPublic | System.Reflection.BindingFlags.Instance) | ||
| .GetValue(ewma) as System.Collections.Concurrent.ConcurrentDictionary<string, double?>; | ||
|
|
||
| Assert.True(scores.ContainsKey(metric)); | ||
| Assert.Null(scores[metric]); | ||
| Assert.True(scores.TryGetValue(metric, out var value)); | ||
| Assert.Null(value); | ||
| } | ||
| } | ||
|
|
| .GetField("_ewmaScores", System.Reflection.BindingFlags.NonPublic | System.Reflection.BindingFlags.Instance) | ||
| .GetValue(ewma) as System.Collections.Concurrent.ConcurrentDictionary<string, double?>; | ||
|
|
||
| Assert.True(scores.ContainsKey(metric)); |
Check warning
Code scanning / CodeQL
Dereferenced variable may be null Warning
scores
this
Show autofix suggestion
Hide autofix suggestion
Copilot Autofix
AI about 2 months ago
To ensure that a NullReferenceException does not occur, the assignment resulting from the reflection call must be guarded by a null check. Specifically, after GetField().GetValue(ewma), the result should be checked before dereferencing. If the result is null, the test should fail, indicating the presence of an unexpected issue (e.g., field name/type mismatch, signature change, etc.). The appropriate fix is to add a check before using scores. If scores is null, an informative assertion should fail the test immediately, helping developers diagnose reflection issues. This check should be applied in every block where similar reflection usage occurs—namely, in all three tests (Ewma_Constructor_InitializesScoresToNull, Ewma_Update_UpdatesScoreCorrectly, and any other present if applicable).
-
Copy modified line R34 -
Copy modified line R56
| @@ -31,6 +31,7 @@ | ||
| .GetField("_ewmaScores", System.Reflection.BindingFlags.NonPublic | System.Reflection.BindingFlags.Instance) | ||
| .GetValue(ewma) as System.Collections.Concurrent.ConcurrentDictionary<string, double?>; | ||
|
|
||
| Assert.NotNull(scores); // Fail test if reflection yields null | ||
| Assert.True(scores.ContainsKey(metric)); | ||
| Assert.Null(scores[metric]); | ||
| } | ||
| @@ -52,6 +53,7 @@ | ||
| .GetField("_ewmaScores", System.Reflection.BindingFlags.NonPublic | System.Reflection.BindingFlags.Instance) | ||
| .GetValue(ewma) as System.Collections.Concurrent.ConcurrentDictionary<string, double?>; | ||
|
|
||
| Assert.NotNull(scores); // Fail test if reflection yields null | ||
| Assert.Equal(15.0, scores["A"]); | ||
| } | ||
|
|
| .GetField("_ewmaScores", System.Reflection.BindingFlags.NonPublic | System.Reflection.BindingFlags.Instance) | ||
| .GetValue(ewma) as System.Collections.Concurrent.ConcurrentDictionary<string, double?>; | ||
|
|
||
| Assert.Equal(15.0, scores["A"]); |
Check warning
Code scanning / CodeQL
Dereferenced variable may be null Warning
scores
this
Show autofix suggestion
Hide autofix suggestion
Copilot Autofix
AI about 2 months ago
To fix the problem, we need to ensure that the scores variable is not null before dereferencing it; this means adding a check after the reflection/casting code. The single best fix is to add an assertion that scores != null before accessing scores["A"]. This way, if the reflection fails, the test fails with a clear message rather than throwing a NullReferenceException. Apply this change only to the region where scores is accessed without null checks (lines 54-55 in Ewma_Update_UpdatesScoreCorrectly). No new imports or definitions are required; just an extra Assert.NotNull(scores); before the offending dereference.
-
Copy modified line R55
| @@ -52,6 +52,7 @@ | ||
| .GetField("_ewmaScores", System.Reflection.BindingFlags.NonPublic | System.Reflection.BindingFlags.Instance) | ||
| .GetValue(ewma) as System.Collections.Concurrent.ConcurrentDictionary<string, double?>; | ||
|
|
||
| Assert.NotNull(scores); | ||
| Assert.Equal(15.0, scores["A"]); | ||
| } | ||
|
|
|
/azp run |
|
Azure Pipelines successfully started running 1 pipeline(s). |
| } | ||
|
|
||
| _iterations++; | ||
| metricName = _ewmaScores |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
What does this return if this is reached before any scores are set? For example if GetBestMetric is called enough times so that _interactions is greater than _minInteractionsForDecision, but Update hasn't been called yet.
Description
Describe the changes in this PR.
Related issues
Addresses AB#170732
Testing
Describe how this change was tested.
FHIR Team Checklist
Semver Change (docs)
Patch|Skip|Feature|Breaking (reason)