Open
Conversation
…eDatasets config The StorageConfiguration.cacheDatasets setting existed but was never wired into the data loading path. Every query re-read FHIR resource data from S3 via Delta Lake. This change applies .cache() to datasets returned by DynamicDeltaSource.read(), gated by the cacheDatasets flag. Cache invalidation is already handled by CacheableDatabase.invalidate() which calls spark.sqlContext().clearCache().
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
A yolo implementation of #2561 :
This pull request introduces support for the
$sqlquery-runFHIR operation, enabling execution of SQL queries defined in FHIR Library resources. It adds new providers, configuration options, and parsing logic to handle SQLQuery Libraries, their parameters, and dependencies. The changes also ensure that the new operation is configurable and properly registered within the server.New SQL Query Operation Support:
Added new providers:
SqlQueryRunProviderandSqlQueryInstanceRunProviderto handle both type-level and instance-level$sqlquery-runoperations for FHIR Library resources, including parameter binding and output formatting. ([[1]](https://github.com/aehrc/pathling/pull/2579/files#diff-7441f4e261837a0c47fe3ba576c977a207bbdc04c5aaadb30c844232af85cdebR177-R184),[[2]](https://github.com/aehrc/pathling/pull/2579/files#diff-7441f4e261837a0c47fe3ba576c977a207bbdc04c5aaadb30c844232af85cdebL233-R248),[[3]](https://github.com/aehrc/pathling/pull/2579/files#diff-7441f4e261837a0c47fe3ba576c977a207bbdc04c5aaadb30c844232af85cdebR277-R278),[[4]](https://github.com/aehrc/pathling/pull/2579/files#diff-5acafa29251cd0573c336431c5e7f133d74246a4f80c9281d3216e3dbc3b83bbR1-R160))Registered the new SQL query run providers within the
FhirServerinitialization logic, gated by a new configuration flag. ([server/src/main/java/au/csiro/pathling/FhirServer.javaR397-R402](https://github.com/aehrc/pathling/pull/2579/files#diff-7441f4e261837a0c47fe3ba576c977a207bbdc04c5aaadb30c844232af85cdebR397-R402))Configuration and Capability Statement:
Introduced a new configuration property
sqlQueryRunEnabledto enable or disable the$sqlquery-runoperation. ([server/src/main/java/au/csiro/pathling/config/OperationConfiguration.javaR73-R75](https://github.com/aehrc/pathling/pull/2579/files#diff-d253d305020a8cff17b5936c099e7e2e7d731120bcc273135f5973795605eaccR73-R75))Updated the conformance/capability statement logic to advertise the
$sqlquery-runoperation when enabled. ([server/src/main/java/au/csiro/pathling/fhir/ConformanceProvider.javaR507-R509](https://github.com/aehrc/pathling/pull/2579/files#diff-dd39c622cbec81a9a667a477be7664401ec1272b7a9d1445a2bd2a5791c8c1e5R507-R509))SQLQuery Library Parsing:
Added
SqlQueryLibraryParser, which parses FHIR Library resources conforming to the SQLQuery profile, extracting SQL text, ViewDefinition dependencies, and parameter declarations. ([server/src/main/java/au/csiro/pathling/operations/sqlquery/SqlQueryLibraryParser.javaR1-R143](https://github.com/aehrc/pathling/pull/2579/files#diff-1f739feb2f4b660e33bb7305adbdbbd1cbb1bbc1e9531ef64010f931193a8495R1-R143))Introduced supporting classes:
ParsedSqlQuery(holds parsed SQL, dependencies, parameters) andSqlParameterDeclaration(represents a named, typed parameter). ([[1]](https://github.com/aehrc/pathling/pull/2579/files#diff-2e4ab2cf0cdde8df9f81de693a4b02979a8b9aad69dc64988fa3690fd1523a63R1-R39),[[2]](https://github.com/aehrc/pathling/pull/2579/files#diff-dcc4e4e7afa0be80e5b6930026fd9a0d0f141902bbe9983bb1b8ca72d636987aR1-R35))