Ingest clinical data using the FHIR service
With the healthcare data solutions in Microsoft Fabric, you can ingest data directly from your Fast Healthcare Interoperability Resources (FHIR) server by using the export process. You can configure the export process to make your FHIR data ready for ingestion. By deploying the Azure Health Data Services – Data Export capability, and by deploying the associated prebuilt Microsoft Azure components, you can import data that's sourced from an Azure Health Data Services - FHIR service into your bronze lakehouse in its raw NDJSON format.
This capability uses the FHIR $export API to efficiently export the FHIR data and store it on an Azure Data Storage Gen2 container on a recurring basis. As part of the configuration of this capability, you can configure a Fabric shortcut to the export folder as you did when using the bring-your-own-storage ingestion approach. After the data exports from the FHIR server, the system picks up the data and ingests it into the bronze lakehouse. Then, it flattens the data into the silver lakehouse.
To use the FHIR data ingestion capability, you must complete the following steps:
Have a working Azure Healthcare Data Services FHIR server.
Deploy the healthcare data solutions in Microsoft Fabric Azure Marketplace offer.
Configure the FHIR Server export destination.
Update the global configuration in the Admin lakehouse with information about your deployed Azure components.
Create a shortcut in the external folder that points to your Azure Data Storage container export.
To deploy the prebuilt companion Azure components, you need to deploy the Healthcare data solutions in Microsoft Fabric Azure Marketplace offer.
The following screenshot shows the installed Azure components in the resource group that you specify in your Azure subscription.
The system uses these components when you implement the FHIR export service notebook to help facilitate the use of the $export API from your FHIR server. You can configure permissions on your FHIR server and a few of these components to allow interactions during the export. Additionally, you can set the storage account on the FHIR server that it uses for export to point to the storage account that you provisioned with the Azure offer. In the following unit, you go through these steps in detail.
As part of the Healthcare data foundations solution that you deployed, it contains an Admin lakehouse. This lakehouse contains the global configuration JSON files, and you must update it with the name of the Azure Key Vault that deployed as part of the Azure offer.
The system deploys the msft_clinical_ahds_fhirservice_export pipeline as part of the solution deployment. This pipeline runs the following notebooks.
The first notebook ahds_fhirservice_export is new and is specific to exporting data from the FHIR service. The other two notebooks are the ones that you previously used in the other pipelines for ingesting data from the bring-your-own-source or bring-your-own-storage approach.
When you run the ahds_fhirservice_export notebook in the export pipeline, the following tasks run:
Get access from Key Vault information to trigger the Azure Function that's installed as part of the Azure offer.
Use the function key that's retrieved from the Key Vault to trigger the Azure function.
Use the function to request bulk export from the FHIR server by using the URL for the FHIR server from the global configuration.
Use the function to poll the FHIR server to determine if the bulk export is complete. If the function isn't complete, the server retries until it receives a success status or if the max_polling_days is reached.
After the bulk export from the FHIR server completes, you can proceed to run the bronze ingest notebook followed by the silver flatten notebook. Essentially, after the export, the process is the same as other approaches to ingest data.
Make sure that you consider the following security recommendations after deploying the healthcare data solutions that the Azure Marketplace offers:
Turn off public access to Azure Key Vault.
Follow the best practices for ensuring the Rotation of Function App Key.
Use managed identities for OneLake private endpoints (instead of shared access signature).