Migration using SFM DataImporter API

The preferred method for data migration to SFM is to use the SFM DataImporter API directly from the third party EHR system as opposed to using file export and SFM FileUploader for import.

The SFM DataImporter API is used to import organization data and patient data to SFM. This is the same API as the SFM FileUploader is using.

The organization and pasient files are compressed and encrypted as described here: Export file format and structure. The only difference is that the files does not need to be stored on the file system, but may reside in memory.

When performing a migration, that is running a set of file uploads, the migration has to be given a SessionId (this is automatically done by the SFM FileUploader when that is in use). SessionId should be a unique identifier of type GUID, and it should be part of the meta data in the requests. See example below.

The files are uploaded using the SFM.DataImporterAPI in the folowing sequence:

The organization file
Each pasientfile

Using the API

The SFM.DataImporterAPI uses the TUS protocol for the upload. This is a protocol that is suitable for file uploads, and makes it possible to resume an upload if it should be interrupted. Read more : Resumable file upload protocol v.1.1

The TUS protocol is built on HTTP for "resumable file uploads". This means that the file upload can be interrupted and resumed at any time. The interruption itself can be user-controlled by the user deciding to pause upload, or by accident, e.g. communication problems or server downtime.

When uploading a file, the client sends a POST request to the SFM.DataImporterAPI to initiate the upload. If the API approves the upload, a positive response is returned with 201 Created and upload URL in location. Upload URL is a unique identifier for this upload. The API accept parallel calls, but no more than 5-10 parallel calls should be posted at a time.

Example (bearer token shortened):

    POST /files/ HTTP/1.1
    Authorization: Bearer
    eyJhbGciOiJSUzI1NiIsImtpZCI6IkI0Q0FFNDUyQzhCNkE4OTNCNkE4NDBBQzhDODRGQjA3MEE
    0MjZFNDEiLCJ4NXQiOiJ0TXJrVXNpMnFKTzJxRUNzaklUN0J3cENia0UiLCJ0eXAiOiJKV1QifQ
    Tus-Resumable: 1.0.0
    Upload-Length: 964736
    Upload-Metadata: name
    ZDE0ZDZkZmItNDg0Yi00ZGM0LTgyNzItMWNkODVjNjhmM2Y5LnBhdA==,contentType
    YXBwbGljYXRpb24vb2N0ZXQtc3RyZWFt,FileType UGF0aWVudEZpbGU=,SessionId
    M2E2ODM2NWYtNGRlYy00ZWNmLWIzZTctMDRiZWRiZGQxMDk2
    Host: 127.0.0.1:8080
    Content-Length: 0
    Accept-Encoding: gzip, deflate
    Connection: close

The POST request contains metadata such as:

file name: d14d6dfb-484b-4dc4-8272-1cd85c68f3f9.pat,
contentType: application / octet-stream
FileType: PatientFile

Response:

    HTTP/1.1 201 Created
    Date: Wed, 13 Oct 2021 13:12:00 GMT
    Content-Length: 0
    Connection: close
    Location: /files/6242e436005b45818557f851beed0d78
    Upload-Expires: Wed, 13 Oct 2021 18:12:00 GMT
    Tus-Resumable: 1.0.0
    Strict-Transport-Security: max-age=15724800; includeSubDomains

Response contains the upload URL in the Location header.

Then follows a HEAD request to check if the server has already received data: Request:

    HEAD /files/6242e436005b45818557f851beed0d78 HTTP/1.1
    Authorization: Bearer
    eyJhbGciOiJSUzI1NiIsImtpZCI6IkI0Q0FFNDUyQzhCNkE4OTNCNkE4NDBBQzhDODRGQjA3MEE
    0MjZFNDEiLCJ4NXQiOiJ0TXJrVXNpMnFKTzJxRUNzaklUN0J3cENia0UiLCJ0eXAiOiJKV1QifQ
    F3ErIwddDrSs7DWm3Pug
    Tus-Resumable: 1.0.0
    Host: 127.0.0.1:8080
    Accept-Encoding: gzip, deflate
    Connection: close

Response:

    HEAD /files/6242e436005b45818557f851beed0d78 HTTP/1.1
    Authorization: Bearer
    eyJhbGciOiJSUzI1NiIsImtpZCI6IkI0Q0FFNDUyQzhCNkE4OTNCNkE4NDBBQzhDODRGQjA3MEE
    0MjZFNDEiLCJ4NXQiOiJ0TXJrVXNpMnFKTzJxRUNzaklUN0J3cENia0UiLCJ0eXAiOiJKV1QifQ
    F3ErIwddDrSs7DWm3Pug
    Tus-Resumable: 1.0.0
    Host: 127.0.0.1:8080
    Accept-Encoding: gzip, deflate
    Connection: close

Upload-Offset contains 0 which means that the server has not received data for this file before.

Then follows the actual file upload in a PATCH request:

    PATCH /files/6242e436005b45818557f851beed0d78 HTTP/1.1
    Tus-Resumable: 1.0.0
    Upload-Offset: 0
    Upload-Checksum: sha1 zxQHcWM8gI34UX+3KS2MUNJe/lQ=
    Content-Type: application/offset+octet-stream
    Host: dataimport.qa.forskrivning.no
    Content-Length: 524288
    Accept-Encoding: gzip, deflate
    Connection: close

    [data]

After successful upload, the client receives the following response:

    HTTP/1.1 204 No Content
    Date: Wed, 13 Oct 2021 13:12:57 GMT
    Connection: close
    Tus-Resumable: 1.0.0
    Upload-Offset: 524288
    Upload-Expires: Wed, 13 Oct 2021 18:12:00 GMT
    Strict-Transport-Security: max-age=15724800; includeSubDomains