Qualiware-Azure Data Ingestion

Purpose

The purpose of this document is to explain how to run initial data upload into Azure and synchronize QLM & QEP content to be used by the AI agent.

Please setup/create the properties in Repository Administrator. Launch the Repository Administrator and go to respective repository. Click on properties to add new property.

  1. Azure.Blob.Account. A storage account provides a unique namespace in Azure for your data. Every object that you store in Azure Storage has an address that includes your unique account name. The combination of the account name and the Blob Storage endpoint forms the base address for the objects in your storage account.  If the account name is not set, synchronization will not be done or attempted.
  2. Azure.Blob.Container. A container organizes a set of blobs, similar to a directory in a file system. A storage account can include an unlimited number of containers, and a container can store an unlimited number of blobs.
  3. Azure.Blob.Key. This is the access key of the storage account, and can be copied from the azure portal under storage account->security + networking->Access keys->Key
  4. Azure.Blob.Url. This is the Blob service endpoint and can be copied from the azure portal under storage account->settings->Endpoint->BlobServices
  5. Azure.Blob.IncludeTemplates.  A list of all templates to Include for the Synchronization.  Value is a coma separated list of templates. suggested value for this property is  ‘*’.
  6. Azure.Blob.ExcludeTemplates.  A list of all templates to exclude for the Synchronization.  Any object of those templates will not be visible by the AI engine. Value is a coma separated list of templates.  Add below suggested value for this property.

Suggested Value: 

Recommended exclude templates (copy-ready)


Schemas

Default Schema

Default behavior is to include all the data fields/attributes from objects and upload them into Azure.

There is possibility to setup  the schema to describe and guide what fields should be imported.  Default schema is included which contains audits information, description and short description and the generic relations information (Sends, sends to, frame, frame by…).

 Initial synchronization

After ensuring all above configurations are defined, run the function “SyncAllObjectToAzureBlobServices” from QLM command execution window. Go to Tools->Command Language->Command

SyncAllObjectToAzureBlobServices()

This function triggered from QLM client will take all object in the Base configuration of a repository and format its information based on schemas and push them into Azure blob services as per properties set into repository administrator.  This function can be run only one time or any time a schema is change or when large amount of data is imported.

  Synchronization

If the Azure.Blob.Account is set then each object will be synchronized on save, the content of a diagram will also be synchronized when a diagram is saved to update relationship