Application Insights

An overview of how Application Insights are used in the validation pipeline.

Overview

Application Insights is an Azure service used to collect logs. It is used in the pipeline to collect the logs of each service and allow users to track the status of jobs by viewing and querying logs.

Storing Logs

The first requirement is an Application Insights instance on Azure. This service generates a connection string which can be added to each application. In Python applications such as those in the validation pipeline, logs are pushed using a custom logger. By adding the connection string to the custom logger, the logs are automatically sent to Azure using the AzureLogHandler library.

#setup logger
logging.basicConfig()
logger = logging.getLogger("azure-logger")
logger.setLevel(logging.INFO)

connect_str = os.getenv('APPLICATION_INSIGHTS_CONNECTION_STRING')
handler = AzureLogHandler(connection_string=connect_str)
handler.setLevel(logging.INFO)
logger.addHandler(handler)

Viewing Logs

Once logs are sent to Azure, the next step is the ability to view them. Each Application Insights instance can be viewed in Azure. There are many functionalities, but the most important here are the logs, and more specifically the traces category.

Incoming logs are sorted under this category. This is an example of a log instance:

Logs have an added field called customDimensions. These are used in the application to define custom fields. In doing this, it is possible to attach the job UUID to the log.

self.properties = {'custom_dimensions': {'UUID': container_name}}
self.logger.info('This is a log!', extra=self.properties)

Another possible use of this service is Alerts. With alerts it is possible to set up automatic notifications when a service goes down or a job fails.

Querying Logs

Based on the aforementioned customDimensions, it is possible to create a query to get all logs that contain that specific UUID:

traces 
| extend uuid = tostring(customDimensions["UUID"])
| where uuid == "338de6f55b"

The above query gets all logs where the UUID matches "338de6f55b". In Azure, this gives the following result: